8

Answers and questions on SciFi Stack Exchange often contain excerpts (quotes, quotations) from the Harry Potter novels. We have over a thousand questions. Could you estimate how much of the novels these excerpts cover? How long till the full text of the novel (although with many typos) can be reconstructed from just these excerpts?

  • 6
    I don't think the full test of any of the novels will ever be reconstructed. There will always be certain parts that are just useless as quotes. – CandiedMango May 2 '14 at 9:41
  • 4
    We Potter fans can't possibly be expected to know our math(s) well enough to be able to answer this question! Ten points from Gryffindor! (I don't know if you're a Gryffindor, but as a Slytherin, I'm obligated to take points from Gryffindor under all circumstances, fair or not.) :) – Slytherincess May 2 '14 at 9:56
  • 4
    @Slytherincess - you're not allowed to take points off on Meta. – DVK-on-Ahch-To May 2 '14 at 17:05
  • 2
    @Slytherincess - also, no Slytherin members are allowed to take points. Ever. – DVK-on-Ahch-To May 2 '14 at 17:05
  • 3
    @DVK - A "certain disregard for the rules" prompts me to take 10 POINTS FROM RAVENCLAW!!!! – Slytherincess May 2 '14 at 23:13
  • 1
    When can we get a real point system for Harry Potter fans and different houses on Science Fiction & Fantasy? – user3459110 May 15 '14 at 13:07
11

Work in progress/we have a long way to go

I’m working on a little script which can get a decent estimate. When/if it’s done, I’ll post the results here, and I’ll put the code/an explanation on my website.

Right now I have:

  • the complete HTML for every question and answer posted in the Harry Potter tag
  • a stripped out version that only includes quoted text (anything in a <blockquote> , including spoiler text)
  • the quotes broken down into distinct sentences, which I can start comparing to the books

There are ~165,000 words in the quotes, but that includes comments, citations, and interviews/supplementary material which isn’t in the main books.

For comparison, the official word counts for the books (via the HP Lexicon pages) is:

Stone    |    76,944
Chamber  |    85,141
Prisoner |   107,253
Goblet   |   190,637
Order    |   257,045
Prince   |   168,923
Hallows  |   198,227
---------+---------
TOTAL    | 1,084,170

So there’s an upper bound of ~15% coverage, but it’s likely to be much less than that. I’m not a betting man, but I’d guess that once I subtract interviews, Pottermore and Beasts/Quidditch, then we’ll be left with about 5% of the book text.

For the curious, I’m writing in Python, with PyStackExchange to get the questions, NLTK to break the quotes into sentences, and fancy hot chocolate to keep me awake. Next up is trying to get good copies of the books in text files, and using difflib to start matching up lines.

I don’t have any more time to devote to this today, and I have important exams coming up, so I might not be able to work on this for a while, but there you are.

  • Wow, I didn't know I'd inspire you for this. I'll keep posting more quotes in future answers then, to bring the coverage up. – b_jonas May 4 '14 at 13:57
  • 1
    -1 for hot chocolate. You should obey stereotype and do coffee (or red bull or whatever that travesty is called) – DVK-on-Ahch-To May 4 '14 at 17:02
  • There are dupe qoutes as well mister. And I can see, most of them are dupes... – user3459110 May 15 '14 at 13:06
  • @AwalGarg He did mention that he will get the distinct sentences. – Möoz Apr 21 '16 at 22:40
  • 1
    Those are some very long exams you’ve been doing. Also, may I suggest not writing your code in hot chocolate? It tends to dissipate and melt quicker than you can compile and run it. :-þ – Janus Bahs Jacquet Apr 24 '16 at 8:37
6

You can get the quarterly data dump and process that. However, you get XML files, so ugh.

Instead, you can do this via the API (e.g. start with the questions tagged [harry-potter]) to get JSON and as a bonus get live information. To get you started:

import json
import gzip
import urllib2
import StringIO

def read_url(url):
    req = urllib2.Request("http://api.stackexchange.com%s" % url, headers={"Accept-Encoding": "gzip"})
    u = urllib2.urlopen(req)
    content = []
    while True:
        block = u.read(49152)
        if not block:
            break
        content.append(block)
    return json.loads(gzip.GzipFile(fileobj=StringIO.StringIO("".join(content))).read())

# Get the questions that are tagged [harry-potter]
matching_questions = []
while True:
    data = read_url("/2.2/search/advanced?order=desc&sort=activity&tagged=harry-potter&site=scifi")
    for question in data["items"]:
        matching_questions.append(question["question_id"])
    if not data["has_more"]:
        break
    # Note that without an API key, you'll probably run out here, because
    # there are a lot of questions.
    print "Queries left:", data["quota_remaining"]

# Get the bodies of the questions.
quotes = set()
for q_start in xrange(len(matching_questions) // 100):
    data = read_url("/2.2/questions/%s?order=desc&sort=activity&site=scifi&filter=!3tz1lVh9XyfTtrm_D" % ";".join(matching_questions[q_start*100:(q_start+1)*100]))
    content = data["items"]["body_markdown"]
    # Crudely assume that any line starting with > is a quote.
    # Even more widely, assume that any quote is a quote from the text.
    # The only feasible way to avoid this (without a human reviewing
    # them all) is to have the complete text available.
    quote = None
    for line in content.splitlines():
        if line.startswith("&gt;"):
            if quote:
                # A multi-line quote.
                quote.append(line[4:])
            else:
                quote = [line[4:]]
        elif quote:
            # The quote has finished.
            quotes.add(" ".join(quote))
            quote = None

At this point, the quotes set has a very rough set of all of the quotes in questions. You'd need to do this for answers as well (pretty much the same, just a different endpoint) and I suppose comments if anyone has ever shoved a quotation in there.

You then have the problems over overlapping quotations and inconsistent quotations (e.g. differences in case, punctuation, typos). A rough way to solve the first problem would be to break all quotes into sentences and assume that complete sentences are always quoted. For the second problem you could have a method that normalises each quote in some way, and remove duplicates that way. I leave this as an exercise for the reader :)

You can then quite easily get a word count for all the quotations, and find a word count for the novels (e.g. if you trust Yahoo! (I wouldn't), or you could ask here). That would give you your (extremely rough) estimate for how much is covered, and you can then do some data explorer exploration to guess how rapidly the number of [harry-potter] questions is growing (and, since we are approximating things everywhere, assume that the ratio of questions to unique quoted content remains constant), and you can get an estimate for 100% reproduction from there.

Apart from all of the roughness of the above, the main flaw is that identified by @simon above - the ratio of questions to unique quoted content will not remain constant, because (I have no data to back this up, but it seems entirely logical) it becomes increasingly unlikely that a new question/answer will use material that hasn't been quoted before, because the unquoted content gets less and less interesting. I suppose in a theoretical year when there are a near infinite number of HP questions, complete coverage would be reached, but the real answer to the question is never.

(You could use something like the above to guesstimate when we would reach some lower number, like 10%, though. It seems reasonable that the site could get to a low number like that).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .