Quotes on the site cover how much of the Harry Potter books?

Question

Answers and questions on SciFi Stack Exchange often contain excerpts (quotes, quotations) from the Harry Potter novels. We have over a thousand questions. Could you estimate how much of the novels these excerpts cover? How long till the full text of the novel (although with many typos) can be reconstructed from just these excerpts?

Answer 1

Work in progress/we have a long way to go

I’m working on a little script which can get a decent estimate. When/if it’s done, I’ll post the results here, and I’ll put the code/an explanation on my website.

Right now I have:

• the complete HTML for every question and answer posted in the Harry Potter tag
• a stripped out version that only includes quoted text (anything in a <blockquote> , including spoiler text)
• the quotes broken down into distinct sentences, which I can start comparing to the books

There are ~165,000 words in the quotes, but that includes comments, citations, and interviews/supplementary material which isn’t in the main books.

For comparison, the official word counts for the books (via the HP Lexicon pages) is:

Novel	Word count
Stone	76,944
Chamber	85,141
Prisoner	107,253
Goblet	190,637
Order	257,045
Prince	168,923
Hallows	198,227
TOTAL	1,084,170

So there’s an upper bound of ~15% coverage, but it’s likely to be much less than that. I’m not a betting man, but I’d guess that once I subtract interviews, Pottermore and Beasts/Quidditch, then we’ll be left with about 5% of the book text.

For the curious, I’m writing in Python, with PyStackExchange to get the questions, NLTK to break the quotes into sentences, and fancy hot chocolate to keep me awake. Next up is trying to get good copies of the books in text files, and using difflib to start matching up lines.

I don’t have any more time to devote to this today, and I have important exams coming up, so I might not be able to work on this for a while, but there you are.

Answer 2

You can get the quarterly data dump and process that. However, you get XML files, so ugh.

Instead, you can do this via the API (e.g. start with the questions tagged [harry-potter]) to get JSON and as a bonus get live information. To get you started:

import json
import gzip
import urllib2
import StringIO

def read_url(url):
    req = urllib2.Request("http://api.stackexchange.com%s" % url, headers={"Accept-Encoding": "gzip"})
    u = urllib2.urlopen(req)
    content = []
    while True:
        block = u.read(49152)
        if not block:
            break
        content.append(block)
    return json.loads(gzip.GzipFile(fileobj=StringIO.StringIO("".join(content))).read())

# Get the questions that are tagged [harry-potter]
matching_questions = []
while True:
    data = read_url("/2.2/search/advanced?order=desc&sort=activity&tagged=harry-potter&site=scifi")
    for question in data["items"]:
        matching_questions.append(question["question_id"])
    if not data["has_more"]:
        break
    # Note that without an API key, you'll probably run out here, because
    # there are a lot of questions.
    print "Queries left:", data["quota_remaining"]

# Get the bodies of the questions.
quotes = set()
for q_start in xrange(len(matching_questions) // 100):
    data = read_url("/2.2/questions/%s?order=desc&sort=activity&site=scifi&filter=!3tz1lVh9XyfTtrm_D" % ";".join(matching_questions[q_start*100:(q_start+1)*100]))
    content = data["items"]["body_markdown"]
    # Crudely assume that any line starting with > is a quote.
    # Even more widely, assume that any quote is a quote from the text.
    # The only feasible way to avoid this (without a human reviewing
    # them all) is to have the complete text available.
    quote = None
    for line in content.splitlines():
        if line.startswith(">"):
            if quote:
                # A multi-line quote.
                quote.append(line[4:])
            else:
                quote = [line[4:]]
        elif quote:
            # The quote has finished.
            quotes.add(" ".join(quote))
            quote = None

At this point, the quotes set has a very rough set of all of the quotes in questions. You'd need to do this for answers as well (pretty much the same, just a different endpoint) and I suppose comments if anyone has ever shoved a quotation in there.

You then have the problems over overlapping quotations and inconsistent quotations (e.g. differences in case, punctuation, typos). A rough way to solve the first problem would be to break all quotes into sentences and assume that complete sentences are always quoted. For the second problem you could have a method that normalises each quote in some way, and remove duplicates that way. I leave this as an exercise for the reader :)

You can then quite easily get a word count for all the quotations, and find a word count for the novels (e.g. if you trust Yahoo! (I wouldn't), or you could ask here). That would give you your (extremely rough) estimate for how much is covered, and you can then do some data explorer exploration to guess how rapidly the number of [harry-potter] questions is growing (and, since we are approximating things everywhere, assume that the ratio of questions to unique quoted content remains constant), and you can get an estimate for 100% reproduction from there.

Apart from all of the roughness of the above, the main flaw is that identified by @simon above - the ratio of questions to unique quoted content will not remain constant, because (I have no data to back this up, but it seems entirely logical) it becomes increasingly unlikely that a new question/answer will use material that hasn't been quoted before, because the unquoted content gets less and less interesting. I suppose in a theoretical year when there are a near infinite number of HP questions, complete coverage would be reached, but the real answer to the question is never.

(You could use something like the above to guesstimate when we would reach some lower number, like 10%, though. It seems reasonable that the site could get to a low number like that).