Is it good that Story Identification questions are so well rewarded?

Question

NOTE: Due to the fact that people seem to be completely mis-reading this question, I'll quote the last paragraph up front:

Just to be clear, I'm not against the concept of story identification questions, and some (very few) of them are quite great.

TL;DR version: "My personal evaluation of intrinsic worth of story-identification questions seems wildly out of sync with the aggregate community based on the # of upvotes they receive. Why?

---

Now, for the full original question:

I am genuinely puzzled by the fact that story-identification questions are so well rewarded. Why? Is it good?

---

First, a couple of stats. As of now, the site has close to 300 of questions tagged story-identification - out of 2000 total. Some of them seem to be mis-tagged (and closed), so let's say 250.

Of those:

• The 50 highest voted ones have 11+ upvotes each

• The second 50 highest voted are 8-11 upvotes each.

• The third 50 highest voted are 6-8 upvotes each.

• 2 highest voted ones are 30 and 28 upvotes (admittedly, 28 upvotes one is a great one that even I up-voted).

• Of the latest 50 tags that I analyzed, the average was 6.7 up-votes, with ~3 standard deviation. Here's the histogram (X-axys is the question's # of up-votes, Y-axis is # of questions with so many up-votes):

  

---

My questions are:

• Somewhat rhethorically: why are those story-identification questions so much up-voted? Are they really THAT useful to so many people or so brilliantly written? I don't know the stats for the entire site but i'd not be surprised if site-wide upvote average is lower than 6.7.

  Personally, I feel that those questions, while perfectly OK to be on-topic, are not a big win for the site and should not be stongly encouraged by lots of (in my humble opinion) undeserved up-votes. If I posted an identification question - which I may have but don't recall - I would honestly expect at most 1 upvote if it was about some obscure work with no great interest as far as overall SciFi field.

• Does the community feel that these story-identification questions are in some way, as a class, helpful to the site? Do the drive many visitors? Do they bring in some special nuggets of knowledge that nobody knew about yet many people would find useful?

Please note that I specifically refer to upvotes on the questions themselves - I don't feel that the ones on the answers are in any way un-deserved.

---

Just to be clear, I'm not against the concept of story identification questions, and some (very few) of them are quite great, either due to the question's innate quality, or a really interesting take on a well known work, or just genuinely interesting (e.g. this). But a vast majority that I see are nowhere near special.

Answer 1

Given that you said in chat

I know I'm repeating myself here, but my question wasn't "are they good" but "I don't get what all the upvoters see in seemingly - IMHO - useless questions". I explicitly said that I don't mind them being asked at all, and some are pretty good.

First, I've seen a lot more upvoting on scifi.SE than some of the other stackexchange sites I've gone to, this is mostly speculation, but I've had a lot more upvotes per answer here than anywhere else. If this fact is true, it accounts for the next part of my response.

I agree with dmckee. The questions are highly accessible, almost everyone can look at them and given that a lot of people seem to upvote questions and answers, a lot of people are going to upvote the question because hey, they know how it feels to half remember a story and they want to help out the person asking the question but they can't answer it so it gets an upvote.

The thing is, this probably happens with "normal" questions, too, but people usually only look at questions they feel they might answer (I am likely to look at a question concerning Doctor Who, but I am not likely to look at a question concerning comic books). Since people often look at questions based on their personal fandom, and story-identification questions are something that everyone could possibly answer, and look at, it causes the story ID questions to have a much higher rate of views and votes.

Personally, a lot of story ID questions seem to come from new users, and I tend to upvote new users more, because they're new, and upvoting new users is a good way to get them to come back to the site. However, if the question doesn't really provide any details beyond "it was a book about time travel" or something, then I'm likely to comment on the question, asking for more detail (or upvote a comment that's already saying the same thing)

Actual Statistical Data Edit I decided to do some calculations (or write a program to do the calculations for me...), using the StackExchange /search/ API. Interestingly, these do not back up my theory. You can look at the stats from the API in this webpage (view source if you wish, these stats are generated from data from the API). I decided to use the top 4 tags (story-identification, star-trek, star-wars, harry-potter), and ended up with 100 questions from each (side effect of the search API).

Conclusion: Average score for story-identification questions were slightly lower than harry-potter questions, and far, far lower than star-trek and star-wars questions. Average views were also far lower (by about 200, at minimum). The highest score received by a story-identification question is 30, the other tags outrank that except for harry-potter, which has a high score question of 25. Highest view count for all other tags are significantly higher, and in terms of lowest views, story-identification only beats out star-wars (side note: with a limit of 30 questions, story-identification questions won with a 67 lowest view count, beating out the second lowest question from Harry Potter, with 60)

The end result: these stats do not actually support the idea that story-identification questions are somehow higher rated, or even higher viewed than other questions. The only thing I do know is that there are more of them (273 questions at this point, with the next highest tag of star-trek having 238 questions).

Answer 2

Speaking as a moderator

Right this minute, 5% of all story-identification questions were rejected (closed and perhaps deleted), compared with our 16.3% rejection rate overall (12.3% if you discount the list and recommendation questions from the site's early days). I conclude that story identification questions are better quality than average.

I've seen the interpretation that story identification are not policed and so poor questions (with too little detail) fall through the cracks. This does not agree with my experience, and in fact is contradicted by your data: those upvotes indicate that most story identification questions do get attention, and those people who upvoted didn't vote to close as “not a real question”.

9.5% of open story-identification are unanswered (as in: have at least one upvoted answer); the overall rate is 4.4%. So, on a subject matter that's inherently difficult, we still manage a decent response rate (90% have an answer that's at least vaguely plausible).

We get the occasional “me too” non-answer on story identification questions. This shows that story identification questions are useful to people other than the asker. I don't have statistics for these, but I think they're more common for story-identification than on average. Here's a sample such non-answer:

Thankyou so very very much for answering the question.
I stayed up nights for years trying to remember the name of that book so i could track down the second book and read it.
The questions was highly relevant and direct as I had exactly the same question now finally answered. Thank you... Now I can get on with my life. :)

However, these pile-on users don't seem to stick on the site (all the examples I found were one-time posters).

Speaking as a user

I subscribe to the story-identification tag, because most questions interest me one way or another: either I know the book and I want a chance at answering, or I don't know the book and I may be interested in reading it (if it's left such a good memory to someone else that they want to find it again, that's a recommendation). Then I apply my usual rules for voting on questions: if it's a well-written, on-topic question that's likely to elicit interesting answers, then I upvote. In other tags, I can only upvote if I know enough about the background (most questions about a specific work don't make any sense if you haven't read/seen/… it). Thus story-identification are inherently more prone to being upvoted by me on average.

Speaking as a community builder

When the site started, I hoped that it would become the go-to site for story identifications. Story identification is a perfect fit for the question and answers format. My model was the Usenet group rec.arts.sf.written, which is well-known as a go-to place for story identification (YASID — Yet Another Story IDentification; if you try to search for these on google you'll miss a lot because Google strips the [YASID] prefix that's traditionally used in the subject of such questions). YASIDs work ok on a Usenet group, but not so great: sometimes the thread diverges and the answer is buried amid hundreds of off-topic possts.

I wouldn't say that we've reached the point where we can advertise on our ability to answer story identification questions. Mind you, our accept rate is a little over 50%, which is almost as high as rasfw's; but rasfw gets more of the difficult requests, because it's the last resort to many. It's a pity, because story identification is great for promotion: this question form attracts both novices (who want their question answered) and experts (who want to show off their culture). I hope that we can organically recruit more experts, so that we can claim to be a place where story identification questions get answered.

Answer 3

I was curious whether they really score higher than other questions, or if everything just scores highly.

This gathered the set of scores for all questions and all story-identification questions:

import json
import gzip
import urllib2
import StringIO
import collections

story_id = collections.defaultdict(int)
everything = collections.defaultdict(int)
page = 1
while True:
    content = json.load(gzip.GzipFile(fileobj=StringIO.StringIO(urllib2.urlopen("http://api.scifi.stackexchange.com/1.1/questions?page=%s" % page).read())))
    if not content[u"questions"]:
        break
    for question in content[u"questions"]:
        if u"story-identification" in question["tags"]:
            story_id[question[u"score"]] += 1
        everything[question[u"score"]] += 1
    page += 1

The average score for all questions is currently 10.2 and for story-identification 7.0 - however there are extreme values in general questions, and aren't for story identification.

If you look at the score distribution (I've trimmed off the questions scoring more than 20):

It looks to me like these questions are more likely to score in the 2-8 range, but less likely to score highly. I don't think it's accurate to say that these questions generally score higher than other questions. It seems more accurate:

• to say that questions in general tend to score more highly than a long-established site (e.g. Stack Overflow) - 81% of questions get at least a 5. I think this is likely because while the site is small, it's fairly easy to look at all questions.
• to wonder if story-identification questions are scoring too highly even though they score (a little) lower than the average question.

For the latter, IMO there are reasons these are easy to gain votes:

• It's difficult for me to judge whether a {random topic} question is well-asked or not if I have no familiarity with the work (I might not even look at the question). It's not hard to judge whether a story-identification question is good (does it include sufficient information that it's likely that it can be answered).
• If you're using "interestingness" as a reason to up vote as well as "well asked", some people (including me) find these questions interesting in general (there's plenty of material in meta about why this is so I won't repeat it here). I would guess that one reason that the questions don't score as highly as other questions in general (even though "well asked" is fairly simple to achieve) is that only a few people feel this way.

Answer 4

There is a certain amount of bikeshed effect here. These questions are highly accessible. That's a feature of crowdsourcing and you just get to live with it.

For my part, I like them because they are about something other than Star Wars, Star Trek, Battlestar, and Harry Potter. To be sure, people are entitled to like what they like, but the recent questions on those universes have been of interest only to fan boys.

The Science Fiction and Fantasy universe is so much bigger than the shows you've seen on TV and at the cinema in that last decade.

Answer 5

It's a common experience to have read or heard or seen a story that we can't remember. And, often, we're not the only ones who remember certain scenes or characters vividly.

I know that about 10 years ago, I was trying to find an old time radio show, A Pail of Air, but kept searching for "bucket" instead of "pail." It took me a long time to find the story because I kept using the wrong word on my searches. Then, later, when I found out it was on a broadcast of "X Minus 1," I found a website that mentioned this is "one of those stories." It's one that everyone remembers parts of, but they forge the title and author.

With that in mind, and considering that many of these questions refer to seeing the show as a child or teen, I think these questions help a lot of people find works they are looking for (perhaps so they can re-read or re-watch) and if it helps the person asking the question, it likely helps others Googling for that info at some later date.

I just wish, when I was looking for the info on A Pail of Air, that SE was around and someone had asked about it. That would have saved me hours of searching and I'm sure these story-id questions will do that in the long run for a number of people.