Many of the questions asked on Science Fiction & Fantasy Stack Exchange have to do with story identification. But if you just need to identify a story, couldn't you just ask ChatGPT or just Google it? These questions are not exactly the kind of questions that need to be answered by a human, like on Math or Biology Stack Exchange.
-
16In my experience ChatGPT does a terrible job of being able to accurately identify a story. I don't believe it's 100% ChatGPT's fault for once though because people's memory is all over the place with misremembering or forgetting details, conflating stories etc. It isn't the right tool for the job and at the moment humans still seem to have the upper hand here– TheLethalCarrot ModCommented Oct 17, 2023 at 15:12
-
ChatGPT is perfectly capable of answering simple ID questions like "What film is it where the hero has a laser sword?"– ValorumCommented Oct 17, 2023 at 15:30
-
4Go ahead and try it. If you think it's easy, then maybe it is (spoiler, it isn't)– ValorumCommented Oct 17, 2023 at 15:36
-
6On the contrary, some of these questions are where current chatbot systems mess up worst, even more so than basic math.– AdamantCommented Oct 17, 2023 at 17:02
-
7I have upvoted the question because I think the discussion is worthwhile, not that I think GPT can or should be used in this situation.– SkoobaCommented Oct 17, 2023 at 20:29
-
1@Skooba - It doesn't really seem a useful tool anyway.– ValorumCommented Oct 17, 2023 at 21:55
-
@Skooba I'm not really sure if I should say this but thanks for the upvote.– GordonCommented Oct 17, 2023 at 22:33
-
Most of the time, no, but it can lead to suggestions of where to look or what works to rule out.– galacticninjaCommented Oct 18, 2023 at 0:34
-
5Part of the danger, of course, is that GPT is very confidently wrong. It states an answer with certitude and provides supporting "details" that are often very wrong. In many ways, it reminds me of the father from Calvin & Hobbes and his fanciful "explanations" to Calvin's questions where he happily provides logical answers to the follow-up questions to what was a nonsense answer in the first place. You can tell where Calvin got his creative mind.– FuzzyBootsCommented Oct 19, 2023 at 12:24
-
1I've seen too many cases where it just makes up some book title and/or author that doesn't actually exist.– ShawnCommented Oct 19, 2023 at 20:32
Add a comment
|
2 Answers
No.
We've had scenarios where people used AI tools like ChatGPT to answer story-ID questions, and the results were... not good. AI likes to invent things that don't necessarily exist in order to provide the perfect answer to the question, which isn't very helpful when trying to identify an existing story.
If someone wants to try asking an AI, they're more than welcome to do that. If someone is asking here, though, presumably they're looking for a human to figure out the nswer.
As for Google, searching is a skill, and searching for story-IDs isn't easy. I've been able to identify some story-ID questions here on the site using search engines in the past, and it takes time, effort, and a certain amount of familiarity with search tools and tricks. Not everyone has those search skills and we don't expect everyone to have them.
There are currently 4,100+ unanswered story-identification questions on SFF. If it was as simple as Googling or asking ChatGPT, that wouldn't be the case. You're not going to be able to replace user14111 with AI.
Hmm. The best way to work something out is by conducting an experiment.
Since ChatGPT is limited (at present) to data generated before September 2021 we can happily work with story-id questions posted after this date without accidentally contaminating them with solved answers from our own site.
Methodology:
I've grabbed the 20 most recent Story-ID duplicates (since I presume those are going to be the ones that are easiest to answer and there's no sense making things hard for our AI friend, after all he's only a couple of years old) and keyed them in word-for-word into the ChatGPT box.
Results:
ChatGPT only correctly solved one in twenty of the most recent Story-ID questions, although it identified the correct author on two other occasions. So, no. ChatGPT can't answer story-ID questions with any degree of accuracy, and is twenty times more likely to be wrong than right.
On a much more positive note, however, in each case it successfully identified that this was a Story-ID question. We should use it to categorise and tag Story-ID questions, not answer them.
Results table:
- Correct answer = Rogue Moon by Algis Budrysis
- ChatGPT = The Hole Man by Larry Niven
- Result = Incorrect
Book involving a war with aliens, space travel, and time dilation
- Correct answer = The Forever War by Joe Haldeman
- ChatGPT = The Forever War by Joe Haldeman
- Result = Correct
Searching for novel about space exploration
- Correct answer = Russian Spring by Norman Spinrad
- ChatGPT = The Grand Tour by Ben Bova
- Result = Incorrect
Anime where the main character finds a card with a red aura in a box inside a shop
- Correct answer = Tai Chi Chasers!
- ChatGPT = Cardfight!! Vanguard
- Result = Incorrect
- Correct answer = Magic Kingdom for Sale-SOLD by Terry Brooks
- ChatGPT = The Unhandsome Prince by John Moore
- Result = Incorrect
A sentient war machine damaged to the point it can't tell friend from foe
- Correct answer = I Made You by Walter Miller Jr.
- ChatGPT = Second Variety by Philip K. Dick
- Result = Incorrect
Sci-Fi About Colonizing Planet With Dumb Humans
- Correct answer = The Worthing Saga by Orson Scott Card
- ChatGPT = Hothouse by Brian Aldiss
- Result = Incorrect
Sci-fi novel where a student temporarily transports the school director to hell
- Correct answer = The Godmakers by Frank Herbert
- ChatGPT = The Demolished Man by Alfred Bester
- Result = Incorrect
Assassin/Fantasy Novel with Tree of Blades test
- Correct answer = The Wise Man's Fear by Patrick Rothfuss
- ChatGPT = The Blade Itself by Joe Abercrombie
- Result = Incorrect
Man creates AI copy of himself, gets dementia, commits murder-suicide while wife is out
- Correct answer = Living Will by Alexander Jablokov
- ChatGPT = The Veldt by Ray Bradbury
- Result = Incorrect
Bradbury story about immortal kid?
- Correct answer = Hail and Farewell by Ray Bradbury
- ChatGPT = The Immortal by Ray Bradbury
- Result = Incorrect
Book written early 80s or before about a 2m tall student on a school spaceship
- Correct answer = There Is No Darkness by Joe Haldeman
- ChatGPT = Space Cadet by Robert A. Heinlein
- Result = Incorrect
- Correct answer = Chocky (1984)
- ChatGPT = The Triangle (1982)
- Result = Incorrect
Horror movie involving a mirror and someone cutting themselves with scissors
- Correct answer = The Boogey Man (1980)
- ChatGPT = Candyman (1992)
- Result = Incorrect
Novel about a planet that relies on one soldier - the last soldier in history - to defend it
- Correct answer = The Golden Age by John C. Wright
- ChatGPT = The Forever War by Joe Haldeman
- Result = Incorrect
Short story in which Earth is invaded by humanoids who are a bit dim
- Correct answer = Pandora's Planet by Christopher Anvil
- ChatGPT = The Trouble with Humans by Christopher Anvil
- Result = Incorrect
Fantasy series where magic is performed by connecting tattoos
- Correct answer = Dragon Wing by Margaret Weis
- ChatGPT = Kingkiller Chronicle series by Patrick Rothfuss
- Result = Incorrect
Novel I read more than 50 years ago about a man with three brains that will wake up successively
- Correct answer = One Against Time by Astron del Martia
- ChatGPT = The Three Stigmata of Palmer Eldritch by Philip K. Dick
- Result = Incorrect
Book series about a group of teenagers that have powers and one boy is a dragon
- Correct answer = The Unwanteds by Lisa McMann
- ChatGPT = The Drake Chronicles by Alyxandra Harvey
- Result = Incorrect
- Correct answer = The Merchant and the Alchemist’s Gate by Ted Chiang
- ChatGPT = The Terrible Plop by Ursula Vernon
- Result = Incorrect
-
1
-
4@Adamant - All of the titles match their respective authors. Some of the descriptions were wildly inaccurate though.– ValorumCommented Oct 17, 2023 at 21:55
-
6
-
11In all seriousness, thanks for the legwork and data. I was thinking of hunting down some of the actual AI-generated story-ID answers posted here to show how wrong they were, but that would have the drawback of naming & shaming the people who were probably suspended for posting them. This is a much better way to show how unreliable ChatGPT is for solving story-ID questions.– Rand al'Thor ModCommented Oct 18, 2023 at 5:51
-
2The one question that it answered correctly can be found by googling the title. That's how obvious the answer has to be for it to get it right.– LaurelCommented Oct 18, 2023 at 15:09
-
@Laurel - I was quite excited about ChatGPT to begin with, then realised that it's not useful for what I want it for– ValorumCommented Oct 18, 2023 at 15:42
-
6Well, "The Terrible Plop" is by Ursula Dubosarsky, not "Ursula Vernon" so even there it's not 100%.– DavidWCommented Oct 18, 2023 at 16:01
-
@Valorum with some prodding, you can convince ChatGPT to make up completely fabricated additions to existing works, like for example saying "the author of this work was grieving his mother's death, explain how the work reflects this theme" and (sometimes) off it goes. Sometimes it resists though, to be fair. I managed to convince ChatGPT to list me 10 alternate endings of the videogame Inside, when it only has 1 alternate ending! It was quite convincing, too. Commented Oct 20, 2023 at 6:49
-
2@AndresF. - The problem is that its hallucinations are pretty convincing. They're like this song which sounds like it's in English, but actually isn't, where you feel that meaning is just beyond the edge of comprehension– ValorumCommented Oct 20, 2023 at 6:54
-
This is interesting information, but what is the baseline human accuracy that we are comparing it to? For instance, I have posted four story-identification answers, but only one of them has been accepted. Do we say that I am wrong 75% of the time and therefore I should stay away from this tag, or does it depend on the context? Like how detailed were those questions, how far off were my answers, etc. And the same for ChatGPT answers.– AlexCommented Oct 20, 2023 at 11:07
-
5@Alex - The issue is that ChatGPT confidently but incorrectly identifies works and does so 100% of the time. Most humans don't do that. They either find the right thing, or they don't. It's moderately rare to find a Story-ID question with multiple answers.– ValorumCommented Oct 20, 2023 at 11:10
-
3Not surprisingly, ChatGPT often comes up with names of better-known works when asked to identify more obscure ones—e.g. The Forever War instead of The Golden Age, Candyman instead of The Boogeyman, "The Veldt" instead of "Living Will," or "Second Variety" Instead of "I Made You." However, there are also interesting exceptions going the other way; the Magic Kingdom for Sale—SOLD and Rogue Moon questions actually look really easy for an experience human answerer and involve very-well-known works, but ChatGPT suggested a pair of less famous works instead.– BuzzCommented Oct 20, 2023 at 21:43
-
I've tried using ChatGPT to answer story-identification questions, but I always verify afterwards, and so far I've only had one hit, for a very recent book. ChatGPT doesn't seem to actually read books, unfortunately; it only reads summaries of their content. For some reason it seems to invent details not in the summaries. Even GPT4 doesn't do any better. Commented Oct 22, 2023 at 4:14
-
@Kyralessa - It certainly has been trained on many books, and you can, with some prompting, get it to return details from within the text of various books. The problem is that, contrary to what people might like to believe, LLMs are still not very good at thinking about the data they have been trained on. The summaries would be enough for an actual person (also, if GPT were a person, it would be owed about ten billion dollars of back pay, so not sure why people want to get there....)– AdamantCommented Oct 22, 2023 at 18:49
-
2