As you may be aware, the artificial intelligence chatbot ChatGPT has been getting some negative attention on Stack Exchange as of late, its usage notably being completely banned on Stack Overflow.

Other sites are now discussing whether to either ban ChatGPT-generated answers or allow them on the condition that they meet quality standards. Two big concerns that have been raised is the issue of plagiarism and citation with respect to chatbot-generated content as well as the potential for a chatbot-generated answer to sound reasonable but contain factual inaccuracies that most honest, knowledgeable human posters would not make. For example, I posed the following question to ChatGPT:

In the original, prime timeline of Star Trek, how does Captain Kirk die?

Many of us know the answer to this question. He dies in the film Star Trek: Generations (1994) while disabling a weapon of mass destruction on the planet Veridian III in the (in-universe) year 2371. Jean-Luc Picard buries him in a makeshift, shallow grave marked with his Starfleet uniform insignia.

ChatGPT's answer was:

In the Star Trek universe, Captain James T. Kirk, played by William Shatner, does not die in the prime timeline. He is a central character in the original Star Trek television series and appears in several Star Trek films.

In the Star Trek films, Kirk is shown to be an aging captain who eventually retires from Starfleet. In the film "Star Trek Generations," which takes place in the prime timeline, Kirk is shown to have died at some point before the events of the film. However, the circumstances of his death are not depicted in the film.

It is worth noting that there are several alternate timelines and continuities in the Star Trek franchise, and in some of these, Kirk's fate may be different. However, in the prime timeline of the original Star Trek series and films, Kirk does not die.

At first this looks like a half-decent middle school essay, but at second glance (more than given to many answers), it is a self-contradictory, factually incorrect piece of garbage. It first claims that Kirk does not die in the prime timeline (factually incorrect). It then mentions the prime timeline film in which the character does, in fact, die, and claims that he died before the film! It claims that the circumstances of Kirk's death "are not depicted in the film", while in fact the film's depiction of Kirk's death is arguably its climax. For its last paragraph, it doubles back and re-establishes its original claim that Kirk does not die.

  • 3
    I suppose your title is a reference to the phrase "elephant in the room", but it confused me for a bit, as I thought you were asking about discussing in our site chatroom.
    – Rand al'Thor Mod
    Commented Dec 17, 2022 at 21:15
  • @Randal'Thor yes, it's a reference to "elephant in the room". Please feel free to rename it if you feel this is too confusing. Commented Dec 17, 2022 at 21:19
  • Currently debating whether I should get a ChatGPT bot to answer this question.
    – Möoz
    Commented Dec 19, 2022 at 0:00
  • 3
    @Möoz do you like having your answers deleted? :p
    – AncientSwordRage Mod
    Commented Dec 19, 2022 at 13:38
  • YouChat's (AI chatbot) answer to the question you asked ChatGPT: "In the original, prime timeline of Star Trek, Captain Kirk dies in 2371 while helping Captain Picard thwart the villain Soran. In the movie Star Trek Generations, Kirk enters the Nexus, a realm of pure pleasure and joy, where he is reunited with his long-dead love, the former Enterprise navigator Lieutenant Commander Janice Rand. However, the villain Soran has infiltrated the Nexus, and Kirk sacrifices himself in order to save the lives of the crew of the Enterprise-D and prevent Soran from destroying a populated planet. [...]" Commented Dec 26, 2022 at 14:45

3 Answers 3


I propose that we follow other network sites in banning AI-generated copy-pasted content. This stuff is (to use a technical term) dangerous bullshit. As you've noted, it looks plausible and well-written, but to an expert eye (sometimes even a non-expert but humanly intelligent eye) it's clearly nonsense. It's one of the facts of SE that people sometimes upvote incorrect answers that look plausible - not every voter has the expertise necessary to judge whether an answer is true or false. So if we allowed this stuff, we'd run the risk of getting upvoted answers that are not only incorrect but were never even written by anyone thinking they were correct - the computer-generated equivalent of shitposting.

I already posted about this in chat:

If you see something that looks at first glance like an answer but to a knowledgeable eye is clearly nonsense, feel free to mod-flag it.

And, sadly, AI-generated answers have already been spotted recently on SFF.

So yes, let's have none of this nonsense, and please be proactive in flagging apparently AI-generated posts. (I have very little experience with AI-generated text or how to tell if a post is AI-generated or not, but there are mod-only spaces where I can get advice from people smarter than me at this kind of thing. Let's not discuss publicly how to detect these posts, beyond the very simple criterion mentioned above, otherwise the people posting them might make them less easily detectable.)

Using AI as a tool to assist generating good quality content might be reasonable, in some cases. I'm not proposing (yet) an outright ban on any answers where any kind of help from AI was used. But using AI, or other tools, to generate a large quantity of answers, without regard to whether those answers are correct and actually answer the question on which they are posted, is not acceptable. Content copy-pasted directly from ChatGPT output must be attributed to its source, or risk falling afoul of existing rules against plagiarism as well as new policies against ChatGPT specifically.

  • 6
    From twitter.com/studentactivism/status/… : "Because if ChatGPT is, as it seems to be, a consummate bullshitter, it's also—definitionally—a bullshitter who doesn't know when its bullshitting. And we all know that that's the most dangerous kind."
    – PM 2Ring
    Commented Dec 18, 2022 at 17:33
  • 1
    @PM2Ring Wait, that's most of us SFF users anyway :)
    – Möoz
    Commented Dec 18, 2022 at 23:49
  • I'm wary of a ban on all use of AI for all time, because we simply do not know what AI is going to look like six months from now (the technology has been moving in leaps and bounds lately). But this particular AI has no notion of truth or even evidence. Of course we should ban it, and (maybe) reassess once its capabilities (and honesty) have (significantly!) improved.
    – Kevin
    Commented Dec 19, 2022 at 8:50
  • 2
    @Kevin Yes, I (and AFAIK other mods/staff across the network) have avoided proposing a ban on all use of all AI for all time. But it seems reasonable to ban use of AI without any human checking to verify and confirm the output, and use of AI without disclosure is banned under existing plagiarism rules.
    – Rand al'Thor Mod
    Commented Dec 19, 2022 at 8:55
  • If the AI posts a correct answer, would we even know it was AI?
    – Skooba
    Commented Dec 22, 2022 at 15:41
  • 1
    @Skooba If it's not edited, we could very likely recognize the "writing style" (for lack of a better concept) of the expository answers given by ChatGPT. It is pretty formulaic. Commented Dec 24, 2022 at 4:44
  • 2
    I've seen two posts now that used Chat-GPT to try to solve [story-identification] questions. Both posts referenced works that simply did not exist, because Chat-GPT made them up; it's just what it does a lot of the time instead of saying it doesn't know. I see no value in keeping these answers (regardless of attribution).
    – Laurel
    Commented Dec 26, 2022 at 22:32
  • 1
    "LANGUAGE!"!!! Wow I think this post had more bad words from Rand than I saw in several years prior, combined :P Tell us how you really feel! Commented Dec 27, 2022 at 18:57
  • 1
    @DVK :-D note that the bad words are hyperlinked to make clear that they're technical terms :-P
    – Rand al'Thor Mod
    Commented Dec 27, 2022 at 19:00

I decided to get some input from the offending party itself, so I've created an account on OpenAI and asked ChatGPT the following:

Should AI-generated posts be allowed on the Science Fiction & Fantasy Stack Exchange?

To which it answered:

It is ultimately up to the moderators and community of the Science Fiction & Fantasy Stack Exchange to decide whether AI-generated posts should be allowed on the platform. However, it is important to consider the potential impacts of allowing AI-generated content on the quality and integrity of the discussions on the platform.

Some potential considerations might include:

  • The quality and accuracy of the content: AI-generated content may not always be of high quality or accurate. It is important to ensure that any content that is allowed on the platform meets the standards of the community and contributes to meaningful and useful discussions.
  • The transparency of the source: It may be important for users to know the source of the content they are interacting with, particularly if it is an AI. This can help users to better understand the context and limitations of the content and make informed decisions about how to engage with it.
  • The potential for misuse or abuse: Allowing AI-generated content on the platform could also present potential risks for misuse or abuse, such as spamming or impersonation. It may be necessary to put safeguards in place to mitigate these risks.

Ultimately, the decision about whether to allow AI-generated content on the Science Fiction & Fantasy Stack Exchange should be based on a careful consideration of these and other relevant factors, and should be guided by the goals and values of the community.

Make of that what you will.

  • 3
    I make of it that it's pretty much a generic statement, not considering that SE is Q&A site, not a forum.
    – Mithoron
    Commented Dec 20, 2022 at 17:05
  • 3
    @Mithoron: If you ask the average person who has never heard of a "Stack Exchange" the same question, I imagine their response would be quite a bit less coherent than what ChatGPT came up with. It's certainly generic, but it's making do with the information available in the prompt. Still, some actual fact-checking would be a welcome improvement to the service.
    – Kevin
    Commented Dec 21, 2022 at 6:10
  • 8
    Basically, it told you to go and ask on meta :-)
    – Rand al'Thor Mod
    Commented Dec 22, 2022 at 16:20
  • 9
    +1 for a great demonstration of how a lot of ChatGPT output can look like useful information while not actually being useful information. Commented Dec 23, 2022 at 19:00

As technology is moving fast, out of curiosity I just asked ChatGPT-4 the same question:

In the original, prime timeline of Star Trek, how does Captain Kirk die?

to which it responded (mostly correctly as far as I can tell):

In the "Star Trek" franchise, Captain James T. Kirk dies in the film "Star Trek: Generations" (1994). In this movie, which is part of the prime timeline of "Star Trek," Kirk dies while helping Captain Jean-Luc Picard stop a mad scientist named Dr. Tolian Soran. Soran's plan involves redirecting the path of an energy ribbon known as the Nexus in order to enter it and return to a paradise-like reality it contains.

Kirk initially appears to die early in the film during an incident aboard the U.S.S. Enterprise-B, but it is later revealed that he was actually pulled into the Nexus. When Picard, who also enters the Nexus, meets Kirk, he convinces him to leave this idyllic reality to help stop Soran. During their efforts to prevent Soran from destroying a star (and consequently a planetary system) to alter the path of the Nexus, Kirk is fatally injured when a metal bridge they are on collapses. His last words express his contentment, as he feels his life had made a difference.

Surprisingly, it also gets the info on the other captains (Archer, Picard, Sisko, Janeway, Burnham) right!

I also tried to trick it:

In the original, prime timeline of Star Trek, how does Captain Obi Wan Kenobi die?

to which it responded:

Captain Obi-Wan Kenobi is not a character from the "Star Trek" universe; he is a significant figure in the "Star Wars" franchise. In "Star Wars," Obi-Wan Kenobi is a Jedi Knight and Master, and he plays a pivotal role across several of the films and series within the franchise.

I still don't think that AI-generated responses should be allowed here as even though it got these tests right, it could fail on others, but it's a datum.

  • 8
    The problem is that the AI doesn't distinguish between reality and plausible sounding nonsense. If I ask it "What was the name of the captain of the ship that brought Qui Gon Jinn and Obi Wan to the Trade Federation ship?" it gives me the answer "The captain of the ship that brought Qui-Gon Jinn and Obi-Wan Kenobi to the Trade Federation ship in "Star Wars: Episode I - The Phantom Menace" is Captain Panaka" which is nonsense. It was actually Maoi Madakor. Panaka is an entirely different character.
    – Valorum
    Commented May 5 at 20:45
  • 1
    The original post was made in December 2022, back when these LLMs were still in their infancy. Most current LLMs will be able to accurately answer the question provided by OP. Commented May 6 at 3:08
  • 5
    @galacticninja - Sure, but I asked the question in my comment yesterday. And it still got it wrong.
    – Valorum
    Commented May 6 at 11:20
  • @Valorum LLMs that have live access to the internet will usually get it correct. Those that can't are likely to hallucinate. Commented May 6 at 13:17
  • 2
    @galacticninja - Bard isn't any better. "The name of the captain of the ship that brought Qui Gon Jinn and Obi-Wan Kenobi to the Trade Federation ship is not mentioned in the movie Star Wars: Episode I - The Phantom Menace or any other canon Star Wars media.". Which is just plain wrong.
    – Valorum
    Commented May 7 at 17:33
  • 1
    @Valorum Here's what I get with Gemini (via Google Search): "In Star Wars, Captain Maoi Madakor piloted the Consular-class space cruiser Radiant VII to the Naboo system in 32 BBY to deliver Jedi ambassadors Qui-Gon Jinn and Obi-Wan Kenobi. The ship's co-pilot was Lieutenant Antidar Williams." Commented May 8 at 0:38
  • 4
    @galacticninja - And a human who has access to the Internet can do even better. In fact, they could read these LLM-generated answers and synthesize their own, which would be benefit our site more than copying and pasting the answer. Even one of the responses shown in the question fails to correct a fundamental misconception (that Obi-Wan Kenobi was never a captain, only a general), and does not actually say how Obi-Wan died (presumably something that the person asking the question would like to know).
    – Adamant
    Commented May 8 at 6:03
  • @galacticninja - Note that the only correct answer there is nearly a direct quote from the beginning of the wiki page on Maoi Madakor: " In 32 BBY, she piloted the Consular-class space cruiser Radiant VII with co-pilot Antidar Williams to the Naboo system in order to deliver the Jedi ambassadors Qui-Gon Jinn and Obi-Wan Kenobi." So this answer is not providing much value above an answer that just links to and quotes the wiki (and indeed, might be doing a "search, quote and reword" strategy behind the scenes).
    – Adamant
    Commented May 8 at 6:09
  • @Adamant [EDIT- This is a reply to your 1st comment.] I agree. What I'm saying is that AI/LLMs have come a long way since the original post was written back in 2022. The example used by the OP is a bit dated now since today's AI models with internet access can easily get that and similar questions right and even provide sources to back them up. Commented May 8 at 6:17
  • 3
    @galacticninja I think your assessment of accuracy is the wrong approach (and I disagree with the original answer that it's accuracy all that matters). If LLMs essentially become better search engines there's no need for scifi.se to exist at all. This site is about humans answering other humans, if we're just cutting the middleman and plugging LLMs into it, why not delete the whole database of Q&A, make scifi.se a frontend for an LLM, and get it over with? And the answer is: because we do not want a generic LLM answer in LLM-speak, we want questions from humans answered by humans.,
    – Andres F.
    Commented May 15 at 21:44
  • 1
    @galacticninja otherwise we might simply become curators/fact-checkers of LLM output... is that what we want this site to become? And keep in mind it takes zero effort to ask a question to an LLM chatbot and paste it here as an answer, placing all the onus on volunteer fact-checkers. At least before LLMs it was harder for human bullshitters to make up bullshit to post here...
    – Andres F.
    Commented May 15 at 21:47
  • 1
    @AndresF. I agree with most of your points. I'm on board with the idea that we shouldn't allow copy-pasting AI/LLM answers without human verification. What I'm saying is that the example provided in the original post is outdated, given the rapid advancements in AI/LLM technology since 2022. Nowadays, AI/LLMs could likely answer that example question accurately. That particular example may not be the best representation of AI/LLM limitations anymore. Commented May 31 at 3:39
  • @galacticninja on that point we agree!
    – Andres F.
    Commented May 31 at 12:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .