22

As you may be aware, the artificial intelligence chatbot ChatGPT has been getting some negative attention on Stack Exchange as of late, its usage notably being completely banned on Stack Overflow.

Other sites are now discussing whether to either ban ChatGPT-generated answers or allow them on the condition that they meet quality standards. Two big concerns that have been raised is the issue of plagiarism and citation with respect to chatbot-generated content as well as the potential for a chatbot-generated answer to sound reasonable but contain factual inaccuracies that most honest, knowledgeable human posters would not make. For example, I posed the following question to ChatGPT:

In the original, prime timeline of Star Trek, how does Captain Kirk die?

Many of us know the answer to this question. He dies in the film Star Trek: Generations (1994) while disabling a weapon of mass destruction on the planet Veridian III in the (in-universe) year 2371. Jean-Luc Picard buries him in a makeshift, shallow grave marked with his Starfleet uniform insignia.

ChatGPT's answer was:

In the Star Trek universe, Captain James T. Kirk, played by William Shatner, does not die in the prime timeline. He is a central character in the original Star Trek television series and appears in several Star Trek films.

In the Star Trek films, Kirk is shown to be an aging captain who eventually retires from Starfleet. In the film "Star Trek Generations," which takes place in the prime timeline, Kirk is shown to have died at some point before the events of the film. However, the circumstances of his death are not depicted in the film.

It is worth noting that there are several alternate timelines and continuities in the Star Trek franchise, and in some of these, Kirk's fate may be different. However, in the prime timeline of the original Star Trek series and films, Kirk does not die.

At first this looks like a half-decent middle school essay, but at second glance (more than given to many answers), it is a self-contradictory, factually incorrect piece of garbage. It first claims that Kirk does not die in the prime timeline (factually incorrect). It then mentions the prime timeline film in which the character does, in fact, die, and claims that he died before the film! It claims that the circumstances of Kirk's death "are not depicted in the film", while in fact the film's depiction of Kirk's death is arguably its climax. For its last paragraph, it doubles back and re-establishes its original claim that Kirk does not die.

5
  • 3
    I suppose your title is a reference to the phrase "elephant in the room", but it confused me for a bit, as I thought you were asking about discussing in our site chatroom.
    – Rand al'Thor Mod
    Dec 17, 2022 at 21:15
  • @Randal'Thor yes, it's a reference to "elephant in the room". Please feel free to rename it if you feel this is too confusing. Dec 17, 2022 at 21:19
  • Currently debating whether I should get a ChatGPT bot to answer this question.
    – Möoz
    Dec 19, 2022 at 0:00
  • 2
    @Möoz do you like having your answers deleted? :p
    – AncientSwordRage Mod
    Dec 19, 2022 at 13:38
  • YouChat's (AI chatbot) answer to the question you asked ChatGPT: "In the original, prime timeline of Star Trek, Captain Kirk dies in 2371 while helping Captain Picard thwart the villain Soran. In the movie Star Trek Generations, Kirk enters the Nexus, a realm of pure pleasure and joy, where he is reunited with his long-dead love, the former Enterprise navigator Lieutenant Commander Janice Rand. However, the villain Soran has infiltrated the Nexus, and Kirk sacrifices himself in order to save the lives of the crew of the Enterprise-D and prevent Soran from destroying a populated planet. [...]" Dec 26, 2022 at 14:45

2 Answers 2

26

I propose that we follow other network sites in banning AI-generated copy-pasted content. This stuff is (to use a technical term) dangerous bullshit. As you've noted, it looks plausible and well-written, but to an expert eye (sometimes even a non-expert but humanly intelligent eye) it's clearly nonsense. It's one of the facts of SE that people sometimes upvote incorrect answers that look plausible - not every voter has the expertise necessary to judge whether an answer is true or false. So if we allowed this stuff, we'd run the risk of getting upvoted answers that are not only incorrect but were never even written by anyone thinking they were correct - the computer-generated equivalent of shitposting.

I already posted about this in chat:

If you see something that looks at first glance like an answer but to a knowledgeable eye is clearly nonsense, feel free to mod-flag it.

And, sadly, AI-generated answers have already been spotted recently on SFF.

So yes, let's have none of this nonsense, and please be proactive in flagging apparently AI-generated posts. (I have very little experience with AI-generated text or how to tell if a post is AI-generated or not, but there are mod-only spaces where I can get advice from people smarter than me at this kind of thing. Let's not discuss publicly how to detect these posts, beyond the very simple criterion mentioned above, otherwise the people posting them might make them less easily detectable.)

Using AI as a tool to assist generating good quality content might be reasonable, in some cases. I'm not proposing (yet) an outright ban on any answers where any kind of help from AI was used. But using AI, or other tools, to generate a large quantity of answers, without regard to whether those answers are correct and actually answer the question on which they are posted, is not acceptable. Content copy-pasted directly from ChatGPT output must be attributed to its source, or risk falling afoul of existing rules against plagiarism as well as new policies against ChatGPT specifically.

9
  • 5
    From twitter.com/studentactivism/status/… : "Because if ChatGPT is, as it seems to be, a consummate bullshitter, it's also—definitionally—a bullshitter who doesn't know when its bullshitting. And we all know that that's the most dangerous kind."
    – PM 2Ring
    Dec 18, 2022 at 17:33
  • 1
    @PM2Ring Wait, that's most of us SFF users anyway :)
    – Möoz
    Dec 18, 2022 at 23:49
  • I'm wary of a ban on all use of AI for all time, because we simply do not know what AI is going to look like six months from now (the technology has been moving in leaps and bounds lately). But this particular AI has no notion of truth or even evidence. Of course we should ban it, and (maybe) reassess once its capabilities (and honesty) have (significantly!) improved.
    – Kevin
    Dec 19, 2022 at 8:50
  • 2
    @Kevin Yes, I (and AFAIK other mods/staff across the network) have avoided proposing a ban on all use of all AI for all time. But it seems reasonable to ban use of AI without any human checking to verify and confirm the output, and use of AI without disclosure is banned under existing plagiarism rules.
    – Rand al'Thor Mod
    Dec 19, 2022 at 8:55
  • If the AI posts a correct answer, would we even know it was AI?
    – Skooba
    Dec 22, 2022 at 15:41
  • @Skooba If it's not edited, we could very likely recognize the "writing style" (for lack of a better concept) of the expository answers given by ChatGPT. It is pretty formulaic. Dec 24, 2022 at 4:44
  • I've seen two posts now that used Chat-GPT to try to solve [story-identification] questions. Both posts referenced works that simply did not exist, because Chat-GPT made them up; it's just what it does a lot of the time instead of saying it doesn't know. I see no value in keeping these answers (regardless of attribution).
    – Laurel
    Dec 26, 2022 at 22:32
  • 1
    "LANGUAGE!"!!! Wow I think this post had more bad words from Rand than I saw in several years prior, combined :P Tell us how you really feel! Dec 27, 2022 at 18:57
  • 1
    @DVK :-D note that the bad words are hyperlinked to make clear that they're technical terms :-P
    – Rand al'Thor Mod
    Dec 27, 2022 at 19:00
10

I decided to get some input from the offending party itself, so I've created an account on OpenAI and asked ChatGPT the following:

Should AI-generated posts be allowed on the Science Fiction & Fantasy Stack Exchange?

To which it answered:

It is ultimately up to the moderators and community of the Science Fiction & Fantasy Stack Exchange to decide whether AI-generated posts should be allowed on the platform. However, it is important to consider the potential impacts of allowing AI-generated content on the quality and integrity of the discussions on the platform.

Some potential considerations might include:

  • The quality and accuracy of the content: AI-generated content may not always be of high quality or accurate. It is important to ensure that any content that is allowed on the platform meets the standards of the community and contributes to meaningful and useful discussions.
  • The transparency of the source: It may be important for users to know the source of the content they are interacting with, particularly if it is an AI. This can help users to better understand the context and limitations of the content and make informed decisions about how to engage with it.
  • The potential for misuse or abuse: Allowing AI-generated content on the platform could also present potential risks for misuse or abuse, such as spamming or impersonation. It may be necessary to put safeguards in place to mitigate these risks.

Ultimately, the decision about whether to allow AI-generated content on the Science Fiction & Fantasy Stack Exchange should be based on a careful consideration of these and other relevant factors, and should be guided by the goals and values of the community.

Make of that what you will.

4
  • 2
    I make of it that it's pretty much a generic statement, not considering that SE is Q&A site, not a forum.
    – Mithoron
    Dec 20, 2022 at 17:05
  • 2
    @Mithoron: If you ask the average person who has never heard of a "Stack Exchange" the same question, I imagine their response would be quite a bit less coherent than what ChatGPT came up with. It's certainly generic, but it's making do with the information available in the prompt. Still, some actual fact-checking would be a welcome improvement to the service.
    – Kevin
    Dec 21, 2022 at 6:10
  • 7
    Basically, it told you to go and ask on meta :-)
    – Rand al'Thor Mod
    Dec 22, 2022 at 16:20
  • 8
    +1 for a great demonstration of how a lot of ChatGPT output can look like useful information while not actually being useful information. Dec 23, 2022 at 19:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .