People cannot reliably tell whether a short story was written by a human or by ChatGPT, and they actually prefer the AI's writing when they think a person wrote it, according to a new study.
Researchers found that readers consistently gave the highest ratings to stories generated by artificial intelligence, but only when they were told the author was human. In one part of the experiment, participants correctly identified AI-written stories just 39.9 per cent of the time, worse than random guessing.
Dr Deena Weisberg, senior author of the study from the Department of Psychological and Brain Sciences at Villanova University, said participants gave the highest ratings to stories they had been told were written by humans but which were actually written by AI. She said this reveals a bias towards narratives written by real people, since people assume creative writing requires uniquely human qualities such as emotional understanding and lived experience, which leads them to underestimate AI's capabilities. She said public assumptions about AI's capabilities are increasingly out of date.
How the study was carried out
Scientists tested more than 2,500 adults aged between 18 and 81 using a series of human-written and AI-generated short stories. Some volunteers were told who had written the story before reading it, while others had to guess for themselves.
The analysis, published in the journal Judgment and Decision Making, found that when participants believed an AI-written story had been penned by a human, they rated it as better written and more engaging than any other combination. Overall, the AI stories also scored higher than those written by real authors for quality and how absorbed readers became in the plot.
When it came to identifying who had written each story, volunteers performed little better than chance. In one experiment they correctly identified AI-generated stories just 39.9 per cent of the time. In another, their success rate was 52 per cent, effectively no better than chance.
Spotting the patterns
Dr Weisberg said familiarity with AI systems appeared to help people recognise the patterns typical of AI-generated writing, such as em dashes and sentence structures like "it's not just X, it's Y." She said this suggests improving AI literacy would be one way to help people navigate the new AI-enabled world they are living in.
The study validates previous research showing that people often prefer AI-generated writing in various formats, including short stories, poetry and essays. Dr Weisberg said AI writing tends to be clearer, more direct and easier to process, while human-written stories are often more subtle and complex. She said the AI versions of the stories usually stated their themes explicitly, rather than allowing readers to infer meaning from the characters' words and actions. She said it is possible that people generally prefer predictability, because difficult or subtle material requires more brain power, which could explain why they rated the AI stories higher.
Other ways to spot AI text
Previous suggestions for spotting AI-generated text include looking for inconsistencies and repetition, such as abrupt shifts in tone or repeated phrases or sentence structures. AI text may also reference specific details without appropriate context or feel basic and formulaic.
Excessive use of buzzwords and jargon can be a sign of AI filling gaps in its knowledge with generic vocabulary. Quick responses could also be a sign of AI-generated text, since people often take time to think and reply. Developers have released AI detection tools to help pick out text that has been generated or improved by AI, for example in student essays or job applications.
