A new study claims that training can help distinguish text produced by artificial intelligence (AI) from text written by humans.
Artificial intelligence technology tools like ChatGPT are making headlines. From writing poetry in the style of 16th century England to writing scientific research papers, chatbots haven’t gone away. In fact, these relatively good agents would continue to learn their craft. Will we be able to not be fooled by the natural language processing models that are becoming a staple of our society?
Fool me once
A research team at the University of Pennsylvania School of Engineering and Applied Science in the United States carried out the largest-ever human study on AI detection to provide some help. They gathered data from the web-based training game ‘Real or Fake Text?’ created by the university itself.
The findings are presented in a peer-reviewed paper at an Association for the Advancement of Artificial Intelligence meeting in February. The study demonstrated that we can learn to detect human-written and machine-generated text.
“We’ve shown that people can train themselves to recognize machine-generated texts,” stated Chris Callison-Burch, associate professor at the Department of Computer and Information Science (CIS) in a news item. “People start with a certain set of assumptions about what sort of errors a machine would make, but these assumptions aren’t necessarily correct. Over time, given enough examples and explicit instruction, we can learn to pick up on the types of errors that machines are currently making.”
Why is the new AI tool ChatGPT causing so much chaos? |
A little training goes a long way
“AI today is surprisingly good at producing very fluent, very grammatical text,” explained study co-author Liam Dugan, a PhD student at CIS. “But it does make mistakes. We prove that machines make distinctive types of errors — common-sense errors, relevance errors, reasoning errors and logical errors, for example — that we can learn how to spot.”
“People are anxious about AI for valid reasons,” added Prof. Callison-Burch, who led the research. “Our study gives points of evidence to allay these anxieties. Once we can harness our optimism about AI text generators, we will be able to devote attention to these tools’ capacity for helping us write more imaginative, more interesting texts.”
“My feeling at the moment is that these technologies are best suited for creative writing,” he continued. “News stories, term papers, or legal advice are bad use cases because there’s no guarantee of factuality.”
Dugan sees the positive in all this: “There are exciting positive directions that you can push this technology in. People are fixated on the worrisome examples, like plagiarism and fake news, but we know now that we can be training ourselves to be better readers and writers.”
So how good are you? To find out, play any of the game’s four categories (short stories, news articles, recipes, presidential speeches) containing thousands of texts! And remember, while honing your recognition skills, you’re also contributing to academic research.
Source: CORDIS
Leave a Reply