AI chatbots, known for their tendency to cause hallucinations, could also induce hallucinations in humans, researchers claim.
Specifically, interactions with chatbots may increase the formation of false memories if the AI model provides incorrect information.
Computer scientists from MIT and the University of California, Irvine recently decided to investigate the impact of AI chatbots on human memory, citing an increase in research on AI-human interaction and an increase in scenarios in which humans engage with chatbots.
Focusing specifically on the possibility of using AI agents during interviews with witnesses to crimes (which, to our knowledge, is not currently practiced), the authors describe their findings in a preprint paper titled “Conversational AI powered by large-scale language models amplifies false memories in eyewitness interviews.”
Of particular concern is the dangerous potential for AI to contribute to the formation of false memories.
“Of particular concern is the dangerous potential for AI to contribute to the formation of false memories,” explain study authors Samantha Chan, Pat Pataranuthapong, Aditya Suri, Wazir Zulfiqar, Patty Mars, and Elizabeth F. Loftus. “This concern is further amplified by the known, yet unresolved, tendency of AI models to hallucinate or generate false information, whether intentionally or not.”
Previous studies have already pointed out the possibility that AI model interactions can mislead people. For example, a 2023 study, “The Inevitable Social Contagion of False Memories from Robots to Humans,” found that misinformation provided by social robots can become false memories, even if the robot provides warnings for the misinformation. And in its GPT-4o system card released last week, OpenAI noted the risks of anthropomorphism and emotional reliance on AI models, and how those risks can be amplified when models accurately mimic human voices.
Researchers from MIT and the University of California, Irvine investigated the memory effects of chatbot interactions in more detail by recruiting 200 participants and conducting a two-phase experiment.
In the first phase, participants watched a two-and-a-half-minute silent, unpauseable surveillance video of an armed robbery to simulate the experience of witnessing the crime. They were then presented with one of four experimental conditions “designed to systematically compare different mechanisms that affect memory.”
These included a control condition where participants simply answered follow-up questions with no intermediate steps, a survey condition where participants answered intentionally misleading questions via Google Forms, a pre-scripted chatbot condition where participants answered the same misleading questions from the Google Forms survey via an interface identified as a “Police AI Chatbot,” and a generated chatbot condition where participants answered misleading questions but the “Police AI Chatbot” provided feedback generated by GPT-4.
The process of reinforcing the generated chatbot conditions is explained as follows:
Then, in the second phase of the study, participants were assessed a week later to see if the induced false memories persisted.
In essence, the known risks of creating false memories (e.g., deliberately misleading questions) are exacerbated when the AI agent endorses and reinforces misconceptions. Consistent with previous research, false memories increased in the study condition in which researchers asked misleading questions, and after one week, those memories were present in 29.2 percent of respondents.
That means we don't need AI systems that create false memories in people when they ask leading, misleading questions: that's always been possible.
But adding generative AI chatbots magnified the problem of false memories.
“The novel contribution of our study is that we examined the impact of a generative AI chatbot on immediate false memory formation,” the paper explains. “In particular, the generative chatbot condition induced nearly three times the false memories observed in the control group and about 1.7 times more than the survey-based method, with 36.8% of responses misinterpreted as false memories one week later.”
The authors argue that their findings show how AI-driven interactions can affect human memory and “highlight the need for careful consideration when deploying such technologies in sensitive situations.”
The code for the research project is publicly available on GitHub.
