New sources of inaccuracy? A conceptual framework for studying AI hallucinations

---

Introduction

AI hallucinations are inaccurate outputs generated by AI tools, such as ChatGPT, Gemini, and Claude, that appear plausible but contain fabricated or inaccurate information (Augenstein et al., 2024). These inaccuracies can emerge from AI systems without deliberate human intent to deceive. Unlike traditional (human) communicators, AI lacks in intent or epistemic awareness that would allow it to recognize or prevent the generation of hallucinated content. Platform guardrails (e.g., internal filters) and retrieval-augmented generation (RAG) systems (e.g., the “search the web” function from many AI tools) can improve factual accuracy, yet hallucinations persist because these models generate language by predicting the next most likely word based on statistical patterns in training data. This characteristic makes AI hallucinations different from human-driven misinformation, which current research attributes to cognitive bias, motivated reasoning, or attempts to deceive.

AI hallucinations are at the boundary of what scholars define as misinformation (Schäfer, 2023). Are they just technical errors, or do they act like human-generated misinformation in influencing public decision-making? Existing interventions such as fact-checking (Krause et al., 2020), accuracy nudges (Pennycook et al., 2020), or alignment strategies (Gabriel, 2020) often assume the presence of an intentional communicator and are insufficient to address outputs that are not rooted in belief, persuasion, or manipulation. Here, I adapt the supply-and-demand framework from communication research: Supply refers to how messages are generated, and demand to how people interpret and react to them. On the supply side, addressing hallucinations requires multi-layered effort from knowledge boundaries, data limitations, generative mechanisms, and misaligned optimization goals. On the demand side, the tone and authoritative style from hallucinated outputs may invite trust or shallow processing and thus call for collaboration with human-computer interaction scholars and practitioners on levels of individuals (micro-level), groups (meso-level), and society (macro-level). As of August 2025, OpenAI states that the latest ChatGPT (GPT-5) makes “significant advances in reducing hallucinations” and is “significantly less likely to hallucinate” than prior models (OpenAI, 2025), but the performance remains uneven across tasks and contexts (often called the “artificial jagged intelligence”; see Fridman, 2025; Karpathy, 2024). As long as the core next-token prediction mechanism remains, hallucinations persist as an ongoing technical challenge for the supply side and an epistemic risk for the demand side. This framework allows for a more systematic analysis of how AI hallucinations are produced, interpreted, and potentially amplified across different domains of public communication.

Why AI hallucinations matter

AI is now widely embedded in processes of public knowledge formation such as online search, customer service, journalism, and scientific research (Reid, 2024). At least 46% of Americans report using AI tools for information seeking (IPSOS, 2025), though the real number may be higher. One survey found that while 99% of Americans had used a product with AI features, only 64% recognized they had done so (Maese, 2025). As a result, users may unknowingly rely on AI-generated content and assume it functions like a traditional information source. On the other hand, studies suggest that even the best-performing AI tools still generate false information at a non-zero baseline rate, regardless of how they are used (Kalai & Vempala, 2024; Vectara, 2024). However, this prevalence varies significantly by topic, and higher reliability may be achieved in domains with consolidated knowledge; for instance, there is episodic evidence that the rate of hallucinated academic references is considerably lower than prior studies (0.6%) when ChatGPT was queried on scientific topics with broad, established consensus (Volk et al., 2025).

The consequences of AI hallucinations are already visible across domains. In healthcare, OpenAI’s Whisper system fabricated misleading content in medical conversation transcriptions (Koenecke et al., 2024). Air Canada’s chatbot misled a customer about bereavement fares, leading to legal consequences (Moffatt v. Air Canada, 2024). In academia, hallucinated citations appeared in legal filings (Zhao, 2024) and editing tools introduced systematic terminology errors (Oransky, 2024). Media outlets have published AI-generated content containing historical inaccuracies (Owen, 2025; Reilly, 2025). Meanwhile, some scholars suggest that unpredictable or imaginative outputs may yield creative value (e.g., Pilcher & Tütüncü, 2025). The prospect of AI-driven creativity is promising, yet without a clear understanding of how such output emerges, the expected purposeful creativity remains, in practice, unintended and uncontrolled divergence. Also, given generative AI’s more prevalent role as personal organizer, health guide, and learning partner, where reliability is expected (IPSOS, 2025; Zao-Sanders, 2025), unintentional hallucinations still present ongoing risks for public understanding and decision-making.

AI hallucinations are technically and conceptually different from human misinformation

Before moving into an explication of why AI hallucinations are different, I would like to revisit the common definitions of misinformation. This paper adopts a broader definition of misinformation as content that contradicts the best available evidence, whether caused by lack of knowledge, unintentional errors, or deliberate deception (Scheufele & Krause, 2019). This umbrella term covers the narrower subsets of disinformation as intentional falsehoods (Fallis, 2015) and conspiracy theories as motivated rejections of consensus and attribution of secret intent (Uscinski et al., 2016), which can all impede effective communication (Jamison et al., 2020).

AI hallucinations result from multi-layered technical vulnerabilities that are different from how human misinformation emerges. Figure 1 illustrates these risk layers with the Swiss cheese analogy to demonstrate how vulnerabilities in each layer, when aligned, can lead to hallucinated outputs. First, training data often contain biases, omissions, or inconsistencies (Chen et al., 2024; Loukissas, 2019), which may embed systemic flaws into outputs. Recent interventions like RAG also face issues such as conflicting sources and poisoned retrieval (Fan et al., 2024; Zou et al., 2024). A popular case, as mentioned at the start of this essay, is Google AI treating April Fool’s satire as a fact. There is also a possibly degenerative “AI-on-AI” feedback loop where AI-generated inaccuracies will pollute future training data, leading to a phenomenon known as “model collapse” from a scarcity of fresh, human-generated content (Shumailov et al., 2024). Second, the training process is opaque, with limited information about what data are used and how they shape the model’s internal representations. This lack of traceability makes it difficult to explain or audit why a model produces specific outputs (Liao & Vaughan, 2024; Thirunavukarasu et al., 2023). Finally, downstream gatekeeping struggles to filter subtle hallucinations due to budget, volume, ambiguity, and context sensitivity concerns (e.g., Lu, 2025; Zhao, 2024). These layered vulnerabilities indicate that hallucinations are structurally inevitable, and they reflect a different production logic than human-generated misinformation. Among the three layers, gaps (or more intuitively, the holes in the cheese) in training data or weaknesses in gatekeeping can sometimes be observed from model prompts (e.g., asking AI something that it has not been trained on) or outputs (e.g., AI tools failed to detect and stop outputting harmful contents). But the training process remains largely opaque, making its vulnerabilities difficult to isolate or audit directly.

AI hallucinations also diverge conceptually from human misinformation in terms of agency and intent. Traditional frameworks for misinformation research focus on human actors and their motivations, beliefs, or knowledge boundaries. AI hallucinations, however, emerge from human-machine interactions (e.g., user prompts and model responses) that challenge this assumption. A circular spectrum is used here to illustrate the distributed agency of misinformation production. On the bottom arc, human-initiated inaccuracies involve deliberate deception or flawed prompts (e.g., Westerlund, 2019; Zamfirescu-Pereira et al., 2023), while human-influenced inaccuracies stem from value alignment and guardrails (e.g., when well-intentioned interventions produce unintended falsehoods; see Thorbecke & Duffy, 2024). In contrast, AI hallucinations are placed on the top arc, where human control is minimal. Although users initiate the process, such as entering prompts into AI tools, the specific falsehoods are emergent properties of the system architecture. These hallucinations are generated without explicit belief systems, epistemic intent, or communicative goals. This conceptual distinction challenges human-centric models and demands new conceptual tools for communication research and practice.

Not just a bug: An agenda for addressing the supply side of AI hallucinations

Understanding the “supply” of AI hallucinations involves examining the upstream conditions that lead to their generation before any audience interaction occurs. Four key areas of concern stand out. These four areas reflect three major vulnerabilities of hallucination risk discussed above (training data, training process, and gatekeeping). I further divide training data concerns into knowledge boundary concerns, where reliable human knowledge is lacking, and data logistics concerns, where credible knowledge exists but is inaccessible to the model.

Knowledge boundaries and uncertainty

AI tools often provide answers even when dealing with unsettled science or topics without credible ground truths (Augenstein et al., 2024). While science communicators already struggle with conveying uncertainty (Beets, 2024; Dunwoody et al., 2018; Peters & Dunwoody, 2016), it remains unclear how generative AI will navigate ambiguity. Interfaces may rely on small disclaimers (e.g., “AI may make mistakes”), but their effects on public trust are unknown. Even when scientific consensus exists, AI systems can mislead through (over)simplification or metaphor, repeating old issues in human-led communication (National Academies of Sciences, Engineering, and Medicine, 2024). A key question is whether AI’s simplifications are normatively helpful or distortive, particularly when they seem definitive.

Data logistics and biases

Training data of LLMs in powering AI systems often contain gaps (“data voids”) (Golebiewski & Boyd, 2018), systemic bias (Crawford, 2021), and quality inconsistencies (Wood & Forbes, 2024). These limitations may reinforce inequalities and generate biased outputs (e.g., Chen et al., 2024). Conversely, AI systems demonstrate greater reliability on topics supported by extensive, high-quality training data and a strong expert consensus, resulting in higher accuracy and fewer hallucinations for well-established scientific domains such as clinical models (Singhal et al., 2023). Yet privacy constraints (Voigt & von dem Bussche, 2017) and platform opacity (Brennen et al., 2025) restrict transparency, making it difficult to retrieve training data or assess AI content quality systematically. This lack of access leaves many findings anecdotal and impedes empirical study of how AI outputs vary across user groups or contexts (Krause et al., 2025).

Opacity of AI processes

The internal operations of LLMs remain opaque or “black boxes” (Bender et al., 2021; Weidinger et al., 2022). These outputs result from layered interactions between data, training processes, training objectives, and user prompts. Fine-tuning efforts that target one domain often lead to unintended distortions in others due to fragile interdependencies within the system (Betley et al., 2025). This opacity makes it difficult to anticipate how changes in each layer could propagate, and harder still to diagnose or isolate the origin of errors. Communication researchers can explore not just what is true but why systems generate certain responses and how people interpret them (Liao & Vaughan, 2024; Ozmen Garibay et al., 2023; Schäfer, 2023).

Gatekeeping and alignment trade-offs

Fact-checking struggles with subtle hallucinations like fake citations (Zhao, 2024), and efforts to align models with human values may sacrifice factual precision (Thorbecke & Duffy, 2024). Institutional policies often react after the fact (Haggart, 2023), and domain-specific tolerance for error varies (Lu, 2025; Rahman et al., 2025). Communication research can help define how “acceptable error” is negotiated across settings and audiences.

In sum, these upstream vulnerabilities manifest differently across sectors. For instance, knowledge ambiguity is particularly salient in public health and science communication, and the alignment trade-offs may be more visible in journalism. A more universal concern is the opacity of AI systems. This lack of visibility (see middle cheese slice in Figure 1) makes hallucinations difficult to trace; thus, transparency and explainability in model development are especially warranted.

More persuasive than misinformation? An agenda for studying the demand side of hallucination

A key question for the demand side is: what attributes make AI hallucinations persuasive to users? Hallucinations could be perceived as credible due to their fluency, coherence, and authoritative tone (Zhang et al., 2023). This speculative persuasiveness still warrants empirical validation. To structure such an inquiry, I propose a macro-meso-micro framework for future research and practices, which offers a heuristic lens to interpret how hallucinations emerge, circulate, and persist across institutional, group, and individual contexts (Krause et al., 2024; Serpa & Ferreira, 2019).

Macro-level: Institutional roles and media credibility

Traditional misinformation intervention methods, including fact-checking and accuracy nudges, were designed to counter misinformation with clear human sources (Costello et al., 2024; Pennycook et al., 2021; van der Linden, 2023). Hallucinations, on the other hand, are often digital artifacts that lack an identifiable author or agenda. These gaps in attribution and accountability may make hallucinated content less likely to be directly challenged. AI-specific solutions such as computational fact-checking or human-AI hybrid verification systems (Narayanan Venkit et al., 2024) still require testing across different content domains. Disclosure studies show mixed effects on perceived accuracy (Bien-Aimé et al., 2025; Li et al., 2025), suggesting a need for new transparency and credibility frameworks tailored to AI-generated information.

Meso-level: Group dynamics and online dissemination

Hallucinations can spread through group-level mechanisms like filter bubbles, echo chambers, or motivated reasoning (Cinelli et al., 2021; Hart & Nisbet, 2012; Jamieson & Cappella, 2008), even without a coordinated intention (Garrett, 2017). Case studies, such as the resignation of an academic editorial board due to hallucinated content (Oransky, 2024), suggest resistance to hallucination exists under certain conditions, but broader patterns remain unclear. Future research should trace how hallucinations are socially reinforced or rejected, and whether domain-specific vulnerabilities exist (Alkaissi & McFarlane, 2023; Schäfer, 2023).

Micro-level: Digital literacy, trust, and user behaviors

Users favor fast, accessible information sources, such as search engines (Hargittai, 2010), and the same may apply to the current AI tools. AI’s fluency and confident tone align with cognitive preferences for easily processed content (Markowitz, 2024; Petty & Cacioppo, 1986). Existing work shows that users form trust in AI based on fluency, tone, and perceived authority, often overlooking accuracy when corrections are absent (e.g., Anderl et al., 2024). These features encourage shallow engagement and may facilitate sycophantic outputs that confirm user expectations (Sharma et al., 2023). Even digitally literate users often rely on surface cues (Guess et al., 2020; Sirlin et al., 2021), and younger audiences may misjudge credibility (Menchen-Trevino & Hargittai, 2011; Wineburg et al., 2025). Besides literacy, trust in AI varies by personal political orientation (Yang et al., 2023), application domain (Eom et al., 2024), and national context (Greussing et al., 2025). Future research should examine how the hallucination feature of AI would affect trust, verification behaviors, and continued AI usage. What makes AI hallucinations distinct is their co-production: Individuals’ vague, under-specified, or conflicting prompts can increase the likelihood of hallucinated content (Zhang et al., 2023). This suggests a need for user education and interface design that encourages more structured and verifiable prompt construction.

Such tiered approaches have been applied or tested in journalism (e.g., byline disclosures) (Bien-Aimé et al., 2025), health (e.g., disclaimers of diagnosis) (Scaff et al., 2025), and research (e.g., AI use guidelines in academic writing) (Kwon, 2025). At the micro-level, such measures target users’ overreliance on fluency or trust cues. More attention is needed for meso-level interventions such as human-AI cross-checking to constrain hallucination spread (e.g., Nguyen et al., 2018), as well as the macro-level regulatory standards tailored to domain-specific risks (Eom et al., 2024). The recent U.S. congressional moratorium on state-level AI regulation and the subsequent bipartisan repeal (Morgan & Shepardson, 2025), for example, underscores the growing political consensus on AI regulations, but also reveals the difficulty of translating broad concerns into actionable governance frameworks.

Conclusion

As I have shown, hallucination is a new form of inaccuracy that is technically and conceptually different from misinformation. As a result, addressing hallucinations requires more than correcting factual errors. It calls for an integrated agenda that links upstream supply-side risks, such as training data flaws, system opacity, and weak gatekeeping, with downstream demand-side vulnerabilities in inadequate trust, limited AI literacy, and skewed collective interpretation. This agenda engages communication research alongside cognitive psychology, science and technology studies, and human-AI interaction, not only to mitigate harm but to understand how people assign meaning to AI-generated information. Identifying vulnerabilities, tracing effects, and adapting institutional responses are necessary to sustain credibility and public trust. However, this supply-and-demand framework in analyzing AI hallucinations remains provisional. First, the categories of “supply” and “demand” may overlap in practice, especially when users and AI co-produce hallucinations through prompt-response interactions. Second, prior summaries on misinformation interventions acknowledged that no single intervention point is sufficient to mitigate information risks (National Academies of Sciences, Engineering, and Medicine, 2024). By extension, addressing AI hallucinations will also require attention across multiple levels of the information environment. This framework should therefore be viewed as an initial organizing heuristic that invites refinement and empirical validation through future research.

There was such a warning over two decades ago: “Humans are not secure…If it [a transhuman AI] thinks both faster and better than a human, it can probably take over a human mind through a text-only terminal” (Yudkowsky, 2002). While today’s AI systems may not yet reach transhuman intelligence, the fluency, speed, and persuasive power are already challenging the stability of human knowledge-making processes. Addressing AI hallucinations involves more than detecting and correcting falsehoods; it now requires a forward-looking how the information ecosystem may evolve in response to a new generative computational agent.