Disagreement as a way to study misinformation and its effects

---

Implications

Experts and the public both consider misinformation a significant societal concern (Altay, Berriche, Heuer, et al., 2023; Ecker et al., 2024; McCorkindale, 2023), yet it remains a vague concept with inconsistent definitions (Adams et al., 2023; Altay, Berriche, & Acerbi, 2023; Southwell et al., 2022; Vraga & Bode, 2020) and a weak relationship with the societal or individual problems studied (Altay, Berriche, & Acerbi, 2023). These problems include polarization, erosion of institutions, problematic behavior, and individual and public health issues (Adams et al., 2023; Rocha et al., 2023; Tay et al., 2024). We refer to these problems as misinformation effects to indicate that they are assumed to be the effects of the spread of misinformation.¹Based on our analysis, we should refer to these as disagreement effects. However, for readability, we will use misinformation effects throughout the paper. The gaps in prevailing definitions and theories of misinformation (Pasquetto et al., 2024) impact the analysis of misinformation effects (Altay, Berriche, & Acerbi, 2023), the development of intervention strategies (Aghajari et al., 2023), and the quantification of effects as a way to evaluate interventions (Southwell et al., 2022). We propose here disagreement as a more effective framework for studying these effects.

Given the variety of definitions for misinformation (Adams et al., 2023; Krause et al., 2022), we focus on the prevailing conceptualization: false and misleading information about factual matters that—whether intentionally or unintentionally—leads to or reinforces false beliefs (e.g., Altay, Berriche, Heuer, et al., 2023; Humprecht et al., 2020; Kennedy et al., 2022; Lazer et al., 2018; Southwell et al., 2022), serving as both a cause and effect of the aforementioned problems (see Figure 1).²Therefore, we exclude research that examines types of misinformation that link to problems without individuals holding false beliefs, such as inherently harmful information (e.g., toxic or discriminatory language). While misinformation focuses on factual matters consumed at the individual level, the misinformation effects involve and are shaped by normative and societal factors beyond misinformation itself, such as opinions, values, epistemic positions, social norms, and ingroup and outgroup identity (see Figure 2).³We are aware that recent studies sometimes use broader analytical frameworks, as we discuss in Conceptual Contribution 1. However, the primary focus in misinformation research remains on mitigating the spread of false and misleading information (Aghajari et al., 2023; e.g.,Kozyreva et al., 2024). Therefore, this is the standard against which we compare our proposed approach. In particular, the inclusion of misleading information makes identifying misinformation a highly context-sensitive task, often relying on a weak form of objectivity that can approach pure subjectivity (Krause et al., 2022; Uscinski, 2023). To clarify, we stand firmly behind scientific objectivity, viewing objectivity as epistemic rather than as metaphysical absolute value, borrowing from Frost-Arnold’s (2023) definition: “a measurement of the degree to which bias is managed within a community of believers” (see also Longino, 2002). This subjectivity, combined with the conceptual misalignment, results in four key limitations when studying misinformation effects.

First, since misinformation is only one of many interdependent factors influencing beliefs and other misinformation effects (Altay, Berriche, & Acerbi, 2023; Tay et al., 2024), the current analytical framework provides a limited explanation for these effects: Someone can hold false beliefs without consuming misinformation (Altay, Berriche, & Acerbi, 2023), and the prevalence of misinformation does not necessarily indicate the prevalence of its effects (Southwell et al., 2022). Second, this weak relationship with misinformation limits the development of effective intervention strategies to counter misinformation effects and their associated harm (Aghajari et al., 2023), with experts skeptical about the efficacy of interventions to misinformation as a way to mitigate its effects (Acerbi et al., 2022; Altay, Berriche, Heuer, et al., 2023; Murphy et al., 2023). Third, the subjectivity in identifying misinformation, along with its negative connotation, creates a conflict of interest for the researcher and renders misinformation a highly value-laden concept (Kharazian et al., 2024; Uscinski, 2023; Williams, 2024; Yee, 2023). Fourth, due to the context dependency in identifying misinformation (and limitations in data), there is no reliable method to quantify misinformation, false beliefs, and misinformation effects (Southwell et al., 2022). This limitation means we cannot answer straightforward questions such as “Are we seeing less or more misinformation or misinformation effects over time?” For example, many misinformation experts believe we live in a post-truth era (Altay, Berriche, Heuer, et al., 2023), but we do not have a method to verify it.

Given the limitations of the prevailing misinformation concept, developing a more effective framework for understanding how misinformation effects emerge, evolve, and can be mitigated is essential. Here, we propose framing misinformation effects as symptoms (and effects) of social disagreement, characterized by conflicting attitudes among individuals and communities. First, disagreement is a holistic concept that aligns better with misinformation effects. As a consequence, disagreement is necessary for these effects, while the prevalence of misinformation itself or false beliefs is not. Second, disagreement is shaped by disagreement factors, such as values, beliefs, and epistemologies, which are consistent with factors recent misinformation research has identified as significant contributors to misinformation effects (e.g., Altay, Berriche, & Acerbi, 2023; Aghajari et al., 2023), but they are inherently excluded by the prevailing concept. Third, disagreement acknowledges that individuals with differing values and epistemologies may reach different conclusions about the truth, requiring researchers to assess when disagreement causes harm. Like parents who may not focus on who is right or wrong when their children argue, a disagreement researcher canignore the topic and still study the social dynamics and effects. Fourth, due to its conceptual alignment and because its identification does not require contextual analysis, disagreement offers a more practical way to measure (the risk of) misinformation effects, which can be fully automated, as we demonstrate. In this paper, we argue that, compared to misinformation, disagreement can offer greater explanatory power in analyzing misinformation effects, increased productiveness in developing new intervention strategies, and improved predictive precision in quantifying these effects (see “cognitive values” in Douglas, 2009), as we explain in the following three subsections. Additionally, in the Findings section, we demonstrate how this framework addresses three key research questions: 1) Why are Republicans more susceptible to misinformation than Democrats? 2) How can we counteract effects like vaccine hesitancy and ideological polarization? and 3) Do we live in a post-truth era? While the main focus of our paper is conceptual, our quantitative findings demonstrate that the proposed disagreement framework can also be applied to quantitative analysis. Due to space constraints, these quantitative contributions are detailed in the Appendix.

The purpose of this paper is neither to challenge misinformation research nor to discourage researchers from adopting a normative stance. Rather, since disagreement encompasses and incorporates misinformation as a factor, it can serve as a bridging concept (Chadwick & Stanyer, 2022) between in-depth, localized research focused on the spreadof misinformation (Pasquetto et al., 2024) and research focused on its effects. Our suggestion to focus on disagreement should not be mistaken for epistemological relativism (i.e., the idea that knowledge is only valid relative to a specific social or cultural situation), which is problematic as it would prevent researchers from making objective claims (Douglas, 2009; Longino, 2002). While researchers often rely on weak objectivity when assessing whether a text negatively affects someone’s beliefs, there is commonly strong scientific evidence to label the associated beliefs and behaviors as false—such as racist attitudes or vaccine refusal. Thus, not all positions within a given disagreement are equal. However, as we argue in this paper, even when scientific evidence clearly supports one side of a disagreement, the proposed framework remains valuable for understanding and mitigating its effects. At the end of this section, we provide recommendations for researchers on how to navigate these challenges.

Disagreement provides better explanations for misinformation effects

By elevating the conceptual focus from the content of information at the individual level (misinformation) to the overarching epistemic tensions between different communities (disagreement), the analytical framework aligns more effectively with the misinformation effects, see Figure 2. Because disagreement inherently recognizes the significance of normative and societal elements (see Figure 3) and of conflicting attitudes when studying the misinformation effects, it provides distinct explanations when compared to the concept of misinformation, see Table 1. For example, if one is interested in analyzing the influence of (fake) experts on vaccine hesitancy, the misinformation concept requires researchers to explicitly include such interdependent factors (Harris et al., 2024). In contrast, the holistic disagreement framework inherently incorporates these elements, compelling researchers to compare and evaluate the significance of individual factors for the misinformation effects under study. Taking factors such as values, identity, and epistemologies into consideration, it becomes less surprising why individuals may arrive at conclusions about given data and believe certain ideas that scientists deem misinformation (Douglas, 2009; Furman, 2023). For instance, misogynists may believe stories that target women, racists may believe stories that target immigrants, or Republicans may be more susceptible to misinformation than Democrats due to their stronger disagreement with science as an institution and its members (see Conceptual Contribution 5). Understanding the strength, nature, and subject matter of disagreement can help analyze these problems and develop intervention strategies.

Disagreement identifies interventions against misinformation effects

Being one of many drivers, addressing misinformation alone will not eliminate the misinformation effects. Despite the significance of other factors, there are relatively fewer studies that focus on interventions at the societal or normative level, such as leveraging social norms and their perception to address collective action problems (Aghajari et al., 2023; Andı & Akesson, 2021; Nyborg et al., 2016) or increasing trust in institutions (Acerbi et al., 2022).

Because the disagreement framework asks distinct questions (see Table 1), it can help identify new interventions for addressing misinformation effects. Additionally, as a holistic framework, it facilitates the balancing of a combination of interdependent strategies, recognizing that individual interventions may have limited impact (Altay, Berriche, Heuer, et al., 2023; Bak-Coleman et al., 2022). To illustrate, we first reevaluate nine well-studied intervention strategies (Kozyreva et al., 2024) using the disagreement framework (Conceptual Contribution 6). We find that four of these strategies may be less effective because simply being aware of a disagreement does not necessarily change one’s belief (Kelly, 2005). Second, we demonstrate how the questions from Table 1 can be used to develop and balance (new) intervention strategies (see Conceptual Contribution 7). For vaccine hesitancy, the proposed framework prompts questions such as “What are the reasons for individuals to disagree with governmental recommendations to get vaccinated?” and “How can individuals be motivated to get vaccinated while accepting their disagreement with governmental recommendations?” At times, calling out misinformation can help align beliefs or actions. However, considering disagreement factors, strategies like focusing on shared goals may be more effective (Friedman & Hendry, 2019). For example, instead of labeling vaccine misinformation, social media platforms could identify conflicting posts, which we argue is easier to automate than detecting whether a post is true or not and then append those conflicting posts with notes emphasizing shared health goals such as: “We observe the circulation of rumors about vaccines. We all want to save lives.”

Disagreement measures misinformation effects

In January of 2024, the World Economic Forum (WEF) identified misinformation and its effects as one of the top risks faced in this year (World Economic Forum, 2024). How can we determine the long-term effectiveness of intervention strategies, assess whether emergent technologies like artificial intelligence (AI) or social media have increased the problem, or compare its progression between countries (Humprecht, 2019)?

Due to its subjectivity and context dependency, identifying misinformation requires human judgment and topic analysis (see Conceptual Contribution 2). As a consequence, measuring misinformation can only be done for known narratives and requires significant effort (Southwell et al., 2022). Even approaches leveraging AI rely on human-labeled datasets used for pretraining such models. These limitations contrast sharply with the considered concern and rapid dissemination. Moreover, when viewing misinformation as a general, unspecific problem—for example, the concern that we live in a post-truth era (Altay, Berriche, Heuer, et al., 2023)—there is currently no approach to track its change over time other than relying on the opinions of experts (Altay, Berriche, Heuer, et al., 2023).

Quantifying the (risk of) misinformation effects through disagreement strength instead of counting misinformation comes with two key advantages: First, its independence of context allows to automate measurements across topics and times; in particular, it enables capturing misinformation effects as a general problem, including effects stemming from unknown or emerging narratives (e.g., polarization). Second, if misinformation effects are more accurately captured by the social phenomenon of disagreement than by instances of misinformation, as we argue here, then disagreement serves as a better indicator.

To demonstrate the first advantage, we conducted a longitudinal disagreement measurement in text in letters to the editor of The New York Times (NYT) from 1950–2022 based on a lexicon-induced (negative) sentiment analysis approach (see Quantitative Finding 1). We found an increase in disagreement since 2006, which aligns with the prevailing assumption that we live in the post-truth era (Altay, Berriche, Heuer, et al., 2023).  A large portion of the dataset used (1950–2007) originates from a prior study on conspiracy theories (Uscinski & Parent, 2014). While our disagreement measurement did not require any human labeling, the authors of the original study manually classified each letter to determine whether it contained misinformation (engagement with conspiracy theories).

Due to the lack of ground truth data for misinformation effects, we can only provide preliminary quantitative evidence that disagreement serves as a better indicator. In Appendix B, we include a comparison between disagreement and misinformation measurements for two longitudinal text datasets (letters to the editor of the NYT and vaccine-related tweets on Twitter). For both datasets, our analysis reveals a significant correlation, providing preliminary evidence that disagreement and misinformation are associated and may approximate the same effects. Furthermore, we found that vaccine-related disagreement decreased between January and March 2021 and increased between April and September 2021, confirming vaccine hesitancy analyses by Liu & Li (2021) and Chang et al. (2024), respectively. More data on misinformation effects is needed to validate whether disagreement serves as a better indicator of these effects than simply counting instances of misinformation or related concepts. Likely, the ideal measure depends on the research endeavor and might even be a combination of various approaches. Disagreement offers a framework for longitudinal, cross-national, and topic-agnostic analyses of misinformation effects, addressing a need identified by experts (Altay, Berriche, Heuer, et al., 2023).

In addition to social media and letters to newspaper editors, the disagreement measurement approach can be applied to other text data where individuals express their attitudes, enabling us to track misinformation effects in both specific and broader contexts. For example, it can be used with Wikipedia edit histories, commentary on government websites, reviews of health-related books, online forums, and comments on news sites, among others. Additionally, measuring unspecific disagreement can serve as a warning system for emerging misinformation effects. Instead of text analysis, disagreement can be analyzed through surveys, either by directly assessing perceived disagreement or indirectly through questions about values, beliefs, and interests (Voelkel et al., 2024; Akiyama et al., 2016).

Recommendations for misinformation researchers and practitioners

Identifying and studying misinformation is complex and context-sensitive. Some scholars argue that misinformation should not be studied at all (Williams, 2024), while others call for clear, context-free definitions (J. Uscinski et al., 2024). We propose a different approach: incorporating context by studying the effects within the overarching disagreement framework. Therefore, the purpose of this paper is neither to challenge misinformation research nor to discourage researchers from taking a clear stance within a disagreement. Compared to evaluating negatively connotated misinformation,⁴It is important to note that not all misinformation has a negative impact, despite its negative connotation. For example, falsely claiming a sports event like the Olympics occurred in a different location may not cause harm. determining whether two communities are in disagreement requires less evidence and is less value-laden, allowing for analysis without normative judgment. This neutral perspective is particularly relevant for studying misinformation effects that do not fit neatly a simple true/false dichotomy, such as polarization and rumors, or when access to data and communities is insufficient to assess whether specific information is misleading—such as in longitudinal, cross-community, and cross-topic analyses and measurements. At this high level, studying disagreement is like studying the climate (e.g., earth temperature). Both are inherently neutral phenomena, but they can become problematic when they change or reach certain levels, making it crucial for researchers to understand how their intensity relates to potential harm. Even when researchers focus on the spread of misinformation itself—rather than its effects—the disagreement framework can still add value by capturing the overarching social dynamics and key factors that explain why individuals believe and share information that researchers evaluate as misleading. Additionally, disagreement serves as a bridging concept (Chadwick & Stanyer, 2022) between in-depth, localized research (Pasquetto et al.,2024) and broader, high-level analyses, as well as with related disciplines such as philosophy, computational linguists, psychology, and political science (e.g., Christensen, 2007; Lamers et al., 2021; Rosenthal & McKeown, 2015; Weinzierl et al., 2021; Landemore, 2017). Social crises drive social resistance, inevitably resulting in disagreement, but often also leading to rumors, false beliefs, and misinformation (Prooijen & Douglas, 2017). Disagreement can drive positive change (e.g., women’s rights, Black Lives Matter) or negative outcomes (e.g., distrust in democracy). Depending on scientific evidence and researcher’s normative stance, the goal may be either to resolve the disagreement independent of absolute positions or to bring the disagreeing community into alignment with the researcher’s epistemology. When taking a stance against claims (e.g., “vaccines cause autism”) or communities’ actions (e.g., vaccine hesitancy), we recommend that researchers explicitly define the potential harmful consequences of these claims or actions. Clearly articulating these harms can help justify the need to bring the disagreeing community into alignment with the researcher’s position.

Findings

Conceptual Contribution 1: Conceptual misalignment between the narrow focus of misinformation and the broad spectrum of misinformation effects.

The focus on misinformation and its underlying causal pathway (misinformation—individuals holding false beliefs—misinformation effects) does not account for the temporal and positional relativity of truth and fails to capture the normative and societal contributors to misinformation effects. For certain misinformation effects such as polarization or rumors, the current concept is not merely narrow but also misdirected. Most obviously, excluding the individuals holding true beliefs overlooks the impact that conflicting viewpoints (from experts and peers) have on our understanding of knowledge and belief. Adopting an attitude in favor of specific (mis)information not only positions oneself for a particular community but also against the opposing one (Pereira et al., 2021).

Figure 2 categorizes misinformation effects based on their matter (factual and normative) and level of occurrence of the underlying social dynamics (individual and societal). For example, ideological polarization is defined as “a situation under which opinions on an issue are opposed to some theoretical maximum” (Au et al., 2022, p. 1331). By definition, this corresponds to an effect involving normative matters (opinions) at the societal level, which does not necessarily require the presence of misinformation or false beliefs. While effects like problematic behavior due to false beliefs can occur at the individual level, effects like polarization, rumors, and environmental issues only manifest when a large number of people are involved. Moreover, many effects, such as rumors, polarization, distrust, and psychological effects (e.g., distress and discomfort) (Ilies et al., 2011; Rocha et al., 2023; Susmann & Wegener, 2023) do not primarily hinge on individuals holding false beliefs but rather on individuals or communities holding conflicting attitudes. These misinformation effects become problematic only when associated activities cause harm. For instance, polarization or distrust in institutions can be beneficial to a certain extent. The concept of misinformation, often seen as inherently negative, fails to acknowledge these nuances.

Given the broad range of social, political, and psychological phenomena representing misinformation effects, it becomes evident that interdependent factors beyond misinformation significantly influence these effects (Southwell et al., 2020). For example, a survey among misinformation experts highlights partisanship, identity, confirmation bias, motivated reasoning, and distrust in institutions as major contributors (Altay, Berriche, Heuer, et al., 2023). While sometimes acknowledged as context, they are often only peripherally considered as contributors to the spread of misinformation rather than directly to its effects. Our literature review suggests that these factors are as crucial as misinformation itself and should be directly incorporated into analyses. Figure 3 categorizes these factors by their nature and level of occurrence. They include opinions (Ancona et al., 2022; Facciolà et al., 2019; Koudenburg & Kashima, 2022; Niu et al., 2022), the epistemic position of individuals who hold true or false belief such as experts (Bongiorno, 2021), fake-experts (Harris et al., 2024; Lewandowsky et al., 2017; Schmid-Petri & Bürger, 2022), and influencers in general (Lofft, 2020), dispositional beliefs (Altay, Berriche, & Acerbi, 2023), the framing of information and experiences (Goffman, 1974; Starbird, 2023; Zade et al., 2024), ingroup and outgroup identity (Altay, Berriche, Heuer, et al., 2023; Pereira et al., 2021; Van Bavel et al., 2024), number of people involved in disagreement related to social norms and perceived norms (Aghajari et al., 2024; Andı & Akesson, 2021; Andrighetto & Vriens, 2022), shared and conflicting values and aesthetics (Aghajari et al., 2023), attitudes (Susmann & Wegener, 2023; Xu et al., 2023), social epistemologies (Bernecker et al., 2021; Furman, 2023; Pennycook & Rand, 2019; Uscinski et al., 2024; Wakeham, 2017), trust in institutions (Humprecht, 2023), disagreeing views (Mazepus et al., 2023; Uscinski, 2023) and so on.

Conceptual Contribution 2: Identifying misinformation relies on human judgment.

The conceptual misalignment has expanded the focus from scientifically false information to misleading information. However, whether and how information changes one’s belief is not self-evident but context-dependent and somewhat subject to researchers’ judgment. Consequently, there is no reliable, generalizable procedure to identify misinformation (Uscinski, 2023), limiting the approaches to automate quantification of misinformation and its effects (Southwell et al., 2022). Furthermore, in combination with the negative connotation of misinformation, it can create a conflict of interest for researchers (Uscinski, 2023) and epistemic injustice in misinformation detection (Neumann et al., 2022).

Conceptual Contribution 3: Disagreement aligns with misinformation effects.

In alignment with misinformation effects, disagreement is shaped by disagreement factors, involving both factual and normative matters, like opinions, values, and beliefs (Frances & Matheson, 2018; Furman, 2023) and can occur at both individual and societal levels. Unlike misinformation, it acknowledges the significance of conflicting attitudes in studying and mitigating misinformation effects (Mazepus et al., 2023), making it a necessary condition for these effects.

Disagreement occurs when at least two individuals adopt conflicting doxastic (belief-related)⁵Other types of conflicting attitudes can be converted to a doxastic disagreement (Zeman, 2020). attitudes toward the same content (Zeman, 2020). These attitudes—belief, disbelief, and suspension of judgment—can vary in confidence. The strength of disagreement depends on the disparity in attitudes, confidence levels, and the number of people involved, serving as a risk indicator for misinformation effects. Disagreement factors, which create and influence disagreements, correspond with contributors to misinformation effects (see Figure 3). Misinformation can be seen as a disagreement factor that corresponds to evidence (Frances & Matheson, 2018).

Conceptual Contribution 4: Disagreement is necessary for misinformation effects, while misinformation itself is not.

In the prevailing concept of misinformation, individuals holding false beliefs link misinformation to its effects. However, false beliefs cannot exist without true beliefs held by others, leading to a disagreement between those who believe the misinformation and those who do not—forming two disagreeing communities. Typically, researchers align with the true-belief community. Instead of dividing by false/true beliefs, communities can also be split between those benefiting from and those harmed by the misinformation. For example, the claim that vaccines cause autism aligns with the views of those who believe vaccines are unsafe. These divided communities are crucial: misinformation only matters if some believe it and seek to persuade others to believe it (i.e., disinformation), while new ideas accepted by everyone and in particular the scientific community are considered knowledge gain, not false belief. Hence, disagreement is necessary (but not sufficient) for false belief and the misinformation effects.⁶False belief is necessary but not sufficient for misinformation.

Conceptual Contribution 5: Republicans are more susceptible to misinformation due to their disagreement with science.

Studies indicate that Republicans are more susceptible to misinformation than Democrats (Pennycook & Rand, 2019; Pereira et al., 2021). The disagreement framework may explain this difference. Differences in values, beliefs, and epistemic methods lead to varying conclusions about which information is relevant, which questions to ask, and how to interpret data (Douglas, 2009; Furman, 2023; Longino, 2002). Since Republicans tend to disagree more with the scientific community than Democrats (Evans & Hargittai, 2020; Lee, 2021), we can expect Republicans to differ more frequently from scientists in assessing information veracity. Conversely, if individuals with background assumptions more aligned with Republicans were responsible for distinguishing between true and misleading information, Democrats could be perceived as more susceptible to misinformation. However, we do not argue that misinformation or false beliefs are arbitrary or unworthy of study, as discussed in the Implications section.

Conceptual Contribution 6: Disagreement framework reveals limitations of current intervention strategies.

We reevaluate nine main intervention strategies from a recent high-profile paper (Kozyreva et al., 2024) viewed within the disagreement framework. Four of these strategies (row 3 in Table 2) may be less effective because they merely inform individuals that other individuals, communities, and institutions (e.g., science, media, government) disagree with their beliefs. However, simply being aware of a disagreement does not necessarily change an individual’s belief (Kelly, 2005; Southwell et al., 2019); it largely depends on their epistemic relationship with the disagreeing community (Frances, 2014). If individuals do not perceive peers, fact-checkers, scientists, or the majority (social norm) as epistemically superior (better positioned for judgment), it is reasonable for them to remain steadfast in their beliefs. Thus, if individuals disagree with science, such as on the safety of vaccines, there are broadly three possible explanations: they are not aware of the scientific consensus on vaccine safety, they do not perceive scientists as epistemic superior, or they do not respond to disagreement in a reasonable way—unreasonable from a scientific perspective. Regardless of someone’s relationship with the disagreeing community, intervention strategies like media literacy tips can help reduce disagreement by aligning individual epistemologies with scientific epistemology.

Conceptual Contribution 7: Disagreement offers new intervention strategies exemplified for vaccine hesitancy and ideological polarization.

Mitigating the spread of misinformation about vaccines and other factual matters can help decrease effects such as vaccine hesitancy or ideological polarization. However, there are many other reasons beyond false beliefs why individuals might disagree with governmental recommendations to get vaccinated or why societies might develop strongly opposed opinions (Au et al., 2022). In Appendix A, we demonstrate how the proposed disagreement framework, along with the related guiding questions and measurement approaches (see Table 1), can help identify and balance intervention tactics.

Quantitative Finding 1: Disagreement increased since 2006.

Disagreement offers a way to measure the risk of misinformation effects. Based on the widely accepted idea that we live in the age of misinformation (Altay, Berriche, Heuer, et al., 2023), we expect general, unspecific disagreement to be higher in the current decade compared to previous times. Figure 4 shows the time series of yearly disagreement scores from 1950–2022 in letters to the editor of the NYT. Disagreement has increased since 2006, providing preliminary evidence for the idea that effects of misinformation have increased in the past years. We used an automatic and scalable approach based on lexicon-induced (negative) sentiment analysis to measure disagreement in the text of letters to the editor of the NYT. Disagreement was measured by averaging the negative sentiment scores of about 1,000 letters per year, normalized to a scale between 0 and 1 across all years. For the years 1950–2007, we used the letters dataset from Uscinski & Parent (2014) to compare with their manually derived yearly misinformation score. To extend the coverage to 2022, we expanded the dataset with letters accessed through ProQuest.⁷https://www.proquest.com/ The approach is detailed in Appendix B along with a longitudinal disagreement measurement of vaccine-related tweets.

Methods

As misinformation researchers ourselves, we grapple with the issues of the prevailing analytical framework, as raised by scholars both within and outside the field (Pasquetto et al., 2024). Aiming to address the critique and to move the field forward, we chose the following approach. First, in reviewing the literature on misinformation, we aimed to precisely identify the gaps between definitions and theories of misinformation (causes) and the problems under study (effects). Second, we reviewed related literature on belief formation and epistemology to synthesize an analytical framework that aligns better with misinformation effects. Finally, by comparing the two frameworks on three typical research questions (Conceptual Contributions 5 and 7, and Quantitative Finding 1), we aimed to demonstrate the effectiveness of the identified disagreement framework.

Limitations and recommendations for future research

Our argument for using disagreement over the misinformation concept is based on a fundamental comparison of common notions of both concepts. We define disagreement in a basic, intuitive way (Zeman, 2020), where at least two individuals hold different attitudes or confidence levels toward a specific proposition. We did not explore other notions of disagreement, leaving the ideal definition for studying misinformation effects to future research.

We use a conceptualization of misinformation that is narrow and only includes the kind where individuals believe it. Misinformation that doesn’t include potential changes in belief is unlikely to spread widely in society or cause something like vaccine hesitancy. Once individuals believe the misinformation or actively seek to persuade others, it then can be captured by our proposed disagreement framework.

Building on our framework, we also introduce disagreement measurement as a risk indicator for misinformation effects. The outcome and interpretation depend heavily on the method and the social ecosystem analyzed. Our approach measures expressed disagreement in aggregated text data, which majority attitudes and neglects minority perspectives in society. Using negative sentiment analysis provides a rough estimate suitable for longitudinal comparison. We leave the development of more accurate measures and the exploration of alternative sources for analyzing disagreement to future research. Therefore, we view our empirical results as a demonstration rather than a benchmark for disagreement measurement.