Increasing accuracy motivations using moral reframing does not reduce Republicans’ belief in false news

---

Implications

Academics, policymakers, and the general public are highly concerned about the spread of misinformation, and a great deal of research has sought interventions to combat misinformation (Lazer et al., 2018). These interventions have mostly been cognitive in nature. For example, past work has found that providing fact-checker warning labels (Pennycook et al., 2020) or corrective information before (Cook et al., 2017) or after (Lewandowsky et al., 2020; Persily & Tucker, 2020; Wittenberg & Berinsky, 2020) exposure reduces belief in falsehoods, as does providing simple digital literacy tips (Guess et al., 2020), engaging in reasoning (Bago et al., 2020), or activating an accuracy mindset (Salovich et al., 2022; Thakral et al., 2021). Similarly, prompting participants to consider accuracy increases the quality of news they share (Epstein et al., 2021; Pennycook et al., 2021).

Here, we consider an alternative approach to combatting misinformation that is rooted in motivation. A large literature on politically motivated reasoning, starting from Kunda (1990), argues that accuracy motives are in tension with “directional” (e.g., identity-based) motives that drive people to believe identity-congruent information and disbelieve identity-incongruent information. Thus, it could be that people believe false ideologically-congruent claims because they are motivated to believe these claims. If so, then increasing the strength of their accuracy motives should help to counteract these directional motives and thus reduce belief in ideologically congruent falsehoods (and increase belief in ideologically incongruent truths). 

In this paper, we attempt to increase the strength of accuracy motives among conservative Republicans in the United States, as prior research finds this group is more distrusting of mainstream news sources (Brenan, 2022; Gottfried, 2021; Pennycook & Rand, 2019), and thus have a higher possibility of turning to and internalizing (Garrett & Bond, 2021), as well as sharing (Grinberg et al., 2019; Guay et al., 2022; Guess et al., 2019) political misinformation, as compared to Democrats. To do so, we developed an intervention that frames accuracy as consistent with conservative ideology. In the United States, conservatives tend to place a higher value on group loyalty, respect for authority, and purity than liberals (Graham et al., 2009; Haidt & Graham, 2007). Persuasive appeals that frame political issue positions as consistent with these values have been shown to lead conservatives to report higher levels of support for a number of political positions, such as environmental protection/climate change (Bayes et al., 2020; Feinberg & Willer, 2013; Wolsko et al., 2016), same-sex marriage (Feinberg & Willer, 2015), immigration (Voelkel et al., 2022), making English the official language of the United States (Feinberg & Willer, 2015), countering vaccine hesitancy (Amin, et al., 2017), and economically progressive political candidates (Voelkel et al., 2022). Beyond moral values, messaging that invokes a common identity has been shown to be more persuasive (Chu et al., 2021; Jiang et al., 2010). Therefore, messages that frame having accurate political beliefs as being consistent with conservative political identity could be a fruitful means for reducing belief in fake news among conservative Republicans.

Specifically, we tested whether framing accurate engagement with political information as consistent with conservative ideology and values affects conservative Republicans’ ability to tell truth from falsehood when evaluating political news headlines. We followed most work on online misinformation in focusing on headlines rather than full articles because on social media, people mostly just read the headlines that appear in the newsfeed and only rarely click out to read full articles. As the headline typically contains the key claim, prior work has shown that participants can typically do fairly well at determining the accuracy of true versus false headlines, especially when they stop to think carefully (for a review, see Pennycook & Rand, 2021).

As expected, we found that (i) the conservative identity-framed intervention increased the self-reported value of accuracy, and (ii) participants who reported a higher value on accuracy were less likely to believe false headlines. Unexpectedly, however, there was no significant effect of the treatment on participants’ accuracy judgments of either true or false headlines. This ineffectiveness stands in contrast to other interventions—such as fact checks (Cook et al., 2017; Lewandowsky et al., 2020; Pennycook et al., 2020; Persily & Tucker, 2020; Wittenberg & Berinsky, 2020), digital literacy tips (Guess et al., 2020), deliberation manipulations (Bago et al., 2020), or financial incentives (Rathje et al., 2023)—which have been shown to significantly affect accuracy judgments. 

What might explain why the intervention we tested was ineffective? One interpretation is that accuracy motivation is only correlated with—but does not have a causal effect on—accuracy judgments, perhaps driven by some omitted third variable such as an increased ability to focus, identify patterns, or exert cognitive effort. In this account, those who exert greater cognitive effort, both indicate placing more value on accuracy and are less likely to believe false headlines, but the link is not causal. Another interpretation is that the link between valuing accuracy and disbelieving false news is causal, but the effect of our intervention on accuracy valuation, or the association between accuracy valuation and accuracy judgments, or both, are too weak to show an effect of our intervention.

It could also be the case that the small increase in accuracy value we produced was a demand effect. That is, our manipulation, which was very explicit on the value of being accurate, increased participants’ perception that it would be reputationally advantageous to signal they valued accuracy but did nothing to affect their actual valuation of accuracy. According to this explanation, the intervention only increased the desire to signal value, but that motivation was insufficient to also impact participants’ actual accuracy judgments in the study. Still, another possibility is that the intervention was sufficient to increase participants’ actual valuation of accuracy but that more is required for participants to translate that valuation into actual changes in accuracy judgments. For example, individuals may also require knowledge of common features of fake versus true headlines in order to capitalize on their valuation of accuracy and more reliably discern truth from falsehood. 

Whatever the reason, our results suggest that identity-consistent accuracy messaging may not be a promising tool for meaningfully reducing susceptibility to misinformation among conservative Republicans in the United States, at least as implemented here. This lack of efficacy is somewhat surprising, given the large body of evidence reviewed above for the effectiveness of this approach in persuading people on political issues. In a related vein, a recent study found that two different descriptive social norms interventions did not reduce sharing intentions for fake news (Epstein et al., 2021) despite the well-established power of descriptive social norms to affect a wide range of other behaviors (Cialdini et al., 2006). Together, these results raise questions about the role of motivation in driving belief in misinformation, as well as how well interventions based on social identity can be applied to the challenge of misinformation. 

This study has several limitations. First, we only conducted our research among conservative Republicans in the United States. It is possible that an identity-based approach could be more effective with other groups. Second, while the intervention tested here combined several identity-based methods (e.g., invoking political values, religious and political identity cues, etc.), it is possible that a different approach to increasing accuracy motivation that more effectively harnessed conservative identity, or which harnessed different identities, could move participants more effectively. Further work should test if this is the case.

Findings

Finding 1: Conservative Republicans who reported valuing accuracy more were less likely to believe false headlines.

As a manipulation check, we measured how much value participants placed on accuracy using our “importance of accuracy” scale consisting of an aggregate of 12 items, each using a 100-point scale (see Appendix Table 1). This importance of accuracy scale included questions about the importance of rejecting false information even when that information is ideologically consistent, and of accepting true information even when that information is ideologically inconsistent. Participants’ self-reported value of accuracy was significantly positively correlated with discerning true headlines from false headlines (interaction between value of accuracy and headline veracity (b = 0.227, 95% CI = [.186, .269]), p < .001);¹Disaggregating the “importance of accuracy” aggregate into subscales, we see the following: valuing the rejection of false information has no relationship with discernment for true discordant headlines (b = -0.015, p = .460), but does for concordant fake headlines (b = -0.145, p < .001). In contrast, valuing the acceptance of inconvenient truths had both a relationship with discernment for true discordant headlines (b = 0.05, p = .004) as well as concordant fake headlines (b = -0.056, p = .002). this was also the case when constraining to only those in the control condition (b = 0.203, 95% CI = [.152, .254], p < .001). Particularly, participants who placed more value on accuracy were less likely to believe false headlines (b = -0.252, 95% CI = [-.288, -.215], p < .001); and there was no relationship between valuing accuracy and believing true headlines (b = -0.024, 95% CI = [-.057, .009], p = .150). Further, the relationship between valuing accuracy and truth discernment (the difference in accuracy ratings for true relative to false headlines) showed no moderation by whether the headline was concordant or discordant with participants’ political identity (interaction between value of accuracy, headline veracity, and headline concordance (b = 0.008, 95% CI = [-.058, .074], p = .819). The association between valuing accuracy and truth discernment was significant—and the effect was of similar magnitude—both for headlines that were politically concordant (interaction between valuing accuracy X real, b = -0.224, 95% CI = [-.274, -.176], p < .001) and discordant (interaction between valuing accuracy X real, b = -0.231, 95% CI = [-.285, -.177], p < .001). These associations conceptually replicate prior work (Arechar et al., 2023) and confirm our intervention’s motivating assumption that those who are more concerned with accuracy are, in fact, better at discerning true versus false headlines.

Finding 2: An intervention that frames having accurate beliefs as consistent with conservative identity and values increased the value that conservative Republicans reported placing on accuracy.

By design, the components of the importance of accuracy scale are closely related to the contents of the intervention. This allows the scale to operate as a manipulation check, ensuring that the intervention successfully induces participants to place greater importance on accuracy. Indeed, when comparing responses to the importance of accuracy scale in the conservative-identity intervention treatment condition relative to the control, we found a small but significant positive effect of the treatment (b = 2.08 95% CI = [1.03, 3.133], p < .001, Cohen’s d = -0.173), with results nearly identical when adding demographic controls (b = 2.08, 95% CI = [1.05, 3.1], p < .001). As portrayed in Figure 1, this shows that our intervention was successful at increasing the value that our conservative Republican participants placed on accuracy, as we would expect, given this scale was designed to be closely related to the contents of the intervention.

Finding 3: The conservative identity-framed intervention did not make conservative Republicans better at identifying false versus true headlines.

When predicting participants’ accuracy ratings for the headlines, there was no significant two-way interaction between treatment and headline veracity (b = 0.01, 95% CI = [-.043, .063], p = .704). Thus, the intervention did not make participants more sensitive to the actual objective veracity of the headlines when judging their accuracy—or, put differently, the intervention did not improve participants’ overall truth discernment. There was also no significant three-way interaction between treatment, headline veracity, and whether the headlines were right-leaning or left-leaning (b = 0.03, 95% CI = [-.03, .09], p = .342), such that the treatment’s effects (or, in this case, non-effects) did not significantly vary by concordant versus discordant with the participants’ politics. More specifically, there was no significant treatment effect on accuracy ratings of left-leaning true headlines (b = -0.008, 95% CI = [-.06, .042], p= .730), which we expected the treatment would increase belief in, or right-leaning false headlines (b = -0.015, 95% CI = [-.081, .051], p = .653), which we expected the treatment would decrease belief in. All results are qualitatively equivalent when including demographic controls or when restricting to participants who spent more than median time reading the treatment or control essays (see Appendix Tables 2–6). As noted above, these results are surprising given previous work, as well as the above correlational relationship between trait accuracy value and discernment. As discussed above, this may be due to an unobserved third variable driving the correlational relationship, or it may be that our treatment was not strong enough to move performance despite moving reported value. Still, given the length and consistent messaging of the treatment, this null result is surprising.

For completeness, we also note that participants rated true headlines as more accurate than false headlines on average (b = 0.515, 95% CI = [.611, .418], p < .001), and rated concordant headlines as more accurate than discordant headlines (b = 0.214, 95% CI = [.081, .347], p = .002). These patterns replicate results found in previous studies (for a review, see Pennycook & Rand, 2021), suggesting that the null effect of the intervention was not due to idiosyncrasies of the headline set nor the experimental paradigm we used.

Methods

We collected data from online survey participants recruited through a large panel of previously surveyed Amazon Mechanical Turk workers who were vetted for attentiveness and Cloud Research participants, (information on the panel this study was run on can be found in Appendix Table. 7). We only included participants who self-identified as Republican and conservative (indicating 5–7 on a scale from 1 = Extremely liberal to 7 = Extremely conservative) and passed three attention checks (18% were dropped pretreatment for inattention). The final dataset included 2,009 participants (mean age = 41, 51% female, 82% White). We conducted this study in September and October of 2021. The study was pre-registered (https://osf.io/9suw7/?view_only=d56f0e59b9c34df5824ebd3dacace34d).

Participants were randomly assigned to the treatment or control condition. Participants in the treatment condition read an essay that connected accuracy to conservative identity and values, titled “Truth, an American Value.” Based on previous work (Feinberg & Willer, 2015), the essay tied accuracy to moral values found to be held more strongly by conservatives than by liberals, such as group loyalty (“Patriotic Americans put honesty and truth over their political party”), authority (e.g., “A patriot believes what is true, not what she or he would like to believe”), and purity (e.g., “We must protect the sanctity of the truth”). The essay also included references common to many American conservatives’ identity, such as quotes from the Bible (e.g., “Therefore each of you must put off falsehood and speak truthfully to your neighbor, for we are all members of one body,” Ephesians 4:25). The essay emphasized the need to accept true information even when inconvenient (“If the truth supports an opposing view, loyal, responsible Americans don’t dismiss this information”), and to reject false information even if one might be inclined to believe it (“When we accept information without question, just because it favors our position, it corrupts the virtue of our side”). (The full example of the essay can be found in Appendix Figure 2). Participants in the control group read an article of a similar length but about an unrelated, non-political topic (i.e., the history of neckties). 

Participants then proceeded to the main task: rating the accuracy of a series of actual news headlines. Based on a previous experimental paradigm (Pennycook & Rand, 2019), all participants rated a total of 20 headlines about political news (see Figure 3 for an example), randomly selected from a large pool of 100 headlines. The headlines were sampled such that half were true and half were rated false by the professional fact-checking website Snopes.com. Additionally, half were right-leaning and half were left-leaning (based on a pre-test in which a separate sample of participants indicated how good for the Democrats versus Republicans the headline would be if it was true).²Exact wording: Assuming the above headline is entirely accurate, how favorable would it be to Democrats versus Republicans? Taken from: Chen, Pennycook & Rand, 2021. Participants rated the accuracy of each headline on a 4-point scale from 1 (Not at all accurate) to 4 (Very accurate). Headlines were displayed in a random order.

Finally, for our importance of accuracy scale, participants answered 12 items, each using a 100-point scale, measuring their self-reported value of accuracy, including the importance of rejecting false information (e.g., “People should reject false political information, even if the information makes them feel comfortable”) and accepting true information (e.g., “It is important to accept true political information, even if it feels wrong”). We piloted this paradigm several times, and the results of the piloting are reported in the supplementary materials (all items and subscales can be found in the Appendix). 

The main analysis tested the effect of condition on truth discernment. To do so, we predicted accuracy rating using an OLS regression with two-way clustered standard errors clustered on participant and headline. We included a dummy variable for treatment, a center-coded dummy variable for headline veracity, a center-coded dummy variable for headline political lean (concordant versus discordant), and all interactions. A significant positive two-way interaction between treatment and headline veracity would indicate that the treatment increases sensitivity to the veracity when rating accuracy (i.e., increases truth discernment). A significant negative two-way interaction between treatment and concordance would indicate that the treatment decreases preferential belief in concordant headlines. A significant three-way interaction would indicate that the effect on the treatment on truth discernment varies based on concordance.