In 2010, the journal Nature (2010, 133) published an editorial stating: “there is a growing anti-science streak … that could have tangible societal and political impacts.” About a decade later, the foreword to Naomi Oreskes's book Why Trust Science? begins with “Science confronts a public crisis of trust” (Macedo Reference Macedo and Oreskes2019, 1). The widespread concern is that politics now overwhelms science. This makes the communication of science—on such topics as climate change, evolution, genetically modified foods, obesity, stem-cell research, and more—a political challenge. COVID-19 reactions epitomized politicization, with partisanship becoming such a driver of health decisions that it was “pernicious enough to threaten the health of citizens” (Gollwitzer et al. Reference Gollwitzer2020, 1195). A burgeoning literature explores three distinct approaches to communication on scientific topics in politicized settings (Bayes et al. Reference Bayes2020), with each facing major political hurdles due to polarization.
First is research examining how scientific experts can influence opinions. While some have shown these effects (van der Linden et al. Reference Van der Linden2015), others question the influence of communications by scientists and/or experts (that is, when such sources serve as expert cues) (Akin and Scheufele Reference Akin, Scheufele, Jamieson, Kahan and Scheufele2017, 25). The pessimism stems from an expectation that on politicized topics, ideology and/or partisan identities lead people to reject science communication counter to their beliefs and identities (see, for example, Kahan Reference Kahan2015). An alternative approach in light of these challenges is framing moral values, such as patriotism or harm prevention, to shift preferences among those who are sensitive to the value in question (Feinberg and Willer Reference Feinberg and Willer2019). This work builds on moral foundations theory, which suggests frames need to resonate with values that differ by party, such as appealing to Republicans' concern for patriotism and Democrats' concern for harm prevention (Wolsko, Ariceaga, and Seiden Reference Wolsko, Ariceaga and Seiden2016). Finally, recognizing that people often emulate others, studies show that appealing to descriptive norms—what people in one's surroundings do—can alter attitudes and behaviors (Cialdini Reference Cialdini2007; Dwyer, Maki, and Rothman Reference Dwyer, Maki and Rothman2015; Moehring et al. Reference Moehring2022), even when complying with norms incurs personal costs (Pickup, Kimbrough, and de Rooij Reference Pickup, Kimbrough and de Rooij2021). Even here, though, Raymond, Kelly, and Hennes (Reference Raymond, Kelly and Hennes2021, 11) explain: “Intense partisanship can make it more important to emphasize the prevalence of a behavior or belief within the relevant political group.”
Scholars have followed suit, focusing on how targeted messages influence particular groups. As a 2017 National Academies of Sciences, Engineering, and Medicine (2017, 56) consensus study emphasizes: “Tailoring scientific messages for different audiences is one approach to avoiding a direct challenge to strongly held beliefs.” Vaccine communication studies explore subgroup appeals based on such factors as partisan cues (Pink et al. Reference Pink2021), values (Bokemper et al. Reference Bokemper2021; Lunz Trujillo et al. Reference Lunz Trujillo2020), and needle sensitivity (Lunz Trujillo et al. Reference Lunz Trujillo2020). While such targeted appeals are effective, they present practical trade-offs. Targeted messaging requires the acquisition of curated information and the ability to reach the targeted audience. When targeted messages reach nontargeted audiences, they can backfire (Hersh and Schaffner Reference Hersh and Schaffner2013).
Here, we explore the extent to which generalized communication can be effective. We do so in the challenging domain of COVID-19 vaccine uptake. This is a difficult area given extreme differences in attitudes about COVID-19 vaccines—particularly at the initial stages of their distribution. Differences in baseline vaccine attitudes emerged based on age, education, income, race, and gender (Lazer et al. Reference Lazer2021). Perhaps most importantly, partisanship remained the most stable and sizable gap, exemplifying the extreme politicization that defined COVID-19 (Clinton et al. Reference Clinton2021; Druckman et al. Reference Druckman2021; Green et al. Reference Green2020).
Even so, Motta et al. (Reference Motta2021) and Palm, Bolsen, and Kingsland (Reference Palm, Bolsen and Kingsland2021) conducted foundational studies that suggest general messages can work, with a focus on COVID-19 vaccine uptake prior to Food and Drug Administration (FDA) approval. These authors report a host of messages—such as safety, social norms, values, and health consequences—have effects across party lines. Our work builds on theirs in three distinct ways. First, we collected data immediately after FDA vaccine approval, when the decision was no longer hypothetical. At the same time, politicization was extremely high and trust in health institutions was falling (Hegland et al. Reference Heglandforthcoming). Secondly, we focus on the aforementioned three messaging approaches previously identified in the literature, with ecologically realistic messages that overlap with, but do not match, the other studies. Third, we systematically test a wider range of potential moderators that coincide with the aforementioned gaps; we do so by using recently developed machine-learning methods. In sum, we offer a novel test of generalized messaging that includes three central science communication approaches and a large set of possible moderators during a time of extreme politicization, when individuals faced an immediate pending decision regarding the vaccine. Our central question, therefore, is whether nontargeted approaches, crafted for general audiences, can be effective in such politicized situations—even influencing partisans from both sides.
Before turning to our studies, we offer two caveats. First, vaccine intentions emerge from a complex set of psychological, social, and institutional factors (Brewer et al. Reference Brewer2017); we focus on the effects of communications, notwithstanding other influences. Secondly, we recognize targeted messages can play an essential role; we simply seek to explore whether generalized messages—in our case, based on science cues, moral frames, or descriptive norms—can as well.
Data and Methods
We conducted an online survey experiment (N = 24,682) from December 16, 2020, to January 10, 2021. Thus, our data collection began after the FDA's December 11 approval of the Pfizer vaccine and just before the December 18 approval of the Moderna vaccine. This makes the decision to get vaccinated no longer hypothetical (as in prior studies), but one that respondents faced. We recruited via PureSpectrum, an online survey panel aggregator, using quota sampling to approximate the race/ethnicity, age, and gender distributions within each state. A total of 24,682 respondents who both passed two closed-form attention checks and one open-ended attention check and did not indicate that they had already been vaccinated against COVID-19 were retained for analysis.
We randomly assigned participants either to read a treatment message that provided a rationale to get the vaccine or to a control condition with no message. For the expert cue messages, we employed two distinct treatments. One suggested that most scientists recommended taking the vaccine. This is the type of message that those who emphasize a need for targeted approaches would not expect to be broadly effective because there is some skepticism as to the efficacy of relying on scientific perspectives (National Academies of Sciences, Engineering, and Medicine 2017). The other provides cues from the respondent's physician, in line with work suggesting doctors—and in our case, a doctor close to the respondent—could play an important persuasive role (Uslu et al. Reference Uslu2021). The moral framing messages follow prior work by appealing either to one's duty to what is right for the country (patriotism) or to preventing harm to themselves and others. Prior research typically identifies conservatives as being more sensitive to patriotism and liberals as more sensitive to harm prevention (see, for example, Feinberg and Willer Reference Feinberg and Willer2013). We are interested in whether these values only work among particular ideological subgroups, or whether their effects are generally similar across the population. Finally, the fifth treatment appealed to descriptive norms by posing a hypothetical where most people the respondent knows have taken the vaccine (Jaeger and Schultz Reference Jaeger and Schultz2017). Additional details regarding the survey and exact question wordings used in the experiment are in the Online Appendix. Even though the aforementioned prior work suggests possible heterogeneous effects, we note that the messages were designed to be general.
All respondents then reported how likely they would be to take the vaccine on a seven-point scale from extremely unlikely to extremely likely. For our main analysis, we make this outcome binary, considering respondents to be vaccine “resistant” if they report that they are “extremely unlikely” to take the vaccine. Those who are vaccine resistant are likely to reject vaccination even as vaccination norms spread (Palm, Bolsen, and Kingsland Reference Palm, Bolsen and Kingsland2021), posing challenges for achieving herd immunity. That said, we conduct a parallel analysis using the full seven-point scale as a continuous outcome in the Online Appendix, and we find the same substantive results. We estimate average treatment effects overall and in comparison with one another, adjusting for multiple comparisons (Bretz, Hothorn, and Westfall Reference Bretz, Hothorn and Westfall2011). This establishes which messages are effective for reducing vaccine resistance and which (if any) are significantly more effective than others, on average.
We then test for the extent to which any of these treatment effects systematically vary over a wide array of demographic covariates, including race, gender, age, education, household income, partisan and ideological identities, and the severity of the COVID-19 outbreak in respondents' counties (see the preceding discussion of vaccine gaps). For a full list of variables included in the main and supplemental specifications (for example, social media exposure) see the Online Appendix. Rather than prespecifying which of these covariates ought to be associated with treatment effects, we systematically explore the data for such effects using the generalized random forest (Athey, Tibshirani, and Wager Reference Athey, Tibshirani and Wager2019; Wager and Athey Reference Wager and Athey2018). This is an extension of the commonly used random forest algorithm (Breiman Reference Breiman2001) that, rather than maximizing the difference in outcomes at each split, maximizes the difference in average treatment effects at each split. The algorithm guards against overfitting by randomly partitioning the observations into a splitting subsample, which is used to fit the tree in a given iteration, and an estimating subsample that is used to derive the predicted effects from that tree for that iteration. Put another way, a tree in the algorithm satisfies the “honesty condition” when each observation is used to either determine splits in the tree or estimate effects, but not both (Wager and Athey Reference Wager and Athey2018). This means that any conditional average treatment effects that are identified by the algorithm are those that were robust to confirmatory, held-out estimation. In addition, it allows for predicted treatment effects at the individual level.
Results
As shown in Figure 1, 22 per cent of respondents exhibit vaccine resistance when receiving no message (in the control), but this percentage significantly decreases in every treatment condition except patriotism, which falls just short of our threshold for statistical significance (adjusted p = 0.096). For example, the percentage drops to 19 per cent (adjusted p < 0.01) when respondents are given the harm reduction message and to 17 per cent (adjusted p < 0.001) in the physician endorsement condition. Given the relatively low baseline levels of resistance in the control group, this 5-percentage-point decrease corresponds to a 23 per cent reduction in vaccine resistance.
Fig. 1. Average treatment effects for vaccine resistance.
Notes: Control estimate with 95 per cent uncertainty interval shown with dashed line in shaded band; treatment condition estimates with 95 per cent uncertainty intervals shown with point ranges. Effects significant at p < 0.05 (adjusted for multiple comparisons) darkened.
Also of particular note is that the messages that moved people the most were those that cued the respondent's physician or the scientific community. As shown in Figure 2, these have significantly greater effects than the patriotism message, though they are not significantly different in effect relative to the messages concerning harm reduction and people the respondent knows (the next-largest difference is between messages concerning respondents' personal physicians and people they know, with an adjusted p-value of 0.075). This result cuts against the prevalent sentiment that “people rarely make decisions based only on scientific information” (National Academies of Sciences, Engineering, and Medicine 2017, 3). Rather, our findings suggest that circumstances involving high personal stakes, such as potentially severe health consequences, may prompt citizens to defer to subject-matter experts. This is true even in politicized settings. The result also contrasts with work suggesting norms play a more powerful role in messaging on scientific issues (Bayes et al. Reference Bayes2020). In our results, descriptive norms do significantly reduce vaccine resistance, but the scientific and medical communities hold sway as well.
Fig. 2. Testing for differences between treatment effects.
Notes: Cells represent differences in vaccine resistance between primary (y-axis) and comparison (x-axis) conditions. Differences significant at p < 0.05 (adjusted for multiple comparisons) darkened.
Next, we explore whether messages had differential effects on particular subgroups of our sample using the generalized random forest. We find very little evidence of heterogeneous effects. Table 1 shows, for each treatment–control comparison, the proportion of predicted individual-level treatment effects that are significantly distinguishable (or not) from 0, as well as the share that are statistically distinguishable from the average treatment effect. Table 1 shows that treatment effects are substantively homogeneous at the individual level. Virtually everyone in the population is predicted to react similarly to each message.
Table 1. Proportions of predicted individual-level effect types in each condition on vaccine resistance
This is apparent in two respects. First, predicted effects go in consistent directions. For essentially every respondent where treatment effects are predicted to be statistically distinct from 0, the effects are associated with reductions in vaccine resistance. Secondly, extremely few respondents are predicted to have treatment effects that significantly differ from the overall average effects. The highest such shares are in the resistance outcome for the physician and scientists recommend conditions, in which just under 5 per cent of respondents are predicted to respond to the message either more or less strongly than the average respondent.
Neither political ideology nor partisanship moderated the moral framing messages in a systematic manner, which is perhaps the greatest departure from expectations implied by previous literature (Feinberg and Willer Reference Feinberg and Willer2019) and is discussed further in the Online Appendix. Instead, the characteristic that we find moderates effects to the most substantial degree is household income—and only in select conditions. In the maximally heterogeneous slices of these data—household income moderating the effects of the physician and scientists recommend treatments (shown in Panel A of Figure 3 for the resistance outcome; corresponding tests on the likelihood outcome show substantively similar results)—household income is clearly associated with the rank ordering of predicted effects (see Panel B of Figure 3). However, as indicated by Table 1, scarcely any of these respondents are predicted to be significantly more or less sensitive to the given message than the average respondent.
Fig. 3. Testing for treatment effect heterogeneity.
Notes: Most important variables (Panel A), sorted individual treatment effects (Panel B), and group average treatment effects by household income quartile (Panel C) for the resistance outcome.
Moreover, these differences are primarily attributable to a ceiling effect: fewer high-income respondents report vaccine resistance in the control condition, leaving less room for their counterparts in treatment conditions to exhibit differences. This is shown in Panel C of Figure 3, which shows control condition means and group average treatment effects by household income quartile. Those in the lowest (highest) quartile are predicted to see significantly higher (lower) than average reductions in vaccine resistance, but they are also starting from a higher (lower) than average baseline. Put simply, higher-income respondents have lower predicted reductions in vaccine resistance because they are already reporting lower rates of vaccine resistance in the control condition, not because they are less sensitive to pro-vaccine messaging.
Discussion
Our results have three significant implications. First, in politicized science environments, succinct messages crafted for general audiences can work, having homogeneous effects across a range of respondent characteristics, including partisanship and ideology. That is to say, while there are significant differences in baseline attitudes concerning COVID-19 vaccination by political identities, there are not significant differences in sensitivity to pro-vaccine messages. The findings reveal the types of communication that can be effective among entire populations (Druckman and Lupia Reference Druckman and Lupia2017). This should not be taken as a criticism of targeted messages, which have a crucial role to play. From a normative perspective, our results reveal a downside insofar as the groups most resistant to vaccines (for example, Republicans) are not affected more than those that are less resistant (for example, Democrats). In that sense, homogeneous effects are not always ideal, and generalized messages need to be joined with targeted appeals. Understanding sources of heterogeneity matter (Callaghan et al. Reference Callaghan2019).Footnote 1 More generally, Callaghan et al. (Reference Callaghan2021) show that if marginalized populations do not have their concerns precisely addressed, then inequalities can be exacerbated. Our point is to clarify the practical usefulness of generalized approaches as part of a communication strategy.
Secondly, despite pessimism that direct messages from scientists or other expert cues can broadly persuade (National Academies of Sciences, Engineering, and Medicine 2017), we find that they can—even in a highly politicized context and even among conservatives, who tend to exhibit less trust in science (Gauchat Reference Gauchat2012; but see Lee Reference Lee2021). In fact, we find that cuing subject-matter expertise proves most effective in the case of COVID-19 vaccines, reducing vaccine resistance by as much as 23 per cent. To be clear, our findings reveal that expert cues matter relative to no message; they do not have a significantly stronger effect than messages from lay sources (see also Motta et al. Reference Motta2021). The same homogeneity applies for the other message types, which, as explained, many view as constrained to precise populations (see also Motta et al. Reference Motta2021; Palm, Bolsen, and Kingsland Reference Palm, Bolsen and Kingsland2021). That said, it remains an open question whether these messages would stand up to competing communications, particularly those that introduce scientific uncertainty or attempt to undermine trust in expertise (Bolsen and Druckman Reference Bolsen and Druckman2015; Bolsen and Druckman Reference Bolsen and Druckman2018; Druckman Reference Druckman2017; Merkley Reference Merkley2020). Future work also would benefit from exploring whether the psychological processes underlying these distinct messaging approaches are similar to one another.
Thirdly, we began the article by pointing out the possibility that partisan or ideological priors limit the potential for homogeneous effects. In contrast, using recently developed methods at the intersection of causal inference and machine learning, we find generalized messages exhibit homogeneous effects with regard to not only partisanship and ideology, but also a host of other variables. This echoes recent political science work (Coppock, Leeper, and Mullinix Reference Coppock, Leeper and Mullinix2018; Motta et al. Reference Motta2021; Palm, Bolsen, and Kingsland Reference Palm, Bolsen and Kingsland2021) but using a distinct method and a large set of covariates. Surely heterogeneity exists with other message types and/or covariates, but it is not inevitable. In sum, rather than working to identify a variety of successful messages, each tailored to a particularly responsive subgroup, our results suggest it can be more efficient to identify the most persuasive messages overall and broadcast them widely.
Supplementary Material
Online appendices are available at: https://doi.org/10.1017/S0007123422000424
Data Availability Statement
Replication data for this article can be found in Harvard Dataverse at: https://doi.org/10.7910/DVN/4QEWQK
Acknowledgments
The authors thank Dominik Stecula, participants at the 2021 meeting of the American Political Science Association, three anonymous reviewers, and the editor for helpful advice and feedback on earlier drafts of this manuscript.
Financial Support
This research was supported by The National Science Foundation (Grant Numbers 2029292 and 2029792), the Knight Foundation, and the Russell Sage Foundation.
Competing Interests
Dr Perlis has received consulting fees from Burrage Capital, Genomind, RID Ventures, and Takeda, outside of the present work; he also holds equity in Psy Therapeutics and Outermost Therapeutics, outside of the present work. No other authors have potentially competing interests to declare.
Ethical Standards
This research was approved by the institutional review boards at Rutgers University (#Pro2020000977), Harvard University (#IRB20-0593), and Northeastern University (#20-04-12).
