With democracy under threat in many countries around the world, how the public reacts to democratic change is a crucial question. A long-established and indeed still-vibrant literature holds that experiencing more democracy boosts democratic support in the public—in short, that democracy creates its own demand (see, e.g., Lipset, Reference Lipset1959; Welzel, Reference Welzel2013; Wuttke et al., Reference Wuttke, Gavras and Schoen2022). In contrast to this classic theory, one prominent recent article, Claassen (Reference Claassen2020), argues that democratic support behaves thermostatically: that increases in democracy yield an authoritarian backlash in the public, while democratic backsliding prompts the public to rally to democracy's cause.
In support of this thermostatic argument, Claassen (Reference Claassen2020) offers evidence based on recent advances in modeling public opinion as a latent variable. Drawing on aggregated responses to more than 60 distinct questions on attitudes toward democracy and its alternatives that were asked in thousands of national surveys, the article estimates a latent variable of democratic support in more than a hundred countries over as long as 30 years. These data constitute a broader evidentiary base than that employed in any earlier work on this topic. Analyses of these data support the article's conclusions that democratic support behaves thermostatically in the public, that “increases in democracy dampen public mood, while decreases cheer it,” and that democracy “does not appear to create its own demand” as the classic theory has it (Claassen, Reference Claassen2020, 51).
But, inevitably, the aggregated survey data, the resulting estimates of democratic support, and ultimately the results of the article's analyses embody a particular set of choices. These choices, known as “researcher degrees of freedom,” have attracted growing attention among political scientists in recent years (see, e.g., Wuttke, Reference Wuttke2019; Breznau et al., Reference Breznau, Rinke, Wuttke, Nguyen, Adem, Adriaans, Alvarez-Benjumea, Andersen, Auer, Azevedo, Bahnsen, Balzer, Bauer, Bauer, Baumann, Baute, Benoit, Bernauer, Berning, Berthold, Bethke, Biegert, Blinzler, Blumenberg, Bobzien, Bohman, Bol, Bostic, Brzozowska, Burgdorf, Burger, Busch, Carlos-Castillo, Chan, Christmann, Connelly, Czymara, Damian, Ecker, Edelmann, Eger, Ellerbrock, Forke, Forster, Gaasendam, Gavras, Gayle, Gessler, Gnambs, Godefroidt, Grömping, Groß, Gruber, Gummer, Hadjar, Heisig, Hellmeier, Heyne, Hirsch, Hjerm, Hochman, Hövermann, Hunger, Hunkler, Huth, Ignácz, Jacobs, Jacobsen, Jaeger, Jungkunz, Jungmann, Kauff, Kleinert, Klinger, Kolb, Kołczyńska, Kuk, Kunißen, Kurti Sinatra, Langenkamp, Lersch, Löbel, Lutscher, Mader, Madia, Malancu, Maldonado, Marahrens, Martin, Martinez, Mayerl, Mayorga, McManus, McWagner, Meeusen, Meierrieks, Mellon, Merhout, Merk, Meyer, Micheli, Mijs, Moya, Neunhoeffer, Nüst, Nygård, Ochsenfeld, Otte, Pechenkina, Prosser, Raes, Ralston, Ramos, Roets, Rogers, Ropers, Samuel, Sand, Schachter, Schaeffer, Schieferdecker, Schlueter, Schmidt, Schmidt, Schmidt-Catran, Schmiedeberg, Schneider, Schoonvelde, Schulte-Cloos, Schumann, Schunck, Schupp, Seuring, Silber, Sleegers, Sonntag, Staudt, Steiber, Steiner, Sternberg, Stiers, Stojmenovska, Storz, Striessnig, Stroppe, Teltemann, Tibajev, Tung, Vagni, Van Assche, van der Linden, van der Noll, Van Hootegem, Vogtenhuber, Voicu, Wagemans, Wehl, Werner, Wiernik, Winter, Wolf, Yamada, Zhang, Ziller, Zins and Żółtak2022). The concern extends beyond cases of “p-hacking,” in which researchers sift through many different options to find a set of choices that yield statistically significant results. It turns out that no intentional fraud or misconduct is required to render results unreliable. Researchers may make only a single set of entirely reasonable choices, that is, they may take only a single walk through the metaphorical “garden of forking paths”; find a result that supports their expectations; and “get excited and believe it” without even considering that a different set of entirely reasonable choices would provide different results (Gelman and Loken, Reference Gelman and Loken2014, 464).
Here, we illustrate the importance of taking seriously researcher degrees of freedom by replicating the primary analyses of Claassen (Reference Claassen2020) while varying two aspects of the measurement of democratic support. The first is the manner in which ordinal survey responses are coded as supportive or not supportive of democracy. The second is the attitude ascribed to survey respondents who did not answer items regarding their support for democracy. We document a range of reasonable alternatives to the choices made in Claassen (Reference Claassen2020) on each of these issues, and demonstrate that analyses employing these alternatives provide little to no support for the conclusions drawn in that article. The fragility of the published results to alternate reasonable choices demonstrates the importance of taking researcher degree of freedom seriously and that the question of the relationship between democratic institutions and democratic support remains unsettled.
1. Researcher degrees of freedom and democratic support
The dependent variable considered in Claassen (Reference Claassen2020) is the public's support for democracy, a latent variable. This latent variable was estimated from aggregated responses to many survey questions asked in many countries in many years. Here we examine the consequences of two choices made early in the process of collecting and aggregating all of those survey data: one regarding how responses were coded as supporting democracy and a second involving how missing responses were treated.
First we consider the question of how to code survey responses. The latent variable model used in Claassen (Reference Claassen2020) requires responses to be dichotomous: either, in this case, democracy supporting or not. This is straightforward for the few survey items that gave respondents only two options, for example, in the Pew Global Attitudes surveys, “a democratic form of government” or “a leader with a strong hand.” But the vast majority of the survey items employed are ordinal and so require a cutpoint to be chosen to split respondents between those who support democracy and those who do not.
There are at least three reasonable options for recoding these ordinal responses into a dichotomous variable of democratic support. The first coding rule is the most demanding. It stipulates that only those who supply only the most democracy-supporting response of those available can be considered unflinching supporters of democracy; anything but the highest response indicates a lack of support. A second coding rule takes the opposite tack, with all but the lowest response considered as indicating at least some support for democracy and only the lowest value considered as not supportive. The last splits the difference, considering answers above the median, e.g., the two highest values on a five-point scale, to indicate that the respondent supports democracy, with the median and below being unsupportive.
In the event, Claassen (Reference Claassen2020) resorts to a mix of the above three rules. Most survey items were dichotomized following the “above the median” rule (see Claassen, Reference Claassen2020, Appendix 1.3). Although not documented in the article's appendix on “Microlevel Coding of Survey Responses,” the other two rules were employed as well. The most demanding rule was applied to the four-point item asked in Pew Global Attitudes surveys, “How important is it to you to live in a country where honest elections are held regularly with a choice of at least two political parties? Is it very important, somewhat important, not too important or not important at all?” For this question, rather than including respondents who gave both responses above the median—“very important” and “somewhat important”—only those respondents who answered “very important” were entered as supporting democracy. And for the three-point item in the Asia Barometer asking whether respondents thought “a democratic political system” would be very good, fairly good, or bad for their country, the “all but the lowest” rule was applied.Footnote 1 Adding this mixed rule to those described above gives us a total of four coding rules.
The second choice involves how to treat missing responses, that is, when survey respondents either indicate that they “don't know” or simply refuse to answer the question. Here, too, there is a range of reasonable choices. We identify four possibilities. One might reasonably assume that a missing response is equivalent to answering with a lack of support for democracy since the respondent did not provide a democracy-supporting response. Claassen (Reference Claassen2020, Appendix 1.3) takes this first approach. But one might conversely and equally reasonably consider a missing response to suggest support for democracy; the respondent, after all, did not provide a democracy-opposing response. A third possibility is that non-responses indicate opposition to the current regime. That is, their meaning depends on the context. In democracies missing responses indicate an unwillingness to supply an honest but socially unacceptable rejection of democracy, but in autocracies missing responses evince support for democracy coupled with a fear of reprisal. One final possibility is that non-responses occur at random. In that case, a non-response tells nothing about the respondent's views toward democracy and can simply be dropped from the sample. There are, then, at least four reasonable treatments of survey non-responses. Together, the four coding rules and four missing-data treatments described above give us 16 combinations of plausible researcher choices.
2. Consequences for inference
How does the selection among these 16 garden paths influence our conclusions regarding the public's response to changes in democracy? To find out, we collected all of the survey data used in Claassen (Reference Claassen2020) from their original sources. We then applied each combination of coding rule and non-response treatment and generated 16 sets of cross-national time-series estimates of democratic support using the latent-variable model employed in the article. Finally, we used each of these sets of estimates to replicate the article's analyses.
Figure 1 is a “small multiple” plot (see Solt and Hu, Reference Solt and Hu2015) showing the results of replicating Model 1.1, the principal model of Claassen (Reference Claassen2020, 47), with each of the 16 combinations of coding rule and treatment of survey non-response. In the top panel, the dots represent point estimates for the coefficients for change in liberal democracy; according to the thermostatic theory, these coefficients should be negative. In the bottom panel, the dots depict point estimates for the coefficients for lagged level of liberal democracy; according to the classic theory, these coefficients should be positive. In both panels, the whiskers show the associated 95 percent confidence intervals. Each coding rule is represented by a different color, while the four non-response treatments are shown in separate grouped estimates from left to right.Footnote 2
Figure 1. The effects of democracy on the change in public support.
Notes: Replications of Claassen (Reference Claassen2020, 47), Table 1, Model 1.1. The mixed coding rule employed in Claassen (Reference Claassen2020) along with that work's assumption that non-responses indicate a lack of support for democracy yield a larger negative point estimate of the coefficient for change in liberal democracy than most other combinations and a larger point estimate of the coefficient for the lagged level of liberal democracy than all other combinations. In error-correction models like these, both coefficients must be interpreted together; see Figure 2.
The darkest, left-most coefficients in the left-most group of estimates represent the combination of mixed coding rule and treatment of non-responses as unsupportive of democracy, that is, the combination employed in Claassen (Reference Claassen2020). It replicates the results reported in that article exactly. More importantly for our purposes, this combination yields a larger negative point estimate of the coefficient for change in liberal democracy than most other combinations. It also yields a larger point estimate of the coefficient for the lagged level of liberal democracy than all of the other combinations. But like the coefficients of constitutive terms of multiplicative interactions (see, e.g., Brambor et al., Reference Brambor, Clark and Golder2006), the coefficients of error-correction models like Model 1.1 are properly interpreted together rather than separately (see Williams and Whitten, Reference Williams and Whitten2012).
Figure 2 is similar to Claassen's (Reference Claassen2020, 48) Figure 5. It depicts simulated effects, in differences rather than levels for ease of interpretation, of a one standard deviation increase in democracy on the public's support for democracy using the 16 sets of regression coefficients presented in Figure 1 with the four different coding rules appearing in the columns and the four different non-response treatments in the rows.Footnote 3
Figure 2. Simulated effects of democracy on changes in public democratic support.
Notes: Simulated effects are estimated using coefficients from the models presented in Figure 1. The solid lines indicate the mean simulated effect; the shaded regions indicate the 95 percent confidence intervals of these effects.
The upper left pane simulates the combination of the mixed coding rule and the treatment of survey non-responses as indicating a lack of support for democracy matches, the same combination employed in Claassen (Reference Claassen2020). The initial drop and slow recovery in the mean of these simulations of public democratic support was the evidence presented for the “thermostatic response of public opinion” and the claim that there is “little evidence that democracy generates its own demand” (Claassen, Reference Claassen2020, 48). But the other panes, with a few exceptions, show smaller dips, quicker recoveries, and continued increases; these findings lead to very different conclusions. Indeed, most of the analyses—11 of 16—show statistically significant increases in democratic support within three decades, the sort of generational change predicted by the classic theory since Lipset (Reference Lipset1959). None of them show statistically significant declines that would lend credence to the argument that democratic support responds thermostatically.
3. Discussion
Claassen (Reference Claassen2020) claims that a thermostatic response to democratic change is required to explain “democracy's fading allure” among publics. It is far from clear that support for democracy is indeed declining (for recent evidence against, see, e.g., Wuttke et al., Reference Wuttke, Gavras and Schoen2022). But if it is, the fragility of the results presented in Claassen (Reference Claassen2020) to alternate reasonable coding decisions and non-response treatments suggests that other explanations are worth revisiting. One possibility not considered in the specification adopted in Claassen (Reference Claassen2020) is that rising economic inequality undermines support for democratic institutions (see also, e.g., Solt, Reference Solt2012; Magalhães, Reference Magalhães2014, 84). This and other theories of democratic support warrant further investigation.
Regardless, the fragility of the findings in Claassen (Reference Claassen2020) that is documented above has profound implications. The thermostatic theory of democratic support presented in Claassen (Reference Claassen2020) suggests that would-be autocrats working to erode democracy constitute little cause for concern: “should elected leaders start dismantling democratic institutions and rights, public mood is likely to swing rapidly toward democracy again, providing something of an obstacle to democratic backsliding” (Claassen, Reference Claassen2020, 51). To the contrary, that theory implies, it is those who attempt to deepen democracy who pose the threat, as “extending democratic rights and legal protections to minorities” will trigger a public backlash against democracy (Claassen, Reference Claassen2020, 51). If the theory is true, the appropriate positions for those who value democracy are, in short, complacency and satisfaction with the status quo.
That the evidence for such a perverse outcome depends entirely on researcher degrees of freedom indicates that defending democracy requires taking a very different course. Alternate combinations of plausible coding rules and non-response treatments yield no support for the thermostatic theory. In fact, most show that increases in democracy work, over time, to increase the public's support, just as suggested by the classic theory that has been advanced in scholarship over more than a half century. These findings imply that those who want to advance democracy should do just that: they should push ahead.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/psrm.2024.16. To obtain replication material for this article, please visit: https://doi.org/10.7910/DVN/JHE1YF.
Acknowledgements
Yue Hu acknowledges the support of the National Natural Science Foundation of China [72374116]. Frederick Solt acknowledges the support of the Office of the Provost, University of Iowa. The three authors thank the editors and anonymous reviewers for their helpful comments.
Open access
