Survey experiments, including a range of designs employing hypothetical vignettes, are increasingly common in the field of Chinese politics.Footnote 1 However, there is little research on how well vignette designs capture actual behaviour in the real world. Jens Hainmueller, Dominik Hangartner and Teppei Yamamoto together provide the first “study that externally validates vignette analysis against a behavioral benchmark.”Footnote 2 They note that “there has been surprisingly little effort to examine how well stated-preference experiments capture real-world decisions.”Footnote 3 This paper employs original survey data featuring both hypothetical vignettes and questions regarding actual behaviour in order to assess the external validity of descriptive and causal inference in survey designs. The survey was conducted in a probability-proportional-to-size sample of 1,897 rural residents in three provinces of China, and focuses on mobilization of citizens in response to grievances involving land rights. We hypothesize that differences between responses to hypothetical survey questions and real-world behaviour will be greatest when the political costs of behaviour are relatively high. We find that as the costs of taking action increase, over-estimation of legal mobilization in hypothetical questions relative to the behavioural benchmark increases, undermining valid descriptive inference. We also find that these differences affect the validity of causal inference, overestimating the effect of Chinese Communist Party (CCP) membership and legal knowledge while underestimating the effect of political connections on legal mobilization. The survey provides a valuable window into an increasingly prevalent approach to research and makes a contribution by identifying conditions under which hypothetical vignettes are less likely to produce valid inference. It engages a rich literature on disputing and legal mobilization in the field of Chinese politics and helps to resolve debates over the role of political connections and legal knowledge.
Vignette Designs
This study focuses on vignette designs that employ hypothetical actors or situations to which the respondent is asked to react. As Elizabeth Martin notes, “the use of vignettes permits an investigator to gather data that could otherwise … be collected for only a small number of cases.”Footnote 4 She also highlights that “the question of whether [respondent] evaluations of hypothetical situations relate to judgments in real life remains an issue.”Footnote 5 Thus, valid descriptive inference requires further investigation.
Challenges to valid descriptive inference exist when measuring behaviour that is costly in the real world. One example of the general problem, which is not unique to hypothetical or experimental approaches, is the over-reporting of voting behaviour and vote preferences.Footnote 6 John Fuh-sheng Hsieh and Emerson Niou investigate responses to questions about support for Taiwan independence, unification with China, or the status quo in surveys conducted in Taiwan.Footnote 7 They focus on the expression of preferences that involve potentially high costs in the real world and reveal that such stated preferences are highly conditional. “When respondents are allowed to express preferences over conditional alternatives, the breakdown of public attitudes changes.”Footnote 8 The authors find that support for Taiwan independence may be lower, conditional on the cost of independence, such as military conflict or economic crisis.
A second aspect of externally validating hypothetical choices is causal inference. Hainmueller, Hangartner and Yamamoto highlight the importance of assessing whether “the reasons that explain the hypothetical choices are the same as the reasons that explain the behavioural choices.”Footnote 9 Distortion is a concern if (mis)stating preferences is correlated with independent variables used to explain the outcome of interest. They compare the results of a survey experiment on the acceptance of immigrants in Switzerland with data from Swiss referenda on the naturalization applications of immigrants and find that the single vignette, “perhaps the most widely used design in the social sciences,” underestimates the effect of country of origin on acceptance of immigrants.Footnote 10 The present study, like those reported by Jordan Louviere, compares revealed and stated preferences and explicitly addresses external validity, “that is, whether stated preference models capture the underlying process demonstrated in the revealed preference data.”Footnote 11
Legal Mobilization in China
This study builds on a well-developed body of theoretical and empirical literature on grievances and legal mobilization in the Chinese context.Footnote 12 This substantive area provides a useful window on the methodological issues that are the focus of this research note. Ethan Michelson, building on Richard Miller and Austin Sarat's early work on grievances and the “pyramid” of approaches to dispute resolution, employs behavioural data collected in 2002 in the “first survey of its kind on grievances and dispute processing in rural China.”Footnote 13 Respondents from 2,902 households reported on a wide range of 17 different possible grievances, and 55 per cent reported at least one grievance in the preceding five years.Footnote 14 Moreover, Michelson finds that political connections significantly increase the likelihood that a respondent will escalate the dispute by taking action through the official justice system.Footnote 15 Mingming Shen and Yuhua Wang focus specifically on economic disputes in China using a representative survey of 6,116 rural residents conducted in 2003.Footnote 16 They find that only 6 per cent of respondents reported having a dispute in the preceding 20 years and, of those, only 11 per cent took any action to resolve it.Footnote 17
Shen and Wang anticipate the small number of aggrieved respondents and innovate by presenting respondents with a hypothetical vignette involving a debt dispute and asking what action they would take. While they find support for Michelson's conclusions regarding the importance of connections,Footnote 18 they conclude that legal knowledge is another key factor explaining dispute behaviour.Footnote 19 Mary Gallagher concurs, theorizing that “One precondition for legal mobilization is knowledge and awareness of codified rights.”Footnote 20 Drawing on a survey “using a hypothetical vignette of a common labor dispute,” she finds that “specific knowledge of labor law” is significantly associated with both taking action and the choice to go to court in particular.Footnote 21
Studying the actual behaviour of private enterprises, Yuen Yuen Ang and Nan Jia argue that “know-who” (political connections) dominates “know-how” (legal knowledge) in shaping firms’ behavioural responses to actual disputes.Footnote 22 Similarly, Lily Tsai and Yiqing Xu highlight the importance of political connections to explain why citizens take action when they experience real-world dissatisfaction with government services.Footnote 23 Meng Ieong identifies a positive relationship between cadre status and using political connections to resolve real-world disputes.Footnote 24 Thus, there is debate in the field over the role of legal knowledge versus political connections in shaping dispute resolution practices.
Scholars employing hypothetical vignettes have been more sensitive to concerns about valid descriptive, as opposed to causal, inference. Gallagher notes that with respect to “a hypothetical problem, respondents overestimated their willingness to take action.”Footnote 25 Jinhua Cheng and Xiaogang Wu, using data from the 2005 Chinese General Social Survey, reveal a large gap between taking legal action in response to actual and hypothetical disputes. Respondents were more likely to take legal action in response to hypothetical situations involving both civil and administrative grievances.Footnote 26 In other words, responses to hypothetical vignettes pose a problem of valid descriptive inference. We offer a rigorous assessment of this gap in the next section.
There has been less scholarly attention to the problem of valid causal inference with hypothetical vignettes. Wu and Cheng present parallel analyses of legal action in response to actual and hypothetical grievances. For example, exploring the relationship between socio-economic status and legal mobilization, they find that data analysis based on reported behaviour and hypothetical scenarios demonstrates different effects of income on taking legal action.Footnote 27 Building on the work reviewed in this section, our research goes a step further to systematically explore concerns about valid causal inference when using hypothetical vignettes in theorizing about legal knowledge, political connections, income and other covariates’ effects on legal mobilization.
Research Design
We hypothesize that the gaps between responses to hypothetical survey questions and real-world behaviour will be greatest when the costs of the behaviour are relatively high. We consider dispute resolution mechanisms located higher up on Miller and Sarat's pyramid (what Michelson calls the dispute pagoda) to be more politically costly. Thus, escalating from direct negotiation to third-party mediation, petitioning the government, litigating in court and, ultimately, protesting, increases in political cost. Empirically, we distinguish political costs from costs in time and money. We explore these issues using a survey of citizen grievances and access to justice in rural China undertaken with the Research Center on Contemporary China of Peking University in three provincial-level units: Hunan (n = 638), Chongqing (n = 624) and Inner Mongolia (n = 635). The sample includes two counties in each province, three towns in each county and two villages in each town; the selection of villages (36 in total) employed PPS (probability proportionate to size) sampling based on per capita income. In each village, 72 households were selected randomly, sampling 2,592 households in all. With a valid response rate of 73.2 per cent, the final sample size was 1,897 households. The face-to-face survey asked about respondents’ real-world experiences with land-related grievances that occurred in the respondent's household over the preceding ten years. It also asked respondents to respond to one version of a hypothetical vignette on the same issue.
Table 1 compares respondents with and without real-world land disputes on theoretically relevant characteristics.Footnote 28 Respondents are similar on all characteristics, as measured by the t-test, with the exception of a small (4 per cent) difference in length of residence in village.Footnote 29 Respondents who have resided in the village longer are more likely to have experienced a real-world dispute. Party members and those with political connections appear equally susceptible to experiencing real-world disputes or not experiencing them, as reflected in the table.
Table 1: Characteristics of Respondents with and without Real-world Land Disputes
Source:
Author's survey.
To enhance ecological validity, the study team drew upon in-depth interviews when designing the hypothetical vignette in order to closely reflect actual grievances common in rural areas of central China at the time of the study. The vignette reads:
Five years ago, Mr Wang left to work as a migrant labourer and allowed the six mu of land contracted [for 30 years] by his family from the village collective to go fallow. The village collective repossessed the land and re-contracted it to Mr Chen. Last year, Mr Wang returned to the village and demanded his land back. However, Mr Chen believed that the village collective's contracting of the land to him had followed relevant procedures, and he did not want to return the land to Mr Wang. If you were Mr Wang, what would you do?
Subsequent questions address mobilization by asking the respondents what actions, if any, they had taken in the case of real-world grievances or would take in response to the hypothetical grievance. Possible actions include doing nothing or pursuing negotiation, mediation, petitioning, litigation and/or other action, including protesting.Footnote 30 Table 2 provides descriptive statistics on actions taken.
Table 2: Taking Action in Hypothetical and Real-world Land Disputes: Descriptive Data
Source:
Author's survey.
The behavioural benchmark is based on the 208 respondents who reported experiencing one or more actual grievances over land and provided detailed responses on the most recent grievance in the survey. We further validated these real-world grievances in two ways. First, the research team conducted in-depth, follow-up interviews with a two-county subsample of respondents who had experienced real-world grievances, giving us a high degree of confidence in responses to experiential questions. Second, we also conducted interviews with representatives of the local court and justice bureau regarding the disputes that had occurred in one of the sub-sample counties. These 208 disputes provide the basis for the behavioural benchmark using revealed, as opposed to stated, preferences about legal mobilization.
Results
We first investigate the decision to take action in response to a grievance. Table 2 shows the action(s) – if any – respondents say they would take in response to the hypothetical vignette regarding the reallocation of land among households in the village. It also shows the actions that respondents who experienced real grievances over land report they did take. The first panel in Table 2 presents data on responses to all real grievances over land. The second panel presents data on responses to real grievances over reallocation, i.e. the specific type of land grievance in the hypothetical vignette. The columns on the right-hand side of the table report the chi-square, p-value and raw percentage of over-reporting.
Table 2 shows that hypothetical and real-world responses to grievances differ significantly. Depending on the type of action, the gap between the hypothetical and real cases ranges from 9 to 27 percentage points for all real land grievances.Footnote 31 Similarly, when the hypothetical and real grievances both involve land reallocation specifically, the gap between responses ranges from 5 to 26 percentage points. Thus, relying on hypothetical vignettes to infer that nearly half of all households would petition the government and more than a third would litigate in reallocation disputes produces invalid descriptive inferences. Focusing on respondents with real reallocation grievances, less than a third attempted petitioning and only 10 per cent attempted litigation.Footnote 32
The data also allow us to identify the specific factors potentially undermining valid descriptive inference. In the real world, the costs of behaviour are highly salient. When taking action over grievances, costs include time, money and political risk, among others. Table 3 presents reasons given by respondents for not taking any given action for both hypothetical and real land grievances.Footnote 33 Strikingly, the lack of power (specifically, “the power of the other party is too great”) is the single most important reason (reported by 27 per cent of respondents) for not taking action in the context of real-world grievances, while it is a minor factor (reported by only 3 per cent of respondents) in the hypothetical vignette. The data show, perhaps surprisingly, that lack of knowledge is less salient for aggrieved households in the real world (9 per cent, compared to 24 per cent responding to the hypothetical vignette), suggesting that experiencing a real grievance may drive searches for knowledge and knowledge acquisition, something unanticipated by respondents to the hypothetical vignette. Similarly, time, money and the risk of losing connections are not obstacles to taking action in the context of real-world land disputes. Finally, not surprisingly, 21 per cent of respondents to the hypothetical vignette cite the lack of importance (specifically, “too trivial”), while only 8 per cent of respondents with real-world grievances cite this reason for taking no action, i.e. “lumping it.”
Table 3: Reasons for Taking No Action in Hypothetical and Real-world Land Disputes
Source:
Author's survey.
We anticipate that using hypothetical vignettes, the estimated effects of theoretically important factors like Party membership, political connections and legal knowledge on taking action may be systematically misestimated owing to the gap between hypothetical and real-world measures. Therefore, it is useful to compare results in tests employing both stated-preference and behavioural responses to explain these different types of mobilization.
We investigate the effect of Party membership, political connections and legal knowledge on stated preferences about resolving grievances, while controlling for demographic factors (education, income, residence) and county fixed effects. The analysis builds on the existing literature on disputing in China to present logistical regression analysis of taking action in the context of a grievance.Footnote 34
Figure 1 shows how the theoretical implications vary between real and hypothetical responses in the decision to take any action in response to a grievance. The alternative is to “lump it,” i.e. do nothing. Most important for our understanding of politics, political connections (guanxi 关系), through both familial and social networks, are significant predictors of taking action in the real world that do not appear to be significant when the analysis is based on responses to hypothetical vignettes. Table 3 reinforces this point by revealing the inverse relationship: lacking power is a prominent factor in not taking action in the real world but not in responses to hypotheticals. While not reaching conventional levels of significance, Party membership has a negative effect in the real world but a positive effect in the hypothetical world.Footnote 35 Legal knowledge is not a significant predictor of taking action in the real world, and its sign is negative, while it is a positive and significant predictor of taking action in the hypothetical scenario. Education is a positive and significant factor in taking action in both real and hypothetical situations. Income is a negative, although not significant, predictor of taking action in the real world, while it appears to be positive and significant in hypothetical responses. Finally, status as a long-standing village resident (in residence in the current village since 1994) is a significant predictor of taking action in the real world but not in hypothetical situations.Footnote 36
Figure 1: Taking Any Action versus No Action in Real-world and Hypothetical Land Disputes
As a further step in evaluating the validity of causal inference employing hypothetical vignettes, we check whether the differences in the marginal effects of covariates are statistically meaningful following the method proposed by Trenton Mize and colleagues.Footnote 37 The results are largely consistent with those presented in Figure 1. The difference in the marginal effect of family connections between real and hypothetical situations is statistically significant at p < 0.001. Family connections to government are an important explanation for taking action in the real world, but the model using data from the hypothetical vignette misestimates this effect. The results show that Party membership and legal knowledge are statistically significant predictors of taking action only in the hypothetical scenario and not in the real-world, although the difference in marginal effects between the two samples does not attain statistical significance. These results are largely owing to the fact that for respondents who experienced actual disputes, the association between Party membership and legal knowledge, on the one hand, and taking action in the real world, on the other, demonstrates quite wide variation (also evident in Figure 1). Finally, differences in marginal effects of income and education in real and hypothetical situations are statistically significant at p < 0.5.
Figure 2 compares explanations for each specific type of action (negotiation, mediation, petition, litigation and other methods such as protest), using both hypothetical and experiential data.Footnote 38 Direct negotiation, mediation and petition are the most common actions for citizens experiencing grievances over land in the real world (see Table 2). Real-world and hypothetical data support different conclusions about the effects of legal knowledge and political connections on these specific actions. Legal knowledge does not have a significant effect on taking any of the possible actions in the real world. By contrast, in the hypothetical scenario, it has a positive, significant effect on every action except mediation. In reports of real-world behaviour, political connections via family members have a significant, positive effect on the choice to petition the government. The positive effects on the choices to mediate and adopt other methods, like protest, are also positive but do not reach conventional levels of statistical significance.Footnote 39 In analyses based on hypothetical vignettes, the sign is reversed (i.e. negative) for the effect of family political connections on mediation, petitioning and other methods. In the case of adopting other methods, the negative effect of family connections is statistically significant at the 0.01 level, which could lead to the incorrect causal inference that political connections reduce the likelihood of mobilization.Footnote 40
Figure 2: Choice of Dispute Resolution Mechanism in Real-world and Hypothetical Land Disputes
Using real-world behavioural measures of taking action, Party membership makes petitioning and other actions less likely, although the relationship is not statistically significant at conventional levels. By contrast, based on hypothetical vignettes, Party membership appears to embolden respondents to take other actions, including protest, against the government. Similarly, using real-world data, respondents with incomes in the top quartile are significantly less likely than those with lower incomes to pursue other actions like protest, while hypothetical data show the opposite. Respondents with higher incomes are significantly more likely to pursue mediation in the hypothetical scenario. Perhaps in the hypothetical situation, respondents imagine that financial resources are useful in the course of mediation. Indeed, Table 2 shows that in the hypothetical scenario, some respondents cite the lack of money as a reason for not taking action, whereas no respondents with actual grievances cite the lack of money as a reason for taking no action.
In a final test of the hypotheses, Figure 3 presents the results of a model of the effects of covariates on the total number of actions that respondents took in the real world or would take in response to the hypothetical scenario.Footnote 41 Relative to the real-world benchmark, the analysis of hypothetical responses overestimates the effects of legal knowledge and income and underestimates the effect of family political connections.
Figure 3: Effect of Covariates on Total Number of Actions Taken in Real-world and Hypothetical Land Disputes
Discussion
The findings of this study suggest that when behaviour is costly in the real world, hypothetical vignettes may not produce valid descriptive or causal inference. Descriptively, aggrieved citizens in the real world are less likely to take action, especially politically costly action like petitioning or litigating. This study also identifies threats to valid causal inference. Family political connections are important in explaining some more costly forms of legal mobilization in the real world, while Party membership and legal knowledge are less important. Using hypothetical vignettes, estimation of the effects of legal knowledge, political connections, Party membership and income on legal mobilization may be inaccurate.
Researchers introduce vignettes to address a range of methodological challenges. They are useful when the phenomenon of interest occurs in a relatively small number of cases in the population.Footnote 42 They also represent an advance over less contextualized survey questions, since they provide relevant framing information for questions about behavioural choices.Footnote 43 Nevertheless, this study identifies concerns about drawing valid descriptive and causal inference using single vignette designs. Employing paired vignettes or conjoint designs can improve external validity in stated preference surveys. Paired vignettes and conjoint designs invoke explicit comparisons across specific aspects of the choice context. Hainmueller and colleagues argue that respondents are more engaged when answering questions about paired conditions. “Less motivated respondents have a tendency to look for cues to provide reasonable answers that are easy to select with little thought to avoid the cognitive work required for optimal question answering.”Footnote 44 Varying features hypothesized to be relevant aspects of the choice context in paired vignettes and conjoint experiments may help to improve validity when using stated preferences to estimate the incidence and causes of real-world behaviour.
Acknowledgements
Support for this research was provided by the University of Washington Asian Law Center. The authors would like to thank Chris Adolph, Bruce Dickson, James Long, Victor Menaldo, Christopher Parker, Lingling Qi, Mingming Shen, John Wilkerson, Clair Yang and participants at the conference on “Advancing Chinese Social Science Studies with Diversified Research Methods,” held at Peking University, and seminar participants at Tsinghua University and the University of Washington.
Biographical notes
Susan WHITING is associate professor of political science at the University of Washington.
Xiao MA is assistant professor of political science in the School of Government at Peking University.
