Quantitative survey

For the first stage in the mixed-methods approach, I designed an anonymous 5 min-questionnaire, which was answered by in total 70 participants of the ‘Behavioural Exchange 2019 (BX2019)’ – conference in London. Like Vivalt and Coville (Reference Vivalt and Coville2019), I made use of such a unique opportunity to conduct a survey among participants of a professional gathering. According to its organizers, at BX2019, about 1200 of ‘the world's leading policy-makers, academics and practitioners gathered to explore new frontiers in behavioural science’ (BIT, 2019).Footnote ² The sampling of survey participants was conducted in a randomized manner. For this, I approached individuals at many different times and locations during the two conference days 5–6 September 2019 and distributed the pen-and-paper questionnaire in person. A potential sampling bias could arise if certain subgroups from my target population were not present among conference attendees (Krumpal, Reference Krumpal2013). Yet as my study specifically focuses on academic researchers, behavioural insight team members and public servants, and all three groups are represented in the survey, this seems unlikely.Footnote ³

Participants had three options of returning the completed form: either directly in person to me, via email, or in one of the boxes placed at the entrance hall of the conference venue. After the conference, the questionnaire was moreover sent out as online survey via www.onlineumfragen.com, a Swiss survey platform, in order to increase sample size.Footnote ⁴

The questionnaire comprised 12 questions (see Supplementary Appendix A) and had been pilot-tested among researchers with experience in collaborative research. While questions 1 to 4 refer to general aspects of collaborative experiments which were posed to all survey participants, the rest of the survey focuses on own experiences. Consequently, the group of those who indicated in question 5 to have no own experience in collaborative research was at this stage directly navigated to the final question asking for their affiliation. During the analyses of those questions referring to own experiences in collaborative research, only respondents are included who have at least conducted 1–2 experiments themselves. For clarification regarding the respective numbers of relevant respondents, all tables presented in this article depict the absolute number of respondents next to the relative share.

Response rates to the survey were 28.3% (43 of 152 handed out survey forms) and 10.4% (27 of 260 individuals contacted via email), respectively. Unfortunately, I have no information about non-respondents. In case systematic differences between respondents and non-respondents occurred, the study's validity would be threatened by non-response bias (Krumpal, Reference Krumpal2013). However, apart from time restrictions (which would affect all groups of conference attendees similarly), the most probable motivation to participate in the survey is that responders were particularly affected by the topic, while non-responders might not have seen much relevance of the questions for themselves. Since it is not the aim of the survey to provide a representative picture of opinions on collaborative research in general but rather to focus on personal experiences individuals made during their collaborative research, this potential self-selection does not seem to limit the data's scope to answer the research question. Moreover, to prevent social desirability bias, all survey questions have been thoroughly checked. None of them gives reason to suspect that ‘due to self-presentation concerns, survey respondents underreport socially undesirable activities and overreport socially desirable ones’ (Krumpal, Reference Krumpal2013, p. 2025). By choosing two of the least intrusive data collection modes – a self-administered paper-and-pencil questionnaire during the conference and a web-survey afterwards – one of the main driving factors for social desirability bias was kept at the minimum.

Respondents’ characteristics

This section presents respondents’ characteristics from of the anonymous survey. Out of 70 survey respondents, 60 (86%) provided information about their affiliation (Table 1).Footnote ⁵ Some 32% of them are academic researchers, another 45% public employees – employed either in the capacity of a public servant or as a member of a behavioural insight team, and 23% work for a private company.Footnote ⁶

Table 1. Descriptive statistics

Note: Out of 70 respondents, only 60 provided information about their affiliation. The percentages in brackets refer to the share in this subgroup.

Table 1 also provides an overview of the number of experiments which respondents have conducted. It shows that among public employees a much smaller share than among academic researchers has no own experiences with experiments on behavioural insights. Almost half of the sample of public employees are considerably experienced: they have conducted at least three own experiments. This reflects that all employees from behavioural insight teams are included in this subgroup, and their job mainly consists of designing and implementing experiments. In addition to that, public employees most likely only attend an expert conference when it is practically relevant for their work, whereas for academics it is much more common to also choose a conference in order to get a good overview about current research on a topic even if oneself has not conducted own research in that field yet.

Most survey respondents indicate to have conducted experiments in Anglo-Saxon countries like the UK, the US, Australia, Canada or Ireland, with an overwhelming majority of experiments being located in the UK. This might be due to the fact that the BX2019-conference was organized by the BIT and many of the behavioural insight team members in my sample likely stem from the BIT. The second group of countries, in which experiments were conducted, includes European countries like the Netherlands, Denmark, Lithuania, Finland, Belgium, Germany and France. Respondents also reported a few collaborative experiments in Afghanistan, Guatemala, Philippines, Kenya, Sierra Leone and Georgia.

In terms of policy fields of own experiments, two fields clearly stand out: education (18) and health and nutrition (16) (see Figure 1, right column). In the second tier, taxation (9), consumer protection (9) and energy and environment (8) attracted frequent attention. The least often experiments were conducted in agriculture (2).

Figure 1. Policy fields: relevance ranking and own experiments.

I also asked respondents to rank from 1 to 5 which policy field they consider most important to test behavioural insights. As the method of scaling, I chose the Borda count (Borda, Reference Borda1784) which is applied worldwide in elections and sport competitions. This method is recommended for cases when one wants to produce a combined estimate of the best item, instead of discarding information by just counting which item was put on rank 1 most often. Under the Borda count, points are assigned to items based on their ranking position: 1 point for last choice, 2 points for second-to-last choice and so on. Consequently, in my study, rank 1 was assigned 5 points, rank 2 received 4 points and so on. After the ranking, the point values from all survey participants were totalled, and the item with the largest point total was assigned the overall rank 1 of the full sample. As the last step, in order to make total points comparable between subgroups and the full sample, I standardized the number of total points by the maximum number of points that was achievable if every participant in the (sub)group had ranked this item on rank 1. The maximal sum of points from 70 valid responses in the full sample was 350.

Interestingly, for some policy fields, the relative frequency of own experiments stands in contrast to what respondents themselves assess as most relevant fields to test behavioural insights. While health and nutrition together with education make up the top 2 in both rankings, energy and environment is assessed as much more relevant than it is mirrored by actual experiments. Especially when compared to the relevance assessment of consumer protection, which has the same number of own experiments but ranks much lower in the relevance assessment, energy and environment seems to be under-researched. The contrary is true for taxation and foreign relations. While they achieve the second tier of policy fields with most frequently conducted experiments, they have not been attributed much relevance in the assessment ranking. Based on relevance assessments, foreign relations score the next-to-last rank.Footnote ⁷ However, it is worth noticing that for some respondents, the formulation of the question might have installed a lower bound due to two reasons: First, respondents were asked to name their top 5 according to the frequency of own experiments. For those who conducted experiments in more than five policy fields, certain policy fields might be underrepresented in their answer. Second, if a respondent conducted more than one experiment in a certain field, it can still only be counted once since I have no data about the number of experiments in each respective policy field. The same is true for the question of which interventions have been tested.

For interventions tested in own experiments, a top 5 emerges (see Figure 2, right column): social norms (20), simplification (16), increase in ease and convenience (16), letter design (15), and – with a little distance – reminders (12). All these interventions can be considered as minimally intrusive. They would not meet much resistance when discussed with policy partners, potentially in contrast to nudges like eliciting implementation intentions, disclosure, or changing the default rule, which attracted much less attention by survey respondents. This pattern fits to what has been documented in the literature elsewhere. For example, in more than 100 trials conducted by the two main behavioural insight teams in the US, a change of default settings was only tested twice, both in one trial (DellaVigna & Linos, Reference DellaVigna and Linos2020). However, when it comes to the relevance assessment under the Borda count, the respondents of this study clearly see default rules as the most important intervention to be tested. Defaults rank substantially before simplification, increase in ease and convenience, and social norms and can hence be considered under-researched with respect to the respondent's own priorities (see Figure 2, left column). Interestingly, the contrary is true for reminders and letter design: Both interventions do not achieve a high position in the assessment ranking but have been researched in own experiments quite frequently.Footnote ⁸

Figure 2. Interventions: relevance ranking and own experiments.

Qualitative interviews

In the second stage of the mixed-methods approach, I collected insights from selected experts by semi-structured interviews. Such an approach is recommended when the study's aim is to gain sophisticated insights into aspects of social reality (Hoffmeyer-Zlotnik, Reference Hoffmeyer-Zlotnik and Hoffmeyer-Zlotnik1992) and wants to capture nuances that cannot be detected by a quantitative approach (Glennerster et al., Reference Glennerster, Walsh and Diaz-Martin2018).

Influence on study design

Development of the research question

Asked how the research question in their collaborative research was derived, only a small minority (13%) of respondents indicated that a knowledge gap identified by the researcher was the starting point (see Table 2). In contrast, 40% reported that a knowledge gap identified by the public body has played this role, and another 48% referred to consultations between the researcher and the public body. This matches results from an earlier survey by Pomeranz and Vila-Belda (Reference Pomeranz and Vila-Belda2019) which focused on collaborations with tax authorities. They find that in 40% of cases the research idea emerged from jointly exploring topics of common interest between researchers and their public partner.

Table 2. Research question and selection of the sample

Notes: Values may sum up to less or more than 100% due to rounding. Of the 70 survey participants, only those with own experiences in collaborative research were asked these questions.

All qualitative interviews confirmed that the public body has a decisive influence on designing the research question. As Ruth Persian (Reference Persian2020) clearly puts it for studies conducted by the BIT: ‘The research question in a way is set by the public sector partner.’ This view is shared by Jaap Drooglever (Reference Drooglever2021) from the Ministry of Internal Affairs and Kingdom Relations: ‘The municipalities decide in the end. They are responsible if something goes wrong, so that means that they look very closely on what the set-up is and if it fits their standards.’ Academic researchers like Christian Gillitzer (Reference Gillitzer2020) made similar experiences in collaborative research: ‘This was more of a partnership where the ideas and proposals were directed by the Australian Taxation Office: they had a business need and something they wanted to be evaluated. Our scope and role were to refine and design the intervention such that it could be tested scientifically.’ ‘In my experience, it works best when they come to you and they want to do something’, Johannes Haushofer (Reference Haushofer2021) from Stockholm University confirms. ‘There is openness for suggestions, but it's within a pretty narrow parameter space.’

While many researchers in academia aim to influence public policy by publishing their study results, researchers in collaborative research experience it the other way round: ‘impact comes first’ (Sanders et al., Reference Sanders, Snijders and Hallsworth2018, p. 156). ‘That's a key thing that I always stress to people if they want to start working with an institutional partner: to make sure you listen to what they care about. I look at it like a Venn diagram of the things that are academically relevant and publishable and the things that are relevant for the policy partner’, Dina Pomeranz (Reference Pomeranz2020) from Zurich University says.

Yet, there also seems to be some scope for researchers to increase their influence over time. Paul Adams (Reference Adams2020), a former manager of the behavioural insight unit at the Financial Conduct Authority (FCA) in the UK, recounts: ‘In some of the earlier trials, the policy interventions were mostly designed by the policy makers. Over time, when results were not as positive as expected, we started to develop a bigger role earlier in the process, and used some other techniques to help design and develop more effective interventions.’

Differing priorities

The great influence of the public body on study design might have meaningful consequences if priorities of public employees and researchers in testing behavioural interventions substantially differed. In these cases, collaborative research with a public partner would lead to a systematically different research focus than other academic work, potentially along a research agenda that is sub-optimal (Levitt & List, Reference Levitt and List2009).

Interventions to be tested

This study shows that academic researchers and employees from a public body do yield such different priorities regarding the interventions to be tested (see Figure 3). While both groups agree that changing the default rule, one of the most effective but also most controversial nudges (see Reisch & Sunstein, Reference Reisch and Sunstein2016; Sunstein et al. Reference Sunstein, Reisch and Rauber2018; Jachimowicz et al., Reference Jachimowicz, Duncan, Weber and Johnson2019), is the most important intervention, assessments differ for the other interventions in their respective top 5.

Figure 3. Ranking of interventions – subsamples public employees and academic researchers.

For public employees, ‘increase in ease and convenience’ is the second most important intervention (43% of maximal points), whereas among academic researchers this intervention does not even achieve their top 5. The same is true for reminders, which receive 24% of maximal points (rank 5) among public employees but score very low with academic researchers (7% of maximal points).

Academic researchers, on the other hand, assess several interventions as much more relevant than public employees: feedback (29% vs 11%) as well as ‘financial incentives versus nudges’ (28% vs 14%). Both interventions make it into the researchers’ top 5 but are not ranked highly by public employees. In addition to that, timing (17% vs 8%) and disclosure (7% vs 0%) are considered more important by researchers than by public employees. Given the great influence of the public body on study design, this will likely lead to these interventions being under-researched in collaborative experiments. This interpretation was confirmed by several qualitative interviews. One interviewee put it in a nutshell: ‘We did have early on a conference where we met with many senior people from the public partner. There we came on with proposals. And they were receptive. But ultimately the things that went ahead were dictated primarily by their business needs.’Footnote ¹¹

Policy fields of interest

Interestingly, with respect to policy fields, less pronounced differences can be observed. Researchers and public employees overall agree that health and nutrition as well as education rank on the first two spots, followed with some distance by energy and environment, work and pension, and consumer protection (see Figure 4). This mirrors the pattern observed in the full sample with the sole modification that in the full sample data, consumer protection ranks before work and pension (see Figure 1, left column). When compared with the AEA RCT registry, the most popular database in economics to pre-register an RCT, one can also observe that the pattern closely fits those policy fields in which most RCTs have actually been conducted.Footnote ¹²

Figure 4. Ranking of policy fields – subsamples public employees and academic researchers.

Yet there are also some differences between the two groups. While public employees assess transportation as an important policy field, this field is not deemed very important by academic researchers (20% vs 14%). On the other hand, public finance (7% vs 18%) is considered relatively more important by academic researchers than by public employees.

Overall, there seems unity that defence, foreign relations and agriculture are not very relevant for testing the application of behavioural insights. However, in practice, respondents from the sample indicated to have conducted own experiments in these policy fields: foreign relations were nominated seven times, agriculture two times and defence at least once (see Figure 1).Footnote ¹³

Motivation for collaborative research

Several interviewees explicitly mention that the perspectives of academic researchers differ from those of the public body while in the quantitative data differences in the motivation for collaborative research are not clearly observable (Table 3). This might be due to the fact that the in-depth interviews were able to uncover some aspects which were not listed as answering options in the survey. ‘I have realized when collaborating with academics quite often, because interests are different, they might be more interested in testing a theory or in finding interesting results that can be published nicely. We got a few projects where we take more the research angle, but ultimately if I have to run the 50th social norms tax trial, it will probably not be intellectually stimulating but if I think that's actually the most effective thing, then that's ultimately what I would do’, Ruth Persian (Reference Persian2020) from BIT explains. Johannes Haushofer (Reference Haushofer2021) describes the same phenomenon from the academic point of view: ‘Public partners are often interested in a quick turnaround and quick results, and they're maybe less interested in understanding mechanisms. They just want to know if it works. As a researcher, you care a lot about why it works. You want a really clean design. That's often not a priority of the public partner.’ Wilte Zijlstra (Reference Zijlstra2020) from the internal behavioural insight team at the Authority for the Financial Markets in the Netherlands adds: ‘We're not a university. Our main task is to promote fair and transparent financial markets. We do research together with academics. But research is not our only aim.’

In sum, while the public body is interested in finding out what works, researchers moreover want to dive into the whys of the causal relationship (Czibor et al., Reference Czibor, Jimenez-Gomez and List2019) in order to inform ongoing theoretical debates (Christensen & Miguel, Reference Christensen and Miguel2018). In clinical research, these categories are used to distinguish pragmatic trials from explanatory trials with the former reportedly being most attractive for policy makers (Patsopoulos, Reference Patsopoulos2011). On the upside, the documented great influence of the public body ensures that the research endeavour is designed in a way that it will create substantial political interest and impact policy decisions (Karlan & Appel, Reference Karlan and Appel2018). On the downside, it gives rise to confirmation bias.

Table 3. Greatest advantage of collaborative research

Notes: The full sample also comprises of those respondents who did not indicate their affiliation. Values may sum up to less or more than 100% due to rounding. Absolute numbers in brackets.

^a In the open text-field option, some respondents enlisted reasons that could be summarized into a new category: creating direct benefits of research for policy institutions and the wider public. This was not provided answering category in the questionnaire.

This bias occurs when the motivation for evaluating a programme or a new policy is to produce external evidence for observations that have been already made.Footnote ¹⁴ If this was the case, the research endeavour would be designed in a way to favour expectancy congruent information over incongruent information (Oswald & Grosjean, Reference Oswald, Grosjean and Pohl2004) and not to search for the true underlying behaviour.

Constraints within the public administration

To gain a deeper understanding of the constraints that public employees are facing, the qualitative interviews also addressed the question what reservations interview partners met within their public administration while pursuing an experimental test of interventions. In the following, their answers are presented along the themes that occurred during the interviews.

Scientific standards

The anonymous survey of this study unveils that in cooperative research the highest scientific standards are regularly not met. A majority of respondents indicates that they had to move away from an ideal scientific approach at least 50% of the time. One quarter of respondents experienced this even in the majority of cases or always (see Table 4).

Table 4. Scientific standards and risk aversion

Notes: Respondents were asked (i) how often they had the impression they had to move away from an ideal scientific approach in order to accommodate the requirements of their cooperation partner, and (ii) how frequently they had the impression that their research opportunities were limited by a high degree of risk aversion in the public cooperation partner in comparison to experiments with other partners. The full sample comprises of 70 respondents, ten of whom did not indicate their affiliation. Only survey respondents with own experiences in collaborative research were asked these questions. Values may sum up to less or more than 100% due to rounding. Absolute numbers in brackets.

^a In the online survey, respondents who indicated to be a public servant were not asked these questions. Answers in the subsample of public employees hence mainly stem from members of a behavioural insight unit.

^b The answering options ‘never’ and ‘always’ include the statement that the respondent only conducted experiments with public partners.

Moreover, there is tentative evidence that public employees, in particular behavioural insight team members, feel the pressure to accommodate the needs of their cooperation partner at the cost of high scientific standards more strongly than academic researchers. While the subsample of academic researchers is evenly split between those who report that they had to move away from an ideal scientific approach never or only in the minority of cases, and those who did this at least 50% of the time, the proportions within the subsample of public employees are clearly different. More than 70% of public employees indicate to have experienced such a pressure at least 50% of the time, while only 30% report this for a minority of cases or never. The answers of public employees to this question are mainly driven by members of behavioural insight teams, since a sample split in the online survey prevented respondents who had indicated that they were a public servant from getting asked that question at all.

For behavioural insight teams, two types can be distinguished: (i) internal units dedicated to applying behavioural insights within their organisation, and (ii) external units like the BIT which started as a British government institution and became a social purpose consulting company in 2014 working for public bodies all over the world.Footnote ¹⁵ Since both types of behavioural insight units are on the payroll of the public body, they are more closely connected to its interests than external actors like academic researchers.

This observation was also confirmed by the qualitative interviews. Paul Adams (Reference Adams2020) recounts that during his time in the behavioural insight team at the Financial Conduct Authority in the UK they were likely to go along with the public body's constraints, for example by implementing mainly low-risk nudges: ‘At the start of the seven year period, we would say, that's fine, let's just do it anyway to get the experience and to try something out. But we sort of realised actually this was not a sensible way to do things. So our approach changed a little bit over time. I think early on we were happy to be more flexible.’ Another interviewee of a behavioural insights unit, who wants this quote to be used anonymously, adds: ‘Obviously we cherish our autonomy. But I also know I might have to collaborate with this person again sometime.’

Academic researchers, on the other hand, seem to experience more freedom when it comes to setting the terms and conditions of the research project. Dan Ariely (Reference Ariely2019), founding member of the Center for Advanced Hindsight, says: ‘For me, experiments with public partners do not mean a step back in scientific rigor. Because I am also happy to say no to an experiment. It's helpful that we are an outside company.’ A similar view is put forward by Dina Pomeranz (Reference Pomeranz2020) from Zurich University: ‘Basically, for quantitative research we need a certain number of observations. If they don't have that, we cannot do the study.’ Also for Johannes Haushofer (Reference Haushofer2021) from Stockholm University, there are non-negotiables: ‘If someone's not willing to randomize, mostly that's the end of the conversation for me.’

For behavioural insight teams like the BIT, it is more about explaining the statistical needs in plain language and gathering understanding: ‘It's being very clear upfront about what standards you have and why. It's good to explain very carefully and be prepared to have these conversations several times. You will have to tell different people. Explain concepts you have. Power calculations is not something that's intuitive to many people but it's finding a way to making it intuitive’, Alex Sutherland (Reference Sutherland2020) summarizes.

When the interviewees were explicitly asked whether they were free to walk away from an experiment in case of quality concerns, one interviewee admitted: ‘Walking away from an entire project is difficult. There are contractual obligations. I mean, there are implications, not for me personally, but for the project itself.’ Another one added: ‘We could not walk away at any time, no. Once we've invested the political capital and the energy of our partner, walking away from a trial is probably not going to go down very well. That's why we were always very careful in our negotiations and be very clear that there is a contracting process, and there's always a stop-go-decision at the very end.’

Behavioural insight team members also seem to be more willing to adjust to the time constraints set by the public partner. ‘One difference to academia is the timelines we are working on can often be very different. We might be trying to turn around a trial in a number of days or weeks rather than months or years’, Alex Sutherland (Reference Sutherland2020) from the BIT describes.

Wilte Zijlstra (Reference Zijlstra2020) from the behavioural insight team of the Dutch Authority for the Financial Markets, even sees himself as an advocate of his organization's interests: ‘When it gets too academic, I tell my colleagues at the behavioural insight team: Don't forget about the practical aspects. Think about what is relevant for supervision.’ Jaap Drooglever (Reference Drooglever2021) from the Ministry of Internal Affairs and Kingdom Relations adds: ‘Most of the municipalities are really far from thinking in the way the behavioural insight team does. These are two different worlds of thinking. But it can be addressed: by a lot of dialogue.’

Quite fittingly many interviewees mentioned that communicating research results in a way that makes it accessible to a non-scientific audience is a core skill of a behavioural insight team. ‘It's knowing where to put the information you are producing. We definitely don't prioritize publishing journal articles but we do prioritize to make sure that whatever is produced is accessible to people who are not specialists, who are not researchers, and we make it as easy as possible for them to understand the implications of results that we find’, Alex Sutherland (Reference Sutherland2020) says. It is hence not surprising that three out of six interviewees working on behavioural insights within a public body held positions as communications professionals in their organisation before (Maser, Tangen, Zijlstra).

From a positive perspective, Ruth Persian (Reference Persian2020) from the BIT points out: ‘The advantage of being external is having a fresh pair of eyes. We can suggest things and get away with them, where civil servants might be a bit more hesitant because they will still be around at the end of the project.’ Yet she also is very aware of certain constraints: ‘We are a consultancy. So ultimately, if the government partner refuses or feels very uncomfortable with a certain design, then we might have to adapt our approach.’

A systematically different research agenda of cooperative research could arise from that. As Delaney (Reference Delaney2018) sketches out in his recent article on the BIT: ‘There is a danger that behavioural insights trials will accumulate a large amount of local information on projects specifically selected for their suitability for treatment and with outcomes determined by local agency pressure.’ This study provides some first explorative evidence to support this apprehension.

Risk aversion of public bodies

As the most important reason for the pressure on adjusting scientific standards, the qualitative interviews identified a high degree of risk aversion in the public body, while the quantitative data on this aspect is not as clear cut. When respondents were asked how often they had the impression that their research opportunities were limited by a high degree of risk aversion in their public partner in comparison to experiments with other partners, only a slight majority experienced this more frequent or always (see Table 4). However, the answering pattern might have been influenced by the formulation of the question which did not ask for the absolute frequency in which respondents experienced high risk aversion but for a comparison to experiments with other partners. If some respondents had only conducted experiments with public partners before and did not feel represented by the two answering options which incorporated that fact (namely: never and always), they would very likely be inclined to pick the answering option equally frequent since they have no comparison.

However, interestingly, again a difference in answering patterns of academic researchers and public employees can be observed. Even though subsample sizes are small, it is remarkable that the greatest share of public employees indicate that they always experienced high risk aversion. In contrast, not a single academic researcher chose that option. This might be due to the fact that, as discussed above, researchers feel free to walk away from a cooperation if their required standards are not met.

Notably, during the qualitative interviews, the topic of risk aversion in the public body was frequently raised. ‘Public servants can lose a lot and do not have much to win. They can win for the country but not for themselves. They need to step out of their comfort zone a big way’, Dan Ariely (Reference Ariely2019) from Duke University says. Dina Pomeranz (Reference Pomeranz2020) from Zurich University shows understanding for constraints resulting from that: ‘Responsible leaders of course have to be risk averse to some degree. If the payoff is small and it's not worth taking any risks for it, they are unlikely to agree to a collaboration. If knowing the answer to the research question is of value to them, they tend to be more willing to take the risk.’ Alex Sutherland (Reference Sutherland2020) from the BIT adds: ‘It's the fear of the unknown. If people have done other kinds of evaluations like quasi-experimental designs, they may be familiar with those kinds of things. But changing something that's been done and planned, going against the status quo, that's more difficult. The process of change might feel alien to them.’

Experiments might also bring about unwanted evidence. ‘Science is kind of risky. You don't know what the answer is going to be. When you experiment, it might result in something that is contrary to what I would like it to be’, Wilte Zijlstra (Reference Zijlstra2020) from the Authority for the Financial Markets summarizes his colleagues’ reservations. Alex Sutherland (Reference Sutherland2020) from the BIT consents: ‘It requires a great deal of faith from the delivery partner knowing that the end result of the evaluation could be that their intervention was not successful in improving outcomes. It's really hard if they have to sell that it hasn't really worked or – as worst outcome – that it made things worse.’ A similar view is put forward by Johannes Haushofer (Reference Haushofer2021) from Stockholm University: ‘For public servants, it takes a lot of courage to run a study. Because trials can produce bad results for your program, it can fall apart, and then you've spent a lot of money and nothing to show for it. It's a risky proposition. And so if you do it with partners that you know and trust, that can make it easier.’

Researchers looking for collaboration can actively address this fear: ‘One aspect that they highlighted about why they accepted this proposal as opposed to some other ones is that it had a lot of safeguards about how we would avoid unintended outcomes. The proposal was careful to protect their institution from potential problems’, Dina Pomeranz (Reference Pomeranz2020) recounts her very first collaborative experiment with a public partner. Another way out of this is ‘building trust and reputation’, Dan Ariely (Reference Ariely2019) from Duke University emphasizes. ‘Even with being well known as a researcher, I do so much free advice and trust building. Normally, I give an introduction into behavioural economics, describe a small problem I am working on and sketch out a project that will come in 20 years. I call it lubricating the trust machine.’

Yet the public partner does not only need to build trust into the researcher but also into the intervention. For this, best practice examples from the public sector help, Lindsey Maser (Reference Maser2020) from the City of Portland emphasizes: ‘Since behavioural science and running RCTs is still fairly new in US government, and government tends to be very risk-averse, it's incredibly helpful to have examples of successful applications from other governments. It's easier to get approval to try something another government had success with than to try something that's never been done and might fail.’ Paul Adams (Reference Adams2020), formerly at the behavioural insights unit of the UK financial regulator, confirms: ‘You can then rely on external expertise to back your approach.’ A view that is also shared by Jaap Drooglever (Reference Drooglever2021) from the Ministry of Internal Affairs and Kingdom Relations in the Netherlands: ‘Experiments give more arguments for the internal process to say: OK, we have to do it another way because the result from this experiment is that we have to redesign something.’

A decisive role in the process plays the media, several interviewees emphasized (Ariely, Reference Ariely2019; Maser, Reference Maser2020; Pomeranz, Reference Pomeranz2020; Tangen, Reference Tangen2020; Zijlstra, Reference Zijlstra2020; Aki, Reference Aki2021). ‘As soon as anything gets out, you then tend to get a lot of questions and that creates a lot of work and essentially media requests. And so organisations in the public sector are generally relatively risk averse in terms of putting things up publicly’, one interviewee describes. Thomas Tangen (Reference Tangen2020) from the Norwegian Tax Authority personally had to deal with bad publicity after cooperating with academic researchers on a tax trial (published by Bott et al. (Reference Bott, Cappelen, Sørensen and Tungodden2019)): ‘It actually became a media issue afterwards. Because some lawyers argued that we treated people differently. Some taxpayers were told “We know you have some money abroad” and others were told “You should just declare everything”, we did not say what we knew. That was a big discussion afterwards.’ Even though the Director General publicly defended the experiment and no legal issue followed, attitudes within the administration changed: ‘The notion was that we have to be very cautious, because at the end of the day we are dependent on people's trust. We still are doing experiments but we have to have that discussion every time’ (Tangen, Reference Tangen2020).

Pre-registry

In the scientific community, publishing a pre-analysis plan for experimental research has become a highly recommended means of quality control (World Bank, 2020).Footnote ¹⁶ Yet in collaborative experiments with a public partner, it does not seem to be commonly used. In the anonymous survey, almost 45% of study respondents indicate that they have never pre-registered any of their collaborative experiments before (see Table 5). Another 14% did this less frequently than with other partners. For 18 of the 21 respondents, who chose either one of these two answers, the affiliation is known. The vast majority of them are public employees who have been shown to yield a great influence on the study design of cooperative experiments. It hence seems that another distinct feature of cooperative experiments is a tendency to not pre-register.Footnote ¹⁷ Interestingly, however, the small minority of respondents who states to always pre-register are not the academic researchers: none of them indicates to have done so in contrast to two public employees.

Table 5. Frequency of pre-registering

Notes: Respondents were asked how frequently they pre-registered their experiments with public partners compared to experiments with other partners. For public servants, the question in the online survey was slightly modified, namely: ‘How frequently did you register a pre-analysis plan for your experiments (e.g., in the AER RCT registry)?’ The answering options were: never, in the minority of cases, 50% of the time, in the majority of cases, always. All three public servants picked never and are hence listed in this table in the corresponding category. The full sample also comprises of those respondents who did not indicate their affiliation. Values may sum up to less or more than 100% due to rounding. Of the 70 survey participants, only those with own experiences in collaborative research were asked these questions. Absolute numbers in brackets.

^a The answering options ‘never’ and ‘always’ include the statement that the respondent only conducted experiments with public partners.

When comparing the answering options less frequent, equally frequent and more frequent, most respondents indicate that they pre-register cooperative experiments equally frequent (36%) or less frequent (14%) than experiments with other partners. None of them indicates that this was more frequent the case (0%). For those answering options, all answers can be accounted for by affiliation. The analysis shows that researchers and public employees contribute to the results in the same absolute numbers. It has to be noted, however, that among public employees almost exclusively members from behavioural insight teams are present, since for public servants these answering options were not available in the online survey.

Several qualitative interviews confirmed the tendency to not upload a pre-analysis plan for cooperative experiments. According to the interviewees, three main hurdles prevent from pre-registering their experiments at established platforms: time pressure, confidentiality issues and the aim of keeping experimental subjects and the media uninformed that a trial is underway (Adams, Reference Adams2020; Persian, Reference Persian2020; Sutherland, Reference Sutherland2020; Tangen, Reference Tangen2020; Zijlstra, Reference Zijlstra2020; Aki, Reference Aki2021; Drooglever, Reference Drooglever2021). Wilte Zijlstra (Reference Zijlstra2020) from the behavioural insight team at the Dutch Authority for Financial Conduct weighs the pros and cons: ‘Setting up a pre-analysis plan helps you with your design. But, again, it costs time. And you are operating under a deadline.’ ‘We use our advisory board for the function of a pre-analysis check. We will consult them about the way we have designed experiments and whether they approve of it or not’, Jaap Drooglever (Reference Drooglever2021) from the Dutch Ministry of Internal Affairs and Kingdom Relations describes. Yet, he also clarifies that they would not make anything public before the experiment starts: ‘I don't think we will upload a pre-analysis plan publicly because I don't know how who will react. But there are no real barriers to do so.’

For the work of the BIT, Ruth Persian (Reference Persian2020) explains: ‘We do have research protocols for every single experiment that we run, but usually we do not pre-register publicly. Often this is because of confidentiality issues with our partners. Social science registries are also quite a new development – but the importance of pre-registration is definitely something we are conscious of and are thinking about.’ Persian also thinks that more pre-analysis plans will be openly published in future: ‘To be honest, I think it's partly a resource problem that we don't do that by default. If we come up with a process, it's probably not that much work.’

Publication of results

Yet it is not only lacking pre-registration that gives rise to concerns with regard to research transparency. According to the interviewees, a substantial number of trials with public partners does not even get published after completion. This finding is in line with the results of a recent meta-study by DellaVigna and Linos (Reference DellaVigna and Linos2020). They document that as much as 90% of the trials conducted by the two largest Nudge Units in the United States have not been published to date, neither as working paper nor in any other academic publication format. According to Sanders et al. (Reference Sanders, Snijders and Hallsworth2018), two problems follow from this: (i) trials that are not published at all produce a public file drawer problem, especially when null and negative results are selectively held back, and (ii) trials that are published with insufficient details regarding their methodology prevent any quality control since the reliability of their results cannot be assessed appropriately.

‘We want to push for greater transparency in our work. We engage with people on this point quite frequently. Yet being a commercial research organization, the incentives are not towards publishing. The incentives are to get the job done’, Alex Sutherland (Reference Sutherland2020) from the BIT describes the tradeoff. Ruth Persian (Reference Persian2020) also puts attention to personal factors: ‘That's all great if you actually get something out of the publication. And our publication on my CV is of course great. But it was also a lot of work that has to happen next to our day job.’ Helen Aki (Reference Aki2021), too, describes that she has to prioritize tasks given her limited time resources: ‘We don't publish all the trials we conduct. And the honest reason is the time it takes to really write a report that is suitable for a public audience and get through the approval processes and everything else. Ultimately we're a government organisation and we're here to make change for New Zealanders. And so it's hard to make the time to do the kind of work for publishing.’ Wilte Zijlstra (Reference Zijlstra2020) from the behavioural insight team at the Dutch Authority for Financial Conduct made similar experiences: ‘If you want to publish externally, you think about: how will this be interpreted and understood? So it's a cost-benefit assessment: how much extra time would it cost to get an external publication and is it worth the time? For reports internally, people know the context, you don't have to explain everything.’ He also sees the risk that some firms would use null or negative results for legal complaints against the regulator: ‘When you get court cases, they can use published reports against us. They would quote us: “you're saying we should do this. But you're also saying it doesn't work”.’

Jaap Drooglever (Reference Drooglever2021) from the Ministry of Internal Affairs and Kingdom Relations has quite the opposite impression. He perceives tight boundaries around the possibilities of holding results back from being made public: ‘To keep certain information as classified, you need really good arguments, based on law. And I can't really see such arguments for this pilot, on the contrary: We want to share this way of working, and our results. So other than the privacy aspect of a pilot, there are no real barriers to be as transparent as we can.’

In the academic literature, publication bias is a well-known phenomenon: If researchers have a greater tendency to submit, and editors a greater tendency to publish studies with significant results, the publicly available evidence will be systematically skewed (Franco et al., Reference Franco, Malhotra and Simonovits2014). That's why researchers strongly advocate to clearly codify an agreement like a memorandum of understanding (MOU) with the cooperation partner beforehand that all findings can be shared publicly (Karlan & Appel, Reference Karlan and Appel2018; Pomeranz & Vila-Belda, Reference Pomeranz and Vila-Belda2019).

It is hence not surprising that for academic researchers like Dina Pomeranz (Reference Pomeranz2020) from Zurich University publishing the results of an experiment is a non-negotiable prerequisite for any cooperation: ‘It is important to always set clear terms ex ante of what can get published, for example general results can be published, but no individual data. For scientific integrity it is important that the institution does not have a veto power at the end if the results are not what they hoped for.’ Christian Gillitzer (Reference Gillitzer2020) from Sydney University, too, did not experience any constraints with respect to publication, just different priorities of the public partner: ‘They're not primarily interested in making contributions to academic literature. Once the initial report was written with the findings, there was a conference call, and after that they briefed their senior people and then considered essentially the case closed. We contacted them with some follow up questions during the revision process to the paper, and they were receptive and helpful and wished us well. But they had gotten out of what they wanted to do and had moved on.’ From the public partner's perspective, the disengagement might be efficient and rational (Karlan & Appel, Reference Karlan and Appel2018). Yet in sum it contributes to a situation where the publication of results from collaborative experiments is not considered the default and hence another essential requirement for high-quality evidence informing policy decisions is not met.

Short- and long-term effects

A third aspect of quality control is to check whether effects are sustainable over time by comparing short-term to medium- or long-term effects. In collaborative research with public partners, this means, too, does not seem to be commonly used. Quite the contrary: the vast majority of study respondents (73%) indicate that the maximum time of observation to measure a (long-term) effect in any of their collaborative experiments was 12 months (see Table 6). Only 9% measured an effect after more than 24 months. Even though these results do not provide evidence that the period of observation in cooperative studies is shorter than in other research, it does show that in the field of cooperative research another criterion expected from high quality evidence is not satisfied.

Table 6. Maximum period of observation

Notes: Values may sum up to less or more than 100% due to rounding. Of the 70 survey participants, only those with own experiences in collaborative research were asked these questions. Absolute numbers in brackets.

However, as the qualitative interviews point out, time might bring about some improvements. According to Dan Ariely (Reference Ariely2019) from Duke University, the concentration on short-term effects could be a phenomenon mirroring the relative young age of testing behavioural interventions with a public partner: ‘We started with low-hanging fruits to show success. There was a focus on short-term effects. Now the field will develop into longer term experiments.’ Yet long-term studies also need a public partner to go along. While some researchers mention that policy makers find long run effects of great importance (Czibor et al., Reference Czibor, Jimenez-Gomez and List2019), others document exactly the opposite (Sanders et al., Reference Sanders, Snijders and Hallsworth2018). This study contributes to the latter view by finding a high institutional impatience when it comes to the time span of research projects. Whether policy makers will truly provide enough administrative resources to measure long-term effects, remains an open question.

‘I think it depends a lot on the person. The key is to find partners who share the interest in learning the answers. It also depends on the institutions. Some institutions have a tradition of research and innovation. Others have less of a culture of learning’, Dina Pomeranz (Reference Pomeranz2020) summarizes. Jaap Drooglever (Reference Drooglever2021) from the Ministry of Internal Affairs and Kingdom Relations made similar experiences: ‘We asked those municipalities, which we know are very open to really innovative ways of working. And that is important because you are doing something which isn't usual business. So it has to be conducted in an organisation where there's space for it.’ It seems that for public bodies the same is true what Karlan and Appel (Reference Karlan and Appel2018) document for organizations from the international development context: ‘The overarching lesson ( . . . ) is to choose carefully. Seek out partners who genuinely want to learn about their programs and products; who are ready, willing, and able to dedicate an appropriate amount of organizational capacity to research; and who are open to the possibility that not all answers will be rosy.’

Internal quality control

On the upside, the qualitative interviews also revealed that some institutions have installed their own processes of quality control. At the BIT, an internal research team reviews all research protocols and their proper set-up (Persian, Reference Persian2020). Being the Chief Scientist and Director of Research and Evaluation of the BIT, Alex Sutherland (Reference Sutherland2020) ensures the overall standards and quality: ‘We are trying to operate a similar sort of standards as external researchers. We have power protocols. We pre-specify. We have quality ensurance reviews of our analyses. We also often collaborate with universities who keep us accountable and shine a critical eye over our methods and tools.’

Collaborating with external academics was also the way chosen by the internal behavioural insight team of the Financial Conduct Authority (FCA): ‘For all our research publications at the FCA, they had to be peer reviewed by an external academic to make sure that the methods were academically sound and rigorous. So all of the publications we put out had to have that external check’, Paul Adams (Reference Adams2020) recounts. The results would then be published in an FCA-own ‘Occasional Paper’ series (Financial Conduct Authority, 2020).

Other interviewees describe that their ministries installed advisory boards with external academics and stakeholders (Aki, Reference Aki2021; Drooglever, Reference Drooglever2021) or have contracts with external research consultants like the BIT for quality control (Maser, Reference Maser2020; Aki, Reference Aki2021). ‘In times, when less funding is available, we design and run the trial and BIT provides some advice along the way, and then reviews our analysis afterwards to confirm if we've done it correctly, or missed some deeper level findings’, Lindsey Maser (Reference Maser2020) from the City of Portland describes. That there are several feasible ways of quality control, is also acknowledged by Johannes Haushofer (Reference Haushofer2021) from Stockholm University: ‘I think having academics involved is one approach of ensuring high scientific standards, or having a high quality outside organization involved, like an NGO that evaluates studies, or even a company.’ However, what becomes clear is that it should always be made sure that experts from outside the public body are part of the quality control process.