R3.1. Which designs involve a randomized manipulation?

Some commentators disagreed with the fundamentals of our analogy. For instance, Hart & Schatschneider objected to comparing within-family genetic studies to RCTs on the grounds that the former “do not have a manipulation,” whereas Kaplan & Bird implied that only RCTs have a “randomizing element.” We disagree. Although there is no artificial manipulation of the genome in human behavioral genetics, the conception of every human involves a natural randomized manipulation of genetic material. Indeed, it's ironic that Hart & Schatschneider warned against “forcing the language of experiments … on within-family designs,” because the language of experiments actually comes from genetics! In his foundational work on experimental designs, Fisher named experimental “factors” after “Mendelian factors,” and strove to create randomization schemes that mimicked the randomization of genetic inheritance. In their comment, Pingault, Fearon, Viding, Davies, Munafò, & Davey Smith (Pingault et al.) quoted Fisher (Reference Fisher1952) on exactly this point:

R3.2. Which designs are complicated by causal stimulus heterogeneity?

In our target article, we discussed the ways in which causal effects can be non-uniform, in that they can produce heterogeneous effects across individuals because of moderation by other causal factors. Lynch et al. incisively raised the issue of another source of heterogeneity, causal stimulus heterogeneity, when not everyone in the “treatment” group receives the same causal treatment. Causal stimulus heterogeneity is patently a problem when using polygenic scores (PGSs), which aggregate effects across very many single-nucleotide polymorphisms (SNPs): two people might have equivalently high or low PGS values, but arrive there via non-overlapping sets of SNPs. Lynch et al. cleverly analogized a PGS to a “a drug with thousands of ingredients of small efficacy, where each pill has one ingredient or an alternative at random according to a defined chance procedure.”

Ross et al. raised a similar concern, noting that it applies not only to PGSs, but also to associations with individual single-nucleotide polymorphisms (SNPs), because each SNP “tags” information about other genetic variants (including unmeasured variants) that are in linkage disequilibrium (LD) with the focal SNP included in a genome-wide association study (GWAS). They provided a similar analogy to Lynch et al.:

Related concerns were raised by Borger, Weissing, & Boon (Borger et al.) (“… thousands of single nucleotide polymorphisms (SNPs) are considered simultaneously”), and by Pingault et al., who also analogized polygenic influences to a drug cocktail, the formulation of which differs across people (“… in this case, the ‘treatment’ is not well defined (in content or timing) … [it is] like a drug RCT consisting of the simultaneous administration of hundreds of compounds”).

Lynch et al. correctly pointed out that one can still conclude, on the basis of an appropriately randomized study, that a heterogeneous causal stimulus “works,” in that it has a non-zero average treatment effect. But, figuring out how it works is be especially challenging:

We agree that causal stimulus heterogeneity does make mechanistic understanding considerably more difficult. But, as we will further explain, we do not think this is a problem that is unique, or uniquely difficult, to the study of genetic causes.

Most discussions of causal stimulus heterogeneity implicitly or explicitly contrasted the messiness of PGSs and SNP arrays with the supposed homogeneity of the causal stimulus in RCTs. Lynch et al., for instance, wrote: “In most RCTs, individuals in the treatment group receive the same, or as similar as possible, treatment or causal stimulus, such as a drug or educational intervention (causal stimulus homogeneity).” Siegel made a similar claim, writing that “RCTs of treatments that prove to be highly efficacious directly demonstrate their greater uniformity of therapeutic effects.”

From our perspective as clinical psychologists who have worked to deliver “empirically supported” psychotherapy to patients in clinical practice, who have been therapists on RCTs of novel psychotherapeutic interventions, and who have implemented educational interventions in our own classrooms, these characterizations of the alleged homogeneity of environmental interventions sound disconnected from the reality of social and behavioral science. Environmental interventions are, on the whole, more like a PGS than they are like lithium: they are cocktails “with thousands of ingredients of small efficacy,” the formulation of which differs across people.

This is perhaps most obvious in the case of therapeutic and educational interventions delivered one-on-one. In an RCT of cognitive behavioral therapy, for instance, the “underlying treatment … may in a real sense differ for every single unit,” because the content of every session is tailored specifically to the individual (Smith, Reference Smith2022, p. 656). But even interventions that are not psychotherapeutic might be, in practice, implemented in a highly idiosyncratic way (see, e.g., an ethnography of welfare case workers by Watkins-Hayes, Reference Watkins-Hayes2009). And, many interventions in the social and behavioral sciences package together multiple services, not all of which are taken up (or taken up in the same way) by every participant. Consider the High/Scope Perry Preschool Program (HPPP), which we discussed in our target article. This intensive intervention combined attending preschool for 2.5 hours, 5 days a week, for 2 years, home visits by teachers for 1.5 hours per week, and monthly small groups for parents. Given the complexity of the intervention, “one is still left without actually being able to pinpoint what it was in the Perry Preschool Project that actually influenced later adult outcomes” (Schneider & Bradford, Reference Schneider and Bradford2020, p. 52).

In this way, a binary variable reflecting the presence or absence of intent-to-treat in an environmental RCT does not always, or even usually, give us granular information about the relevant difference-maker(s) or assure that the relevant difference-maker(s) are experienced homogenously across people. Rather, a binary treatment indicator often represents a gross simplification (Heiler & Knaus, Reference Heiler and Knaus2021). The situation with oft-cited naturally occurring environmental exposures may be of even lower resolution: “treatments” like being drafted into the military (Angrist, Reference Angrist1990) or living in a region of Holland occupied by the Nazis (Stein, Susser, Saenger, & Marolla, Reference Stein, Susser, Saenger and Marolla1972) are hardly homogenous or “well-defined.”

Thus, even as philosophers and plant geneticists extol the homogeneity of environmental interventions in the social and behavioral sciences, interventionists themselves paint quite a different picture: “We must expect, study and capitalize on the heterogeneity that characterizes most effects in science” (Bryan, Tipton, & Yeager, Reference Bryan, Tipton and Yeager2021, p. 986). In fact, environmental interventionists sound remarkably like behavioral geneticists: “The researcher faces a tough trade-off between interpretability and statistical power or, put differently, between learning about the effects of the underlying heterogeneous treatments and the sample size available for studying each treatment” (Smith, Reference Smith2022, p. 656). These quotes illustrate that the problem of causal stimulus heterogeneity, while definitely a formidable challenge to mechanistic understanding, is not a challenge that is unique to the study of genetic causes, but is rather a difficulty that besets most studies in the social and behavioral sciences.

In light of this shared challenge, we wholeheartedly agree with Bondarenko that the “‘second-generation’ goals of causal inquiry in the context of human behavior cannot be achieved by genetics alone, nor do genetically informed research designs provide the only possible path toward mechanistic understanding.” Indeed, at no point in our target article did we suggest that genetics can achieve anything alone, nor did we suggest (as Smith & Downes alleged) that we “hope to supplant” environmental studies. Although we are optimistic that deeper measurements of the genome and further advances in fine-mapping and gene prioritization methods will result in a higher resolution understanding of genetic difference-makers, we also think that behavior genetics should incorporate more of the conceptual and methodological tools developed by environmental interventionists who are taking heterogeneity seriously, including methods for causal inference when there are multiple versions of treatment (VanderWeele, Reference VanderWeele2022; VanderWeele & Hernan, Reference VanderWeele and Hernan2013). As Durlauf & Rustichini highlight, formal theoretical structures are needed to reveal sources of heterogeneity and generate deeper causal explanations of how biopsychosocial systems produce human behavior. Again, we agree with commentators like Taylor, Weiss, & Marshall that the “genome [is] one resource (among many) used by the developmental system to grow,” and indeed, our entire discussion of average difference-makers as non-unitary causes is based on the idea that genes “operate within intricate causal systems” (target article, abstract). As we will discuss next, we also believe that, by virtue of being a difference-making component of the causal system, genetic data may play a key role in helping identify complex etiological models.

R3.3. Which designs give actionable information?

One word that recurred throughout the commentaries was “actionable.” Turkheimer intimated that knowledge of genetic causes was not actionable because the effect sizes are too small: “The actionable part of the [genotype-phenotype] correlation, estimated as a real number in the form of a PGS, is … under 5% for even the most studied traits…” The 5% figure refers to the within-family effect size of an educational attainment polygenic score (PGS) on years of education in North American samples who have “European” genetic ancestry. Effects of this magnitude were trivialized as “weak” and “small and indeterminate.”

We disagree with this characterization. An R ² of 5% is approximately equal to a correlation (r) of 0.22, or to (assuming equal-sized groups) a Cohen's d of 0.46. By comparison, a study of Swedish children who were adopted into better socioeconomic circumstances found that adoption increased IQ scores, relative to the children's siblings who were reared by their biological parents (Kendler, Turkheimer, Ohlsson, Sundquist, & Sundquist, Reference Kendler, Turkheimer, Ohlsson, Sundquist and Sundquist2015) by around 4.5 IQ points, or a Cohen's d of 0.34. This effect of “family environment” (certainly a heterogeneous causal stimulus!) was described as “a significant advantage in IQ” (Kendler et al., Reference Kendler, Turkheimer, Ohlsson, Sundquist and Sundquist2015, p. 4612). Another study of children randomized to foster care, rather than to severely deprived institutional care, found that foster care increased average IQ scores at age 12 by d = 0.41 (Almas, Degnan, Nelson, Zeanah, & Fox, Reference Almas, Degnan, Nelson, Zeanah and Fox2016). Yet another study examined the effects of a major educational reform in Sweden, which increased the number of years of compulsory schooling, abolished academic tracking at grade six, and rolled-out a unified national curriculum (Meghir & Palme, Reference Meghir and Palme2005). Researchers leveraged the gradual implementation of the reform across different areas in Sweden and concluded that the reform increased years of schooling by about 3.5 months, d ~ 0.2. What these examples illustrate is that the estimated causal effect of the educational attainment PGS rivals the effect sizes that we observe when people experience radically sweeping changes to their environmental context.

Most effect sizes for specific environmental interventions are even smaller than 5%, and this is exactly what we would expect given the causal complexity of human behavior. Kraft (Reference Kraft2020), for instance, reviewed all of the educational studies funded by the U.S. government's Investing in Innovation fund, and found that the median effect size was d = 0.03. They concluded: “effects of 0.15 or even 0.10 SD should be considered large and impressive” (p. 248). Yeager and Dweck (Reference Yeager and Dweck2020) similarly summarized: “In the real world, single variables do not have huge effects. Not even relatively large, expensive, and years-long reforms do. If psychological interventions can get a meaningful chunk of a .20 effect size on real-world outcomes in targeted groups, reliably, cost-effectively, and at scale, that is impressive” (p. 1281). We agree. Our conclusion that genetic effect sizes are, for some phenotypes, impressive does not stem from grandiosity about genetics, but rather from humility about the difficulty of specifying any cause, artificially manipulated or naturally varying, that accounts for even a few percentage points of the variance in complex behavior.

Others pointed out that, unlike the results of some RCTs, causal genetic effects are not directly actionable because we cannot artificially manipulate the genome on the basis of that information. Pingault et al. wrote: “Thus, while RCTs can provide actionable evidence of a specific intervention's efficacy, a within-family genetic association only indicates the effect of inheriting one variant or another.” Markon similarly pointed out that “counterfactual theory … takes an unactionable epistemological stance: even if a counterfactual account informs about what would have occurred had things been different, it does not inform about what one can do now, given things as they are.” Kaplan & Bird concurred: “[t]he ‘shallowness’ of the causal knowledge gained in RCTs does not prevent them from being useful guides to practice … the situation in behavior genetics is nothing like this. Unlike in the case of RCTs, we cannot change the genetic variants associated with the phenotypic variation – and even if we could, doing so would be wildly irresponsible.” We made a similar point in our target article, writing that “even if we concede that, at a conceptual level, genes could cause average differences in human behavior, at a practical level, it is not readily apparent what we would do with this knowledge.… [W]e cannot (and should not) readily apply knowledge of genetic causes to change the genomes of large swathes of the population in the hopes of changing their outcomes” (target article, sect. 1.2, para. 6).

Given that we are not planning to change people's genotypes, Kaplan & Bird ask, “what would we gain from even an accurate finding that a particular genetic variant was associated with downstream effects?” This is a curious question to pose, as it is precisely the question that we address, at length, in the target article on which they are supposedly commenting. Our short answer is that we agree with Lewontin (Reference Lewontin1974) (quoted by Turkheimer): “The analysis of causes in human genetics is meant to provide us with basic knowledge we require for correct schemes of environmental modification and intervention” (p. 409).

And, so far, the project of devising correct schemes of environmental modification and intervention has been far less successful than commonly imagined (Kraft, Reference Kraft2020). Given that there is clearly room for improvement for successful identification of environmental intervention targets, we agree with Bondarenko that one “important role for genetic data” is as “controls in the study of environmental variables … when applied with care, genetic controls may help address some of the worries that our ‘first-generation’ knowledge of environmental factors does not meet a stringent epistemic standard.” We also agree with commentators who pointed out that that genetic results can be usefully exploited in Mendelian randomization studies (Pingault et al.) for the study of phenotypic causation (Pingault, Richmond, & Davey Smith, Reference Pingault, Richmond and Davey Smith2022), which “given their speed and relative low cost” are a “useful first-step to guide future randomization/intervention studies” (Haworth & Wootton).

Finally, simply knowing that something has genetic causes – even if you can't identify them or manipulate them – can and has changed clinical practice. For example, genetic research on substance use disorders (SUDs) contributed to a paradigm shift in conceptualizing addiction as a chronic disease that resides within the body of the individual rather than as a moral deficit that resides within their will (Hall, Carter, & Forlini, Reference Hall, Carter and Forlini2015; Volkow & Koob, Reference Volkow and Koob2015). Now, educating patients about genetic effects on addiction is a standard part of psychoeducation that can reduce stigma and increase motivation for treatment (Hassan et al., Reference Hassan, Ragheb, Malick, Abdullah, Ahmad, Sunderji and Islam2021; Ray, Reference Ray2012). Causal knowledge has the power to produce important changes in our intentions and actions as scientists and clinicians, whether those causes can be manipulated or not.

R3.4. Which designs have external validity?

Most commentators who were critical of our target article had the intuition that RCTs in the social and behavioral sciences were valuable research endeavors, and objected to our comparing within-family genetic studies to them. A few commentators, however, brought up that RCTs also have their limitations, most prominently, that they sacrifice external validity and generalizability for the sake of internal validity. Shen & Feldman, for instance, pointed out that causal knowledge built from within-family comparisons does not necessarily generalize on a population scale. Borger et al. also decried the “limited ability…to generalize,” and Siegel warned that GWAS results “may not apply to other ethnic groups, a non-uniformity that may exacerbate health disparities.” Syed offered a particularly trenchant summary of the problem:

For those interested in “humans in general” (Byrne & Olson), causal knowledge that is not built from culturally and demographically representative samples – and therefore not expected to apply equally across subgroups – is inherently flawed. As Syed said, “no diversity in, no causes at all.” In contrast, others were less interested in “humans in general” knowledge, and were instead concerned with the “individual making important life-choices” (Miller). And, in an interesting counterpoint to Syed's commentary, Bourrat suggested that, in some cases, “local causal knowledge can be more useful for explanation and intervention than more generalisable knowledge.” Similar to Cerezo's emphasis on triggering conditions, Bourrat pointed out that accounting for context can reveal specificities about a causal relationship that are masked when aggregating more generally.

A medical example of the unique contribution of local causation comes from oncology research, where chemotherapy was found to have no average impact on patient survival in a cohort of individuals with stage IB lung cancer, but rather improved patient survival only in those with a tumor size greater than 0.4 centimeters (Strauss et al., Reference Strauss, Herndon, Maddaus, Johnstone, Johnson, Harpole and Green2008). This is what Bakermans-Kranenburg and Van Ijzendoorn (Reference Bakermans-Kranenburg and Van Ijzendoorn2015) referred to as the “hidden efficacy of interventions.” The effect of chemotherapy on patient survival cannot be expected to generalize across all individuals with stage IB lung cancer, but it makes a significant difference for some people.

This example illustrates that locality is not always a curse and that non-portable causes can still sometimes be useful. Finding that a genetic effect holds within a particular ancestral group but not another, or is manifest under certain social conditions but not others, or applies within families but not between them, can be valuable because it allows us to build causal knowledge within family or cultural systems.

The tension between Syed's and Bourrat's commentaries – both of which we think make incisive and valuable points – highlights what Richard Levins described as “the contradictory desiderata of generality, realism, and precision…” (Levins, Reference Levins1966, p. 431). The more we attempt to generalize, the more we collapse over dimensions of variability that meaningfully influence the causal relationship; the more we specify local variables, the more we restrict the applicability of our findings (see Yarkoni [Reference Yarkoni2022] and related commentaries for a discussion about the precision of estimation/breadth of generalization trade-off).

With these contradictory desiderata in mind, we strongly agree with Syed that randomization is not enough: who is being randomized across what dimensions of human experience? We endorse their conclusion that “diversity is central to first-generation studies” and that “the racial/ethnic diversity of samples included in behavior genetic studies… must be central to any effort to build generalizable causal knowledge.” Similarly, we agree with Byrne & Olson that, behavior geneticists have a responsibility to make “analytic choices” that “enhance the visibility of context,” in particular, sampling across sociocultural contexts that have heretofore been largely excluded from genetics research. And, we agree with Eftedal & Thomsen that “[b]ehavioral genetics should broaden its empirical scope beyond single-culture WEIRD samples.”

Accordingly, we applaud recent efforts within the field to conduct GWASs in non-European ancestral groups (Gulsuner et al., Reference Gulsuner, Stein, Susser, Sibeko, Pretorius, Walsh and McClellan2020; Pereira, Mutesa, Tindana, & Ramsay, Reference Pereira, Mutesa, Tindana and Ramsay2021), advance methodology to increase the integration of diverse samples in GWASs (Mathur et al., Reference Mathur, Fang, Gaddis, Hancock, Cho, Hokanson and Johnson2022), improve discovery of within-family genetic effects (Howe et al., Reference Howe, Nivard, Morris, Hansen, Rasheed, Cho and Davies2022), develop equitable partnerships among international institutions that promote resource sharing and shared-infrastructure development (Martin et al., Reference Martin, Stroud, Abebe, Akena, Alemayehu, Atwoli and Chibnik2022), and build more representative biobanks like the Trans-Omics for Precision Medicine Program that allow for genetic discovery within cultural subgroups (Popejoy & Fullerton, Reference Popejoy and Fullerton2016). These initiatives represent important steps toward building more representative causal knowledge. Finally, we also echo Tarchi et al.'s warning that an absolute requirement for a genetic study to have a causal identification strategy or to provide mechanistic understanding might have undesirable consequences for representation in genetics: “to consider ‘association studies’ as secondary or even detrimental should be critically evaluated,” lest we exacerbating problems of exclusion in genetic research.