Considerable missing data with imputation of low prevalence events

It is well-known that those who experience adverse events are less likely to complete outcome measures (see e.g. Ioannidis et al., Reference Ioannidis, Evans, Gøtzsche, O'Neill, Altman and Schulz2004). It is also known that this phenomenon is exacerbated when researchers and instructors/clinicians are associated with the program (Allen, Chandler, Mandimika, Leisegang, & Barnes, Reference Allen, Chandler, Mandimika, Leisegang and Barnes2018); the authors themselves have recently published on this issue (Goldberg & Tucker, Reference Goldberg and Tucker2019). Thus, it is important to carefully consider what the data attrition actually reflects in the work by Hirshberg et al. (Reference Hirshberg, Goldberg, Rosenkranz and Davidson2020; as high as ~23% in their largest sample). Regardless of course completion numbers, the larger percentage of missing data is concerning because it could be masking harm (Schulz & Grimes, Reference Schulz and Grimes2002). The authors still considered that data were missing at random after adjusting imputation for just gender and participation year. Although these variables were predictive of missingness, they are insufficient to address potential causes for missingness when investigating harm (see e.g. Sterne et al., Reference Sterne, White, Carlin, Spratt, Royston, Kenward and Carpenter2009). Moreover, harm outcomes are expected to be low-prevalence events, making data imputation even more problematic.

Limited measurement of harm

The authors rightly assert that there is no consensus about ‘harm’ in psychosocial interventions. Acknowledgement of this lack of consensus is not, however, justification to ignore the many ways in which psychosocial interventions may potentially cause harm (Linden, Reference Linden2013). Hirshberg et al. (Reference Hirshberg, Goldberg, Rosenkranz and Davidson2020) focus their analyses on the deterioration of illness and the emergence of new symptoms as endorsed on a medical symptom checklist. It is worth noting that under-detection of adverse events is likely with traditional simple checklists (see Allen et al., Reference Allen, Chandler, Mandimika, Leisegang and Barnes2018). It is of additional note that MBSR guidelines not only encourage participants to practice outside of formal activity of the courses but also recommend that practice continue beyond the course (Santorelli, Meleo-Meyer, Koerbel, & Kabat-Zinn, Reference Santorelli, Meleo-Meyer, Koerbel and Kabat-Zinn2017). Continued practice outside the supportive period during which training is being provided, along with encouragement for participants to engage in meditation retreats (Santorelli et al., Reference Santorelli, Meleo-Meyer, Koerbel and Kabat-Zinn2017), makes the post-intervention period especially important to monitor. People cannot report harms if they are not asked about them and it is very possible that more serious harms only emerge when the practice is implemented without the support of MBSR instructors. It is essential that longer follow-up periods are examined prior to claiming MBSR does no harm.

Potential sample bias

The overall level of impairment (take Global Severity Index - GSI, for example) is significantly higher in the community sample than either of the two groups in the randomized controlled trials (RCTs; v. MBSR, Cohen's d = −0.99, t(2309) = −10.03, p < 0.001; v. WLC, Cohen's d = −1.17, t(2271) = −9.86, p < 0.001), and is above the T value (i.e. 60) commonly used to detect psychological distress (M = 61.78, s.d. = 42.67). Thus, the community sample represents a large selection of individuals who sought out and paid to receive MBSR, presumably, with the aim of helping them to manage their psychological distress. As this represents a non-blinded intervention, it likely suffers from inflated effect size estimates (pre-post change: Community d = −0.70; RCT d = −0.07) and attrition rates (~23% missing data in Community v. ~10% missing data in RCT; see Hróbjartsson, Emanuelsson, Thomsen, Hilden, and Brorson, Reference Hróbjartsson, Emanuelsson, Thomsen, Hilden and Brorson2014). Self-selection, desire to improve, along with financial and time investment in the treatment, makes it likely that participants were inclined to give favorable ratings at discharge. Those who did not receive benefits are more likely to have discontinued treatment earlier on in the process.

Collation of data across potentially different trials

The authors mention that they collated data from three trials and give two grant numbers but no trial protocols or publications. We were unable to find waitlist-controlled trials published under the grant U01AT002114-01A1. We found one waitlist and Health Enhancement Program (HEP; MacCoon et al., Reference MacCoon, Imel, Rosenkranz, Sheftel, Weng, Sullivan and Lutz2012) controlled trial under P01AT004952 that measured GSI; curiously enough they failed to report GSI results (Goldberg et al., Reference Goldberg, Wielgosz, Dahl, Schuyler, MacCoon, Rosenkranz and Davidson2016). We failed to find any information on two of the three trials. We also failed to identify a prospective protocol for the analyses presented by Hirshberg et al. While we recognize the time commitment of such efforts, it is widely recognized in the scientific community that trial reporting needs to be much more detailed and comprehensive to avoid major biases (Schulz, Altman, & Moher, Reference Schulz, Altman and Moher2010). Lack of transparency and pre-registration of detailed analysis plans increase chances of bias, compromising confidence in the results. Furthermore, data from trials were combined as if it was a single trial without any consideration for potential (unknown to us) variability among trials; such amalgamation can lead to bias and overprecise estimates of effect (Tierney et al., Reference Tierney, Vale, Riley, Smith, Stewart, Clarke and Rovers2015).

Inappropriate conclusion regarding preventative potential of MBSR

The authors use the deterioration of the wait-list group, compared to the improvement of the community group as a means of saying that community MBSR is preventative of harm. The comparison is inappropriate for three reasons. The first is that one cannot reasonably compare an active, open-label intervention to the control arm of an RCT. The second is that the estimation of effect in the community MBSR group is likely an overestimate, as discussed above. The third is the fact that participants allocated to wait-list groups in active randomized controlled trials are known to exhibit the nocebo effect (i.e. deterioration of symptoms generated by negative expectations pursuant to not getting the active intervention; Furukawa et al. Reference Furukawa, Noma, Caldwell, Honyashiki, Shinohara, Imai and Churchill2014; Gold et al. Reference Gold, Enck, Hasselmann, Friede, Hegerl, Mohr and Otte2017). In the study in question, participants in the wait-list condition deteriorated on the medical symptom checklist by Cohen's d = −0.64; well within the range of the observed nocebo effects (see e.g. Furukawa et al., Reference Furukawa, Noma, Caldwell, Honyashiki, Shinohara, Imai and Churchill2014). A recent meta-analysis showed that the difference in outcomes of active interventions can vary as much as d = 0.50 when compared against active controls v. waitlists (Furukawa et al., Reference Furukawa, Noma, Caldwell, Honyashiki, Shinohara, Imai and Churchill2014). A comparison with a control group such as the HEP, to control for non-specific effects, and used in at least some of the three trials analysed by the authors, would have provided the kind of evidence desired while mitigating the nocebo effect.

For all these reasons we do strongly disagree with the assertion that ‘these data provide strong evidence against claims that MBSR may increase harm’. The evidence presented in this article is insufficient to support the claim. Moreover, the authors appeal to a logical fallacy; absence of evidence rather than evidence of absence – not finding evidence of adverse effects in MBSR does not mean that there are none. We are also concerned that the abstract, often the only section people read, fails to acknowledge any of the limitations that undermine confidence in the results.

The claims made by the authors provide a much sought-after reassurance that is as yet unfounded in our view. A series of recent studies have suggested that adverse events can and do happen in meditation-based interventions including those with a focus on mindfulness (Cebolla, Demarzo, Martins, Soler, & Garcia-Campayo, Reference Cebolla, Demarzo, Martins, Soler and Garcia-Campayo2017; Farias, Marald, Wallenkampf, & Lucchetti, Reference Farias, Marald, Wallenkampf and Lucchetti2020; Schlosser, Sparby, Voros, Jones, & Marchant, Reference Schlosser, Sparby, Voros, Jones and Marchant2019). None of this is to say that people should not undertake MBSR or that MBSR is likely to be harmful. Much to the contrary, MBSR is likely to be helpful for most people for a wide array of issues (de Vibe et al., Reference de Vibe, Bjorndal, Fattah, Dyrdal, Halland and Tanner-Smith2017). However, methodological and reporting quality needs to improve before making strong evidential claims about adverse effects in interventions with high public health relevance such as MBSR.