Introduction
It has been over 10 years since the paper of Hotopf et al. (Reference Hotopf, Churchill and Lewis1999) on pragmatic trials. This understandable call was partly a response to the increasing realization that even well-conducted randomized controlled trials did not necessarily address the needs of everyday care. Most trials involved participant groups so selected that they would be rare in routine care, and evaluated treatments administered so rigidly as to further remove the practice from reality and, finally, measured outcomes that were of questionable importance to clinicians, policymakers or recipients of care. The article by Hotopf et al. (Reference Hotopf, Churchill and Lewis1999) highlighted some of the benefits of pragmatic trials but with few examples to point to at that time. Most lessons had been learnt from colleagues in cancer, heart disease and perinatal medicine. Since that time, however, there have been many developments regarding our understanding of what pragmatism really means in trials and in how to conduct these types of studies in mental health. Further papers followed that highlighted a developing understanding of these types of studies within mental health care evaluation with increasing numbers of examples to cite (March et al. Reference March, Silva, Compton, Shapiro, Califf and Krishnan2005; Stroup, Reference Stroup2011).
Despite this, ‘pragmatic’ remains a much misunderstood concept within trials and is often used as a broad, fashionable and, often, unhelpful label. Over the last years it became apparent that the concept of pragmatism in randomized trials was not binary (Stroup & Geddes, Reference Stroup and Geddes2008). It has been recognized that considering randomized controlled trials as purely explanatory (studies that focused on whether, in ideal circumstances, a treatment would have any effect) or pragmatic (a study which investigated whether the treatment, given in real-world circumstances, really did have clinically meaningful effects) was too simplistic. The concept of explanatory and pragmatic trials being at opposite poles was dismissed in favour of the idea that there was a continuum between maximally pragmatic or fully explanatory.
Recently there has been a genuine advance in our understanding of pragmatism within trials (Thorpe et al. Reference Thorpe, Zwarenstein, Oxman, Treweek, Furberg, Altman, Tunis, Bergel, Harvey, Magid and Chalkidou2009). The authors defined 10 domains encapsulating explanatory and pragmatic approaches and covering the most important aspects of a trial design (Table 1).
Table 1. The 10 domains relevant to explanatory and pragmatic trials a
Thorpe et al. (Reference Thorpe, Zwarenstein, Oxman, Treweek, Furberg, Altman, Tunis, Bergel, Harvey, Magid and Chalkidou2009) went on to define and explain each of these domains in detail and have left us with an elegant tool by which pragmatism could be estimated. These attractive diagrams (Fig. 1) can be created for those designing trials or assessing trials at the protocol stage of the funding application process to give some impression of applicability to clinical circumstances. Estimates are made regarding each parameter and this can result in ‘open’ or ‘closed’ diagrams illustrating how pragmatic or explanatory a trial is likely to be (Fig. 2).
Fig. 1. The blank Pragmatic–Explanatory Continuum Indicator Summary (PRECIS) ‘wheel’.
Fig. 2. Two examples of trials.
Understandably, this has triggered papers outlining specific tools for both trials and studies within systematic reviews (Tosh et al. Reference Tosh, Soares-Weiser and Adams2011). These tools, with various degrees of reliability, can produce numerical estimates of the applicability of trials to the real world (Table 2).
Table 2. Scoring trial protocols estimating degrees of pragmatism a
ACHIEVE, Achieving healthy lifestyles in psychiatric rehabilitation; ERP, enhanced relapse prevention; DYD, Down Your Drink; PTSD – Yoga, Post-traumatic stress disorder – Yoga; CATIE, Clinical Antipsychotic Trials of Intervention Effectiveness; FIAT, Financial Incentives (for) Adherence Trial; TREC-SAVE, Tranquilizǎcção Rápida - Ensaio Clínic: Segurança no manejo da Agressão: Viabilidade e Ética na contenção e isolamento.
a From Tosh et al. (Reference Tosh, Soares-Weiser and Adams2011) and corrected to remove reference errors in the source table.
Mental health now has a series of trials to illustrate, to greater or lesser degrees, a more pragmatic approach. Previously the speciality had no choice but to employ explanatory trials in policy making. Currently, however, with increases in skills and experience, decision makers begin to have options. The last decade has seen a series of trials that falls much more into a pragmatic category. For example, Clinical Antipsychotic Trials of Intervention Effectiveness (CATIE; Lieberman et al. Reference Lieberman, Stroup, McEvoy, Swartz, Rosenheck, Perkins, Keefe, Davis, Davis, Lebowitz, Severe and Hsiao2005) and Cost Utility of the Latest Antipsychotic Drugs in Schizophrenia Study (CUtLASS; Jones et al. Reference Jones, Barnes, Davies, Dunn, Lloyd, Hayhurst, Murray, Markwick and Lewis2006) are important ground-breaking studies with greater levels of pragmatism than had previously been seen in such large mental health trials. Other randomized studies such as BALANCE (Bipolar Affective disorder: Lithium/ANticonvulsant Evaluation; Geddes et al. Reference Geddes, Rendell and Goodwin2002) and the series of Tranquilizǎcção Rápida – Ensaio Clínic (TREC; Alexander et al. Reference Alexander, Tharyan, Adams, John, Mol and Philip2004; Huf et al. Reference Huf, Coutinho and Adams2012) trials further developed techniques in pragmatism within mental health. All juggle the differing needs of researcher, clinician, policymaker and service user within a trial. In these studies the concept of pragmatism has been declined in different ways, but some key common characteristics include the following. First, all these studies investigated already marketed medicines, with the aim of answering comparative effectiveness. Second, the choice of control group interventions was mainly based on prevailing clinical practice. Third, patient populations were included minimizing exclusion criteria and being focused more on clinical presentation than on formal diagnoses. Fourth, the choice of outcome measures was based on pragmatic reasoning around what really outcome means for practising doctors. Finally, the sample size was calculated to show superiority, which implies that value is attributed to treatments that are associated with some degree of advantage over standard ones.
Clearly, critical issues may arise from such study designs. For example, the concept of ‘real life’ cannot be unambiguously defined, and therefore the results of pragmatic trials carried out in European countries might be hardly applicable to a very different system of care, such as, for example, a very low-income system of care. The same criticism may be applied to the choice of control treatments in pragmatic trials, as prevailing clinical practice may highly differ and therefore what may be relevant is a specific system of care may not be similarly relevant in others. Internal validity might be another critical aspect, as in some circumstances it is difficult to identify the ‘active ingredients’ of effective interventions assessed under a highly pragmatic design.
Progress has also been made in relation to reporting of trials. The CONSORT (Consolidated Standards of Reporting Trials) statement – a tool intended to improve quality of reporting and minimize risk of bias – has been extended to take into account pragmatism of trials (Zwarenstein et al. Reference Zwarenstein, Treweek, Gagnier, Altman, Tunis, Haynes, Oxman and Moher2008). Additional text was added specifically for eight items, with detailed information on background, participants, interventions, outcomes, sample sizes, blinding, participant flow and generalizability of findings. These extensions represent an advance towards transparent and complete reporting of trials in general, and, in relation to the pragmatism of a trial, a guide for clinicians and policymakers for judging applicability to everyday clinical practice (Zwarenstein et al. Reference Zwarenstein, Treweek, Gagnier, Altman, Tunis, Haynes, Oxman and Moher2008). This progress came from a change in culture, not least of which was in psychiatry, where farsighted researchers and clinicians anticipated the broad and hitherto often overlooked needs of the many stakeholders for trial data (Tansella et al. Reference Tansella, Thornicroft, Barbui, Cipriani and Saraceno2006).
The next 10 years
There is more to do. Legislation could dramatically change the landscape for pragmatic trials.
The regulatory authorities
The issue of pragmatic versus explanatory is a reflection of the aim or objectives of the trial. Pragmatic trials aim to provide answers to real-world questions perceived to be relevant to patients, clinicians and policymakers. Common to all such trials are measures of patient-valued outcomes. Regulatory trials do not necessarily aim to answer these types of questions. Regulatory trials aim to obtain regulatory approval. Currently, phase III trials are often carried out by industry and are highly explanatory, while phase IV trials may more often be carried out by groups of physicians who have a clinical dilemma and are more pragmatic (Barbui & Bighelli, Reference Barbui and Bighelli2013b ).
The Food and Drug Administration in the USA and the European Medicines Agency (EMA) are regulatory bodies that establish rules to get new medicines approved and marketed. The current explanatory standard of phase III studies, therefore, is a consequence of current rules and requirements issued by these agencies. Changing regulatory requirements could greatly encourage a more real-world and pragmatic set of designs within phase III studies (Wood, Reference Wood2006). For example, in Europe new drugs can be evaluated with no comparison with active alternative treatments. As a consequence, phase III studies compare new drugs with placebo to make these new drugs eligible for registration – often not the question troubling the ‘real world’ of clinical care in which there may well be viable alternative drugs. If comparisons with other drugs are made within phase III studies, these usually rely on demonstrating therapeutic ‘non-inferiority’ – in agreement with current EMA requirements. If the scientific community agrees that active–control superiority is desirable, clinical trials should be designed and powered to generate the evidence and European pharmaceutical legislation should then be induced to incorporate this requirement, at least in addition to placebo-controlled trials (Garattini & Chalmers, Reference Garattini and Chalmers2009). It has recently been suggested that regulatory authorities may require at least one two-arm head-to-head pragmatic trial in order to demonstrate superiority of the investigational product over an active comparator (Barbui & Bighelli, Reference Barbui and Bighelli2013a ).
Conduct of studies
Increasingly, government funders of studies have become aware that the trials they have supported did not have the population-wide impact that they had intended (Duijnhoven et al. Reference Duijnhoven, Straus, Raine, de Boer, Hoes and De Bruin2013). Explanatory studies are less likely to achieve wide impact for the general population because, by definition, those eligible for the explanatory study are a highly specific group of people from which most of the general population is excluded. As a result taxpayers’ money has funded studies that help far fewer taxpayers than had been originally intended. Broad-brush stipulation from funders – that real-world applicability is required in trial design (NHS National Institute for Health Research, 2013) – and then authoritative governance to ensure that funding panels rigorously adhere to this stipulation can greatly help shift research culture and tradition. Additional legislation can help more. A case illustration helps. In Italy, the Gruppo Italiano per lo Studio della Streptochinasi nell'Infarto Miocardico (GISSI) trial was a highly pragmatic study investigating intravenous streptokinase in early acute myocardial infarction (Gruppo Italiano per lo Studio della Streptochinasi nell'Infarto Miocardico, 1986). This enrolled 11806 patients in 176 coronary care units. Subsequent to the first highly influential paper of 1986, debate focused around the need for more similar studies. In 2004 a Ministerial Decree was issued recognizing the public health value of the original study and the need of establishing rules to help implement pragmatic independent phase IV clinical trials (Tognoni & Franzosi, Reference Tognoni and Franzosi2005). The Decree states that if a set of conditions is met (study coordinating centre is independent from drug companies, study results can be disseminated autonomously, there is no personal financial interest in studying the drugs included in the trial, study drugs are already in the market and are studied within the licensed indications), then the Italian health system supports the conduct of the trial in three ways. First, drug costs are paid by the Italian health system; (2) there are no fees for submitting the study protocol to the local ethics committees; (3) continuing medical education credits are provided to local investigators. The impact of this Decree on the development of pragmatic experimental studies has not been assessed yet, but from a cultural and theoretical viewpoint it has already been seen as a major advance in Italy.
Conclusions
When Hotopf et al. (Reference Hotopf, Churchill and Lewis1999) wrote about pragmatic trials in psychiatry their tone was almost wistful as regards other more advanced specialities. Things have changed and are changing further. We now have excellent examples of truly pragmatic trials in mental health. Funders have helped encourage more emphasis on the need for such studies but ‘local’ and trans-national regulations could help more. Consumers of the evidence should have a greater voice in generating the research agenda and, as this happens (Lloyd & White, Reference Lloyd and White2011), the questions generated are, assuredly, more likely to be answered by a pragmatic approach to trials.
If we do not devalue the ‘pragmatic’ label through misuse (Sackett, Reference Sackett2013), the next decade will see a burgeoning of mental health trials with a much more emphasis on showing if treatments really work in everyday care.
Acknowledgements
This paper received no specific grant from any funding agency.
Declaration of Interest
None.
