The term ‘evidence-based medicine’ has developed an intimidating connotation in recent years. It is all about us. Britain has always been known for its devotion to pragmatic empiricism, and doctors, in particular, now seem to be living in a virtual furor empiricus. Everything has to be tested empirically. It has been made clear that if consultants do not practise ‘evidence-based child psychiatry’ or if they cannot practise it because the necessary evidence does not exist, they may have resources removed from them. So we need to look carefully at this word ‘evidence’. The more narrowly or rigidly it is defined, the more difficult it will be to justify our activities; the more broadly it is defined, the easier it will be. In this paper I shall explore the meaning of the term ‘evidence’ and its relevance to the field of child psychiatry.

EVIDENCE-BASED CHILD PSYCHIATRY

Sackett et al (Reference Sackett, Richardson and Rosenberg1997) define evidence as “ external clinical evidence from systematic research”. They suggest that the best type of ‘evidence’, the gold standard, is the randomised controlled trial (RCT) or, better still, the meta-analysis of several such trials. If no RCT has been carried out to guide a clinical decision, one should follow the trail to the next best available external evidence to judge whether a treatment does more good than harm, and work from there. The strongest evidence comes from “at least one systematic review of multiple well-designed randomised controlled trials”, and the weakest from “opinions of respected authorities based on clinical evidence, descriptive studies or reports of expert committees” (Reference Muir GrayMuir Gray, 1997, p. 61). Sackett et al (Reference Sackett, Richardson and Rosenberg1997) draw a sharp distinction between such ‘evidence’ and ‘clinical expertise’, which they define as “the proficiency and judgement that individuals acquire through clinical experience and clinical practice”. Although this distinction between external scientific evidence and ‘clinical expertise’ has gained wide currency, and may be appropriate in general medicine and surgery, it has the serious disadvantage that it conveys a message that clinical experience does not produce evidence.

Most consultant child and adolescent psychiatrists know, of course, that evidence derived from RCTs represents the ‘gold standard’ for evidence. However, in ordinary common parlance they are likely to define evidence as any information useful in making a clinical decision. Does this include experiences with similar patients? Most psychiatrists would indeed think of several relevant clinical experiences as reasonably strong evidence.

Further relevant information or ‘evidence’ may be derived from the individual case about which a decision has to be made. If, during an assessment of a seven-year-old boy with the features of attention-deficit hyperactivity disorder, it becomes clear that there is a marked improvement in behaviour when it has been indicated to his parents that they need to be firmer in their control of him, it may be reasonable to embark on a course of parent counselling or parent management training without initially using stimulant medication (a scientifically substantiated treatment for this condition), at least until the effect of a psychological approach has been assessed.

Some resolution of this issue concerning the nature of evidence may be achieved if one moves away from the sharp distinction proposed by Sackett et al (Reference Sackett, Richardson and Rosenberg1997) between scientific evidence and clinical expertise. When faced with a clinical problem, one can use information or evidence derived from the systematic study of groups of similar cases. One can also use information or evidence derived from previous experience of similar cases and from the assessment of the unique child and family that one is seeing. Rather than making a distinction between scientific evidence and clinical expertise, it might be better to think first of external evidence and then of case evidence. Case evidence refers to information derived from the unique features of the child and family in question. Bringing these two types of evidence together might help to solve the problem of what Szatmari (Reference Szatmari1998) has called “the two solitudes - the (different) spaces in which parents and professionals live, (which are) divided by the term ‘evidence’”.

Both types of ‘evidence’ are relevant, and the relationship of data to theory is similar, if not identical. In clinical research one moves inductively from observations of groups of similar cases to theory, which one then attempts to test using hypothetico-deductive methods, hoping to achieve generalisable findings. In the individual case, one moves inductively from the data obtained in an assessment, combining these with information derived from findings from generalisable research, to a theory about this particular case. One then tries, using a hypothetico-deductive model, to test this theory by checking it against other information or by attempting an intervention and observing to see whether or not the result is consistent with the theory. These are, or should be, interlocking processes in which evidence from unique cases feeds into generalisable research activity, and vice versa.

LIMITATIONS OF THE RCT

The use of such case evidence and other forms of clinical evidence is essential because, unfortunately, at this time and at least for a very long time into the future, it is likely that RCTs will have only a limited place among the range of influences affecting treatment interventions. There is a variety of reasons for this:

1. (a) In management, the symptoms that children show may need to be considered as reactions to predicaments or complicated and problematic life situations. Now, RCTs focus on symptoms and diagnosis and, in general (although this is not the case for some interventions, such as various multi-focal approaches for conduct disorder), ignore the predicaments. It is not sensible to focus on symptoms but to make no attempt to deal with predicaments where symptoms are clearly secondary.

2. (b) Randomised controlled trials, with a few exceptions (e.g. Reference Klein, Abikoff and KlassKlein et al, 1997), are generally carried out with the intention of improving the outcome of one specific disorder, and yet most children seen in clinical practice have comorbid disorders. In the recently published Audit Commission study, 95% of more than 17 000 children whose attendances were studied had more than one diagnosis (Audit Commission, 1999).

3. (c) The great majority of RCTs are carried out on individuals whose disorders reach DSM research diagnostic criteria, and yet the disorders shown by many children who are socially impaired by their behaviour and emotional problems do not fulfil such criteria. Angold et al (Reference Angold, Costello and Farmer1999) have found that over 50% of children attending clinics in the Great Smoky Mountains Study do not reach DSM or ICD research criteria for a diagnosis and yet half of these are significantly impaired in their social functioning.

EXISTING FINDINGS FROM WELL-CONDUCTED RCTs

These limitations reduce the relevance of the findings from RCTs in our field, but this is not to suggest that clinically relevant information from such trials does not exist because it certainly does. To summarise briefly, because the information has been well reviewed elsewhere (e.g. Reference Target, Fonagy, Roth and FonagyTarget & Fonagy, 1996), we know from well-conducted trials that medication produces symptom reduction in attention-deficit hyperactivity disorders (Reference Weiss and LewisWeiss, 1996), psychotic disorders (Reference McClellan and WerryMcClellan & Werry, 1994), obsessive-compulsive disorders (Reference DeVaugh-Geiss, Maroz and BiedermanDeVaugh-Geiss et al, 1992), tics, Gilles de la Tourette's syndrome (Reference Shapiro, Shapiro and FulopShapiro et al, 1989), enuresis (on a short-term basis) (Reference Blackwell, Currah, Kolvin, MacKeith and MeadowBlackwell & Currah, 1973), as well as depressive and anxiety disorders (Reference Emslie, Rush and WeinbergEmslie et al, 1997). We know that family therapy is better than routine support and weight maintenance treatment for adolescent girls with anorexia nervosa (Reference Russell, Szmukler and DareRussell et al, 1987) and in the management of some psychosomatic disorders such as asthma (Reference Lask and MatthewLask & Matthew, 1979). Psychoanalytical psychotherapy improves compliance in difficult-to-control diabetes (Reference Moran, Fonagy and KurtzMoran et al, 1991). Cognitive-behavioural therapies are effective in obsessive-compulsive disorders (Reference MarchMarch, 1995), bulimia in older adolescents (Reference Wilson, Eldredge and SmithWilson et al, 1991), general anxiety disorders and some phobias (Reference Kendall and Southam-GerowKendall & Southam-Gerow, 1996), post-traumatic stress disorders (Reference Smith, Perrin, Yule and GrahamSmith et al, 1998), depressive disorders (Reference Harrington, Wood, Verduyn and GrahamHarrington et al, 1998), non-organic pain disorders (Reference Sanders, Shepherd and CleghornSanders et al, 1994), the enhancement of social skills (Reference Beelman, Pfingset and LoselBeelman et al, 1994) and the psychiatric consequences of sexual abuse (Reference Jones and RamchandiJones & Ramchandi, 1999). Interpersonal therapy is promising in the treatment of adolescent depression (Reference FombonneFombonne, 1998). Behavioural therapies are useful in the treatment of enuresis (Reference Dische, Yule and CorbettDische et al, 1983). Parent management training reduces conduct problems in young boys with conduct disorder (Reference Kazdin, Siegel and BassKazdin et al, 1992) and multi-systemic family therapy (Reference Henggeler, Schoenwald and BorduinHenggeler et al, 1998), parent management training and functional family therapy improve conduct disorders and reduce subsequent delinquency in older boys and girls (Reference KazdinKazdin, 1997).

It is possible to draw some general conclusions from the findings of these studies. First, different conditions respond to different therapies. There is no longer any room for clinicians who are guided by only one theory and who are only prepared to apply one type of treatment to all problems they see. Second, although effect sizes are larger in so-called ‘laboratory’ studies with volunteer samples (Reference Weisz, Donenberg and WeissWeisz et al, 1995), they are relatively small, in the region of 0.2-0.3, even in the best outcomes obtained in clinical samples. Third, when either might have been applied, the evidence for the effectiveness of behavioural therapies is greater than the evidence for non-behavioural therapies, including non-behavioural family therapy, individual psychodynamic psychotherapy and counselling. This may be because non-behavioural therapies have been less well evaluated, and of course unevaluated therapies are not necessarily ineffective.

The implications of this finding of the superiority of evidence for behavioural therapies are, as I see them, threefold. First, in situations where behavioural or non-behavioural types of therapy might be used, and the superiority of the former has been established, these should be preferred. The second implication is that there is a need for more outcome research with non-behavioural methods, using techniques that, if found to be effective, could be applied realistically in National Health Service settings. There is little point in the conduct of research on treatments that are either prohibitively expensive or demand unrealistic levels of compliance from family members. Third, where the efficacy of behavioural treatments has been established, non-behavioural treatments should not be applied except as part of well-designed research studies examining outcome using standardised measures.

BARRIERS TO IMPLEMENTATION OF EXISTING FINDINGS FROM TREATMENT RESEARCH

As the results of the Audit Commission report (Audit Commission, 1999) make clear, in most clinics a wide range of treatments is on offer. Nevertheless, it is also sadly clear that in a number of clinics the most effective known therapies are not being offered. The extent of this shortfall in the provision of the most effective therapies is not known, but it is clearly substantial. Why should this be? What are the barriers to the delivery of the most effective known forms of treatment?

First, although our classification system is as good as one might hope for, given the limited knowledge on which it is based, it is still not strong enough to allow diagnosis to be more than a very rough guide to intervention. The motivation of the child and other family members to receive help, the presence of mental illness or personality problems in one or both parents, the social circumstances in which the family is living, the resources available and many other considerations may all be more powerful determinants of treatment than diagnosis, and together they are almost bound to be so.

Second, psychiatrists, psychologists, social workers and psychotherapists have inadequate access to the best available evidence. The textbooks do not provide it. Published reviews are often out of date, even at the time they appear. The FOCUS initiative is beginning to fill the knowledge gap with evidence-based briefings (Royal College of Psychiatrists' Research Unit, 1999), but the gap will continue to exist for many years. In addition, one may expect more up-to-date and thoroughly researched information to appear on Internet websites, such as that on evidence-based mental health.

Third, and perhaps most importantly, the resources to provide the forms of treatment best supported by evidence are lacking. There is a lack of professionals trained in evidence-based therapies because there is no one competent to supervise them. This is particularly the case for cognitive-behavioural therapy. Trainees in child and adolescent psychiatry still have too little supervised experience in the most effective therapies. The situation is changing, but many trainees still get little opportunity to develop skills in the use of medication and behavioural therapies.

Fourth, given that inevitably some consultants have been trained in forms of therapy less effective than others that have since emerged, there is a lack of opportunity to re-train under supervision. One- or two- day workshops are unlikely to be adequate, and longer attachments may be required during which consultants can observe therapies unfamiliar to them and, at the very least, undertake therapy using role-play until it is clear that they have mastered the necessary skills.

Fifth, there is the problem of the institutionalisation of particular types of therapy, especially psychoanalysis and family therapy, but one can also see similar trends developing in some forms of behavioural, cognitive-behavioural and problem-solving therapies. The whole apparatus of training institutions - their hierarchical structures, training committees, their certificates of attendance, diplomas, even degrees, their stringent admission and exclusion criteria and the financial investment that trainees have to make to become accredited - creates a system permeated by beliefs, attitudes and values that are extraordinarily impervious to new ideas and evidence that does not fit with the received dogma. There are at least a few forward-thinking, authoritative psychoanalysts who agree with this view (Reference Fonagy and FonagyFonagy, 1999).

WHEN EXTERNAL SCIENTIFIC EVIDENCE DOES NOT EXIST

There are many situations in which the clinician needs to make decisions where the external scientific literature does not assist. First, and most commonly, a child may be showing two or more comorbid disorders, and RCTs only rarely address comorbidity. In these circumstances, a sensible approach may involve identifying the symptoms that the family regard as most problematic and using the scientific evidence relating to the disorder that these symptoms reflect. Of course, if there are reasonable grounds for assuming that one disorder is primary and the others are secondary (e.g. depressive disorder resulting in conduct disorder symptoms), it will be sensible to tackle the primary disorder first.

A second example arises when a child is showing an unusual type of problem such as a gender identity disorder for which RCTs do not exist. In these circumstances, as Muir Gray (Reference Muir Gray1997) suggests, it will be reasonable to use information derived from specialist clinics. Third, child mental health professionals spend a great deal of their time “managing the context” (S. Kingsbury, personal communication, 1999). A visit or a telephone call to a school to discuss ways in which exclusion may be avoided, putting a lone mother in touch with a self-help group and arranging after-school activities for a boy entangled in a delinquent subgroup would all fall into this category.

QUALITATIVE RESEARCH METHODS

What types of disciplined inquiry between the RCT and what Cronbach & Suppes (Reference Cronbach and Suppes1969) have described as “ casually assembled fragments of evidence” might provide us with useful, relevant knowledge? Barnes et al (Reference Barnes, Stein and Rosenberg1999), have suggested that the sharp division between audit and research should be reduced. It might, indeed, be possible to use some of the findings of audit to examine not just the efficacy but also the effectiveness of interventions. This is an attractive idea, but one that there is no space here to explore. Instead, the use of qualitative methods will be considered.

What is qualitative research? Most of those who have defined qualitative research have done so by contrasting it with quantitative research. However, as we shall see, many of those engaged in qualitative research (e.g. Reference Hammers ley and BrannenHammersley, 1992) do not see the two types of research as categorically different. They see both types of methodology as part of a common endeavour to try to expand knowledge through disciplined inquiry.

First, although Strauss & Corbin (Reference Strauss and Corbin1998) define qualitative research as “any type of research that produces findings not arrived at by statistical procedures or other means of quantification”, this definition is not widely accepted by those engaged in qualitative work, who virtually all give the number of subjects they have investigated, and often in their conclusions describe certain explanations as ‘more’ or ‘less’ likely, or certain occurrences as ‘frequent’ or ‘infrequent’.

A second distinction is that those engaged in qualitative research proceed by inductive methods, that is by moving from observable data to theory, whereas quantitative researchers proceed by deduction, testing theory by experiment and observation. As Medawar (Reference Medawar1984) pointed out, those engaged in scientific, biological research are constantly moving from observation and experiment to theory, and back again.

Those engaged in qualitative research are said to focus on events in natural settings, whereas quantitative research is undertaken in experimental settings. This begs several questions. Is a child psychiatric clinic a ‘natural’ setting or an ‘experimental’ one? If a social scientist goes into schools and observes quantifiable behaviour, as Rutter et al (Reference Rutter, Maughan and Mortimore1979) have done, are they carrying out work that is ‘experimental’ or ‘natural’?

Qualitative research is said to be strong on validity, and quantitative research strong on reliability. This distinction seems unnecessarily derogatory to both approaches. Surely all disciplined inquiry must involve observations that are repeatable (reliable), and use valid methods that produce data relevant to the purpose for which they were designed. The findings of qualitative research need to be reliable if they are to be generalisable, and findings from quantitative work need to be valid if they are to be of any use at all.

Qualitative research is said to be committed to an idealist epistemology and quantitative researchers committed to a view that the world can be described realistically. Again, this distinction is artificial. Picasso is quoted as saying “Anything that can be imagined is real”, and quantitative researchers who think they have discovered real reality or truth are usually merely blind to historical, cross-cultural and other evidence of the social construction of truth.

Having criticised the idea that these two types of research are categorically different, it must be allowed that those engaged purely in qualitative research do, in general, have different areas of interest, use different methods and achieve different types of outcome or results when compared with those engaged in purely quantitative work. But, more and more, those engaged in relevant research use both approaches. Most problems need to be addressed by both types of method, and increasingly a mix of methods is employed (Reference Brannen and BrannenBrannen, 1992).

QUALITATIVE RESEARCH: QUALITY CONTROL

Mays & Pope (Reference Mays, Pope, Mays and Pope1996) emphasise the need for those engaged in qualitative research to make explicit their theoretical framework, and describe the sampling strategy, fieldwork and context in which their work is carried out as clearly as possible. They should make their data and methods of data analysis available for independent scrutiny. They should test for reliability of observations and use quantitative methods to test qualitative conclusions where this is appropriate.

The research method needs to be described clearly and fully so that anyone wishing to replicate the study must be able to do so on the basis of published material. If a research worker wishes to establish a finding, enough subjects must be interviewed or observed to make it unlikely that investigation of more subjects would produce conflicting evidence. The data, often audio- or video-taped, should be available for checking by an independent observer. Finally, any arguments that go against the conclusions should be considered seriously and discussed; in particular, deviant cases that go against general conclusions need to be taken into account.

THE USE OF QUALITATIVE METHODS

How might qualitative research clarify issues in treatment delivered by child and adolescent psychiatrists? First, let us take drop-outs and non-attendees. Children who are referred show non-attendance and early drop-out rates varying from 5 to 30%. A number of quantitative studies (e.g. Reference KazdinKazdin, 1996) have been carried out to investigate this phenomenon. Combining these findings with those obtained using qualitative methods might illuminate the problem further. Unstructured interviews that aimed to understand the hopes and expectations (or lack of hope and expectation) of this group of children and their families, as well as the meaning to them of attendance at a clinic, might produce information that changed the content of treatment offered or the setting in which it was offered. Of course, such qualitative research would need to be followed by quantitative work to see whether, using the theories derived from the qualitative studies, it was possible to improve rates of attendance at clinics or deal with such problems in an entirely different way. As I see it, the development of theory is, in itself, a sterile exercise without some attempt at verification, and this implies testable theory. But quantitative work might well be more productive if it was more frequently preceded and inspired by good qualitative studies.

There are several examples where a mix of qualitative and quantitative methods has produced useful information. Skuse et al (Reference Skuse, Bentovim and Hodges1998), in a quantitative study, identified family violence as an important predictor of later perpetrating behaviour in sexually abused boys. They were greatly helped to formulate their hypotheses by systematically conducted qualitative assessments carried out by psychoanalytical psychotherapists. Jones & Ramchandi (Reference Jones and Ramchandi1999) have described how a series of studies, funded by the Department of Health, using a mixture of qualitative and quantitative methods has provided useful information to guide interventions where there is a suspicion of child sexual abuse. Finally, qualitative work could help to clarify another area in which there is a substantial amount of cognitive dissonance. What are reasonable and what are unreasonable expectations of the effectiveness of a child psychiatric service? Exploration of this issue using qualitative methods with managers, mental health professionals and those working in other agencies, such as social work and education, could be useful in reducing such dissonance.

Such work might enable us to reduce expectations to a more realistic level, especially in severely disturbed children. Interventions should be valued if they enable those working in different agencies to achieve common understanding of problems and similar expectations of their development and agreement on a realistic plan for the future (Reference Wiener, Withers and PatrickWiener et al, 1999). Good inter-agency collaboration by no means necessarily brings with it better outcomes for children and families, as the Fort Bragg study showed (Reference BickmanBickman, 1996), but such collaboration does improve satisfaction both in families and in professional staff, and this in itself is a worthwhile achievement.

Behaviour and emotional problems are inevitable. They cannot be cured: they may be helped to improve, although perhaps only temporarily: most of our treatments achieve only marginal benefit and some do not work at all. To paraphrase part of a recent editorial in the British Medical Journal (Reference SmithSmith, 1999), the best child and adolescent mental health service “…will not be one that provides everything for everybody, but rather one that determines how much it wants to spend on such services, and then provides explicitly limited evidence-based services in a humane and open way without asking the impossible of its staff.”

Clinical Implications and Limitations