Methodological issues and the nature of evidence

In our view, one of the limitations of much of the literature on evaluation research is that it gives insufficient attention to the problems involved in undertaking evaluations. The problems include a less than sceptical approach to data and to data collection, and a tendency to approach evaluation work from a single perspective, giving inadequate attention to the plurality of interests which are commonly present in public services.

When evaluating a policy or service, data obviously need to be collected and analysed. It is therefore important to be aware of the ontological and epistemological foundations on which the data are being built. Ontology is a branch of metaphysics and is the study of ‘being’. It concerns itself with the question of whether things exist and (if they do exist) the nature of those things. Epistemology is a related philosophical field: it is the study of knowledge, and helps us to distinguish between an unsupported assertion (and its more subtle relation the undersupported assertion) or mere opinion on the one hand, and a justified belief on the other hand.

These issues are important because they help us to be sceptical about what might appear at first sight to be valid evaluations, especially when evaluating processes about which there might be significant disagreements. Consider, for example, the evaluation of various ways of assessing and resolving interpersonal conflict between members of a management team trying to work together in the planning and delivery of public services. Different observers of the managers’ behaviour patterns might easily make totally different judgements about the sorts of conflict that exist, or about whether there was any conflict at all. One observer might see the raising of voices in a meeting as clear evidence of conflict and hostility. Another might see it as no more than a normal exchange of views and perspectives being expressed assertively by those present. These different perceptions by the observers point to the inevitable subjectivity of such judgements, even where the observers are taking care to be as objective and systematic as possible.