If you have decided to do some impact evaluation, or to try out the process, you may want to go straight on to Chapter 4. If you are interested in some of the difficulties inherent in evaluating impact and in gathering and interpreting evidence, read on.

A metaphor and a model

The language and underpinning concepts of performance measurement are often arcane and sometimes confusing. This is probably because the basic metaphor of performance indicators and targets and the associated vocabulary were developed by engineers when looking at system efficiency and appropriated by accountants to simplify and ‘measure’ the complex world of managing organizations.

It is hardly surprising that the concepts of performance measurement work reasonably well when the library is considered as a more or less efficient operating system. Unfortunately for the accountants (often operating in their guise of management consultants), libraries do not exist simply to perform efficiently. The performance metaphor begins to break down when we look at what the service is trying to achieve and how well it is doing (or moving beyond easily measured ‘outputs’). In suggesting a way forward, we would like to stress that performance measurement based on system efficiency is important for libraries and should run alongside any work on evaluating impact. Efficiency is important – but so is effectiveness.

Figure 2.1 outlines the traditional performance evaluation model. In this model, performance indicators are arrived at by looking successively at inputs, processes and outputs. Pursuing the engineering example, if you are in the business of manufacturing televisions, your inputs can be all the components that go together to make televisions. The processes will then be everything that you do in your production line to assemble the televisions. Your outputs are the completed and working televisions. In this model, if you want to look at the outcomes from the production process you can resort to aggregated outputs, by counting the total number of television sets produced (and probably also the proportion of these that work when switched on!).

The engineering process model works reasonably well in complex organizations like libraries – but only up to a point. No amount of aggregation in library settings will usually tell you much about impact. Our next example shows where the problem lies.