The model of evaluation
Key question
We use the ideas of ‘realist evaluation’ which is described by Pawson and Tilley in ‘Realistic Evaluation’ (Sage 1997). This approach aims to develop evaluations that answer this key question:
Which intervention works best for whom in which circumstances and settings?
At the outset and before the project begins, we ask organisations to state why they think what they are proposing will answer the need they have identified. This should be based on their experience of the people they work with and the kind of work they do. Their informed guess will be their first attempt at explanation or the start of their theory building. By collecting data, guided by the theory, they can test out whether their explanation has a bearing on reality. The evidence should tell them how right or wrong they are, as well as helping them answer the question above.
We encourage managers to think of evaluation as a process of understanding. The current theory is the first point of understanding – it tries to sum up what they know now. As evidence becomes available, we expect ideas and probably projects to change. Evaluation should help them track how and why their ideas change.
The three key components of the theory - context, mechanism, outcomes
1. Context
All projects take place in ‘real’ life and the situation of the project will have an impact on how it functions.
For it to be successful or to work in some way, it needs to have the right conditions. The point of the project is not to change these conditions – some of these can’t be changed – but an important part of understanding how it works is to have some idea of the influence that these conditions, circumstances or situations have on the interventions.
If we go back to the key question: 'which intervention works best for whom in which circumstances and settings?', the context part of your explanation tells others the key characteristics of your users, (‘the whom’ part of the question), and also the situation or circumstances that may affect the project (‘the circumstances and settings’ of the question)
2. Mechanism
This part of the explanation attempts to explain why a particular intervention works, (the ‘what works best’ part of the key question).
There are two parts to this concept.
Internal trigger
This describes the motivation of the user to change. All projects rely on this inner trigger to affect change in individuals. Imagine a smoking cessation programme where the key intervention is showing the smoker a picture of a foetus of a mother who smokes and comparing this to a picture of a foetus of a non-smoking mother. Do you think this is going to have much effect on a teenage boy? Probably not. It is much more likely to work on pregnant women, because it relies on the concern of a mother for the unborn child to trigger a motivation to change.
External resources
A description of the intervention and the resources needed to change the situation
3. Outcomes
Having identified the problems and outlined a solution, how do you know if you have had an impact?
By developing outcomes. These aim to predict the observable change in the situation, that can be reasonably linked back to the intervention.
This is easier said than done! One of the difficulties in setting outcomes is that we often set targets that are unreasonable to expect from the intervention.
