One of the characteristics of a high-quality theory of change is that it uses robust data to measure change over time and to test whether the theories within it are achieving the desired outcomes and impact.

So how do you decide which indicators and which data sources should be included to demonstrate the efficacy of your theory of change?

In our experience, a structured data mapping approach can help. These are some of the questions that we work through with clients as part of the development of their theory of change:

1. Relevance: how does the data that is being collected map against each dimension of the theory of change? Which indicators provide evidence about inputs, outputs and activities, and which indicators relate to outcomes and impact?
2. Quality: what is the quality of the data that is being collected? For instance, how is it collected, from how many people, by whom, does it use a standardised and/or validated tool or definition or approach, is it free from bias?
3. Frequency: how often is the data collected? Is data being captured to assess sustained impact over time?
4. Intersectionality: is data collected in a way that it is possible to scrutinise whether people’s experience and the outcomes achieved varies by key characteristics, such as gender, ethnicity or disability?
5. Accessibility and usability: how easy is it for practitioners, decision-makers and researchers to access and use the data?
6. Analysis: what type of analysis do you hope to undertake using the data? Is it fit-for-purpose? A key question here is whether it is sufficiently large to be able to draw reliable conclusions.
7. Ethics: is the data collected ethically and in compliance with effective practice around information sharing, information governance and data protection? Are appropriate quality assurance processes in place?

A data mapping exercise using these questions will help to identify how robustly a theory of change can be monitored over time. It will also help to identify any potential gaps in data. This means that a further three follow-up questions can be asked:

1. In an ideal world, what other evidence would you need to help you evaluate the theory of change?
2. To what extent can this data be collected, either by changing existing data collection systems or introducing new ones?
3. What is the likely cost of this and what additional training or staff time may be required to collect this on an ongoing basis?

We have successfully deployed this data mapping approach for theories of change for a wide variety of projects. It can apply to services, programmes or interventions. It can also be applied to system-wide or place-based initiatives that involve multiple partners and complex interactions.

To find out more contact Colin Horswell, Managing Director.