Evaluating Impact
Evaluation Systems Change

Evaluating Systems Change

Community builders eager to make progress on complex issues must move beyond projects and programs and seek instead to change the systems that underlie the challenges. As Karen Pittman, CEO of the Forum on Youth Investment noted in a Tamarack event in 2015:

While programmatic interventions help people beat the odds, systemic interventions change their odds.

The increased focus of social innovators on changing systems must be matched by a an increased focus by evaluators. This is anything but straightforward. The concepts and methods of traditional evaluation emerged when the emphasis was on assessing projects and their effects on people. They are often unsuitable for the more complex, larger scale and dynamic work of systems change.

Happily, the field of evaluation is evolving. There are plenty of excellent examples of evaluating systems changes and helpful sets of principles, frameworks and methods. There is, however, still plenty more to do: most would-be changemakers still struggle to get a handle on the results and learnings of their work.

Tamarack is devoting the second half of 2018 to explore two important developments in the emerging field of systems evaluation. These include: (a) how to plan an evaluation of systems change and (b) some emerging methodologies for evaluating systems change.

Papers

Evaluating Systems Change: An Inquiry Framework

Social innovators, evaluators, and community changemakers are increasingly focused on changing complex systems, but often struggle to describe either the systems itself or what they hope to achieve. This paper is designed to give clarity on how to approach the evaluation of systems change and provides three types of results that social innovators and evaluators should consider “mission-critical” to their work.

Read the Paper

Webinars

The Systems Change Evaluation Canvas
Featuring Mark Cabaj & Galen MacLusky

This webinar features Mark Cabaj and Galen MacLusky introducing the Systems Change Evaluation Canvas, walking through important things to consider when planning systems change, and showing how the canvas can be applied to a real-world scenario. 

Access the Recording

You can also access the Systems Change Evaluation Canvas tool

Planning an Evaluation of Systems Change
Featuring Meg Hargreaves & Mark Cabaj

In this webinar, Meg Hargreaves and Mark Cabaj explore the ideas and steps required to properly evaluate ‘system changes’ that emerge when tackling complex issues. Using her experience in systems change evaluation, and her publication "Evaluating Systems Change: A Planning Guide", Meg provides a foundation for understanding and approaching this important topic. 

Access the Recording

For more on Planning Systems Change Evaluation, see Evaluating Systems Change: A Planning Guide

Evaluating Systems Change: A Results-Based-Framework
Featuring Mark Cabaj

Almost anyone who is interested in making progress on tough economic, social and environmental issues is committed to changing the system. Yet, many social innovators — and the evaluators that support them — struggle to describe what systems they want to change and what they mean by ‘results’. This webinar explores an archetypical results framework that can be used to inform the thinking of any effort to ‘change’ systems.

Access the Recording

Tools

The Systems Change Evaluation Canvas

How can you evaluate a systems change initiative? The diversity of methods and approaches available can be overwhelming, but ultimately what is most important is what is appropriate for your context and your community. 

Using the Systems Change Evaluation Canvas will help you prepare an evaluation scope of work, understand key considerations when designing an evaluation for a systems change effort, and make decisions about evaluation design that reflect the needs of your context and community

Access the Tool

Articles

Outcome Harvesting: Principles, Steps, and Evaluation Applications

This article features Mark Cabaj reviewing expert Ricardo Wilson-Grau’s book, Outcome Harvesting: Principles, Steps, and Evaluation Applications, and describes one of the newest and most impactful methodologies to emerge in the field of social change and innovation - Outcome Harvesting.

The review helps differentiate Outcome Harvesting from other more traditional approaches to evaluation, and sets the stage for changemakers to approach and make use of this exciting new development which asks three questions of every outcome of a change initiative:

• What was the outcome? (action, behaviour, change)
• Why is the outcome significant?
• What was the initiative's contribution to the outcome?

Access the Article

Resource Review

Outcome Harvesting

This article features Mark Cabaj reviewing expert Ricardo Wilson-Grau’s book Outcome Harvesting: Principles, Steps, and Evaluation Applications, and describes one of the newest and most impactful methodologies to emerge in the field of social change and innovation - Outcome Harvesting.

The review helps differentiate Outcome Harvesting from other more traditional approaches to evaluation, and sets the stage for changemakers to approach and make use of this exciting new development.

Access the Review

The Tyranny of Metrics

This article features Mark Cabaj reviewing expert Jerry Z Muller's book The Tyranny of Metrics which concerns the practice of reducing things like individual and organizational performance to standardized metrics. 

While Muller is not staunchly anti-metric, and sees the benefit of pulling quantitative data in support of performance, he takes on three main beliefs

• Outcomes and performance can be reduced to standardized measurements even in complex challenges
• Making metrics public or transparent increases the chances that institutions will be accountable for their actions and results.
• The best way to motivate people is by attaching financial and reputational reward and penalties to their ability to meet quantifiable performance targets.

Access the Review

Additional Resources

Here are a number of resources that provide a summary of principles and methodologies that social innovators and evaluators may find useful in their efforts to change systems:

Systems Change: A Guide to How to Do It - A very accessible, introductory level guide to explore the basic ideas of systems change prepared by the Lankally Chase Foundation.

Evaluating Systems Change: A Planning Guide - A systematic approach to planning an evaluation of systems change initiatives.

Evaluating Complexity: Propositions for Improving Practice - A list of ten principles and associated methods developed by the Foundation Strategy Group (FSG) for evaluating complex change initiatives, including systems change.

Systems Concepts in Action: A Practitioners Toolkit - A first rate resource that explores the application of systems methods to investigate, evaluate, and intervene in complex and messy situations.