The video System Thinking and Evaluation, by Kylie Hutchinson, Chris Lovato and Bev Parsons is an excellent introduction to evaluating systems change. It describes how an evaluation of a hypothetical initiative to improve nutrition in a community must both ‘zoom in’ to explore the programmatic effects of the effort (e.g., improved health of program participants) and ‘zoom out’ to assess influence and change on factors in the larger systems that affect their individual health (e.g., urban design which affects levels of physical activity, the quality of industrial food production, the culture of portion sizes). The video also reminds us that deep and durable progress on complex issues depends on our ability to reshape the deeper systems that contribute to those problems in the first place.
This creates a challenge for the field of evaluation. In a recent series of Tamarack sponsored workshops to introduce the concept and practices of Principles-Focused Evaluation to participants in four Canadian cities, Michael Quinn Patton reminded everyone that evaluation emerged with a strong focus on ‘zooming in’ on programmatic interventions. The field has not kept pace with the ever increasing focus by social innovators on 'zooming out' to change systems.
Happily, the field of systems evaluation is adapting. In the last decade, evaluators have created or adapted methodologies, principles, and even outcome frameworks for specific types of systems change efforts (e.g., workforce development, childcare, health care). The American Evaluation Association has a strong community of practice on the topic and the website Better Evaluation has a diverse set of resources on the topic.
One area that is not well developed is the practice of planning systems change evaluations. This includes describing the intervention – or parts of the intervention – to be assessed, nor landing on the users of the assessment, their purpose and questions, and preferences for things like methods, communicating styles, deadlines, and budgets.
Evaluating Systems Change: A Planning Guide, fills the gap. The Guide was prepared by Meg Hargreaves, one of the most experienced and well known evaluators of systems change initiatives in North America. It reflects her first rate understanding of system theory and evaluation, as well as her skill at making the work of evaluation accessible and practical.
The resource begins with a review of definitions, characteristics and thinking of systems theory, and then provides a series of questions that need to be answered to put together an evaluation of an effort to change a system. These include:
- System – What are the boundaries, relationships, perspectives of the system we are trying to change?
- System Intervention – What are the governance arrangements, intended outcomes, and theory of change for this initiative?
- System Evaluation – Who are the evaluation stakeholders of this initiative, their purposes, and preferred methods?
The answers to these questions provide the material required to develop an evaluation scope of work, which can then guide evaluators in their efforts to develop more specific evaluation designs, metrics, methods, budgets and work plans.
The Guide does not propose specific methods for evaluating systems change. These must be developed on a case by case basis. I have participated in evaluations that have assessed efforts to change energy systems, housing systems, and justice systems, and each has required methods customized to their unique issues and contexts. At the same time, Hargreaves offers a table that includes qualitative and quantitative methods that match the varying types of systems innovators are trying to change, ranging from the simple to the complex.
This Guide is not the only resource upon which social innovators and evaluators should rely. They will want to complement its general approach to scoping out an assessment with more specific tools and resources that reflect their specific situations. They may even want to go deeper on some of the questions posed in the Guide: there are, for example, a steady flow of new ways to describe the systems that people may want to change, that continue to emerge.
These qualifications aside, Evaluating Systems Change: A Planning Guide, is still the best all-around resource on the topic for practitioners and must-have an evaluation library.
Learn More:
- Read Evaluating Systems Change: A Planning Guide
- Watch a recording of Mark's webinar Planning an Evaluation of Systems Change featuring Meg Hargreaves
- Dive deep into evaluating systems change at new workshop in Calgary from November 13-14 - Evaluation + Design: Evaluating Systems Change
- Watch the video System Thinking and Evaluation by Kylie Hutchinson
- Read Mark's blog Evaluating Systems and Social Change: New Developments and A Ground-breaking Program by SFU
- Visit our page on Evaluating Systems Change for curated events, resources, and links to take your learning further
