Planning Systems Change Evaluation

Posted on September 21, 2018
By Mark Cabaj

Blank sheet with writing utensils around it-521135-editedThe video System Thinking and Evaluation, by Kylie Hutchinson, Chris Lovato and Bev Parsons is an excellent introduction to evaluating systems change. It describes how an evaluation of a hypothetical initiative to improve nutrition in a community must both ‘zoom in’ to explore the programmatic effects of the effort (e.g., improved health of program participants) and ‘zoom out’ to assess influence and change on factors in the larger systems that affect their individual health (e.g., urban design which affects levels of physical activity, the quality of industrial food production, the culture of portion sizes). The video also reminds us that deep and durable progress on complex issues depends on our ability to reshape the deeper systems that contribute to those problems in the first place.

This creates a challenge for the field of evaluation. In a recent series of Tamarack sponsored workshops to introduce the concept and practices of Principles-Focused Evaluation to participants in four Canadian cities, Michael Quinn Patton reminded everyone that evaluation emerged with a strong focus on ‘zooming in’ on programmatic interventions. The field has not kept pace with the ever increasing focus by social innovators on 'zooming out' to change systems.

Happily, the field of systems evaluation is adapting. In the last decade, evaluators have created or adapted methodologies, principles, and even outcome frameworks for specific types of systems change efforts (e.g., workforce development, childcare, health care).  The American Evaluation Association has a strong community of practice on the topic and the website Better Evaluation has a diverse set of resources on the topic.

One area that is not well developed is the practice of planning systems change evaluations. This includes describing the intervention – or parts of the intervention – to be assessed, nor landing on the users of the assessment, their purpose and questions, and preferences for things like methods, communicating styles, deadlines, and budgets.

Evaluating Systems Change:  A Planning Guide, fills the gap. The Guide was prepared by Meg Hargreaves, one of the most experienced and well known evaluators of systems change initiatives in North America. It reflects her first rate understanding of system theory and evaluation, as well as her skill at making the work of evaluation accessible and practical.

The resource begins with a review of definitions, characteristics and thinking of systems theory, and then provides a series of questions that need to be answered to put together an evaluation of an effort to change a system. These include:

  • System – What are the boundaries, relationships, perspectives of the system we are trying to change?
  • System Intervention – What are the governance arrangements, intended outcomes, and theory of change for this initiative?
  • System Evaluation – Who are the evaluation stakeholders of this initiative, their purposes, and preferred methods?

The answers to these questions provide the material required to develop an evaluation scope of work, which can then guide evaluators in their efforts to develop more specific evaluation designs, metrics, methods, budgets and work plans.

The Guide does not propose specific methods for evaluating systems change. These must be developed on a case by case basis. I have participated in evaluations that have assessed efforts to change energy systems, housing systems, and justice systems, and each has required methods customized to their unique issues and contexts. At the same time, Hargreaves offers a table that includes qualitative and quantitative methods that match the varying types of systems innovators are trying to change, ranging from the simple to the complex. 

This Guide is not the only resource upon which social innovators and evaluators should rely. They will want to complement its general approach to scoping out an assessment with more specific tools and resources that reflect their specific situations.  They may even want to go deeper on some of the questions posed in the Guide: there are, for example, a steady flow of new ways to describe the systems that people may want to change, that continue to emerge.

These qualifications aside, Evaluating Systems Change: A Planning Guide, is still the best all-around resource on the topic for practitioners and must-have an evaluation library.

Learn More:

Evaluating Community Impact, Evaluation, Mark Cabaj, Systems Change, Blog

Mark Cabaj

By Mark Cabaj

Mark is President of the consulting company From Here to There and an Associate of Tamarack. Mark has first-hand knowledge of using evaluation as a policy maker, philanthropist, and activist, and has played a big role in promoting the merging practice of developmental evaluation in Canada. Mark is currently focused on how diverse organizations and communities work together to tackle complex issues, on social innovation as a "sub-scene" of community change work, and on strategic learning and evaluation.

Related Posts

Unleashing Youth Expression Through Arts-Based Evaluation

Learning Indigenous Ways of Knowing and Being

Vibrant Communities Calgary’s Top 10 Highlights of 2021