In a recent article, evaluation expert Mark Cabaj reviewed The Tyranny of Metrics by Jerry Z Muller. In his book, Jerry takes on three underlying assumptions about how we typically use data:
Outcomes and performance can be reduced to standardized measurements even in complex challenges
Making metrics public or transparent increases the chances that institutions will be accountable for their actions and results
The best way to motivate people is by attaching financial and reputational rewards and penalties to their ability to meet quantifiable performance targets
Mark's review of The Tyranny of Metrics highlights some of the practical issues that come up when an organization becomes over reliant on simple metrics to track and manage progress on complex issues. For example, an emphasis on a simple key performance indicator (KPI) may cause an organization to focus more effort on supporting the work that directly impacts that KPI at the expense of other work that may ultimately be just as important, but is not closely monitored. Mark, and ultimately Jerry Muller himself, is quick to explain that it is not the data or the desire to collect it that is the issue here, but rather the "inappropriate, excessive, and simplistic application of standardized metrics to decision-making" that can creep into our work.
Both Mark Cabaj and Jerry Muller are ultimately concerned with ensuring that organizations are intentional about what and how they are measuring progress and impact. Bad faith reporting, where an organization does not believe a metric has any relevance or impact but continues to report it, leads to lower morale, lost staff time, and bad decisions. This review is a great introduction to anyone who wants to dig deeper into this problem and equip themselves with the information it will take to do something about it.
If you're interested in learning more about the role of evaluation in the context of complex systems change, we're hosting a unique two-day workshop Evaluation + Design: Evaluating Systems Change in Ottawa, ON from May 22-23. We'll talk in depth about the challenges of measuring impact on complex systems, and equip you with the practical tools, principles, and resources needed to truly make an impact.
- Read the full review The Tyranny of Metrics
- Learn more about our upcoming two-day workshop Evaluation + Design: Evaluating Systems Change where we'll be exploring tools from both the evaluation and design field that will help you evaluate complex systems change efforts.
- Check out our new Systems Change Evaluation Canvas, a tool designed to help you build an evaluation of complex systems change efforts. You can also view a free webinar recording of Mark Cabaj and Galen MacLusky introducing the canvas and going through it piece by piece.