Rethinking How We Evaluate and Use Data

Posted on April 15, 2019
By Duncan Field
tyranny of metricsEveryone, regardless of whether you consider yourself an evaluator or not, uses metrics. We track our finances, look at gas mileage and so much more. In a community change context, we often find ways to track the performance of things like marketing and communications, program delivery, and budgeting.

In a recent article, evaluation expert Mark Cabaj reviewed The Tyranny of Metrics by Jerry Z Muller. In his book, Jerry takes on three underlying assumptions about how we typically use data:

  • Outcomes and performance can be reduced to standardized measurements even in complex challenges

  • Making metrics public or transparent increases the chances that institutions will be accountable for their actions and results

  • The best way to motivate people is by attaching financial and reputational rewards and penalties to their ability to meet quantifiable performance targets

Mark's review of The Tyranny of Metrics highlights some of the practical issues that come up when an organization becomes over reliant on simple metrics to track and manage progress on complex issues. For example, an emphasis on a simple key performance indicator (KPI) may cause an organization to focus more effort on supporting the work that directly impacts that KPI at the expense of other work that may ultimately be just as important, but is not closely monitored. Mark, and ultimately Jerry Muller himself, is quick to explain that it is not the data or the desire to collect it that is the issue here, but rather the "inappropriate, excessive, and simplistic application of standardized metrics to decision-making" that can creep into our work.

Both Mark Cabaj and Jerry Muller are ultimately concerned with ensuring that organizations are intentional about what and how they are measuring progress and impact. Bad faith reporting, where an organization does not believe a metric has any relevance or impact but continues to report it, leads to lower morale, lost staff time, and bad decisions. This review is a great introduction to anyone who wants to dig deeper into this problem and equip themselves with the information it will take to do something about it.

If you're interested in learning more about the role of evaluation in the context of complex systems change, we're hosting a unique two-day workshop Evaluation + Design: Evaluating Systems Change in Ottawa, ON from May 22-23. We'll talk in depth about the challenges of measuring impact on complex systems, and equip you with the practical tools, principles, and resources needed to truly make an impact.

Learn More:

Evaluating Community Impact, Evaluation, Mark Cabaj

Duncan Field

By Duncan Field

Duncan is a researcher, writer, and editor with an interest in supporting community organizations. He works as a Community Animator to create and produce engaging content and learning materials, and help maintain Tamarack’s online spaces. Duncan comes to Tamarack by way of York and Queen’s University, with experience studying Psychology, Philosophy, and Law. Most recently, Duncan worked in the charitable sector designing and producing e-learning courses, as well as building online communities of practice.

Related Posts

Unleashing Youth Expression Through Arts-Based Evaluation

Learning Indigenous Ways of Knowing and Being

Vibrant Communities Calgary’s Top 10 Highlights of 2021