The team received a striking number of questions leading up to and following the webinar, demonstrating a keen interest amongst members to further their learning in evaluation. In follow-up to this webinar, Melanie Hientz from LivingSJ and Andrew Taylor from the Ontario Nonprofit Network kindly agreed to respond to some of the questions posed by webinar attendees.
Melanie Hientz, LivingSJ
Q: What's the number one tip you would give an organization seeking to evaluate its work?
A: Evaluation is how we track the degree to which we’re meeting our outcomes, and ensure fiscal accountability to our funders. To evaluate effectively, we must work closely with boards a
nd staff to first define the impact they would like to achieve and the outcomes they aim to accomplish. Establish a clear and mutually agreed upon definition of what success looks like, a carefully articulated path of how this success will be achieved, as well as the measures that will be used to track progress. Working with boards and staff through a theory of change process can enhance the way people think about evaluation, instilling a better understanding of – and confidence in – evaluationprocesses, and the value that effective evaluation brings an organization.
Q. Can you speak to how you develop an evaluation and disseminate the results in a way that measures how specific programs (ex. Saint John’s Promise Partnership) achieve their goals and contribute to poverty reduction?
A: When evaluating community change or poverty reduction efforts, population- and program-level data help us to identify which interventions are working, and the degree to which we’re achieving our outcomes. Living SJ captures population-level data from a number of our partners (e.g. the school district’s grade 2 literacy levels), while individual programs also contribute data (e.g. early learning programs). Attribution of outcomes can become challenging in areas where there are multiple programs working on a similar priority (e.g. closing the education achievement gap), so in these cases, using both program evaluation and population-level data can help paint that broader picture and determine which interventions are most successful.
Andrew Taylor, Ontario Nonprofit Network (@AndrewTaylorTNC)
Q: Can you speak to how (or whether it’s even possible) to get buy-in from front-line staff, board members, funders, businesses, and volunteers?A: I totally think it is possible. The key is to understand, respect and address their misgivings about evaluation. Front line staff often worry that they will have to do all the evaluation work, or that they will be personally judged. Board members worry that evaluation will eat up time that would be better devoted to the organization’s mission, and that it won’t generate anything useful. These worries are not unfounded. Make sure you have an evaluation plan that will lead to action, and will generate excellent value for the time and energy you are asking your staff to invest.
Q: How do you create the safe space to bring funders to the table and have those discussions if you are worried about the budget being cut?
A: Try to make the conversation less about your program, and more about the funder’s reasons for asking for evaluation. Find out what they hope to do with evaluation, and then show them how your approach will help them get there. For example, if your worries about budget cuts stem from the fact that your numbers of clients are low, help them see how your evaluation will highlight the unique challenges involved in serving your client population and generate insights they can apply to their work with other grantees.
Q: How do we mitigate evaluation fatigue? We ask for evaluations after each term - how can we mix it up and make it more interesting for respondents?
A: Spend less time doing more evaluation, and more time making sure you’ve learned as much as you can from the data they have already provided. If they see that their input has led to action, they will be less fatigued. If you turn their input into action, that will lead to you to identify new evaluation questions that will be fresh and interesting for them.
Further your Learning
- Full recording: 'Evaluations That Work: What the Non-Profit Sector Can Learn from the Ontario Nonprofit Network and Vibrant Communities'.
- Five minute teaser: 'Evaluations That Work: What the Non-Profit Sector Can Learn from the Ontario Nonprofit Network and Vibrant Communities'.
- Presentation slides: Evaluations That Work
- ONN's blog: Making Evaluation Work for Nonprofits: Our Theory of Change
- Webinar highlights: Twitter Storify
