Young Caregivers Association strives to support young caregivers – individuals between 5 and 24 of age who take on caregiving responsibilities in their family – by providing opportunities to form connections with peers, relieve stress, and develop life and personal skills through a weekly drop-in program called ‘Caregivers Night Out’ (CNO).
During my placement with Young Caregivers, I was tasked with conducting a program evaluation of the CNO program, focused on its ability to develop young caregivers’ skills and ability to cope with stress, and manage their caregiving responsibilities.
What are Evaluation Questions and Why Do They Matter?
During the development of a program evaluation, specific evaluation questions must be chosen to provide structure to the evaluation and define the key issues that will be evaluated (Centers for Disease Control [CDC], 2018). Evaluation questions help stakeholders identify specific strengths, weaknesses, and areas requiring improvement in a program (Newcomer et al., 2015).
Selecting Types of Evaluation Questions.
There are two main types of evaluation questions, which are often chosen based on an organization’s goals and intent for a program evaluation:
Process Evaluation Questions – are used to address the core components of a program and the extent to which they are being implemented as planned (Morrison & Harms, 2018). For example, ‘How are weekly CNO sessions structured?’
Outcome Evaluation Questions– seek to determine a correlation between some aspect of the program and an intended program outcome or impact. For example, ‘How does participants’ ability to cope with stress change as a result of participating in the CNO?’ Desired program outcomes are typically short, medium, and long-term in scope; each with their own time expectancy of being met (Morrison & Harms, 2018).
Overall, establishing your evaluation questions involves deciding whether they will be process or outcome-focused, as well as ensuring that they are (CDC, 2018):
- Central to stakeholders and program staff.
- Meet program needs.
- Address program goals and objectives.
- Can be answered with available resources.
- Support decision-making in an efficient manner.
- Deliver useful information to increase program strengths.
My Experience Developing Evaluation Questions in Practice.
As a student, learning how to develop evaluation questions in class and applying them in practice with the Young Caregivers Association took a lot of time and critical thought, but was a great learning experience.
The first step I took towards developing evaluation questions required meeting with relevant stakeholders to review the goals and objectives of the program. This involved establishing desired improvements/strengths, as well as the process and outcome components of a ‘logic model’, which was completed on this template. I then brainstormed potential evaluation questions that were specific to the program’s desired outcomes.
Through discussions, I also learned that the stakeholders wanted to focus their evaluation at the level of the CNO ‘participant’ – young caregivers ranging from 5-24 years of age. This meant I had to take the evaluation questions that my team developed and simplify them on our chosen data collection tool (a survey) to ensure that the youngest program participant could answer them. In order to make the questions more age-appropriate, we simplified the wording of some of the questions and created Likert Scale and multiple-choice response options, rather than solely using open-ended ones.
Overall, this experience taught me the importance of knowing who your audience when selecting the data collection tools to answer your evaluation questions - ensuring that you create tools and questions that are answerable and that will ultimately lead to informative responses for your stakeholders to learn from.
This blog is part of a series in collaboration with Brock University. Written by a student in the ‘Program Evaluation in Professional Practice’ course led by Dr.’s Corliss Bean and Meghan Harlow, this blog details a student’s experience developing evaluation questions towards a program evaluation in placement with the Young Caregivers Association.
Take Your Learning further:
- Centers for Disease Control and Prevention (2018, August). Program Evaluation. Adolescent and school health. https://www.cdc.gov/healthyyouth/evaluation/index.htm
- Morrison, J., & Harms, A. (2018). Advancing evidence-based practice through program evaluation: A practical guide for school-based professionals. Oxford University Press.
- Newcomer, K., Hatry, H., & Wholey, J. (2015). Handbook of practical program evaluation. John Wiley & Sons, Incorporated.