Guidelines for Program Evaluation Proposal
As outlined in your syllabus, you are required to devise a 6-8 page evaluation proposal. Do NOT include references, Cover Sheet, or appendix in your page total.
First Step: Choose a Health Education / Promotion Program to evaluate. You will not actually evaluate the program, but you will propose an evaluation plan that could be implemented to evaluate the program. Example programs could be:
- A sexual education training given at your sorority / fraternity or dorm aimed at changing knowledge of STDs
- A group fitness program at a gym or YMCA – are participants satisfied with the program? (process evaluation)
- A four week healthy eating program focused on increasing fruit and vegetable intake delivered to 4thh grade students at a local school.
- A sexual violence awareness program at your dorm.
*Think of a program that you know of – maybe something that you attended that focused on knowledge, attitudes and/or health behavior change.
For the purpose of this guide, I will go through each section using the example of a process evaluation measuring consumer satisfaction of the group fitness program at the University of Memphis Rec Center.
(See page 165 for consumer satisfaction, process evaluation)
Considerations for Content and Design of the Assignment
- Executive Summary 1. Identify your topic and purpose of the evaluation, health program decisions to be made based on evaluation results, purpose of the health program being evaluated, and provide a brief synopsis of what the report, would entail IF you had implemented the proposal. How will the lessons learned from the inquiry be used to improve public health effectiveness?
Do this part last. The Executive Summary should be a summary of your entire proposal. They are usually written last.
B. Introduction
- Introduce the program that you will evaluate and a rationale for why you will be evaluating this program.
Example: The purpose of this evaluation proposal is to evaluate the consumer satisfaction of the group fitness program at the University of Memphis Rec Center. The University of Memphis Rec center is a student gym that offers the following group fitness classes … they are open at the following times…, it is free of charge for students… etc. Describe the program in detail.
Rational: The Rec center is considering offering more classes, changing the variety of classes, and changing times that classes are offered. They would like an evaluation of current consumer satisfaction to help inform these decisions. The purpose of this evaluation proposal is to evaluate the consumer satisfaction of the group fitness program at the University of Memphis Rec Center. What does the evaluator what to know? What is the rational for conducting this evaluation. Who will be using the evaluation results. (See chapter 2, page 43).
Include what level of evaluation this plan proposes (process, impact, outcome). See pages 33-35. My example is a process evaluation. If you are measuring changed in knowledge, attitudes or behaviors, it is an impact evaluation. Measuring changed in actual health status is an outcome evaluation.
- Through incorporation of literature review, situate your research in the context of relevant research in the field. Demonstrate an awareness of current conversations and attitudes in the literature regarding the specific aspects of the field that your research encompasses. So if you are evaluating a program for infant mortality you would provide relevant statistics on the problem of infant mortality global to specific to your program, you would identify risks of this health problem, identify similar programs and their effectiveness and limitations.
Conduct research on the health problem that your program is trying to address and include statistics and risk factors in this section. Also research other programs that are similar and report on their findings. Were they successful?
Example: Since I am evaluating consumer satisfaction of the Group Fitness Program at the University of Memphis Rec Center, I am going to provide background research about physical inactivity and related health consequences. I will include statistics about the prevalence of people that are physically inactive, specifically college age students. I will use CDC.gov and HealthyPeople.gov to find this data. I will also talk about other group fitness programs that have been effective at increasing college student physical activity, or related programs if I can’t find something that specific. I will do a library search to find related articles or studies. I will site all of these sources in my references.
- Identify and explain the main objectives of the evaluation. What will be evaluated? (That is, what is the program and in what context does it exist?) What aspects of the program will be considered when judging program performance? (try to include operational definitions for variables and indicate the overall objectives of the evaluation)
The purpose of this evaluation proposal is to determine the degree of consumer satisfaction of students that take group fitness classes at the UofM Rec Center. This will help determine if changes to the existing program are recommended. To meet this goal, 3 objectives were identified based on selected objects of interest related to “consumer satisfaction.” These include: 1. Satisfaction with times of classes, 2. Satisfaction with variety of classes … etc. Define what we mean by each variable… time classes are offered, variety of classes offered, quality of classes offered, etc.
Objective 1: The evaluator will survey students in the selected sample to assess satisfaction of times of group fitness classes offered at the Rec Center by May 30, 2019.
Objective 2: The evaluator will survey students in the selected sample to assess satisfaction of the variety of group fitness classes offered at the rec center by May 30, 2019.
Etc…
Refer to Chapters 3 and 4 – operational definitions for variables.
- Identify the parties who will be affected by your research, and explain how various audiences have been or will be included in your project.
Current students that use the UofM Rec center group fitness classes will be impacted as the research my lead to a change in the program to better meets students’ needs. This may result in increased participation in physical activity…etc.
Although not the target population, students who are non-users may also be impacted as changes could result in more people using the Rec center… Etc.
C. Methodology
- Identify your evaluation design and key stakeholders. Research Design (Qualitative, Quantitative, or Mixed) Describe to the best of your ability.
Key stakeholders: The key stakeholders for this evaluation include the University of Memphis Rec Center staff, University of Memphis administration, and University of Memphis students. They are key stakeholders because …. Explain / describe.
Evaluation Design: I propose a mixed method evaluation design (both quantitative and qualitative) for this process evaluation (consumer satisfaction).
The quantitative aspect of this evaluation will include surveying students to quantify their satisfaction with the Group Fitness Classes at the Rec Center related to the three objects of interest that have been selected to comprise overall satisfaction: times of classes, variety of classes, quality of classes. The design will be non-experimental as there will be no control group. (see appendix for survey instrument)
(see chapter 9)
The qualitative aspect of this evaluation will include conducting three focus groups of 8 students each. These focus groups will ask open ended questions related to the objects of interest to identify additional areas of interest. (see appendix for focus group guidelines)
- Participants (describe the particular population involved, the sample size if applicable, how the sample will be drawn from the population)
Refer to chapter 11.
For my example, I could use a volunteer, non-probability sample which would be less scientifically rigorous and limit my use of the data as it would not be generalizable to all users of the group fitness classes at the Rec Center.
Or, I could get a list of all student users of the group fitness classes at the rec center and randomly select my sample, so that it would be generalizable to the entire target population (all student users of the group fitness classes at the rec center).
Either way, describe which method you propose in detail.
- Identify all instruments used for collecting information AND attach them in the Appendix. So if you are using focus groups, attach your focus group guidelines and questions, etc. Instrumentation (type of data, discuss type of measure or instrument (i.e. interviews, observations, surveys, etc.)
You must create an instrument for this piece and attach it to your paper in the appendix. This can be a survey or focus group protocol, etc. depending on what type of evaluation you are performing (qualitative, quantitative or both). See chapter 4 for survey development (quantitative). I am attaching a focus group question guide to help if you want to create a focus group guideline. This would include a list of open ended questions to ask your group. You can also use observation (pages 101-102) or other qualitative approaches (see pages 220-221). See chapter 10 for more on qualitative evaluation.
- Clearly present the methods through which information will be collected. If necessary, provide separate methodologies for various aspects of data collection. Procedure (discuss the way in which the evaluation will be conducted and include the steps you will follow in administering or using any instruments, etc.) List and briefly explain the sampling procedures employed for each method.
Example: For the survey implementation, after selecting my sample I will use survey monkey to implement the survey via student email addresses.
For the focus groups, I will recruit 30 students from the Rec Center group fitness classes to participate in one of three focus groups. I will recruit 50% male and 50% female (etc. – be sure to include representation from all of the demographics you are interested in evaluating).
Provide more detail here… what exact steps will you take.
- Explain how information was/will be stored and how any data collected will be analyzed.
Example: Once collected the survey data will be entered into a database and the electronic surveys will be stored for accountability. The focus group recordings will be stored on a password protected hard drive and used for data analysis. All survey materials will remain confidential.
Refer to chapter 12 for data analysis ideas. I will find the mean, mode, and median for each survey question… etc.
Refer to chapter 10 for qualitative evaluation data analysis if you are using a qualitative design (i.e. focus group) (page 221 and 222).
- Defend the reliability and validity of the methodological techniques you are proposing.
Refer back to validity and reliability of study designs (page 184-190) and reliability and validly of instruments (chapter 5 and chapter 6), and address both the validity and reliability of the instrument(s) you created and the study design you propose. What steps would you take to ensure validity and reliability.
- List any potential limitations of the study.
Any threats to validity? Will you be able to generalize the data? What could make the design stronger?
E. References
- A complete list of references should be included. All works cited within the text should appear in the reference page.
- All references, both in-text and bibliographic, should adhere to the American Psychological Association (APA) style.
Tip: The more concepts that you address from the text, the stronger your paper will be. Show me in your proposal that you understand the concepts covered throughout the course. Every section provides you with an opportunity to highlight your understanding of the course content. You are welcome to define key concepts in your paper and then tell me how they will apply to your proposal.
I am happy to answer questions or meet with students to go over in detail!