Home / Blog / Public Health Program Evaluation Explained

Public Health Program Evaluation Explained

April 14, 2024

Uncategorized

A public health professional talks to a colleague in a corridor

Strained resources in healthcare facilities endanger the entire community. Public health programs work to prevent crises like these. They strive to keep populations healthy, improve quality of life, and reduce long-term healthcare costs.

Public health professionals evaluate public health program evaluations to identify ways that they can be improved and to decide how they will be developed in the future. A Master of Health Administration (MHA) degree program teaches aspiring public health professionals how to evaluate a program and how to apply public health program evaluation tools.

The Practice of Health Program Evaluation

Typically, health programs include policies, initiatives, interventions, preparedness efforts, and infrastructure efforts. They are complex, requiring time, resources, and funding.

The U.S. Department of Health and Human Services (HHS) has been a longtime source of funding for public health programs and initiatives. However, all programs have certain objectives to meet to justify their continued operation. For example, a program designed to decrease influenza cases must show that it makes a measurable impact to warrant funding and resources.

Quality researchers and evaluators are other important elements in the process. Those who possess a terminal degree, such as a Doctor of Public Health (DrPH) or Doctor of Philosophy (PhD), have the most opportunity. When comparing a DrPH and PhD, outcomes may differ. A DrPH can qualify a graduate to become a public health director whereas a PhD can qualify a graduate to work as a research director who focuses on analysis.

The Evaluation Process

The practice of health program evaluation applies a systematic method for collecting, analyzing, and using statistics and data to determine efficiency and effectiveness. An evaluation assesses what is required to run a program in relation to its quality and impact, including what works well and what needs improvement.

Additionally, evaluations provide public accountability and transparency, both of which can affect policy decisions and community support. The success of a public program in one part of the country may spark the launch of the same program in a different part of the country facing the same issues. On the other hand, the failure of a program to live up to expectations may prevent public health officials from launching it elsewhere.

Advance Your Public Health Career with an MPH Program

Pursue Your Degree Online From Tulane University

Find Out More

How to Evaluate a Program

Public health programs vary in size, scope, operation, and other key attributes. Those who wish to learn how to evaluate a program should familiarize themselves with the following:

Evaluation design

Evaluation considerations

Evaluation types

Evaluation change measures

Data collection strategies

Common evaluation challenges

Centers for Disease Control and Prevention’s framework for program evaluation

Types of Public Health Program Evaluation Design

An evaluation design is used to determine how a program affects participants, including their behaviors, knowledge, and healthcare outcomes. The three evaluation designs are as follows:

Experimental design. Experimental design determines if the new program is better than the current process by randomly assigning participants to a control or treatment group. Experimental design is considered the most effective and accurate of the three options.

Quasi-experimental design. Unlike experimental design, quasi-experimental design does not randomly assign participants; however, it may compare the treatment group to a similar group that is not involved in the program. A quasi-experimental design estimates what results a program will yield when conducting an experimental design is not possible.

Nonexperimental design. Nonexperimental design has no control group, meaning that the outcomes of the program or intervention lack the comparative element. However, this design still yields actionable findings that can be used to improve the program and develop best practices.

Types of Public Health Program Evaluation

Several different frameworks can be used to evaluate disease prevention and health promotion programs.

Formative Evaluation

During the development and implementation phases, formative evaluation is used to glean information about how to improve the program and/or achieve its goals.

Process Evaluation

Process evaluation uses data collection to determine if a program was successfully implemented as it was originally designed. If not, the evaluation determines how the program’s current operations differ compared with the original program’s operations. The result of the process evaluation answers two questions:

What services are being delivered?

Who is receiving the services?

Outcome Evaluation

Every public health program has goals or outcomes that it aims to achieve. An outcome evaluation determines if the program’s goals were met. A positive outcome evaluation can be used to justify expanding the program in the future. For example, when a program launched with the goal of decreasing a certain virus is successful, professionals seek to replicate that positive outcome.

Impact Evaluation

The goal of any program is to make a specific impact on a target demographic. An impact evaluation uses data to gauge the impact that the program had and the changes that resulted from it. For example, data can reveal the long-term impact of anti-smoking programs.

Performance Monitoring

Performance monitoring is the only type of program evaluation that is ongoing for the duration of the evaluation. Using specific data that gauges program success, evaluators can determine how well the program is performing.

Cost-Benefit Evaluation

A cost-benefit evaluation determines the cost-effectiveness of a public health program by weighing the cost of the program against the benefits and outcomes.

Public Health Program Evaluation Change Measures

The data and evidence collected from a program are used to evaluate it and measure changes over time. Examples are as follows:

Status change: Measurement of the improvement of a population’s healthcare indicators and status

Environmental changes: Factors in environmental variables that have changed, such as the removal of barriers between a population and healthier lifestyle choices

Change in knowledge: A population’s acquisition of new knowledge that enables them to make educated decisions

Change in behavior: A population’s adoption of new behaviors or amending of existing behaviors

Affective change: A population’s change in feelings or outlook about a particular issue or behavior

Public Health Program Evaluation Data Collection Strategies

Data is what drives an evaluation, so carefully considering what data is collected and the method of collection is important. Public health program evaluation typically uses quantitative and qualitative data to answer research questions and gauge program effectiveness. Quantitative data is numbers-based; qualitative data is language-based and open to interpretation. Common data collection strategies include the following:

Interviews

Surveys and questionnaires

Focus groups

Observation

Progress tracking

Other Public Health Program Evaluation Considerations

When devising how to evaluate a program, a few considerations should be made to ensure that the evaluation is comprehensive, including the following:

Resource inventory. The strategy should reflect the organization’s timeline, budget, and other available resources.

Program goals and objectives. Goals and objectives must be clearly defined for the program to properly be evaluated. Goals must have measurable indicators with a clear outcome. Objectives should be on a clearly defined timeline, address a specific population, and include relevant program activities.

Engage stakeholders. Stakeholders and other relevant parties can assist with the data collection process by ensuring that only accurate, verifiable information is collected.

Public Health Program Evaluation Challenges

No program evaluation is immune to challenges. However, evaluators will find that it is easier to maintain the evaluation’s integrity if they are aware of some common challenges beforehand, including the following:

Location challenges. Populations in rural or sparsely populated areas may lack transportation or access to programs, thus hindering the evaluation.

Determining effectiveness. The typical public health program addresses a combination of individual, environmental, systematic, and policy changes, potentially making it difficult to determine the effectiveness of a strategy.

Population diversity. Populations consist of a range of different people from different backgrounds, meaning that they have different feelings and opinions on certain healthcare issues. This can make the measurement of effective change extremely difficult.

Contextual factors. Public health issues have several factors that influence them, including economic and social drivers, environmental determinants, and genetic influences. Considering the contextual factors is critical when developing an evaluation plan.

Proving prevention. All public health programs share the same end goal: preventing negative healthcare outcomes. However, prevention is one of the most difficult metrics to measure. Most evaluation programs measure the activities and processes that were implemented as a means of prevention.

Measuring outcomes. Changes in status, environment, behavior, knowledge, and outlook will occur when a program or an intervention is rolled out; however, it is difficult to determine the percentage of those changes that can be attributed directly to the program.

Centers for Disease Control and Prevention Framework for Program Evaluation

The CDC divides the practice of health program evaluation into six steps that should be taken at the beginning of the public health program’s rollout. The steps are as follows:

Engage all stakeholders.

Describe the basic program.

Fine-tune the evaluation design.

Collect evidence.

Explain conclusions.

Ensure implementations and discuss lessons learned.

Here is a general overview of the six steps and their roles in a public health program evaluation.

Step 1: Engage Stakeholders

The individuals or organizations invested in a program’s outcome are known as stakeholders. Program evaluation is an effective way to show stakeholders that their interests are being considered.

Some stakeholders are a significant source of program funding, or they can advocate for program expansion.

To help shape evaluations, engage stakeholders by asking what they envision, including the following:

Program outcomes and activities that are essential to retaining their involvement

Most important questions to answer with the evaluation

Types of data to collect

Resources the stakeholder can contribute for the evaluation

Which points the stakeholder wishes to be involved in

Preferred method of communication for updates

What the stakeholder intends to do with the evaluation results

Engaging stakeholders is an essential part of the evaluation process, especially if the program relies on their funding or resources. The more stakeholders involved, the more difficult it is to address their wants and needs.

Step 2: Describe the Program

A description of the program to stakeholders and organizations should clearly illustrate the relationship between its activities and its desired outcomes. According to the CDC, this is best accomplished by using a graphic representation of the program road map, also known as a logic model.

The following elements are typically factored in when constructing a logic model:

Inputs: Essential resources needed to implement programmatic activities

Activities: Actions taken by staff and how the program uses resources

Outputs: Tangible deliverables that are a result of activities

Outcomes: Measurable changes that occur in the environment, populations, and so forth as a result of outputs and activities

Impacts: Outcomes that are long term or permanent

Moderators: Any contextual factors that can influence outcomes and impacts but are beyond the control of the program

Step 3: Focus the Evaluation Design

The third step determines which parts of the program are under scrutiny. The following assessment standards should be applied:

Utility: Determines which stakeholders or organizations need evaluation information and how it will be used

Feasibility: Determines how much time, money, and other resources can be invested in the evaluation

Propriety: Determines which parties need to be involved for the evaluation to be considered ethical

Accuracy: Determines which evaluation design will result in accurate information

According to the CDC, utility and feasibility are the two most important standards of assessment for public health program evaluation.

Step 4: Gather Credible Evidence

Evaluation judgments and subsequent recommendations must be supported by relevant data and information. Examples of credible evidence are quantitative or qualitative data collected as a result of a survey, an observation, or an experiment. Most credible evidence is culled from multiple sources using a mixture of methods.

Regardless of the type of data or its source, it must share one common denominator: Stakeholders must find it relevant and trustworthy. The entire evaluation may be compromised if stakeholders do not agree on what qualifies as credible evidence.

The best practices for gathering credible evidence include the following:

Rationalizing with stakeholders on how information is sourced and its validity

Developing clear procedures of data collection and ensuring that staff is adequately trained to process data

Conducting quality assurance on collected data that confirms its legitimacy and accuracy

Determining how much data needs to be collected to establish a judgment

Ensuring that data is appropriately safeguarded and accessible only by authorized parties

Step 5: Justify Conclusions

The fourth step generates one or more conclusions of the program evaluation; this fifth step reinforces those conclusions by linking them to credible evidence and supporting them with data. Additionally, the conclusions must be justified according to stakeholder standards.

The following activities may be used to justify conclusions:

Using different methods of analysis to summarize findings

Determining the significance of results and how they should be interpreted

Comparing results using alternative methods

Identifying alternative explanations for how results were generated

Step 6: Ensure Use and Share Lessons Learned

The final step in the CDC framework determines if the evaluation met its primary goal. After the evaluation, stakeholders review its key findings to inform future program decisions, such as whether to expand it nationally.

Additionally, the evaluation results are distributed to improve the process. Follow-up meetings with stakeholders for feedback can inform future evaluations.

Public Health Program Evaluation Tools

Program evaluation is complex, with numerous elements to consider; this is why the CDC supplies several public health program evaluation tools on its website to aid the process. The main evaluation tools include the following:

A resource that breaks down logic models in detail, including how to best develop one and how to use them

A breakdown of indicators and performance measures, including how they can be implemented and examples

A tool that helps with the development of evaluation reporting

A resource section covering the value of economic evaluations and how they are incorporated

A section covering evaluation databases and data resources

A resource that highlights the use of health communication tools

Strategies, interventions, and program evaluation best practices

Webinars and podcasts that discuss program evaluations

Public Health Program Evaluation Example

Launched in 2014, “The Real Cost” campaign marked the first smoking prevention effort from the U.S. Food and Drug Administration (FDA). For stakeholders, the main desired outcome was to prevent youth from adopting the habit.

To accomplish this, a paid media strategy aimed at the at-risk target audience was executed using evidence-based best practices to discourage smoking and tobacco use. TV ads were tested on a large demographic of at-risk youth to determine the effectiveness of the messaging before rolling it out to a wider audience.

Additionally, the FDA hired RTI International to measure relevant indicators of success for the first two years of the campaign. Most notably, the FDA wanted to measure the change in knowledge about the dangers of smoking and the thoughts and feelings that the target demographic had regarding smoking. RTI International reported that over 90 percent of the target audience saw the initial wave of anti-smoking ads.

Ultimately, “The Real Cost” campaign was a resounding success. It prevented up to 587,000 youth (ages 11-19) from trying cigarettes and saved more than $53 billion by reducing medical care costs and disability claims related to smoking.

Using this public health program evaluation example as proof of concept, the FDA subsequently launched an e-cigarette and smokeless tobacco prevention campaign.

Promote Community Wellness in the Field of Public Health

Public health program evaluation is a critical aspect of promoting the health and wellness of communities. By carefully evaluating health promotion and disease prevention programs, public health workers can prove their value and effectiveness to stakeholders and the public. Further, the practice of health program evaluation can be used to improve programs and justify their expansion.

The best way to get involved and learn how to evaluate a program is by pursuing a relevant degree path, such as the Online MPH in Community Health Sciences offered at Tulane University. During the program, students will become well versed in areas such as epidemiology, behavioral science, and biostatistics.

Learn more about the program to find out how it can support your professional goals in public health.