When developing an overdose response plan, completing a formative evaluation helps you assess whether the plan is realistic and well matched to the situation. Think through the following with your response team:

An economic evaluation helps assess the costs of a program relative to its effects. This can be a cost-benefit analysis or a cost-effectiveness analysis. Cost benefit analysis requires an estimate of the costs of undertaking a program and the estimates of its benefits. These costs and benefits are then translated into comparable monetary units. The alternative approach, cost-effectiveness analysis requires monetizing only the program costs. Its benefits are expressed in terms of the costs of achieving a given result. 

In the context of an overdose response plan, an economic evaluation can help teams decide how best to respond with the resources available. Think about the considerations below:

Process evaluation determines whether the plan is implemented as intended. It helps document and assess program activities to ensure consistency across different settings, considering both operational execution and service utilization. For an overdose response plan, implementation can be evaluated in two ways:

One-time process evaluation: collect input from staff and stakeholders at a single point in time to check that activities align with the plan.
Continuous process monitoring: Track a few key indicators at set intervals (or as needed) to provide timely feedback and guide adjustments.

Below is a logic model framework that illustrates response protocols when using the ESSENCE syndromic surveillance system. This is an example of a continuous process monitoring to detect anomalies in the data by following a set protocol.

• A successful process evaluation examines the following six components:
• Assess similarities in demographics and characteristics (age, sex, race/ethnicity, area of residence, prescription history, prescribers, where overdosed) 
• Obtain records of interest: EMS, ED and hospital records, law enforcement reports, toxicology testing results, and death certificates, if applicable 
• Patient disposition (i.e., mortality and morbidity), if available, can be leveraged to assess lethality 
• If narratives are available, look for indications that the same substance was used 
• Check retrospective data to see if there have been previous instances of the same event (same county, same level, etc.) 
• Check neighboring areas, such as counties 
• If available, look at granular geographic data for potential clustering and proximity to a major highway 
• Use mapping to identify potential clusters of cases in the jurisdiction 

Some examples of questions that a process evaluation for an overdose response plan can answer are:

• Does the public health agency have adequate number of qualified staff to perform the required functions?
• Does the public health agency coordinate effectively with community partners who are crucial to implementing the plan?
• Are the resources being used efficiently?
• Since some states have a centralized overdose response plan while others are decentralized, one question could be to understand the variation in performance between sites and addressing potential barriers, if any.
• Are partnering organizations satisfied with the interaction with each other and the overall functioning of the program?

An outcome evaluation examines the progress in the outcomes the program intended to address (i.e., the right side of the logic model in Process Evaluation). Outcomes are the program’s objectives for change and reflect the goals for the program set by the stakeholders. The challenge in conducting an outcome evaluation is translation of these goals and objectives into outcome measures that can be captured.

Like other programs, implementation of an overdose response plan can result in short-term, intermediate, and long-term outcomes. While long-term outcomes are considered to have the most practical importance, short-term and intermediate outcomes are the easier to assess and, during implementation, can provide early indication of whether the program is moving in the desired direction. Examples of each can be found below:

Short-term outcomes

• Earliest possible identification of an overdose alert.
• Timely communication of the situation to partners and public.
• Timely coordination of activities with partners.
• Earliest possible identification and investigation in case of an event that impacts the public.
• Ongoing implementation and evaluation of the plan.

Intermediate outcomes

• Improve data sharing within sector (public health, public safety, behavioral health)
• Improve overdose response plan process (communication, efficiency, etc.)
• Increase collaboration and data sharing
• Ongoing implementation and evaluation of the plan.

Long-term outcomes

• Reduce or prevent morbidity and mortality from overdose events.
• Better quality of life among individuals with a substance use disorder.
• Constantly evolving overdose response plan to better meet the needs of public.

Impact evaluation assesses the difference between the outcomes occurring with implementation of the program and those that would have occurred if the program was not implemented. The goal here is to determine whether the program has produced the desired effects in the target population. Therefore, impact evaluation has a huge potential to influence high-level decisions and is sometimes considered synonymous with program evaluation.

While the main question answered through impact evaluation is whether the program produced the desired effects, there are other questions that can be answered also. Some common questions that can be answered in the process of evaluating the impact of an overdose response plan are listed below:

• What is the difference in the desired outcomes that is attributable to the overdose response plan?
• Are there any unintended positive or negative consequences of implementing the overdose response plan?
• What is the impact of overdose response plan on outcomes for different subpopulations in the region where the plan is being implemented?
• Is there a chance that provision of higher quality services and more resources would result in better outcomes?
• Does overdose response plan implementation match with the intended plan for implementation?
• How does the fidelity of the plan vary across time, regions, and individuals (staff and/or population)?
• Is greater fidelity of implementation associated with larger program effects?

As it requires comparing the outcomes of the program to those that would have occurred in the absence of the program, impact evaluation can be challenging. While a randomized control design is the ideal choice for impact evaluation, there may be several constraints to evaluating an overdose response plan using this design. The impact of an overdose response plan can be evaluated by comparing data on the same group of individuals before and after implementation of the plan. This could include all individuals residing in a particular ZIP Code, county, council district or state. Another approach is to compare a region where the overdose response plan is being implemented to a region where it is not. However, these two regions should be comparable in ways that would produce same results for both regions if the plan was not implemented. Multiple comparison group designs such as these can be used to measure program effects by using approaches to reduce or eliminate potential bias. These can include multivariate regression analysis to statistically adjust the effect estimate for differences between the groups, using fixed effects designs to eliminate biases associated with difference between groups on characteristics that are stable among a group, etc.