Collecting Data for Management, Monitoring & Evaluation

By reviewing this section of the guide first responders will be able to:

1. Understand the importance of collecting data for management, monitoring, and evaluation of deflection and diversion programs

2. Develop strategies for using available data source to manage and monitor local programs

Data collection is done for several purposes:

Data collection is completed for several purposes: 1) Case management, 2) Program monitoring, and 3) Evaluating effectiveness.

Case management involves collecting data about individual participants and activities so that program staff can identify participants, track progress and participation throughout the program, make informed decisions about participants, and share information with relevant partners as needed. Program monitoring helps program staff and management assess how well the program is working and identify opportunities for improvement. Program monitoring takes place on an ongoing basis as the program is being administered and can involve reviewing outputs/activities and client-level outcomes. Impact or outcome evaluations examine the effectiveness of a program on key outcomes after sufficient time has elapsed following program participation or adoption. Collecting data is an essential part of program implementation because it is necessary for operating the program, revising on an ongoing basis, and ultimately demonstrating the effectiveness of the program.

In all cases, program planning and implementation activities should include strategies for collecting data on participants, program outputs/activities, and key outcomes. While many practitioners report measuring performance indicators, studies have shown that few organizations conduct formal evaluations due to limited capacity. This is often due to challenges in transferring knowledge into practice and insufficient staff expertise and the complexity of evaluating human change processes.[69] Still, practitioners can help to initiate data collection efforts to support evaluation efforts that might come later. Fully designing an impact evaluation is beyond the scope of this guide so we focus on the measurement of key outputs and outcomes that will be useful to monitoring and demonstrating the potential effect of the program; for sites that are interested in more robust evaluation, research partners can enhance ongoing evaluation activities by helping first responders to develop evaluation methodologies or measurement strategies.

Section A: Data Sources for Program Monitoring & Evaluation

Three main sources of information are available to program staff
to monitor and prepare for evaluation of the program:

1. Law Enforcement 911 Emergency Call Data And Incident-Based Data

These data are integrated into routine first responder practice in most communities and can be used to measure key outputs (e.g., field contacts) and outcomes (e.g., overdose events). Often data is available for long periods prior to adoption of a deflection program, allowing program staff to monitor changes before and after the adoption of new practices. Agencies frequently have some capacity to prepare and analyze these data. When using 911 emergency call data or incident-based reports, it is important to understand the context of these data; there are cautions about the accuracy of certain measures due to officer reporting decisions in the field. This data also does not measure all the output and outcome measures that would be important to monitoring and evaluation, but can be a very useful supplement to programmatic data. 

2. Program-Specific Case Management Data

Case management data includes information about specific program participants that is collected as part of the day-to-day operations of the deflection program. This often includes information about participant characteristics (i.e., demographics, referral information) and programmatic decision-points or activities. Case management systems vary in terms of their complexity (e.g., simple spreadsheets to relational databases) and the scope of information they collect. While the primary purpose of case management data is to help program staff and supervision to manage decisions about participants, the data collected through case management is a critical source of information that can be used to measure program outputs and outcomes for both monitoring and evaluations.

NOTE: Depending on the program structure, some case management data may be subject to privacy protections under HIPAA or CFR42 Part 2. When creating a first responder deflection or diversion program, leadership should consider how to most effectively protect sensitive information, and also how this will affect reporting and evaluation in the program.

3. Original Data Collection Activities, Including Surveys Or Interviews, Usually Performed By External Researchers
The advantage of these sources is that they can be designed to measure program-specific outputs and outcomes that are not collected through other sources. For example, interviews with participants can collect information about drug use – a key outcome measure – that police data or even case management data would not capture. In contrast to incident-based data or case management data, original data collection activities do not need to occur in an ongoing way but can be administered for a short time to gather specific information and then discontinued. Researchers can help design strategies to collect information in ways that produce reliable and valid measures and involve a lower burden for program staff and participants. We do not focus on providing guidance around this source of information because it necessitates research partners and is less useful for ongoing program monitoring.

Section B: Measuring Key Outcomes & Outputs

Outcome measures are tied to the program’s specific goals. Deflection and diversion programs often seek to reduce overdoses, justice-system involvement, and criminal behavior. Program staff can measure these outcomes through police or program management data. When monitoring diversion and deflection model outcomes, most will be measured at either the individual-level or the aggregate-/community-level over a set observation period. For example, the number of overdose events can be tracked through self-report (individual participants) or emergency calls for service (community). Similarly, program staff can record how often an individual is arrested, or look at overall arrest rates in a community or target area; for example, a program might report the number or rate of arrests per population for all participants in the program or for implementation sites.

Next, another essential part of program monitoring and evaluation is measuring outputs. Outputs are measures of the specific activities or services that the program delivers. Although there is variation in the specific kinds of activities and outputs that deflection and diversion implementers will engage in, deflection initiatives are commonly seeking to engage people in the program (e.g., number of referrals, number of contacts), deliver resources (e.g., number of Naloxone kits distributed, number of test kits distributed), provide services (e.g., number of recovery coaching sessions), refer them to treatment providers, and guide them through program steps or stages. Most of these activities are client-based and therefore outputs are often derived from case-management data. 

To identify key outputs and outcomes, we recommend drafting a client flow diagram, which will help identify specific opportunities for reporting or monitoring. Below, we provide a simple client flow diagram for a Self-Referral Program (the client flow is bold, and examples of data points that could be used for evaluation are below in italics):

Client Flow PAARI Promising Practices Guide

Section C: Practical Uses for Data

To better understand data collection efforts, it is helpful to see how the data can be used in practical ways. Deflection and diversion programs and published process evaluations give good examples of how data can be used to monitor programs. A few example questions will illustrate the straightforward process of 1) question or problem; 2) data collection; 3) analysis; 4) dissemination; and 5) action for change. 

Example 1.
How well does the program connect people to treatment?

Through case management data, program staff will know the number intakes, referrals, and enrollments in treatment. They can therefore assess, on an ongoing basis, the percentage of referrals that have been enrolled in treatment. Staff can further examine this metric across different groups, sites, or other distinctions that are meaningful to program staff. These results can be shared with partner organizations. If connections are lower than expected, or decreasing over time, or varying across subpopulations, program staff can use that information to advocate for improvements. 



Example 2.
How well does the program utilize diversion for individuals that have committed eligible offenses?

Police data on arrests can be linked to program management data to determine the percentage of arrests that have a referral. An agency that arrests 60 people for possession offenses during a given timeframe and makes 30 referrals to program case management staff, has a referral rate of eligible individuals of 50%. Program staff can assess how well this meets expectations, changes over time, or varies across areas of the city or site. This data can be shared with managers across the department and external partners. They can act to create awareness of the referral process or reinforce expectations through supervisors.

Section D: Additional Resources for Data Collection/Performance Measurements

  • Police Assisted Addiction Recovery Initiative (PAARI) convened experienced researchers and evaluators from the field of deflection, and PAARI partners, to develop and recommend a set of standards for programmatic data collection. PAARI’s “Standards for Deflection Data Collection” guide is a great resource to review. 
  • The Wisconsin Deflection Initiative developed a comprehensive guide on Deflection Performance Measures to provide suggested quantitative measures as a framework to measure progress and success of deflection initiatives.  
  • The Police Treatment and Community Collaborative (PTACC) has recommended core measures for deflection and pre-arrest diversion.