Data and Methods

The processes to be used to collect and analyze data will also be structured by these three components of the evaluation approach.

1. Context Evaluation Data Collection Process: The primary data collection method to be used in the

context evaluation will be key informant interviews. Key informants include NH Bureau of Drug and Alcohol Services (BDAS) staff, leadership, coalition members, resource partners, and community members including youth and families. Once stakeholders have agreed upon the evaluation questions, a semi-structured key informant interview protocol will be designed to collect this information. In addition, to key informant data, the evaluation process will entail review of documents (e.g. written guidelines, financial reports, and meeting minutes) to track inputs, structural developments and contextual changes. While a central evaluation team will be needed to take the lead on these responsibilities, the process is participatory and necessitates input from multiple groups and stakeholders. Data collection activities in this component of the evaluation will be ongoing throughout each Plan year and the results will be summarized along with findings and lessons learned from the other aspects of the evaluation in an annual Strategic Prevention Enhancement Plan Progress Report.

2. Process Evaluation Data Collection Process: Several data collection methods will be applied in the

process evaluation component of this plan. For the purpose of tracking program activity and aggregate outputs including data on events, participants and resources used, we will anticipate the use of the improved

Performance Management System (P-WITS) that is currently in development with by the BDAS. An improved P-WITS has been identified as a priority through the first phase of the State’s Strategic Prevention Enhancement grant activities. In addition, to tracking program performance from an activity and output perspective through P- WITS, periodic surveys of Prevention System Partners members will be conducted to assess the status in key areas and processes that are characteristics of successful partnership-based systems including:

• Formation (e.g. respected conveners, sense of a Shared Vision, framework established and communicated for acting on vision)

32

• Functioning (e.g. Strong and Collaborative Leadership (in addition to staff), diverse participation with clear understanding of potential contributions, open discussion/resolution of potential conflicts

• Operations (e.g. staff skilled in community engagement, organizational stability, diversified funding) • Actions (e.g. adapts to changing circumstances and lessons learned; aware of/leverages community assets) • New Community-Based Capacity (e.g. increased community awareness and understanding of efforts;

expansion of partners/participants committed to vision, increased media attention to prevention issues) In addition to these approaches, the process evaluation will include key informant interviews, participant surveys, and document review. Semi-structured key informant interviews (completed at the same time as the context evaluation) will gather more in-depth information about the characteristics of activities, level of partner involvement, satisfaction with the process, influencing factors, lessons learned and recommendations for improvement. Documents such as meeting minutes and program plans will provide additional information on implementation success and the level of partner engagement.

3. Outcome Evaluation Data Collection Process: The data collection sources and techniques utilized to

monitor trends in substance use indicators will address the core measures addressed in the Logic Model as well as consequence/social indicator data. The data sources include the following:

• Youth Risk Behavior Survey (YRBS): The NH YRBS is administered bi-annually to a statewide, random sample of public high school students, as well as to a ‘local sample’ of high schools on an opt- in, voluntary basis. In the latter case, the complete student enrollment in 9th through 12th grade are administered the YRBS –historically comprising about 75% of NH high schools. Measures to be obtained from the YRBS include:

o Age of onset of drug use (including alcohol, marijuana, and tobacco) o Past 30-day use (including alcohol, marijuana, and tobacco)

o Perception of risk or harm (including alcohol, marijuana, and tobacco)

o Perception of disapproval of use by parents (including alcohol, marijuana, and tobacco)

Beginning in 2011, the NH YRBS instrument also included additional questions addressing prescription and over-the-counter drug use including lifetime use, 30-day use, perception of harm, personal and parental perception of disapproval, and ease of access.

• National Survey on Drug Use and Health (NSDUH); In addition to the YRBS, state and sub-state estimates of substance use from this national survey will continue to be monitored with particular emphasis on prevalence estimates for the 12-25 year old population.

• NH Department of Health & Human Services, Web-Based Reporting and Query System (NH WRQS) and Web-based Interactive System for Direction and Outcome Measures (WISDOM) – The NH BDAS is currently working with the NH Division of Public Health Services on development of improved systems for public reporting and query of public health data including substance abuse prevention and treatment-related information. These systems are being supported with Strategic Prevention

Enhancement resources and will contain publically- available ‘dashboards’ and data at the community, regional and state level on hospital and emergency department admissions including substance-abuse related admissions, behavioral health-related admissions, and other measures of health status including behavioral risk factors. These information systems will facilitate tracking and trending of population health outcome measures and targets as specified by this Strategic Plan.

Education collects and reports information on individual school drop-out rates and school safety

including counts of incidents of in-school and out-of-school suspensions and expulsions and the reasons for them (including drug- and alcohol-related incidents).

• NH Bureau of Liquor Enforcement Retailer Compliance Rates: Reports of retailer violations as a proportion of licensed establishments at the state, regional and community level.

• U.S. Drug Enforcement Agency: Reports of prescription take-back events, dates, locations, participating communities, and number of pounds collected.

• Department of Safety, Uniform Crime Reporting: annual statistics for alcohol- and drug-related arrests. The key informant interviews and partnership surveys previously described will also be utilized to enrich the outcome data with specific information about impacts on the state and local prevention systems, new resources and partnerships, generated level of engagement of specific sectors, and evidence of cultural and attitudinal change.