Structural sensitivity analysis: model map

To test the model’s sensitivity to the selected geography, I generated 9 additional population density maps based on the 2010 United States Census merged with HRSA Medically Underserved Area maps using the QGIS GIS mapping software (QGIS, 2017). The population density maps are shown in Figure 3-4 and the underlying MUA maps are shown in Figure 3-5. I selected the maps from a wide array of regions in the US including the North East, South East, Midwest, Mountain West and California, and include highly urbanized regions (such as Figure 3-4, bottom middle), and rural regions (such as Figure 3-4, bottom left). Models were run 30 times each for one year, initialized at the start of 2013, and buprenorphine recipients, opioid overdose deaths and diversion outcome variables were compared against baseline model runs. I normalized diversion and opioid overdose death outcome variables to the population size to allow for direct comparison. The buprenorphine recipients outcome variables for each set of runs was also compared against 2013 unique buprenorphine recipient data.

There were no significant differences in opioid overdose deaths when using different starting maps. Regions 1, 3 and 6 had significantly fewer buprenorphine recipients than the baseline Region 0 (Figure 3-4 top left, top right and middle right, respectively), with mean values 12%, 6% and 7% lower than the mean number of recipients per 100,000 population. Regions 1, 3, 5, and 7 (Figure 3-4 top left, top right, middle middle, bottom left) had significantly lower diversion (mg per 100,000

population) than the baseline region, with mean values 14%, 10%, 7% and 15% lower.

Regions 4 and 8 (Figure 3-4 middle left, bottom middle) had significantly higher diversion (mg per 100,000 population) than the baseline region, with mean values 7%

and 6% higher. In general, more densely populated regions had higher diversion than less densely populated regions despite normalization, and may signal a systematic bias in how demand for diverted buprenorphine is modeled in remote and urban areas.

The number of unique buprenorphine recipients at year end was low for all regions. Model confidence intervals did not cover the actual year-end value adjusted for model population in any region. Inadequate calibration to the year-end target in the baseline region reported in Section 3.4.3 resulted in the use of poorly calibrated

parameter values in all regional model runs.

Figure 3-4: Alternative population density maps for geographic sensitivity testing. Population density is measured in people per square mile. Maps are numbered sequentially left to right, top to bottom.

Figure 3-5: Additional Medically Underserved Areas (MUA) maps for geographic sensitivity analysis. Darker regions are MUAs. Maps correspond to population density maps in Figure 3-4.

3.4.5 Recalibration

The original research plan called for calibration on 2013 data, external validation on 2014-2015 data, and sensitivity analysis followed immediately by policy

experimentation. Rather than perform policy experiments on a model that failed tests of external validity, I chose to iterate and use all available data for model calibration and to perform policy experiments on a better calibrated, though non-validated model. Failure to validate large simulation models or poor results on tests of model fit tend to be a challenge when striving to simulate complex social systems.

To recalibrate the model, I made three choices—

1. to initialize the model in 2013 with the number of people with opioid dependence calculated the same way as the 2014 and 2015 populations,

2. to tune provider preference parameters to privilege fit to total unique buprenorphine recipients over fit to provider patient census in Stein, et al (2016),

3. to use a non-stationary opioid overdose death rate (the death rate changes over time).

I chose to recalculate the initial population with OUD in 2013 because calculating the population in a different way resulted in an apparent decrease in people with OUD from 2013 to 2014 due to a modeling artifact. I removed this spurious effect by

calculating the population the same way for all model years. The modeling artifact was

introduced in the first place because of the considerable lag between early modeling efforts in iterations 1-3 of the model and validation efforts on model iteration 4.

I chose to recalibrate the model by tuning provider preference variables because these variables have considerable leverage over the unique buprenorphine recipients outcome measure as shown in structural sensitivity analysis (see Section 3.4.4.2).

Further, Stein’s regression model on 2009-2011 patient census data showed that patient census levels were rising even in 2011 (Stein BD et al., 2016).

Lastly, to fit trends in opioid overdose deaths in 2013-2015, I chose to

parameterize the model with a non-stationary opioid overdose mortality rate. It may have been possible for a non-stationary opioid overdose mortality rate to have arisen endogenously by introducing considerable model complexity. I could have chosen to model people’s shift from prescription opioids to heroin, from oral to injection drug use, and other transitions from lower risk to higher risk behaviors, but I would likely still have had to introduce exogenous factors that drove up overdose mortality, such as changes to price and purity of heroin and the introduction of synthetic fentanyl (see Section 3.4.2, Face Validation). Rather than introduce this complexity, which I felt would have little relevance to capacity and access equity research questions, I chose the simpler option of a time-varying opioid overdose mortality rate.

The following parameters were tuned to fit the number of unique

buprenorphine recipients in 2013, 2014 and 2015: theoretical patient capacity for

non-specialists with a high waiver and for non-non-specialists with a low waver not on the SAMHSA searchable list, and percentage of patients in treatment at baseline each year.

Table 3-14 shows all post-recalibration parameter values including crude mortality rates.

The initialization data and input data parameterizations reported in Sections 3.3.1 and 3.3.2 record parameter settings after calibration and recalibration was complete.

Table 3-14: Parameter values changed during recalibration and the final values of these parameters.

Theoretical patient capacity

Non-specialist, high waiver, on SAMSHA list

Non-specialist, low waiver, not on SAMHSA list

30 + random exponential(220) patients random normal(20, 7) patients

Percentage of patients in treatment at baseline

Opioid overdose crude mortality rate (not in treatment)