VALIDATION METHODS AS USED IN PHASE 3

2.4.5.5.1 SYSTEM DYNAMICS MODEL STANDARD VALIDATION METHODS IMPLEMENTED

Table 2.17 below lists the SDM standard method model validation tests (in the first column), the procedures I used to implement those tests (in the second column) and the test definition (in the third column).

103 Table 2.17 Validation Methods Implemented

Name of test What I did (see Chapter 3 Section 3.5 for results) Definition[123] (p. 189-195)} and[105] (p. 123) Parameter Confirmation - Theoretical I reviewed literature for numerical estimates of the constants in

the model.

“Evaluating the constant parameters … [against generalized knowledge about the system that exists in the literature] both conceptually and numerically.”[123]

Parameter Confirmation - Empirical Implemented using the following methods: SSM-S, CM-S, SIM-S and Stakeholder Dialogue.

“Evaluating the constant parameters against knowledge [obtained directly from] the real system, both conceptually and numerically.”[123]

Structure Confirmation - Empirical Implemented using the following methods: SSM-S, CM-S, SIM-S and Stakeholder Dialogue.

“Comparing the model structure with information (quantitative or qualitative) obtained directly from the real system being modeled.”[123]

Structure Confirmation - Theoretical Related models were identified but addressed different problems, prohibiting formal comparison.

“Comparing the model structure with … [generalized knowledge about the system that exists in the literature].”[123]

Behavior Reproduction I checked the model to see if it was able to reproduce the reference mode. I also checked that the model was able to reproduce both failure and success modes as described in the problem statement.

“Does the [simulation model’s] behavior match any historical data and/or the reference mode?”[105]

Extreme Policy I implemented values for each policy that represented extreme conditions and checked that the model behavior was reasonable.

“When policies are pushed to extremes are the [simulation model’s] behaviors reasonable?”[105]

Mode Reproduction Ability I implemented several past policies in the model to see if it was able to reproduce behavior consistent with historic behavior.

“With different past policies, does the [simulation model] yield behaviors consistent with other [examples] of the system”[105]

Behavior Prediction I checked that the model reproduces the anticipated behavior for future/hypothetical situations.

“Does [the simulation model] reproduce the anticipated behavior in future/hypothetical situations?”[105]

Anomalous/Surprise Behavior I simulated the model under various conditions and found anomalous behaviors. I studied these causes of these behaviors in the model. When these behaviors were indeed anomalous, corrections were made to remove them. Other times, they were surprise behaviors that led to insights.

“Have odd behaviors been studied to show that either: 1) they are anomalous, needing [simulation model] corrections to remove them? 2) the [simulation model] yields insights into a previously unrecognized mode?”[105]

Perspectives Boundary Adequacy Implemented using the following methods: SD-S, Stakeholder Dialogue Suitability and Methods Suitability.

“Do the models support debate on different perspectives in the [Problem Statement] concerning: 1) choice of modelling approach used? 2) System dynamics issue addressed? 3) Goals to be achieved? 4) Policies for doing so?”[105]

104 Name of test What I did (see Chapter 3 Section 3.5 for results) Definition[123] (p. 189-195)} and[105] (p. 123)

Structure Boundary Adequacy Implemented using the following methods: SSM-S, CM-S, SIM-S, Methods Suitability and Stakeholder Dialogue.

“Do the models contain sufficient and appropriate variables, policies and feedback loops to address the issue that they are being built to study?”[105]

Dimensional Consistency Used the “check units” feature in Vensim to confirm that the right-hand side and left-hand side of each equation passed the dimensional consistency test.

“Checking the right-hand side and left-hand side of each equation for dimensional consistency”[123]

Extreme conditions in equations I implemented extreme values for constants in the model and checked that the outputs of each policy were reasonable.

“Are the outputs of policies reasonable if the inputs take extreme values?”[105]

Behavior Sensitivity I implemented sensitivity analysis without policies turned on and checked that the model output was plausible. Specifically, I performed a Monte Carlo analysis with 200 runs for each of the constants in the model, each in a range of 50% their original values.

“Are the previous behavior tests compromised by the plausible changes in parameter values?”[105]

Behavior Boundary

Adequacy/Structure Sensitivity

I implemented multiple equally-likely structures in the model during model development. As I tested each one, those

producing behavior that did not match the behavior described by interviews or operational thinking or those where a simpler model produced the same behavior were abandoned.

“Does the [simulation model] contain sufficient and appropriate variables, policies and feedback loops to address the issue when this is tested by adding new pieces of relevant structure and examining the resulting behavior?”[105]

Policy Sensitivity I implemented sensitivity analysis for the model with various policies turned on.

“Are the suggested [policies and recommendations] robust to plausible parameter changes?”[105]

Policy Boundary Adequacy I added possibly relevant structure and tested its impact on behavior, under different policy scenarios.

“Does the addition of more possibly relevant structure change the [policies and recommendations]?”[105]

Norms/Values Boundary Adequacy Implemented using the following methods: SSM-S, CM-S, SIM-S and Stakeholder Dialogue.

“Do the models support debate concerning, and represent the behavior of the relevant actors': 1) goals (are the desired states acceptable?) 2) Policies (are the actions based on discrepancies between goal and actual conditions acceptable within the culture?)”[105]

Roles Boundary Adequacy Implemented using the following methods: SSM-S, CM-S, SIM-S and Stakeholder Dialogue.

“Are the feedback links in the models consistent with the abilities of current actors in the system to access, interpret and employ information?”[105]

Italics are used to denote the use of newly-developed validation methods described in Appendix E.

105 2.4.5.5.2 SIMULATION MODEL SATURATION

This test consulted SIM1, the Causal RIQs, the problem statement, and the dynamic hypothesis (i.e., the Conceptual Model) as described in Appendix E.

The problem statement and dynamic hypothesis were consulted to see: how consistent was SIM1 with the structure and behavior that it was meant to represent.

Also, variables and relationships in SIM1 were checked, that they make sense in terms of the descriptions provided in the quotations and interpretations (i.e., proper time frame, proper designation as stock/flow/constant, proper scope). Then, the full quotation was reviewed with SIM1 in hand, to make sure that the causality described in the quotation is covered in SIM1.

When goals and policies were discussed, I checked that they were also found in the simulation model and that they were within scope of the cognitive limitations apparent in the quotations.

Finally, the model was run using the structural aspects described in these quotations and the run results were compared with behavioral expectations mentioned in those quotations.

Modifications to SIM1 were made, as needed in this test. Once complete, the model was called SIM2.

2.4.5.5.3 STAKEHOLDER DIALOGUE REVIEWING SIMULATION MODEL In this step, stakeholders interacted with the simulation model. The simulation model permits a high-level discussion of the structural theory in the model and the behavior patterns it produces. Stakeholders interacted with the model as they would with a rough draft flight simulator – 1) checking for bugs (i.e., looking at anomalous simulation runs and discovering causes from flawed formulations, whether conceptually or mathematically), and 2) discussing how best to organize the dashboard and outcome graphs.

Stakeholders engaged were: national experts in primary care, primary care clinicians and experts in health services research and SDM in health. One-on-one meetings were held with each stakeholder. Notes were taken during each discussion and model modifications were made after reflecting on those notes.