NEWLY-DEVELOPED VALIDATION METHODS

This section presents the newly-developed validation methods. These methods are described in detail in Appendix E. In this section, I introduce these methods by providing a brief overview of the goals of each method. For ease of the reader, I use the full method name in this section;

hereafter, each method’s corresponding acronyms are used. I will explain later in the chapter (Table 2.13) how these methods map to those listed in Table 2.6. Additional information regarding this mapping is also provided in Appendix E.

Table 2.10 below lists the new methods, their acronyms and focus questions as well as the data set used and level of analysis. Model-specific validation tests are used in the various phases of model development. Meta-level validation tests enable periodic reflection on the meta-level validity – aspects that relate to the overall project.

84 Table 2.10 Formal Validation Methods Designed for Textual Mental Databases

Method name Acronym Focus Question

Data Set Used

Level

Model Development Set Model Validation Set Model Meta

Shared Mental Model Saturation SMM-S Do the different clinics agree on the

structure of the system? x x

Conceptual Model Saturation CM-S

Does an additional clinic agree with SMM2 on the structure of the system?

x x

Simulation Model Saturation SIM-S

Does an additional clinic agree with SIM1 on the structure and behavior of the system?

x x

System Dynamics Saturation SD-S

Does an additional clinic confirm the usefulness of SDM as the modeling approach?

x x

Methods Suitability N/A

What are the trade-offs of the specific modeling approach used in this research?

x

Data Suitability N/A

Are the data upon which the model is based used within the scope of their limitations?

x

Stakeholder Dialogue Suitability N/A

Have stakeholders been

appropriately engaged throughout the research project?

x

Three of these methods (CM-S, SIM-S, and SD-S) use a set of interviews that was left unused throughout the model development process. This subset was selected based on participant characteristics (in this dissertation, having clinicians, clinical staff and management personnel practicing in one context; i.e., all participants from one clinic). Once selected, these interviews were set aside until model validation (herein, the model validation set).

These methods use tests which are not pass/fail per se. The first three tests are used to identify strengths and weaknesses of the model. The remaining four tests are used to identify strengths and weaknesses in the overall research approach. All these tests are performed as described in Appendix E and results of these tests are then presented along with model findings. It is then up to the audience to judge.

Shared Mental Model Saturation (SMM-S) is proposed here to have been reached when it has been demonstrated that (1) the shared mental model appears to describe the system

structure underlying the problem statement and (2) the addition of one more data source from the saturation reserve is not likely to modify the existing shared mental model (in the case of this dissertation, this would be another interview or clinic set of interviews).

85 The process of CLD Combination draws on the bank of CLDs representing individuals’ mental models as interpreted via Purposive Text Analysis. SMM-S goes one step back to check the shared mental model against the individual clinic CLDs. In so doing, it mitigates against the risk that essential elements were lost in the process of CLD Combination.

Conceptual Model Saturation (CM-S)is proposed here to have been reached when it has been demonstrated that the addition of one interview is not likely to modify the Shared Mental Model²⁴. I start with the existing Shared Mental Model (that was just verified via SMM-S).

That model is revised as needed during CM-S, until the addition of one more interview is not likely to modify it. The method involves coding a new set of interviews from scratch (the model validation set of interviews). This mental data is rigorously analyzed to validate the structure of the Shared Mental Model. At this point, once the CM-S Test is passed, the CLD is referred to as the Conceptual Model.

This method provides a formal way of reflecting on the conceptual model. In so doing, there is the recognition that even researchers’ mental models of what they study are hard to change.

This process helps to surface researcher assumptions and to allow participants’ views to question existing assumptions including on important concepts such as model structure, system boundaries and parameters.

Simulation Model Saturation (SIM-S) is proposed here to have beenreached when it has been demonstrated that the simulation model does not contain radical departures from the

participants’ interviews. Participant perspectives are accessed in the results of CM-S and used here to validate the structure and behavior of the simulation model.

This method provides a formal way of reflecting on the simulation model (e.g., its purpose, structure, parameters and behavior) by way of comparison with the problem definition, dynamic hypothesis and participant perspectives. For structure, this reflection involves exploring the limitations and boundaries of the model; the limitations being concepts in quotations which were simplified in the model and the boundaries being concepts which were left out of the model, treated as outside the system, as constants, or as aggregated concepts.

For behavior, this reflection includes exploring how the model is able to generate the scenarios described in participants’ interviews. In so doing, this method checks that the large number of elements created in translating the conceptual model into the simulation model is consistent with the way participants perceive the workings of the system.

24 This term is now capitalized since the shared mental model has been validated via the SMM-S test, producing SMM2. SMM2 is being validated in this test.

86 System Dynamics Saturation (SD-S) is proposed here to have been reached when the extent to which the validation set interviews verify that the target group sees SDM as a useful way of addressing the issue has been demonstrated (CptV4 - culture). This is demonstrated by rigorously analyzing relevant segments of the coded transcripts generated in CM-S, segments which, demonstrate participant perceptions of issues which SDM is used to study (e.g., mental models, systems with causal structures which include significant time delays, feedback loops and non-linearity).

This test verifies that participants’ mental models contain causality and delays and that system structure is perceived to be causing their problem. Furthermore, this test verifies that these participants understand that they have mental models as characterized in SDM, that these models are important, that these models can (and indeed must) change, and that doing so involves emotional engagement. In verifying these things, this test demonstrates the extent to which participants are likely to feel that, when presented with it, SDM research is appropriate and acceptable even though they were not directly exposed to it prior to their interview.

Methods Suitability is proposed here to be determined by reflecting on the methods used in the research. This test relates to both the modeling process (i.e., how diverse understandings are brought together) and the modeling language(s) (what sorts of models are used to represent those understandings) used in the research project. In so doing, the researcher provides a documentation of the trade-offs considered in the project’s overall methodological approach and justifies the researcher’s reasoning, exposing the methodological validity claims to evaluation by others.

Data Suitability is proposed here to be determined by reflecting on the three types of data that are possible, the data availability, and their limitations. This test relates both to model documentation and model validation. It should be started prior to model development and updated during model development. In so doing, the researcher provides a documentation of the data which were used in model development and validation, exposing the data validity claims to evaluation by others.

Stakeholder Dialogue Suitability is proposed here to be determined by reflecting on the research project and the extent to which stakeholders have been engaged. In so doing, the researcher provides a documentation of the project’s approach to stakeholder dialogue and justifies the researcher’s reasoning, exposing the validation claims using evidence from stakeholder dialogue to evaluation by others.

87 Table 2.11 identifies the type(s) of validity that I have found each method to assess.

Table 2.11 Validity Types Ascribed to Each Newly-developed Validation Method

Types of validity

CptV1 CptV2 CptV3 CptV4 FV1 FV2 FV3 EV1 EV2 DV1 DV2 DV3

Shared Mental Model Saturation (SMM-S) x x x x x System Dynamics Saturation (SD-S) x Conceptual Model Saturation (CM-S) x x x x x Simulation Model Saturation (SIM-S) x x x x x x x Stakeholder Dialogue Suitability x x x x x x x x

Methods Suitability x x

Data Suitability x x x

(CptV) Conceptual Validity, (FV) Formulational Validity 1-3, (EV) Experimental Validity 1-2, (DV) Data Validity. See Lane[105] for more information

Each cell marked with an “x” in Table 2.11 above corresponds to one row in Table 2.10 below.

In Table 2.12, the first column lists the validity type, defined in Appendix D. The second column identifies the method. The third column provides the question(s) that each newly-developed method contributes to addressing for each particular type of validity. The final column identifies the data that are compared in addressing the question(s) including the version of the model and the data source(s).

88 Table 2.12 Questions that Each Newly-developed Validation Method Addresses, Sorted by Validity Type

Validity Type Method Question Data Compared

CptV1 Variables &

boundaries

SMM-S Are the variables and boundaries of the shared mental model well-developed and validated?

SMM1 compared to the problem statement and CLDs at one level of aggregation lower than SMM1

CM-S Are the variables and boundaries of the Shared Mental Model well-developed and validated?

SMM2 compared to the problem statement and model validation set interviews

SIM-S Are the variables and boundaries of the simulation model well-developed and

validated? SIM1 compared to the model validation interviews

Stakeholder

Dialogue Are the variables and boundaries of the models well-developed and validated? Models compared to stakeholder perspectives

CptV2 Links

SMM-S Are the relationships in the shared mental model well-established and validated? SMM1 compared to the CLDs at one level of aggregation lower than SMM1

CM-S Are the relationships in the Shared Mental Model well-established and validated? SMM2 compared to the model validation set interviews SIM-S Are the relationships in the simulation model well-established and validated? SIM1 compared to the model validation set interviews Stakeholder

Dialogue Are the relationships in the models well-established and validated? Models compared to stakeholder perspectives CptV3

Saturation

SMM-S Have new and relevant data regarding the elements in the shared mental model ceased to emerge?

SMM1 compared to the CLDs at one level of aggregation lower than SMM1

CM-S Have new and relevant data regarding the elements in the Shared Mental Model

ceased to emerge? SMM2 compared to the model validation set interviews

CptV4 Culture

SMM-S Are bounded rationality and cultural acceptability evident in the relationships? SMM1 compared to the CLDs at one level of aggregation lower than SMM1

CM-S Are bounded rationality and cultural acceptability evident in the relationships? SMM2 compared to the model validation set interviews SIM-S Are bounded rationality and cultural acceptability evident in the simulation model? SIM1 compared to the model validation set interviews SD-S Does the target group see SDM research as a useful way of addressing the issue? The research project undertaken thus far considered

through the lens of the model validation set interviews Stakeholder

Dialogue

Do stakeholders see SDM research as a useful way of addressing the issue? Are

bounded rationality and cultural acceptability evident in the models? Models compared to stakeholder perspectives Methods

Suitability

Does the modeling process address cultural acceptability and cognitive limitations

and use a non-coercive approach? Reflection

89

Validity Type Method Question Data Compared

FV1 Language limitations

SMM-S Do the relationships in the shared mental model distort aspects of the clinic CLDs? SMM1 compared to the CLDs at one level of aggregation lower than SMM1

CM-S Do the relationships in the Shared Mental Model distort aspects of the model

validation set participants’ statements? SMM2 compared to the model validation set interviews

SIM-S

Have important elements been omitted or distorted because of the procedures used in the creation of the simulation model (i.e., "the discipline of the programming language"[105] (p. 120))?

SIM1 compared to the model validation set interviews Stakeholder

Dialogue

Do the relationships in the model distort aspects as they are conceptualized by

stakeholders? Models compared to stakeholder perspectives

Methods Suitability

Have important elements or relationships been omitted or distorted because of the

procedures used in the model development process? Reflection

FV2 Conceptual equals simulation

SIM-S How consistent is the simulation model with the structure and behavior that it was meant to represent?

SIM1 compared to the problem statement and dynamic hypothesis

Stakeholder Dialogue

How consistent is the model with the structure and behavior that it is meant to

represent? Models compared to stakeholder perspectives

FV3 SDM guidelines

Stakeholder Dialogue

How well do the models conform to SDM guidelines? (e.g., CLD grammar, stock and

flow diagram and model grammar) Models compared to stakeholder perspectives

EV1 Structural design

SIM-S Do experiments in the simulation model challenge or support its structure? SIM1 compared the model validation set interviews Stakeholder

Dialogue Do experiments in the simulation model challenge or support its structure? Models compared to stakeholder perspectives EV2 Insights

gained

Stakeholder Dialogue

Are the model's analytical insights at an acceptable level of quality (i.e., usefulness,

accuracy, robustness)? (discuss with stakeholders) Models compared to stakeholder perspectives DV1

Mental data

Data Suitability

Are the mental data upon which the model is based used within the scope of their

limitations? Reflection

DV2 Written data

Data Suitability

Are the written data upon which the model is based used within the scope of their

limitations? Reflection

DV3

Numerical data

SIM-S Is the numerical data for use in the simulation model reliable/appropriate, available

and sufficient? SIM1 compared the model validation set interviews

Data Suitability

Are the numerical data upon which the model is based used within the scope of

their limitations? Reflection

(CptV) Conceptual Validity, (FV) Formulational Validity 1-3, (EV) Experimental Validity 1-2, (DV) Data Validity. Please see Lane[105] for more information.

90 The proposed model validation methods described here correspond to tests in the standard SDM method, as shown in Table 2.13 below. From right to left, the table presents: newly-developed validation methods, their foci, the standard method tests that they address, and the purposes to which the tests contribute. This table presents the new validation methods in terms of the tests to which they correspond. This presentation is based on combining two tables above (Table 2.9 and Table 2.11).

Table 2.13 New Formal Validation Methods in Their Context

Purpose of validation test

Name of test

Focus of test New Validation Method

Structure Behavior Culture Shared Mental Model Saturation System Dynamics Saturation Conceptual Model Saturation Simulation Model Saturation Stakeholder Dialogue (Suitability) Methods Suitability Data Suitability

Relevance to and

For Stakeholder Dialogue Suitability, suitability is in parentheses because, while Stakeholder Dialogue is not a new validation method, Stakeholder Dialogue Suitability is a new method. It is the Stakeholder Dialogue that verifies each of these tests and Stakeholder Dialogue Suitability that verifies that appropriate stakeholder groups were engaged appropriately.

91

2.4 METHODS

This section summarizes the standard and newly-developed SDM methods as they were used in this dissertation. Table 2.14 below orients the reader to the sub-sections that follow: the four phases and two cross-cutting steps. As the research progressed through the phases, new model iterations were developed. This table also lists the model development and model validation methods as well as model iterations and how they correspond to the various phases of this work. This table is based on Table 2.1 and Table 2.3 presented earlier. The purpose of this table is to include the corresponding locations of the methods and results sections.

In the sections below, Section 2.4.1 presents a brief description of the iterative improvement of the problem definition and dynamic hypothesis. Sections 2.4.2 to 2.4.4 and 2.4.6 describe the data collection and analysis methods employed during phases one through four. Section 2.4.5 details the iterative process of model validation implemented across these research phases.

92 Table 2.14 Methods Summary, Model Iterations, Validation Methods & Locations in the Thesis

Study

Cross-Phase Step: Iterative Improvement of Problem Statement & Dynamic Hypothesis

Scoping

Cessation (Section 2.4.5.1 and 3.5.1) Data Suitability* (Section 2.4.5.2 and 3.5.2) Methods Suitability* (Section 2.4.5.3 and 3.5.3) SD-S* (Section 2.4.5.7 and 3.5.6) Stakeholder Suitability* (Section 2.4.5.8 and 3.5.7) Phase 1

* = Newly-developed methods for SDM, proposed in this dissertation.

93 2.4.1 CROSS-PHASE STEP - ITERATIVE IMPROVEMENT OF THE PROBLEM

DEFINITION & DYNAMIC HYPOTHESIS

In the scoping study (Appendix C), a problem statement (i.e., purpose) was drafted based on preliminary qualitative and quantitative analyses. This problem statement was periodically updated throughout the model development and validation process. Questions considered along the way, to develop (and iteratively update) the problem definition, are listed in problem definition above. With each phase, additional results provided the opportunity for reflection and for the creation of a more explicit definition of the problem.

My dynamic hypothesis²⁵[113] (p. 86) was also updated along the way. All of the model development and model validation steps leading to the development of the Theoretical Model refined it.

In document The Journey toward the Patient-Centered Medical Home: A Grounded, Dynamic Theory of Primary Care Transformation (Page 84-94)