PHASE 1 – MENTAL MODEL ELICITATION

Purposive sampling[170, 171]²⁶ was used to select five primary care clinics deemed to be representative (in their context) of the larger organization (the 10 HSDO clinics) – interviews from these clinics are the model development set. Purposive Text Analysis was performed on all clinician and MA interviews at these five clinics (n=20 interviews). Interviews from a sixth clinic (n=10 interviews) were set aside for validation once the model was developed – these are the model validation set. The remaining HSDO interviews were set aside for saturation reserve: three clinics for saturation reserve associated with model development and one clinic for saturation reserve associated with validation. Thus, in total, interviews from 80% of clinics were designated for model development and interviews from 20% of clinics were designated for validation. These designations are presented in Table 2.15 below.

Table 2.15 Designation of Clinics for Model Development & Validation

Analysis Set

25 A dynamic hypothesis explains the problem behavior as it is endogenously produced via feedback structures among key system variables. See Section 2.2.2.1.2 for more on dynamic hypotheses.

26 To be clear, there is no methodological relationship between purposive sampling and Purposive Text Analysis. Purposive sampling was used two times: first, (by HSDO managers) selecting respondents and second, (by me) segmenting the mental database (as in Table 2.15).

94 Clinics were selected based on their context, with the goal of maximizing the variation in contextual factors. HSDO management personnel characterized each clinic’s context.

Descriptions of the five selected clinics are presented in Table 2.16 below.

Table 2.16 Clinic Characteristics

Clinic A Clinic B Clinic C Clinic D Clinic E

Urbanization Suburban Urban Suburban Urban Suburban

Distance from UHC Mid-range Close Far Mid-range Far

Patient Diversity High Low Moderate Moderate Moderate

Training Clinic (residency) Rotation Continuity Rotation Rotation Rotation MD Availability (most) Full Time Part Time Full Time Mix Full Time

Ancillary Specialty Many None Many Many A few

Market Competition High Moderate Low High Moderate

Clinicians interviewed practicing in these clinics span all the practice clusters (described in the Appendix C Scoping Study). Therefore, it is assumed that using these clinics increases the likelihood that the analysis captures the range of perspectives held among front-line employees within the HSDO.

Purposive Text Analysis is used to develop CLDs from these interviews. Whereas interviews were used for thematic coding in the scoping study, they are now used for model

development. To satisfy the assumptions of purposive text, the verbatim transcripts of the interviews must represent:

Participants’ “sophisticated [or first-hand] knowledge of the system”[117]

Participants interviewed in the Mixed Methods Project were: one clinician and one medical assistant (herein, MA) from each care team in the HSDO’s 10 community clinics as well as relevant managers. The community clinics Quality Director as well as clinic center managers performed the selection using purposive sampling. Selection criteria were: 1) to have one clinician and one MA from each team 2) to capture a range of approaches to implementation and 3) to capture a range in the length of involvement in the

transformation process (e.g., initial implementation, introduction into a clinic mid-way through implementation).

Candid discussions, where participants “are not grand-standing or taking rhetorical positions that they do not believe in strongly”[117] (p. 314)

Interviews were performed in a private room at the respective clinic. Participants had the option to terminate the interview at any time. They signed a consent form to participate.

95 While the research team was embedded within the HSDO, I (the interviewer) was not. It was made clear to respondents that HSDO staff would not have access to respondent interviews and that their statements would not be presented in an identified manner at any time. The semi-structured interviews were gathered with ethical approval from the London School of Hygiene and Tropical Medicine as well as from the University of Utah.

By employing a systematic coding procedure that treats the data in a consistent manner, Purposive Text Analysis overcomes the temporal and spatial distance between data source and researcher which may introduce biases (i.e., the researcher’s own assumptions about the system investigated; e.g., variables considered important for one researcher might be ignored by another)[117].

Purposive Text Analysis is a method that was first developed and applied by Kim in a study that generated a stock and flow diagram from a series of verbatim transcripts from the US Federal Reserve Board’s Open Market Committee (an important U.S. policy-making group) meetings[117]. The stock and flow diagram represents the policy makers’ mental models communicated and shared during the meeting leading to their collective decision. When the researcher cannot verify the diagram with the original stakeholder, systematic coding and documenting allows the researcher to leave a documentation of data-to-diagram linkage and, where feasible, creates an opportunity for the diagram to be examined by others[117].

This dissertation uses this same method to generate CLDs. The coding procedure can be summarized as follows:

1. Define the problem of focus.

2. Select data segments within the problem boundary. Each data segment consists of one argument and its supporting rationales.

3. From each data segment, identify the cause variable, effect variable, and the polarity of the relationship.

4. Represent each causal relationship in a simple words-and-arrow diagram.

5. Collect and merge the words-and-arrow diagrams into a collective CLD. In doing so, collapse similar variables using a common variable name.

6. Assign unique identifiers to data segments and CLD variables and causal links. As the coding progresses, document the data segments where each CLD variable or causal link is elicited.

96 Purposive Text Analysis proceeded as follows: clinician and MA interview transcripts were coded to identify portions where structural relationships were discussed, focusing on statements describing a cause and effect relationship. Simultaneous to coding in computer-aided qualitative data analysis software (NVivo[176]) respondent mental models were visualized in system dynamics modeling software (Vensim[177]) as the coded causal relationships were then translated into a CLD.

An example illustrating how purposive text analysis was used is presented below in Figure 2.10. One MA describes one aspect of task shifting within her team as she describes whether she is allowed to remain in the examination room during patient visits. This figure presents the statement and shows how it was subdivided into four causality arguments, and a corresponding words and arrow diagram, in the words of this MA.

Once individual CLDs were completed, variable names were standardized across all CLDs. This was achieved by entering variable names from one individual CLD into MS Excel and then updating that list (by adding and/or modifying the variable names) as additional CLDs were reviewed. The final set of variable names was then used to create a standardized CLD for each individual.

These CLDs were then pruned. Pruning elicits the factors that contribute most to system behavior over time. As per SDM theory, accumulation in stocks is what people can see, and it happens via delays and feedback structures[178]. Yearworth[116] recommends retaining only linkages involving delays and/or loops of three or more links. In this step, pruning was relaxed to also retain loops of two links (even when they did not contain delays). In this dissertation, this is referred to as mild pruning.

97 Figure 2.10 The Purposive Text Analysis example of arguments 1 to 4

98 2.4.3 PHASE 2 – DEVELOP CONCEPTUAL MODEL

Preparing CLDs for CLD Combination involved: coding 20 interviews to create 20 corresponding CLDs and mildly pruning them into 20 individual CLDs (Phase 1).

In Phase 2, CLD Combination was implemented as individual CLDs were combined to develop a shared mental model. Specifically: the 20 mildly pruned individual CLDs are combined into 10 Team CLDs, which are further combined into 5 Clinic CLDs, and then finally combined and fully pruned into 1 shared mental model for all participants in the model development set (this model is called SMM1).

That shared mental model was then validated. Validation methods used were: SMM-S Test, CM-S Test, and Stakeholder Dialogue (see Section 2.4.5.4 for descriptions). After validation, the new model version was referred to as the Conceptual Model (also known as SMM3).