Data sources 1 Aim

The primary data source for Aim 1 came from a cross-sectional survey of 120 middle managers in 120 health centers (one middle manager per health center) representing 10 Midwestern and West Central states who participated in the HDC. The survey represents an ideal data source for the study, and the data are unique because they pertain to an innovation that was simultaneously implemented in multiple health care organizations. Studying

multiple organizations improves the generality of results; studying a single innovation permits the use of a single and consistent measure of implementation effectiveness. Further, the survey fulfills Weiner, Amick, and Lee’s (2008) criteria for measuring organizational readiness for change: The survey specifically focused on respondents’ attention to HDC implementation, aggregated individual middle managers’ appraisals of their health centers’ capabilities as a whole, and surveyed multiple organizations. Secondary data of this kind are scarce given the novelty of the research topic.

The survey was conducted between March and December 2004 by the National Opinion Research Center, the University of Chicago, and the MidWest Clinicians Network using the standards of Dillman’s Total Design Method (1978). Dillman’s Total Design Method involved identifying each aspect of the survey process that might affect the quality or quantity of responses and modifying the process accordingly. Specifically, the cost of completing the survey (i.e., respondents’ time, effort, emotional toll) was minimized to the

51

extent possible through careful survey design, and the benefits of responding to the survey were emphasized (i.e., by completing the survey, middle managers would contribute to research that has the potential to improve future HDC efforts). Telephone prompting, up to two additional survey mailings via express delivery, and letters of support from Bureau of Primary Health Care officials were used to increase response.

The HDC began in 1999, and the survey was conducted in 2004. As such, the survey allowed sufficient time for middle managers to decide how committed they were to HDC implementation and to act on this decision. The surveys were sent to 149 eligible middle managers. The final overall response rate was 81 percent (N = 120). The study period was calendar year 2003 through 2004 because survey questions specifically requested responses regarding this period. Surveys were also sent to CEOs (n = 103). I used CEOs’ responses to survey questions regarding organizational size and location in lieu of middle managers’ responses because CEOs were most likely to have accurate information regarding these measures. I accounted for common method variance associated with using middle managers’ responses alone to construct all other variables. This method is described in detail in section 4.5.1 below.

4.3.2. Aim 2

Semi-structured interviews were conducted with 16 of the middle managers included in Aim 1. The interviews shed light on concepts that cannot be captured in surveys and allowed me to explore the underlying reasons for relationships identified in Aim 1 analyses. To ensure variation in the key constructs, I selected middle managers based on their level of commitment to HDC implementation and the effectiveness of HDC implementation in their

52

health centers using the following method: (1) I used a publicly available list of 149 health centers that participated in the HDC to call middle managers. I attempted to contact middle managers who no longer worked in the health centers using forwarding information provided by the health center. (2) I requested middle managers’ consent to identify their Aim 1 survey responses to determine whether they could be classified into one of the following quadrants: Figure 4.3. Criteria for inclusion in qualitative study

Commitment to HDC implementation High (≥3.33) Low (≤2.33) Implementation effectiveness High (≥6.86) 4 4 Low (≤5) 4 4

I was unable to reach 89 of the middle managers, three declined to participate in the

qualitative study, and fifty-seven middle managers consented to participate in the qualitative study. Of the 57 middle managers who consented, the sixteen with the most extreme

implementation effectiveness and commitment to HDC implementation scores were selected for participation in the qualitative study: Thresholds for “high” and “low” commitment to HDC implementation and implementation effectiveness were determined based on the distribution of responses from the 57 middle managers who consented. Implementation effectiveness was classified as “high” for middle managers who scored in the top twenty-fifth percentile and “low” for those who scored in the bottom twenty-fifth percentile. For example, the maximum score for implementation effectiveness was 11, and the maximum score for commitment to HDC implementation was 5; middle managers in the high implementation effectiveness/high commitment to HDC implementation quadrant scored a minimum of 6.86 for implementation effectiveness and 3.33 for commitment to HDC implementation (see figure 4.3). Two of the middle managers selected for participation in the qualitative study no

53

longer worked in the health centers in which they completed Aim 1 surveys, but I reminded them to respond to interview questions based on their experience in the health center in which they implemented the HDC. Indeed, I reminded all of the middle managers whom I interviewed to respond to interview questions based on their experience in the HDC in the study period (calendar year 2003 through 2004) to the extent possible.

Interviewing multiple middle managers allowed me to identify patterns that are robust against idiosyncratic features of any given middle manager’s experience, thereby addressing selection and selection-treatment interaction biases (Cook & Campbell, 1979). Middle managers who could not be classified into one of the above quadrants were excluded from participation in the qualitative study. The consent (Appendix 1) and interview guide (Appendix 2) were approved by the Institutional Review Board at the University of North Carolina at Chapel Hill.

One- to two-hour interviews were conducted via telephone and were scheduled based on the participant’s convenience. I conducted interviews in a private office with a closed door at the University of North Carolina at Chapel Hill. Interviews were recorded and transcribed to enhance data reliability. My interviewing training and experience minimized middle

managers’ evaluation apprehension and my own expectancies as an interviewer (Cook & Campbell, 1979).

I followed an interview guide (Appendix 2) based on the theoretical framework (figure 3). Using the interview guide helped to avoid instrumentation bias (Cook & Campbell, 1979). Basing the interview guide on the theoretical model reduced the bias associated with

inadequate preoperational explication of constructs that may threaten the validity of study findings (Cook & Campbell, 1979). Including several questions in the interview guide

54

targeting the same construct addressed the threat of mono-operational bias (Cook &

Campbell, 1979). Prior to conducting interviews with the sixteen middle managers selected for inclusion in the qualitative study, I conducted pilot interviews with two middle managers in health care organizations that are unaffiliated with the HDC. My observations and

feedback from pilot interview subjects were used to refine the interview guide. Pilot

interview data were not included in final analyses. During HDC middle manager interviews, questions were amended and supplemented based on middle managers’ responses.

4.4 Measurement