Leading for Quality in Healthcare: Development and Validation of a Competency Model

Development and Validation of a

Competency Model

Andrew Garman, PsyD, MS, CEO, National Center for Healthcare Leadership, and professor, health systems management, Rush University; and Linda Scribner, BA, CPHQ, director of quality and clinical outcomes management, Methodist Dallas Medical Center

E x E C u T I V E S u M M A R Y

Increased attention to healthcare quality and impending changes due to health reform are calling for healthcare leaders at all levels to strengthen their skills in leading quality improvement initiatives. To address this need, the National Asso-ciation for Healthcare Quality spearheaded the development and validation of a competency model to support healthcare leaders in assessing their strengths and planning appropriate steps for development. Initial development took place over the course of several days of meetings by an advisory panel of quality profession-als. The draft model was then validated via electronic survey of a national sample of 883 quality professionals. Follow-up analyses indicated that the model was content valid for each of the target samples and also distinguished differing levels of job scope and experience. The resulting model contains six domains spanning three organizational levels.

I N T R O D u C T I O N

As delivery of high-quality healthcare continues to grow more complex, so do the roles of the professionals lead-ing these efforts. Recent years have seen increased focus on the leadership ele-ments of the quality professionals’ roles; initiatives such as the Comprehensive Unit-Based Safety Program (CUSP; Pro-novost et al. 2005), crew resource man-agement, Lean Six Sigma, and Malcolm Baldrige emphasize the key elements of leadership and management needed for success (see Garman et al. 2011). Given the impending changes associated with the Affordable Care Act, leaders are likely to be charged with implementing these quality improvement initiatives within a context of increasing emphasis on resource efficiency. While the oppor-tunities to improve may be tremendous, threading the value needle will likely test the mettle of all leaders of quality efforts in the years to come.

In preparation for this new era for the quality professional, the National Association of Healthcare Quality (NAHQ) began an initiative to investi-gate the leadership development needs of the profession. Their efforts yielded a competency model that is specific to leadership in quality and holds implica-tions for professionals across the career path. Development and validation of this model are described in the follow-ing section.

M E T h O D

Development of the competency model began in June 2008. Members of the NAHQ board agreed to serve as the advisory panel for developing a leader-ship model and convened a two-day

series of meetings to develop the draft. The meetings proceeded through three phases: clarification of goals, definition of scope, and competency identifica-tion. In the competency identification phase, subcommittees were formed to represent the perspectives of present-state, future-present-state, and senior leadership. Using the Health Administrators Leader-ship model (HAL) (Garman, Tyler, and Darnall 2003) as their seed model, the subcommittees reviewed this and other published models against the goals and scope criteria developed during the phase one and two meetings. These sep-arate reviews were then compiled during a large group meeting to form the first draft. These results were summarized overnight and disseminated the follow-ing mornfollow-ing for further discussion. The revised model contained 21 competen-cies organized into six domains. This model was circulated to the commit-tee a third time several weeks after the original meeting, and feedback was col-lected via teleconference. At that point the board agreed by consensus that the model reflected the elements they believed would be essential to quality leadership moving forward.

With the draft of the model final-ized, the next step was to conduct a con-struct validation study. To accomplish this, an electronic survey was developed for distribution to quality professionals, using methods adapted from Williams and Crafts (1997). The survey asked respondents to review each of the com-petency descriptions and rate it accord-ing to level of perceived importance to their quality leadership role using a five-point scale of relative importance (1 = No importance, 5 = Extremely

a series of content validity ratios (CVR) (Lawshe 1975) were calculated. CVRs indicate the proportion of respondents who agree that a particular competency is very or extremely important. This analysis indicated that all competencies were above the recommended threshold value of 0.49 (range: 0.66 for “Finan-cial Acumen” to 0.97 for “Professional Ethics”).

Model Structure

The association between competen-cies and domains was examined using principal components factor analysis, a statistical technique that tests the extent to which items on a survey tend to be rated similarly by respondents (higher levels of association suggest that the items hang together more closely, form-ing associations). To test whether the domains originally defined would map to the competencies we originally speci-fied, the analysis was set up to fit the data to a six-domain solution. Results of this analysis, shown in Exhibit 2, supported the original structure for 17 of the 21 competencies and identified four areas in need of reconciliation. First, the systems thinking competency, which was originally in the organiza-tional awareness domain, loaded more heavily onto the fosters positive change domain. Second, the lifelong learning competency, originally in the profes-sionalism domain, loaded relatively equally onto the self-management and professionalism domains and slightly higher onto self-management. Third, the professional ethics competency, which was originally under the pro-fessionalism domain, loaded more strongly onto the self-management important). An open-ended question

followed each competency, to allow respondents to comment on clarity, word choice, or other concerns with the way the competency was defined. Respondents were also asked to answer a set of demographic questions concern-ing the nature of their positions and their institutional settings.

Invitations to complete the survey were sent to all members of NAHQ (n = 4,445), using a list maintained by this organization and a list of non-NAHQ members holding the Certified Professional in Healthcare Quality (CPHQ) credential (n = 2,704), which was provided by the Healthcare Quality Certification Board.

R E S u LT S

A total of 883 quality professionals responded to the survey, for a response rate of 12 percent. Respondent demo-graphics are provided in Exhibit 1. For respondent role, four categories were created to collapse roles to a manage-able number: senior executive leader-ship (CEO, president, chief of staff, chief or VP of medical affairs, chief nursing executive, chief operating or operations officer, chief financial officer); senior quality leadership (president or vice president for quality services, quality improvement, patient safety divisions/ departments, physician quality leader (department or service line); mid-level quality leadership (director or manager of quality, patient safety, and/or risk management); and direct contributor (quality coordinator, quality specialist, data analyst, data abstractor).

To determine the validity of the model and its component competencies,

domain. Fourth, drive for results, originally under the passion for positive change domain, loaded equally strongly onto two other domains: performance improvement and self-management. These four findings were vetted with the original advisory panel, who by consen-sus agreed that the survey results made conceptual sense and should be used to guide revision of the model. These final

revisions yielded the model shown in Exhibit 3.

Subgroup Analyses

Our next step was to develop a more refined understanding of how the competencies may differ in their rela-tive importance as a function of orga-nization setting, orgaorga-nization size, experience level, and position level. E x h I B I T 1

Respondent Demographics

N (%)

Professional

NAHQ member 254 (29)

Certified professional in Health Quality (CPHQ) 160 (18)

Both 469 (53)

Years working in quality

Less than 1 year 14 (2)

1–2 years 46 (5)

3–5 years 117 (13)

6–10 years 169 (19)

11–15 years 137 (16)

More than 15 years 400 (45)

Hospital bed size (hospital-based professionals) 604 (68)

Less than 50 61 (10) 50–99 53 (9) 100–199 110 (18) 200–299 102 (17) 300–399 83 (14) 400–499 46 (8) 500 or more 149 (25)

Job level (by categorizable role, respondents in hospital settings) 521 (59)

Senior executive leadership 14 (3)

Senior quality leadership 44 (8)

Mid-level quality leadership 251 (48)

Exh

IBIT

2

Factor L

oadings Based on a Principal Components Analysis for the Competencies on the Original Six Domains

Factors Competency Fosters P ositive Change Communicating Organizational Awareness Self-Management Future F ocus Per formance Improvement

Advocates for and adapts to change

0.73

0.22

Partners for change

0.66

0.27

Cultivates a quality-supportive climate

0.66

0.28

Systems thinking

0.58

0.39

Drives for results

0.41

0.40

0.41

Verbal communication skills

0.82

Written communication skills

0.75 0.23 0.26 Educating 0.22 0.58 0.34

Listening and receiving feedback

0.29

0.53

0.40

Strategic planning

0.83

Strategic thinking and alignment

0.23 0.82 Financial acumen 0.64 0.26 Professional ethics 0.23 0.75

Manages personal limits

0.62

0.38

Resilience and self-restraint

0.31 0.62 0.31 Lifelong learning 0.25 0.24 0.45 0.44 Future focus 0.32 0.68 Consumer advocacy 0.30 0.67 Managing data 0.81

Analytic thinking /decision making

0.75

Develops a knowledge-rich environment

0.25

0.26

0.39

0.53

Note:

E x h I B I T 3

Leadership in Quality: Final Model Structure and Associations with Respondent Demographics Significant Associations Domains/

Competencies Experience Job Level

Organization Size

Interaction Terms

I. Fosters positive change ** ** **

Advocates and adapts to change

Partners for change ** **

Cultivates a quality-supportive climate **

Drives for results ** ** **

II. Communicating

Verbal communication skills Written communication skills

Listening and receiving feedback ** Educating

III. Organizational awareness ** ** **

Strategic planning ** **

Strategic thinking and alignment ** ** ** **

Financial acumen **

Systems thinking ** ** **

IV. Self-management **

Professional ethics Manages personal limits Resilience & self-restraint

V. Professionalism / Professional values Consumer advocacy

Future focus ** **

Lifelong learning

VI. Performance improvement Managing data

Analytic thinking / knowledge-based decision-making

** ** **

Develops a knowledge-rich environment

Note: Complete and current descriptions of each of the competencies are available online: www.nahq.org/membership/

leadership/devmodel.html

For organization setting, although the survey contained a question about primary practice setting, within-category response counts were too small for meaningful separate analysis of these results. For this reason, we focused only on respondents who provided a bed count for their organizations in the demographics section of the sur-vey. A total of 267 (31%) respondents indicated “not applicable” and were categorized as nonhospital settings. The remaining 69 percent were cat-egorized as hospital settings. A series of statistical analyses (ANOVAs) was conducted to determine whether there were differences between hospital and nonhospital groups on the importance of the competencies and competency domains. These analyses yielded no sta-tistically significant differences between these groups, providing some evidence for generalizability across organiza-tional settings.

To examine the influence of orga-nization size and experience level, we followed a similar approach. For organi-zation size, we focused only on hospi-tal settings and only considered those respondents who reported bed counts associated with their organizations. These analyses suggested significant differences as a function of organization size for four competencies within the organizational awareness and fosters positive change domains (strategic planning, strategic thinking and align-ment, partners for change, and drives for results), as shown in Exhibit 3. Level of experience was also significantly asso-ciated with five competencies (strate-gic thinking and alignment, financial acumen, systems thinking, analytic

thinking/knowledge-based decision making, and listening and receiving feedback). For all significant effects, the direction of the effect was for larger organizations and higher experience lev-els to be associated with higher levlev-els of importance for the competency/domain.

To analyze competencies accord-ing to position level, subgroups were formed according to responses to the demographic question “The primary functional role (not necessarily the title) of my current position related to qual-ity or patient safety is: ___.” Responses were categorized into the four role levels described previously (senior execu-tive leadership, senior quality leader-ship, mid-level quality leaderleader-ship, and direct contributor). For the competency analysis, respondents were selected only if they reported a bed count in the demographic section (indicating they worked in a hospital setting). This analysis yielded statistically significant associations for 5 of the 21 competen-cies: one from the revised organizational awareness domain, two from the revised fosters positive change domain, and one each from the performance improve-ment and professionalism/professional values domains. Those five competen-cies were strategic thinking and align-ment, drive for results, systems thinking, analytic thinking/knowledge-based decision making, and future focus. In all cases, higher organizational level was associated with greater perceived impor-tance of the competency, suggesting that these five competencies would be particularly appropriate foci for leader-ship development programs that center on career progression to higher organi-zational levels.

D I S C u S S I O N A N D C O N C L u S I O N

Several important limitations of this work are important to keep in mind when interpreting the results. First, although the sample size obtained for the content validity study was large, the small response rate means the responses could differ in important ways from the population from which they were drawn. Second, because the focus of this project was on leadership in quality, the model should not be considered a com-plete description of all critical elements of a quality professional’s job. Effective-ness in a quality leadership role will typically also require mastery of a con-siderable knowledge base and technical competencies that are beyond the scope of the present effort. Third, competency models are only as useful as the leader-ship-development efforts they support. When they are incorporated into devel-opment programs based on sound adult learning principles, competency models can be powerful facilitators of indi-vidual change, but by themselves these models do little to help people develop. Last, the effectiveness of even the most skilled quality professionals ultimately will be bounded by the level of collabo-ration they experience from the other leaders and clinicians they work with. Attention to leadership as a team sport and the need for a continuous learning focus will continue to be essential in supporting quality improvement gains in healthcare organizations.

These limitations notwithstanding, results of this study suggest that the Quality Leadership model and its com-ponent competencies can be considered content-valid descriptions of the areas

quality leaders need to master to be effective in their roles and thus represent a useful model for leadership develop-ment within the quality profession. This appears to hold true regardless of setting (hospital or other) or of other profes-sional characteristics such as experience, job level, or organization size. Addition-ally, the relative importance of a num-ber of the competencies and domains increased as a function of higher posi-tion level and larger, more complex organizational settings, suggesting spe-cific areas that may be particularly useful foci for leadership-development efforts.

Based on the patterns of results described previously, we constructed the graphic model depicted in Exhibit 4. The pyramid shape captures the hier-archy of the competencies according to the analyses described in the prior sections. In particular, professional-ism/professional values appears as the base to indicate the critical role that this domain (and professional ethics in particular) plays across all quality posi-tions. The next level contains founda-tional skills that, while still relevant to all positions, start to look different at different organizational levels. The top level, which contains organizational awareness and fostering positive change, is the most closely associated with higher organizational levels.

Taken together, these competen-cies illustrate several important ways in which successful leadership of quality efforts may differ from a more general definition of leadership effectiveness. In comparison to other widely recog-nized healthcare leadership competency models (for a review, see Garman and Johnson 2006), there appears to be a

greater relative emphasis on developing and maintaining a culture of continuous process improvement (partnering for change, cultivating a quality-supportive climate) and the competencies necessary for the analytic work associated with data-driven decision making (manag-ing data, analytic think(manag-ing, develop(manag-ing a knowledge-rich environment). It is perhaps not surprising that both empha-ses map closely to areas focused on by the Malcolm Baldrige award criteria (National Institute of Standards and Technology 2010).

Conversely, competencies with an external focus (e.g., community relations, board relations) or longer-term focus (e.g., fundraising, talent development), or a focus on day-to-day operations (e.g., human resource management, information technology management) are absent or deempha-sized. So the model does appear to trade

a narrower focus for greater depth of focus on quality leadership specifically.

In terms of developing current and future quality leaders, competency mod-els can support these efforts in a variety of ways. Competency models are par-ticularly helpful for identifying areas in which further development may provide the greatest relative payoff. For example, the competency definitions can be used as a template for creating developmen-tal 360-degree feedback programs in preparation for on-the-job develop-ment. Similarly, the model can be used as a framework for a development planning discussion between leaders and their direct reports, using templates such as those provided by Garman and Dye (2009). Models that are adopted by a hospital or health system’s senior leaders can be particularly useful for aligning the developmental agenda of the entire organization. For example, E x h I B I T 4

an employee and organizational devel-opment department can cross-walk an organization-endorsed competency model against internally provided train-ing and development programs to better communicate these opportunities to employees and to identify potential gaps in their offerings. Additionally, assess-ments such as 360-degree feedback can be aggregated to the organization level to identify and begin to address broader skill gaps.

In conclusion, this model appears relevant and useful for leaders and edu-cators interested in developing capacity in the area of quality leadership. In sup-port of open dissemination, and in the spirit of viewing quality as everyone’s responsibility, all competency descrip-tions and supplementary material have been made available on the NAHQ website. Our hope is that the availability of this model will help organizations attend to the leadership development needs of their quality professionals.

R E F E R E N C E S

Garman, A. N., and C. F. Dye. 2009. The Healthcare C-Suite: Leadership Development at the Top. Chicago: Health Administration Press.

Garman, A. N., and M. P. Johnson. 2006.

“Leadership Competencies: An Introduc-tion.” Journal of Healthcare Management 51 (1): 13–17.

Garman, A. N., A. S. McAlearney, J. Robbins, P. Song, M. McHugh, and M. I. Harrison. 2011. “High-Performance Work Systems in Health Care Management, Part 1: Development of an Evidence-Informed Model.” Health Care Management Review 36 (3): 201–13. http://journals.lww.com/ hcmrjournal/Fulltext/2011/07000/High_ performance_work_systems_in_health_ care.1.aspx#.

Garman, A. N., J. L. Tyler, and J. S. Darnall. 2003. “Development and Validation of a 360-Degree Feedback Instrument for Healthcare Administrators.” Journal of Healthcare Management 49 (5): 311–25. Lawshe, C. H. 1975. “A Quantitative Approach

to Content Validity.” Personnel Psychology 38 (4): 563–75.

National Institute of Standards and Technol-ogy. 2010. 2010–2011 Health Care Criteria for Performance Excellence. Accessed May 30, 2011. www.nist.gov/baldrige/pub lications/upload/2011_2012_Health_ Care_Criteria.pdf.

Pronovost, P., B. Weast, B. Rosenstein, J. B. Sex-ton, C. G. Holzmueller, L. Paine, R. Davis, and R. Hava. 2005. “Implementing and Validating a Comprehensive Unit-Based Safety Program.” Journal of Patient Safety 1 (1): 33–40.

Williams, K. M., and J. L. Crafts. 1997. “Induc-tive Job Analysis: The Job/Task Inventory Method.” In Applied Measurement Methods in Industrial Psychology, edited by D. L. Whetzel and G. R. Wheaton, 51–88. Palo Alto, CA: Davies-Black Publishing.

P R A C T I T I O N E R A P P L I C A T I O N

Janet Holdych, PharmD, CPHQ, director of quality, Catholic Healthcare West, San Francisco, California

T

Chasm called out the US healthcare delivery system for not consistently pro-viding high quality care to all people. The report concluded that fundamental changes were needed in order to provide safe, effective, patient-centered, timely, efficient, and equitable care (IOM 2001). Operationalizing this vision has led to

a decade of change in the healthcare field as healthcare providers, payers, regula-tors, and policy makers work toward improving the quality of care and reducing costs through an increased focus on applying evidence to healthcare, implementing information technology, and aligning payment policies with quality improvement (e.g., pay for reporting, value-based purchasing). Healthcare organizations have felt the impact through increased data collection and reporting requirements and the implementation of healthcare quality improvement initiatives. This increased atten-tion to hospital quality performance has driven organizaatten-tions to make certain their leaders have the skills and tools they need to participate in or lead quality of care improvement work.

The National Association for Healthcare Quality (NAHQ) competency model that Garman and Scribner describe is a useful tool that supports healthcare leaders in assessing their ability to lead and drive performance improvement. Fortunately, Cath-olic Healthcare West was an early adopter of the model; we used the NAHQ compe-tencies as a starting point toward the development of our organizationwide Quality Leader competency model. Its development was deemed a priority in response to the increased focus on hospital quality and what appeared to be a shortage of qualified quality leaders to fill vacant leadership positions in our hospitals. The Quality Leader competency model follows in the footsteps of other competency models our internal Organizational Development department established for our organization’s C-suite positions. We recognized that our success to achieving our mission of providing high-quality, affordable healthcare to the communities we serve was dependent on our people. To best leverage our talent, we needed to clearly understand the specific behaviors, knowledge, competencies, and skills that these leaders must demonstrate in order to be effective in their given role.

We became aware of the draft NAHQ competencies when the survey was released in 2008. Collaboration with NAHQ provided us with a starting point for our compe-tency model and allowed us the opportunity to provide additional responses to their survey and vetting process. Our competency model was designed to assist our orga-nization in developing incumbents’ skills, identifying top talent for succession, and assessing the organization’s overall bench strength. More important, it allowed us to develop competencies needed as we progressed toward leadership development.

As part of the process for developing our competency model, we solicited partici-pation from job experts (quality leaders within our organization) and executive team members. The process included research and data collection, focus group sessions, drafted competency models for functional and job-specific levels, and validation with job experts, stakeholders, and the executive team. The validation process was thor-ough, and as a result, we have a fully integrated model. Supplementary documents to support the model were also developed, including a quality leader job descrip-tion; a quality leader interview guide that contains behavioral interview questions; and a development catalogue consisting of courses, activities, books and periodicals, resources, professional organizations, and other development aids that are applicable to each of the defined competencies. Keeping competencies as the focus of these

tools helps ensure that individuals are hired against the same criteria they will be developing and measured on in their jobs. In addition, quality leaders within our organization complete an individual development plan annually that the executive they report to reviews and the system office uses to identify potential development activities for the upcoming year.

Our end product is a competency model that defines the competencies and attributes that the hospital quality leader must demonstrate in order to be effective in their role. We believe that effective leadership and adequate resources in the quality department can lead to an effective program that supports and accelerates progress toward improved patient care. Garman and Scribner conclude that they hope their competency model will help organizations attend to the leadership development needs of their quality professionals, and we can attest that it already has.

R E F E R E N C E

Institute of Medicine (IOM). 2001. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academies Press.