Methodological Issues in Comparing Hospital Performance: Measures, Risk Adjustment, and Public Reporting

(1)

Methodological Issues in

Comparing Hospital

Performance: Measures, Risk

Adjustment,

and Public Reporting

Harlan M. Krumholz, MD

Yale University School of Medicine July 31, 2015

(2)

(3)

(4)

Comparison of health care spending in Taiwan and

the World’s major countries

(5)

NHI public satisfaction

ratings 1995-2014

(6)

(7)

(8)

(9)

(10)

It would be great if Prof. Krumholz

could share the US experiences in

developing hospital report cards

regarding:

(11)

1. How to select appropriate

indicators/performance metrics to

better represent the performance of

hospitals.

(12)

(13)

• Published “Evaluating the Quality of Medical

Care” in 1966

• Separates quality measures into 3 categories: • Structure

• Process • Outcome

(14)

Types of Quality Measures Structure Assumes “good medical care will follow” Process Outcome

(15)

Types of Quality Measures Structure Assumes “good medical care will follow” Process Assesses “whether ‘good’ medical care has been applied” Outcome

(16)

Structure Measures – Pros & Cons

Pros

• Measure important foundational concepts • Easy to measure and compare

(17)

Structure Measures – Pros & Cons

Pros

• Measure important foundational concepts • Easy to measure and compare

Cons

• Relationship to quality is assumed

• Improving performance may not improve outcomes

(18)

Structure Measures

Define the presence or absence of specific care resources Include:

• Facilities and equipment • Number of beds

• EHR capabilities

• Professional and organizational resources • Licensure and certification

• Joint Commission accreditation • Policies and procedures

(19)

Process Measures

Assess whether or not specific care is performed Include:

• Clinical and preventive care • CMS “Core” measures

(20)

Process Measures

Examples

Applicable to hospitals:

• Aspirin administration to heart attack patients • Timing of antibiotic initiation for pneumonia

(21)

Process Measures – Pros & Cons

Pros

• Evolve from evidence-based practice guidelines • Combine consensus-based information with clear

(22)

Process Measures – Pros & Cons

Pros

• Evolve from evidence-based practice guidelines • Combine consensus-based information with clear

instructions for improving care

Cons

• Can be burdensome to collect

• Exclude some patients from measurement • May not correlate well with outcomes

(23)

Outcome Measures

Assess what actually happens to the patient • Survival

• Unintended treatment effects • Symptom relief

Include:

• Acute clinical events • Health status

(24)

Outcome Measures

Examples

Applicable to hospitals:

• 30-day all-cause mortality and readmission following AMI, heart failure and pneumonia

(25)

Outcome Measures – Pros & Cons

Pros

(26)

Outcome Measures – Pros & Cons

Pros

• Measure what matters most to patients

Cons

• Must be influenced by care

• Requires adequate risk adjustment to account for differences in case mix

• Influenced by many factors • More difficult to measure

(27)

What makes a good outcome measure?  Well thought out measure concept

 Considering potential applications in design

(28)

A well thought out

measure concept

 Addresses measurement and performance gaps  Measurement gap: Not currently measured

 Performance gap: variation in performance not explained by variation in patient case mix

(29)

Considering potential

applications

in design  Clinical care

 Quality improvement  Accountability

(30)

Measure

specifications

 Population to be measured (cohort)

 Result to be measured (outcome)

 Outcome attribution (who is responsible)

(31)

Measure

specifications

 Population to be measured (cohort)

 Result to be measured (outcome of interest)

(32)

Measure

specifications

 Population to be measured (cohort)  Result to be measured (outcome)

(33)

Measure

specifications

 Population to be measured (cohort)  Result to be measured (outcome)

(34)

Measure Cohort

 Reliably identifiable  Clinically coherent

(35)

Measure Outcome

 Reliably identifiable

(36)

Outcome attribution

 Ownership of care

(37)

2. The mechanism to select risk factors

for each outcome indicators,

especially to get/build consensus from

medical professionals.

(38)

Outcome

Hospital

Care

Patient Status on Presentation

Other

Effects

(39)

Patient

Status on

Presentation

Expected

Outcome

(40)

Expected

Outcome

Observed

Outcome

(41)

Purpose of

risk adjustment

 To help define EXPECTED OUTCOME  Level the playing field

 Account for patient-level factors that influence outcome but do not reflect care quality

(42)

 Avoid risk factors that

 Represent complications of care

 Represent healthcare system attributes  Will potentially mask disparities

(43)

3. The statistic models used for risk

adjustment, and their strength and

weakness.

(44)

Approach to risk adjustment

 Account for patient case mix

 Account for clustering of patients at measurement unit (hospital, group practice, etc.)

 Represent healthcare system attributes (e.g., discharge disposition)

(45)

Approach to risk adjustment

 Account for patient case mix

 Account for clustering of patients at measurement unit (hospital, group practice, etc.)

 Represent healthcare system attributes (e.g., discharge disposition)

(46)

4. The issues regarding quality of data,

especially when claim data is the

major source for measuring

performance.

(47)

5. The policy/approach that help

medical professionals and consumers

to understand report card, especially

the result of adjusted outcomes.

(48)

Aspects of outcome measures specific to accountability:  Higher threshold of scientific reliability and validity  Must be fair to measured entities

 Should assign performance categories with high degree of confidence

(49)

Aspects of outcome measures specific to accountability:

 Higher threshold of scientific reliability and validity

 Must be fair to measured entities

 Should assign performance categories conservatively

(50)

 Higher threshold of scientific reliability and validity  Must be fair to measured entities

 Should assign performance categories conservatively

(51)

(52)

(53)

CMS’s hospital AMI 30-day all-cause mortality measure  Reported as risk-standardized mortality rate (RSMR)  Measures all-cause mortality after AMI admissions

 Captures deaths within 30 days of admission  Publicly reported on HospitalCompare.gov

(54)

Risk-standardized mortality rate (RSMR) = Hospital’s “predicted” deaths

Hospital's “expected” deaths

 “Predicted” = number of deaths within 30 days predicted on basis of hospital’s performance with its observed case mix

 “Expected” = number of deaths expected on basis of the nation’s performance with that hospital’s case mix

national unadjusted mortality rate

(55)

Performance categories for public reporting

 Hospitals categorized as “Better”, “Worse”, or “No Different” than the national rate

 Use 95% interval estimate (like 95% confidence interval) to define categories

(56)

(57)

6. Advice to accelerate/promote

public reporting of hospital

(58)

(59)

(60)

(61)

(62)

(63)