1
Three-Star Composite Rating Method
CheckPoint uses three-star composite ratings to enable consumers to more quickly and easily interpret information about hospital quality measures. Composite ratings combine individual measures into a single rating to summarize the overall quality of care for a specific clinical condition or quality topic. The method used for CheckPoint
composite ratings combines the methods used by HealthPartners1 and The Joint Commission2. The method uses a four-step process of:
• Determine whether each measure within the composite is statistically different from the Wisconsin average for that measure;
• Calculate the Quality Score for each measure, based on the statistical difference;
• Calculate the Composite Score for the group of measures assigned to the composite; and • Assign a three-star Composite Rating.
1) Determine statistical difference for each measure
A confidence interval is calculated for each measure and then compared to the state average or a Target Range for that measure.
a) Calculation of Confidence Intervals
i) Confidence intervals will be calculated for measures with a denominator >25.
ii) When a confidence interval is supplied with the data source, that confidence interval is used to determine statistical significance. This includes Centers for Medicare and Medicaid Services (CMS) mortality, readmission and infection measures. It also includes mortality and patient safety indicators calculated with Agency for Healthcare Research and Quality (AHRQ) software.
iii) When a confidence interval does not already exist one is calculated using the Wilson Score Interval method. This method is more reliable than the Normal Approximation Interval method, particularly when sample sizes are small and/or the performance level is near 100 or zero.
Wilson Score Interval Method Calculation
b) Compare to State Average or Target Range
i) If the confidence interval was supplied with the data the individual hospital’s confidence interval is compared to the state average for that measure.
ii) If a confidence interval was calculated using the Wilson Score Method the confidence interval is compared to a Target Range.
(1) When the direction of desired improvement is an increase in the measure rate, the target range consists of an upper and lower target limit computed as follows:
(a) If the state average is greater than or equal to 0.95 then the upper target limit and lower target limit will both be set to 0.95.
(b) If the state average is less than 0.95 and greater than or equal to 0.90, then the upper target limit is set to 0.95 and the lower target limit is set to the state average.
2 Examples:
State Average Lower Target Limit Upper Target Limit
0.98 0.95 0.95
0.92 0.92 0.95
0.84 0.84 0.92
(2) When the direction of desired improvement is a decrease, the then target range consists of an upper target limit and lower target limit computed as follows:
(a) If the state average is less than or equal to 0.05 then the upper target limit and lower target limit will both be set to 0.05.
(b) If the state average is greater than 0.05 and less than or equal to 0.10, then the lower target limit is set to 0.05 and the upper target limit is set to the state average.
(c) If the state average is greater than 0.10 then the upper target limit is set to the average of the state average and zero, and the lower target limit is set to 0.05.
Examples:
State Average Lower Target Limit Upper Target Limit
0.04 0.05 0.05
0.08 0.05 0.08
0.28 0.05 0.14
c) Classify the Difference Between Hospital Performance and the State Performance
Compare confidence interval for hospital, on each measure, to the state average or target range and classify as “No Different”, “Better Than” or “Worse Than” the overall state performance.
i) HCAHPS analysis is based on “Always” survey responses and utilizes the following logic for comparing each survey domain, overall satisfaction and willingness to recommend1:
(a) If the hospital score is >75 the hospital is “Better Than” other hospitals in the state.
(b) If the hospital score is >70 or <75 the hospital is “No Different” from other hospitals in the state. (c) If the hospital score is <70 the hospital is “Worse Than” other hospitals in the state.
ii) For measures where lower performance is better, such as mortality, infections and readmissions, the following logic is used2:
(a) If the state average or target range for the measure overlaps a hospital’s lower and upper confidence intervals the hospital is “No Different” from other hospitals in the state.
(b) If the state average or target range is entirely above a hospital’s upper confidence interval the hospital is “Better Than” other hospitals in the state. If a hospital has a rate of zero, they will be scored as “Better Than” regardless of the sample size and confidence interval.
3
iii) For measures where higher performance is better, such as process measures, the following logic is used2:
(a) If the state average or target range for the measure overlaps a hospital’s lower and upper confidence intervals the hospital is “No Different” from other hospitals in the state.
(b) If the state average or target range is entirely below a hospital’s lower confidence interval the hospital is “Better Than” other hospitals in the state. If a hospital has a rate of 100 they will be scored as “Better Than” regardless of their sample size or confidence interval.
(c) If the state average or target range is entirely above a hospital’s upper confidence interval the hospital is “Worse Than” other hospitals in the state.
2) Calculate a Quality Score for Each Measure
A Quality Score is assigned to each measure based on their statistical difference from the other hospitals in the state.
a) Calculation of Raw Quality Score
4
iii) Measures that were “Worse Than” other hospitals receive a score of 0.0 Example: If Target Range = 85.2-96.5%
Hospital Numerator Denominator Hospital Rate Statistical Difference Raw Quality Score A 24 25 95% “No Different” 0.5 B 42 61 69% “Worse Than” 0 C 99 99 100% “Better Than” 1.0 D 50 53 94.3% “No Different” 0.5 E 9 12 75% NA – Sample is <25 NA F 20 20 100% “Better Than” 1.0
b) Weighting of Raw Score
i) The weighted score for HCAHPS domains/questions equal the Raw Quality Score x 1.0.
ii) The weighted score for process measures equal the Raw Quality Score x 1.0. Process measures are measures that indicate whether the appropriate care was provided.
iii) The weighted score for outcome measures equal the Raw Quality Score x 1.5. Outcome measures are measures that reflect the patient’s response to care, including mortality, readmissions and hospital acquired conditions.
Example for Stroke:
Measure
Raw
Quality Score Weight
Weighted Quality Score
STK-2 - Antithrombotics on discharge 1.0 1.0 1.0
STK-3 - Anticoag for atrial fib/flutter 0.5 1.0 0.5
STK-5 - Early antithrombotics 0.5 1.0 0.5
STK-6 – Discharged on statins 0.5 1.0 0.5
STK-8 – Patient education 0 1.0 0
STK-10 – Assessed for rehab 1.0 1.0 1.0
Stroke mortality 0.5 1.5 0.75
Weighted Score 4.25
3) Calculate a Composite Score for a Group of Measures
a) A hospital must participate in all measures that comprise the Composite Rating to receive a composite rating. If a hospital has chosen to not report data for any of the composite measures they will be reported as “DNR” – Did Not Report.
b) Each composite will have three or more measures. A list of the measures used for each Composite Rating is provided in Appendix A
c) A hospital must have a denominator >25 for the measure to be included in the Composite Score. At least half of the measures must have valid denominators to calculate a Composite Score. If the hospital participates in all of the measures but does not meet the minimum sample size for at least half of the measures their composite will be reported as “+” – Insufficient Sample Sizes to Calculate a Composite. d) The Composite Score is the sum of the weighted scores divided by the total possible weighted points.
X X
5 Example for Stroke:
Measure
Raw
Quality Score Weight
Weighted Quality Score
STK-2 - Antithrombotics on discharge 1.0 1.0 1.0
STK-3 - Anticoag for atrial fib/flutter 0.5 1.0 0.5
STK-5 - Early antithrombotics 0.5 1.0 0.5
STK-6 – Discharged on statins 0.5 1.0 0.5
STK-8 – Patient education 0 1.0 0
STK-10 – Assessed for rehab 1.0 1.0 1.0
Stroke mortality 0.5 1.5 0.75
Totals 7.5 4.25
Composite Score 0.57 (4.25/7.5)
4) Assign a Composite Rating
A three-star rating is assigned using the Composite Score calculated in step 3. a) Composite Scores <0.33 receive a one-star rating
b) Composite Scores >0.33 and <0.67 receive a two-star rating c) Composite Scores >0.67 receive a three-star rating
0 0.33 0.67 1.0
Exhibit 3: Composite Ratings
Example and Composite Key Weighted Quality Score Composite Rating Comments Hospital A 0.59 2 star Hospital B 0.85 3 star Hospital C 0.25 1 star
Hospital D DNR Hospital did not participate in all
measures
Hospital E + Hospital participates in all measures but did not have a sample size of >25 in at
least half of the measures
Hospital G NA Hospital does not provide the services
rated by this composite
References:
1. HealthPartners ; Cost and Quality Assessment Documentation -
https://www.healthpartners.com/ucm/groups/public/@hp/@public/documents/documents/cntrb_033165.pdf
2. The Joint Commission – 2011 Health Care Professional Quality Report User Guide -
http://www.qualitycheck.org/assets/Hospital_Prof_USER_GUIDE_June%202011.pdf
X X
6 Appendix A – Composite Ratings and Component Measures
Composite Rating Component Measures
Birth Cesarean section
Early elective delivery (PC-01)
Newborn Screening Turnaround Time Heart Attack PCI within 90 minutes (AMI-8a)
Heart attack 30-Day mortality Heart attack 30-Day readmission
Heart Failure Left ventricular failure assessment (HF-2) Heart failure 30-Day mortality
Heart failure 30-Day readmission
Infections Central line associated blood stream infection Catheter associated urinary tract infection Colon surgery surgical site infection
Abdominal hysterectomy surgical site infection Mortality Acute stroke mortality
Gastrointestinal hemorrhage mortality
Abdominal aortic aneurysm mortality (2013 data excluded pending source software issue resolution)
Coronary bypass mortality Coronary angioplasty mortality Carotid endarterectomy mortality Craniotomy mortality
Hip replacement mortality Hip fracture mortality Pneumonia mortality Heart failure mortality Heart attack mortality Patient Satisfaction Overall satisfaction
Patient would recommend hospital Doctor communication
Nurse communication
Patients received help they needed Staff explained medications
Pain control
Hospital was quiet at night Patient room was clean
Staff provided discharge instructions Pneumonia Appropriate antibiotic (PN-6)
Pneumonia 30-Day mortality Pneumonia 30-Day readmission Readmissions Pneumonia 30-Day readmission Heart failure 30-Day readmission Heart attack 30-Day readmission All Cause Readmission
Stroke Antithrombotics on discharge (STK-2)
7
Composite Rating Component Measures
Discharged on statin (STK-6) Patient education (STK-8)
Assessment for rehabilitation (STK-10) Stroke mortality
Surgery Antibiotics within 1 hour of incision (SCIP-Inf-1) Appropriate antibiotics (SCIP-Inf-2)
Antibiotics stopped within 48 hours of surgery (SCIP-Inf-3) Clot prevention given (SCIP-VTE-2)
