This issue brief, the third in a series prepared by Breakaway Policy Strategies for FasterCures, discusses how “quality” is defined in health care. It explains how performance is measured by various public and private entities that pay for health care, identifies the key
organizations that develop and evaluate measures, and discusses the main uses of performance data and concerns with the validity and usefulness of the data in practice.
Introduction
As discussed in A Closer Look at Alternative Payment Models, the second brief in the FasterCures Value and Coverage Issue Brief Series, the United States is transitioning from a fee-‐for-‐ service (FFS) payment system—one that pays doctors and hospitals for each individual service provided— into a system that rewards providers for quality while controlling costs. Studies have shown that at least 20 to 50 percent of all prescriptions, visits, procedures and hospitalizations in the U.S. are inappropriate either as the overuse, underuse or misuse of what has been shown to be effective and beneficial care.1,2,3 This has fueled demand by consumers and health care policymakers for information regarding the performance of health care providers and institutions.
Developing and Evaluating
Performance Measures
Performance measurement is a quantitative way to
measure quality, and can be defined as “whether or
how often a process and outcome of care occurs.”
4In certain clinical areas, such as cardiac and intensive care, performance measurement has been associated with improvements in providers’ use of evidence-‐based strategies and patients’ health outcomes.5,6 Aside from quality
improvement, capturing performance data can also increase transparency in the quality of care provided and serve as the basis for accreditation or certification for provider groups or organizations.
Agencies like CMS and the Agency for Health Care Research and Quality (AHRQ)7, as well as non-‐profit private developers such
as the Joint Commission8 and the National Committee for
Quality Assurance (NCQA)9 are the organizations responsible for the development of many of the performance measures in use today. Professional societies such as the American Heart Association, the American College of Cardiology, and the American College of Surgeons also develop measures. After measures are developed, they sometimes undergo evaluation by the National Quality Forum (NQF)—a public/private, multi-‐ stakeholder organization that endorses general standards for measurement and specific measures.10 The most widely used performance measures are those endorsed by the NQF or those developed by the NCQA, known as the Healthcare Effectiveness Data and Information Set (HEDIS)—which have been vetted by stakeholders and tested for reliability, feasibility and validity.
Defining and Measuring Quality
The Institute of Medicine (IOM) describes quality as
multidimensional and inclusive of concepts that go well beyond safety.
IOM defines “high-‐quality care” as care that is
safe, effective, patient-‐centered, timely, efficient,
and equitable (with no disparities between racial or
ethnic groups)
11.
In the late 1990’s, the American Medical Association (AMA) began a program to develop physician-‐level performance measures to be used for quality improvement. By bringing together physicians and experts from various medical specialties, clinical process measures for several areas ofPage 2 of 6
medical practice were developed. The AMA’s program, now known as the Physician Consortium for Performance Improvement (PCPI), continues to lead efforts in developing, testing and implementing performance measures for use at the point of care.12
Organizations that define and measure Quality
Organization Involvement with Quality Measurement
NCQA A private, nonprofit that reviews and accredits health insurance plans. Created the Healthcare Effectiveness Data and Information Set (HEDIS), a set of health plan performance measures used for both public reporting and
accreditation.
AHRQ Federal agency within HHS which aims to improve quality, safety, efficiency, and effectiveness. AHRQ initiatives include: the National Quality Measures Clearinghouse (NQMC), which provides information on specific evidence-‐based health care quality measures, and the Consumer Assessment of Health Providers & Systems (CAHPS), a comprehensive series of patient satisfaction surveys regarding health care services.
The Joint Commission
An independent not-‐for-‐profit that accredits more than 20,000 health care organizations and programs in the United States. States and CMS require hospitals and other health care organizations to be accredited by the Joint commission in order to participate in Medicare and Medicaid.
NQF A private, nonprofit that builds consensus around quality improvement priorities and evaluates and endorses quality standards and measures.
There are several quality metrics that allow a user to quantify the quality of health care services by comparison to specific criteria. These metrics include: process measures, outcome measures, patient experience measures, and structure measures.13 Detailed descriptions of these measures is
included in A Closer Look at Alternative Payment Models, the second brief in the FasterCures Value and Coverage Issue Brief Series.
Process measures, which look at improvement and assess the performance of activities shown to contribute to positive patient outcomes, are the most commonly used of the quality measures. The measures can be used for any number of things, including quality improvement, accountability or research—or some combination of the three. The entity performing the measurement must determine the purpose and intended use of the measure and can then use the measurement to help identify problems, establish baseline results, and/or drive quality and performance improvement. Measurement results can be expressed as a rate, ratio, frequency distribution or score for average performance and is often interpreted in comparison to a set standard.
Primary Uses of Performance Data
Performance measurement data is used primarily
for quality improvement initiatives and
accountability.
Both the public and private sectors have developed quality improvement initiatives, such as pay-‐for-‐ performance programs (P4P). P4P is a term that describes payment models that offer financial incentives to providers who achieve or exceed specified quality benchmark. (P4P initiatives are discussed in greater detail in A Closer Look at Alternative Payment Models.) Under most payment models, payments to physicians and hospitals are adjusted on the basis of whether the providers achieve a pre-‐determined set of quality measures.In theory, providing the public with access to
performance data should allow patients to make
informed choices about their care and be more
involved in their medical decision-‐making. It should
also allow providers to identify areas for
improvement and motivate them to make those
improvements. Indeed, some studies have shown
that publicly reporting provider performance data
can result in quality improvements.
14 Such data,however, is not always readily accessible by patients—either patient are not aware that the data is available, it is not the exact information they need, or it is not presented in an understandable way.15
Commercial health plans also make performance data publicly available by classifying providers into different value tiers and encouraging consumers to choose certain providers by offering lower cost-‐sharing.
Page 3 of 6
Performance Measurement in Practice
There are many examples of performance measurement in practice. A few examples include:•
The Physician Quality Reporting System (PQRS),
a voluntary reporting program implemented in
2007 by the Centers for Medicare and Medicaid
Services (CMS),
16offers a financial incentive to
eligible professionals for voluntarily reporting
data on specific quality measures applied to the
Medicare population.
The program uses measures developed by several sources—the majority coming from the AMA PCPI. For 2014, there are 285 measures in total, 110 of which are individual quality measures.17Though there is a financial incentive to
encourage participation in the PQRS, less than
30 percent of eligible providers actually report
data to CMS.
This low participation rate may be due to the concerns that many physicians have regarding the validity of this data and the credibility and accuracy of public reporting, especially in regards to outcomemeasures.18 Despite these concerns, participation in PQRS has been growing.
• CMS uses performance data to offer bonuses to Medicare
Advantage (MA) plans using a star ratings system. The program, implemented under the Affordable Care Act (ACA), pays MA plans bonuses based on the “Medicare 5
star program,” which rates plans online on a scale of 1 to 5 stars. Stars are awarded based on performance measures taken from CMS administrative data, HEDIS measurement data, and CHAPS survey data. CMS also uses performance measurement data in many of its P4P programs, including the End-‐Stage Renal Disease (ESRD) Bundled-‐Payment and Quality Incentive Program.19
• The use of performance measurement in improving specific
types of care has also produced some promising results. For example, there have been significant changes in cardiovascular care in last 10 years. Key changes in this treatment area came from a decision by CMS to support the measurement of care provided to patients with an acute myocardial infarction. Using reported performance data, CMS was able to identify gaps in the quality of care and facilitated and supported efforts to improve
cardiovascular care. Hospitalizations for acute myocardial infarction dropped by more than 25 percent and
hospitalizations for heart failure fell by more than 30 percent.20,21 Post-‐hospitalization mortality due to acute
myocardial infarction also decreased by over 20 percent.22
• CMS has also created online tools to aid in consumer
decision-‐making, including Hospital Compare, Nursing Home Compare and Physician Compare. These sites aid patients in making informed decisions about their health care based on publicly available provider quality data.23 The
program was designed to encourage providers to improve the quality of their care through accountability.
Information made available includes, for example, that regarding readmission, complications and death, timely and effective care, use of medical imaging, and surveys of patients’ experiences.
Page 4 of 6
Assessing the Validity and Usefulness
of Performance Measures
While the purpose of performance measures is to improve quality and promote transparency, even proponents believe it is important to:
• ensure that measures are appropriate; understand the
scientific basis underlying and the strengths and limitations of each measure; and
• reduce inaccurate inferences about provider
performance.
Many agree that scientifically rigorous and valid measures of performance can truly improve value in health care.24 However,
despite the widespread acceptance and use of performance measures in recent years, some argue that there has not been a sufficient corresponding increase in the quality of care.25 There are a few possible explanations for this:
• concerns about the strength of the evidence
underlying the performance measures,26
• the ways in which measures are used to encourage
providers to improve care,27
• limitations of the amount and type of existing data.28
The measures also may not be suitable for clinically important subpopulations—meaning it is easier to achieve in practice, but will have little or no impact on the group of patients who need improvement most,29 or they may not account for a patient’s or clinician’s personal preferences for certain services.30
Another concern is that the majority of the current
quality improvement initiatives focus too heavily on
process measures—instead of outcome measures—
which do not always result in improved care or
value of care for the patient.
Process measures look at improvement and assess the performance of activities shown to contribute to positive health outcomes for patients while outcome measures look at the effects that care had on patients.One example of a program that did not improve
care was the Medicare Premier Hospital Quality
Incentive Demonstration, the largest test of both
public reporting and P4P, which failed to have any
significant impact on the value of care in three key
clinical conditions and neither reduced patient
mortality nor cost growth.
31,32 While the hospitalsdemonstrated improved scores on mostly process measures, the program did little to improve patient outcomes.
However, shifting to using more outcome measures to improve quality is not as easy as it sounds. Patients’ health outcomes are not solely based on the quality of care they receive, but also their previous risk factors, chance events, or social
determinants of health.33,34 As with process measures, there are
also concerns regarding the validity of outcome measures based on the source of data—mainly claims data which may fail to identify preexisting conditions and complications that occur after hospital admission.35
Modifications to Measures over Time
Though concerns exist regarding the validity and usefulness of performance measurement, programs and measures can be modified over time. As those developing and evaluating these measures continue to assess how these work in practice, modifications can be made directly to the measures, by finding new or multiple ways for providers to satisfy the measures, or by creating exclusions for specific circumstances—as in the example above. The future of evidence-‐based performance measurement rests on the ability of stakeholders to make such modifications.One example of how performance measures can be modified involves the Medicare National Pneumonia Project. The Project used performance measures related to timing of antibiotic treatment for patients coming to hospitals with community-‐ acquired pneumonia.36 The Project adopted a measure to administer antibiotics within four hours of a patient first arriving to the hospital. Use of this measure, however, failed to take into account that often, pneumonia cannot be diagnosed during an initial evaluation and that the appropriate standard of care for a stable patient is to withhold treatment until a more certain diagnosis can be made. After studies failed to show that the four hour time window for antibiotic administration decreased mortality in stable patients,37 the Joint Commission relaxed the time window to 6 hours and created a new carve-‐ out for “diagnostic uncertainty” that can be used to exclude certain patients from this measure.
Page 5 of 6
Conclusion
Rewarding performance in our health care system is
gaining momentum as a way to improve the quality
and value of care. Public and private payers
continue to measure not only the under and over
use of services, but also to assess the quality of
certain services and interventions, using the results
for public reporting and P4P programs.
It is important for performance measures to take into account theperspectives of various health care stakeholders—including the patient, the purchaser, and the provider. Though there are hurdles to overcome in perfecting the use of performance measurement in practice, current efforts have demonstrated the potential of these measures if used and modified appropriately along the way.
1 “The Urgent Need to Improve Health Care Quality.” Consensus
Statement-‐ September 16, 1998. Institute of Medicine National
Roundtable on Health Care Quality. JAMA. 280 (1998): 1000-‐1005.
2McGlynn EA, Asch SM, Adams J, et al. “The quality of health care
delivered to adults in the United States.” N Engl J Med. 348 (2003);
2634-‐45.
3 Kerr EA, McGlynn EA, Adams J. “Profiling The Quality Of Care In
Twelve Communities: Results From The CQI Study.” Health Affairs. 23.3
(2004): 247-‐256.
4 Statement preceding each physician measurement set authored by
AMA, JCAHO, and NCQA. Web. June 2005. Available at
http://www.ama-‐assn.org/ama/pub/category/2946.html. Accessed
March, 2014.
5 Krumholz HM, Wang Y, Chen J, et al. “Reduction in Acute Myocardial
Infarction Mortality in the United States: Risk-‐Standardized Mortality
Rates from 1995-‐2006.” Journal of the American Medical Association,
302.7 (2009): 767-‐73.
6 Chassin MR, Loeb JM, Schmaltz SP, et al. “Accountability Measures—
Using Measurement to Promote Quality Improvement.” New England
Journal of Medicine, 363 (2010): 683-‐688.
7 “AHRQ at a Glance.” Rockville, MD: Agency for Healthcare Research
and Quality, www.ahrq.gov/about/mission/glance/index.html
(Accessed March 2014).
8 “Facts About The Joint Commission.” Oakbrook Terrace, IL: The Joint
Commission. N.d. Web. April 2013.
9 Desirable Attributes of HEDIS.” Washington, DC: National Committee
for Quality Assurance. n.d. Web. April 2013. Available at:
www.ncqa.org/tabid/415/Default.aspx (accessed March 2014).
10 National Quality Forum. n.p., 2014. Available at:
http://www.qualityforum.org/who_we_are.aspx (accessed March 2014).
11 “Crossing the Quality Chasm: A New Health System for the 21st
Century.” Institute of Medicine. Washington, DC: National Academies
Press, 2001.
12 “About the PCPI.” American Medical Association. n.p., n.d.. Available
at: http://www.ama-‐assn.org/ama/pub/physician-‐resources/physician-‐ consortium-‐performance-‐improvement/about-‐pcpi.page?. Accessed March, 2014.
13 “Selecting Quality Measures.” Agency for Healthcare Research and
Quality. n.p., n.d. Available at:
http://www.qualitymeasures.ahrq.gov/tutorial/selecting.aspx. Accessed March 2014.
14 Ferris RG and Torchiana DF. “Public Release of Clinical Outcomes
Data—Online CABG Report Cards.” New England Journal of Medicine
363 (2010): 1593-‐1595.
15 Totten AM, Wagner J, Tiwari A, et al. “Closing the Quality Gap:
Revisiting the State of the Science (Vol. 5: Public Reporting as a Quality
Improvement Strategy, Evidence Reports/Technology Assessments.”
Agency for Health Care Research and Quality. 208.5 (2012).
16 Berenson R, Kaye D. “Grading a physician’s value-‐ the misapplication
of performance measurement.” N Engl J Med. 369 (2013): 2079-‐2081.
17 Measures Codes.” Centers for Medicare and Medicaid Servvices.
N.p., 2013. Available at: http://www.cms.gov/Medicare/Quality-‐ Initiatives-‐Patient-‐Assessment-‐
Instruments/PQRS/MeasuresCodes.html. Accessed March 2014).
18 “Physician performance measurement & reporting introduction.”
Value-‐based Purchasing Guide. n.p., 2011.
19 Van Lare JM and Conway PH. “Value-‐Based Purchasing—National
Programs to Move from Volume to Value.” New England Journal of
Medicine. 367 (2012): 292-‐295.
20 Chen J, Normand SL, Wang Y, et al. “National and Regional Trends in
Heart Failure Hospitalization and Mortality Rates for Medicare
Beneficiaries, 1998-‐2008.” JAMA. 306.15 (2011): 1669-‐1678.
21 Wang OJ, Wang Y, Chen J, et al. “Recent Trends in Hospitalization for
Acute Myocardial Infarction.” American Journal of Cardiology, 109.11
(2012): 1589-‐1593.
22 Krumholz HM, Wang Y, Chen J, et al. “Reduction in Acute Myocardial
Infarction Mortality in the United States: Risk-‐standardized Mortality
Rates from 1995-‐2006.” JAMA.302.7 (2009): 767-‐73.
23 “Hospital Compare.” “Nursing Home Compare.” “Physician
Compare.” Medicare.gov. n.p., n.d. Available at:
http://www.medicare.gov/nursinghomecompare/search.html;http://w ww.medicare.gov/hospitalcompare/search.html;
http://www.medicare.gov/physiciancompare/search.html.
24 “Guidance for Measure Testing and Evaluating Scientific
Acceptability of Measure Properties.” Washington DC: National Quality
Forum. January 2011.
25 Pronovost PJ and LIlford R. “A Road Map for Improving the
Performance of Performance Measures.” Health Affairs (Millwood).
30.4 (2011): 569-‐73.
26 Tricoci, P., JM. Allen, J.M. Kramer, R.M. Califf, and S.C. Smith, Jr.
Page 6 of 6
Guidelines.” Journal of the American Medical Association. 301.8
(2009): 831-‐841.
27 Werner, Rachel and Dudley, R. “Medicare’s New Hospital Value-‐
Based Purchasing Program is Likely to Have Only a Small Impact on
Hospital Payments,” Health Affairs. 31.9 (2012): 1932-‐40
28 Roski J and McClellan M. “Measuring Health Care Performance Now,
Not Tomorrow: Essential Steps to Support Effective Health Reform.”
Health Affairs. 30 (2011): 4682-‐689.
29Hayward, R. “All or Nothing Treatment Targets Make Bad
Performance Measures.” The American Journal of Managed Care.13.3
(2007): 126-‐128.
30 Tinetti, M.E., et al. “Health outcome Priorities Among Competing
Cardiovascular, Fall Injury, and Medication-‐Related Symptom
Outcomes.” Journal of the American Geriatrics Society. 56.8 (2008):
1409-‐1416.
31 Ryan AM, Nallamouth BK and Dimik JB. “Medicare’s Public Reporting
Initiative On Hospital Quality Had Modest or No Impact on Mortality
From Three Key Conditions.” Health Affairs. 31.5 (2012): 585-‐592.
32 Ryan AM. “Effects of the Premier Hospital Quality Incentive
Demonstration on Medicare Patient Mortality and Cost.” Health
Services Research. 44.3 (2009): 821-‐842.
33 Bradley et al., Schwarz M Cohen AB, Restucciai JD, et al. “How Well
Can We identify the High Performing Hospital?” Medical Care Research
and Review. 68.3 (2011): 290-‐310.
34 Werner RM, Bradlow ET, and Asch DA. “Does Hospital Performance
on Process Measures Directly Measure High Quality Care or Is it a
Marker of Unmeasured Care?” Health Services Research. 43.5.1 (2008):
1464-‐1484.
35 Glance LG, Newman M, Martinez, EA, et al. “Performance
Measurement at a “Tipping Point.”” Anesthesia and Analgesia. 112.4
(2011): 958-‐966.
36 Wachter RM, Flanders SA, Fee C, Pronovost PJ. “Public reporting of
antibiotic timing in patients with pneumonia: lessons from a flawed
performance measure.” Ann Intern Med.149 (2008): 29-‐32.
37 Wachter R, Flanders S, Fee C, Pronovost P. “Public reporting of
antibiotic timing in patients with pneumonia: lessons from a flawed
per formance measure.” Annals of Internal Medicine. 149 (2008): 29-‐
