Risk Adjustment for Pediatric Quality Indicators
Karen Kuhlthau, PhD*; Timothy G. G. Ferris, MD, MPH*‡; and Lisa I. Iezzoni, MD, MSc§
ABSTRACT. The movement to measure medical care quality has been accelerating, spurred on by evidence of poor quality of care and trials of interventions to improve care. Appropriate measurement of quality of care is an essential aspect of improving the quality of care, yet some quality measures may be influenced by patients’ attributes unrelated to quality of care. Risk adjustment is the term commonly applied to those methods that ac-count for patient-related attributes, making measure-ment of health care quality as comparable as possible across providers or organizations seeing different mixes of patients.
The measurement of quality of care for children poses specific challenges. In addition to these measurement challenges, analysts must ensure that quality compari-sons among doctors, groups of doctors, hospitals, or health plans are not adversely affected by the likelihood that different types of patients seek care in different places. Although some techniques designed to adjust performance measures for case mix were developed for both adults and children, other systems are specific to childhood circumstances. The theoretical issues involved in risk-adjusting childhood outcomes measures for new-borns in the neonatal intensive care unit were reviewed recently. Here, we go beyond the neonatal intensive care unit setting to consider risk adjustment for pediatric quality measures more broadly. In particular, we 1) re-view the conceptual background for risk-adjusting qual-ity measures, 2) present policy issues related to adjusting pediatric quality measures, and 3) catalog existing risk-adjustment methodologies for pediatric quality mea-sures. We conclude with an overall assessment of the status of risk adjustment for pediatric quality measures and recommendations for additional research and application.Pediatrics2004;113:210 –216;risk adjustment, pediatric quality indicators, child.
ABBREVIATIONS. QI, quality indicators; ICU, intensive care unit; NICU, neonatal ICU; ICD-9-CM,International Classification of
Dis-eases,Ninth Revision, Clinical Modification; CDS, Chronic Disease Status; PICU, pediatric ICU; PRISM, Pediatric Risk of Mortality; HEDIS, Health Employer Data and Information Set; CAHPS, Con-sumer Assessment of Health Plans Study; HCUP, Health Care Cost and Utilization Project; APR, all patient-refined; DRG, diag-nosis-related group.
If comparing quality of care across doctors, hospi-tals, or health plans, we must ensure that the groups of patients are sufficiently similar to make the comparison meaningful and fair. As for adult patients, some pediatric providers or health plans have patient populations with a more severe burden of illness, especially chronic conditions, than do oth-ers.1,2 Even with the best care, these children are
more likely to do poorly (ie, have worse outcomes) than healthier children.
Quality indicators (QIs) are usually described as either outcome or process measures.3For example, if
we measure health care quality for children with chronic conditions, we may choose 2 different ap-proaches. Process measures examine whether the right care was delivered. In contrast, outcome mea-sures evaluate how well children do after receiving care. Distinguishing process and outcome measures is important for risk adjustment: Both the policy implications and the methods used generally differ between these types of measures.
Quality measures should isolate factors under the control of the unit of observation (eg, providers or hospitals). Risk adjustment involves accounting for patients’ characteristics that influence the quality measure but are not under the control of the pro-vider. To the extent that risk-adjustment techniques are able to adjust for differences in the population that is served, risk adjustment will insure that quality comparisons are valid. The rationale for risk adjust-ment is generally obvious for outcome measures. However, even some process indicators (eg, vac-cination) are not entirely under the control of pro-viders.
Risk adjustments typically include adjustment for health status (and may include socioeconomic status) and other intrinsic patient factors. The technique used can vary by the unit of observation (eg, pro-vider or institution), outcome/process measure of interest, and the type of data used. Decisions re-garding when to use risk adjustment for quality measures often can be made on conceptual grounds. Ideally, comparisons of patients’ outcomes should account for all factors that predispose the patient to a better or worse outcome. In practice, the inclu-sion of all factors is neither practical nor necessary
From the *Center for Child and Adolescent Health Policy, MassGeneral Hospital for Children, Department of Pediatrics, Harvard Medical School, Boston, Massachusetts; ‡Institute for Health Policy, Division of General Medicine, Massachusetts General Hospital, Department of Medicine, Har-vard Medical School, Boston, Massachusetts; and §Division of General Medicine and Primary Care, Department of Medicine, Harvard Medical School, Beth Israel Deaconess Medical Center, the Charles A. Dana, Re-search Institute, and the Harvard-Thorndike Laboratory, Boston, Massachu-setts.
Received for publication May 13, 2003; accepted Oct 2, 2003.
This article was commissioned by the Agency for Healthcare Research and Quality and presented at Quality Measures for Children’s Health Care: Assessing the State of the Science and Practice—A Strategy Development Meeting; February 5– 6, 2002; Rockville, MD.
Address correspondence to Karen Kuhlthau, PhD, Center for Child and Adolescent Health Policy, MassGeneral Hospital for Children, 50 Staniford St, Ste 901, Boston, MA 02114. E-mail: firstname.lastname@example.org
because some factors matter more than others, and inclusion of the most predictive factors in statistical models generally accounts for a majority of the ad-justment.
The appropriateness of adjusting quality measures for process-of-care (eg, giving an immunization) measures is more complicated. Many process mea-sures require a positive action by patients (eg, bring-ing an infant for immunization). These measures raise special concerns. Education, motivation, where-withal (eg, transportation, child care, and time off from work), preferences for care and outcomes, cul-tural concerns, and a variety of other factors influ-ence these patient actions. Different health plans and providers see different mixes of patients along these critical dimensions. Because these factors affect im-munization rates and they are largely out of the control of the provider, one might argue that risk adjustment is indicated. Yet, an accreditation body making this measure public knowledge in the inter-est of accountability may want to hold all providers to the same standard of care regardless of the patient population (eg, all children should be immunized). In this context, risk adjustment may be viewed as either unnecessary or even counterproductive. Nev-ertheless, providers with the more disadvantaged patient population may be required to invest more resources per patient to achieve comparable immu-nization rates. These providers therefore may be un-able to focus equally on other issues (eg, counseling for smoking cessation) or may need more funds to cover the additional work necessary to achieve sim-ilar performance.
There are alternative ways of creating comparable groups without risk-adjusting quality measures. Risk stratification refers to splitting populations into specific policy-relevant groups to facilitate compari-sons.4Risk stratification is an alternative to risk
ad-justment and is desirable when a patient character-istic that is a potential risk factor could also result from differences in the way that patients are treated (ie, their quality of care). Assuming that data are available on the patient characteristic by which the quality data should be grouped (eg, race), it is pref-erable to perform risk stratification first (eg, compare performance across plans or providers separately for patients grouped by racial categories). If perfor-mance is similar for different plans or providers within each patient stratum, then it would be reason-able to combine patients across strata and risk-adjust for that characteristic, assuming that the conceptual model provides a valid, causal rationale for that char-acteristic being a risk factor (see Romano5for a
discus-sion of the policy issues).
Even when risk adjustment is applied, it does not adjust perfectly for all factors. Thinking about poten-tial risk factors that were not accounted for will assist in interpreting comparisons across plans or provid-ers. Identifying risk factors for poor outcomes or processes also assists in the process of improving care by providing targets for quality-improvement interventions.
ISSUES IN RISK-ADJUSTING PEDIATRIC QUALITY MEASURES
Many characteristics make children special: rela-tively low mortality and morbidity rates; specialized pediatric services (eg, neonatal intensive care unit [NICU] care); use of providers and institutions out-side the traditional health care delivery system (eg, school health clinics); and the need for adult help and supportive environments. These considerations often make risk adjustment more challenging for children than adults. Importantly, these factors also compli-cate efforts to define pediatric quality measures re-gardless of risk. Several factors are especially impor-tant when designing risk adjustment for pediatric quality measures.
The first factor is the epidemiology of childhood disease and chronic conditions. Except for specialty clinics, the number of children with any particular serious health condition is likely to be small, espe-cially for individual providers. Even for relatively common chronic conditions such as asthma, num-bers of patients may be too small for individual providers to support QIs or adjust for risk. Most children are hospitalized relatively rarely; hospital-based quality measures will be less useful than for an adult population. Nonetheless, comparisons be-tween hospitals or hospital units are still useful in children, especially when their care is concentrated (such as comparing NICU outcomes or the results of congenital heart disease surgery). These comparisons require appropriate adjustments.6
Second, the meaning and use of diagnoses, drugs, and procedures are sometimes different for children compared with adults. Although diagnoses can serve as important risk adjusters for the high percentage of older adults with multiple conditions, symptoms, developmental milestones, and functional status may be more pertinent for children. The standard diagnosis coding nomenclature in administrative da-tabases, the International Classification of Diseases,
Ninth Revision,Clinical Modification(ICD-9-CM)7does
not contain codes for many of these developmental or functional attributes. ICD-9-CM codes for symp-toms and signs give little indication of condition severity or chronicity. In addition, the meaning of codes may be different for children and adults (eg, murmur and sleep disturbance). Most studies of cod-ing completeness and accuracy have targeted Medi-care populations,8,9as have most regulations
involv-ing auditinvolv-ing and oversight of codinvolv-ing and medical record documentation. Diagnosis coding and medi-cal record documentation may be even less accurate and clinically meaningful for children than adults.
Researchers recently developed risk-adjusted tools based on pharmacy claims. Patterns of medication use are different in adults and children.10,11 Many
may seek diagnoses to obtain services (eg, attention-deficit disorder or dyslexia) through special educa-tion or other programs.
Third, most health care encounter databases clude services covered by one private or public in-surer. However, children often use services from diverse settings within the health care delivery sys-tem and outside of it. Use of school and public health clinics is generally not included in administrative databases; medical records in private doctors’ offices do not link automatically with those in school or public health settings. Thus, children who use early intervention, public health, and school-based clinics may seem healthier than they actually are simply because existing data sets do not include all health-related information. These potential biases could compromise risk adjustment.
A related concern is that children use both the pediatric care system (including pediatric subspe-cialists, general pediatricians, and children’s hospi-tals) and the adult care system. Because the pediatric care system likely serves children with more severe conditions, risk adjustment may be particularly im-portant when comparing pediatric and adult provid-ers.
Finally, family and community factors strongly influence child utilization and health status. Risk therefore should be measured, at least in part, at the family or parental level. Familial attributes may be particularly important for process-of-care measures. Because of the complexity of identifying and quan-tifying child outcomes, process measures are likely to be used more commonly than outcomes measures as QIs for children. Timely delivery of preventive ser-vices is measured easily and is a common QI for children. Although virtually all children should be immunized, characteristics of the family (eg, living with mother and maternal grandmother, cash assis-tance eligibility, father living in home with child, presence of an additional preschool sibling, and use of Supplemental Nutrition Program for Women, In-fants, and Children) are associated with reduced immunization rates.12,13 As noted above,
stratifica-tion may be the appropriate first step for analyzing differences in process measures, but few databases contain information on these critical familial and environmental risk factors. One promising possible solution makes use of data on socioeconomic status obtained from the census to adjust performance mea-sures such as physician profiles.14
The prominent role of the family and environment in which children live is tied to the concerns about equity in the delivery of medical care raised by the Institute of Medicine.15 Unless these contextual
fac-tors are recognized and adjusted for, providers and plans seeing disproportionate numbers of relatively sick and vulnerable children could fare poorly in quality measurement; they may seem as if they are giving substandard care. This could unfairly penal-ize some providers and make others less inclined to care for disadvantaged children. On the other hand, as noted above, one would not want to hold provid-ers to a lower standard of care for any subpopula-tions. As such, the decision about risk adjustment
depends on both the conceptual model guiding its use and the quality measure’s purpose. Because the requisite data on these risk factors are generally un-available, comparisons of performance among pro-viders caring for children across the socioeconomic spectrum must proceed cautiously.
RISK-ADJUSTMENT METHODOLOGIES FOR CHILDREN
Here we briefly review existing techniques and examples for adjusting pediatric quality measures. Risk adjustment varies depending on the specific quality measure; we present several available pedi-atric quality measures only as examples. A more complete review of pediatric quality measurement per se is in Beal et al.16We consider risk adjustment
in 3 categories: 1) generic risk adjustment using ad-ministrative data; 2) risk adjustment for specific con-ditions or settings using administrative or clinical data; and 3) risk adjustment found in major quality-measurement data sets.
Generic Risk Adjustment
Several risk-adjustment methodologies use ICD-9-CM diagnosis codes and/or pharmacy claims from administrative data to create predictions of future costs for a defined population. These methodologies, sometimes called “case-mix adjustments,” were orig-inally intended to assist in setting capitated payment levels. Although they have been used primarily to predict costs, they sometimes have been used to assess provider performance (such as comparing hospitalization rates across physicians). Examples of these risk-adjustment systems include the Adjusted Clinical Groups,17 Diagnostic Cost Groups,18 the
Chronic Illness and Disability Payment System,19
and the Clinical Risk Groups.20 Most of these
ad-justment techniques were developed with the in-volvement of pediatricians in specifying the clinical algorithms (eg, grouping together clinically homoge-neous ICD-9-CM diagnosis codes). Some specifically tested their algorithms on child populations (eg, by using Medicaid or private insurance databases).21
One strength of these methodologies for adults is their ability to quantify the combined effect of mul-tiple, coexisting conditions. Because children typi-cally do not have as extensive comorbidities as adults,22 comorbid diagnoses may not be as useful
for adjusting pediatric performance measures. Yet, these risk-adjustment methodologies have been shown to predict subspecialist visit use in adult23
and child24,25 populations. Perrin et al26 used these
adjustment tools to predict child hospitalizations and emergency department visits. On the other hand, these adjustments have only marginal influence on quality comparisons for which care processes for a single disease (eg, asthma) are being considered.27
med-ications for the treatment of diseases of children.28
Fishman and Shay29 used the CDS methodology to
develop a pediatric-specific model. The increasing ease of obtaining pharmacy information make this risk adjustment with pharmacy data a potentially important tool for populations with health insurance plans (public or private) that offer prescription drug benefits.
Risk Adjustment for Specific Conditions or Settings
Risk Adjustment for Neonatal Outcomes
Morbidity and mortality for newborns, particu-larly those admitted to NICUs, are important child outcomes. Richardson and colleagues reviewed the literature on neonatal risk-scoring systems that were designed to predict neonatal mortality, length of stay, and total costs.30 More recently, Richardson et
al6 reviewed issues of risk adjustment for
perfor-mance measurement in the NICU. Several illness-severity measurement systems for adjusting neonatal outcomes have been developed, and two, the Clinical Risk Index for Infants31 and the Score for Neonatal
Acute Physiology,32,33 have undergone rigorous
in-dependent evaluation.34 Some systems (including
the Neonatal Therapeutic Intervention Scoring Sys-tem and the Berlin score) include treatments as part of the score, whereas others (including the National Institute of Child Health and Human Development neonatal network model35) include race. The
inclu-sion of treatment and race variables may create un-wanted biases in application of these systems to quality comparisons. Despite publications docu-menting multiple uses for these systems, they have not been extensively applied to compare provider performance outside of a limited set of research stud-ies. Nonetheless, these studies suggest several im-portant considerations if they are to be applied broadly to compare NICU performance. First, these systems have been used successfully to adjust per-formance comparisons between different NICUs.36 –38
Second, the performance of these systems varies by both the population of newborns and the time period to which they have applied. Empirically calibrated adjustment systems such as the Score for Neonatal Acute Physiology reflect ICU practices that were common when the system was developed or brated. These systems require updating and recali-brating with therapeutic advances. Third, these sys-tems require data from blood tests (eg, arterial blood gas), and differences between institutions in the tim-ing of blood draws might affect the risk-adjustment scores.
Despite these caveats, risk adjustment for neonatal outcomes is sufficiently well developed and vali-dated to be ready for more widespread use in NICU performance comparisons. Considerable improve-ments in NICU care would likely result from wide-spread adoption.6,36
Risk Adjustment for Pediatric ICU Outcomes
Methods for predicting mortality in the pediatric ICU (PICU) have also been developed and studied extensively. The most commonly available tools
include Pediatric Risk of Mortality (PRISM),39
Dy-namic Objective Risk Assessment,40Pediatric
Logis-tic Organ Dysfunction, and Paediatric Index of Mortality. Dynamic Objective Risk Assessment is an adjustment system that permits daily updating of mortality risk as assessed by the PRISM system. The Societie Francaise d’Anesthesie et de Reanimation maintains a web site at which risks of mortality from each of these instruments can be calculated and com-pared (http://www.sfar.org/scores2/scores2.html). PRISM and subsequent revisions (PRISM III) have undergone the most extensive testing and use. PRISM scores, similar to the neonatal scores, rely primarily on data from physiologic measurements made within the first half or full day of admission to a PICU. PRISM has been used to understand differ-ences in PICU admissions between teaching and nonteaching hospitals and the effect of pediatric in-tensivists on mortality rates.41PRISM also has been
used to compare hospital length of stay between different institutions.42 The PRISM system permits
comparison of outcomes between different PICUs, and although not currently widely used, PRISM or a system similar to it could be widely used to compare quality of care across PICUs.
Risk-Adjusting Surgical Mortality for Congenital Heart Disease
Surgery for congenital heart defects in children is relatively common at regional centers and, similar to adult cardiac procedures, is technically demanding. For these reasons the outcomes of congenital heart surgery are a logical target for comparisons between institutions.43Several groups of researchers have
ex-amined mortality after surgery for congenital heart defects. Risk of mortality seems to be most closely related to the type of procedure.44 – 46
Jenkins and et al47 developed a method of
risk-adjusting in-hospital mortality for children ⬍18 years old undergoing surgery for congenital heart disease. They derived the Risk Adjustment for Con-genital Heart Surgery measure by using a combina-tion of expert consensus and empirical methods to determine relative risks of in-hospital death based on specific surgical procedure and other clinical charac-teristics. This system is relatively new and therefore has not been extensively tested, but it may be applied to comparisons of cardiac surgery, which then could be used to help improve care48 as was done for
coronary artery bypass graft surgery in New En-gland.49
Risk Adjustment Using Major Quality-Measure Sets
Here we discuss risk adjustment for selected pedi-atric quality measures found in commonly used mea-surement sets. Again, our review is not exhaustive but illustrative, focusing on commonly used mea-sures or those for which the risk-adjustment tech-nique illustrates a salient point.
Health Employer Data and Information Set
qual-ity measures. As with the adult qualqual-ity measures, HEDIS does not use risk adjustment in the way we have used the term throughout this article. For some measures, instead of including an entire population or condition (such as all persons with asthma) within a measure, HEDIS restricts the denominator to in-clude only those who meet strict definitions of eligi-bility. For example, the diphtheria-tetanus-acellular pertussis immunization measure excludes patients with encephalopathy. This approach minimizes the need for posthoc risk adjustment. But, because de-nominators are restricted, the measure applies to only a fraction of the population. HEDIS measures do not adjust for socioeconomic risk, although they can be reported by insurance plan (Medicaid versus private).
HEDIS also includes the Consumer Assessment of Health Plans Study (CAHPS) survey.50,51The
pub-licly reported CAHPS scores results are not risk ad-justed, although the survey does contain questions about respondent (parent) and child characteristics that could be used to adjust responses (eg, parental education or child health status). The CAHPS team found that including these elements for adjustment of pediatric CAHPS scores does have a small but significant impact on CAHPS scores.52 Risk
adjust-ment of adult CAHPS data has been shown to have small but significant effects on quality scores.53
Rand’s Pediatric Quality Measures
Rand developed a comprehensive set of indicators for quality of care for children and adolescents as a part of a larger effort to collect quality-assessment tools addressing many dimensions of performance for conditions/events such as preventive services, family planning, otitis media, sickle cell disease, and acne.54Risk adjustment is largely absent from Rand’s
approach, although similar to the HEDIS measure set, indicators of treatment and assessment are con-ditional on diagnoses and previous treatments. In other words, the Rand measures tightly define to whom the measure should be applied and therefore greatly reduce the need for risk adjustment. For ex-ample, “in patients requiring chronic treatment with oral corticosteroids, a trial of inhaled corticosteroids should have been attempted.” In addition, they re-strict denominators, eliminating children with con-tradictions to treatment (eg, concon-tradictions to immu-nization).
Health Care Cost and Utilization Project QIs
The Health Care Cost and Utilization Project (HCUP) QIs were created in 1994 by the Agency for Healthcare Research and Quality and rely on hospi-tal administrative discharge data. The HCUP QIs use all patient-refined (APR) diagnosis-related groups (DRGs) to risk-adjust certain hospital quality mea-sures. In choosing their risk-adjustment methodol-ogy, the Agency for Healthcare Research and Quality considered: the classification and analytic approach; how the method was developed; costs and feasibil-ity; empirical performance (discrimination and cali-bration); whether the method is “open” (ie, has pub-lished logic detailing the methodology); and whether
the methodology had achieved official recognition. The choice of APR DRGs was driven in part by Medicare’s reliance on DRGs.
Only selected HCUP QIs are risk adjusted. Risk adjustment is applied by using regression tech-niques. HCUP identifies 7 pediatric QIs. For pediat-ric heart surgery volume, the instructions do not recommend risk adjustment. For 5 child measures, HCUP recommends adjusting by age and sex (pedi-atric asthma admission rate, pedi(pedi-atric gastroenteritis admission rate, hospitalization for bacterial pneumo-nia, hospitalization for urinary tract infection, and perforated appendix). Risk adjustment using APR DRGs is recommended only for pediatric heart sur-gery mortality.
GAPS AND IMPORTANT RESEARCH AREAS FOR PEDIATRIC RISK ADJUSTMENT
How much risk adjustment is adequate to “level the playing field?” The answer to this question is: It depends on the quality measure itself and how the information will be used. Even with the relative so-phistication of risk-adjustment methods for NICU, PICU, and congenital heart disease deaths, for exam-ple, it is still probably unwise to make final decisions about the quality of NICU, PICU, or heart surgery programs on this basis alone. Unmeasured risk fac-tors may still differ across providers (eg, differences in “do-not-resuscitate” practices, at least partially a reflection of patients’ preferences for life-sustaining care). Risk-adjustment methods that account for the most important risk factors are more likely to facili-tate drawing inferences about quality of care than methods that account minimally for pertinent risk factors.
Risk adjustment for pediatric quality measures has received little scrutiny. The areas that have generated more attention (NICU and PICU outcomes) repre-sent acute, high-technology care, rarely needed by children. Nonetheless, these adjustment systems are, in general, well tested and validated. Given the well-developed status of ICU outcomes adjustment meth-ods, large-scale risk-adjusted comparisons of ICU outcomes could be used to motivate improvement in the quality of NICU and PICU care. On the other hand, more work is needed on applying risk adjust-ers in divadjust-erse practice settings, in different sociode-mographic populations, and across a variety of qual-ity dimensions. As new qualqual-ity measures are derived, developers must consider which risk factors are pertinent and produce risk-adjustment strategies appropriate to each particular measure.
future resource consumption and setting capitated payments. Considerable effort has gone into their development and statistical validation; they rely on readily available administrative databases. Testing their validity for risk-adjusting pediatric quality measures seems reasonable, especially for measures relating to resource consumption (eg, hospitalization rates). One attraction of these approaches for pediat-rics is their consideration of the totality of all diag-noses: because relatively few children might have one specific diagnosis, these risk adjusters overcome problems of small disease-specific sample sizes. The obvious drawback is their focus on resource con-sumption.
As noted at the outset, risk adjustment becomes necessary when people with different risk profiles are not assigned randomly to the “unit of observa-tion” (ie, the groups whose performance one wants to compare). Relevant units of observation for chil-dren may differ from those for adults. For chilchil-dren we may care more about schools, health department jurisdictions, provider groups, or communities than hospitals. In addition, we need to better understand the impact on health risks and thus risk adjustment of the large numbers of children with no health care or with care only for prevention and/or acute illness or injuries. Given current data and information sys-tems, it may prove impossible to capture pertinent risk factors for children across different units of ob-servation. Therefore, research must consider how unmeasured risk factors affect perceptions of relative quality across diverse units of observation.
Closely related is the need to know more about the role of family and community characteristics as risk adjusters for pediatric quality measures. As noted above, children depend heavily on their families and communities, and these factors therefore are poten-tially important risk adjusters. Information from the census and other sources can provide important community information. Parental information may be available in the same data sets as child informa-tion when parents and children are insured by the same plan. When parents report on quality, parents can answer questions about themselves and their families.
Linking data systems among institutions that could provide information about child health, such as schools, public health departments, and children’s hospitals, would be an enormous undertaking. How-ever, sharing data will be important to obtaining community-level views of child health quality and various risk factors. Defining relevant roles for vari-ous organizations (eg, school and public health de-partments, Medicaid programs, State Children’s Health Insurance Program, pediatric health centers, and children’s hospitals) and the data available from each might be a good starting point. These issues extend beyond the narrow topic of risk adjustment, highlighting barriers to obtaining comprehensive in-formation about child health, let alone the quality of pediatric care. Risk adjustment is limited inherently by inadequate or unavailable data.
A variety of technical issues also need more study. Many efforts to compare provider performance must
contend with small sample sizes; individual provid-ers frequently treat too few patients of a given type each year to generate accurate and stable estimates of their long-term average performance with such pa-tients. Better ways to estimate expected outcomes, such as empirical Bayes and hierarchical modeling, need more investigation. Simultaneous concerns about risk adjustment and small sample sizes can trap comparative efforts between “a rock and a hard place.” The main strategy for increasing the accuracy of estimates is including more cases, but this gener-ally increases population heterogeneity and makes risk adjustment both more necessary and more prob-lematic.
Thinking about valid risk adjustment is premature without assuring the validity of the quality measure to be risk adjusted. Risk adjustment aims to control for patient factors outside provider control so that residual differences in performance could, poten-tially, reflect quality differences. Few measures cur-rently used as ostensible QIs have undergone such scrutiny. For example, risk-adjusted NICU and PICU mortality rates are putative quality measures, but no one really knows whether differences across institu-tions reflect quality problems that, if resolved, would lower mortality rates. Research about validity of quality measures themselves should be linked di-rectly to risk-adjustment research: they should go hand in hand. Methods must be validated across time and settings of care (eg, academic health centers versus community facilities). Changes in patterns of care (eg, diagnostic testing or therapeutic interven-tions) and differences across settings could affect the type of data available for risk adjustment and the relationships between the risk factors and outcome of interest. Risk adjusters should generally be up-dated periodically, requiring an ongoing research commitment and investment.
Finally, we need to better understand how risk-adjusted quality measures will be used and dissem-inated. Risk adjustment will never be perfect; it is impossible to adjust for all relevant risk factors. Un-derstanding better how to disseminate admittedly imperfect information to clinicians, parents, plans, purchasers, and even older children requires more study. Given the ultimate goal of improving child health care, it is critical that battles over measure-ment methods not unduly divert resources and at-tention, as has happened sometimes in adult settings.
We thank Gerard Anderson, PhD, Anne C. Beal, MD, MPH, John Co, MD, MPH, Denise Dougherty, PhD, Jonathan Finkel-stein, MD, MPH, Paul Fishman, PhD, Christopher Forrest, MD, PhD, John Neff, MD, Murray M Pollack, MD, MBA, and Barbara Starfield, MD, MPH for their thoughtful comments. We dedicate this paper to the memory of our much-valued colleague Douglas Richardson, MD, MBA, who gave generously of his time and insight.
1. Ferris TG, Perrin JM, Manganello J, Chang Y, Causino N, Blumenthal D. Switching to gatekeeping: changes in expenditures and utilization for children.Pediatrics. 2001;108:283–290
adolescents covered by Medicaid.Arch Pediatr Adolesc Med. 1996;150: 503–507
3. Brook RH, McGlynn EA, Cleary PD. Quality of health care. Part 2: measuring quality of care [editorial].N Engl J Med. 1996;335:966 –970 4. Fiscella K, Franks P, Gold MR, Clancy CM. Inequality in quality:
ad-dressing socioeconomic, racial, and ethnic disparities in healthcare. JAMA. 2000;283:2579 –2584
5. Romano PS. Should health plan quality measures be adjusted for case mix? [letter; comment].Med Care. 2000;38:977–980
6. Richardson D, Tarnow-Mordi WO, Lee SK. Risk adjustment for quality improvement. Pediatrics. 1999;103(1). Available at: http:// pediatrics.aappublications.org/cgi/content/abstract/103/1/SE1/255 7. Practice Management Information Corp.International Classification of
Diseases, Ninth Revision, Clinical Modification. Fourth Ed. Los Angeles, CA: Practice Management Information Corp; 1995
8. Lamont EB, Lauderdale DS, Schilsky RL, Christakis NA. Construct validity of Medicare chemotherapy claims: the case of 5FU.Med Care. 2002;40:201–211
9. Cooper GS, Yuan Z, Stange KC, Dennis LK, Amini SB, Rimm AA. Agreement of Medicare claims and tumor registry data for assessment of cancer-related treatment.Med Care. 2000;38:411– 421
10. Ferris TG, Saglam D, Stafford RS, et al. Changes in the daily practice of primary care for children.Arch Pediatr Adolesc Med. 1998;152:227–233 11. Stafford RS, Saglam D, Causino N, et al. Trends in adult visits to
primary care physicians in the United States.Arch Fam Med. 1999;8: 26 –32
12. Alessandrini EA, Shaw KN, Bilker WB, Schwarz DF, Bell LM. Effects of Medicaid managed care on quality: childhood immunizations. Pediat-rics. 2001;107:1335–1342
13. Wood D, Donald-Sherbourne C, Halfon N, et al. Factors related to immunization status among inner-city Latino and African-American preschoolers.Pediatrics. 1995;96:295–301
14. Fiscella K, Franks P. Impact of patient socioeconomic status on physi-cian profiles: a comparison of census-derived and individual measures. Med Care. 2001;39:8 –14
15. Committee on Quality Health Care in America, Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC: National Academy Press; 2001
16. Beal AC, Co JPT, Dougherty D, Jorsling T, Kam J, Perrin J, Palmer RH. Quality measures for children’s health care.Pediatrics.2004;113:199 –209 17. Weiner JP, Starfield BH, Steinwachs DM, Mumford LM. Development and application of a population-oriented measure of ambulatory care case-mix.Med Care. 1991;29:452– 472
18. Ash AS, Porell F, Gruenberg L, et al.An Analysis of Alternative AAPCC Models Using Data From the Continuous Medicare History Sample. Waltham, MA: HCFA University Health Policy Research Consortium, Brandeis and Boston University; 1986
19. Kronick R, Zhou Z, Dreyfus T. Making Risk Adjustment Work for Everyone.Inquiry. 1995;32:41–55
20. Muldoon JH, Neff JM, Gay JC. Profiling the health service needs of populations using diagnosis-based classification systems.J Ambul Care Manage. 1997;20:1–18
21. Shenkman EA, Breiner JR. Characteristics of risk adjustment systems. Gainesville, FL: Institute for Child Health Policy; 2001
22. van den Akker M, Buntinx F, Metsemakers JF, Roos S, Knottnerus JA. Multimorbidity in general practice: prevalence, incidence, and determi-nants of co-occurring chronic and recurrent diseases.J Clin Epidemiol. 1998;51:367–375
23. Salem-Schatz S, Moore G, Rucker M, Pearson SD. The case for case-mix adjustment in practice profiling. When good apples look bad.JAMA. 1994;272:871– 874
24. Kuhlthau KA, Ferris TG, Beal AC, Gortmaker SL, Perrin JM. Who cares for Medicaid-enrolled children with chronic conditions?Pediatrics. 2001; 108:906 –912
25. Forrest CB, Reid RJ. Prevalence of health problems and primary care physicians’ specialty referral decisions.J Fam Pract. 2001;50:427– 432 26. Perrin JM, Kuhlthau K, Gortmaker SL, Beal AC, Ferris TG. Generalist
and subspecialist care for children with chronic conditions.Ambul Pe-diatr. 2003. In press
27. Shields AE, Finkelstein JA, Comstock C, Weiss KB. Process of care for Medicaid-enrolled children with asthma: served by community health centers and other providers.Med Care. 2002;40:303–314
28. Von Korff M, Wagner EH, Saunders K. A chronic disease score from automated pharmacy data.J Clin Epidemiol. 1992;45:197–203
29. Fishman PA, Shay DK. Development and estimation of a pediatric chronic disease score using automated pharmacy data.Med Care. 1999; 37:874 – 883
30. Richardson DK, Tarnow-Mordi WO, Escobar GJ. Neonatal risk scoring
systems. Can they predict mortality and morbidity?Clin Perinatol. 1998; 25:591– 611
31. Network IN. The CRIB (clinical risk index for babies)—a tool for as-sessing initial neonatal risk and performance of neonatal intensive care units.Lancet. 1993;342:193–198
32. Richardson DK, Gray JE, McCormick MC, Workman K, Goldmann DA. Score for neonatal acute physiology: a physiologic severity index for neonatal intensive care.Pediatrics. 1993;91:617– 623
33. Richardson DK, Phibbs CS, Gray JE, McCormick MC, Workman-Daniels K, Goldmann DA. Birth weight and illness severity: indepen-dent predictors of neonatal mortality.Pediatrics. 1993;91:969 –975 34. Pollack MM, Koch MA, Bartel DA, et al. A comparison of neonatal
mortality risk prediction models in very low birth weight infants. Pediatrics. 2000;105:1051–1057
35. Horbar JD, Onstad L, Wright E. Predicting mortality risk for infants weighing 501 to 1500 grams at birth: a National Institutes of Health Neonatal Research Network report.Crit Care Med. 1993;21:12–18 36. Horbar JD. The Vermont Oxford Network: evidence-based quality
im-provement for neonatology. Pediatrics. 2002;103(1). Available at: http://pediatrics.aappublications.org/cgi/content/abstract/103/1/ SE1/350
37. Roblin DW, Richardson DK, Thomas E, et al. Variation in the use of alternative levels of hospital care for newborns in a managed care organization.Health Serv Res. 2000;34:1535–1553
38. Phibbs CS, Bronstein JM, Buxton E, Phibbs RH. The effects of patient volume and level of care at the hospital of birth on neonatal mortality. JAMA. 1996;276:1054 –1059
39. Pollack MM, Patel KM, Ruttimann UE. PRISM III: an updated Pediatric Risk of Mortality score.Crit Care Med. 1996;24:743–752
40. Heard CM, Fletcher JE, Papo MC. A report of the use of the Dynamic Objective Risk Assessment (DORA) score in the changing pediatric intensive care environment.Crit Care Med. 1998;26:1593–1595 41. Pollack MM, Cuerdon TT, Patel KM, Ruttimann UE, Getson PR,
Leve-town M. Impact of quality-of-care factors on pediatric intensive care unit mortality.JAMA. 1994;272:941–946
42. Ruttimann UE, Pollack MM. Variability in duration of stay in pediatric intensive care units: a multiinstitutional study.J Pediatr. 1996;128:35– 44 43. Erickson LC, Wise PH, Cook EF, Beiser A, Newburger JW. The impact of managed care insurance on use of lower-mortality hospitals by children undergoing cardiac surgery in California.Pediatrics. 2000;105: 1271–1278
44. Jenkins KJ, Newburger JW, Lock JE, Davis RB, Coffman GA, Iezzoni LI. In-hospital mortality for surgical repair of congenital heart defects: preliminary observations of variation by hospital caseload.Pediatrics. 1995;95:323–330
45. Jenkins KJ, Gauvreau K, Newburger JW, Spray TL, Moller JH, Iezzoni LI. Consensus-based method for risk adjustment for surgery for con-genital heart disease.J Thorac Cardiovasc Surg. 2002;123:110 –118 46. Hannan EL, Racz M, Kavey RE, Quaegebeur JM, Williams R. Pediatric
cardiac surgery: the effect of hospital and surgeon volume on in-hospital mortality.Pediatrics. 1998;101:963–969
47. Jenkins PC, Flanagan MF, Jenkins KJ, et al. Survival analysis and risk factors for mortality in transplantation and staged surgery for hypo-plastic left heart syndrome.J Am Coll Cardiol. 2000;36:1178 –1185 48. Mavroudis C, Jacobs JP. Congenital heart disease outcome analysis:
methodology and rationale [letter, comment].J Thorac Cardiovasc Surg. 2002;123:6 –7
49. O’Connor GT, Plume SK, Olmstead EM, et al. A regional intervention to improve the hospital mortality associated with coronary artery bypass graft surgery. The Northern New England Cardiovascular Disease Study Group.JAMA. 1996;275:841– 846
50. Homer CJ, Fowler FJ, Gallagher PM, et al. The Consumer Assessment of Health Plan Study (CAHPS) survey of children’s health care.Jt Comm J Qual Improv. 1999;25:369 –377
51. Carman KL, Short PF, Farley DO, Schnaier JA, Elliott DB, Gallagher PM. Epilogue: Early lessons from CAHPS Demonstrations and Evaluations. Consumer Assessment of Health Plans Study. Med Care. 1999;37(3 suppl):MS97–MS105
52. Kim M.Adjusting Pediatric CAHPS Scores to Ensure Fair Comparison of Health Plan Performance [dissertation]. Cambridge, MA: Harvard University; 2002
53. Zaslavsky AM, Zaborski LB, Ding L, Shaul JA, Cioffi MJ, Cleary PD. Adjusting performance measures to ensure equitable plan comparisons. Health Care Financ Rev.2001;22:109 –126
Karen Kuhlthau, Timothy G. G. Ferris and Lisa I. Iezzoni
Risk Adjustment for Pediatric Quality Indicators
Updated Information &
including high resolution figures, can be found at:
This article cites 45 articles, 10 of which you can access for free at:
This article, along with others on similar topics, appears in the
Permissions & Licensing
in its entirety can be found online at:
Information about reproducing this article in parts (figures, tables) or
Karen Kuhlthau, Timothy G. G. Ferris and Lisa I. Iezzoni
Risk Adjustment for Pediatric Quality Indicators
located on the World Wide Web at:
The online version of this article, along with updated information and services, is
by the American Academy of Pediatrics. All rights reserved. Print ISSN: 1073-0397.