• No results found

Evaluation of the Agency for Healthcare Research and Quality Pediatric Quality Indicators

N/A
N/A
Protected

Academic year: 2020

Share "Evaluation of the Agency for Healthcare Research and Quality Pediatric Quality Indicators"

Copied!
11
0
0

Loading.... (view fulltext now)

Full text

(1)

ARTICLE

Evaluation of the Agency for Healthcare Research

and Quality Pediatric Quality Indicators

Matthew C. Scanlon, MDa, J. Mitchell Harris, II, PhDb, Fiona Levy, MD, MBAc, Aileen Sedman, MD, FAAPd

aDivision of Pediatric Critical Care, Medical College of Wisconsin, Milwaukee, Wisconsin;bNational Association of Children’s Hospitals and Related Institutions, Alexandria,

Virginia;cDivision of Critical Care Medicine, University of Texas Southwestern Medical Center, Dallas, Texas;dDivision of Pediatric Nephrology, Department of Pediatrics,

University of Michigan Health System, Ann Arbor, Michigan

Financial Disclosure: Dr Sedman is a paid medical advisor to the National Association of Children’s Hospitals; the other authors have indicated they have no financial relationships relevant to this article to disclose.

What’s Known on This Subject

PDIs were created by the AHRQ in 2006 on the basis of expert opinion. The purpose was to define potentially preventable complications in hospitalized children, such as noso-comial infections, iatrogenic pneumothorax, hemorrhage, and decubiti.

What This Study Adds

This study is the first to use the AHRQ quality indicator software and apply it to 76 children’s hospitals to calculate rates of the events. Chart review was conducted by pediatric clinicians to calculate the accuracy of the indicators as to whether the complication occurred before hospitalization and the rate of preventable occurrences.

ABSTRACT

OBJECTIVES.Pediatric quality indicators were developed in 2006 by the Agency for Healthcare Research and Quality to identify potentially preventable complications in hospitalized chil-dren. Our objectives for this study were to (1) apply these algorithms to an aggregate children’s hospital’s discharge abstract database, (2) establish rates for each of the pediatric quality indicator events in the children’s hospitals, (3) use direct chart review to investigate the accuracy of the pediatric quality indicators, (4) calculate the number of complications that were already present on admission and, therefore, not attributable to the specific hospital-ization, and (5) evaluate preventability and calculate positive predictive value for each of the indicators. In addition, we wanted to use the data to set priorities for ongoing clinical investigation.

METHODS.The Agency for Healthcare Research and Quality pediatric quality indicator algo-rithms were applied to 76 children’s hospital’s discharge abstract data (1 794 675 discharges) from 2003 to 2005. Rates were calculated for 11 of the pediatric quality indicators from all 3 years of discharge data: accidental puncture or laceration, decubitus ulcer, foreign body left in during a procedure, iatrogenic pneumothorax in neonates at risk, iatrogenic pneumotho-rax in nonneonates, postoperative hemorrhage or hematoma, postoperative respiratory failure, postoperative sepsis, postoperative wound dehiscence, selected infections caused by medical care, and transfusion reaction. Subsequently, clinicians from 28 children’s hospitals reviewed 1703 charts in which complications had been identified. They answered questions as to correctness of secondary diagnoses that were associated with the indicator, whether a

complication was already present on admission, and whether that complication was preventable, nonpreventable, or uncertain.

RESULTS.Across 3 years of data the rates of pediatric quality indicators ranged from a low of 0.01/1000 discharges for transfusion reaction to a high of 35/1000 for postoperative respiratory failure, with a median value of 1.85/1000 for the 11 pediatric quality indicators. Indicators were often already present on admission and ranged from 43% for infection caused by medical care to 0% for iatrogenic pneumothorax in neonates, with a median value of 16.9%. Positive predictive value for the subset of pediatric quality indicators occurring after admission was highest for decubitus ulcer (51%) and infection caused by medical care (40%). Because of the very large numbers of cases identified and its low preventability, the indicator postoperative respiratory failure is particularly problematic. The initial definition includes all children on ventilators postoperatively for ⬎4 days with few exclusions. Being on a ventilator for 4 days would be a normal occurrence for many children with extensive surgery; therefore, the majority of the time does not indicate a complication and makes the indicator inappropriate.

CONCLUSIONS.A subset of pediatric quality indicators derived from administrative data are reasonable screening tools to help hospitals prioritize chart review and subsequent improvement projects. However, in their present form, true preventability of these complications is relatively low; therefore, the indicators are not useful for public hospital comparison. Identifying which complications are present on admission versus those that occur within the hospitalization will be essential, along with adequate risk adjustment, for any valid comparison between institutions. Infection caused by medical care and decubitus ulcers are clinically important indicators once the present-on-admission status is determined. These complications cause significant morbidity in hospitalized children, and research has shown a high level of preventability. The pediatric quality indicator software can help children’s hospitals objectively review their cases and target improvement activities appropriately. The postoperative-respiratory-failure indicator does not represent a complication in the majority of cases and, therefore, should not be included for hospital screening or public comparison. Chart review should become part of the development process for quality indicators to avoid inappropriate conclusions that misdirect quality-improvement resources.Pediatrics2008;121:e1723–e1731

www.pediatrics.org/cgi/doi/10.1542/ peds.2007-3247

doi:10.1542/peds.2007-3247

Key Words

quality, indicator, AHRQ, complications

Abbreviations

AHRQ—Agency for Healthcare Research and Quality

PSI—patient safety indicator PDI—pediatric quality indicator POA—present on admission PPV—positive predictive value

Accepted for publication Feb 25, 2008

Address correspondence to Aileen Sedman, MD, FAAP, University of Michigan Health System, Ann Arbor, MI 48109-0825. E-mail: asedman@umich.edu

(2)

H

EALTH CARE QUALITYand safety metrics have gained increasing importance in the last decade with the growing attention on quality-based regulations, public reporting, and pay for performance. For instance, as of December 2006, 27 states had regulations related to mandatory reporting of adverse medical events to state agencies.1Similarly, the Center for Medicare and Med-icaid Services has stated its intent to use quality mea-sures to guide reimbursement to hospitals.2

In response to the growing demand for quality metrics, a number of agencies and groups have advanced new indicators for measuring quality and safety. These include the Joint Commission’s National Patient Safety Goals and the Agency for Healthcare Research and Quality’s (AHRQ’s) patient safety indicators (PSIs), inpatient quality indicators, and pediatric quality indicators (PDIs).3,4

Central to the use of any of these metrics is the premise that the measures are meaningful for use with the targeted population. That is, a set of measures in-tended to accurately reflect the quality of a pediatric population should be valid and reliable and, ideally, should guide both improvement and accountability ac-tivities.5,6That a published measure set is a valid assess-ment of quality care should not be assumed; evaluations of the AHRQ PSIs in the pediatric population revealed potential value in some areas for guiding improvement projects, yet there were significant issues that limited their utility for public reporting.7,8

Subsequent to the release of the AHRQ PSIs and follow-up published reviews,7,8a pediatric-specific set of measures was released by the AHRQ.9 The PDIs were designed exclusively for pediatric patients and intended to “screen for problems that pediatric patients experi-ence as a result of exposure to the health care system and that may be amenable to prevention by changes at the system or provider level.”9 The aim of this project was to assess the validity and reliability of the AHRQ provider-level PDIs in a cohort of children’s hospitals by using chart-review methodology.

METHODS

AHRQ PDI software was applied to the National Associ-ation of Children’s Hospitals and Related Institutions case-mix data set that included 1 794 675 pediatric hos-pital discharges from 2003 through the end of 2005 from 76 hospitals. The indicators analyzed were (1) accidental puncture or laceration, (2) decubitus ulcer, (3) foreign body left in during a procedure, (4) iatrogenic pneumo-thorax in neonates, (5) iatrogenic pneumopneumo-thorax in nonneonates, (6) postoperative hemorrhage and hema-toma, (7) postoperative respiratory failure, (8)

postop-erative sepsis, (9) postoppostop-erative wound dehiscence, (10) selected infection caused by medical care, and (11) transfusion reaction.

The PDIs use a logistic regression risk-adjustment model that takes into account the age, gender, reason for admission (diagnosis-related group), and comorbidities of individual cases. Regression coefficients used in the models were developed by using only pediatric cases from Health Care Utilization Project state inpatient da-tabases, and there is a different set of coefficients for each individual PDI. Previously, any AHRQ quality in-dicators that could have been applied to pediatric cases used the same coefficients developed by using an all-age database, so this method is an improvement. There are also some indicator-specific stratifications that are part of the AHRQ PDI software that attempt to further the abil-ity to compare indicator rates across providers9(Table 1). We calculated actual number and rates of PDI events per 1000 discharges. Discharges flagged by the PDI algo-rithm as having a potentially preventable event were identified; then, lists of those discharges were sent to 28 children’s hospitals (see “Acknowledgments”) that had volunteered to participate in a detailed chart review. In 2006, the mean number of admissions per hospital in the 28 hospitals was 11 696, and 82% of the hospitals have been trauma certified.

For each PDI there were 10 chart reviews by each of the 28 hospitals. Hospitals were asked to review the most recent cases first; therefore, case selection was chrono-logical and not random. However, for a number of PDIs, some hospitals did not have 10 cases over 3 years and, therefore, reviewed fewer cases. Area-level PDIs (asthma admission rate, diabetes short-term complica-tion rate, gastroenteritis admission rate, perforated ap-pendix admission rate, and urinary tract infection ad-mission rate) were excluded from this study, as were the 2 cardiac surgery provider-level PDIs (cardiac surgery volume and mortality rates) because the cardiac physi-cians were performing a separate analysis of those indi-cators to compare them to physiologic databases. A total of 1703 charts representing 1890 potentially preventable events were reviewed by the participating 28 children’s hospitals.

For each PDI-positive screen, the participants at each hospital received patient and encounter identifiers. For each of the reviewed cases, the clinician obtained a core data set including:

● patient demographic data (age, gender, mortality, and length of stay);

TABLE 1 Additional PDI-Specific Stratifications

Accidental puncture or laceration: risk category based on number of procedures, type (diagnostic versus therapeutic), and intensity (minor versus major) Decubitus ulcer: 2 stratification categories—high-risk patients (eg, paralysis, spina bifida, patients on mechanical ventilation) and all other patients Iatrogenic pneumothorax in neonates at risk: stratification categories—birth weight in 500-g increments

Postoperative hemorrhage and hematoma: 2 stratification categories—high-risk patients (coagulapathy) and all other patients

Postoperative sepsis: 3 stratification categories—high-risk patients, intermediate-risk patients (based on diagnoses that depress immunoresponse), and all other patients Selected infection caused by medical care: 3 stratification categories-high risk patients, intermediate-risk patients (based on diagnoses that depress immunoresponse),

(3)

● the coded principal diagnosis and all secondary diag-noses; and

● the coded safety event identified by the software. For each of the reviewed PDIs, a set of general and PDI-specific questions were applied. These questions were developed by a panel of pediatric experts before the initiation of data collection and included a clinical as-sessment of preventability. If the complication was deemed “not preventable,” the reviewer was asked to list the reason why. There was no formal training in iden-tifying preventability, although discussions took place during teleconferences, and problematic cases were used as examples. After completion of review, the data were submitted back to the National Association of Children’s Hospitals and Related Institutions for analysis without inclusion of any protected health information.

Descriptive and comparative analysis was per-formed by using SPSS Base 12.0 (SPSS Inc, Chicago, IL). Cases that were deemed present on admission (POA) were removed from the analysis of preventabil-ity. For example, a child who had a chronic Broviac line in place and was admitted with an infection in the line would have “infection caused by medical care” counted as already present (POA) and would not be counted as occurring within the hospitalization; there-fore, such an event was not preventable during the hospitalization. There were also cases for which sec-ondary complication codes were recorded incorrectly, such as when an absorbable suture was purposely left in and then coded as a “foreign body left in during procedure.” In both these examples, the statistics were applied after the incorrect cases were removed. Posi-tive predicPosi-tive values (PPVs) were calculated for each PDI by dividing the true-positive results over the sum of the true-positive and false-positive results.10In each case, the lower PPV estimates used only those events identified as clearly preventable in the calculation. In the high estimates, those events classified as being of uncertain preventability were included with the clearly preventable in the numerator of the equation.

RESULTS

Numerical results are displayed in Table 2. The rate of events per 1000 discharges for the 3 years in the 76-hospital database ranged from a low of 0.1/1000 for PDI13 (transfusion reactions) to a high of 35/1000 for PDI9 (postoperative respiratory failure). The median rate for all 11 PDIs was 1.82/1000, with only 2 PDIs having a rate of⬎8/1000. Thus, 9 of 11 PDIs are low volume or extremely low volume. The number of events reviewed by the 28 participating hospitals ranged from 7 for PDI13 (transfusion reaction) to 295 for PDI1 (accidental punc-ture or laceration). Hospitals were limited to reviewing their own charts, and if they had no or only a few cases to review, then they could not complete 10 charts for each PDI. Nonetheless, chart reviews were completed on ⬎7% of the database’s total number of identified PDI events, ranging from 51% for PDI11 (postoperative wound dehiscence) to 3% for PDI12 (selected infections caused by medical care).

Chart reviewers were required to be clinicians (ie, physicians or nurses with clinical experience). Discus-sions about criteria for preventability were held during group teleconferences with example cases reviewed. Re-viewers were asked to justify why cases were rated as nonpreventable in the data-collection tool. No formal interrater reliability tests were performed.

The AHRQ PDIs are identified on the basis of coded secondary diagnoses or procedures. Therefore, for each reviewed PDI, clinicians were asked to indicate whether the secondary diagnosis or procedure associated with the identified event was correct. The rate of incorrect sec-ondary diagnoses/procedures ranged from 0% for PDI13 (transfusion reaction) to 25.7% for PDI9 (postoperative respiratory failure) (Table 2).

For those cases in which the secondary diagnosis or procedure associated with the identified event was cor-rect, investigators were then required to identify whether the event that indicated a positive PDI screen was present at the time of admission to the hospital. This is critical, because a majority of hospitalized children have chronic disease and, therefore, often have chronic intravenous lines in place that become infected at home or have decubiti that form outside the hospital. These are important clinical problems; however, the PDI system is meant to count problems that occur because of inade-quate hospital care. Therefore, conditions that are POA must be removed from the data to give hospitals appro-priate information. This POA status ranged from 0% in PDI4 (iatrogenic pneumothorax in neonates at risk) to 43% for PDI12 (infection caused by medical care) (Table 2).

Finally, for those cases in which the event was not POA, investigators were charged with determining whether the potential events identified by the PDI soft-ware were preventable. Events were classified as clearly preventable, clearly not preventable, or uncertain. Events identified as clearly preventable ranged from 0% for PDI13 (transfusion reaction) to 51.45% for PDI2 (decubitus ulcer). Events identified as clearly not pre-ventable ranged from 19.31% for PDI12 (selected infec-tions caused by medical care) to 80% for both PDI4 (iatrogenic pneumothorax in neonates at risk) and PDI13 (transfusion reaction).

For each PDI chart review, clinicians were asked to comment on problems with the indicators to determine if the indicator actually captured an event and why. If the clinicians categorized the event as nonpreventable, they were asked why they felt the event occurred. The comments are summarized for each PDI in the para-graphs below. Brief definitions of the denominator and numerator are included. The full definitions are avail-able in the online manual.9The total number of events in 76 children’s hospitals over 3 years is listed after the title of each PDI. The rates per 1000 discharges, rates per year in 76 children’s hospitals, and average rate per children’s hospital per year are listed in Table 2.

PDI1: Accidental Puncture/Laceration (n3168)

(4)
(5)

had a secondary diagnosis code for accidental puncture or laceration during a procedure.

Seven percent of the children had the problem before they were admitted to the hospital (ie, POA). This is common in children’s hospitals, to which many patients are referred for complications that occur in other insti-tutions. Clinician reviewers felt that these events were not clearly preventable 68% of the time. Of 295 events, 20 occurred during lysis of adhesions, for which opera-tive notes described significant scarring present in chil-dren who had multiple surgeries before this particular event. Thirty-eight percent of the time, the laceration occurred during gastrointestinal procedures and 29% of the time during cardiovascular or thoracic surgery. Com-ments often included documentation that these were children who had congenital abnormalities such as gas-troschisis, omphalocele, diaphragmatic hernias, cloacal defects, and cardiac defects who were coming back into the hospital for 1 of multiple procedures that had previ-ously involved significant scarring. Although the lacer-ation or puncture was not a therapeutic part of the surgery, clinicians felt that they may have been unavoid-able for the surgeon to do what was necessary. Incidents that were clearly listed as preventable were often com-plications of line or device placements that subsequently punctured vessels, lungs, or the gastrointestinal tract.

PDI2: Decubitus Ulcer (n1688)

The denominator is all medical and surgical patients aged 0 to 17 years with a length of stay⬎5 days, and the numerator is any patient with decubiti noted on his or her record. In 40% of the cases, children were identified as having the decubiti before they were admitted; there-fore, the event was not preventable within the hospital-ization. For events not determined to be POA, clinician reviewers listed the decubiti as not clearly preventable during the current hospitalization 49% of the time. A number of cases were questioned by the clinicians be-cause of confusion at what level skin disruption actually should be defined as a decubitus. For example, a child with myelogenous leukemia came in with cutaneous involvement and subsequently had skin breakdown that was coded as decubitus in the chart but was not consid-ered preventable by the clinician reviewer. Also, there is confusion between skin irritation or breakdown occur-ring because of irritation from friction or tape and actual decubiti, which clinicians usually associate with pressure injury. Therefore, especially in small infants, there is significant variation in coding. A number of decubiti occurred on the occiput of infants after cardiac surgery or who were on extracorporeal membrane oxygenation, but the majority occurred on the sacrum or heels. Al-though this study did not include what percentage of children needed skin grafts and extensive therapy to recover, we know from other chart reviews that this is the case for many children who suffer from decubiti.

PDI3: Foreign Body Left in During Procedure (n153) The denominator is all medical and surgical patients aged⬍17 years, and the numerator is those individuals

who are coded to have a foreign body accidentally left in during a procedure. The indicator was POA 19% of the time, often a result of transfers because of complications at other hospitals.

This is a rare event (0.089/1000 patients) for which reviewers felt the trigger code was incorrect 22% of the time and was felt to be preventable only 44% of the time. A majority of these events do not represent actual foreign bodies (such a sponges “forgotten” by the sur-geon); instead, they represent catheters, screws, drains, etc that “broke off” during procedures or drains or su-tures that were purposely left in the body. Of 72 cases, 5 actually involved retained sponges, several of which were found at sponge count and removed before the end of anesthesia; therefore, they were not truly left in the body postoperatively, although they prolonged the op-erative time. Over a 3-year period, reviews indicated that only 3 cases occurred in which a physician truly forgot a foreign body in the patient. These were often noted to include errors in the sponge count and were later found incidentally on radiograph. Of more impor-tance, these data highlight the fact that we must have a way of collecting and analyzing data concerning equip-ment that breaks off during normal usage so that the manufacturers can be made aware of potentially resolv-able defects.

PDI4: Iatrogenic Pneumothorax in Neonates at Risk (n65) The denominator is all neonates with a birth weight of ⱕ2500 g; infants who weighed⬍500 g were excluded. Also, children who had a code indicating thoracic or cardiac surgery were excluded. The numerator is those infants coded with the diagnosis of iatrogenic pneumo-thorax. There were no cases POA. Clinician reviewers felt that pneumothorax in premature infants was not clearly preventable 90% of the time in the charts re-viewed. The exclusions should leave primarily prema-ture infants who sustain barotrauma from high ventila-tor pressures. However, 3 of 11 pneumothoraxes that were reviewed occurred after patent ductus arteriosus repair; several others procedures included sternotomy for placement of pacemakers, and, therefore, should have been excluded because entering the chest was an expected part of the surgery.

PDI5: Iatrogenic Pneumothorax in Nonneonates (n646) The denominator is all medical and surgical admissions of children aged 0 to 17 years excluding neonates, and the numerator is any child with iatrogenic pneumotho-rax excluding those with any thoracic surgery.

(6)

iatrogenic, although the first occurrence was associated with cardiothoracic surgery. Thirty percent of the pneu-mothoraxes, the majority of which were associated with central line placement, were considered preventable.

PDI8: Postoperative Hemorrhage or Hematoma (n483) The denominator is restricted to patients aged 0 to 17 years who had elective (ie, scheduled) surgery; the nu-merator includes patients who had a hemorrhage docu-mented with either a procedure to control hemorrhage or drain a hematoma. Of these events, 2.8% were POA. Clinician reviewers felt that 87% were not clearly pre-ventable. More than half of the reviewed cases were of cardiac surgeries, most of which involved cardiopulmo-nary bypass, heparinization, and subsequent bleeding. Clinicians often stated that they were uncertain whether these hemorrhages/hematomas were preventable, not-ing the difficulty of the surgeries and the fact that bleed-ing was a known complication. Other cases that were coded as being uncertain or not preventable included resection of neoplastic tumors that were very vascular or reperformance of surgeries in children who had had severe congenital defects (eg, diaphragmatic hernia) and now had significant scarring when revisions were per-formed. Clinicians listed preventable hemorrhage when a procedure was not frequently associated with severe bleeding, such as cleft palate repair, tonsillectomy, or hernia repair, or when placement of lines caused signif-icant hemorrhage or hematoma.

PDI9: Postoperative Respiratory Failure (n6098)

The denominator is all surgical patients aged 0 to 17 years except for children with cardiac surgery and those who had a primary diagnosis of respiratory or circulatory disease. The numerator included patients who had elec-tive (scheduled) surgery and had mechanical ventilation for ⬎96 hours postoperatively or were required to be intubated 1 or 2 days after the index procedure for respiratory failure. Of these cases, 16.9% were POA, often transfers from other institutions.

This PDI had a very high rate of 35 cases per thou-sand. Clinicians felt that these events were not clearly preventable 86% of the time, many because children were expected to be on ventilators for ⱖ4 days if they had major thoracic (such as tracheal reconstruction), neurosurgical, abdominal, or spinal surgeries. Twenty eight of the 290 reviewed cases had dorsal spinal fusions with planned intubation of ⬎4days. A number of the neurosurgical cases involved removal of extensive tu-mors from patients who postoperatively had decreased consciousness or increased intracranial pressure and re-mained intubated for that reason. Even when clinicians thought the prolonged stay on a ventilator was prevent-able, it often involved trying to decide if the child had fluid overload postoperatively, requiring prolonged in-tubation, or was getting too much sedation (a difficult call when a child is in significant pain). In general, clinicians felt that this indicator, as it is presently de-fined, is not helpful as far as identifying a preventable complication in the pediatric setting.

PDI10: Postoperative Sepsis (n4367)

The denominator is all surgical patients aged 0 to 17 years (excluding neonates) with a hospital stay of ⬎4 days who did not have sepsis or infection coded as a principal diagnosis when they were admitted and sub-sequently had a code for sepsis in the secondary diagno-sis field. The numerator was any patient with 1 of 20 different codes that indicated sepsis, such as streptococ-cal septicemia or systemic inflammatory response syn-drome due to infectious process with organ dysfunction. Forty percent of the patients had infection of some type POA, and of the remaining events, clinicians thought that the sepsis was not preventable 74% of the time. Many were patients who were admitted for a procedure but had long-term severe disease (ie, malignancy, chronic peritoneal or hemodialysis, chronic tracheosto-mies, post liver or kidney transplant), had a procedure and in the process were found to have infection—many fungal infections. There was confusion when there was questionable bloodstream infection in a line postopera-tively but the child was not truly septic but simply ceiving a “septic workup.” Despite the false-positive re-sults and the confusion about line infections being defined as sepsis, clinicians felt that the infection was directly related to the procedure and could have been prevented 25% of the time. The clear-cut cases usually involved children who came in relatively well and sub-sequently developed wound or line infection and be-came septic. Considering there were 4367 cases identi-fied in our 3-year data set, preventing 25% of them (even taking into account the percentage with incorrect triggers and POA) would have saved 177 children per year from a significant complication.

PD 11: Postoperative Wound Dehiscence (n102)

The denominator is all abdominopelvic surgical patients aged 0 to 17 years except for those with lengths of stay of⬍2 days. There were also exclusions for infants with gastroschisis or omphalocele, with which there are often staged procedures for closure, and patients with immu-nodeficiencies. The numerator is patients with the code for reclosure of disruption of abdominal wall.

In the 3-year period there were 102 cases, and 10% were POA. Clinician reviewers thought that 66% of the remaining events were not clearly preventable. There were a number of patients with diaphragmatic hernia in which the wound was left open purposefully and was closed in stages when there was decreased swelling. Several clinicians noted that dehiscence occurred when children has severe crying and coughing, sometime oc-curring after extubation, and concluded that better se-dation and pain management might have prevented the dehiscence.

(7)

injec-tion, or transfusion or caused by a vascular device, im-plant, or graft. Forty-three percent were POA, often in children who had chronic lines or ports in place and were infected at home or in other facilities. These rep-resent an enormous problem, although they were not strictly preventable during their hospitalization.

The majority of the bloodstream infections occurred secondary to an intravenous line. Forty percent were felt to be clearly preventable by the reviewing clinicians; this represents⬃605 cases per year of children who have a preventable infection, even taking into account the per-centage with incorrect triggers and POA status.

PDI13 Transfusion Reaction (n15)

The denominator was all medical and surgical patients aged 0 to 17 years excluding neonates. The numerator is any patient with a code for ABO incompatibility reac-tion, Rh incompatibility reacreac-tion, or mismatched blood. Twenty-nine percent were POA. This is extremely rare (15 cases in 3 years in 76 hospitals), and none of the 7 cases reviewed were considered preventable; they were reactions to correctly typed blood known to occur even with the best typing available because of untypeable antigens or antibodies. The reactions were usually tran-sient fevers or rashes.

DISCUSSION

The need for improvement in the safety and quality of patient care is well established. Both children and adults have been identified as experiencing preventable harm in the inpatient environment.11–15In response, providers, payers and patients have sought to improve health care. The ability to accurately identify patient safety and qual-ity events is essential for guiding these improvement efforts while holding providers accountable for safe care. Children have unique characteristics that require pe-diatric-specific patient safety and quality indicators.16,17 To that end, a number of general and specific indicators have been suggested for use in children.7,9,18Study of the ARHQ PSIs suggests that there is potential utility in using some of these indicators to guide safety efforts.7,13 How-ever, chart review of the PSIs also revealed significant limitations that suggested limits to their appropriateness for public reporting.8

Recognizing the limitations of the PSI for a pediatric population, the AHRQ developed and released provider-and area-level PDIs. In keeping with other AHRQ quality indicators, the PDIs use administrative discharge data to identify events, which may represent preventable pa-tient harm. Conceptually, the PDIs could help direct organizations in their improvement efforts. In addition, government and private pay-for-performance initiatives might leverage the AHRQ PDIs as accountability mea-sures. Thus, we chose to perform a chart review of those admissions to a cohort of pediatric hospitals that were identified as having a positive screen using the provider-level PDIs.

Our analyses revealed several significant findings. First, the study confirmed the wide spectrum in the prevalence of PDI events with half of the PDIs being very

low-volume or rare events. Low volume is not a problem by itself if the event is clinically very significant and highly preventable, but this is not the case. Transfusion reactions occurred at a rate of 0.01 per 1000 hospital discharges, and none were found to be preventable. In contrast, the PDI screen identified postoperative sepsis and postoperative respiratory failure as more frequent events, with rates of 29.41 and 35.04 per 1000 dis-charges, respectively. This would suggest that prevent-able postoperative complications of respiratory failure and sepsis are not uncommon and would merit a com-mitment of resources to decrease their incidence. How-ever, both of these PDIs were judged to have low pre-ventability, especially postoperative respiratory failure, and so this might actually misdirect a lot of hospitals’ quality-improvement activities.

At the time of this study, the AHRQ PDI methodology did not account for the presence of the secondary diag-noses at the time of hospital admission. That is, a patient may be admitted to a tertiary hospital with a diagnosis that triggers the PDI screen, which may have been ac-quired elsewhere. However, the lack of robust methods for handling POA cases creates an opportunity for incor-rect attribution of harm to a hospital. This can best be appreciated with PDI2 (decubitus ulcer). The chart re-view revealed that nearly 60% of the ulcers identified with the software were present at the time of admission to the hospital identified as having the potential harm event. To be clear, these decubitus ulcers are very real and are sources of significant harm and cost to patients. However, the incorrect attribution of harm to a hospital that may have merely “inherited” this diagnosis may result in significant harm to the hospital, too. The plan for the Center for Medicare and Medicaid Services to no longer reimburse hospitals for decubitus ulcers acquired in hospitals beginning in October 2008 increases the stakes for correct assignment of these harm events.2,19 Medicare has implemented a new billing form, the UB-04, to replace the previous UB-92 form; with this change, a POA indicator has been added.20Updated ver-sions of the PDI will take advantage of these new fields and include improved ways of identifying conditions that are POA. Ideally, through improved documentation and thoughtful assessment of patient safety and quality screening tools, this issue will no longer limit the mean-ingfulness of tools such as the PDIs.

(8)

find-ings were then leveraged to calculate the PPV for each measure. Defined as true-positive results over the sum of true-positive and false-positive results, this epidemio-logic concept brings about a clinically relevant question: if a test is positive, how likely is it that a patient has the disease? In the case of the PDIs, the question becomes: if a PDI screen is positive, how likely is it that the patient has a preventable event? As was true throughout our evaluation of the PDIs, a large variation in the PPV was identified.

The variation in PPV was aggravated, in part, by the percentage of uncertain preventable events for any given PDI. A low and high range of PPVs was calculated for each of the PDIs by excluding and including events classified as uncertain preventability in the true-positive portion of the equation. If events classified as uncertain preventability are excluded from the true-positive por-tion of the PPV calculapor-tion, then only 1 of the PDIs, decubitus ulcer, had a PPV of ⬎50%. Or, for every 2 positive screens, only 1 indicated a preventable harm event to a patient. If the events classified as uncertain preventability are included in the true-positive portion of the PPV calculation, then 4 of the 11 studied PDIs have a PPV of ⬎75%. In the case of PDI12 (selected infections during medical care), the PPV was ⬎80%, which suggests that this may be a useful indicator to screen for preventable harm in patients.

There are some potential limitations to this study. First, there was no formal assessment of interrater reli-ability between the clinicians at the various centers who were performing the evaluations.

A second potential limitation lies in the lack of a strict definition for preventability between the reviewers. In part, this lack of standard definition may explain the theme of being unable to determine preventability. However, the patient-safety literature reflects similar methods for identifying preventability, with the excep-tion of using⬎1 reviewer to examine each chart.15,21,22 Despite the potential for either overassignment or un-derassignment of a given event as preventable on the basis of a single reviewer, the recurrent nature of assign-ment to an unpreventable category for some of the PDIs would suggest that the problem is related to PDIs falsely identifying harm events as preventable.

A third potential limitation is the study design and the decision to not evaluate charts that were not screened as positive for the PDIs. If studied, a determination of the true-negative and false-negative rates could be made. Instead, this limitation provides grounds for future work.

From the perspective of a patient or family member, this study could be viewed as largely irrelevant. If a child suffers harm during medical care, whether at a specific hospital or before admission or whether preventable or not, then the harm is a real entity to be dealt with by the patient, family, and caregivers. However, from the per-spective of payers, regulators, hospitals, and physicians, the findings of this study are important. As with our review of the AHRQ PSIs in a pediatric population, in-fections caused by medical care remain a significant

problem and, once POA is accounted for, could be pub-licly reported and compared between hospitals.

With this in mind, the low preventability coupled with the POA issue suggests that at the present time, AHRQ PDIs are unsuitable for public reporting. A focus on these indicators as a tool for accountability may result in penalizing hospitals for issues that are beyond their control and the opportunity cost of misallocating re-sources to improving the same issues.

Measurement of care to improve quality is an impor-tant and laudable endeavor. The AHRQ has used expert opinion to drive the evolution of the process. However, the experts on the AHRQ panel lacked a clear consensus as to what measures were accurate enough for public reporting. Our study reinforces that lack of consensus and confirms that chart review would be very useful during the development process and in the future should be a mandatory component in measure development before the release of new measures.

CONCLUSIONS

This evaluation of the AHRQ PDIs supports that infec-tions caused by medical care and decubitus ulcers re-main significant sources of preventable harm in hospi-talized children. Once POA status is consistently recorded, the data on these indicators will become much more accurate. Additional refinement of the indicator “infections caused by medical care” could specifically look at the rate of infection in central venous lines only, which could be carefully analyzed from administrative data. Decubitus ulcers will have to be rated according to an accepted scale of severity. These steps, along with adequate risk adjustment, would increase accuracy and clinical significance. At the present time, the indicator “postoperative respiratory failure” is inaccurate and should not be used.

Chart review should become an essential and man-datory component of the development of administra-tive-based quality indicators along with expert opinion. Until this is done, broad use of the indicators for public reporting and pay for performance may serve as its own source of preventable harm.

ACKNOWLEDGMENTS

(9)

Mor-gan Stanley Children’s Hospital of New York-Presbyte-rian (New York, NY), Cincinnati Children’s Hospital Medical Center (Cincinnati OH), Children’s Hospital of Philadelphia (Philadelphia, PA), Children’s Hospital of Pittsburgh (Pittsburgh, PA), Vanderbilt Children’s Hos-pital (Nashville, TN), Children’s HosHos-pital of Austin (Aus-tin, TX), Driscoll Children’s Hospital (Corpus Christi, TX), Children’s Medical Center of Dallas (Dallas, TX), Cook Children’s Medical Center (Fort Worth, TX), Texas Children’s Hospital (Houston, TX), Covenant Health Ser-vices (Lubbock, TX), Christus Santa Rosa Children’s Hos-pital (San Antonio, TX), Children’s HosHos-pital and Re-gional Medical Center, Seattle (Seattle, WA), and Children’s Hospital of Wisconsin (Milwaukee, WI). We also thank John Muldoon and Dr Marlene Miller for extensive support and comments and Charles Olson for preparation of the article.

REFERENCES

1. National Academy for State Health Policy. Patient safety tool-box for states. Available at: www.pstooltool-box.org. Accessed July 5, 2007

2. Center for Medicare and Medicaid Services. Details for: FY 2007 inpatient prospective payment system final rule top of form [press release]. Available at: www.cms.hhs.gov/apps/media/press/release. asp?Counter⫽1922. Accessed August 20, 2007

3. The Joint Commission. 2008 national patient safety goals. Available at: www.jointcommission.org/PatientSafety/NationalPatientSafety Goals. Accessed July 5, 2007

4. Agency for Healthcare Research and Quality. AHRQuality indicators: introduction. Available at: www.qualityindicators. ahrq.gov/introduction.htm. Accessed July 5, 2007

5. Palmer RH, Miller MR. Methodologic challenges in developing and implementing measures of quality for child health care.

Ambul Pediatr.2001;1(1):39 –52

6. Agency for Healthcare Research and Quality. Using measures. Avail-able at: www.qualitymeasures.ahrq.gov/resources/measure㛭use. aspx. Accessed June 15, 2007

7. Sedman A, Harris JM 2nd, Schulz K, et al. Relevance of the Agency for Healthcare Research and Quality patient safety indicators for children’s hospitals. Pediatrics. 2005;115(1): 135–145

8. Scanlon MC, Miller M, Harris JM, et al. Target chart review of pediatric patient safety events identified by the Agency for

Healthcare Research and Quality’s patient safety indicators methodology.J Patient Saf.2006;2(4):191–197

9. Agency for Healthcare Research and Quality. Pediatric quality indicators overview. Available at: www.qualityindicators.ahrq. gov/pdi㛭overview.htm. Accessed June 15, 2007

10. Altman DG, Bland JM. Diagnostic tests 2: predictive values.

BMJ.1994;309(6947):102

11. Brennan TA, Leape LL, Laird N, et al. Incidence of adverse events and negligence in hospitalized patients: results from the Harvard Medical Practice Study I.N Engl J Med.1991;324(6): 370 –376

12. Leape LL, Brennan TA, Laird NM, et al. The nature of adverse events in hospitalized patients: results from the Harvard Med-ical Practice Study II.N Engl J Med.1991;324(6):377–384 13. Miller M, Elixhauser A, Zhan C. Patient safety events during

pediatric hospitalizations. Pediatrics. 2003;111(6 pt 1): 1358 –1366

14. Kaushal R, Bates DW, Landrigan C, et al. Medication errors and adverse drug events in pediatric inpatients.JAMA.2001; 285(16):2114 –2120

15. Woods D, Thomas E, Holl J, Altman S, Brennan T. Adverse events and preventable adverse events in children.Pediatrics.

2005;115(1):155–160

16. Scanlon MC. Human factors and ergonomics in pediatrics. In: Carayon P, ed.Handbook of Human Factors and Ergonomics in

Health Care and Patient Safety. Mahwah, NJ: Lawrence Erlbaum

Associates; 2007

17. Mangione-Smith R, McGlynn EA. Assessing the quality of healthcare provided to children.Health Serv Res.1998;33(4 pt 2):1059 –1090

18. Scanlon MC, Mistry KP, Jeffries HE. Determining pediatric intensive care unit quality indicators for measuring pediatric intensive care unit safety. Pediatr Crit Care Med. 2007;8(2 suppl):S3–S10

19. Pear R. Medicare says it won’t cover hospital errors.New York

Times. August 19, 2007:1

20. Healthcare Information and Management Systems Society. UB-04 FAQs. Available at: www.himss.org/content/files/200612㛭UB-04㛭 FAQs.pdf. Accessed August 1, 2007

21. Temple ME, Robinson RF, Miller JC, Hayes JR, Nahata MC. Frequency and preventability of adverse drug reactions in pae-diatric patients.Drug Saf.2004;27(11):819 – 829

(10)

DOI: 10.1542/peds.2007-3247 originally published online May 12, 2008;

2008;121;e1723

Pediatrics

Matthew C. Scanlon, J. Mitchell Harris II, Fiona Levy and Aileen Sedman

Indicators

Evaluation of the Agency for Healthcare Research and Quality Pediatric Quality

Services

Updated Information &

http://pediatrics.aappublications.org/content/121/6/e1723 including high resolution figures, can be found at:

References

http://pediatrics.aappublications.org/content/121/6/e1723#BIBL This article cites 13 articles, 4 of which you can access for free at:

Subspecialty Collections

atistics_sub

http://www.aappublications.org/cgi/collection/research_methods_-_st

Research Methods & Statistics

b

http://www.aappublications.org/cgi/collection/medical_education_su

Medical Education

following collection(s):

This article, along with others on similar topics, appears in the

Permissions & Licensing

http://www.aappublications.org/site/misc/Permissions.xhtml in its entirety can be found online at:

Information about reproducing this article in parts (figures, tables) or

Reprints

(11)

DOI: 10.1542/peds.2007-3247 originally published online May 12, 2008;

2008;121;e1723

Pediatrics

Matthew C. Scanlon, J. Mitchell Harris II, Fiona Levy and Aileen Sedman

Indicators

Evaluation of the Agency for Healthcare Research and Quality Pediatric Quality

http://pediatrics.aappublications.org/content/121/6/e1723

located on the World Wide Web at:

The online version of this article, along with updated information and services, is

by the American Academy of Pediatrics. All rights reserved. Print ISSN: 1073-0397.

References

Related documents

The value of the assets is still measured by the acquisition cost (without revaluation) despite the plants‟ constant growth and transformations. At the end of the reporting

Patients and Methods: This was a 5 year prospective study comprised of thirty-one survivors of childhood acute lymphoblastic leukemia with a mean age of 11.8 (4.6) years,

In this previous human study involving young females, it was found that six weeks of aerobic dance exercise at three times per week, one hour per session combined with

Pusa 256 (bold seeded) and Pusa 261 (small seeded), raised under unirrigated and irrigated field conditions, were examined for Rubisco, soluble protein and N contents in the leaves

In this study, we describe our own technique with the use of a fl exible cannula and present adverse events and patient comfort in a large population of patients under- going

Figure 5 (A) Fluorescent microscopic images of FITC uptake by U87-MG glioma cells in vitro incubated with FITC solution (a), FITC loaded VPGs (b), FITC loaded PGA-ODA VPGs (c) and

The 42 CC15-MRSA-V+SCCfus in this study belonged to ST1535, harbored SCC mec V, agr II, and the novel SCCfus composite genetic element similar to the isolates obtained from retail

Studies were selected for inclusion in the meta-analysis according to the following criteria: 1) pretreatment NLR (baseline NLR) was measured from serum; 2) all