Auditing Effectiveness of Electronic Health Records Version of Sunday, May 17, 2009
Contact: Farrokh Alemi at
[email protected] Abstract
Aims: We propose to evaluate the impact of an organization’s electronic health records on cost and quality of care. The Recovery Act provides incentives for clinicians to adopt “certified” electronic health records and put it to “meaningful use.” Approximately, 20 billion dollars of incentives are provided. There is a time-sensitive need to examine if the stimulus funds are leading to the desired results. We propose to automatically collect and analyze the necessary data for evaluation of the impact of the EHR on cost and quality of care.
Approach: We will integrate data from the hospital’s (a) administrative claims data, (b) cost reports and (c) key managers’’ report of changes in organization of care to calculate risk-adjusted cost of care and risk-risk-adjusted inpatient quality indicators. A quasi-experimental design with multiple observations before and after implementation of the electronic health record will be used to display the changes in outcomes of care. If there has been a significant change in outcomes of care, then a causal analysis will be done to see if the change is due to the introduction of EHR. The analysis will control for possible alternative explanations of why changes in outcomes of care have occurred.
Specific Aims
The goal of this project is to accelerate the availability of data on impact of Electronic Health Records (EHRs) on cost and quality of care. The Recovery Act authorizes the
expenditure of nearly $20 billion dollars for adoption of EHRs. No clear method of
investigating the impact of these expenditures is available. Our proposed project creates the infrastructure needed so that the necessary data is collected and automatically made available across a large number of organizations as part of the implementation of electronic health records.
The specific aim of this project is to evaluate impact of EHR on cost and quality of care within one organization. The software relies on (1) administrative claims data, (2) cost reports to CMS and (3) input from key mangers within the organization. Key managers provide
information on data not available in cost reports and timing of implementation of various interventions, including the EHR. The data needed for the analysis is readily available in hospitals that have implemented EHR. We will use Activity Based Costing method to allocate cost reports made to the Center for Medicare and Medicaid Studies to specific departments within the organization that have implemented the EHR system. In this fashion, it is possible to know if within the departments that have implemented the EHR cost of care per patient has gone down.
We will also use the Hospital Quality Indicators developed by the Agency for Healthcare Research and Quality to measure changes in quality of care before and after implementation of the EHR. These data will be analyzed to (1) establish a statistically significant change in
alternative concurrent interventions. In particular, Causal Analysis (including examination of counterfactual claims) will be done to establish a link between the change in outcomes and the introduction of EHR. The project will test the software within the Washington Hospital Center. Research Design and Methods
Healthcare organizations have difficulty auditing the impact of electronic health records. First, it is not clear what an electronic health records is. It contains many systems and there are many levels for implementation of each system. Second, implementation of an EHR function does not mean that it is being used. Third, the start time of electronic health records is not clear as some patients may receive the service before others do. Fourth, there may be a lag for the effect of electronic health records as clinicians learn to use the system. Fifth, the patient outcomes are not clear as there are multiple measures of quality of care with different levels of sensitivity and specificity. Finally, sixth the impact on productivity is not clear as measures of cost of care are often not available within the EHR. The purpose of this proposal is to overcome these difficulties and put in place an infra-structure that will (1) enable health care organizations to collect the necessary data from extant sources, (2) assist in merging of the data within and across organizations, (3) enable causal analysis of comparative effectiveness of electronic health records.
The project will rely on the following sources of data available at most institutions that have implemented electronic health records:
1. Data on cost of care per patient:
a. Total cost of care within the organization will be based on the organization’s budget supplemented with market data for cost of buildings and donated services. b. The number of patients served will be based on reports of data within the
organization’s EHR as well as the organization’s annual budget reports.
2. The data for quality of care over time will be based on claims-based measures of quality problems developed by the Agency for Healthcare Quality and Research. These measures will be assessed based on data within the electronic health records. For years prior to implementation of electronic health record, these data will be based on administrative billing data.
3. The level and timing of the implementation of the electronic health record will be based on date of changes in functionality of the electronic health record as verified by extent of use of the function within the EHR.
4. A survey of healthcare managers will establish alternative explanations for changes in healthcare outcomes and dates of these changes.
Two distinct but highly related challenges must be addressed in order to analyze the proposed data. The first, data within the electronic health record must be prepared for analysis. There are many redundant data within electronic health record. These data items often contain missing information. The first step is to create new constructs that are more robust (i.e. have less missing values) and that integrate data from multiple fields within the electronic health record. To accomplish this, some solutions rely on syntax-based approaches, which can be effective only in controlled environments, while others employ sophisticated semantic approaches such as ontologies, which are more suitable for open world scenarios but don’t have a principled way of dealing with incomplete, uncertain data. A suitable and scalable solution is needed for consistently integrating multiple fields in and outside the electronic health record with incomplete, uncertain data. For example, the patient’s census is available in two different sources. It is available by counting the number of unique patients within EHR as well as through reports of the budget of the organization. These data, however, are not exactly the same
as not all patient data are entered into the EHR and estimates of billed care is often different from number of patients served. A probabilistic ontology will clarify the semantic meaning of these terms and how a consensus estimate can be obtained from these two overlapping fields of data.
The second step to analyze this data is to perform probabilistic causal analysis. The implementation of electronic health records is one of many events that might affect patient outcomes. A simple before and after study could erroneously attribute the change in outcomes to the electronic health record. In contrast, we propose a method of analysis that checks for causal link between the introduction of electronic health record and changes in health outcomes. This project will prepare the necessary software necessary to collect supplemental data and to report the data automatically. We will work with the EHR industry to understand how the software should be modified so that it can be used by their installed base.
The Approach
Study design: We propose to evaluate the impact of EHR through naturally occurring quasi-experiments, with intervention and control groups observed over several time periods. The intervention group will be patients whose data were maintained in the EHR. The control group will be the remaining patients, risk-adjusted for baseline differences between control and experimental group. If the control group does not exist (a situation in which all patients are using the EHR), a matched synthetic group is organized based on the features of the current patients and outcome of care prior to EHR implementation. Figure-1 shows an example. In this Figure, data are organized into monthly observations before and after the known dates of implementing a component of EHR. The time of implementation is shown as month zero. The X-axis shows time from the implementation. The Y-axis shows the cost of caring for an average patient (cost of care per patient is an example of an outcome examined. A complete list of outcomes examined and how these outcomes are measured are presented in a later section). In this example, the control group is constructed from the patients who are receiving care within the same organization but do not currently have access to the EHR. Their cost of care is risk-adjusted for differences in the characteristics of patients in the experimental and control groups. The line titled “Best of the control group” is the Lower Control Limit for the control group and is calculated as 3-standard deviations below the risk-adjusted average cost of care for patients in the control group. The worst of the control group is calculated as 3-standard deviations above the risk-adjusted average cost of the control group. If the EHR group is between the best and worst of the control group, then no statistically significant change has been made. If not, then there is a statistically significant change that might be attributed to the implementation of this component of EHR (a later section shows how alternative explanations are examined before making this causal attribution). Figure-1 shows that shortly after implementation of this component of EHR, the cost per patient increased; but 3-months later the cost per patient decreased.
Figure 1: Cost of Care per Patient after Implementation of a Component of EHR This study design allows one to detect if there has been a statistically significant change in care outcomes. The attribution of this change in outcomes to the introduction of EHR requires additional analysis. Key managers within the hospital will be asked to describe competing explanations that could explain the change in outcomes. A causal analysis is done to examine which of the competing explanations is most predictive of change in outcomes.
Procedure for Causal Analysis: In recent years, scientists have made significant progress in making causal inferences from observational data, the type of data in an electronic health record. 1,2 Investigators have developed statistical procedures to distinguish association between two variables from cause and effect. Of particular interest is the work of Judea Pearl, who has shown that causal analysis can be done through simultaneous equations as well as graphs and probability models. In causal analysis, several measures are examined that go beyond the typical association studies. These include:
(1) Precedence: Causes must precede effects. In our context, we need to examine the impact of EHR implementation on future outcomes.
(2) Mechanism: There must be a clear mechanism linking the cause to the effect. In our context, we plan to show the mechanism by showing how use of the EHR component (e.g. use of drug-drug interaction component) has affected treatment decisions (e.g. use of medications) and in turn led to better outcomes.
(3) Counterfactual: There must be evidence that improvements in outcomes are not possible if the cause is not present. In the context of EHR, the counterfactual can be tested by comparing observed improvements in outcomes while using EHR to risk-adjusted projection of outcomes when EHR was not used. The counterfactual assumption is met if observed outcomes are significantly better than forecasted outcomes.
(4) Control of Confounding: The statistical procedure used should control for alternative explanations (e.g. hiring of new clinicians, purchase of new equipment, changes in published recommended treatment) for why treatment decisions might have changed during the same time period when EHR was implemented. This is typically done by examining time periods in which EHR was implemented but the alternative intervention was not. The detail of when various alternative
explanations of changes in treatment were implemented is available through survey of key managers within the organization.
In causal analysis, unique procedures are needed in order to separate out the effects of various causes and explanations. Figure 2 shows how EHR might affect treatment decisions and in turn affect health care outcomes.
Treatment Decisions Treatment Outcomes Severity of Illness Alternative Interventions to Change Treatment Use of EHR Component Implementation of EHR
Figure 2: Impact of EHR on Health Outcomes
In causal analysis, simultaneous equations are used to link multiple causes (independent variables) to an effect (dependent variables). For example, an equation is set to model the influence of treatment decisions, Di, severity, Si, on patients’ outcomes, Oi:
Similarly, a separate equation is used to measure the impact of use of a specific component of EHR, Ui, alternative interventions to change treatment, Ai, and severity of the patient’s illness on decision to select a specific treatment:
The simultaneous analysis of both equations enables one to control for the confounding impact of severity on both treatment decisions and on treatment outcomes. It also allows one to separate the impact of alternative interventions from the impact of the EHR on treatment decisions. The proposed software will (1) report if there have been major changes in quality and outcomes of care, (2) report if these changes can be attributed to EHR implementation versus other contemporary alternative explanations. The reports from the software will be available as a dashboard for continuous monitoring of impact of EHR.
Measurement of EHR Implementation Stage: An electronic health record is a complex system with many components. It is important to accurately measure the implementation of the electronic health record. In the past, implementation has been defined based on self-reported surveys of availability of the software. Palacio, Harrison, and Garets suggest seven levels in implementation of EHRs: 3
0. None of the ancillary systems installed.
1. Ancillary radiology, laboratory and pharmacy installed
2. Clinical Data Repository, Controlled Medical Vocabulary, CDSS inference engine, and Document Imaging installed
3. Clinical Documentation (Flow sheets), CDSS error checking, PACS available outside radiology
4. Computerized Physician Order Entry, Clinical Decision Support (Clinical Protocols) available
6. Physician Documentation (Structured Templates), full CDSS variance and compliance, full R-PACS available
7. Medical record fully electronic, HCO able to contribute, CCD as a byproduct of EMR, and data warehousing/mining available
These seven stages of implementation of EHR are modified by us to reflect (1) variations among each department’s implementation of EHR, (2) difference in use versus availability of the component, and (3) differences in integration of data across systems. In order to improve the measure of implementation of electronic health records we will calculate for each department and within each stage, the percent of time the relevant data could and was available at the time of review by the clinician. In this sense, a system that is not partially integrated or that has significant lag in transfer of data will be considered to have not reached the higher stages of implementation.
Measurement of Cost of Care: The source of data for the measurement of cost of care per patient is the data reported by the health care organization to the Center for Medicare and Medicaid Services (CMS). Medicare requires institutional providers to submit an annual report through a Fiscal Intermediary to the Healthcare Provider Cost Reporting Information System (HCRIS). This report contains a standardized format for data on facility characteristics, utilization data, operating cost and charges by cost center (in total and for Medicare patients). We will use Activity Based Costing (ABC) procedures4,5 to allocate these costs to cost centers that have implemented EHR and costs centers that have not done so. These procedures allocate indirect costs to various cost centers based on different cost drivers:
• Total personnel cost is allocated to the cost center proportional to time spent by the clinicians within the cost center; this is determined by review of claims data.
• Within the personnel cost, the cost of organization’s overall management is allocated proportional to the budget of the cost centers. The cost of various centers is available through HCRIS.
• The cost of major equipment (e.g. computed axial tomography) is allocated proportional to billing for these services. The billing information is available through examination of claims data.
• The cost of operations is allocated to various cost centers proportional to their census. The census information is available through the claims data.
• The cost of major equipment is allocated to the cost centers based on the use of these equipments. For example, the cost of EHR is allocated proportional to use of EHR. The data on use of EHR by different cost centers is available through the EHR. For another example, the cost of Computed Axial Tomography is allocated to cost centers requesting these scans. Not all of the data needed for the analysis of the cost are available in HCRIS. For example, cost of buildings and volunteers are not reported in HCRIS, nor are they available in the electronic health record. Some organizations no longer pay for their land or building; many accounting reports do not reflect the economic value of volunteers. To assess the costs not reported to HCRIS, the software will survey key managers within the organization. Here are some examples of costs that are allocated based on survey data:
• The cost of minor equipment (e.g. office space) is allocated proportional to cost of renting office equipment. The number of offices and office equipment within them are collected through survey of key managers.
• The cost of building, including cost of maintenance and utilities, is allocated proportional to the square footage used by the cost center. When the cost of building is missing, it will be estimated based on cost of renting equivalent office space in the locality and the square footage used by the organization.
• The extent of use of volunteers is estimated through survey of key managers. The cost of volunteers is estimated proportional to the cost of equivalent paid personnel.
The cost per patient for a cost center in a particular month is determined by dividing the portion of the total cost of care allocated to the cost center by the census of the cost center. The ABC cost per patient is a comprehensive measure and reflects many components usually ignored in calculation of costs through other methods. For example, it includes the cost of time of clinicians who are being trained as well as the typical cost of trainer’s time. It includes the cost of vacation and idle time as well as direct cost of time of personnel. The ABC cost per patient is also a sensitive measure of cost and is affected by many factors. For example, if the use of EHR reduces the number of patients served, then cost per patient will increase. There are several reports that EHR may initially increase the cost of care. If use of EHR reduces the space allocation for medical records, then the allocation of building cost to specific cost centers changes and cost per patient will reflect these savings.
Measurement of Quality of Care: The measurement of quality of care is based on 28 inpatient quality of care indicators (e.g. mortality rate for patients with acute myocardial infarction, or cesarean delivery rate) developed by the Agency for Healthcare Research and Quality (AHRQ). In addition, we will include 27 patient safety indicators (e.g. foreign body left during procedures) also developed by AHRQ. These indicators allow measurement of potentially avoidable adverse hospital outcomes, inappropriate utilization of hospital procedures and avoidable hospital admissions. AHRQ has developed software that allows the measurement of these quality indicators from claims data. Since administrative claims data are available widely, these data can be assembled among organizations that have implemented electronic health records as well as those that have not done so.
Measurement of Severity (Risk Adjustment): A quick examination of the literature shows widespread use of severity measures (the word severity appears in titles of 319,740 articles in PubMed). The problem is not that severity adjustment is not being done but that what is being done maybe insufficient. The risk adjustment may create an illusion of controlling for differences in patients’ illness but in truth it may not do so adequately. What is an adequate adjustment for severity of the patient’s illness? Statistical significance is not a good criterion for examining if severity adjustment is adequate because in large databases almost any measure of severity will have a statistically significant relationship to outcomes of care. The key, in our view, is that measures of severity must explain a large portion of variation in adverse outcomes of care.
The procedure for measurement of severity is fully detailed in the publication and a pending patent by Alemi et al. on measurement of episodes of illness from claims data.6
Measurement of severity of illness and patient’s prognosis is important because outcomes differ for sicker patients. An evaluation methodology that does not adequately adjust for patients’ severity of illness will erroneously attribute poor outcomes to poor quality when in fact it might be due to patients’ conditions. In essence, such a system will blame the fire on the firemen. A large literature exists for how severity of illness should be measured. We plan to use a procedure that allows us to measure severity of illness from claims data, which is widely available. In the proposed procedure, first the relationship between each diagnosis and patient outcomes are assessed using the entire claims data and regression analysis.
In the above equation, the parameters α, β, … are estimated from the data and are referred to as the “relative score”. The variables ICDxxx, ICDyyy, … are indicator variables that are 1, when the patient has the diagnosis and 0 otherwise. The next step is to standardize the scores so that they range from 0 to 1, using the following formula:
In the above equation, the maximum and minimum refer to the maximum and minimum relative score among all diagnoses. Patients typically present with multiple diagnoses and co-morbidities. The overall severity of the illness is calculated through the following formula; where standardized-severity-score for diagnosis “i” is shown as Si:
Overall severity = 1- ∏i (1 - Si)
For example, if a patient has two diagnoses (one with standardized-severity-score 0.9 and another with standardized-severity-score of 0.5), then the overall severity of the patient’s illness is calculated as:
Overall severity = 1 - (1 - 0.9) * (1 - 0.5) = 0.95
The procedure described above has been shown to radically improve the accuracy of severity measurement within electronic health records. Alemi and Walters applied the proposed severity index to measurement of both in-patient and outpatient Medicaid payments for the patient. The severity index created in this fashion explained 53% of variation in cost of care of patients. By way of comparison, when different severity indices were used to predict length of stay for patients with pneumonia, between 9.8% and 16.9% of the variation in length of stay was
explained.7 The procedure described increased the percent of variation explained by severity by more than 3 folds.
Timeline and Milestones Specific milestones include:
Table 1. Work Plan Schedule Tasks
Month
1-4 Months5-8 Months9-12 Months13-16 Months17-20 Months21-24 Define case study scenarios and
develop initial ontology
Construct knowledge base and design inference algorithms Develop data collection
software to allocate cost and to timestamp interventions Develop data analysis software to conduct causal analysis Implement algorithm for assessing changes in outcomes of care
Evaluate performance of software within Washington Hospital Center
Prepare public release of software & work with EHR industry
Prepare semi-annual project technical progress reports and software documents
The plan includes development of system specifications, design of inference algorithms, programming of data collection tools, programming of data analysis tools, display of data results in dashboard format, development of tools for assessment of quality of care and cost of care,
implementation of the systems within WHC, analysis of impact of WHC’s EHR on cost and quality of care, distribution of the software to EHR vendors and preparation of final reports. Note that according to the work plan, although work with EHR industry is scheduled for the second year, elements of these efforts will necessarily begin in earlier months.
Protection of Human Subjects
We plan to apply for approval of the project by both Georgetown and Washington Hospital Center’s Institutional Review Board. This project develops software that is implemented within an EHR. The data from the implementation in WHC will be used to report the ability of the software to detect changes in outcomes of care and to attribute these changes to specific interventions. No patient specific information is collected. All reported data are aggregate values calculated across patients.
There is a risk that the study, through its publications, may be exposing cost and quality of care at WHC, at a particular point in time. Study publications will not report the hospital within which the data was collected. There is a risk that the identity of the hospital involved may be revealed through affiliation of the authors of the publication. The study publications will report the data as sample results that could be obtained in a hospital if the system was implemented in the hospital.
Inclusion of Women and Minorities
This study does not recruit subjects. It uses data already collected through the electronic health record of Washington Hospital Center.
Targeted / Planned Enrollment Table
This study does not recruit subjects. It uses data already collected through the electronic health record of Washington Hospital Center.
Inclusion of Children
This study does not recruit subjects. It uses data already collected through the electronic health record of Washington Hospital Center.
Resource Sharing Plan
This project develops software that can be implemented within existing electronic health records to detect if these systems are making a difference in quality and cost of care. The software will be available as open source software through the web. In addition, the project will work with six leading EHR vendors to incorporate the software within their existing installed base. After the end of the project, the school’s fund for the situation room will be used to continue to maintain the software on the web.
1 Kenny DA. Correlation and causality. John Wiley & Sons Inc, 1979.
2 Pearl J. Causality: Models Reasoning and Inference Cambridge University Press,
2000.
3 Palacio C, Harrison JP, Garets D. Benchmarking Electronic Medical Records
Initiatives in the US: a Conceptual Model. J Med Syst, 2008.
4 Alemi F, Sullivan T. An example of activity based costing of treatment
programs. Am J Drug Alcohol Abuse. 2007;33(1):89-99.
5 Alemi F, Taxman F, Doyon V, Thanner M, Baghi H. Activity based costing of
probation with and without substance abuse treatment: a case study. J Ment Health Policy Econ. 2004 Jun;7(2):51-7.
6 Alemi F, Walters SR. A mathematical theory for identifying and measuring
severity of episodes of care. Qual Manag Health Care. 2006 Apr-Jun;15(2):72-82.
7 Iezzoni LI, Shwartz M, Ash AS, Mackiernan YD. Does severity explain
differences in hospital length of stay for pneumonia patients? J Health Serv Res Policy. 1996 Apr;1(2):65-76.
