Methodological Considerations
Methodological Considerations
When Evaluating the
I
l
i
f L
S l
Implementation of Large‐Scale
Electronic Health Record Systems
y
Amirhossein Takian M D Ph D Amirhossein Takian, M.D, Ph.D AcademyHealth ARM 2012 Orlando‐ USA 24 June 2012 24 June 2012 METHODOLOGICAL CONSIDERATIONS OF EHR EVALUATIONS 1EVALUATIONS ARE NEEDED TO TEST THE PROMISE OF
HEALTHCARE INFORMATICS
Medicine’s Dickensian Past Healthcare’s Utopian Future Modernising InformatingThe Old System The New System
Integrating Inconsistent Error-Prone Fragmented Evidence-Based Safe Connected Latest IT Policy Inefficient Doctor-Centred Reactive Accountable Patient-Centred Proactive
There is a big need to put the claims for healthcare informatics to the test
The EHR delivery structure in England
(Robertson et al 2010) (Robertson et al. 2010)
24 June 2012 METHODOLOGICAL CONSIDERATIONS OF
Overview of our dataset
(Sheikh et al. 2011) Total no. of site interviews (by WP) Hours of on‐site observations No. of site other documents Other data collected (e.g. field notes outpatient(by WP) observations documents field notes; outpatient surveys; CLICS surveys)
Total: 498 590 498 38 sets of field notes;
WPs1‐3: 310 WP4: 36 WP5: 60 130 CLICS surveys; 4,684 outpatient surveys WP6: 37
7 24 June 2012 METHODOLOGICAL CONSIDERATIONS OF
5 Key Challenges of national EHR evaluations
5 Key Challenges of national EHR evaluations
1 H t k l ti f ti l i l t ti f EHR 1. How to make evaluation of national implementations of EHR less of an afterthought? 2. How to ensure the independence of such evaluations?p 3. Which methodologies/study designs should we be employing? 4. Should these evaluations be predominantly formative, summative or both? 5. How to meaningfully synthesise the findings across such complex context‐bound interventions/evaluations?1. Evaluation as afterthought or forethought?
Evaluations of EHRs still tend to be afterthought, particularly in the context of national, politically‐driven implementations In the context of NPfIT, such evaluations were forced on the government by UK academics government by UK academics These evaluations were therefore commissioned in haste, but still These evaluations were therefore commissioned in haste, but still after key decisions had been undertaken/procurements had been made, etc. Can evaluations be moved onto the front foot? 24 June 2012 METHODOLOGICAL CONSIDERATIONS OF EHR EVALUATIONS 9
2. The independence of evaluations
Our evaluation was commissioned by NHS CFHEP, a semi‐ independent body at University of Birmingham, funded by the DH Although independent, we were still dependant on government bodies to recruit hospitals, obtain relevant documentary evidence/information, etc This was however not always possible: concerns about the This was however not always possible: concerns about the usefulness of independent evaluations, parallel government evaluations and “commercial confidentiality” Major concerns about yet more unwelcome publicity… 24 June 2012 METHODOLOGICAL CONSIDERATIONS OF EHR EVALUATIONS 11
3. Methodological considerations
RCTs as the gold standard for effectiveness, but inherently less suitable for studying safety Conventional designs rooted in a positivist ontology, assuming a single “truth”, uncovered by controlling for confounding variablesg , y g g Suitability and possibility of such approaches in the context of a national EHR implementation: would never be repeated P ti liti t t l f h i li l d hi Practicalities to control for changes in policy, leadership, government, economic climate? 24 June 2012 METHODOLOGICAL CONSIDERATIONS OF EHR EVALUATIONS 13OUR ADOPTED SOCIO‐TECHNICAL VIEW
The Health Human
System The Health
Care System Human Perspectives System Functions Structure Role of Medicines Technology OrganisationWork
What has been put in place?
What is the role of the intervention in the wider
healthcare system? What does the
intervention look like? practices emerge?What new work
Process
What is done that is different?
Processing What processes emerge and change?
Social Interactions How do communication patterns and workflow change? Management of Care Delivery What organisational changes emerge? Outcome
What has been
workflow change? Validity of
Processing Is the service safe
Quality of The Service Is the experience of the
Potential for Change How might the What has been
achieved?
Is the service safe and reliable?
Is the experience of the service satisfactory?
How might the intervention be used in
4. Formative or summative evaluations?
Most experimental studies are summative with interim analyses discouraged/penalised
Our outset plan to undertake both: formative & summative Despite requests from participating hospitals, funders
discouraged us from providing any formative feedback discouraged us from providing any formative feedback Given that these national experiments are never going to be repeated, it can however be argued that a formative evaluation is, if anything, the more important… 24 June 2012 METHODOLOGICAL CONSIDERATIONS OF EHR EVALUATIONS 15
5. Synthesising findings in national EHR
evaluations
evaluations
Systematic reviews can be used, but simplistic attempts at meta‐ l i t f th ti analysis are out of the question Rather a great need to consider using realist synthesis techniques Rather, a great need to consider using realist synthesis techniques,
aiming to understand relevant context‐mechanism‐outcome (CMO) configurations
Such CMO configurations offer insights into the likely transferability/generalizability of findings across settings transferability/generalizability of findings across settings
24 June 2012 METHODOLOGICAL CONSIDERATIONS OF
SO, TO CONCLUDE ...
National EHR programs cannot be dissociated from the national social and political context, as well as the local contexts Implementation not follow a standard pattern, national ones unique!p p q There should be no standard way to evaluate implementation There is a need to balance positivist vs. interpretive methods There is a need for reflexivity and adaptability to avoid reductionism A willingness to embrace new ontological, epistemological and th d l i l ti methodological assumptions: • Evaluation based on an appropriate and meaningful framework? • A shift from value judgements to exploring and interpreting• A shift from value judgements to exploring and interpreting
i / Questions/comments Amir Takian M.D., Ph.D School of Health Sciences & Social Care Brunel University London Room 112, Mary Seacole Building, Uxbridge‐ UK E [email protected] 24 June 2012 METHODOLOGICAL CONSIDERATIONS OF EHR EVALUATIONS 19
