Performance measurement methodology

(1)

Performance measurement methodology

G-I-N conference, Berlin Aug. 24th, 2012

P f J hi S i MD MS

Prof. Joachim Szecsenyi, MD, MSc

AQUA-Institute for Applied Quality Improvement and R h i H lth C Götti G

Research in Health Care, Göttingen, Germany

www.aqua-institut.de; www.sqg.de

University of Heidelberg Hospitaly g p

Dpt. of General Practice and Health Services Research

www.allgemeinmedizin.uni-hd.de

(2)

Agenda

 Sharing the same problems: Implementation of guidelines

 Why measuring performance?y g p

 Indicator development and guidelines

S l f f t f G

 Some examples of performance measurement from Germany

 Take home messages

(3)

Sharing the same problems:

Bariers against implementation of guidelines

 Lack of resources

 Lack of dissemination to the end user

 Dissemination is not enough, implementation is necessary

 Misunderstanding: If the guideline is there, it will implement itselfMisunderstanding: If the guideline is there, it will implement itself

 Professionals may be afraid of loosing power

 Difficulties of changing routines and habits

 Opinion-leaders may not go with the guideline

W i ti b i l (i i b t)

 Wrong incentives may be in place (i.e. reimbursement)

(4)

Your doctor is not good.

Why?

As far as I know,

(5)

Measurement and improvement in healthcare



„If you can

´

t measure it, you can

´

t change and improve it“

(D. Berwick)

(6)

Performance measurement needs indicators



“

A measurable element of practice performance for which there is

evidence or consensus and that can be used to assess the quality,

q

y

and hence change the quality of care provider”

(Martin Lawrence, 1997)

 Indicators show (latin: “indicare”) quality of care, figures need to be interpreted then

 May be used for internal management, auditing and improvement, external comparisons (variation, benchmarking), transparency (public

) ( f f )

(7)

Criteria for good quality indicators



Valid, relevant

Representative, good coverage of

aspects that should be measured



Communicable

Meaning can be easily explained



Discriminative

Detect differences among performance



Discriminative

Detect differences among performance



Sensitive

Can detect change in performance



Influencable

Possible to change if required



Influencable

Possible to change if required

(8)

What „granularity“ in performance measurement are

„g

y

p

we aiming for?

g

(9)

With indicators, you can

 Measure performance on different levels (i.e. health system, system, region cross-sectoral facility single provider)

region, cross sectoral, facility, single provider)

 Measure acticvityy

 Measure health care structure and processes

 Measure health care outcomes

 Clinical aspects

From the patient´s perspective

(10)

Uncontrolled diabetes hospital admissions

and prevalence of diabetes

Source: OECD, data: 2009 or nearest year

KOR R² = 0,05 120 140 MEX FIN 100 120 100 000 o n DEU POL FIN DNK SWE NOR 60 80 s ions per populati o USA PRT CAN DEU SVN ITA IRL GBR NOR ISL 20 40 Admis s CAN ISR NZL 0 0 2 4 6 8 10 12 Prevalence of diabetes (%)( )

(11)

D

l

t f i di

t

f

id li

Development of indicators from guidelines

 Recent (and first) systematic review (8.697 publications identified and screened; 48 relevant publications analysed); Köter et al.

S 2012

Implementation Science 2012

 Only 10 publications gave criteria how to select guidelines for i di t d l t

indicator development:

 Methodological quality of the guideline (using instruments like AGREE)

AGREE)

 Up-to-datedness of the gudieline

 Guideline is applicable for the area/topic of health care in which quality measurement should take place

(12)

D

l

t f i di

t

f

id li

Development of indicators from guidelines

 Criteria for selection of recommendations from guidelines for which indicators should be developed:

 Agency Healthcare Research and Quality (AHRQ)

 Burden of diseaseBurden of disease

 Value for money H d t l (1996)

 Hadorn et al (1996)

 Impact on quality of care

(13)

D

l

t f i di

t

f

id li

Development of indicators from guidelines

 Selection of indicators

 Structured expert panel – methods (i.e. modified RAND appropriateness method) very often usedpp p ) y

 Selection of panelists: criteria rarely published

 Patient participation during QI development is extremely

 „Patient participation during QI development is extremely uncommon“

 Conclusions

„A wide variety of methodological approaches are described in the

li f id li b d QI d l I i l hi h

literature for guideline-based QI development. It remains unclear which methods leads to the „best“ QI, since no randomised controlled or other

(14)

Indicator development process for the

Q

O

(Q

)

UK Quality and Outcomes Framework (QoF)

(15)

Indicator development process in Germany

(Across healthcare sectors, AQUA-Institute / Federal Joint Committee)

 Priorisation process for theme/topic (Federal Joint Committee)

 Analysis of current care, potentials for improvement, data valiable

 Pathway of care

 Scoping workshop with stakeholders (medical and nursing experts, ti t t ti f i l b di t )

patient representatives, professional bodies, payers, etc)

 Search for existing indicators and guidelines

 Interdisciplinary expert panel including patient representatives toInterdisciplinary expert panel, including patient representatives to discuss, modify and evaluate indicators (modified RAND

appropriateness method). Conflict of interests made transparent.

( )

 Report on development process (to be commented by stakeholders)

 Technical and practical piloting with health care facilities, regional level and software companies

and software companies

(16)

Risk adjustment

 Comparisons between doctors (providers, hospitals, practices) may not be fair because the case-mix of the population served is variable not be fair, because the case mix of the population served is variable



Possibble solutions



Exeption reporting (QoF, UK)



Risk adjustment (outcomes)



Stratification



Pre-selected criteria from other research



Logistic regression with quality of care data

(17)

Retirement of indicators

 No established method available yet, but some common considerations

considerations

 Evidence base has changed

 High achievement and no more improvement over (long) time g p ( g)

(plateau)

 Low variablity between providers

C t ff ti i

 Cost effectiveness issues

 Other (better) sources of information on quality of care available

(18)

Some examples of performance measurement

from Germany

 Primary care

 Hospital care

(19)

Peer review / quality circles

 Peers have a strong influence (for the good and the bad) on health professionals

 The power of peers“ (Richard Grol)

 „The power of peers (Richard Grol)

 Since 2010 volunteer peer visit program in hospitals (supported by thep p g p ( pp y national chamber of physicians)

 1995 start of the first national training programme for moderators of peer review groups (quality circles) by the AQUA institute in Göttingen peer review groups (quality circles) by the AQUA institute in Göttingen in collaboration with the Association of SHI physicians

 In 2011 more than 5.000 peer moderators (facilitators) trained, more than 4.000 quality circles active nationwide

 Programs with structured feedback (i.e. rational prescribing) to be used in quality circles

(20)

Example: feedback on prescribing for heart failure

in primary care

Your practice Mean Comparison group (median, quartiles)

Based on DEGAM guideline for heart failure: percentage of patients with the diagnosis heart failure not receiving ACE inhibitors, AT1 blockers or betablockers.

Caution: data capture period 3 months prescriptions from other primary care practices Caution: data capture period 3 months, prescriptions from other primary care practices or specialists not included.

(21)

Results from quality circles on rational prescribing

 High attendance rates of general practitioners

 High satisfaction with group work

 Change of prescribing routines, i.e.

L t ti ll i i t ibi f th ld l

 Less potentially inapropriate precribing for the elderly

 Less potential interactions between drugs prescribed

 Better use of antibioticsBetter use of antibiotics

 Lower costs

Wensing M, Broge B, Riens B, Kaufmann-Kolle P, Akkermans R, Grol R, Szecsenyi J. Quality circles to improve prescribing of primary care physicians. Three comparative studies. Pharmacoepidemiol Drug Saf. 2009

(22)

Example: European Practice Assessment (EPA)

 Comprehensive assessment of the quality of services in ambulatory care facilities by validated quality indicators

 Including self-assessment, interviews with practice managers and staff, staff satisfaction survey, patient satisfaction survey, audit and y p y team meeting with a trained facilitator

B h ki i t th ti ith h i ft

 Benchmarking against other practices with a comprehensive software

 Controlled study shows substantial improvements especially in

 Controlled study shows substantial improvements, especially in organisation of facilities and patient safety issues

Szecsenyi J, Campbell S, Broge B, Laux G, Willms S, Wensing M, Goetz K. Effectiveness of a quality-improvement program in improving management of primary care practices. CMAJ. 2011

(23)

An example from our nationwide hospital

benchmarking system

Community aquired pneumonia (CAP)

Year 2011:

234.957 admissions to 1.322 hospitals

8 quality indicators reported since 2006

since 2006

In 2005 evidence-based guideline for CAP developed by professional organisations developed by professional organisations (PEG, DGP, DGI, CAPNETZ)

(24)

Results for CAP over the last 6 years

(25)

German Hospital Quality Report



Appears yearly



Free download of 2011 report

i

li h i

t

2012

in english in autumn 2012



http://www sqg de/quality



(26)

„Mr. Gorbatchev, tear down this wall“

Public address US president Ronald Reagan, Berlin, June 12th,1987

Ironicall alls like those bet een amb lator and hospital care

 Ironically, walls like those between ambulatory and hospital care not only in Germany prove to be more resistant than the former ‘iron curtain’.

 Performance measurement across healthcare sectors (primary care, secondary outpatient secondary inpatient follow up etc ) worldwide secondary outpatient, secondary inpatient, follow-up etc.), worldwide still in it´s infancy

 Many quality problems occur at the transition between sectors

 Szecsenyi J et al: Tearing down walls: opening the border between hospital and ambulatory care  Szecsenyi J et al: Tearing down walls: opening the border between hospital and ambulatory care

for quality improvement in Germany.Int J Qual Health Care 2012 Apr;24(2):101-4. Epub 2012 Jan 22

(27)

Example:

Colorectal cancer, some of the problems

Not all patients discussed in interdisciplinary tumorborads

Possibble overtreatment with chemotherapy

Not all patients with liver metatasis seen by a specialised surgeon Not all patients with liver metatasis seen by a specialised surgeon

Problems in quality of surgical procedures and pathology

Lack of communication between healthcare providers

N h d d i i ki i l k i f i No shared decision making, patients lack information

Many patients lost for follow-up / after-care y p p

(28)

Examples: Indicators for colorectal cancer

Process Process

% of patients who were discussed in interdisciplinary tumor board

% of patients where coloscopy report was available before surgery

% of patients with a minimum of 12 lymphatic nodes extracted and p y p hystology

% of patients with post-surgical assessment of gut-, bladder-% of patients with post surgical assessment of gut , bladder

(29)

More examples

outcome

30-day mortality after primary surgery

safety

% of patients with application of antibiotics before surgery

patient perspective

% of patients who report shared decison making between doctor and patient about therapeutic options

patient about therapeutic options

area indicator

% of patients with early cancers compared to all new cancers in a region (per year)

(30)

What makes quality improvement work?

 Performance measurement and feedback

 Learn from comparable „best practices“

 Learn from variation

 Multi-perspective (patients, doctors, team, etc.)

 Individualised and needs oriented feedback

Written feedback alone is mostly uneffective Written feedback alone is mostly uneffective

 Personal support and social influence

 Educational outreach visits/facilitators, peers

 Mulifaceted and taylored interventions seem to be more effective

(31)

Take home messages I

 Performance measurement needs indicators

 Indicators can support implementation of guidelines

 Indicator development requires a rigorous process (evidence and consensus)

consensus)

 Guidelines may be basis to generate indicators

 Guideline core recommondations need to be explicit

 Guidelines may have weak points at indication for a procedure, test or prescribing and at interfaces between healthcare sectors

Consider possibble unintended effects of measuring

 Consider possibble unintended effects of measuring

 Cross-sectoral perspective in performance measurement is important, but is innovative and needs further developmentp

(32)

Take home messages II

 Indicator development and performance measurement needs special expertise as guideline development does guideline development does

 These two groups of experts need to collaborate, to develop good performance measures and to develop programmes forp p g

(33)

Thank you!

AQUA– Institut für angewandte

Qualitätsförderung und Forschung im Gesundheitswesen GmbH Gesu d e s ese G b Maschmühlenweg 8–10 37073 Göttingen Telefon: (+49) 0551 / 789 52 -0 T l f ( 49) 0551 / 789 52 10 Telefax: (+49) 0551 / 789 52-10 E-Mail: [email protected] Internet: www.aqua-institut.de www.sqg.de