Other Analytical Techniques. Nick Salkowski SRTR February 13, 2012

Other Analytical

Techniques

Nick Salkowski SRTR February 13, 2012

2

Control Charts and Control Limits

•

Control Charts:



Routinely monitor quality



Distinguish between “in-control” and “out-of-control”

processes



Distinguish between “normal” variation and “assignable

cause” variation



Run until there is an “out-of-control” signal

•

Exceeding Control Limits or thresholds

trigger a response

1 _{NIST/SEMATECH e-Handbook of Statistical Methods, http://www.itl.nist.gov/div898/handbook/}

3

Thresholds and Responses

•

Control thresholds and response plans need to be developed

together



Lower thresholds will produce more false-positive signals,

and are appropriate if the response is minor



Higher thresholds will produce fewer false-positive signals,

and are appropriate if the response is intensive

•

Of course, higher thresholds make it more difficult to signal

when the process is “out-of-control”, too!

4

Statistical Hypothesis Tests

•

Theoretically distinct from Control Charts

•

Test a specific null hypothesis against an alternative



Type I errors – Rejecting a true null hypothesis



Type II errors – Failing to reject a false null hypothesis

•

Adjustments are needed for multiple testing

5

CUSUM Strengths

•

Tracks process continuously using current data

•

Produces a signal after a center has a sufficiently bad run of

outcomes

•

Chart provides a visual summary of center performance over

time

•

"When retrospectively compared to currently available data

reporting, the CUSUM method was found to detect clinically

significant changes in center performance more rapidly, which

has the potential to inform center leadership and enhance

quality improvement efforts."—Axelrod, et al. 2009

American

Journal of Transplantation

9(part 2):959-969

6

CUSUM Limitations

• Data doesn't always appear instantly

 It can take months for a death to appear in the data set!

• CUSUM charts are intended to run until there is a signal

 In-control processes will all signal eventually

• Calculating the CUSUM can be computationally challenging

 _{When the in-control and out-of-control rates are based on}

survival models, the daily hazard for every person at risk must be calculated every day

• CUSUM is a tool for constant quality monitoring: it is best if it is calculated whenever there is new data

 _{Daily computation is probably sufficient}

 _{Much less useful if the CUSUM is calculated every 6 months}

7

Threshold Difficulties

• Thresholds need to be uniquely determined for each program

 _{Simulations are needed}

 _{Predictions about future rates are needed}

• Thresholds will only perform well under a steady state

• If a program changes over time, the thresholds need to change too!

 _{If the number of transplants performed increase, the expected}

graft failure rate per day probably increases

 _{If a program performs more transplants with high expected graft}

failure rates, the expected graft failure rate per day increases

• What does an "out-of-control" program look like?

 _{Double the risk of an "in-control" program?}  50% more risk than an "in-control" program?

8

Funnel Plots

•

Scatterplot of an estimate against a measure of the estimate's

precision

•

Tend to form a funnel shape since low-precision estimates

tend to spread out more than high-precision estimates

•

Good for comparing different centers

•

Good for identifying programs with unusually good or bad

outcomes

9

10

11

Period Analysis Cohorts

• Use different cohorts to estimate different segments of the survival curve, so that the most recent outcomes are used

• For example:

 Use 2011 transplants to estimate survival during year 1  _{Use 2010 transplants to estimate survival during year 2}  _{Use 2009 transplants to estimate survival during year 3}  …

 _{Use (2012-Y) transplants to estimate survival during year Y}

• Long-term survival can be estimated without using old data to estimate initial survival

• Odd behavior at boundaries: 12/31/2010 transplant is used only for 2nd _{year survival, but 1/1/2011 transplant is used only for 1}st _year survival

12

13

Alternative Period Analysis Cohorts

• Possible to define cohorts as all persons at-risk for a particular event during a specific period of time

• For example, all persons at-risk for graft failure during the first 3 years post-transplant between January 1, 2011 and December 31, 2011

 _{Includes all transplants during 2011}

 Includes all transplants during 2008-2010 without a graft failure

before 1/1/2011

 Only failures during 2011 count!

• Left-truncated / Right-censored analysis

• Compatible with longer follow-up outcomes (e.g., 5-year, 10-year)

14

15

Alternative Period Analysis Cohort Limitations

•

Tradeoff between data timeliness and quantity



Shorter time intervals mean more recent data and less

overlap between PSR cohorts, but smaller sample sizes:

fewer events and persons at-risk



Changes in power could require changes to flagging criteria

or produce different flagging probabilities

•

Some failures or deaths could be "lost" during a transition



Occurred too long ago to be included in new cohort

Too recent to be included in the old 3-year cohort