Peer reviewed Performance m etrics in c linical trials
Using Metrics to Direct Performance
Improvement Efforts in Clinical Trial Management
Keith Dorricott, BSc
W e are used to the idea of measurement in the general practice of medicine, such as the vital signs taken after a baby is born or during the course of therapy for an illness in an adult. We would wonder what a medical practitioner was doing if he or she failed to take measurements such as blood pressure, heart rate, cholesterol, etc., when providing medical care and then compare those measurements to established norms.
Measuring is fundamental to our ability to understand and control the world we live in, and this is particularly true for scientific disciplines; it is part of the scientific method. In clinical trial management, if you want to know how enrollment is going for a particular trial, you might look at the enrollment rate or the number of subjects enrolled. You might want to compare these to your initial expectations to see if you are on track and take remedial action if necessary. Without a defined process for measurement—a method, a way to capture data for review—what would be the point of tak- ing the measurements in the first place?
A metric has been defined as “a standard of measurement.”
1Metrics are essentially the definitions of how we collect data on measurement and the value of those measurements once they are made. Many organizations recognize the need to measure (to use metrics), but their measurement systems have typically built up over time and have not been put together from a strategic perspective.
Hammer
2claims that across all the organizations with which he works, there is a wide consensus that they measure too much or too little, they measure the wrong things, and they do not use the metrics effectively.
2This article describes some key considerations for organizations as they review their approach to metrics or as they begin developing a key set of metrics.
The overall approach is shown in Figure 1.
Measurement Needs a Purpose
In the general practice of medicine, there are myriad things you could meas- ure. However, if you attempted to measure everything, there would be a substantial cost and the medical practitioner would be overloaded with all the data. The particular measurements that are useful will depend on the circumstances; measurements of the health of a newborn baby, for example, will be very different from those of someone who has high cholesterol.
Similarly, in clinical trial management, there are many things you could measure. Often, companies attempt to measure and track large numbers of metrics simply because they can.
3If you measured and reported all possible metrics across a set of clinical trials, the cost would be significant. You
This article
describes some key
considerations for
organizations as they
review their approach
to metrics or as they
begin developing a key
set of metrics.
details are important, but so is the overall composition. Treating symp- toms individually without considering them together—and whether there is a common underlying cause—would not be in the best interests of the patient.
Thinking “big picture” from the per- spective of those who are going to use the metrics can help to narrow down the metrics that you plan to collect.
Thinking “big picture”
from the perspective of those who are going to use the metrics can help to narrow down the metrics that you plan to collect.
Part of thinking about the big pic- ture is to select a mix of different met- rics types. Having different types of metrics in your measurement system helps to minimize the chance of sub- optimization.
4,5For example, focusing only on speed could make an activity faster; but if it adversely affects qual- ity, then subsequent activities can be undermined and the overall effect might be to increase the length of the trial. Generating a protocol quickly might be desirable, but not if there are underlying quality issues that mean costly, time-consuming protocol determining the purpose of measure-
ment.
4For example, the purpose of measurement in clinical trial manage- ment might be:
●
●
For a contract research organiza- tion (CRO) to be able to demon- strate oversight for the trials in its control to ensure timely, accu- rate, actionable data.
●
●
To reduce the time to conduct clinical trials.
●
●
To maximize the success of applications of new drugs to regulatory authorities.
As described in the following sec- tions, once you have determined the purpose of your measurement, there are a number of other key considerations.
Think “Big Picture”
As with a masterpiece painting, in the general practice of medicine, the little would be completely confused about
how to interpret all the data and would not have the resources to tackle all the questions that would arise, resulting in inaction. You would have the cost of data collection and reporting, but no outcome. Considering these factors, a typical flow for how a metric might be used is shown in Figure 2.
To get through the steps of selecting and implementing a metric (Figure 1) and the steps involved in using a met- ric (Figure 2) involves many resources and their associated costs. Every met- ric is a balance of that cost versus the benefit you can get out of the metric itself. There are only a relatively small number of key things that are really useful to measure in a given circum- stance (perhaps up to a dozen); these are often termed as the “key perfor- mance indicators.” So how do you go about determining those vital metrics?
A key consideration that will help you focus on the important metrics is
Figure 1 Selecting and Implementing a Metric
Figure 2 Using a Metric Determine the
purpose of measurement
Determine how you will collect, display and use the metrics
Program and validate Select/Define
your metrics and targets
Think
“Big Picture”
Drive value in the metrics – use the
“so what?” test
Have a mix of metric types Measure process
not people performance
Metrics definitions – use of industry
standards Start small
Review and
interpret Root Cause
Analysis
Agree and take actions to get the metric “on track”
Are data
“on track”?
Continue to monitor
No Yes
many sites could have been audited;
and you don’t know the result of the audits.
Of course, you could collect a vari- ety of metrics that would capture those other details, but perhaps there is a sin- gle metric of more value, such as the number of critical observations? Imag- ine you now have the data: There were four critical observations. So what?
Maybe there were 50 audits?
If you cannot think of actions that would result from collection of the data on a particular metric, it may not be of value to use that metric.
Perhaps you could measure the mean number of critical observations per site
Some Metrics Contain More ValueSome metrics are inherently more use- ful than others; they can tell the story that would otherwise need several
“lesser” metrics. A good way to deter- mine if you have selected one of these more powerful metrics is to use the “so what?” test.
4If you were to gather data on that metric, what would you do with it? What action might it drive? If you cannot think of actions that would result from collection of the data on a particular metric, it may not be of value to use that metric.
For example, you might be inter- ested in the quality of work performed at investigator sites in relation to the attention the sites have received from monitors. So you might select the number of investigator site audits in the last three months as a metric (see Table 2). Imagine you now have the data: There were two audits. So what?
You don’t know how many audits there should have been; you don’t know how Some different types of metrics you
should consider are shown in Table 1, along with examples and the risk of sub-optimizing by focusing only on a specific metric type. Note that a metric is typically either a lagging or leading indicator, and an indicator of one or more of the factors of cycle-time, time- liness, efficiency, or quality.
The bad news is that some of the most important things are not always measurable.
6For example, many mea- surements are possible for a newborn baby, but can you measure the instinct of the midwife who looks at the baby and says he looks good and healthy?
The right metrics can certainly help you manage the business, but they will never tell the whole story.
Keeping the big picture view helps you to realize when to be cautious in using particular metrics without oth- ers, or in relying too much on met- rics that might be leading to sub- optimization.
Table 1 Metric Types
Metric Type Description Example Metric Risk of Focusing on
This Metric Type Only Leading Indicator Provides information that you can
act on immediately to get the trial/
process back on track.
The proportion of sites activated versus expected would be a leading indicator for whether subject enrollment is likely to be on track.
Lack of data to help with process understanding and improvement
Lagging Indicator Provides information that you can use for future trials or for baselining for process improvement efforts.
The time taken from “last subject last visit” to database lock.
Lack of data to affect current work and act before negative consequences occur Cycle-time Measures the time taken to complete
a task.
The median time from subject visit to data entry into an electronic data capture system.
Faster cycle-time with poor quality leading to a process needing to be repeated unnecessarily; longer overall cycle times
Timeliness Measures whether a particular
milestone has been met. The number of days between planned and actual dates of the first site activated.
Meeting the timelines, but using excessive resources and not at the required quality level
Efficiency Measures the amount of resource required to complete a task or set of tasks versus that expected.
The difference between the actual final total contract value and the initial baseline contract value for a CRO running a clinical trial.
Process using minimal resources, but not meeting timelines
Quality Measures how well an output from a process meets the requirements of the customer of that process.
The proportion of expedited safety reports that are received by regulatory authorities within the required timelines gives an indication of the quality of the pharmacovigilance reporting process.
High quality outputs, but missing timelines and with high cost
staff reach “Category A” incidents within eight minutes of an emergency call.
9Using the metric to assess man- ager performance led to misreport- ing, such as “Category A” calls being reclassified as “Category B” when the eight-minute goal was not met and
“Category B” calls classified as “Cate- gory A” when crews arrived within tar- get. Also, varying the definition of the start and end times meant that eight minutes to one authority could be 10 minutes to another.
As seen in the case just described, using a metric to measure the perfor- mance of people may induce people to spend their time trying to meet the target—often by any means—rather than to focus their efforts on trying to improve the process itself. It is one of the best ways to make people lose the “big picture” and sub-optimize. It would be better to involve the staff in determining appropriate metrics more clearly related to the purpose of the process—the health outcome for the patient—and then to get them to focus their efforts on using the metrics to understand their process better and to improve the process for everyone.
In Conclusion
Finally, you need to consider how you will display, validate, review, and act on the metrics. There are many sys- tems that can be used for display, from basic Excel® spreadsheets to specific software designed for the purpose.
Ensuring the data are accurate (i.e., validating) is an important step to give confidence to those who are going to use the metrics. All these efforts will be of little value, however, without a process like that shown in Figure 2 to review the metrics so that they can be used to drive decisions and actions.
In the general practice of medi- cine, measurements are fundamental.
Similarly, measuring the clinical trial importance of the definition of a met-
ric. The definition should be written down so that there is no ambiguity;
this definition can be used when try- ing to understand why the metric is at a particular value. There are indus- try organizations that have developed standardized, defined metrics for use in clinical trial management,
7and the potential metrics they provide can
be used as a starting point for met- rics selection. Using standard metrics also makes it easier to benchmark to compare performance across organi- zations and help to drive continuous improvement.
Metrics Should Measure Process Performance, Not People Performance
As you select your metrics, you should focus on the process rather than using the metrics to measure people. By using a metric to measure people, there is a high risk of sub-optimization, as individuals focus on the metric to the exclusion of everything else and “gam- ing” or “cheating” can result.
8Seddon describes the impact of implementing a metric by looking at audit? Imagine you now have the data:
There were two. So what? Here you have some actionable data; having an average of two critical observations per site audit would be a real cause for concern. You would want to understand the root cause, finding out which sites were audited and looking for systemic issues, such as a confusing protocol or poor training of site staff. This one met-
ric has high value, as it gives an indica- tion of quality of work at investigator sites and by monitors. It best matches the purpose of interest to you.
In a similar way, if you are look- ing for a metric to indicate whether a clinical trial is on track, measuring the number of sites that have been acti- vated does not pass the “so what?”
test. Measuring the proportion of sites activated out of the total expected has more value; even better, however, would be to measure the proportion of sites that have been activated out of those expected to be activated at that particular time. This gives an immedi- ate indicator of whether you are on track for activating sites, and there are clear actions you could take. It passes the “so what?” test.
Considering the value inherent in
Table 2 Building Value into Your MetricsPossible Quality Metric Data So What?
Number of investigator site audits
in the last three months 2 Knowing this tells us nothing about the quality.
Number of critical observations
in the last three months 4
Possibly a cause for concern, but we do not know how many audits there were.
Mean number of critical observa- tions per site audit in the last
three months 2
Definitely sounds like a cause for concern. We would want to take action and investigate further.
Increasing Value
Using a metric to measure the performance of people
may induce people to spend their time trying to meet the
target—often by any means—rather than to focus their
efforts on trying to improve the process itself.
mance. BeyeNETWORK. Available at www.
b-eye-network.com/view/7981.
4. Zuckerman DS. 2006. Pharmaceutical Metrics.
Gower.
5. Sullivan L. 2011. Defining “quality that mat- ters” in clinical trial startup activities. The Monitor 25(7): 22–6.
6. Nelson LS, quoted by Deming WE. 1982. Out of the Crisis. MIT Press, p. 121.
7. Metrics Champion Consortium, www.metrics champion.org/default.aspx.
8. Pyzdek T. 2012. Gaming the metrics—use met- rics to guide improvement, not measure the performance of people. Quality Digest. Avail- able at www.qualitydigest.com/inside/quality- insider-column/gaming-metrics.html.
9. Seddon J. 2005. Freedom from Command and Control. Vanguard Education, p. 213.
Keith Dorricott, BSc, is director for operations management, process improvement, and metrics at INC Research in the United Kingdom. He is an active member of the Metrics Champion Consortium, and worked with the organization to launch the Process Improvement Work Group in 2009. Prior to his seven years working on improv- ing the clinical trial process at various contract research organizations, he was technical manager at Eastman Kodak manufacturing. It was at Kodak that he honed his skills in process improvement techniques, including Six Sigma and Lean. He is a Lean Sigma Master Black Belt. He can be reached at [email protected].
to understand and improve processes.
If particular metrics fail the “so what?”
test, then consider removing or replac- ing them.
Ideally, metrics measure the perfor- mance of systems and processes, and analysis of them should help direct your efforts in process improvement.
By careful use of measurement in clini- cal trial management, we extend the scientific method beyond the science of the trials themselves, and that sci- ence reminds us of the really “big pic- ture”—that the fundamental purpose of our efforts as clinical researchers is to improve patients’ lives.
References
1. Merriam-Webster Online Dictionary, www.
merriam-webster.com/dictionary/metric.
2. Hammer M. 2007. The seven deadly sins of performance measurement and how to avoid them. MIT Sloan Management Review 7(43).
3. Nelson G. 2008. Implementing metrics man- agement for improving clinical trials perfor-
