What is a Metric?
Standard of measurement by which efficiency,
performance, progress, or quality of a plan, process, or
product can be assessed
1.
A “quality” metric is a measurement of quality. It can be
either a “leading” or “lagging” indicator.
• A “leading” indicator provides information proactively in order to take action
before contravening a target objective or goal. Can be used to predict an outcome.
• A “lagging” indicator measures performance after the fact and can be used to determine if goals and objectives have been achieved.
Regulatory Guidance
Limited guidance in the GMP’s on what constitutes
quality metrics
ICH Q10 (2007) -
Performance indicators
Measurable values used to quantify quality objectives to reflect the performance of an organization, process or system, also known as “performance metrics” in some regions.
C.02.11 - Canadian GMP’s refers to APR’s 21CFR180(e) - US GMP’s, refers to APR’s
21CFR820.20 – US GMP’s for Medical Devices refers to quality data EU GMP’s – Quality Management refers to Product Quality Reviews
Why do we collect Metrics?
Gauge to measure performance
• Provides feedback and information on your state of control
• Indicator for trends to act proactively not reactively….leading • Allows you to make informed decisions
• Drives improvement (minimize cost and maximize performance) • Good business….good motivator
What do we Measure
What Gets Measured gets Done…
True, but sometimes we lose sight of why we collect metrics or question the value of what we are measuring…..Collecting metrics for the sake of collecting metrics is non value added.
…Your CMO is not in the metrics business
“Not everything that counts can be counted, and not everything that can be counted counts.” Albert Einstein
What Metrics do you Measure at your CMO?
Only those that you need!
…129 line items?? You’re kidding right??Service? Quality? Leading or Lagging?
Consider from a product quality perspective and a GMP perspective 1. Define objective
2. Define metric
3. Set targets
4. Collect data
5. Assess and communicate
6. Modify or change
CMO Organizations
• Metrics are an integral part of our business. Like most companies
we use them to drive performance, identify areas for improvement and strengthening
• Have defined set of metrics based on our assessment of our performance
• Metrics are aligned to our corporate objectives
• Have capabilities to electronically track and trend most data • Have resources to track and trend our data
Common Quality Metrics
• Number of Deviations (trending root cause) – Also consider deviation per batch
• Number of OOS’s (trending root cause)
• Batches Right first time – If the site is running generally >90% then
consider dropping metric. Lower numbers or downwards trend should be closely monitored.
• Deviations Past Due – Completing an investigation (RCA, CAPA Plan and
Approvals) within a timely manner is important. However ensuring that the content of the investigation in complete and thorough is critical. Although there are those that may go past the due date with QA approval,
recommend that it is the exception rather than the rule. Consider Aging?
• CAPA Past due – Keeping track of your CAPA’s is imperative. Make sure
timelines are realistic. Although there are those that may go past the due date with QA approval, recommend that it is the exception rather than the rule. Consider Aging?
Other Considerations
• Stability Past Due - compliance to pull dates
• APR past Due – compliance to schedule (ensure CMO Portion is
defined)
• SOP Training Compliance • EM results and trends • Batch Rejections
• Documentation Errors
• Documentation creation, review and approvals • Audit observation trends
…others
Metrics Formatting
• Consider a 12 month rolling
• Identify what constitutes a positive or negative trend • Select reasonable scales and maintain consistency
• Communicate and reinforce definitions and associated calculations • Automate if possible – eliminate subjectivity
• Include colors to emphasize the good, the bad and the ugly • Include commentary to clarify anomalies or negative trends • Consider measuring the negative not the positive
Examples…
0 1 2 3 4 Ju l-1 0 Au g-10 Sep-10 Oct -1 0 N ov -1 0 D ec -1 0 Ja n-1 1 Feb-11 Ma r-1 1 Ap r-1 1 Ma y-11 Ju n-1 1 # P as t Ta rg et A p p ro va l D at eEU Regional Trend
F G H
0 2 4 6 8 10 12 14 16 18 20 # P as t Tar ge t A pproval Dat e
US Regional Trend
A B C D E
0 0 0
0 0 0
0 0
1
4
1 1 1
0 0 0
2 4 1 0 0 1 2 3 4 5 6 7 8 9
A B C D E F G H
# P as t Ta rg et A p p ro va l D at e Commercial Operations Comparison
Within 10 days Past Target Approval Date (On-Time < X ≤ 10 Days) Within 30 days Past Target Approval Date (10 Days < X ≤ 30 Days) Within 60 days Past Target Approval Date (30 Days < X ≤ 60 Days)) Beyond 60 days Past Target Approval Date (60 Days < X)
a
Examples…
0
4
6
0 0 0
0 2 4 6 8 10 12 14 0 1 2 3 4 5 6 7
A B C D E F
Nu mbe r (# ) P as t D ue Pe rce ntage ( %) Pas t Due
North American Region
CAPA Past Due % Target CAPA Past Due #
0 5 10 15 20 25 30 # of De v. Month
LINCON OPERATIONS
DEVIATIONS ROOT CAUSE TREND FY2011
Procedure External Process Other Material Human Equip/Mech
Data Management
What do you do with the Data?
How is it reviewed, analyzed and assessed internally? If it stopped coming would anyone know?
Who see’s the data?
Is the visibility restricted to Quality management? Is your data shared with the CMO? Is their open dialogue and discussion?
Is the data defined and understood?
Does the internal management team understand what is being measured? Does the CMO have the same level of understanding?
How effectively is it communicated
Does the data get disseminated down to those that have an opportunity to influence or impact the outcome?
Communication and Ownership
• It is critical that results are effectively communicated and understood throughout the organization
• Ensure that there is departmental ownership for their own performance metrics.
…Quality metrics do not just belong to Quality
A well run department ….will be actively involved with the collection and
dissemination of metrics pertaining to their area of operation and have detailed plans in place for remediation …they will not wait to be asked.
Individual contribution and ownership is key to success
Awareness and Ownership
Department Meetings
Are metrics discussed? Are employees engaged? Do employees contribute to CAPA plan? Are employees rewarded for success?
Site Management
What goes on in management meetings? Are metrics presented? Who is presenting the data? What questions are being asked? Does each concern have a defined CAPA plan? Are we holding our managers accountable? Are we being accountable? Are we seeing positive change? Ownership?
Senior Executive
What goes on in a executive review? Who is presenting the data? What questions are being asked? Does each problem have a defined CAPA plan? Are we holding our sites accountable? Are we being accountable? Are we providing resources and
capital to rectify issues? Are we seeing a positive benefit for the meeting?
Ignorance is NOT Bliss!!
The importance of senior management awareness, understanding and engagement cannot be overstated.
Recent regulatory issues suggest that this awareness,
understanding and engagement is either lacking or not as strong as it should be.
“There is a lot of attention at the moment on how FDA is approaching out-of-compliance firm and it’s a very positive development that the media is attuned to the types of failures that are occurring in the pharmaceutical industry, which often can be traced back to inattention by senior
CMO Relationship
A CMO is an extension of your operation. If this
relationship is not strong in your existing operation
what does that mean about your understanding of what
is occurring at your CMO.
Regulators are considering holding drug owners more
accountable for problem at their CMO’s.
– Transfer of information and knowledge is an important first step
followed by open and effective channels of communication
This relationship needs to be actively and responsibly managed. This does not mean to measure everything and anything or pitch your tent.
