Here is an example of how to assess an audit program based on existing records. Newly certified organization ABC was hav- ing trouble getting its internal audit program off the ground. A consultant was hired to improve the internal quality audit program after the organization received findings (one major and one minor) against it in each of its two customer audits during the preceding year. The auditors had received formal training, and there were no dedicated auditors; auditing was a second- ary job function.
After reviewing ABC’s records, the consultant noted the fol- lowing:
1. The audit schedule for the previous year was not completed until June.
2. Only seven of 17 scheduled audits were conducted. 3. There was objective evidence that only one of two opportuni-
ties cited was followed up on.
4. The status of scheduled audits was not tracked.
5. Audit forms were completed inconsistently from person to person.
6. Auditor meetings were not held. 7. Audit results were not regularly reported.
8. Training records were available only for four of six current auditors.
9. Audit forms were poorly designed and did not logically flow. 10. Audit procedures were unclear and insufficiently detailed.
The consultant interviewed the existing auditors, and many expressed dissatisfaction with their levels of training. All confessed to occasional confusion about how to classify audit findings, how to complete various audit working papers and when to pursue formal corrective action. They said it was due to unclear work instructions and poorly designed forms. Some were concerned with having too many audits to do along with their regular workloads.
Before beginning a root cause and corrective action pro- cess analysis, the consultant rated ABC’s internal quality audit program to provide an objective baseline for comparison after improvements were implemented. Figure 3 (p. 44) shows the results of the assessment. A score of 60 is minimally acceptable, 80 or better is the goal and a score of 90 or higher is delightful.
With a rating of 23% (based on the information from ABC’s records), ABC’s audit program was in the dissatisfaction quadrant of the Kano model (see Online Figure 2, which can be found on this article’s webpage at www.qualityprogress.com).
The next year, the following changes were implemented: • Two new auditors were trained, and one worked on off shifts. • Two of the existing auditors received refresher training. • Portions of two audits were conducted during off shifts. • The standard operating procedure for audits was completely
rewritten.
• Audit forms were revised, and auditors were trained to ad- dress previously cited deficiencies.
These changes led to the following results from subsequent customer audits. There were no major audit findings against the quality management system, and there wasn’t any nonconfor- mance issued against the audit program itself. Figure 4 (p. 45) shows the results of the following year’s audit program evaluation.
The improvements made to the audit program were substan- tial, and as expected, they were reflected in the greatly improved score. The new rating was more than three times greater than the original score—well above minimally acceptable requirements and within the margin of error for attaining the goal of 80%.
The score was in the must-be quadrant of the Kano model (see Online Figure 2). There were still improvements that could be made, and some areas’ low scores could easily be used as a roadmap for improvement and delightful performance in the future. —L.C.
AUDITING
Audit system evaluation form / FIGURE 2
Audit system evaluation Date:
Planning (40%)
___ 5%—All audits are scheduled that should be scheduled (per requirements of the standard, regulation or internal policies). ___ 5%—All scheduled audits were conducted.
___ 5%—Percentage that were on time.
___ 5%— All nonscheduled audits (not on the annual schedule, but scheduled to address a concern that was identified) were conducted.
___ 5%—Annual audit plan was approved by site management.
___ 5%—Audit program is tied to quality management system (QMS) evaluation metrics and corporate goals.1
___ 5%—Individual audit plans were approved by lead auditor or quality management with input from process owners. ___ 5%—Audit program is well integrated within the risk management program.
Reporting, records and analysis (30%) Reporting—10%
___ Reported to site-level management. ___ Reported to quality manager. Data analysis—10%
___ Reported data regularly analyzed and acted on. Records—10%
___ Records maintained (audit reports). ___ Records maintained (training records).
___ Records maintained (quarterly or annual summary reports). Implementation and results (30%)
___ 5%—Do we audit across all operating shifts?
___ 5%—Is there an appropriate level of auditor training (requirements + actual)? ___ 5%—Are an appropriate amount of audits dedicated to improvement activities?2
___ 5%—Is a documented and structured method for evaluating the validity and classification of findings? ___ 5%—Are there adequate resources?
___ 5%—Are varying types of audits used, such as process, product, trace, system or element? Subtotal = potentially 100%
+ Value add percentage – Reality check percentage
Total Notes
1. ISO 9001:2008 and ISO 13485:2003 requires an audit program to monitor the effectiveness of a QMS, but it isn’t explained how you know whether the QMS is effective. It’s not enough to simply state that a lack of findings demonstrates that the QMS is effective. Metrics should be developed to assess the QMS, and the audit program should monitor those metrics in addition to compliance to policies and procedures.
2. Improvement auditing is assessed against criteria that could be driven by compliance or a desired future state. It also could evaluate an area for muda (nonvalue-add wastes) or monitor project results against predetermined milestones.