When an audit process lacks standardization, as our original paper-based audits did, different auditors may produce different audit reports given the same source record. Without standardized quality measures, the changes in an organization’s data quality cannot be compared from year to year, nor can audit results be compared from site to site. We described five desirable attributes for a flexible audit support tool designed to simplify and standardize auditors’ work, including multi-user support through networking, file formats for importing and exporting audit data, standard error assessment protocols, audit decision support, and tools for calculating and displaying audit results.
Our prototype system demonstrated the significance of these key attributes as well as the feasibility of implementing them in a simple web-based application. We tested our CAAT during audits of twelve cancer and tuberculosis datasets in nine locations and found the system drastically improved the audit process by decreasing preparation time from over six hours, on average, to less than one hour; enforcing consistent, on-site error coding for all records; and eliminating difficulties reading other auditors’ completed forms, as well as manual error tallying for the post-audit report.
68
Limitations
The CAAT recommendations we describe stem from audit experiences at HIV clinics in Latin America and the Caribbean. The perspective of a single international research network, however, may limit the diversity of experiences that informed the five suggested requirements. Indeed, for single-user audits, a desktop application or other non-networked solution might present less complexity. Considering multidisciplinary audit teams are more common and effective [130].we believe these recommendations represent a necessary though perhaps not sufficient set of features needed for a successful paperless audit tool.
Additionally, our CAAT was limited in both its implementation of our proposed key attributes and its general applicability. We elected not to implement the fourth attribute, Audit Decision Support, as it was redundant given our audit circumstances and we were unable, therefore, to evaluate its usefulness in the field. We also did not write installation instructions or support documentation as the system was a prototype. A full-fledged application should include such support tools and documentation.
Our approach to importing audit data and form designs as REDCap CSV files saved several hours of preparation time as we had already designed the electronic form templates to create REDCap data entry interfaces for CCASAnet sites. Other users wishing to audit datasets for which no REDCap form templates exist would need to create them and would therefore not benefit from these time savings. Nevertheless, the REDCap data and metadata files are built using a straightforward design and coding process (as shown in Table 15 and Table 16 of Appendix B), and require no expertise beyond typing a list of variable names, labels, and form element codes into an MS Excel spreadsheet. We believe designing one such template for several audits of identically formatted data is much faster than creating new paper audit forms for each site.
69
We also recognize the effectiveness of the REDCap-motivated audit interface is dependent on the structure of the audit data. As we determined during the initial paper-based audits, correcting errors in antiretroviral regimen data often requires auditors to rearrange the order of regimens and insert new fields to accommodate additional data points. The TB and Cancer datasets we reviewed using the CAAT contained no variables that were as order-dependent as ARV regimens. Such data would require extended user interface option, possibly with Javascript widgets that allowed drag-and-drop duplication and reordering of medication regimens.
We were unable to conduct a quantitative comparison between the efficiencies of the paper- based and CAAT-driven audit processes because the HIV dataset we audited on paper differed from the TB and Cancer datasets. The quantity and types of data elements varied among datasets, as did the types of source documents we reviewed to validate them. The auditors had also developed greater auditing skill and knowledge of the domain by the second round of audits, which would have biased the results in favor of the audit tool.
Conclusion
Audits are an important quality assessment and intervention, but paper audit forms complicate the audit process, during the preparation for and conduct of the audit, as well as the post-audit analysis. We identified five attributes that would allow a computer application to replace paper audit forms: computer networking for simultaneous application use by multiple auditors, standard formats for importing data and metadata, a structured approach to classifying data errors, support for pre-audit record selection, and the automatic tabulation of audit results. Our prototype CAAT conformed to these specifications and measurably improved the efficiency of our source document reviews. Furthermore,
70
use of an electronic system enforced consistency in our audit forms, error coding, and error tallying, which likely improved the comparability of our audit results across sites.
After extending the user interface to handle different types of data and completing a cycle of code refactoring, we intend to test the applicability of this simple, computer-based audit tool in networks outside of CCASAnet and on datasets of non-HIV data. These steps will allow us to evaluate the validity and comprehensiveness of our five CAAT desiderata and identify extensions to the application that increase its potential for general use. Our experiment demonstrated that a computerized audit tool can simplify the audit process and enable research networks to measure and improve the quality of their data. We hope other similar networks will find a computer-based auditing approach useful for evaluating the quality of their data, standardizing their quality control activities, and identifying areas for process improvement.
71