September 2014 DISCLAIMER:

(1)

(2)

September 2014

Recommended Citation: Government of Kenya. 2014. Data Quality Audit Report, August 2014, Nairobi, Kenya: Division of Health Informatics Monitoring and Evaluation, Ministry of Health, AfyaInfo Project.

This Data Quality Audit Report was derived from the findings of a country-wide data quality audit exercise conducted for the health sector in March 2014. The report and the audit were done with assistance from the AfyaInfo project. AfyaInfo is a technical assistance program to support the Government of Kenya to strengthen their health information systems. The program is implemented by Abt Associates, Inc. in partnership with Training Resources Group, ICF International, the University of Oslo, Knowing Inc., the Kenya Medical Training College, and the University of Nairobi. It is funded by the United States Agency for International Development (USAID), under the AIDS Support and Technical Assistance Resources (AIDSTAR) Sector II IQC, contract number GHH-I-00-07-00064-00 AID-623-TO-11-00005, Kenya Health Information System.

DISCLAIMER:

The author’s views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government

(3)

The management of health services is an important function that has been devolved to the established County Governments to ensure the delivery is closer to the people, and is of continually improving quality. Consequently, National level and County Governments are obligated by law to give periodic reports and evaluate performance of the health outcomes and recommend appropriate actions. This has seen an increased demand for accountability and the need to demonstrate results at Country and County Levels. However, performance can only be adequately monitored if the data used is above reproach i.e. it is reliable and of high quality because this impacts its utility and value to users. Data which are incomplete, inaccurate, out of date, etc. has a negative impact on the quality and utility of the information used to disseminate results or to measure performance over time.

A countrywide Data Quality Audit (DQA) exercise was undertaken in 2010, the results of which went on to inform the implementation of a national web-based routine data collection system, the DHIS2 platform for reporting of routine health facility service delivery and community health services. Since its installation, there has been no other national data quality assessment or audit carried out in Kenya.

The DQA carried out in 2014 is the first nation-wide exercise since the rolling out of the DHIS reporting platform in 2011, whose main purpose was to provide baseline information on the quality of routine data on service utilization being captured in the current system. Additionally, the findings were to provide insights into the development of the Data Quality Improvement strategies and methods for Ministries of Health at County Level as well as National Level.

This report outlines the findings as well as recommendations for improving data quality. I am confident the content will provide appropriate guidance on data quality improvement strategies that can ensure data collected is acceptable for use in measuring performance, in evidenced based decision making and ultimately in improving the quality of health services at the grassroots levels.

Dr. Nicholas Muraguri

(8)

Acknowledgements

The Ministry of Health wishes to acknowledge the contribution of all the core team members who participated in the planning, preparation and implementation of the Nationwide Data Quality Audit (DQA) 2014 as a whole.

We wish to extend our gratitude to the field technical teams that comprised of Health Information officers from Div-HIME as well as several programs & departments within MOH.

Special thanks and appreciation goes to the Director of Medical Services Dr. Nicholas Muraguri as well as the Head of the Division of Health Informatics, Monitoring and Evaluation (Div-HIME) Dr. David Soti who both gave full support and guidance to the process. We especially appreciate the Head of the HIS Unit within Div-HIME, Dr. Martha Muthami who committed time as well as technical direction for the exercise which ensure successful implementation in the field.

We also appreciate the technical and financial support provided by USAID-AfyaInfo in conceptualization of the exercise, field activities, data consolidation and analysis and report writing. We also would like to thank MEASURE-Evaluation for their technical input into the report

(9)

Executive Summary

The country’s health sector performance can only be adequately monitored if the data used is reliable and of high quality. With devolution, national level and counties are obligated by law to give periodic reports and evaluate performance of National and County government with regard to health outcomes and recommend appropriate actions as stipulated in the various governing Acts.

The Data Quality Audit 2014 was the first nation-wide DQA since the rolling out of the DHIS reporting platform in 2011 for routine health facility service delivery and community health services. The last DQA was conducted in 2010 whose results informed the development of the DQA protocol. Since then, only vertical and sporadic DQAs have been conducted by programmes to meet localized needs.

A descriptive cross‐sectional design was utilized to collect data from 178 facilities of

which 33.7 % were GoK level 2 and 3, a further 30.3% were GoK level 4 & 5, and 6 while 15.7% were Faith Based Organisations and 20.2% were privately owned. The assessment utilised both qualitative and quantitative methods to verify the data from source documents for selected indicators against summary data, DHIS data, and Kenya HIV/AIDS Program Monitoring System (KePMS) data collected during the months of July- September 2013. Nine indicators were selected for this assessment taking into account the key programmatic areas in the health sector.

From the findings, it was noted that the reporting rates for the summary sheets/reporting forms for the assessed indicators was fairly high with MOH 711 (Integrated RH, HIV/AIDS, Malaria, TB and Nutrition) and MoH 705 A & B (Outpatient Morbidity) having a reporting rate of about 90%, MOH 515 (Community Health Extension Worker Summary) and MOH 710 (Immunization summary) had the lowest reporting rates ranging from 34.6% to 64.8%. The availability of audit documents ranged from 91.1% for number of women of reproductive age receiving family planning to number of pregnant women referred for ANC at 39.1%. However, the calibre of available documents ranged from the standard registers to improvised counter books to older versions of the registers. The number of fully immunized children had the least complete audit documents at 64%. Notably the private facilities had the highest rate of missing audit documents with availability of documents being as low as 29.4% for some indicators.

(10)

The accuracy of summary sheet data against source documents was calculated as the proportion of facilities assessed found to have matching records; findings were number of HIV+ pregnant mothers receiving preventive ARVs (36.6%), total number of patients currently on prophylaxis – Cotrimoxazole (32.6%), number of facility based maternal deaths (72.6%), number of women of reproductive age receiving family planning (6.2%,), number of pregnant women referred for ANC (32.1%), number of fully immunized children (22.9%), number of children under 5 treated for malaria (31.9%), number of new outpatient cases with high blood pressure (29.5%) and number of children under 5 who are under weight(31.4%) with an average accuracy of 30.8%. The MoH Level 4, 5 & 6 had least accurate data at 20.9%, followed by MoH level 2 & 3, at 26.8%, FBO at 31.8% and private facilities at 36%.

The accuracy of DHIS data against source documents was; number of HIV+ pregnant mothers receiving preventive ARVs (38.8%), total number of patients currently on prophylaxis – Cotrimoxazole (26.5%), number of facility based maternal deaths (75%), number of women of reproductive age receiving family planning (7.2%), number of pregnant women referred for ANC (24.2%), number of fully immunized children(16.1%), number of children under 5 treated for malaria (12.8%), number of new outpatient cases with high blood pressure (21.3 %) and number of children under 5 who are under weight (27.4%).

In addition, the accuracy of DHIS data against the Summary Sheets was established for the period. The number of HIV+ pregnant mothers receiving preventive ARVs had an accuracy of 58% , total number of patients currently on prophylaxis – Cotrimoxazole (32.9%), number of facility based maternal deaths (74.4%), number of women of reproductive age receiving family planning (13.2%), number of pregnant women referred for ANC (25.2%), number of fully immunized children (21.9%), number of children under 5 treated for malaria (12.1%), number of new outpatient cases with high blood pressure (60.7%) and number of children under 5 who are under weight

(35.6%). The average accuracy was 36.9%.

Verification of KePMS data against source documents established that only 19.6% of facilities had KePMS matching data with the source documents for total number of patients currently on prophylaxis – Cotrimoxazole. A comparison of DHIS data to KePMS established that only 28.9% had matching data for total Number of HIV+ pregnant mothers receiving preventive ARVs and 39.1% of the facilities assessed had matching data for total number of patients currently on prophylaxis – Cotrimoxazole.

(11)

Performance at Sub-County level in terms of reporting rate, completeness and timeliness was assessed and found to range from 88% for reports availability, timeliness at 82% and completeness at 77%. Noteworthy is that only the habitually reporting facilities are tracked by the sub counties.

On the qualitative aspect of the DQA, the systems assessment findings on factors affecting data quality were lack of training and support supervision for staff handling data, non-medical staff handling data (casuals), lack of data review measures, complex aggregation procedures, unclear indicator definitions especially for Immunization and HIV/AIDS, Family Planning, Malaria, and underweight with staff not sure what to count, chronic lack of tools resulting to improvising, lack of instructions especially on summary tools; some facilities not utilizing the standard tools and using those of partners, no written guideline available on data collection, aggregation, and manipulation procedures. In addition among the 44 health facilities assessed with Electronic Medical Record Systems (EMRs), the majority of them were installed for managing outpatient service utilization and for financial purposes rather than data generation and were either non-functional in data aggregation or malfunctioned when efforts were made to retrieve data.

The conclusion drawn from this DQA were that the accuracy of summary data and DHIS data against the source documents was generally low and was aggravated by several systemic issues including lack of standardized tools, governance, standard operating procedures, indicator definitions, and unclear roles and responsibilities. There was only a slight improvement of accuracy of DHIS data against summary sheets despite having qualified HRIOs keying in this data due to lack of aggregation instructions and multiple service delivery sites generating data. The low availability of audit documents in private health facilities highlights the minimal inclusion of these facilities in the national HIS.

Among the emerging recommendations were; sensitization and collaborative efforts by all stakeholders in investing in good data quality, development of data quality improvement plans to strategize on addressing the myriad of systemic issues affecting data quality, dissemination of the data quality assurance protocol, investment in technology to ease work load with regards to data and targeted efforts towards data use including data reviews and performance review forums, as well as regular data products generation and dissemination.

(12)

1. INTRODUCTION

1.1 Background

Establishing a robust health information system able to support performance monitoring of health programs and track progressive improvement of health of Kenya citizenry is one of the flagship projects of Kenya Vision 2030. The Constitution of Kenya 2010 states that every person has a right to the highest attainable standard of health which includes the right to health care services including reproductive health and hence the need for transparency and accountability and public participation in monitoring health sector performance. The investments and inputs in health, accompanied with effective and efficient management of resources should translate to better and demonstrable health outcomes.

In this regard, the quality of all health data and health related data needs to be beyond reproach in order to cast an authentic picture of progress and inform sound evidence based decision making process. This is of interest to County and National government who are obligated by law to give periodic reports and evaluate performance of National and County government with regard to health outcomes and recommend appropriate actions as stipulated in the County Government Act (2012) and Intergovernmental Relations Act (2012).

1.2 Health Information System

Health Information is one of the key orientations of the attainment of the Health Policy objectives in order to reach the overarching goal of ‘Better Health, in a responsive manner’. The (draft) KHSSP III has outlined its mission of deliberately building progressive, responsive and sustainable technologically-driven, evidence-based and client-centred health system for accelerated attainment of highest standard of health to all Kenyans. It has further outlined the impact targets which include reduction in neonatal and maternal deaths. A well-functioning Health Information System (HIS) is critical for evidence-based decision making and monitoring of the interventions geared towards the attainment of these targets.

Quality data is needed to inform the design of interventions and to monitor and evaluate plans and quantify progress towards predetermined treatment, prevention, and care targets. Attention to data quality ensures that target-setting and results reporting are informed by valid and sensitive information, and that reporting service providers are collecting and organizing this information in a consistent manner.

(13)

Quality data is data that is reliable, accurate, precise, and complete, provided in a timely manner, is truthful and maintains client confidentiality.

1.3 Data Quality Audit 2010

The last country-wide Data Quality Audit (DQA) was conducted in 2010 and among the key findings were;

 Timeliness of reporting and completeness of data was lower than expected. A comparison of the regions assessed in the three parameters revealed that all were above 60% for completeness, timelines and availability. Lower Eastern had all the three parameters being over 80% while Nyanza and Western had availability and completeness of over 90% though the timeliness was 80% for Western and 78% for Nyanza. The lowest performing region was North Eastern with availability of 79%, Completeness of 65% and timeliness of 69%.

 Data verification was either over-reported or under-reported for most of the indicators assessed. There was over-reporting for Women of Reproductive Age accessing FP commodities (105%), Pregnant Women receiving IPT2 (103%), New-borns with Low Birth Weight (115%), and PMTCT/ART at 122%. The verification revealed under-reporting of data for four ANC visits and Delivery by skilled health attendants in health facilities (99%) and gross under-reporting for maternal deaths in health facilities (90%).

 Data verification documents were not always available pointing to issues with storage of records

 Use of multiple tools to aggregate the data and the lack of data collection tools contributed to discrepancies observed in reported and recounted data

 Failure to use registers as per instructions was also noted while some indicators were not well understood.

Since then, only vertical DQAs spearheaded by programmes have been carried out albeit sporadically and the findings have been at best shared within the specific programmes. There had been no country-wide data quality audit carried out in Kenya since installation and use of the DHIS platform in 2011 for the reporting of routine health facility service delivery and community health services.

1.4 Data Quality Assurance Protocol

From the findings of the DQA report 2010, the Division of Health Information Systems (now the Division of Health Informatics and Monitoring and Evaluation – DivHIME) developed a Data Quality Assurance Protocol whose purpose is to provide a uniform

(14)

approach and framework in which all stakeholders could participate in ensuring data quality. The DQA protocol offers a guideline for the implementation of DQA by all departments in the Ministry of Health, development partners, Non-governmental Organizations (NGOs), private sector, Faith Based Organizations (FBOs) and Community Based Organizations (CBOs) (HIS Policy, 2010 - 2030). Data quality assurance strategies are envisioned to be implemented at all levels of the health system and to have a comprehensive approach to reflect national, regional, county and community coverage. The DQA 2014 will contribute to enriching the protocol and further elucidate on the roles of the different stakeholders in data quality assurance and improvement while highlighting the investments required.

1.5 Purpose of the Data Quality Audit

The delivery and management of Health Service has been devolved to the counties. Each County is responsible for the health outcomes of their inhabitants and therefore need to implement a sound Monitoring and Evaluation framework to keep track of its targets. The quality of data collected to track progress will need to be high to ensure that counties are making well informed decisions on where to deploy resources. This DQA serves to provide baseline information on the quality of routine data on service utilization being captured in the current system and provide insights into the development and/or improvement of the DQI strategies and methods for the national and county governments.

1.6 Objectives

The objectives of conducting the data quality audit were to:  Verify the quality for key indicators at selected sites/levels

 Assess the ability of data management systems to collect, manage and report quality service utilization data.

 Identify corrective measures and develop action plans for strengthening the data management and reporting system and improving data quality

(15)

2. METHODOLOGY

2.1 Assessment Design

A descriptive cross‐sectional design was adopted for this assessment targeting the

service utilisation data collected between the months of July- September 2013. The survey utilised both qualitative and quantitative methods to verify the data from source documents (registers) for select indicators against summary data (reporting forms), DHIS data (software), and KePMS data (software). The assessment also collected qualitative data on the data management systems to determine their ability to collect, manage and report quality data.

2.2 Sampling and Site Selection

The DQA was country-wide exercise conducted in 181 facilities selected across the 44 out of the 47 counties in the country (data from the counties of Garissa, Mandera, and Wajir was not collected due to (in) security circumstances surrounding the period of data collection.

In each county, four (4) facilities were selected, which comprised at least one hospital and 2-3 primary health-care facilities -Health Centres, Nursing homes, Medical Clinics, and Dispensaries. There was purposive inclusion of the county hospitals where necessary. For the remaining three facilities, one sub-county (district) was selected from each county and 3 health facilities were randomly selected based on purposive sampling approach with an effort to make the sample as representative as possible in terms of facility ownership (Government, Private, and FBOs) and type/level of facilities (Hospital, Health Centre, Nursing Home and Dispensary). The sample also included three national referral hospitals (Spinal Injury Hospital, Mathari Referral Hospital, and Moi Teaching and Referral Hospital). Sixty-two percent (62%) of the sampled facilities are government owned while the remaining 38% belonged to private or faith-based organizations. Thirty-eight (38%) of the facilities are hospitals while the remaining 62% of are primary health-care facilities. However, at the time of data collection, some facilities were found non-operational and this altered the proportions slightly resulting in the proportions illustrated inTable 1below.

(16)

Table 1: Distribution of DQA Facilities by Ownership/Level Ownership Number of Facilities in MFL % of Facilities in MFL Number of Facilities in DQA % of Facilities in DQA FBO 1210 14.2% 28 15.7% MOH (levels 2,3) 3775 44.3% 60 33.7% MOH (levels 4,5,6) 276 3.2% 54 30.3% Private 3262 38.3% 36 20.2% Total 8523 100% 178 99.9% 2.3 Indicator Selection

In consultation with different programmatic service delivery areas of the MOH, nine (9) indicators were selected taking into account representation of programmatic areas and types of registers and summary tools used collect data to feed into the indicators. The following programmatic areas and health service delivery units were included as part of the assessment: National AIDS/STD control programme (NASCOP)-two indicators, Maternal and child health (MCH) programme-three indicators, Outpatient diagnosis (OPD)-two indicators, and RH/FP service and community health service unit-one indicator each. The nine indicators and their corresponding registers and summary tools are listed inTable 2below:

Table 2: List of Indicators, Registers and Summary tools Programmatic

Area/ Health

Service Unit Indicators

Summary

Tool Register

NASCOP

Number of HIV+ pregnant mothers

receiving preventive ARVs (ANC only) MOH 711 ANC MOH 405

Total No. of patients currently on

prophylaxis - Cotrimoxazole MOH 711 Daily ActivitySheet

RH/MCH

Number of facility based maternal deaths MOH 717 Maternity MOH₃₃₃

Number of women of reproductive age

receiving family planning MOH 711 Family PlanningMOH 512

Number of fully immunized children MOH 710 Immunization_{MOH 510}

Number of children under 5 who are under

weight MOH 711 CWC MOH 511

OPD

Number of children under 5 treated for

malaria MOH 705 A

Under 5 OPD MOH 204A Number of new outpatient cases with high

blood pressure MOH 705 B Over 5 OPD MOH204B

Community Health Services

Number of pregnant women referred for

(17)

2.4 Assessment Tools

The assessment tool was adapted from the global DQA tool developed by Global Fund. The tool comprises two main components: data verification and data management and systems assessment sections. The data verification section used to assess the availability, completeness, and accuracy of data for each of the audited indicators. The data management and systems assessment section include interview questions to assess the strength of the underlying factors that may affect data quality. Generally, the quality of reported data is dependent on the underlying data management and reporting systems; stronger systems should produce better quality data.

2.5 Data Collection Process

A total of 22 teams each comprising of two officers were trained on data collection techniques using the RDQA tool (Error! Reference source not found.) during a three day workshop held at Lukenya Getaway. The teams had a one-day pre-test/practice on the tools during the training period. The pre-test/practice took place in Machakos Level 5 hospital, Bishop Kioko Mission Hospital and Muumandu Health Centre in Machakos County. An earlier pre-test of the tools by the DQA preparatory committee had taken place at Mbagathi Hospital in Nairobi County.

The data collection teams were deployed to the field to collect data in 44 counties. Each team was assigned to eight (8) facilities selected across two counties and also to collect sub county level data in each of the assigned counties. A total of eight (8) supervisors supported the teams with six (6) supervising the teams in the field and two supervisors assisting with the technical issues arising from the tools.

The period selected for review was July 1st_{, 2013 to September 30th, 2013, which}

corresponds to the 1st_{quarter of the 2013/14 fiscal year. All sampled facilities and}

sub-county health recording offices were visited between March 17 and March 28, 2014 excluding Lodwar County which was visited from 31st _{March to 4}th _{April 2014 due to}

travel logistics.

The data collection was done through document review for data verifications, and key informants interviews for systems assessment. Ideally, routine health data is collected in standardised registers at each facility where health-care services are delivered. Every month the staffs at the different primary health-care facilities collate the data and send monthly summaries on paper to sub-county Health Records and Information Officer (HRIO). The monthly summaries are then entered into web-based District Health Information Software (DHIS2) system by HRIO based within the sub-county health

(18)

office. Larger facilities such as referral hospital and sub-county hospitals have dedicated facility recording officers who themselves enter the facility data into the DHIS2 system. Sub-county HRIOs oversee the input of all data into the DHIS2.

With this background in mind, the data verification was done through recounting data from the source documents for each indicator. In addition, the teams copied the figure in the summary tool for the corresponding month. The recounted and reported values

were entered into

corresponding cells of the excel RDQA questionnaire. As per the values entered,

recounted figure were

compared to the reported values.

Summary statistics of all indicators are also calculated and presented graphically in the dashboard of the tool for each site and aggregation level scores.

In addition the data reported in the DHIS for the period was obtained and compared with the recounted and summary data. The same case applied for data reported through the KePMS.

Systems assessment was conducted through qualitative questions administered to health workers within the facilities to evaluate data management capacity. Information was collected on five areas of data management and reporting systems:

 Monitoring and evaluation capabilities, roles and responsibilities/ training  Indicator definitions and data reporting requirements

 Data collection tools and reporting forms

 Data management processes and data quality controls  Links with national reporting system

(19)

2.6 Data Analysis

The data collection Excel sheets from the different health facilities were aggregated to summarize the data. Due to technical issues with the tools and the magnitude of the data collected, the automatic summation could not take place and compiling the data was done manually.

A single sheet showing the summary of all the recounted data, summary data from the summary tools, DHIS data and KePMS data for the relevant indicators was prepared. A comparison across the different data sources was done and the proportion for accuracies calculated. The data was disaggregated by ownership and level – GoK, level 4, 5, 6; GoK level 2, 3, Faith based facilities and Private. The proportions were weighted in relation to the contribution of level/ownership to the national data.

The qualitative analysis/ system assessment utilized the summary scores obtained from scoring the different aspects of the five key areas under assessment. This was disaggregated into same level and ownership categories with a national outlook presented in a spider diagram. The comments and notes taken during the data collection exercise were analysed thematically and used to beef up the findings of the systems assessment and other qualitative findings. The Sub County analyses were also done to shed light on completeness, availability and timeliness of received reports.

2.7 Ethical Considerations

Relevant authorization from the Cabinet Secretary of Health was sought before the commencement of the exercise. Authorization from the relevant county organs was also sought and from facilities in charges. The data collection teams were briefed on confidentiality and care was taken to ensure that the facility records were treated with utmost care and no records were carried away from the facilities. Where photographs were taken to illustrate the qualitative findings, care was taken not to expose the patients’ details.

(20)

3. FINDINGs

3.1 Background

A total of 181 facilities were visited for the DQA exercise; of these 178 had data that could be used for the DQA. Sixteen percent (16%) were FBO, 34% MoH Levels 2 & 3, 30% MoH Levels 4, 5, & 6 and 20% were privately owned health facilities.

3.1.1. Availability of Services corresponding to Assessment Indicators

Among the facilities assessed, some did not offer some of the services that corresponded to the indicators under assessment and therefore did not contribute data to these indicators. The facilities not providing a given service were excluded from the denominators in the calculation of proportions in the different DQA findings as detailed in Table 3.

Table 3: Proportion of Facilities in the Sample Where the Services Corresponding to Indicators were not Available/Delivered

Indicator Service associated_{with indicator}

Facility Ownership FBO (n=28) MOH (levels 2,3) (n=60) MOH (levels 4,5,6) (n=54) Private (n=36) Number of HIV+ pregnant

mothers receiving preventive

ARVs PMTCT in ANC 10.1% (3) 26.7% (16) 0% (0) 36/1% (13) Total No. of patients currently

on prophylaxis - Cotrimoxazole Care and Treatment 17.8%(5) 36.7%(22) 1.8%(1) 50%(18)

Number of facility based maternal deaths Maternity / Deliveries 7.1% (2) 18.3% (11) 1.8% (1) 22.2% (8) Number of women of

reproductive age receiving

family planning Family Planning

32.1%

(9) 1.7%(1) 0%(0) 11.1%(4)

No. of pregnant Women

referred for ANC Community HealthUnit 25%(7) 28.3%(17) 113%(7) 44.4%

1 (16) Number of fully immunized

children Immunization 0% (0) 5% (3) 0% (0) 16.7% (6) Number of children under 5

treated for malaria

Outpatient for Under 5 years 0% (0) 0% (0) 1.8% (1) 2.8% (1) Number of new outpatient

cases with high blood pressure Outpatient for over5 years 0%(0) 1.7%(1) 1.8%(1) 2.8%(1)

Number of children under 5 who are under weight

Growth Monitoring for Children

0%

(0) 5%(3) 0%(0) 11.1%(4)

1 Ordinarily, Community units which report this indicator are attached to MoH and in some cases FBOs facilities. The finding could be as a result of data collection error

(21)

3.1.2 DHIS Reporting Rates for Indicators Assessed

The reporting rates of data contributing to the nine indicators assessed in the DQA were determined from the DHIS. While the reporting rates for the forms for the facilities in the sample were not calculated, the reporting rates in the DHIS were used as an indicator of the level of reporting for each of the indicator. These are illustrated in Table 4.

Table 4: DHIS2 Reporting Rates for Summary Tools Included in Audit Indicator

Summary Tool (n=expected reports

nationally)

Reporting Rate DHIS22

July 2013 August 2013 September 2013 Number of HIV+ pregnant

mothers receiving preventive ARVs

MOH 711(n=6,805)

Integrated RH,HIV/AIDS, Malaria, TB and Nutrition

90.9% 91.4% 89.8%

Total No. of patients currently on prophylaxis - Cotrimoxazole Number of women of

reproductive age receiving family planning

Number of children under 5 who are under weight Number of facility based maternal deaths

MOH 717(n=6,755)

Service Workload 89.7% 89.7% 88.8%

No. of pregnant Women referred for ANC

MOH 515(n=3,289)

Community Health Extension Worker Summary

54.3% 52.7% 54.6%

Number of fully immunized children

MOH 710 V1(n=5,613) Vaccines and Immunizations (version 1)

87.6% 86.8% 84.4%

Number of children under 5 treated for malaria

MOH 705 A(n=6,815)

Outpatient Summary < 5 years 91.7% 90.8% 91.4%

Number of new outpatient cases with high blood pressure

MOH 705 B(n=6,896)

Outpatient Summary > 5 years 90.1% 89.8% 90.3%

MOH 731-3(2375)

HIV/AIDS Care and Treatment 80% 81.7% 80.3%

2_{Reporting Rates for July-September 2013 as found in DHIS2 on 20th April 2014 calculated as No. of expected}

(22)

3.2 Data Verification

3.2.1 Availability and Completeness of Source Documents

In some facilities assessed, source documents i.e. standard registers, for different indicators were not available or were found to be incomplete for the auditing period.

Table 5 presents by indicator and facility levels/ownerships, the proportion of facilities

that were found at the time of the assessment to have source documents that were available and complete. Overall Pregnant women referred for ANC (39%), Number of fully immunized children (64%), Total number of patients currently on prophylaxis – Cotrimoxazole (73%) and Under 5 children underweight (73%) had the lowest availability rates of source documents.

In general, availability of source documents was found to be lower for private facilities than for government and faith-based facilities. In most cases the community-based service register (MOH514) was not available at facilities’ record offices for the auditing purpose as they were kept by Community Health Extension Workers (CHEWs) or Community Health Volunteers, in their homes. In some facilities HIV/AIDS care and treatment register (PRE-ART and Activity Sheets) were not available in the records offices because partners had custody of the registers.

Where available, the audit teams assessed completeness of a register based on the extent to which the required data elements were filled in. A completeness rate was therefore, defined as the percentage of audited registers that have complete records for the required data elements. The lowest register completeness (64%) was observed for child immunization CWC register (MOH510) while the highest (91%) was for high blood pressure OP register (MOH204B).

Some pages of source documents (such as CWC and OP registers) were also found missing (e.g. torn out) due to mishandling of the tools. Reasons cited for this, was size of the registers and presence of too many handlers including students. The results show that incompleteness of source documents was more pronounced among private facilities compared to government and faith-based facilities.

In several facilities, tally sheets were presented as source documents particularly for immunization, whereas their true purpose is for intermediary use to aggregate numbers for different services provided.

(23)

Table 5: Availability and Completeness of Source Documents

Indicator/Data collection tool (%)* (n=28)FBO

GoK Level 2-3 (n=60) GoK Level 4-6 (n=54) Private (n=36) Weighted/National Number of HIV+ pregnant

mothers receiving preventive ARVs (ANC MOH 405)

Available (%) 91.3 80.4 90.2 56.5 73.1%

Complete (%) 68.4 82.3 87.0 63.6 73.3%

Total No. of patients currently on prophylaxis – Cotrimoxazole (Activity Sheet)

Available (%) 90.0 60.5 86.0 50.0 61.5%

Complete (%) 81.2 90.0 80.9 66.7 79.5%

Number of facility based maternal deaths (Maternity MOH 333) Available (%) 82.6 89.6 88.6 86.4 87.3% Complete (%) 88.2 92.5 89.0 77.8 86.1% Number of women of reproductive age

receiving family planning (FP MOH 512)

Available (%) 84.2 91.2 94.1 90.0 89.8%

Complete (%) 93.3 86.3 89.4 76.9 83.8%

Number of pregnant women referred for ANC (MOH 514) Available (%) 46.7 46.3 33.3 29.4 39.5% Complete (%) 55.4 87.5 90.0 66.5 75.0% Number of fully immunized children (Immunization MOH 510) Available (%) 83.3 89.5 88.2 56.5 75.9% Complete (%) 65.0 74.5 53.3 58.3 66.3%

Number of children under 5 treated for malaria (OPD MOH 204A)

Available (%) 92.0 96.6 72.5 66.7 83.7%

Complete (%) 87.0 94.0 75.7 95.2 92.9%

Number of new

outpatient cases with high blood pressure (OPD MOH 204B)

Available (%) 88.0 87.7 68.6 69.7 80.2%

Complete (%) 90.0 94.6 80.0 100 _95.5%

Number of children under 5 who are under weight (CWC MOH 511)

Available (%) 73.9 87.7 64.7 58.3 73.7%

Complete (%) 75.0 96 72.7 92.3 90.9%

*It should be noted that the ‘n’ shown in the title of columns is the TOTAL facilities assessed in each category. However, the percentages were calculated ONLY for those providing the service as detailed in Table 3.

(24)

3.2.2 Availability and Completeness of Summary Tools

In some facilities, summary reports were also not available or were found to be incomplete during the data verification exercise. It was not possible to access summary reports for HIV/AIDS, community health services, child immunization and underweight indictors in more than 10% of the facilities visited. Some summary reports were incomplete and boxes for recording summary figures found blank e.g. the maternal deaths data. Therefore it was not possible to tell whether there were no maternal deaths or if data was missing.

Unavailability or incompleteness of summary reports was more common for private and faith-based facilities than government facilities. The audit teams observed that some facilities used photocopied forms citing a shortage of reporting tools. In some facilities where there were multiple service delivery points, including community outreach and satellite clinics, other summary forms were used as intermediaries and collectively used to aggregate the final summary tools.

3.2.3 Missing Data/Value for Indicators under assessment for both Source and Summary Tools

The assessment found that not all facilities with audit documents had data available to be audited for the months under review. This is illustrated inTable 6

.

The missing data was especially high for – ‘Number of HIV+ pregnant mothers receiving preventive ARVs’ (private), ‘Total number of patients currently on prophylaxis – Cotrimoxazole’ (FBO, MOH levels 2 &3, and Private) and the ‘Number of pregnant Women referred for ANC’ across all facilities. The recording of ‘Number of fully immunized children’ in private facilities was generally poor.

(25)

Table 6: Proportion of Missing Data/Value for Indicator (source documents and/or summary sheet) in Facilities Where Service is provided

Indicator Facility Ownership FBO (n=28) MOH (levels 2,3) (n=60) MOH (levels 4,5,6) (n=54) Private (n=36) Number of HIV+ pregnant mothers

receiving preventive ARVs

24% (6/25) 11.4% (5/44) 5.5% (3/54) 43.4% (10/23) Total No. of patients currently on

prophylaxis - Cotrimoxazole 30.4% (7/23) 26.7% (8/30) 11.3% (6/53) 55.5% (10/18) Number of facility based maternal

deaths 30.7% (8/26) 14.3% (7/49) 7.5% (4/53) 35.7% (10/28) Number of women of reproductive age

receiving family planning

15.7% (3/19) 0% (0/59) 3.7% (2/54) 9.4% (3/32) No. of pregnant Women referred for

ANC 61.9% (13/21) 25.6% (11/43) 46.8% (22/47) 70% (14/20)

Number of fully immunized children 14.3%

(4/28) 7.0% (4/57) 9.2% (5/54) 63.3% (19/30) Number of children under 5 treated for

malaria 7.1% (2/28) 5% (3/60) 7.5% (4/53) 14.3% (5/35) Number of new outpatient cases with

high blood pressure

3.6% (1/28) 3.4% (2/59) 13.2% (7/53) 14.3% (5/35) Number of children under 5 who are

under weight 28.6% (8/28) 8.8% (5/57) 12.7% (7/54) 40.1% (13/32)

3.2.4 Accuracy of Summary Sheet against the Source Documents

From the recounting done on the available source documents, the assessment determined the extent to which the data on the summary sheets reflected what was recounted in the source documents. The findings are illustrated in Table 7. The accuracy of data for majority of the indicators under assessment was very low especially for ‘Number of women of reproductive age receiving family planning’. The explanation for this was that it was not clear what constituted family planning. Some facilities recorded condoms while others did not and some facilities included men issued with condoms in the summation. It was also not clear on how to treat first time visits and revisits. Another example was the definition of a fully immunized child with

(26)

some facilities recording Measles vaccine at 1 year, 2nd _{Measles vaccine at 18 months}

while others recorded Measles vaccine whether the child was up to date with other vaccines or not. Poor recording of diagnoses of malaria with multiple terms being used also contributed to the inaccuracies between the two records.

It was also noted that amongst the four categorized health facilities, MOH hospitals at levels 4, 5 and 6 were found to have the lowest levels of accuracy for several indicators even where there were designated HRIOs on site. Most of the data reported in the summary sheet did not match what was recounted in the source documents. Some reasons cited included multiple service delivery points, high volume of clients/patients versus low staff numbers, unsupervised recording by students on site and handling of source documents by multiple staff.

Table 7: Proportion of facilities with Source documents data matching Summary Sheet data

Indicator Summary Sheet % Accurate FBO (n=28) MOH (levels 2,3) (n=60) MOH (levels 4,5,6) (n=54) Private (n=36) Weighted National (n=8523) Number of HIV+ pregnant

mothers receiving preventive ARVs MOH 711 32% (8/25) 40.9% (18/44) 20.3% (11/54) 34.7% (8/23) 36.6%

Total No. of patients currently on prophylaxis -Cotrimoxazole MOH 711 21.7% (5/23) 36.7% (11/30) 15.1% (8/53) 33.3% (6/18) 32.6%

Number of facility based

maternal deaths MOH 717

69.2% (18/26) 85.7% (42/49) 49% (26/53) 60.7% (17/28) 72.6% Number of women of reproductive age

receiving family planning

MOH 711 21% (4/19) 1.7% (1/59) 3.7% (2/52) 6.2% (2/32) 6.2%

referred for ANC MOH 515

28.5% (6/21) 34.9% (15/43) 34% (16/47) 30% (6/20) 32.1% Number of fully

immunized children MOH 710

21.4% (6/28) 3.5% (2/57) 3.7% (2/54) 47,4% (9/19) 22.9%

Number of children under 5 treated for malaria

MOH 705 A 35.7% (10/28) 21.7% (13/60) 24.5% (13/53) 42.9% (15/35) 31.9% Number of new

outpatient cases with high blood pressure MOH 705 B 32.1% (9/28) 27.1% (16/59) 28.3% (15/53) 31.4% (11/35) 29.5%

Number of children under

5 who are under weight MOH 711

25% (7/28) 29.8% (17/57) 9.2% (5/54) 37.5% (12/32) 31.4% Average 31.8% 26.8% 20.9% 36.0% 30.8%

(27)

3.2.5 Accuracy of DHIS Data against the Source Documents

The accuracy of DHIS data against the source documents was assessed for the period under review to establish whether the entries matched what was in the source documents. Notably, the proportion of matching data between Source documents and DHIS was very low for all indicators under review. The ‘Number of women of reproductive age receiving family planning’ had the lowest accuracy at 7.2%, while the ‘Number of children under 5 treated for malaria’ had an accuracy of 12.8% while the ‘Number of fully immunized children’ was at 16.1%. The overall average for matching source/DHIS data was at 27.7% for the data of the indicators assessed. The results are illustrated in Table 8. Notable, was the poor performance of MoH levels 4, 5, & 6 in all indicators despite having trained HRIOs and accessibility to Electronic Medical Records.

Table 8: Proportion of facilities with Source Documents data matching DHIS2 Data3

Indicator Summary_Sheet

% Accurate FBO (n=28) MOH (levels 2,3) (n=60) MOH (levels 4,5,6) (n=54) Private (n=35) Weighted National (n=8523) Number of HIV+ pregnant

mothers receiving preventive ARVs

MOH 711 _(8/25)32% _(19/44)43.2% _(3/54)5.5% _(9/23)39.1% 38.8%

Total No. of patients currently on prophylaxis -Cotrimoxazole

MOH 711 _(4/23)17.4% _(9/30)30% _(1/53)1.9% _(5/18)27.8% 26.5%

Number of facility based

73.1% (19/26) 85.7% (42/49) 62.3% (23/53) 64.3% (18/28) 75.0% Number of women of reproductive age receiving family planning MOH 711 21.1% (4/19) 6.8% (4/59) 0% (0/54) 3.1% (1/32) 7.2%

19% (4/21) 25.6% (11/43) 19.1% (9/47) 25% (5/20) 24.2%

Number of fully immunized

children MOH 710 10.7% (3/28) 5.3% (3/57) 3.7% (2/54) 31.6% (6/19) 16.1%

5 treated for malaria MOH 705 A

14.3% (4/28) 11.7% (7/60) 3.8% (2/53) 14.3% (5/35) 12.8%

Number of new outpatient cases with high blood pressure

MOH 705 B _(6/28)21.4% _(15/59)25.4% _(8/53)15.1% _(6/35)17.1% 21.3%

21.4% (6/28) 19.3% (11/57) 9.3% (5/54) 40.6% (13/32) 27.4% Average 25.6% 28.1% 13.4% 29.2% 27.7%

(28)

3.2.6 Accuracy of DHIS Data against the Summary Sheets

The assessment further established the extent to which the data in the DHIS corresponded with the data in the summary sheets for indicators under review. The findings are illustrated in Table 9. Some indicators performed poorly especially ‘Number of children under 5 treated for Malaria’ at 12.1%, “Number of women of reproductive age receiving family planning’ at 13.2% and ‘Number of fully immunized children’ at 21.9%.

Noteworthy was the extent of inaccurate DHIS data against the Summary sheets for MoH levels 4, 5, & 6 despite the availability of trained records officers. Among the reasons for this was multiple aggregation sites and lack clear indicator definitions.

Unlike the results of source documents versus DHIS where the matching records were found to be all below 50%, ‘Number of facility based maternal deaths’ had 74.4% of summary sheets matching DHIS, ‘Number of new outpatient cases with high blood pressure’ had 60.7% while ‘Number of HIV+ pregnant mothers receiving preventive ARVs’ had 58.0% of records matching. The reason for the seemingly high performance on maternal deaths data was mainly because most of the data to be matched comprised of zeros or blanks.

(29)

Table 9: Proportion of facilities found Summary Sheets data matching DHIS data Indicator Summary Sheet % Accurate FBO (n=28 MOH (levels 2,3) (n=60) MOH (levels 4,5,6) (n=54) Private (n=35) Weighted National (n=8523) Number of HIV+ pregnant

mothers receiving preventive ARVs MOH 711 48% (12/25) 68.1% (30/44) 33.3% (18/54) 52.2% (12/23) 58.0% Total No. of patients

currently on prophylaxis -Cotrimoxazole4 MOH 711 26.1% (6/23) 36.7% (11/30) 5.7% (3/53) 33.3% (6/18) 32.9% Number of facility based

69.2% (18/26) 81.6% (40/49) 75.5% (40/53) 67.9% (19/28) 74.4% Number of women of reproductive age

receiving family planning5

MOH 711 26.3% (5/19) 10.2% (6/59) 3.7% (2/54) 12.5% (4/32) 13.2% No. of pregnant Women

23.8% (5/21) 25.6% (11/43) 27.7% (13/47) 25% (5/20) 25.2% Number of fully

immunized children MOH 710

17.6% (5/28) 19.3% (11/57) 20.4% (11/54) 26.7% (8/30) 21.9%

Number of children under 5 treated for malaria6

MOH 705 A 7.1% (3/28) 10% (6/60) 3.8% (2/53) 17.1% (6/35) 12.1% Number of new

outpatient cases with high blood pressure MOH 705 B 67.9% (19/28) 66.1% (39/59) 43.4% (23/53) 53.3% (16/35) 60.7% Number of children under

32.1% (9/28) 36.8% (21/57) 11.1% (6/54) 37.5% (12/32) 35.6% Average 35.3 39.3% 24.9% 35.8% 36.9%

4_{From DHIS2: Total on Cotrimoxazole for September 2013}

5_{From DHIS2: Total of Family Planning includes: Women Receiving Microgynon, Woman Receiving Microlutin,}

Female Sterilizations, IUD insertion, FP injections and all other FP methods

(30)

3.2.7 Verification of KePMS data against Source Documents

KePMS (Kenya HIV/AIDS Program Monitoring System) is an indicator-monitoring database designed to help USG Kenya HIV/AIDS Program Implementing Partners to report on, monitor and evaluate HIV/AIDS care, treatment and prevention programs supported by PEPFAR. KePMS is supported and managed by the AfyaInfo Project from since 2011. KePMS allows in-country USG implementing partners to manage their program-specific data and forward those data to the Kenya USG program managers, where KePMS is able to aggregate and collate the data to meet the reporting needs of the USG program managers. Two indicators whose data is collected through the KePMS were assessed for accuracy against the source documents. The findings are illustrated below.

a. Number of HIV+ pregnant mothers receiving preventive ARVs

Ordinarily the PMTCT data is collected from ANC and Maternity; however, the audit only focused on ANC aspect of the PMTCT which is usually reported in MoH 711 summary sheet as a part disaggregate of total PMTCT data. The MoH 731 form from which KePMS data is reported has no provision for disaggregation of ANC PMTCT data and Maternity PMTCT data. Therefore, considering that the ANC PMTCT data is subset of total PMTCT data, the findings here can only be used as proxy to indicate the extent to which source documents data compares with KePMS data.

Nine out of 69 facilities referred to as USG supported sites were reportedly not offering PMTCT service. Verification factor (Recounted data/reported data*100) of KePMS data against source documents showed under-reporting of PMTCT data at 122%.

b. Total number of patients currently on prophylaxis - Cotrimoxazole

Thirteen out of 69 facilities were reported not to offer the service. Eleven out of 56 (19.6%) facilities had data matching with source documents. However, seven out of these reported zero for both recounted and KePMS data, therefore it was not clear whether this was an error or missing data. The verification factor for KePMS data showed slight over reporting at 98% as illustrated inTable 10

.

Table 10: Verification of KePMS data against Source Documents

Indicator Facilities offering_Service Source/KePMSFacilities with matching data

Verification factor of KePMS against source

documents

PMTCT 60/69 (87%) 13/60 (21.7%) 122%

(31)

3.2.8 Comparison of DHIS data To KePMS data

a. Total Number of HIV+ pregnant mothers receiving preventive ARVs

Out of the 69 supported sites, 45 facilities had data in one of the two systems, i.e. either KePMS or DHIS. Out of the 45, only 13 facilities (28.9%) data in DHIS matched with their data in KePMS. However, it was noted that 31 facilities had blank data in the DHIS while another 31 facilities had either blanks or zeros7_{in the KePMS, though not necessarily in}

both systems.

b. Total number of patients currently on prophylaxis - Cotrimoxazole

From 69 supported sites, 46 had either data in DHIS or KePMS. From these only 18 (39.1%) facilities had matching data in both DHIS and KePMS. Notable was that 27 of the 69 facilities had blanks in DHIS and 33 facilities had either zero or blanks in KePMS data though not necessarily the same facilities for the two systems.

Table 11: Comparison of DHIS data against KePMS

Indicator

No. of Facilities / USG supported sites with blanks

entries in DHIS

No. of facilities / USG supported sites with zeros

or blanks entries in KePMS Facilities with data in either DHIS or KePMS Facilities with DHIS/KePMS matching data PMTCT 31 31 45 13/45 (28.9%) CTX 27 33 46 18/46 (39.1%)

7_{From the way KePMS is organized, a zero or a blank in facility may not necessarily mean that the service is}

not offered or the count is zero; it may mean that there could be more than one USG supported partner supporting the site. Each partner linked to a site will enter the value for the service they support and a zero for the service they do not support but which may be supported by another partner in the same facility.

(32)

3.3 Sub County Level Findings

3.3.1 Sub County Reports Availability, Completeness and Timeliness Systems assessment was done through interviews conducted with Sub-County HRIOs (SCHRIOs) and other data managers at the sub-county level. Ordinarily, all facilities regardless of ownership or level are expected to submit summary reports on key health indicators on a monthly basis. Reporting performance was measured by percentage of reports available, complete (reported on all services offered), and submitted on time (i.e. by 5th_{of every month) from all facilities for the period under audit.}

Percent reports availability: is defined as the actual number of facilities that submitted reports, divided by the total number of facilities that are expected to submit the reports.

Percent reports complete: defined as the number of facilities that submitted complete reports for all key indicators for the services offered divided by the total number of facilities that are expected to submit the reports.

Percent reports timely: defined as number of facilities that submitted reports within a set deadline (5th _{of every month) divided by the total number of}

facilities that are expected to submit the reports.

The assessment teams conducted an assessment for one sub-county in each of the 44 counties visited. Overall all sub-counties average indicate that reports availability rate had the highest performance at 88%, followed by timeliness rate with about 82%, while completeness rate was found to be the least (77%). Figure 2 shows that out of the 34 sub-counties for which complete data were available, 24 counties (70%) had more than 90% availability rate. However, only 14 sub-counties (41%) had more than 90% rate of completeness. This implies that though many facilities submitted reports in time, in most cases their reports were not complete. However, the findings below do not take into account the facilities that do not habitually report, especially the private facilities that in many cases are not included as part of the denominator (of expected reports).

(33)

Figure 2: Number of Sub-counties by proportion of reports Available, Completeness and Timeliness

3.3.2 Systems Assessment- Sub County Level

For systems assessment, the audit teams used the standard data management and reporting systems assessment section of the RDQA tool. Interviews were conducted with those managing data both at facility and sub-county levels and responses recorded in the MS Excel template. Five functional areas were assessed namely

 M&E structure, functions and capabilities

 Indicator definitions and reporting guidelines

 Data-collection and reporting forms and tools

 Data management processes; and

 Links with national reporting system.

Following are the sub-county level findings (Figure 3). Data management processes were the poorest performing at 2.38 at a scale of 0 to 3, while links with the national systems was the best performing at 2.88 on the same scale.

(34)

Figure 4: Sample Checklist Tool used to track reporting performance per health facility for each dataset expected per month (T=Timely; L=Late)

2.48

2.67

2.74

2.38

2.88

I. M&E Structure, Functions and Capabilities II. Indicator Definitions and Reporting Guidelines III. Data-collection and Reporting Forms / Tools IV. Data Management Processes V. Links with National Reporting System

(35)

3.4 Systems Assessment- Facility Level

3.4.1 Overall Scores

Table 12 presents the summary statistics for the five functional areas of system assessment results by facility levels/ownerships. The color-coded values correspond to the level of strength of the five functional areas. The green colour indicates very strong system, while the yellow and red colours correspond to medium and low levels of strength, respectively. The results show that the average score ranged from 2.1 to 2.8, for the ‘Structure, Function and Capabilities’ and for the ‘Link with National Reporting Systems’ functional areas, respectively. The second lowest score (2.3) was recorded for the ‘Data management process’ while the second highest score (2.7) for ‘Data-collection and Reporting Forms/Tools’ functional area. Overall, government level 4-6 facilities and faith-based facilities have relatively stronger data management and reporting systems than private and lower-level government facilities. However, it must be pointed that this has not translated to improved data accuracy as illustrated by data verification results.

Table 12: Facilities Summaries of Scores on Data Management and Reporting Systems Assessment

Assessment of Data Management and Reporting Systems

Service Delivery Points by Level / Ownership M&E Structure, Functions and Capabilities Indicator Definitions and Reporting Guidelines Data-collection and Reporting Forms / Tools Data Management Processes Links with National Reporting System Per Facility overall average GoK, Level 4-6 2.32 2.63 2.68 2.32 2.78 2.56 GoK Level 2-3 2.05 2.42 2.69 2.17 2.80 2.43 Faith-based Facilities 2.06 2.61 2.58 2.45 2.83 2.51 Private Facilities 2.02 2.26 2.47 2.21 2.65 2.32 Average (per functional area) 2.10 2.48 2.66 2.26 2.76 2.44

Colour Code Key

Green 2.5 - 3.0 Strong_Systems Yellow 1.5 - 2.5 Someweaknesses identified

(36)

3.4.2 Monitoring and Evaluation Structure, Functions and Capabilities

a. Human Resource and Training

Most of the facilities visited, particularly at level 2-3 facilities, did not have organizational charts or clearly documented job descriptions for staff managing data. In about 40% of the facilities, the responsibility of recording service deliveries was not clearly assigned to the relevant staff and positions necessary for data recording, aggregating and reporting were not filled (50% of facilities). These findings are illustrated in Figure 5. Occasionally due to high work load, the filling of registers and summary tools was delegated to untrained staff (casual workers). Mistakes resulting from this were evident during the DQA exercise – an example is a dispensary in Kilifi County where the untrained staff listed all children on one page of the register as underweight by simply inserting a ‘Y’ in the column for underweight (Figure 6).

It was observed that health workers feel that data recording and reporting activities are only secondary to their service delivery responsibilities. In addition, some health workers who are responsible for data recording have not received training on indicators definitions, tools and data management. This was reported in about 80% of the facilities. In some facilities, training is given only to HRIOs while most nurses have not received any training. In addition, most HRIOs lack basic technical skills on computer applications and data analysis.

(37)

Figure 6: Snapshot of actual records showing all

children weighed recorded as 'Y' in the Underweight Column (confirmed as erroneous)

It was also reported that regular supervisory visits from the higher-levels, to provide technical assistance and on job trainings were lacking.

b. Data Quality Control

Close to 50% of all facilities visited had no regular data quality control system to identify and correct errors before submitting summary reports, or before entering data to the DHIS system where it was keyed in directly by facilities. Irrespective of facility level and ownership, about half of the facilities indicated that the quality of their data was not consistently reviewed before submission. In most cases data review was done on ad hoc basis when there were concerns raised externally regarding the quality of the data, or in quarterly meetings.

This was evident from the data recording and aggregation errors observed, the likely cause for the disparities in the reported and recounted figures during the data verification. Missing data/blank cells, entering incorrect records in the registers and illegible handwritings were among the major errors observed (See Figure 7).

Lack of uniformity in recording service delivery data was another observed finding. For instance, for malaria treatment the column for recording diagnosis result was filled out with “Malaria”, “Mal”, or “Cl”. In such cases, it was difficult to differentiate between

(38)

Figure 7: Snapshot of register records illustrating poor handwriting, limited space to record important data such as diagnosis or treatment, and poor recording even where space is adequate

clinical and confirmed malaria cases. Similarly, for the underweight indicator in some cases recorded as “normal” instead of writing ‘Y’ for underweight and ‘N’ for not underweight.

Errors were also reported during counting, aggregating and transferring data from source documents into summary tools by HRIOs. Audit teams noted that many inconsistencies between recounts and summary numbers were caused by arithmetic errors. Some facility staff reported that they were not clear on what to count. The audit teams also noted that some registers and summary tools had very small writing space provided to fill out records, and as a result in some cases numbers and letters recorded were illegible. Intermediary aggregations errors from multiple service delivery points led to cumulated errors in the final summary report.

3.4.3 Indicators Definitions and Reporting Guidelines

The Health Informatics Monitoring and Evaluation Division (formerly Div-HIS) of the MOH developed the standard operational definitions for all audited indicators at national level. However, the audit team noted that most facilities had no written documents on indicator definitions. Facility-level recording staff mostly relied on their common knowledge from pre-service training or verbal instructions given by DHRIOs or colleagues for definitions of indicators.

(39)

The audit team observed lack of common understanding of operational definitions for some indicators such as HIV/AIDS treatments, family planning, child immunization and underweight indicators. It was found to be difficult for recording officers to clearly distinguish new patients from revisiting clients for preventive ARVs, family planning, and high blood pressure indicators as outlined below:

i. Number of HIV+ pregnant mothers receiving preventive ARVs

The definition of preventive ARV indicator for pregnant women required counting pregnant women tested HIV positive who received ARV prophylaxis on their first visit to the ANC. However, there

was no clear guideline to ascertain when the clients knew their HIV positive status, and the majority of service providers filled in the column for every visit. This resulted in double counting in cases of treatments during re-visits which are also recorded in the register and included in the

monthly summary

reports. It was also unclear how to count

missing clients whose lab results were positive and received ARV prophylaxis on their subsequent visits.

ii. Total Number of patients currently on prophylaxis - Cotrimoxazole

The definition for ‘patients currently on Cotrimoxazole prophylaxis’ refers to both new patients registered in the reporting period plus existing patients excluding dropouts, transfers and those who died in the previous reporting period. This complex definition was not well understood by some staff who compile summary reports; particularly new employees and those who had not received training on the same. Furthermore, there are no clear instructions given in the MOH711 on what to count and how to aggregate and fill out summary tool. In most cases patients were supplied with drugs that lasted for more than one reporting period (one month) therefore was not clear how to count them. In addition, the manual counting process across periods was complex and tedious. During the verification process, it was observed that most errors occurred in the process of compiling monthly summary reports by aggregating thousands of cases