Note on data limitations and how they were addressed

In addition to the caveats on design noted in the inception report (Witter et al., 2014), a number of additional limitations emerged in the course of the review. These included the following.

Our examination of the quantitative data sources, particularly the HMIS and DHS, showed more data-quality problems than we were expecting (see Annex E for details). This has restricted the range of analyses that we could do and the conclusions that we could draw.

1. The HMIS had the following issues:

 The micro data from before April 2011 have been lost.

 An independent review by Options Consultancy (2015) showed that, in many cases, there were significant inconsistencies between the data recorded in the database and the actual situation recorded in health facility registers.

 Although overall response rates for HMIS are high, with over 90% of facilities reporting each month, the level of non-response for individual variables is much higher. The sample of facilities and variables we checked showed missing values for between 20% and 40% of cases.

These weaknesses mean we can only look at pre-2011 HMIS trends using the few tables that have already been published. In addition, results from the HMIS analysis need to be triangulated with other sources before we can draw conclusions.

2. The DHS had particular quality concerns in the 2008 survey. These are evident from the distributions of ages of the participants in the survey, which do not match the known population profiles from the census. The problems arose from poor supervision of the fieldwork, and it is likely that this is the reason for the apparent inconsistent results between the 2008 and 2013 rounds in, for example, the area of child mortality.

As a result of the weaknesses in the 2008 DHS, we have focused on the 2013 DHS as our main source. We have only used the 2008 survey where necessary, for example to look at changes in

relation to equity issues using the disaggregations by wealth quintile and where the 2008 survey is judged the best available baseline.

3. Concerns about the accuracy of NHA data, especially for household expenditure, which could suggest biases in opposing directions (see Box 3).

Box 3: Limitations of sector trend analysis using NHA data

The expenditure data for the sector analysis in the report have certain limitations, and caution should be exercised in their interpretation. Issues affecting the NHA trend analysis include:

1) The three sets of NHA data (2004–2006, 2007–2010 and 2011–2013) use different methodologies and vary in their scope and quality.

2) NHA data for 2011 and 2012 are not yet available and thus estimated annualised growth rates were used for these years, potentially missing any yearly variation.

3) Household expenditure changes are largely a reflection of price movements, with figures in the NHA for 2005 to 2010 taken from the 2004 Living Standards Survey and adjusted for inflation (the 2013 NHA uses the 2011 Living Standards Survey). Therefore, any change in donor or GoSL financing that is different to inflation will automatically change the composition of health expenditure in the sector.

4) Expenditure figures for GoSL in 2004 to 2006 are three times higher than government accounts report for those years, which are more in line with 2007–2013 NHA data. If these NHA figures are not correct, and the NHA reports do not comment on this anomaly, it artificially suggests a slowdown in health expenditure pre-FHCI for the period 2004–2009. This would therefore overestimate the change that FHCI created when comparing 2004–2009 trends with post-2010 trends.

5) Anecdotal evidence suggests the quality of the NHA in the period 2007–2010 is limited, with some suggestions that double counting occurred. This would suggest that the increase in expenditure

post-2010 is underestimated.

There is also missing data for donor expenditure in 2008; for example, two large donors – the World Health Organization (WHO) and the Global Fund – are missing entries for NHA expenditure. This therefore artificially suggests a large rise in expenditure between 2008 and 2009 pre-FHCI, thus limiting

comparisons with post-FHCI trends.

We have retained and used these data sources but with appropriate triangulation of results, where possible, and cautious interpretation.

In addition, there were data sources that we hoped to use but which were simply not available for a variety of reasons. Issues surround this included the following:

 Some of the quality of care indicators, which were not available in the HMIS or other sources, had to be replaced by more qualitative assessments of changes to quality of care.

 The same is true of staff competence – there are no national sources for this indicator, which cannot therefore be integrated into our analysis and constitutes an important gap.

 We planned to integrate findings from a PhD thesis on informal payments but due to Ebola that PhD was transferred to another setting.

 We had also hoped to use the data collected by the Health For All Coalition (HFAC). The data provided by HFAC were very incomplete both in terms of the months and modules covered. The figures also did not match published tables and included very erratic and unlikely trends that could not be explained.

A third type of limitation to note is the assumptions that are built into particular models. In

particular, for the LIST tool, inbuilt assumptions of the effectiveness of core maternal, newborn and child health (MNCH) interventions are used to convert coverage to outcome changes. These are based on international literature. In the absence of Sierra Leonean evidence, we have used these. However, they may overstate gains if the quality of care provided is below expected levels to

ensure effectiveness (as suggested by the evidence of this evaluation). The infeasibility of disaggregating DHS coverage data for under-five interventions into individual years also meant that we could not do incremental trend analysis here (despite this being important for the LiST model counterfactual). Box 4 sets out further details on the limitations relating to the cost- effectiveness estimates.

Box 4: Limitations of the cost-effectiveness analysis (CEA)

There are two key limitations to the CEA. First, drawing boundaries around what interventions—and their associated costs and outcomes—are, or are not, linked to the FHCI is challenging. Our approach is to try to ensure that what is included on the cost side is matched on the effect side; however, this is inevitably determined to some extent by the available cost and outcome data.

Second, limitations to data sources on both the costs and outcomes of the FHCI means that the true incremental costs and effects of the FHCI are difficult to isolate. The lack of 2011 and 2012 NHA data is the key limitations on the cost side. On the effect side, coverage data is used to model the effect on maternal, newborn and child mortality using the LiST. It is important to acknowledge that LiST is a tool that allows for the modelling of impact, not the actual estimation of impact using data on the impact variables and an appropriate impact evaluation technique. The conclusions that can be drawn from such an exercise are different from those that could be drawn from an actual impact evaluation. It is also worth highlighting a few key limitations of LiST:

 LiST uses inbuilt assumptions about the effectiveness of MNCH interventions to convert increased coverage estimates to mortality reductions. These are based on international literature. In the absence of evidence from Sierra Leone, we have used these. However, they may overstate gains if the quality of care provided is below expected levels to ensure effectiveness.

 Family planning is a difficult intervention to model in terms of maternal lives saved. This is essentially because of lack of data on abortion practices. We take the approach of modelling maternal deaths averted in the context of the increased contraceptive prevalence rate (CPR) of Sierra Leone from 2008 to 2013. To the extent that the FHCI contributed to the increase in family planning over the period (family planning was free before 2010 but increased utilisation of health facilities by women of reproductive age likely increased its use), we are therefore underestimating the demographic impact of the FHCI on maternal deaths.

Finally on the effect side, there are important limitations to the DHS data used to estimate increases in coverage due to the FHCI. Incremental trend analysis cannot be undertaken for the child interventions and for a number of the maternal and newborn interventions. This is because the recall variable for the

indicator is too short to allow for annual disaggregation of the data. When incremental trend analysis is possible, the gradient of the projected counterfactual line is very sensitive to the jump in coverage

estimates between 2008 and 2009. This is the point between the two rounds of the DHS and is more likely a product of data quality problems in one or both surveys and not a real increase at this point.

There is therefore inevitably some uncertainty around our estimates. A sensitivity analysis is performed to understand how the estimate changes when some key assumptions are varied. Comparison with other reductions in mortality estimates are also made to understand whether the modelled estimates are credible in terms of their level. However, it should be acknowledged that not all our assumptions can be tested, and the point estimates for the cost-effectiveness ratio should therefore be interpreted within this understanding.

The fiscal space analysis is presented in more detail in terms of its methods and assumptions in a separate report (OPM, 2016b). In particular, it lays out assumptions relating to future revenue flows and to the expected costs of the FHCI.