Accuracy of the BiH MICS 2000 data

The relevant research questions are:

• Was the data measured by the BiH MICS 2000 survey accurate?

Validity has no single agreed definition but generally refers to the extent to which a concept, conclusion or measurement is well-founded and corresponds accurately to the real world.a The validity of a measurement tool (i.e. test in education) is considered to be the degree to which the tool measures what it claims to measure.

a

122

The BiH MICS 2000 survey sets out to measure aspects of population health. A prime concern is whether the data measured is accurate or not. The accuracy of the data was estimated by analysing sources of error and estimating the quality of data quality. There are two types of errors in household surveys – sampling and non-sampling.

Sampling errors arise when a representative probability sample is taken from a population. The sample of respondents selected in the BiH MICS 2000 survey is only one of the samples that could have been selected from the same population, using the same design and size. Each of these samples would yield results that differ somewhat from the results of the actual sample selected. Sampling errors are a measure of the variability between all possible samples. The extent of variability is not known exactly, but can be estimated statistically from the survey results for selected indicators. Sampling errors were not measured at the time for the BiH MICS survey,a but have since been measured for purposes of this study. The following were measured (where possible) for each of the selected indicators:b

Value (r): This is the value of the indicator as measured in the survey. The denominator values given for each indicator used in the error measurement is presented in a table.

Standard error (se): Sampling errors are usually measured in terms of standard errors for particular indicators (means, proportions etc). Standard error is the square root of the variance. The Taylor Linearization Methodc was used for the estimation of standard errors.

a _{The guidelines for the MICS 1 and MICS 2 survey series contained no specific recommendations on}

measuring or reporting on sampling errors for the surveys. Guidelines were provided for error estimation in MICS 3 and MICS 4.

b _{Some indicators were composites i.e. made up of the answers to more than one question and so this}

calculation was not possible.

c

The Taylor series (linearization) method is the most commonly used method to estimate the covariance matrix of the regression coefficients for complex survey data. It is the default variance estimation method used by many statistical software packages. It is also the method recommended for MICS survey from series 3 onwards and so was used for consistency.

123 Coefficient of variation (se/r) is the ratio of the standard error to the value of the indicator.

Design effect (deff) is the ratio of the actual variance of an indicator, under the sampling method used in the survey, to the variance calculated under the assumption of simple random sampling. The square root of the design effect (deft) is used to show the efficiency of the sample design. A deft value of 1.0 indicates that the sample design is as efficient as a simple random sample, while a deft value above 1.0 indicates the increase in the standard error due to the use of a more complex sample design.

Confidence limits or intervals(C I) were calculated to show the interval within which the true value for the population can be reasonably assumed to fall. For any given statistics calculated from the survey, the value of that statistics will fall within a range of plus or minus two times the standard error (p + 2.se or p - 2.se) of the statistic in 95.0% of all possible samples of identical size and design.

For the calculation of sampling errors in the BiH MICS 2000 survey, SPSS Version 18 Complex Samples module has been used. Sampling errors were calculated for indicators of primary interest, for the national total, and for urban and rural areas.

Non-sampling errors involve non-observation errors when there is a failure to obtain data from a sampling unit or a variable, or measurement errors that arise when the values for survey variables are collected. Non-observation errors are usually fixed in nature, and lead to considerations about bias in survey estimates. Measurement errors are sometimes fixed, but they may also be variable.

Among non-observation errors, two sources of error are most important: non- coverage and non-response. In probability sampling, there must be a well-defined population of elements, each of which has a non-zero chance of selection. Non- coverage arises when an element in the population actually has no chance of selection; the element has no way to enter into the selected sample. Within the BiH MICS 2000 survey, one element in the population that has no chance of selection is people who are of no-fixed abode, as if present in a household when the survey team enter they will not be listed as household members, nor will their temporary dwellings be included as actual households.

124 Non-response refers to the situation where no data are collected for an element response that has been chosen into the sample. This may occur because a household or person refuses to cooperate at all, or because of a language barrier, a health limitation, or the fact that no one is at home during the survey period. The response rates were calculated in the BiH MICS 2000 survey for a selection of questions and the results presented in tabular form.

Measurement errors arise from more diverse sources, including respondents, interviewers, supervisors and data-processing systems. Respondent measurement errors may occur when a respondent forgets information needed and gives an incorrect response, or distorts information in response to a sensitive question. These respondent errors are likely to constitute a bias, because the respondent consistently forgets, or distorts an answer, in the same way, no matter when he or she is asked a question. These errors can also be variable. Some respondents may forget an answer at one moment, and remember it another.

Potential measurement errors were addressed as follows. To ensure fieldwork quality 10% of households were re-interviewed by field supervisors and UNICEF staff conducted independent field checks on a small number of households. Filled questionnaires were checked by the fieldwork coordinator to ensure completeness of data, and those with missing or absent data returned for checking. Data was doubled entered and inconsistencies corrected. Having two separate entity teams allowed comparison at a number, including in determining outliers in question responses (only one part-question found to be an outlier, probably due to different understanding of the mean in the local language between the two teams). An additional consistency check was made looking for evidence of age distortions in the sample and rounding of anthropometry measurements.