Research Question 1a: To assess the SMGL program’s overall impact, I first calculated a difference-in-differences estimate for the two groups. The rationale for
selecting this method was that it controls for secular changes that occur over time.
Having a comparison group enabled me to approximate what changes would have
happened in Kalomo absent the SMGL program (ZamCAT only), while also taking into
account changes that may have occurred due to the ZamCAT implementation alone. To
calculate the net effect of the SMGL program, I first subtracted the baseline percentage
Figure 5. Illustration of a Difference-in-differences Estimate
Run Charts
To account for potential bias in the difference-in-difference method, I examined
changes over time, using run charts to examine trends and shifts in rates of FBB and
delivery with a skilled birth attendant by looking at monthly rates over the entire project
period. Run charts are a simple and visual analytic tool that are used widely in quality
improvement work, including most recently in health care settings, as a more detailed
way to examine improvements over time (Perla, Provost, & Murray, 2011). This analysis
allowed for a finer detection of changes that were correlated with the timing and level of
program implementation in Kalomo. The rules for detecting a trend is a minimum of
seven data points continuously increasing or decreasing for more than 20 observation
Research Question 1b:
First, for each of the experimental groups, I constructed both unadjusted and
adjusted odds ratios (OR) for the likelihood of FBB and use of an SBA after the
intervention was implemented, with the reference group of before SMGL. The adjusted
ORs controlled for several socio-demographic covariates: mother’s age, education, parity,
distance to the health facility, children under 5 in the household, and asset quartile.
Next I used multi-level logistic regression modeling to test the net effect of the
SMGL intervention by comparing the SMGL and comparison sites while controlling for
the same set of moderating factors. Multi-level logistic regression could account for the
clustering of individuals by facility and allowed for the simultaneous examination of the
effects of group-level (facility) and individual-level variables on the individual-level
outcomes.
I regressed the primary individual-level outcomes (FBB and use of an SBA)
against a dummy interaction variable that I created by taking the product of time (pre-
SMGL vs. during SMGL) by group (SMGL vs. comparison sites). I adjusted the analysis
for socio-demographic covariates that may have moderated the effect of SMGL on the
outcomes. The model that I used is shown below: Y = β₀ + β₁T + β₂I + β₃TI + yX + e Where:
X = vector of covariates, including household size, maternal age, maternal
education, facility more than two hours away, 4+ ANC visits, and parity β₀ = a constant, and y is a vector representing the impact of covariates
β₁ = the measure of change in the response variable between baseline and final survey
β₂ = the difference in response variable between the intervention and comparison sites
β₃ = the interaction term that measures the impact of the intervention e = error term
As was done is a similar study, I calculated standard errors that were clustered by
facility (the primary sampling unit) to address the assumption of independent and
identically distributed errors within districts (Mohanan et al., 2013) (Azmat et al., 2013).
In order to account for the sampling plan used in ZamCAT, I had to account for both
stratification (urban/rural) and cluster randomization. To do so I used both stratum and
cluster in the logistic model.
I used SAS Version 9.3 for descriptive analysis and model estimation. The “proc
survey logistic” command in SAS uses the Taylor expansion approximation and
incorporates the sample design information, including stratification and clustering,
Propensity Score Matching Analysis
To account for differences in the Kalomo and comparison site populations, I also
matched individuals in the intervention district with a sample of women from the
comparison districts who had similar observable characteristics. To do this, I calculated
the propensity score for each individual based on the estimated probability that this
person might be in the SMGL group (D’Agostino, 1998). I matched women using the
socio-demographic characteristics and other predictors that were both different between
the two intervention groups and strongly associated with the outcome of FBB in the
overall study population. These included: mother’s age, mother’s tribe, mother’s
education, parity, distance to a health facility, HIV status and household asset quartile.
Individuals in the comparison areas without near matches were excluded. With that data
in hand, I created a comparison group of individuals that did not have exposure to SMGL
but shared the same characteristics as the SMGL-exposed group.
To do the matching, I used the “greedy 5->1” algorithm (Parsons, 2001), meaning
that I rounded the propensity score to 5 significant figures, and selected random pairs that
matched exactly. For the remaining unmatched subjects, I rounded the score to 4
significant figures and made exact matches and then I continued the process for 3, 2 and
1 significant figures until all matches were made. Once a match was made it was never
Objective 2: Program Impact on Immediate and Intermediate Outcomes