Approaches to evaluate the impact of the multifaceted intervention

The evaluation of intervention studies in clinical settings in real time is challenging. This is because changes in practice often taken place gradually over time and randomising cohorts of patients to standard practice vs. best practice has major ethical limitations (339). This is why quasi-experimental designs, such as controlled and uncontrolled before-after and

interrupted time series analysis are more suited to interventions that are carried out in clinical settings (Figure 6.4) (339, 340) .

Figure 6.4: Study design approaches to assess the effect of an intervention on the quality of care.

Uncontrolled Before-after approach

Controlled Before-after approach

Interrupted Time series approach

136 The following sections briefly describe the advantages and disadvantages of the various experimental approaches that could be applied to assess the impact of the intervention and provides a justification for the methods chosen.

Uncontrolled before–after approach

In this study design, data is collected in two time periods, before and after the

introduction of an intervention, in order to try and demonstrate a significant change. However, this study design is relatively weak in terms of evaluating quality improvement interventions (339). It has been shown that the uncontrolled before and after approach overestimates the results of interventions when compared to a controlled approach (341). For example, other confounding factors might lead to changes in the measured outcomes which might be not related to the intervention itself. Therefore, it has been suggested that this approach should not be used to evaluate an intervention, and if it is used, the data must be interpreted with caution (339).

Controlled before-after approach

In this approach, a researcher attempts to evaluate the effect of an intervention in the presence of a control group, where the intervention is not applied. The data is collected before and after the introduction of a quality improvement intervention in both study groups (339). The outcome measurement during the post-intervention phase is then evaluated and a

significant difference between the two groups would be assumed to be due to the intervention. Using this approach would overcome the problem of the sudden and secular changes that are found when using the uncontrolled before-after approach (150). However, it would be

sometimes difficult to find a control group during the baseline period that has similar outcome measurement to the intervention group.

137

Interrupted-Time series approach

This approach is designed to evaluate the effect of interventions at multiple equal time periods before and after interventions (342). When collecting data from a single study site to measure the effect of quality improvement interventions, interrupted-time series approach is always preferred over the simple uncontrolled before-after approach (150). A number of advantages might lead to a preference for using this approach over the before-after approach. Firstly, this is an alternative method to overcome the issues from the uncontrolled before-after design, and when it is difficult to find a comparable control group for the controlled before- after approach (339). Secondly, more information would be obtained by using this approach by allowing researchers to evaluate direct and latent effects of the intervention visually and statistically (343). This could not be evaluated by using the before-after approach. However, the impact of other factors that would occur at the same time as the intervention could not be assessed by this approach. Finally, a significant change that might be found when using the uncontrolled before-after approach might not be due to the intervention itself, and it might be due to an already increasing trend during the pre-intervention period (Figure 6.5). For

example, a UK study was conducted to assess the effect of mailing guidelines on the use of radiographic examination to 376 general practitioners. The simple before-after approach indicated that the intervention had a significant impact, and fewer referral requests were observed after the introduction of the intervention. However, the interrupted time-series analysis showed a non-significant difference (344).

138

Figure 6.5: Uncontrolled before-after vs interrupted time series approaches.

Intervention Pre-intervention _{Post-intervention} O utc om es ’ lev el s

139 For a better understanding of the effect of our intervention on prescribing behaviour, two approaches were utilised to analyse the data:

1- Controlled before-after approach: for this approach, the North West Regional Hospital (NWRH) was assigned as a control site and the RHH was assigned the

intervention site. The RHH was selected as the site for intervention for several reasons. Firstly, the RHH has a well-established AMS program which would facilitate applying intervention strategies, whereas at the NWRH there was no AMS program at the time the study started. Secondly, the study to identify potential barriers to guideline

adherence was conducted at the RHH and as discussed earlier, the whole project utilised an intervention designed to overcome the identified barriers. Finally, the RHH was conveniently located in close proximity to the investigators base.

2- Interrupted time-series approach: this approach was selected because two different intervention strategies were applied for two consecutive periods of time at the study site. Furthermore, the interventions were continuously applied during the whole intervention period. Therefore, the interrupted time-series approach would be useful to evaluate the effect of the different strategies over time, which could not be provided by using the previous approach. However, with the simple interrupted time-series analysis design, the main drawback is that other events might occur at the same time as the intervention and produce the same effect as the intervention. Therefore, using a combination of this method and the controlled before-after approach was deemed the most appropriate way to overcome this threat to internal validity.

140