Adequacy of the project’s methodology: how should an innovative disease management algorithm be developed and evaluated?

Constructing an evidence-based electronic algorithm

The initial construction of IMCI was based on a WHO-based large technical review (Gove, 1997). Several formal technical reviews of IMCI components have been carried out since (World Health Organization, 2014) and another major review process is currently ongoing (Costello AM and Dalglish SL on behalf of the Strategic Review Study Team, 2016). Such formal guideline processes have the advantages of a large capacity to review evidence comprehensively, and to reach formal consensus between different stakeholders. The goal of the development process of e-POCT was different in that we aimed at generating new evidence through the construction of an innovative tool, rather than developing a fully evidence and global consensus based disease management algorithm. An additional risk of formal guideline construction processes is that they result in the perception of an evidence base when there is in fact none. An example is the very cumbersome WHO dehydration scale, which was has been recommended by IMCI since its inception based expert consensus alone. We interviewed several pediatric providers in Tanzania on the clinical symptoms and signs they employ to identify children with dehydration. Almost all admitted

that they found the WHO scale to be of little practical value, though they thought it was ‘evidence-based’; as it is recommended by the WHO.

Given our different aims in developing e-POCT from more classical guidelines development processes, we performed an ‘individual’ structured literature review that allowed to quickly identify key areas for evidence-based improvement or innovation. Overall, we found very little evidence for the majority of disease management components reviewed. On the other hand, such components are routinely employed in pediatric clinical care and we were able to draw on this informal experience from clinicians. For example, little evidence from the

literature could be retrieved on which cutoffs to use for CRP and PCT at peripheral health care level. However, both CRP and PCT have been evaluated and used in pediatric practice for the past 30 years. Ultimately, we relied on this experience to choose cutoffs for our disease management tool. Such an informal consultation process, in contrast to a formal consensus process, also helped to identify quickly research areas at need of additional evidence. The disadvantage of such an informal approach is obviously its subjectivity. Though we attempted construct e-POCT with as much objectivity as possible, and to carefully document our rationale, large areas remain that are based largely on our own experience. Again, we chose this process to allow the generation of new hypothesis and innovation. But for future implementation, the evidence retrieved from this project should be integrated into formal, coordinated guideline review activities (see Chapter 10.6 below). The electronic format implicates specifics in terms of logical flow and content of an algorithm (Rambaud-Althaus, 2015). Algorithms increase the consistency of assessments, as

clinicians have to follow recommendations step-by-step. On the other hand, given that there is less room for the clinician’s interpretation, the logical flow of the algorithm has to be coherent and not delay patient management. A paper algorithm designed for clinicians cannot be simply handed to a software specialist for programming. We therefore developed a detailed paper reference manual to limit the introduction of errors through the software- programming step. Close collaboration of a clinician and software programmer is required for the development of electronic algorithms. e-POCT included background calculations and the integration of data which would have not been feasible in a ‘manual’ consultation. At the same time we paid attention to keep the user in and output screens simple. It is important to include such software features into electronic algorithms as simply translating paper

algorithms into electronic versions will result in little advantage for the end-user. Before using the electronic algorithm version in the clinical trial, we performed careful internal validation and evaluated the algorithm with supervision from medical doctors among 100 pilot patients. In general, there is little regulatory guidance on how electronic algorithms (or electronic decision support systems in general) should be developed and validated (Karnik,

2014). Until such regulatory guidance becomes available, the future creation of an ‘auto- regulatory’ guide by different groups involved in algorithm development and validation may increase patient safety, and the acceptability for scale-up.

Algorithm evaluation

The innovative character of e-POCT called for a safety evaluation through a controlled study design. We did not perform prospective validation studies since our goal was not to measure diagnostic accuracy, but to evaluate our tool against patient outcome. We chose an

individual randomization design to avoid clinician and facility-level cluster effects. This was indeed a source of type I error in the ALMANACH design where intervention clinicians received training on the new tool but no training activities were done in the control arm. In addition, clinicians in the intervention arm may have been more motivated by the fact that they were testing a new electronic tool. Finally, the number of health facilities required for a clustered design would have overloaded the safety monitoring capacities of this study. Given that electronic format guided the clinician through the consultation process and allowed exact monitoring of adherence, our concern for a type II error through contamination of practice was low. However, to be on the safe side, clinicians used one algorithm for two weeks at a time to avoid confusion.

Figure 29 Two study clinicians treating patients at Magomeni health center during the trial.

We chose ALMANACH, instead of IMCI, as a control group since several improvements had already been made to ALMANACH, such as the introduction of urine dipstick testing. On the other hand ALMANACH is likely less powerful for the detection of severe malnutrition than the new version of IMCI (Chapter 9.2), which may have introduced a type I bias. The PEDIATRICK project had already demonstrated a reduction in antibiotic prescription and improved clinical outcome of ALMANACH compared to routine care in Tanzania (Shao et al., 2015). A comparison of e-POCT with routine, IMCI-based care would have hence neither been appropriate nor ethical. We chose a non-inferiority design since we expected benefits

in terms of the disease consultation process and antibiotic prescriptions, but not necessarily in terms of clinical outcome, when compared to ALMANACH. We constructed the primary outcome, clinical failure by day 7 (Table 10) based on previous pneumonia management trials (Hazir et al., 2011; Fox et al., 2013) and the experience from the PEDIATRICK project. The validity of some of the criteria used for non-severe disease classifications should be re- evaluated for future studies (see Chapter 7.5) and a global consensus on outcome

definitions for future studies would be highly desirable. We chose to evaluate e-POCT in its integrity, rather than performing studies to evaluate its individual components. This was out of the recognition that children often present with multiple, overlapping symptoms. This requires an integrated disease management approach, which is a cornerstone of IMCI. A resulting drawback is that the study was not sufficiently powered to draw strong conclusions on the utility of certain tools or subgroups, such as Hb screening, or patients with FWS, or severe disease. This was also because of its design as a safety, rather than an

effectiveness trial. Given its innovative character we decided to evaluate e-POCT in a relatively healthy population (high vaccine coverage and low HIV rates) and in an area that with enough infrastructures to allow effective adverse events monitoring with our available resources. However, this limits the generalizability of the findings. Future evaluations (or at least component evaluations) should be carried out in other settings. However, tools to mitigate high-volume antibiotic overuse are urgently needed. The need to strengthen the evidence in certain sub-parts of the e-POCT algorithm should not delay implementation efforts with careful monitoring and iterative algorithm improvements.

On the spectrum of efficacy to effectiveness trial, this study was certainly more an efficacy trial than an effectiveness trial. Only limited conclusions on the benefit of e-POCT in routine care settings can be drawn. This should be the subject of future research efforts. For the e- POCT components for which safety was not the primary concern (oximeter, Hb, severe malnutrition diagnosis), an effectiveness study would have been more adequate. However, given that novel biomarker testing strategies were integrated, an efficacy study was required as the first step. In general, before routine introduction of future, novel disease management tools, outcome-based clinical trials should be carried out to assess new disease

management tools before routine introduction. Interestingly, the IMCI case management algorithm itself has actually never been assessed in clinical trials. The initial evaluation studies that were carried to measure an effect on childhood mortality directly assessed the IMCI strategy as a whole, including packages for health system strengthening and drug supply (Gera et al., 2016).