Samples

Samples were collected from a control scheme evaluation in northern Xinjiang province in the People’s Republic of China, as described in the previous chapter and in recent reports (van Kesteren et al., 2015). Both necropsy (n=38) and field samples

79

(n=125) were used in the current analysis. One ‘field panel’ sample, from village ‘N’, was removed from further study due to failure of replication.

3.2.2 Cut-off determination

The necropsy panel was used to estimate cut-off values for positivity using the three broad methods described earlier. All cut-offs were interpreted as being the threshold for positivity – meaning that any samples with the same OD as the cut-off were interpreted as being positive. Firstly, the standard protocol based on selecting a point three standard deviations above the mean of a selection of validation samples from a nonendemic area was used (‘predetermined Gaussian’ method). This same approach was then repeated using samples collected from the field in Xinjiang, from dogs which were negative on necropsy (‘Gaussian 1’ method). As the distribution of the OD values of these samples did not conform to the Gaussian distribution expected for this approach, outliers (defined as those points which were greater than 1.5 times the interquartile range above or below the upper or lower quartile, respectively) were identified using the ‘boxplot’ command in R, and were removed, before repeating the process (‘Gaussian 2’ method). Secondly, ROC curve analysis was used in order to select the cut-off point which maximises both the sensitivity (𝑆𝑒) and specificity (𝑆𝑝) of the test simultaneously, using the R package ‘ROCR’ (Sing et al., 2005) (although the Youden index was not calculated here, this method would be expected to give the same cut-off value).

Finally, a Gaussian finite mixture model was created using the R package ‘mixtools’ (Benaglia et al., 2009). Selection of the appropriate number of components was achieved using an iterative bootstrap analysis of the effect of adding an extra class to the model (described above), using a p-value of 0.05 or less to suggest an improved fit. Three methods were then used to allocate samples to a ‘positive’ or ‘negative’ status. The first of these was based on characterisation of the ‘negative’ component, followed by selection of a cut-off point three standard deviations above the mean of this distribution in a similar manner to the Gaussian distribution method described earlier (‘MM1’ method). In the case of a solution including more than two components,

80

different cut-offs were estimated according to the distribution of the data. The second mixture model method was based on allocation of individual samples to either a ‘negative’ or ‘positive’ group according to the modal posterior probability of component membership (‘MM2’ method). The final mixture model approach (‘MM3’) was based upon a ROC curve analysis approach to the mixture model output. ‘Positive’ and ‘negative’ components were identified (in the case of models with more than two components, intermediate groups were ignored), and the posterior probabilities of membership in each of these components for each individual sample were estimated. Samples were then ordered according to OD, and a range of different OD cut-off points were applied to the data. For each cut-off point, the sum of all the probabilities of ‘negative’ component membership for samples classified as negative, and the sum of all the probabilities of ‘positive’ component membership for samples classified as positive were estimated, and these were expressed as a proportion of the total sum of all probabilities within the group in question (in order to ensure equal weighting of negative and positive groups in unbalanced studies). These estimates were then summed, and the cut-off which maximised this total was selected.

In order to assess the potential use of mixture models in the absence of a gold standard test, Gaussian mixture models were also created using the field data, and cut-off points were estimated as described for the MM1 and MM3 methods (allocation according to modal probability was not performed as it does not result in a cut-off, and so could not be validated). The 𝑆𝑒, 𝑆𝑝, and overall accuracy (proportion of samples correctly classified) of each method was estimated using the necropsy panel results. Finally, the effect of a selection of these different methods on the coproantigen prevalence estimate from the field data (stratified by village, due to the stratified sampling approach used) was also estimated. The Rogan-Gladen approach (Rogan and Gladen, 1978) described above was used to give a point estimate of the true prevalence of infection, and exact Blaker confidence intervals (accounting for test sensitivity and specificity) were calculated (Blaker, 2000; Reiczigel et al., 2010), using the ‘epi.prev’ command in the ‘epiR’ package for R (Stevenson et al., 2013).

81

3.3 Results

The ‘Gaussian 1’ method, based on a panel of faeces from 43 dogs from nonendemic areas, estimated a cut-off OD of 0.065. A total of 38 faecal samples were collected by necropsy in Xinjiang, of which 16 (42%) were found to contain Echinococcus spp. The number of Echinococcus worms present amongst infected dogs ranged from 2 to over 10,000, with a median of 100, as shown in Table 3.1. A total of 22 dogs were negative on necropsy, and were used in the ‘Gaussian 2’ approach to give a cut-off OD of 0.331. After the removal of three high OD outliers, the ‘Gaussian 3’ approach gave a cut-off of 0.180. ROC curve analysis including all 38 samples suggested that a cut-off of 0.117 maximised the overall accuracy of test classification.

Application of mixture models to the necropsy data identified two components, as detailed in Table 3.2. Based upon the distribution of the ‘negative’ component, a cut-off of 0.149 was estimated with the ‘MM1’ approach. When the adjusted ROC curve approach was applied to the posterior probabilities of sample component membership (the ‘MM3’ approach), the optimal cut-off was found to be 0.117.

Table 3.1. Distribution of worm burdens and coproantigen ELISA OD values amongst

the 16 Echinococcus spp positive dogs identified by necropsy.

OD Number of Echinococcus worms Number of Taenia spp

0.117 2 1 0.088 3 7 0.155 10 1 0.176 20 2 0.252 50 2 0.171 50 1 0.087 50 Not recorded 0.461 100 0 0.240 100 0 0.373 100 7 0.396 300 4 0.462 500 5 0.571 >5,000 6 0.793 >10,000 0 0.680 >10,000 6 0.665 >10,000 2

82

Table 3.2 Properties of components identified by the mixture models. Components have been ordered according to their means (smallest to largest).

Dataset Mixture Model component Proportion Mean OD in component OD standard deviation Necropsy ‘negative’ 0.45 0.075 0.025 ‘positive’ 0.55 0.335 0.198 Living dogs ‘negative’ 0.81 0.091 0.041 ‘positive’ 0.19 0.339 0.132 ‘negative’ 0.22 0.044 0.012 ‘intermediate’ 0.59 0.108 0.034 ‘high’ 0.19 0.336 0.132

When a mixture model was applied to the field data (collected from living dogs), the optimal number of components was found to be three, with an ‘intermediate’ component between the negative and positive ones identified in the necropsy data (see Table 3.2). As this was unexpected, a mixture model was first created with only two components, as had been used for the necropsy data (‘MM1a’), which gave a cut-off of 0.215. As the status of dogs in the intermediate component of the three component model was not clear, cut-off methods were applied including it as both a negative (‘MM1b’) and as a positive (‘MM1c’) group. These gave cut-off points of 0.079 and 0.209, respectively. Finally, the adjusted ROC curve approach (‘MM3’) applied to these models, which gave an optimal cut-off of 0.200 for the two component model and 0.180 for the three component model.

Figures 3.1 and 3.2 show the overall distribution of OD values for all samples (necropsy dogs and field dogs), both in the form of a histogram and a kernel density plot (created using the ‘density’ command in R). Cut-off estimates are overlaid (some of the less reliable and more problematic cut-offs have been excluded, for ease of interpretation).

83

Table 3.3 shows the estimated sensitivities and specificities of the different classification methods. The average of the sensitivity and specificity is also presented (rather than the overall classification accuracy of the test), in order to ensure that negative and positive samples have equal weighting. Finally, Table 3.4 and figure 3.3 show the effect of some of the different cut-offs on the point estimates of the coproantigen prevalence and the Rogan-Gladen ‘true’ prevalence (and exact 95% confidence interval) for the six villages visited. The ‘Gaussian 1’ and three-component mixture model approaches were not evaluated here, due to suspected limitations in their applicability. The ‘predetermined Gaussian’ approach was primarily included for comparison purposes rather than due to its suspected validity, since this is the current approach used for estimation of a cut-off point.

84

Figure 3.1. Distribution of OD values for all necropsied dogs. The top graph shows

results for those with no Echinococcus spp. adult worms on intestinal inspection, and

85

Figure 3.2. Distribution of OD values for all necropsied dogs (top) and all live dogs sampled in the field (bottom).

86

Table 3.3. Test characteristics using different methods of classification (sensitivity, specificity and overall accuracy estimated from necropsy data). The ‘predetermined Gaussian’ method is a Gaussian method using nonendemic controls; and the ‘Gaussian 1’ and ‘Gaussian 2’ methods use necropsy data (with the latter excluding outliers). The ‘MM1’ approach estimates a Gaussian cut-off from the ‘negative’ component of a mixture; the ‘MM2’ approach allocates individuals to mixture model components according to modal probability; and the ‘MM3’ approach uses a modified ROC curve-type approach to identify a cut-off which maximises the probability of membership in the ‘negative’ and ‘positive’ mixture model components.

Method Cut-off Sensitivity _{(n=16 positives)} Specificity _{(n=22 negatives)} Accuracy

Predetermined Gaussian

(3sd above mean of nonendemic panel tested separately) 0.065 16/16 = 100% 8/22 = 36% 68%

Estim a ted fr om ne cr os py pane l Gaussian 1

(3sd above mean of necropsy negative panel from field) 0.331 8/16 = 50% 22/22 = 100% 75%

Gaussian 2

(as for Gaussian 1 but ‘outliers’ removed) 0.180 11/16 = 69% 19/22 = 86% 78%

ROC curve

(maximising Se + Sp; based on necropsy data from field) 0.117 15/16 = 94% 17/22 = 77% 86%

Necropsy MM1

(3sd above mean of ‘negative’ component) 0.149 14/16 = 88% 17/22 = 77% 82%

Necropsy MM2

(component allocation according to modal posterior probability) N/A 14/16 = 88% 17/22 = 77% 82%

Necropsy MM3

(ROC curve evaluation of posterior probabilities) 0.117 15/16 = 94% 17/22 = 77% 86%

Estim a ted fr om livin g dog s

Field data MM1a

(two component model as for MM1) 0.215 11/16 = 69% 19/22 = 86% 78%

Field data MM1b

(three component model;

as for MM1 with intermediate group classified as negative)

0.079 16/16 = 100% 11/22 = 50% 75% Field data MM1c

(three component model;

as for MM1 with intermediate group classified as positive)

0.209 11/16 = 69% 19/22 = 86% 78% Field data MM3a

(ROC curve evaluation of two component model) 0.200 11/16 = 69% 18/22 = 82% 75%

Field data MM3b

87

Table 3.4. Effect of different cut-offs on point coproantigen prevalence (upper percentages) and estimated true prevalence (according to the Rogan-Gladen method) and exact 95% confidence intervals (using the Blaker method), whilst accounting for test sensitivity and specificity (using the Reiczel method) (lower percentages) for six villages in Xinjiang.

Method used (cut-off) Village A B C N Q T Predetermined Gaussian (0.065) 13/19 = 68% 13% (0 – 60%) 18/20 = 90% 73% (13 – 95%) 13/21 = 62% 0% (0 – 46%) 12/20 = 60% 0% (0 – 43%) 19/26 = 73% 26% (0 – 66%) 11/19 = 58% 0% (0 – 39%) Gaussian 2 (0.180) 1/19 = 5% 0% (0 – 21%) 8/20 = 40% 48% (13 – 90%) 2/21 = 10% 0% (0 – 30%) 1/20 = 5% 0% (0 – 19%) 5/26 = 19% 10% (0 – 44%) 2/19 = 11% 0% (0 – 33%) ROC /Necropsy MM3 (0.117) 6/19 = 32% 12% (0 – 46%) 14/20 = 70% 67% (35 – 89%) 8/21 = 38% 22% (0 – 52%) 3/19 = 15% 0% (0 – 20%) 12/26 = 46% 33% (8 – 61%) 8/19 = 42% 27% (0 – 60%) Necropsy MM1 (0.149) 3/19 = 16% 0% (0 – 25%) 10/20 = 50% 42% (10 – 74%) 3/21 = 14% 0% (0 – 19%) 1/20 = 5% 0% (0 – 2%) 6/26 = 23% 1% (0 – 30%) 5/19 = 26% 6% (0 – 42%) Field data MM 1a (0.215) 3/19 = 16% 4% (0 – 46%) 10/20 = 50% 66% (28 – 100%) 3/21 = 14% 1% (0 – 39%) 1/20 = 5% 0% (0 – 19%) 6/26 = 23% 17% (0 – 52%) 5/19 = 26% 23% (0 – 66%)

Field data MM3a (0.200) 2/19 = 11% 0% (0 – 26%) 8/20 = 63% 43% (5 – 89%) 2/21 = 10% 0% (0 – 24%) 1/20 = 5% 0% (0 – 11%) 5/26 = 19% 2% (0 – 39%) 2/19 = 11% 0% (0 – 26%)

88

Figure 3.3. Unadjusted (top) and adjusted (bottom) estimates of the coproantigen prevalence for the different villages, using different cut-offs. Bars indicate 95% confidence intervals.

89

3.4 Discussion

The current paper details an attempt to improve the method of classification of canine faecal samples when using an Echinococcus coproantigen ELISA test. As well as reviewing two general approaches in common use currently, a number of novel methods based on mixture modelling are described. It is hoped that these methodologies will offer some prospect for improvements in canine echinococcosis surveillance, by both assisting in the selection of an appropriate approach to diagnostic test interpretation, and by illuminating some of the limitations associated with dichotomous interpretation of any data on a continuous scale, such as ELISA OD data.

3.4.1 Gaussian approaches

Based on the results shown here, the ‘predetermined Gaussian’ approach to cut-off determination, based upon selection of a cut-off three standard deviation above the mean OD of samples taken from a nonendemic area, does not perform well as a diagnostic test. The estimated cut-off using this strategy was low, and therefore the test specificity was also low. Although the sensitivity was 100% in this particular case, this strategy does not implicitly incorporate positive samples in its calculation, and therefore this could be an incidental finding. The low specificity resulted from the nonendemic negative panel generally having lower OD values than negative samples from the necropsied dogs. This may have resulted from variations in ELISA conditions (as the nonendemic samples were not tested at the same time as the others presented here, as per the usual protocol for the predetermined Gaussian approach), or may suggest that the variation and mean of these samples was lower than that observed in the field data (see Table 3.5). It is possible that dogs from nonendemic communities differ in various ways from those in endemic communities, and as such may not present an optimal panel for selection of a cut-off point to apply to field data. One possible way in which these dogs may differ is in terms of concurrent worm burdens – in particular, other cestodes such as Taenia spp. Although preliminary analysis of the necropsy data used here gave no evidence of an association between Taenia spp

90

burdens and OD values (following adjustment for the effect of Echinococcus spp burden – data not shown), previous studies have suggested that this may exist at a low level (Deplazes et al., 1992; Allan et al., 1992; Allan and Craig, 2006). The presence of this cross-reactivity could possibly result in greater variation in OD values amongst Echinococcus negative animals from endemic communities compared to those from nonendemic communities. One other possibility is that of misclassification of the necropsy panel (meaning that necropsy negative samples may not have been true negatives) – which, if true, would be expected to improve the estimated specificity of the predetermined Gaussian method. This issue is dealt with below.

The ‘Gaussian 1’ approach suffered the opposite problem to the predetermined Gaussian method – giving a high cut-off, and therefore a low sensitivity. The cause of this was the presence of a number of high-OD outliers amongst the otherwise relatively Gaussian-distributed negative samples from the necropsy panel, which increased both the standard deviation and estimated mean of these OD values. As the presence of these outliers violated the basic assumption of the Gaussian approach, the three most extreme values were removed (although a further two remained outside the expected Gaussian distribution). This demonstrates the dangers of not visually inspecting the distribution of data before applying a technique such as this. The reason for the outliers is unclear, but as they are rarely seen when using a panel of negative dogs from a nonendemic area (author’s personal observation), they may represent dogs with low burdens which have been overlooked during necropsy. Alternatively, they may be taken from dogs which have been recently dosed with a cestocidal drug: as seen with other cestodes (Deplazes et al., 1990; Allan et al., 1990), it has been found that it can take 2-4 days for E. granulosus coproantigens to disappear following treatment (Jenkins et al., 2000). This demonstrates a potential limitation with the use of field data as a negative control panel (see Table 3.5). Another possibility is that the distribution of negative samples truly does not follow a Gaussian distribution, as has been suggested from a study of coproantigen ELISA OD densities for fox faeces in France (where samples were diagnosed using the ‘gold standard’ of necropsy and sedimentation and counting technique (WHO/OIE, 2001d; Eckert,

91

2003)) (Raoul et al., 2001). If this was the case, any approach based upon the Gaussian method (as well as the Gaussian mixture model) will be flawed.

As well as the specific issues described above, one clear limitation of any Gaussian cut-off method is that the distribution of ‘positive’ OD values is not accounted for at all. As such, selection of a cut-off aims to maximise the test specificity without any consideration of the impact this has on the sensitivity. An ideal coproantigen ELISA test would have complete separation between negative and positive OD values, and so this would not be an issue (indeed, this is often seen when evaluating the test using nonendemic negative samples and high burden or spiked positive samples). However, this is not likely to be the case in the field situation, in the presence of negative dogs with high concurrent worm burdens (and therefore possible cross reactions, as described above), and positive dogs with low worm burdens (as described below) or prepatent infections.

3.4.2 ROC curves

Traditional ROC curve analysis of the necropsy data appeared to give the best results of all of the methods assessed here, with a very high sensitivity and a good specificity (although definitive conclusions are difficult to make based on such a small sample size). ROC curve analysis is also the only method detailed here which specifically allows the determination of a cut-off point according to the requirements of the test (Greiner et al., 2000), and which does not make any assumptions about the frequency distribution of OD values amongst negative or positive samples. Although the current approach has aimed to maximise both the sensitivity and specificity, in some cases (such as monitoring for introduced infection in a nonendemic community, where any possible positive dogs need to be identified quickly), it may be of greater use to select a cut-off giving a maximal sensitivity, even if this results in a reduced specificity. Additionally, as the sensitivity and specificity are explicitly estimated as part of the ROC curve estimation approach, a greater appreciation may be gained of the limitations in test interpretation.

92

Despite these positive aspects, it should be noted that the cut-off determined in the nonparametric ROC curve analysis used here can only take the value of one of the OD values in the panel investigated – which can lead to some loss of accuracy in the overlapping area between negative samples and positive samples (which is the area of interest), especially when sample sizes are relatively small (as was the case in the current study). For example, both ROC curve-based approaches towards classification of necropsy data identified the cut-off for positivity as 0.117 (to 3 significant figures), which relates to one particular sample (which had an OD of 0.11685). Therefore, all values with an OD of 0.11685 or more were classified as positive. The sample with the next highest OD value to this one had an OD of 0.11585 – meaning that it could equally be stated that the cut-off for negativity was 0.116 (i.e. all samples with an OD of 0.11585 or less were classified as negative). Whilst both of these methods give the same result when applied to the necropsy data here, the use of these slightly different cut-off interpretations to field data could give different results if intermediate OD values between 0.116 and 0.117 were present. An alternative to this would be to use parametric ROC curves, which assume that both negative and positive samples follow Gaussian distributions. This approach may not be appropriate with the relatively small sample sizes in this case, and may be problematic if the distribution of OD values amongst infected individuals (see below) or uninfected individuals (see above) did not follow a Gaussian distribution. Another approach is ‘two-graph ROC’ (TG-ROC) analysis, which can allow the estimation of an ‘intermediate range’ (Greiner et al., 1995).