To investigate this, I considered the MKNN approach and rather than placing a unit stake per win classification, I instead staked a measure of the average stake placed by all the tipsters that agreed with this classification. I considered both the mean and median as measures of average, since the mean may be skewed by outliers.
Betting Strategies Match Number Profit Propor tion 2000 4000 6000 8000 0.0 0.2 0.4 0.6 0.8 1.0 Unit Tipster Mean Tipster Median
Figure 5.18: Crowd Sourcing A Stake Strategy
Figure 5.18 shows how following the average stake strategy of the tipsters does yield a better return on investment in this case. Note that the scales of the streams are normalised so that 1.0 corresponds to the maximum profit achieved under that stake strategy.
The flat strategy ROI was 1.44% whilst the median strategy resulted in a ROI of 1.70% and the mean strategy a ROI of 2.16%.
Performing once again the now familiar t-test to see if the mean profit was different from zero was unfortunately inconclusive at the 5% level in both cases. The median approach achieved a mean profit of 1.5654 units with a corresponding 95% CI of [−0.9798,4.1006]
and p = 0.2258. The mean approach achieved a mean profit of 1.6444 units with a
corresponding 95% CI of [−0.6105,3.8993] and p= 0.1526.
5.4
What About The Tipsters?
One final thing to consider is: how did the tipsters fare over the same period of final test data? Figure 5.19 shows the profit streams of the tipsters who achieved a mean profit significantly different from zero using the t-test at the 5% level.
There were only five such tipsters and, surprisingly, they achieved very high return on
investments, with the best being 16.26% by “SecondWalz” which hadp= 0.0274 in the
associated t-test. The worst return on investment obtained was a still respectable 5.15%
5.4. What About The Tipsters? Chapter 5. Real World Investigation
Significant Tipster Profit Streams
Match Number Profit 2000 4000 6000 8000 0 500 1000 1500 2000 2500 jgm1967 uncjrod jlof10 SecondWalz jopo16
Figure 5.19: Significant Tipster Profit Streams
None of the five significant tipsters remained active over the entire test period, which is interesting because there were tipsters who did remain active over the entire period who did not show up as being significant. In hindsight, it would seem obvious to have just copied these significant tipsters and made a fortune, however in reality we do not have the luxury of seeing the bigger picture at earlier time steps. For example, although these five tipsters were significant over the entire test period, over shorter intervals they may not have been significant and in fact other tipsters who were not significant over the entire test period may have shown up as being significant over these shorter periods. This raises another important question. I decided to break up the test data into 10 blocks, akin to 10 fold cross validation, which meant that the classifiers were trained at the beginning of each block and used to classify test examples until the end of that block.
But what if something game-changing happened within the block? Ideally you might
retrain the classifier over much shorter intervals, the most extreme case being retraining after each new data point is encountered. This is a very time consuming process. Most notably, each retraining requires relearning the best hyper-parameters. This is discussed further in Section 6.2.2.
6
Discussion
In this final chapter, we summarise the achievements and findings of the project in Section 6.1 and provide several suggestions for future work in Section 6.2.
6.1
Summary Of Findings
We set out to see if we could spot the wisdom in the crowds and extract reliable infor- mation from it.
By conducting experiments in the simulated world, we found that it was possible to identify tipsters who were behaving non-randomly through the use of hypothesis tests performed using historical data on the strike rates and profit streams of the tipsters. The Naive Bayes and weighted K nearest neighbour classifiers consistently performed the best out of all the classifiers considered in each of the experiments, managing to achieve precision rates in excess of those achieved by the favourite strategy. We found the least successful approaches be the linear and Gaussian support vector machines, sometimes behaving little better than guesswork.
Considering the TennisInsight population, we found that the 17% minority of active tipsters on the site appeared to neither be acting randomly nor simply following the favourite strategy. Obtaining a significant positive return on investment was found to be a much stronger condition than having a significantly non-random strike rate, with only 4% of active TennisInsight tipsters achieving a significant ROI whilst 70% had a significant strike rate.
The best performance obtained by the classifiers on the TennisInsight test data set was that of the modified weighted K nearest neighbour approach, with a precision rate of 88.6% and recall rate of 9.1%. This approach also achieved the highest observed ROI of 2.16% using a crowd sourced stake strategy, in the face of a mean over-round of 2.69%. Unfortunately, this positive return was not found to be statistically significant and so it remains to be seen whether the approach would yield a profitable long-term strategy.