Results and discussion Vapour exposure
5.2 Analysis of the QCM array using Artificial Neural Network (ANN)
A sample of 12 images of two occupied lecture venues is used to test the audience response system. In this 12 image sample, over 250 volunteers presented over 1100 colour poll sheet readings in only two different environments varying in venue slope, lighting, width and
63 height. Many of the participants of the experiment came from a computer science background, and the primary colours used for the colour poll sheets experiment were red, black, green, and blue.
A baseline of actual results was manually counted and recorded to make sure the detected results were true positive results and evaluate the accuracy of the detected results of the program. The evaluation expansion of the detection algorithm adds to the algorithm running time and is a useful system option when testing if the algorithm is performing at an acceptable accuracy level.
A top-down analysis of the results was conducted, scrutinising the total results first and then dissecting the results further to determine the major areas affecting the system results.
Out of a total of 1103 actual polling results, the template algorithm had a detection rate of 62%.
The reason for this was then to filter out the wrong detections using a colour detection algorithm. If the midpoint of a detected rectangle does not fall within the colour ranges stipulated then we can reject this point and only count the detected rectangles that lie within the specified colour ranges. Of this total detection rate, there was only about 61%
True Positive (TP) detected resulting in a total precision of 68% and total F-measure of 0.67 (with alpha at 0.9) and a balance F-Measure of 0.63
Table 5.1: Average Detection results
In the next section of this chapter we take a further look at the error rates according to different variables, mainly:
- Class Environment (CSC and ZOO)
- Poll sheet colour (Red, Green, Blue and Black) - Class Size/Number of participants
Picture Precision Recall F-Measure Balanced F
Average 0,68 0,62 0,67 0,63
64 By inspecting these various independent variables, we can assay which variables will result in better total results and lower error rates. Various adjustments to the algorithm can be introduced to increase the detection rate and reduce the error rates, which will be investigated later in the chapter.
5.2.1 Detection rate analysed on the Environment
Further analysis of the total results uncovers a few details with regards to what makes this detection rate so low. The second thing that comes to mind is a look at the different picture results by the environment. We have only two distinct environments CSC and ZOO.
Table 5.2: Recall results per environment (env.)
The table 5.2 above can show that there is a marked difference between the CSC environment and the ZOO environment signalling that the environment would cause a large enough difference in the recall rate.
The reason the two environments have different results is due to width and elevation of the seating arrangement in environments. The CSC environment is narrower and steeper in comparison to the ZOO environment, this allows the poll sheets to appear bigger and therefore easier to detect mainly because they are relatively close to the camera in comparison to the poll sheets in the ZOO environment. The CSC environment holds fewer individuals in comparison to the ZOO environment. The width of the ZOO environment
Picture Env. Recall
P3 CSC 0,68
P6 CSC 0,85
P7 CSC 0,66
P8 CSC 0,68
P9 CSC 0,70
P11 CSC 0,87
P1 ZOO 0,73
P2 ZOO 0,40
P4 ZOO 0,69
P5 ZOO 0,42
P10 ZOO 0,43
P12 ZOO 0,32
65 and the gentle slope of the seats within this environment makes the colour poll sheets at the back appear smaller in the picture and therefore harder to detect resulting in a lower detection rate in comparison to the CSC environment.
However, the precision and recall rates increase dependent on the environment. For all the CSC environment regardless of the number of people in the classroom, the results are much better than the ZOO environment. The precision rates for all CSC images are above 70% and recall rates above 66%. The ZOO environment precision rates are as low as 32%, and recall rates are as low as 32%.
CSC Environment
Table 5.3: CSC Precision and Recall and F-Measures
Mean and standard deviation on the CSC environment
Table 5.4: CSC Mean and Standard Deviation
ZOO Environment
Table 5.5: ZOO Precision and Recall and F-Measures
Picture Env. Precision Recall F-Measure Balance F
P3 CSC 0,76 0,68 0,75 0,71
P6 CSC 0,74 0,85 0,75 0,80
P7 CSC 0,70 0,66 0,70 0,68
P8 CSC 0,77 0,68 0,76 0,72
P9 CSC 0,83 0,70 0,81 0,76
P11 CSC 0,80 0,87 0,80 0,83
Picture Accuracy Precision Recall F-Measure Balanced F
Total CSC 3,05 4,59 4,43 4,57 4,49
Mean CSC 0,51 0,77 0,74 0,76 0,75
Variance CSC 0,01 0,00 0,01 0,00 0,00
Standard Dev. CSC 0,08 0,04 0,10 0,04 0,06
Picture Env. Precision Recall F-Measure Balance F
P2 ZOO 0,41 0,40 0,41 0,41
P4 ZOO 0,59 0,69 0,60 0,63
P5 ZOO 0,32 0,42 0,33 0,36
P10 ZOO 0,73 0,43 0,68 0,54
P12 ZOO 0,85 0,32 0,73 0,46
66
5.2.2 Detection rate analysed on the Poll sheet colour
The results shown by the colour poll sheet program are tabulated in Table 5.6 below as follows:
Table 5.6: Red Poll Colour Analysis (A Col. Red meaning Actual Colour Red, D Col. Red meaning Detected Colour Red)
As can be seen from the table 5.6 above CSC environment presents a higher detection rate of the two environments. In the row with a detection of 0%, there were no red-poll sheets responses in this poll. The average detection percentage for red colour poll sheets is 64%. The standard deviation associated with the detection of the Red colour is
24.75%.
Picture Env. A Col. Red D Col. Red Detection %
P1 ZOO 71 55 77%
P2 ZOO 10 6 60%
P3 CSC 13 9 69%
P4 ZOO 33 25 76%
P5 ZOO 16 6 38%
P6 CSC 21 20 95%
P7 CSC 7 5 71%
P8 CSC 79 49 62%
P9 CSC 54 37 69%
P10 ZOO 6 2 33%
P11 CSC 0 0 0%
P12 ZOO 59 21 36%
Total 369 235 64%
Picture Env. A Col. - Green D Col. - Green Detection %
P1 ZOO 27 17 63%
P2 ZOO 105 44 42%
P3 CSC 84 44 52%
P4 ZOO 36 25 69%
P5 ZOO 53 15 28%
P6 CSC 19 19 100%
P7 CSC 11 4 36%
P8 CSC 21 9 43%
P9 CSC 19 10 53%
P10 ZOO 4 0 0%
P11 CSC 1 1 100%
P12 ZOO 8 1 13%
Total 388 189 49%
67
Table 5.7: Analysis per Green Poll Colour (A Col.-Green meaning Actual Colour Greed, D Col.-Green meaning Detected Colour Green)
When it comes to the green poll sheet, the average detection percentage is only 49%.
green was used more in the ZOO environment which the colour poll sheet program had lower detection results. The standard deviation for the detection of the Green colour is 29.16%
Table 5.8: Analysis per Black Poll Colour (A Col.-Black meaning Actual Colour Black, D Col.-Black meaning Detected Colour Black)
The average detection rate for the black poll sheet colour was 58%. The standard deviation for the detection of the Black colour is 27.70%
Picture Env. A Col. - Black D Col. - Black Detection %
P1 ZOO 6 2 33%
P2 ZOO 13 3 23%
P3 CSC 17 16 94%
P4 ZOO 15 9 60%
P5 ZOO 13 5 38%
P6 CSC 10 5 50%
P7 CSC 14 9 64%
P8 CSC 4 2 50%
P9 CSC 0 0 0%
P10 ZOO 67 31 46%
P11 CSC 46 38 83%
P12 ZOO 1 0 0%
Total 206 120 58%
Picture Env. A Col. - Blue D Col. - Blue Detection %
P1 ZOO 12 11 92%
P2 ZOO 4 0 0%
P3 CSC 6 2 33%
P4 ZOO 32 21 66%
P5 ZOO 21 19 90%
P6 CSC 26 20 77%
P7 CSC 3 3 100%
P8 CSC 22 17 77%
P9 CSC 1 1 100%
P10 ZOO 0 0 0%
P11 CSC 0 0 0%
P12 ZOO 1 0 0%
Total 128 94 73%
68
Table 5.9: Analysis per Blue Poll Colour (A Col.-Blue meaning Actual Colour Blue, D Col.-Blue meaning Detected Colour Blue)
The average detection rate for the blue poll sheet was 73% which is the highest detection rate of all four colours. From the Table 5.9 above we can see that blue colour has a relatively high detection rate in any environment. The standard deviation for the detection of the blue colour is 36.80%
5.2.3 Detection rate analysed on the Number of participants/Class size
To ascertain whether some participants affect the accuracy attained by the program, we could try and prove whether the error rates increase the more participants are present.
Most of the experiments have about 70 - 120 students in the participation.
The Table 5.10 below shows precision and recall results.
Table 5.10: Precision and Recall per Class
The number of participants does not seem to affect the results drastically, with higher numbers not correlated with the higher results. As the graph in figure 5.1 shows, there is no correlation between the class size and the precision or recall rate of the table arranged according to class size.
Env. # of Participants Precision Recall
ZOO 116 0,68 0,73
ZOO 132 0,41 0,40
CSC 105 0,76 0,68
ZOO 116 0,59 0,69
ZOO 75 0,32 0,42
CSC 75 0,74 0,85
CSC 32 0,70 0,66
CSC 113 0,77 0,68
CSC 69 0,83 0,70
ZOO 77 0,73 0,43
CSC 45 0,80 0,87
ZOO 69 0,85 0,32
69 Figure 5.1: Precision vs. Recall participant graph
According to the chart above in figure 5:1 the blue line shows the actual results sorted in ascending order. However, the detected results do not indicate any pattern similar to that of the real results. Therefore we can conclude the number of participants does not affect results of the program. There is also no correlation between, a higher number of students and greater or lower error rates.
Ideally, a situation in which one detects more results and then filters these to obtain a greater number of actual results leads to a better algorithm.
To answer one of the research questions, Can we use the coloured poll sheets to achieve an accuracy of 85% within the timeframe of 60 seconds?
We can analyse any variables changed to reduce the error rates within the system. Once we have identified the main variables to change and improve on, an enhancement will result in higher accuracy and lower error rates. This improvement will help answer the question as to whether an enhancement can produce as accurate a response system as clickers.
0,00 0,10 0,20 0,30 0,40 0,50 0,60 0,70 0,80 0,90 1,00
36 47 69 76 77 80 106 116 116 121 127 132
Chart Title
Precision Recall
70
5.2.4 Detection analysis on research question
A few samples of the program detections depicted as follows.
Figure 5.2 below depicts a samples size of a medium-sized classroom with not that many people.
Figure 5.2: Example of Small classroom illustration
Figure 5.3 below depicts a samples size of a medium-sized classroom with more participants in the same environment as the smaller classroom.
Figure 5.3: Example of a Medium-sized classroom detection
Figure 5.4 depicts a samples size of a large-sized classroom with more participants in a different environment.
Figure 5.4: Example of an extensive classroom detection
71