Study E

This study is intended to investigate the extent to which shorter Flexilevel tests may provide comparable measures of an examinee’s proficiency as a CBT test. It should be noted that elements of this work were discussed in Pyper et al. (2015a).

Participants. A group of 18 first year online distance learning Computer Science students took part in a test. The test was summative, and related to their knowledge and understanding of internet technologies and, in particular, ASP.NET.

Methodology. The test was presented to students online via the web application developed as part of this programme of research (please see Appendix B). It was invigilated by a remote live invigilation service to ensure students were not accessing materials that were not permitted during their test.

The remote online invigilation service uses live proctors; the service allows institutions to examine test-takers in their home environment and have them invigilated by means of a live proctor, using screen sharing technologies combined with live audio and video feeds. The service works in conjunction with any preferred assessment application but does not include assessment as part of its functionality.

The service is available 24 hours a day, seven days a week, allowing participants to select a time to take part within a 48-hour window.

On the day of the test, participants were required to log in at the time of their pre-selected slot. An email reminder was sent 24 hours prior to the test. Once participants logged into the remote live invigilation service, they were prompted to download and run the software that would connect their webcam and desktop to a live proctor.

After a connection had been established, the following authentication and environment checks were carried out:

1. An identity check was performed in which participants were asked to present a form of photo identification (e.g. their student ID). The proctor checked that

93 the photo and name on the card matched with the participant who had scheduled the slot and was present onscreen via webcam. Nothing was recorded or captured; it was simply logged as having been checked.

2. Participants had the option of uploading their photo in advance or having their photo taken by the proctor as part of the authentication process. If the photo was uploaded in advance, the proctor compared this to the individual participant onscreen.

3. Following authentication, participants were asked to pan over their work area using their web cam and hold a reflective surface to the camera to ensure there were no unauthorised materials or persons present. If all checks were completed to the satisfaction of the proctor, the participant was invited to log into the assessment application.

Once the test was running, the proctor monitored the participant’s desktop, their environment and also their conduct. None of the proctors reported any unusual examinee behaviour, technical or usability problems with the application.

The test was limited to 40 minutes and contained 40 items overall. The test items were subjected to calibration using data from the performance of an earlier cohort of students.

An initial stage outside of the scope of the study, but relevant to the coverage of topics in the module, was presented first for every student. This stage contained 10 items. Then, as with previous studies, the examinees were randomly assigned to one of two different groups. Half of the participants were assigned to Group 1, and the second half to Group 2. Group 1 was presented with the Flexilevel test followed by the CBT; and Group 2 was presented with the CBT followed by the Flexilevel test. For the students there was no distinction between these two test stages. The Flexilevel and CBT stages consisted of 10 and 20 items respectively.

Examinees also took part in a web scripting coursework as part of their studies that covered the same subject domain as was covered in the online tests introduced here.

Results. Table 4-13 shows the range of scores obtained and the mean score for each stage of the test. It can be seen that the scores for both of the stages varied relatively

94 widely. The responses from three participants were not included in the analysis as they did not complete one or more components of the study.

Component Minimum Maximum Mean Std.

Deviation Flexilevel Test

(out of 10)

5.0 9.5 7.83 1.27

Conventional CBT Test (out of 20)

6.0 19.0 12.46 3.37

Web Scripting coursework (out of 30)

10.0 28.0 23.33 5.70

TABLE 4-13: SUMMARY OF STUDENTS’ PERFORMANCE FOR EACH SECTION OF THE TEST AND ON THE COURSEWORK (N=15)

Figure 4-10 illustrates the correlation between the Flexilevel and conventional CBT test scores.

FIGURE 4-10: A SCATTERPLOT CHART TO SHOW THE SCORES ACHIEVED BY STUDENTS IN EACH OF THE STAGES (N=15)

The Pearson’s product-moment correlation coefficient was calculated for the data presented in Table 4-13. This is reported in Table 4-14 below.

95 Component

Flexilevel Test CBT Test Web Scripting coursework

Flexilevel Test (out of 10) Pearson Correlation * 0.574 0.734 Sig. (2-tailed) p≤0.05 p≤0.01 Conventional CBT Test (out of 20) Pearson Correlation 0.574 * 0.662

Sig. (2-tailed) p≤0.05 p≤0.01 Web Scripting coursework (out of 30) Pearson Correlation 0.734 0.662 *

Sig. (2-tailed) p≤0.01 p≤0.01

TABLE 4-14: PEARSON’S PRODUCT-MOMENT CORRELATION FOR ALL THREE ASSESSMENT CONDITIONS (N=15)

It can be seen from Table 4-14 that the scores achieved by examinees in the Flexilevel test and the conventional CBT test are significantly correlated. Furthermore, the correlations between the Flexilevel test and the Web Scripting coursework scores, and the conventional CBT test and the Web Scripting coursework scores are also statistically significant.

Findings. These results further support the notion that the Flexilevel test can provide comparable results to a standard CBT test whilst only presenting half the items.

Interestingly, in this study the Flexilevel test was a better predictor of examinee performance in the practical component (i.e. the Web Scripting coursework) than was the conventional CBT test.