Following the methodology presented in Chapter 3, the learning material described in chapter 4 was evaluated. The proposed method of testing was to ask students to run some or all of the Learning Material and to provide their reactions to the style of the learning through the use of a questionnaire. Questionnaire design needs to be undertaken logically and carefully, and has to be addressed in terms of ensuring as far as possible that any results are non-biased and that the results can be shown to be fair and equitable.
An evaluation strategy was therefore developed that comprised a pilot phase, the design of a suitable questionnaire, organisation of a series of practical sessions within schools and colleges and the detailed analysis of the results obtained.
5.2 Evaluation strategy
As explained in Chapter 3 four options were considered in the process of evaluating the experiment: Serial, Parallel, a survey of Teaching Staff, and running an experiment in class. The first three of these options were discounted in terms of requiring too much time or that they had the risk of introducing too many variable factors that could have brought the accuracy of the evaluation process into question. The decision therefore was to evaluate the learning material via experiments carried out in the classroom.
Various options can be used to obtain user feedback. Interviews and observations are two main tools used but a questionnaire has advantages of saving time both in data collection and data processing. A questionnaire should also minimise any difference in approach or bias resulting from the use of any other data collection tools.
!
Cohen et al in their (2007:505) book, Research Methods in Education, suggest that the researcher needs to consider five aspects in their approach to a statistical evaluation:
1 What!scales!of!data!are!there?!
2 Are!the!data!parametric!or!nonJparametric?! 3 Are!descriptive!or!inferential!statistics!required?!
4 Do!dependent!and!independent!variables!need!to!be!identified?! 5 Are!the!relationships!considered!to!be!linear!or!nonJlinear?!
Cohen et al, (2007) explain that interval scales and ratio data generally provide parametric data, and as the participants would be selected by virtue of their current choice of AS, A Level, or BTEC courses, in Computing or ICT it was considered a suitable method to compare the results of the survey as one block of data from which a descriptive analysis of the nature of the participant responses would be valid.
The Likert type scales provide ordinal results that also allow the respondent
flexibility in providing their response(s) through an interval scale. The term ‘interval scale’ as used by Cohen refers to the ‘regular and equal interval between each data point’
(2007:502). Further Cohen et al (2007) cite Cohen and Holliday (1996): that it follows logically that this exact and same interval between data points interval scales are also referred to as equal-interval scales. It was also considered that dependant and independent variables need not be identified and that the relationships of the responses were not
significant at this stage if they showed linear or non-linear relationships within the study. The responses received have no meaning in terms of pure number. Respondents were free to select values from these scales so a purely quantitative tool was used to provide the merest suggestion of qualitative subtlety. Any differences in the marks awarded in response to the questionnaire were at the sole discretion of the participant. Acknowledging that there are risks of respondents tending to ignore the extreme ends of any marking scale; and also that they may chose the safety factor of giving a middle point mark (Cohen et al, 2007:327). The decision was to use a scale of 0 to 9, which provides ten scale divisions and importantly, no middle mark. Evaluation was to be undertaken at schools and colleges who contributed by allowing their students to access the Learning Unit and to freely provide feedback. Care would be taken to introduce the material and the researcher to the students.
Ethical considerations meant that it was essential to emphasise the completely voluntary nature of the evaluation process, and how respondents were able to exercise their right to have their evaluations removed from the study at any time.
The use of numerical answers allows the researcher to gather considerable quantities of quantitative data, but being simply ‘numbers on a sheet’ concerns will remain about how accurate and valid the results are. The alternative face of research is ‘qualitative’ and that requires a different approach to obtaining the responses from participants. The questionnaire was designed around a single sided A4 sheet of paper so participants would not feel it was going to be too onerous a task to complete. The main questions were confirmed in the design phase, three open answer questions were included. Using a measure of both qualitative and quantitative aspects adds the benefit of improving the statistical validity of any research exercise (Cohen et al, 2007) so this was included in the design of the
questionnaire as mentioned previously in Chapter 3 under Mixed Methods. 5.3 Pilot phase
Pilot testing for any data gathering research task is regarded as ‘crucial’ by Cohen et al (2007:341). Cohen sees the testing of any questionnaire as an opportunity not to be missed and lists most areas of the questionnaire as being tested during this process. The questionnaire designed for this experiment was trialled accordingly and at the same time the opportunity to ensure that the material also worked as intended in college and school
computer laboratories whatever IT and network installations were in use.
With one of the LOs complete the opportunity to run a pilot test was arranged to run at Cronton Sixth Form College, a part of Riverside College, Runcorn. The purpose of the pilot test was twofold. Firstly, to assess that the design of the LO and approach adopted to presenting the material was appropriate for the target age group. This was important to establish before significant additional work had been undertaken in authoring the remaining LOs. Secondly, a pilot test allowed the approach to evaluation and questionnaire design to be assessed before full-scale evaluation was to be undertaken. The learning material was loaded onto personal computers in one of Cronton College computer laboratories and
!
fourteen students from a mix of three courses; AS and A Level Computing, and BTEC Level 3, Information and Communications Technology, participated in the pilot test. The learning material was presented to the students with a brief introduction about both the project and the required ethical considerations. After the material was used participants were asked to complete a questionnaire, the design of which was based on Research Methods in Education (Cohen et al, 2007) and followed the information gathering diagram in Fig 5.1.
!
The Introduction notes and an example of the questionnaire as used in the pilot test are included at Appendix B and Appendix C.
Added after pilot test
!
Two abbreviations are used in this diagram: PEL standing for Prior Experience and Learning, and ‘L10’ which indicates that numerical answers would be collected using a Likert scale of 10 intervals (0 – 9).
Following this pilot test the responses were analysed in terms of the rating scale marks and the written responses to the open answer questions. In both the scale ratings and the written comments received the responses were interpreted as being positive and
encouraging which provided positive feedback to continue with the authoring of the remaining LOs and the feedback provided was useful enabling a minor refinement to be added to the questionnaire, which will be described in detail in the next section.
!
5.4 Evaluation design
The questionnaire was the prime means by results obtained from the research would be collected and analysed. The opportunities to ask myriad questions can tempt the unwary into a long questionnaire, but the focus of the research was to discover if the style of the learning material answered the Research Question:
Will learners be more interested, and therefore, enthused to study technology based subjects if the teaching of those subjects is directly related to their daily use of such
technology?
The risk of bias (Cohen et al, 2007), in any direction, must be understood whether designing a questionnaire or in any interaction to obtain feedback for analysis. Even the ideological standpoint of the researcher can inadvertently create bias. Having considered the information needed to establish the relevance of the experiment, risk in this area was
hopefully minimised
The aspect of the learners’ level of prior experience and knowledge relevant to this area of learning was not tested before they used the material developed for this experiment, but learners would be asked to indicate by a self-evaluation of their own level of prior knowledge in the subject area prior to using the Learning Materials.
Using a mixed-method approach to the evaluation was determined in Chapter 3. . The questionnaire was designed to address five key areas, these were:
• Design!and!layout!! • Their!prior!knowledge!!
• Explanations!and!technical!content! • How!a!text!message!storyline!helped,!and!
• If!the!progress!checks!and!the!research!questions!were!helpful.! Using the diagram in Fig 5.1, the question structure was refined using iteration to control the number of and the length of questions while obtaining sufficient and suitable response data. Nine questions were designed and worded so that they followed the same ‘style’. This Likert Scale was used in assessing the responses statistically. Marking on a Likert Scale limits responses to statements or values, so the next design question is how many degrees of variation to allow for the scale. Using values was considered to be more efficient than worded responses in terms of final data collation. Scales may be coarse, using a short scale, often based on five degrees; or a longer and finer scale allows finer differences to be obtained and provide greater variation for evaluation. The middle point mark was avoided by using an even number of degrees on the scale (Cohen et al, 2007:326 et seq). The design was finalised as a ten-point scale from ‘0’ (low) to ‘9’ (high). Respondents were given the following guidance for each of the questions: “Please circle your preference marks using the scale of 0 – 9 where:
0 = did not like / lower / lowest & 9 = best / better / liked very much”
In addition to these scale questions, seeking the benefits of the mixed-methods approach it was decided to include three open text questions. Three open questions were chosen to complement the nine questions asking for numerical answers. The first and second of these questions would ask participants for their ‘most liked’ and ‘least liked’ aspects of the material:
! • “Which!aspect!did!you!like!most!about!the!material!in!the!Learning!Unit! /!Object?”! ! • “Which!aspect!did!you!like!least!about!the!material!in!the!Learning!Unit! /!Object?”! ! And,!a!‘catchJall’!question!leaving!comments!completely!open!to!the!respondent:! • “Please!add!any!other!comments!you!feel!might!be!relevant:”!
These three open questions proved an interesting and valuable resource to assist with the analysis of the data. The version of the questionnaire used for the trial test is shown in Fig 5.2: Questionnaire for Pilot Test.
The final version of the questionnaire changed little; the only addition being one final question asking if the respondent used a mobile phone, in order to establish if the
supposition of these students being regular users of this technology. A copy of the final questionnaire is at Appendix B.
Fig 5.2: Questionnaire for Pilot Test
After completing the six Learning Objects and a minor revision to the questionnaire the introduction sheet was revised to suit ant new locations for the testing sequence in light of experiences obtained during the pilot phase.
!
Students at two establishments in the North Cheshire area evaluated the material. These Colleges both provide courses relevant to the material, and the Colleges were very happy to assist.
5.5 School and college sessions
Two colleges were chosen within which to carry out the evaluation of the completed learning material. These were Cronton Sixth Form College at Runcorn and Priestley College in Warrington.
5.5.1 Cronton Sixth Form College
Cronton College is part of Riverside College, Runcorn, England. Riverside College is a further education college that was established from the merger of Halton College and Widnes & Runcorn Sixth Form College in August 2006. Riverside College now has two campuses in Widnes and one in Runcorn (Cronton College). The college provides a range of courses including GCSEs, A Levels, BTECs, Apprenticeships and Access courses. This College participated in the pilot test described earlier when 14 of their students tested the material and after running the materials the students tested the questionnaire. They remained willing to participate in the full evaluation. A total of 44 of their students took part in a full evaluation held in March 2013, and staff and students at the college were most welcoming throughout the evaluation.
Ofsted inspected the College in 2010 and awarded them a grade 2 marking. Since then the college has many new staff and new leadership. The Ofsted report includes some details of the demographics relevant to the surrounding catchment area:
The$college$recruits$a$high$proportion$of$learners$from$areas$of$social$and$ economic$deprivation$and$many$learners$aged$16$to$18$arrive$with$relatively$low$ GCSE$grades.$$Two$schools$in$the$local$area$have$sixth$forms.$$The$percentage$of$ learners$from$minority$ethnic$groups$is$around$4%,$which$is$about$twice$the$ proportion$resident$in$the$borough.$$Unemployment$rates$are$similar$to$national$ averages$but$more$people$have$been$unemployed$in$the$long$term$and$many$have$ poor$health.$(Ofsted:$2010)$ !
5.5.2 Priestley College, Warrington –
Priestley Sixth Form and Community College is a sixth form college located on Loushers Lane, Warrington, England. It also offers adult courses, professional training on another site, and is an associate college of the University of Salford. A total of 83 students from the College participated in the evaluation. Ofsted last inspected Priestley College in 2010 awarding a grade 1 marking. The College is delightful to visit and the Ofsted report mentions the local demographics as:
…$The$vast$majority$of$learners$were$white$British,$with$about$5%$of$minority$ ethnic$heritage.$$The$majority$of$learners$aged$16A18$study$level$3$courses,$with$a$ small$proportion$studying$at$level$2,$and$a$few$at$level$1.$$The$college$offers$ courses$in$most$curriculum$areas,$including$health$and$social$care,$science,$ mathematics,$information$and$communication$technology,$humanities,$social$ sciences,$arts,$languages,$sports$and$business$(Ofsted:$2007).$
Testing at the two establishments provided a total of 127 evaluations in addition to the 14 responses conducted during the pilot run. These participation details are summarised in Table 5-1: Colleges participating in the research
The researcher attended Cronton Sixth Form College where LO material was loaded onto personal computers in the computer labs, and the sessions were introduced personally. At Priestley College the learning material was loaded onto their Moodle powered Virtual Learning Environment (VLE). The researcher did not attend the evaluation sessions at Priestley College; these sessions were introduced by their staff. This helped minimise inadvertent bias from the researcher
Making the Learning Material available over the college’s VLE allowed students at Priestley College a more flexible schedule in that they were able to access the materials within their own time or during periods apart from their timetable. All responses were paper-based, with students completing a questionnaire and returning them at the end of their interaction with the material.
!
Participating College or School
Course Date Numbers in
study: Pilot Test, Cronton, July 2012 A AS &
BTEC
2/7/2012 14
Cronton Sixth Form College (part
of Riverside College, Runcorn); A AS & BTEC 26/3/2013 44 Priestley College, Warrington; A AS &
BTEC 12 – 15/3/2013 83 Totals - Pilot - Main - Total 14 127 144
Table 5-1: Colleges participating in the research
Responses were collated by the researcher and safely filed in accordance with the policies of the University of Salford.
5.6 Results
Using a mixed method approach as explained previously allows the responses to be considered and compared from both aspects of qualitative and quantitative data. This process has been shown to contribute a level of validation and reliability to the evaluation process (Cohen et al, 2007). Firstly, the quantitative test data was reviewed in terms of how the respondents rated their experience of using the Learning Objects. Secondly, the
qualitative data from the questionnaire will be used, as indicated in Chapter 3, to confirm the validity and reliability of the quantitative responses; and this may also elicit nuances of interest captured by the three open questions included in the questionnaire.
The first stage in analysing the data from the questionnaire responses was to plot the answers to all nine quantitative questions onto a single graph as shown in Fig 5.3: Composite Graph of response volumes. The trend curves resulting from this indicate that the various cohorts of students show little variation and so it was decided to interpret the results ‘en bloc’.
Fig 5.3: Composite Graph of response volumes
The response to each of these nine questions will now be evaluated in detail. 5.6.1 Question 1 – ‘Design and screen layout’
The first question was a general question asking about the design and layout of the screen. It asked respondents to score the "colour choice for background and/or text". The results are presented in Fig 5.4: Question 1 Responses
! 0! 5! 10! 15! 20! 25! 30! 35! 40! 45! 50! 1! 2! 3! 4! 5! 6! 7! 8! 9! R esp on se s' Question'Number'
Composite'Graph'all'questions'and'responses'
Series 1! Series 2! Series 3! Series 4! Series 5! Series 6! Series 7! Series 8! Series 9!!
Fig 5.4: Question 1 Responses
The majority of responses were in the third quartile, and more were above this sector than were below it.
5.6.2 Question 2 – ‘Prior Knowledge’
The second question sought to determine the prior knowledge of the participants. It asked respondents to state: “What level of knowledge in this area would you say you had prior to using this material?”
Fig 5.5: Responses to Question 2
The responses to this question are presented in Fig 5.5 which shows that there are many more responses above the centre line with peaks for values returned of: ‘5’ and ‘7’. So the subject matter may not have been overly challenging.
5.6.3 Question 3 – ‘easy to read or follow’
The next set of three questions were concerned with the quality of the explanation of the technical content within the learning material. Question 3 asked whether the material was "easy to read or follow?" The results from the respondents are presented in Fig 5.6: Question 3 responses.