Methods adopted for this research

As stated previously, iterative cycles of prototyping were employed to test and modify the software support system, with students from the case study. These students were studying the same module, Team Project, but were drawn from three different years of

three different undergraduate degree programmes: Business Information Technology, Business Information Systems and E-Commerce. Further details of the case are given in Chapter 4. At the same time feedback from students was gathered to provide answers to the research questions posed. It is important for methods used to be reliable, so that readers may be assured that the results obtained represent the true picture, and so that any researchers in the future may carry out identical studies and obtain the same results. Although it is difficult to reproduce qualitative studies identically, efforts can be made to minimise variations in participants, interviewer technique and potential bias, to maximise reliability (Silverman 2000). In this section the methods adopted for this research are introduced to indicate how attempts were made to ensure a degree of reliability. In Chapter 5 (beginning on page 109), where the cycles of prototyping are described, there is a fuller description of the specific data collection methods employed for each cycle of prototyping.

As suggested by Silverman, generalisability from a case study may be achieved through using qualitative research methods, combined with quantitative measures, with appropriate sampling techniques, and careful analysis to tease out structures in basic social order (Silverman 2000:102). The aim of this research was to test the usage of the artefact with a sample of students, in order to generalise to a wider population, and to identify factors affecting their performance.

Questionnaires, focus groups and interviews were used to gather feedback from the students. The questionnaire (Appendix 3) was used for each cycle of the prototyping, as a primary means of gathering feedback from as many as possible of the students involved in the testing, to discover views of the concept of an agent system as well as their opinions of its functionality. It was designed with both open and closed questions. Closed questions were used to ascertain its usefulness, whether the interface was easy to use and self-explanatory, and whether respondents thought that this sort of system would be useful for online students as well as co-located students. Open questions were used to enable the respondents to provide more details, such as suggestions for additions, changes or other functions the system could provide. There was also an opportunity to report difficulties they have observed in team project working (Appendix 3 includes the two versions of the questionnaire used). This questionnaire was administered over a two-week period, immediately after the teams

had completed their use of the output from the system, which took about three weeks. The questionnaire was designed to provide quantitative data upon which decisions on continuing with the research could be taken, as well as qualitative responses to enable the researcher to elicit how these students used the system.

After students had used the first prototype system, a focus group session was conducted. This was timed for after the output from the system had been used, because feedback from all of the students was important to gauge the success of the system, and to elicit suggestions and opinions from the team members. A mixture of team leaders and team members participated in the focus group, thus providing a balanced view of the system. A focus group is a useful way to promote discussion between the participants, whereby ideas from one member may trigger suggestions from another. The researcher attended the session to type the comments from students as the session progressed, but the facilitator, who was an experienced colleague, was able to conduct the session in a manner that was free of researcher bias. A similar focus group session was held after the first trial of the final version of the prototype, and on this occasion the focus group session was tape recorded, and transcribed within a week by the researcher. The researcher was present in the second session, and a different facilitator conducted this session, and who was briefed on the purpose beforehand. Although the presence of the researcher, who was also a team tutor, may have prevented the kind of open discussion required, the researcher was present for practical reasons, and it was thought not to limit useful discussion, which can be a criticism of the focus group technique (Oates 2006).

Starting from the second prototype system, interviews were used as a means of gathering more detailed feedback on how the students used the system, from the perspective of students who took the role of team leaders. There were no interviews carried out after students had used the first prototype system, because it was apparent that the implementation and interface had provided some difficulties, which prevented the students from using the system to its full potential. Apart from the first two questions that were to ascertain whether the team had actually used the system, and whether it was successful, the interview script consisted of open questions, to enable the respondents to provide as much detail as possible to justify their responses. The researcher’s experience and ability to probe for further elaboration of points made was

useful, and given as justification for the researcher conducting all of the interviews herself, even though there is a possibility of researcher bias in the manner in which questions are asked, and responses interpreted. The interviews were conducted after the teams had been able to make use of the output from the system in their team working, and were carried out, by appointment, at mutually convenient times in rooms near to the team working areas.

The following table (Table 2.3) summarises the student teams from the case study involved with the research, the stages of their projects that questionnaires were administered, when interviews with team leaders were completed and when the focus groups were held. The cycles of prototyping comprised three different versions of the student support system, with the final version being tested for a second year with different student teams.

Year of trial 2002 2003 2004 2005 Programming language used Version 1, coded in Prolog Version 2, coded in Java + MySQL Version 3, coded in PHP + MySQL Version 3, coded in PHP + MySQL

Delivery method Internal network Intranet Online Online

Number of teams using it 7 2 of these Sem 2 7 7 in Sem 1 5 of these Sem 2 10 Total number of teams 29 28 24 22 Interviews with team leaders and dates

No interviews Nov and Dec 2003 Team 20 Team 1 Team 10 Team 5 Team 21 Team 9 Team 27

Nov and Dec 2004 Team 13 Team 15 Team 17 Team 7 Team 9 Team 2 Team 8

Nov and Dec 2005 Team 10 Team 14 Team 9 Team 11 Team 2 Team 13 Team 5 Questionnaires to

all team members returned and dates

36 from 7 teams in Sem 1 (Weeks 4 and 5) 14 from 2 teams in Sem 2 (Weeks 4 and 5) 22 from 4 teams in Sem 1 (Weeks 4 and 5) 12 from 3 teams in Sem 1 (Weeks 4 and 5) 35 from 5 teams in Sem 1 (Weeks 4 and 5) Date of focus group and number of students present

Oct 2002 (Week 6) 8 students

No focus group March 2005 (Week 15) 9 students

No focus group

Table 2. 0.3. Summary of research tools used for the cycles of prototyping

It should be noted that the team projects are run over a full academic year, from the end of September to the end of April. Although the system was used at the getting started stage of the team projects in the first weeks of the projects in Semester 1, some

teams chose to use the system again at the beginning of the second semester in February, because their project had changed in scope or their team membership had changed.