Choosing to perform a case study does not immediately yield a set of methods or proce- dures to be followed in order to conduct a study: “[it] is not a methodological choice but a choice of what is to be studied” (Stake, 2000, p. 435).
Merriam suggested that there are three main methods of data collection techniques in case study design: interviews, observations, and document data mining. The choice of methods from these options is informed by the literature, the theoretical framework and the research questions; but there are other considerations and constraints, such as time and cost.
Nonetheless, “in education if not in most applied fields, interviewing is the most common form of data collection in qualitative studies. In some studies, it is the only source of data” (Merriam,2014, p. 86, emphasis in original).
The artefacts produced by CAA testing could be considered documents that could be mined for data. While these documents give some idea of the actions of the lecturers and students, the context of these actions and the details of these actions themselves are not recorded by the CAA system. It was possible to obtain data relating to the length
of time students spent on practice tests and the number of practice tests attempted, for example; likewise, data on the location and times that these tests were attempted were also available. On reflection, the author believes that there could have been some value in obtaining such data in relation to the case study participants. However, there were two concerns that precluded the pursuance of these data.
The main concern was that it was not a straightforward task to isolate individuals’ CAA activity from the system log files. Indeed, it transpired to be sufficiently difficult to obtain raw CAA summative test scores for cohorts to be discouraged from doing so. A request for individuals’ data would have taken a significant amount of an administrator’s time. There was also an ethical concern about obtaining an individual’s data — it would have had to be with the participant’s permission, and this might have had an effect on the recruitment of participants. Given that there transpired to be some difficulty with attracting participants, this was a risk that was not worth taking.
Having ruled out using artefacts from CAA testing, observations were also deemed to be unsuitable for this study.
4.2.1 Ruling out observations
Observations would appear most compatible with the theoretical framework. Kaptelinin and Nardi (2006, p. 31, emphasis added) stated:
“Activity theory maintains that no properties of the subject and the object exist before and beyond activities (e.g. Leont’ev 1978). These properties do not just manifest themselves in various circumstances; they truly exist only in activities when being enacted.”
Jonassen and Rohrer-Murphy (1999, p. 68) argued this case: “first and foremost, whatever the focus, the activity must be studied in real-life practice with researchers as active participants in the process”. Kuutti, Iacucci, and Iacucci further underline why active participation is important:
“One of the epistemological formulations in activity theory has been devel- oped by V. V. Davydov, who separates empirical and substantial generalisa- tions in the following way. Empirical generalisations are based on external observation and they are formulated by classifying and naming observable features into groups that are similar in some respect. The significance and relations between these classes are difficult to identify and uncertain, and the
generalisations themselves are static and lifeless. Substantial generalisations must be based instead on purposeful interactions with the material to be studied. Only in such interactions the hidden connections, dynamics and re- sistance will reveal themselves. Substantial generalisations are thus systemic and dynamic, and further actions can be based upon them.” (Kuutti et al.,
2002, p. 97, emphases in original)
However, there are practical and ethical challenges to using observations in such a study where there are human participants undertaking a task that has long term implications on their success when they are either studying at or working for a university.
How students use computer-aided assessment is one focus of the project. For a simple observation to be possible, the researcher would need to know where and when students use computer-aided assessments. However, since all students are permitted to access the practice tests at times and locations of their choosing, this would require the researcher to be a participant in the social setting. From ethical and practical perspectives, this is not acceptable. Students may undertake these tests spontaneously, at any time of day or night, with others or on their own, and in locations not accessible to the researcher. There is also a problem with regard to the reliability of the data collected. It is quite likely that a student would modify their actions for the presence of a researcher, particularly if they would normally engage in practices that would be considered plagiarism or in some way contrary to expectation. Observations conducted without the student being aware of the presence of the researcher may yield more reliable data, but would be ethically and practically unacceptable.
Similar arguments apply to the lecturers’ case as they prepare CAA tests for students or create new questions for the question bank. It is possible that lecturers might adjust their practices if they were aware of the presence of an observing researcher, for example. Given that observations would likely be obtrusive and would manipulate the actions of both students and lecturers, they were not suitable methods for this study. Although some of the context and detail available in observations would be lost by using interviews to collect data, interviews were less obtrusive than observations and offered the opportu- nity to ask students and lecturers about their thoughts and past experiences with regard to CAA.
While interviews would give the level of detail that a case study would require, the author used questionnaires to gain a broad understanding of the society and culture in which individuals were subjects in the activity of learning at university. The questionnaires
were distributed to get an understanding of the population before interviewing a sample of students and lecturers.
Students are the most numerous end-users of computer-aided assessment. Most students will only ever see the front-end of the system in the test and practice areas. Yet, over the course of one academic year, students’ experiences differ. These are, in part, explained by the different practices they experience, but they also depend on their histories and the culture of the social groups they have become immersed in.
The questionnaires sought to provide some context for their histories and the activity while the interviews served to gain deeper insight into the perceptions and experiences of a small selection of individuals in that population.
For lecturers, computer-aided assessment seems to offer an ideal solution to a common problem. Students often need encouragement to work between lectures, and while as- sessment offers this encouragement it can often be time-consuming. Computer-aided assessment gives students both the motivation to obtain the marks that are on offer and feedback for the work they do. The feedback is an important consideration — it would be time-consuming to offer a similar feedback service if assessing “offline”.
This lecturer study considers how CAA might be effective for lecturers. It observes the emergence of different practices of students and the reasons for this divergence. A questionnaire on practices and opinions towards CAA was followed by interviews to explore the reasons for these practices and opinions.