from the problem definition and knowledge of what is possible and feasible in attaining a desirable solution. An iteration between the first two steps leads to the drawing of an objective-centered solution. Chapters 1 and 2 of this thesis identified the research problems and outlined the objectives of the study. As outlined in Chapter 1, the present study aims to develop and evaluate a semi-automatic computer-aided assessment framework that can potentially extend and improve on current assessment practices by providing opportunities for detailed feedback.
The conceptual framework for the proposed solution is described in Chapter 4. The chapter sets the stage for the research and specifies the components of the semi-automatic framework. The third step involves the design and development of the tools arising from the components specified in the framework. This step produces the prototype tools in which the research contributions are embedded. The activity here includes determining the functionality of the tools, describing the architecture and selecting the appropriate platforms to implement the solutions. The fourth step encompasses demonstrations to prove that the idea works. The tool is used to to solve one or more instances of the problem. Chapters 5 and 6 sets out the design, development and demonstration of the prototype tools. Running examples from primary mathematics are used to demonstrate these functionalities.
The fifth step is the evaluation of the designed solutions. Evaluation is a key step in the research process and centers on observing and measuring how well the tools support a solution to the problem. This involves empirically comparing the objectives to the actual measured results. The evaluation in done in terms of functionality, completeness, consistency, accuracy, performance, reliability and usability. The empirical evaluation of the prototypes with real users is presented in Chapters 7 and 8. The chapters validates the solutions and informs future iterations.
The final step proposes the need for communication to diffuse the resulting knowledge. This involves communicating the problem and its importance, the tool and its utility, novelty, design and effectiveness to different audiences. This thesis, and the scholarly publications outlined in Section 1.7, are means to communicate the outputs of the study.
3.10
Ethical considerations
Access and ethics are critical aspects for the conduct of research studies. Research ethics refer to the standards of behaviour that researcher’s follows in relation to the rights of those who become subject of the study or are affected by it (Thomas et al., 2008). Ethics define
3.10 Ethical considerations 76
the principles of right and wrong conduct in a community or profession and can be used to make choices to guide behavior (Cocks, 2006; Thomson and Schmoldt, 2001). Gaining access involves convincing people that the researcher has decided upon on who should be the participants that would provide information in conducting research (Feldman et al., 2004). This would require the researcher to talk to many people by developing rapport with them and to be in a position to learn from them (Feldman et al., 2004). According to the World Health Organization, it is important to adhere to ethical principles in order to protect the dignity, rights and welfare of research participants (World Health Organization, 2016). As such, all research involving human beings are reviewed by an ethics committee to ensure that the appropriate ethical standards are being upheld.
The evaluation of the prototype tools developed for the present study will involve the use of human participants i.e. children, students and teachers. Saunders et al. (2012) and Bryman and Bell (2012) summarized ethical issues that should be taken into consideration in research studies. This is presented in Figure 3.3.
3.10 Ethical considerations 77
In line with these requirements the following steps will be taken to ensure that the study complies with high ethical standards.
1. Before the data collection commences, approval will be sought and obtained from the Research Ethics Committee of Loughborough University
2. The informed consent of participants will be obtained before involving them in the study.
3. Participants will not be subjected to harm in any way as they use the research tools. Details of the requirements of the study will be clearly communicated to all participants
4. Participants in the respective schools will not be subjected to coercion in any way. All participants will be informed of their freedom to choose whether or not to participate in the study without any consequence
5. The privacy of the research participants will be ensured so that no personal data will be collected from respondents. No individual will be identified based on the data collected. Pseudonyms will be used to refer to participants in discussions relating to the data collected.
6. The participants will be briefed about aims and objectives of the study before the primary data collection process. All the details about the study are summarized in the consent and information forms used for the study (see Appendix A).
7. All research data, including primary materials and raw data such as questionnaires, measurements, recordings and computer captured data will be stored in suitable and durable storage facilities.
These steps help to ensure that the study is conducted with high ethical standards. As mentioned earlier, the context of the study is within elementary mathematics. As a result, children participants will be sought during the evaluation. According to Flewitt* (2005) and Anderson et al. (2008), four additional provisos are specific to research involving children. These are:
1. Children’s competencies, perceptions and frameworks of reference, differ according to factors such as age and environment
2. Children are potentially vulnerable to exploitation in interaction with adults
3. The differential power relationships between adult researcher and the child participant must be taken into consideration
3.11 Chapter summary 78
4. The role of adult gatekeepers in mediating access to children, with concomitant ethical implications in relation to informed consent
To meet these requirements, additional steps need to be taken in addition to those outlined earlier.
• An initial consent letter will be obtained from the gatekeepers (i.e. parents and teachers or others with a duty of care for the child. This permission does not mean the child has consented to participate, but it allows consent from the child to be sought
• Individual consent will be sought from each child rather than as a group. This is because some children in the group could feel pressured to take part against their wishes (Cocks, 2006; Morrow, 2008)
• Question items used for the study will be carefully worded and made simple to facilitate easy understanding by the children
3.11
Chapter summary
This chapter has presented a detailed account of the research philosophy, approach, method, strategy and ethical considerations used for the study. A brief review of different alternatives and available choices were first outlined. After this appropriate choices and rationale for them were presented. A positivist and interpretive philosophical stance was adopted, abduction was selected as the research approach and mixed/multiple method methodological choice was made. Regarding the strategy case-study, experiment and tracking strategies were viewed as appropriate for the investigations. The overall research process was summarized by the complete research framework flow diagram. Building on these choices, the following chapters present the theoretical and practical details of the study.
Chapter 4
The Semi-Automatic Assessment
Framework
This chapter presents the development of a new framework for semi-automatic assessments in education, with a specific focus on elementary mathematics. Chapter 2 provided the background and the need of having CAA systems that can measure assess the product and process of problem solving. It also explored intelligent techniques that can be used in marking and feedback production. A framework has been defined as a broad overview, outline, or skeleton of interlinked items that supports a particular approach to a specific objective (Framework, 2016). The semi-automatic CAA framework provides a combination of approaches to help address the needs. It aims at aiding students to respond compressively to an assessment task and supporting teachers to assess students’ responses consistently and efficiently. This chapter presents in detail the three important techniques that would be used to build tools for the semi-automatic CAA system. These are multi- touch technologies, analytic rubrics and the case-based reasoning AI methodology. The framework will serve as a prelude to the design development and evaluation of the tools. The chapter also provides theoretical background and discussions from the literature regarding the technologies, design alternatives, design decisions and their justifications for the choices. In the next section, a general direction for the study is outlined. This is followed by a broad description of the conceptual framework. After this, discussions on interactive technologies for problem solving, rubrics and the case-based reasoning method for marking and feedback are presented.
4.1 Rationale for a semi-automatic assessment approach 80
4.1
Rationale for a semi-automatic assessment approach
Section 2.8 contrasted the three major assessment approaches i.e. fully automatic, semi- automatic and manual systems. Manual systems were described as allowing for a high degree of personalisation from the teacher but suffer from slow speeds and inconsistencies. Fully automatic systems, on the other hand, are largely consistent have the advantages of instant scoring. However, they have restricted question types and low degree of personalised feedback. Although fully automatic assessment systems for selected responses such as multiple-choice and true-false tasks are now widespread, the use of automatic assessment systems for constructed response question tasks like essays has not yet received widespread endorsement by educators. Critics have advanced several reasons suggesting that automated systems are no match for live teachers. For instance, Yang et al. (2002) argued that automated scoring of constructed responses suffer from an overreliance on the surface feature of responses, insensitivity to the content of responses, creativity and the vulnerability to new types of cheating and test-taking strategies. Dikli (2006) and Yang et al. (2002) articulated the concern that students’ motivation will be reduced if they know that no human will read their responses. In 2013, Humanreaders.org (2013) launched an online petition, calling for schools, colleges and educational assessment programs to stop using automatic scoring for students’ responses to essay tasks in high stake exams. They listed several research findings suggesting that students, parents, teachers, employers, administrators and legislators should not rely on automatic scoring of essays and that automatic assessment does not promote the authentic arts of writing (Humanreaders.org, 2016).
These arguments advanced suggest that human judgement is still considered critical for assessment of complex tasks. The semi-automatic assessment approach which allows teachers to make judgments while the computer supports the teacher in storing and reusing his or her judgments is highly desirable. This creates a system that leverages on the advantages of both the automatic and manual assessment approaches.