Data analysis

3.3.1 Pretest

The pretest is given to the students in the preparation phases. The data that we have are students‟ written work when they are solving the test items and students‟ verbal explanations (video recordings) in the interview session after they take the test. We develop a rubric (see pre and posttest rubric) to rate the students‟ works. The data are carefully analyzed according to the rubric in order to investigate students‟ prior knowledge and to know the starting points of students about the concept of angle. The results of the analysis are used to make some adjustments in the initial HLT to improve the predictive power of it. In addition to that, the results of the pretest are used to select the focus group that consists of students with various level of knowledge about the topic.

3.3.2 First teaching experiment

The aim of the first teaching experiment is to get an insight into how the selected students react on the designed tasks. In this case, the selected students act as a „miniature‟ of the students in the second teaching experiment. We analyze the data in this phase using a task-oriented method in order to know how the predictions of the HLT correspond (or don‟t correspond) with the students‟ actual learning process. The data analysis is performed in the following steps:

1. Video observation

The videos of a lesson are watched with the research questions and the HLT as guidelines. Here, the focus is to find confirmation and counter- examples for the conjectured learning process in the actual learning process.

2. Video observation notes

The interesting fragments in the videos of a lesson are excerpted. Here, the interesting fragments refer to any observable and interpretable activities in the lesson that can be categorized as confirmation or counter-example of the students‟ learning.

3. Dierdorp‟s analysis matrix

The excerpts from the videos of a lesson are analyzed in Dierdorp‟s analysis matrix in order to know how the predictions of the HLT correspond (or don‟t correspond) with students‟ actual learning process.

The results from this analysis are used to calibrate the initial HLT in order to make the HLT ready to use in the second teaching experiment. Ideally, after the task-oriented method, we could perform the „constant comparative method‟ to gain more theoretical insight into the learning process. However, since this is a small scale study, we cannot perform the follow-up analysis due to time restrictions.

3.3.3 Second and third teaching experiments

Similar to the analysis in the first teaching experiment, in these sub-phases we analyze the data using a task-oriented method. The results of the analysis from this phase are used to answer the research questions, generate a conclusion, and revise the HLT.

3.3.4 Posttest

The way we analyze the data from the posttest is similar to what we do in the pretest. However, we also compare the posttest results with the pretest results quantitatively to know in general how well the knowledge gained by the students and qualitatively via interviews to evaluate and examine the development of students‟ learning and understanding of the concept of angle. All the outcomes from this phase are used as additional data for triangulation, answering research questions and drawing the conclusions.

3.3.5 Validity and reliability

According to Bakker‟s and Eerde‟s submitted paper (2013), internal and external validity and reliability seem most relevant in the context of design research. Therefore, in this part of this chapter, we will describe these types of validity and reliability related to the data analysis in this study.

1. Internal validity

In the analysis phase, the internal validity refers to the soundness of the reasoning that has led to the conclusions. In order to improve the internal validity of analysis of this study, we take the following steps:

In the retrospective analysis, we analyze the data using a task-oriented method in order to generate and test the hypothetical learning process in the HLT. We also perform data triangulation with other data, such as students‟ written work, field notes, and video registrations of interviews and lessons in order to strengthen (search for confirmation and counter-examples) the results from the retrospective analysis.

2. External validity

External validity is strongly related to the generalizability of the results. In design research, the generalizability means that others can adjust and perform the current study to their local contingencies. In order to improve the external validity of this study, we utilize the explicit educational materials (HLT, teacher‟s guide, and students‟ worksheets) that can be easily followed by others.

3. Internal reliability

Internal reliability refers to the degree of independence of the researcher of the collection and analysis of the data (Bakker and Eerde, 2013). In order to improve the objectivity of the data analysis, during the retrospective analysis we discuss the critical transcript from the actual learning process with colleagues for peer examination.

4. External reliability

External reliability usually denotes replicability, meaning that the conclusions of the study should depend on the subjects and conditions, and not on the researcher (Bakker and Eerde, 2013). For improving the external validity of this study, we present the study in an explicit way

(how the study has been carried out and how the data are analyzed and the conclusions have been drawn from the data), so the other researcher can track the whole process of this study.