Validity and reliability of the CCIT

The CCIT is based on the CCI and similar to the CCI, but changes were made to make it more suitable for diploma students at a South African university. The CCIT further differs from the original CCI in that it contains questions on anti-derivatives, whereas the CCI did not.

It is important to refine the instrument to fit the target cohort to be tested. The original CCI targeted Calculus I students studying for degrees at international universities. The CCIT targeted diploma students taking calculus as a service subject for science and engineering diplomas. Diploma students differ from bachelor degree students in various ways, the most important being lower entrance level skills and academic performance prior to their university studies. The researcher had to take cognisance of this fact. The material for the intervention had to be pitched carefully in order to stimulate conceptual thinking, but care had to be taken that it was not be beyond the capabilities of diploma students, since students could get discouraged and despondent. Furthermore, the quality of the analyses based on the instrument could be compromised.

It is a well-known principle of educational measurement that the difficulty of the items used to assess student achievement should match the ability of the students taking the assessment. In the context of assessing mathematics achievement, measurement is most efficient when there is a reasonable match between the mathematics ability level of the student

152 population being assessed and the difficulty of the assessment items. The greater the mismatch, the more difficult it becomes to achieve reliable measurement. In particular, when the assessment tasks are much too challenging for most students, to the extent that many students are responding at chance level, it is extremely difficult to achieve acceptable measurement quality (Mullis, Martin, Ruddock, O'Sullivan and Preuschoff, 2009, p. 39).

Although the researcher took this statement into consideration, she did not want to deviate too much from the original nature of Epstein’s CCI test, since the latter was, and still is, an iconic benchmark in calculus concept testing. Also, “quality assessment must be aligned with curriculum and teaching to have any effect” (Epstein, 2007). The curriculum for mathematics as a service course for diploma programmes differ quite substantially from curricula for the degree courses in South Africa. Adapting the test was therefore a challenge, because it involved judgements about how the targeted group would respond to the items, both individually and collectively, but doing so prior to having any corroborative information. A balance had to be found between the high difficulty levels of a ConcepTest, and the needs of the targeted cohort. This challenge required subject expertise, teaching experience and PCK particular to the subject and the context in which it is taught (Dunne, LONG, Craig and Venter, 2012). Input was thus obtained from experienced subject experts on the suitability of items and item structures. The test was hence piloted on students who had already passed M2, and afterwards more adjustments were made to the test. As already mentioned elsewhere, persuading students to participate in the pilot study proved difficult. Not a single person pitched for the pilot test when the first call was made. After making incentives available, eight students who had already passed M2 agreed to write the pilot test in February 2015.

The CCIT consisted of 19 questions and covered functions, derivatives and anti- derivatives. Almost all the questions were multiple choice questions. In one question though, students were expected to identify two units of measurement and write these down. The CCIT was used as a Pre- and Post-test for M2 students (Appendix B). Some M2 groups were selected to be in the experimental group who were exposed to the intervention designed for this research, and other groups were selected as control

153 groups. All were tested using the same CCIT test to evaluate the effectiveness of the intervention. The questions were divided into three categories, namely functions, differentiation and antiderivatives (Table 3.5).

TABLE 3.5: CCIT: Three categories of questions

Category Topic Question numbers

1 Functions 2, 11, 13, 15.1, 16, 18

2 Derivatives 3, 4, 5, 6, 8, 9, 10, 14, 15.2, 17 3 Antiderivatives 1, 7, 12

Criterion validity is regarded as the ability of a measure to correlate with other standard measures of similar constructs or established measures. The CCIT was based on the only Calculus Concept Inventory developed to date, namely the CCI. However, only four of the questions in the CCI were retained. The other questions were replaced with similar questions which were deemed more suitable to diploma students. A confidentiality agreement was signed between the researcher and the author of the CCI (Appendix A). The four questions from the CCI that were used in the CCIT, can therefore not be made public. Only their topics are mentioned. The sources of the other questions are depicted in Table 3.7. The reliability of the test was low, as depicted by Cronbach’s alpha in Table 3.6.

TABLE 3.6: Cronbach alpha of the CCIT

Reliability Statistics

Cronbach's Alpha No of Items

154 TABLE 3.7: Sources of questions of the CCIT

Topic Source

1 Integral and constant of integration

Question 12 of “Constructing Antiderivatives Analytically” (Terrell, 2005, p. 3)

2 Exponential growth Epstein

3 Limit Question 15 (Terrell, 2005, p. 121)

This type of problem links differentiation rules with the limit definition of derivative

4 Slope Gibson (2011, p. 96) Gibson, L.R.

5 Velocity Epstein

6 Instantaneous rate of change

Question 24 of the section The Complete Set of Differential Calculus Questions, (Terrell, 2005, p. 8)

This type of problem prepares students for related rates problems. 7 The definite integral and

area calculation

Question 10 of Section 5.2, “The Definite Integral” (Terrell, 2005, p. 6)

8 The derivative of a function Epstein

9 Maximum of a function read from a graph of the

derivative of a function

Adapted from Question 55 Section 6.1 Antiderivatives Graphically and Numerically (Terrell, 2005, p. 27)

155 10 Fourth derivative of a

trigonometric function

Question 121 Classroom Voting Questions: Calculus I The Trigonometric Functions Q4 (Terrell, 2005, p. 32)

11 Reflection and Horizontal and Vertical shifts

Question 140 from “ The Complete Set of Differential Calculus Questions ” (Terrell, 2005, p. 39)

12 Anti-derivative of a function determined from a graph of the function

Question 20 from “The Complete Set of Differential Calculus Questions”, (Terrell, 2005, p. 7)

13 Amplitude and period determined from a graph of a trigonometric function

Question 209 from the “Pre.student.edition.pdf” (Terrell, 2005, p. 57)

14 Testing graphically the signs of the function, first

derivative and second derivative

Question 38 of the section “The Complete Set of Differential Calculus Questions”, (Terrell, 2005, p. 12)

Also Question 11, (Hughes-Hallett et al., 2005, p. 91) 15 Units of first derivative and

of the inverse of a function

Adapted from Question 8, (Hughes-Hallett et al., 2005, p. 88)

16 Use a graph to read values of a function at two points, and the inverse of a function at two points

Adapted from Question 11, Calculus Readiness Test, Department of Mathematics, Simon Fraser University (SFU)

17 Tangents Epstein

18 Linear functions No 13 of the section on Functions and Change (Terrell, 2005, p. 3) . Also similar to question 13.2, from Schlatter (2011, p. 89).

156 The CCIT was also analysed by experienced mathematics lecturers, and their recommendations were implemented. They were hence asked to rate the expected performance of the students in each question in the questionnaire as either very easy, easy, moderate, difficult or very difficult (Figure 3.1). Most of the questions were rated as difficult or very difficult (73.7%), in line with the questions contained in the CCI test. A further 21.1 % of the questions were rated as moderate and only one question was rated as easy. None of the questions was rated as very easy.

FIGURE 3.1: Difficulty level of CCIT as judged by experts