Expert validation and pilot test result

Initially, 31 items (18 closed-ended, i.e. 11 multiple-choices and seven true or false and the remaining 13 open-ended/workout) have been selected. Informed by:

 literature,

 comment form panel of experts, and

 pilot tests the quality of the items was improved.

A pilot test of the items was conducted with students in a private school in the study area. The pilot test was conducted in two rounds. One intact classroom student (they were 27 in number.) in the first round and another intact classroom (they were 31 in number.) in the second round 58 students participated. The aim of the pilot tests had been to get feedback about the items before they were used in the study. The changes made on the items based on the feedback from the pilot and experts were discussed in the following paragraphs. To present the discussion in a reader-friendly format, the following categorizations were used limit of sequences, the limit of functions, continuity of functions, and derivative.

The limit of sequences part initially has six items (thee closed-ended/multiple-choice and three open-ended/workout). The three multiple choose items (item1.1-1.3) were taken with only little modification on the format and one new item was added. The added item (item 1.4) is designed to address the issue of multiple representations. Only one of the workouts items (item 1.5) was taken, and the remaining two items were removed as the other items address their purpose. For instance, one of the removed items was the item asking to find the limit of the sequence ( ( ) ) which was intended to address the issue of alternative sequence. Now, this purpose was addressed by item 1.4(c).

The limit of function part initially has 12 items (three multiple-choices, five true-false, and four workouts). Two of the multiple items (item 2.3 & 2.4) were taken with little modification, one item is removed, and one new item (item 2.5) was added to

73

address the linguistic issues. As informed by the literature, students confuse the terms undefined, does not exist, indeterminate, and infinity. Thus, the new item was intended to confirm this.

The true-false items were converted to two multiple-choices (item 2.1 & 2.2). From the four workout items, one item (item 4.5) was modified so that it accommodates the purpose of one item from the limit of a function and one item from the application of derivatives (see Table 6).

Table 6: Former description and the two items that were incorporated with item 4.5 4.5. The percent of concentration of a certain drug in the bloodstream t hours after

the drug is administered is given by the function ( ) . Then 4.5a. Evaluate ( ) and interpret this result.

4.5b. Find the time (in hours) at which the concentration is a maximum, and 4.5c. Find the maximum concentration.

…The concentration of a drug in a person‟s bloodstream t hours after it was injected is given by ( ) . Then ( ) Interpret this result

…

…What is the maximum value of ( ) on , -? A. D.

B. E. has no maximum value C.

Why do you think so? _________________________________________

The continuity part initially has six items (two multiple-choice, two true-false, and two workouts). One of the multiple-choice items (item 3.3) was taken as it is. One of the observations during the pilot test was that it was hard to analyse students‟ responses for open-ended items as their response was too diverse and the sample was large in number. Based on this observation instead of open-ended, options were provided so that students select the one they think is the right answer. With this consideration, an item (item 3.1) replaced one of the open-ended items with the opportunity to choose

74

from in order to ease the process of analysis. Item 3.2 was developed based on the two true-false items since the multiple-choice items were observed better to address the intended purpose than true-false items. It is observed that the true-false item has less discrimination power7.

One of the workouts items (item 3.4) was modified to accommodate the purpose of one remaining multiple-choice item. Table 7 presents the modified item (item 3.4) and the former version of this item and the multiple-choice item removed since the purpose is incorporated in this item respectively.

Table 7: Former version and an item incorporated with item 3.4 3.4 . Consider the function ( )

3.4a. Sketch the graph of (discuss basic steps of the graph).

3.4b. What can you say about the continuity of the function exactly at ? (say continuous or discontinuous.).

3.4c. Does the function have a limit value at ? (yes /no) (underline your choice).

3.4d. If you answered in 3.4c above is yes, what is that limit value? 3.4e. Compute f at

… Sketch the graph of the function ( ) and answer the following questions.

What happens to the graph of at the point ? ______________ What is the limit of at ? ________________

What is the value of the function at i.e. ( )? ______________ Is the function (continuous/discontinuous) at the pint ? ________ … Let ( )

√ then ( )

A. 6 B. C. -6 D. does not exist E. -5

7

75

The derivative part initially has seven items (three multiple-choices and four workouts). All the multiple items (item 4.1 to 4.3) were taken without any change. Based on the pilot test result, one of the workouts items (item 4.4), was modified to reduce the number of algebraic operations without affecting the intended purpose to be addressed (see Table 8). The item was intended to address the issue of the chain rule. The item has a low correct response due to the algebraic manipulation errors. Table 8: Item 4.4 and its former description

Differentiate ( √ ₎

Differentiate ( √ ₎

One item (item 4.6), taken with little modification and one other item (item 4.7) was completely replaced due to its low discrimination power. Based on the comment from the panel of experts, and the literature the newly added item (item 4.7) is given in graph to address more multiple representations.

Finally, 21 items (15 multiple-choices and 6 workouts), were selected for final administration (see appendix D). All of the items were adapted from different sources. Accordingly, item 1.1 & 2.3 are adapted from (Areaya & Sidelil, 2012). Similarly, items 1.2, 1.4, & 2.5 are also adapted from (Moru, 2006). Likewise, items 1.3, 1.5, & 4.5 are adapted from (Chung, n.d.). In the same way, items 2.1 & 3.4 are adapted from (Jordaan, 2005). Alike, items 2.2, 3.1 & 3.2 are adapted from (Wangle, 2013); items 2.4 & 4.6 are also adapted from (Bezuidenhout, 2001). Correspondingly, items 3.3, 4.1 & 4.7 are adapted from (Rabadi, 2015). Again, item 4.4, item 4.2, and item 4.3 are adapted from (Jojo, 2011), (GRE, 2008), and (IER & AAU8, 2015) respectively.

The purpose of item 1.1 was to establish students‟ knowledge of the definition of terms and the relations and conditions among these terms. Item 1.2 was aimed to determine students‟ computational ability of convergence of different types of sequences. The difference between item 1.2 and item 1.4 is form of representations.

8

76

Triangulation of the two-items result gave an opportunity to establish students‟ abilities in multiple representations and how consistent their knowledge is. The purpose of item 1.3 was to examine students‟ ability in visualization and coordination of processes.

The purpose of item 2.1 and 2.2 were to examine students‟ concept images of the limit of functions. The distractors were designed to accommodate frequently occurring alternative conceptions as described in the literature. Item 2.3 was aimed to examine students‟ knowledge of the non-existence of a limit at a point. Item 2.4 is also aimed to establish students‟ knowledge of the relationship between limit value and function value and the existence of the limit and continuity of functions. Item 2.3 and 2.4 were designed to observe if students are able to interpret the symbolic expression of limit. Item 2.5 was aimed to establish students‟ linguistic ambiguity in a limit. It also reveals more about students‟ algebraic manipulation skills.

The purpose of item 3.1 was to establish students‟ concept image of continuity. The item was designed to incorporate domain-continuity, limit-continuity, and continuity- connectedness interplay. Item 3.2 was also designed to establish more on the interplay between continuity and the other concepts in calculus differentiation, limit, and being defined. The purpose of item 3.3 was to establish how students understand continuity in the subject matter of limit. In addition, the item was aimed to see students‟ ability to compute the one-sided limits.

Item 4.1 was aimed to establish students‟ visualization of the derivative. Besides, it aimed to see computational ability on procedures of the derivative. Items 4.2 and 4.3 were designed to see students‟ knowledge of the conceptual level and how it goes beyond algebraic manipulation. Moreover, item 4.2 demanded reverse thinking, whereas, item 4.3 addressed students‟ ability to form networks of concepts the limit, continuity, and derivatives.

On all these multiple-choice items, besides the purpose in the objective part as explained above, was intended to establish students‟ ability to explain and justify the reasoning employed together with the ability to communicate in written their

77

mathematical knowledge in a coherent, consistent, and flexible mathematical practice.

The purpose of item 1.5 was to dig students‟ representation of the limit (dynamic- static interplay), co-variation, and infinity (actual or potential). The main purpose of item 3.4 was to see how students treat points of discontinuity both algebraically and graphically. On the way to attain this purpose, it also helped to explore students‟ ability on algebraic manipulations, the existence of a limit at a point where the function is undefined, and how they relate limit and the function values. The purpose of item 4.4 was to explore how students‟ understand the chain rule and their computational ability on rules and procedures of the derivative.

The main purpose of item 4.5 was to see how students extend their knowledge on limit and derivative to a real-life problem. On the way to attain this purpose, it also helped to establish students‟: concept image of infinity, knowledge of coordination of processes, the nature of their limit conception, and knowledge on rules and procedures of the derivative. Item 4.6 is aimed to see how well students‟ knowledge structure is synchronized. It addresses the issue of integration among concepts in calculus, i.e. the limit, continuity, and derivatives. It also addresses the issue of representation forms and symbolic interpretation. Item 4.7 is designed to address three purposes- to see students‟ knowledge on continuity in a closed interval, how they interpret the meaning of derivative of a function at a point, and how they relate continuity and differentiability at the same point. All the open-ended items were labelled as object-level conception demanding of the respective concepts.

Finally, appropriateness of language, the time frame of the test, and workspace, level of difficulty, and discrimination power about each item was addressed based on the feedback from both pilot tests and expert‟s comment. While eighteen items were used for the diagnostic assessment (see appendix D), twelve items were used for the post-test (see appendix F).

78

Similarly, the pre-test items were passed through the same process of a pilot and validation. From initially identified 30 items (most of them taken from EUEE9), through validation and pilot test 25 items (function, sequence, geometry, algebraic computation, and application problems) had been selected and was used for final administration (see appendix E).