Student Perceptions of and Interactions with the SI Tutor

Central to this project is the investigation of how students perceive and interact with our synthetic tutor. Of the 22 students who we interviewed 19 used the system in a configuration that featured an SI tutor (the others used a configuration that just featured the lessons). In collecting interview data with these students it is important to ascertain that the students have had sufficient experience with the system to offer informed comments. That is one of the motivations for conducting interviews over three sessions. Furthermore, in a large majority of the interviews the student either completed the entire lesson within the one hour allotted, or completed most of the lesson activities. The majority of the instances for which most of the activities were not completed were in the first session which was the students first encounter with the system and also contained the additional lesson 0 training activity which required extra time. This indicates that our lessons are at approximately the correct length for students in our target population to work through in an hour, and more importantly that students’ comments in the subsequent interviews were based on significant, if not complete experience with the system. As a side note, it also compares well with the method we proposed previously for estimating completion time when direct observation isn’t possible (Nakamura et al., 2010). In the following two sections the most important themes from student interviews based on their experiences with the synthetic tutor will be discussed.

4.1.1.1 Social Interaction with the SI-Tutor

Social interaction is critical to effective tutoring and we must look for evidence for or against the idea that students may interact with the SI tutor in a social manner. Moreover we must ultimately look for ways to promote this type of interaction. Evidence for this type of interaction is conflicted. Students frequently discussed the SI tutor as if it was a person (“I asked her about…”) however observations based on the video recordings of students using the systems indicate use that is more consistent with it as a video player. Students were observed to rewind, pause and fast-forward the SI tutor or cut it off mid-sentence when it became clear the response was not the one that they were looking for. In a sense this should not be too surprising. Those features are built into the video player and so we must expect students will take advantage of them. The best that we can conclude is that there is opportunity in this type of system for social interaction, but there is no conclusive evidence that that social interaction is occurring.

63

4.1.1.2 Student Framing of the SI Tutor’s Instructional Role

One of the most interesting themes in interviews with students was the idea of using the synthetic tutor “just to make sure” or “to confirm what I knew.” Students repeatedly cast their use of the SI tutor in this light. We can see an important and interesting case in the following excerpt, in which a student has claimed to use the SI tutor to confirm what he knew, but what he knew was wrong. At one point in the student’s third interview the interviewer asked the student about his use of the SI tutor:

I: Ok. Ok. But other than that...do you feel like you used the tutor more or less than last time? S: I mean those are two video clips I watched because I was unclear, I watched a couple more

while I was doing it just to be clear on what I was doing. I think I knew but I wanted to make sure I was clear.

I: mhm

S: So I'd say I used it probably the same, but not for the same purposes. I: Ok.

S: The times before I'd watch it more because I didn't understand what was happening. This

time I watched just to make sure I understood what was happening.

I: Ok. Ok. And it reinforced what you were thinking about the physics? S: Yes. Correct.

This exchange clearly indicates that the student used the SI tutor, or at least believed he used the SI tutor to confirm his understanding. In this same interview the interviewer and the student were discussing the physics of a train crash activity:

I: So in the first activity we give you these two trains that collide. And we ask you to calculate

the force that each one feels right?

S: mhm

I: Can you tell me a little uh, about what you did there?

S: Uh force was just uh, acceleration, no, mass divided by acceleration that’s basically how you

get how many Newtons of force there are.

I: mhm

S: And so, that's basically how I found the force of each one. I: Ok. So uh force was mass divided by acceleration?

64 S: Yeah

This exchange clearly indicates a misunderstanding, or misremembering of Newton’s 2nd law. Looking at the log data for this student’s user account indicates that he selected the

question “What does Newton’s 2nd law say?” via the related questions menu. So he was presented with a correct statement of Newton’s second law, which would not have confirmed what he already knew. Another related interesting exchange occurs later in this interview:

I: Ok. Ok. Uhm, so if they felt the same force and had different masses and you're saying that

the force is the mass divided by the acceleration right?

S: Yes I: Uhm.

S: I think. Hopefully.

I: Ok, so that's what I'm wondering here. Do you remember Newton's Second Law from last

time?

S: Newton's Second Law is uhm, about, net force right? I: uh huh.

S: All the forces acting on an object, that's the net force, and that's Newton's Second Law. I: And how do we calculate the net force?

S: Adding up all the forces being applied, forces this way, that way, gravity, normal force. I: mhm. And what does that equal?

S: Well if it's stationary it should equal zero. I: mhm.

S: But if it's a non zero net force that means the things accelerating. I: Ok.

S: Yeah I think that's right. I: It's accelerating?

S: Yes.

I: More force means more or less acceleration? S: More acceleration.

I: Ok. Uhm...how- when you say that how does that make you feel about the equation force

65 S: That they're interconnected.

I: I guess let me rephrase that- S: Yeah.

I: If you say force equals mass divided by acceleration, and you make the force bigger, will

acceleration- keep the mass fixed- will acceleration get smaller or larger?

S: If you make the force bigger? I: mhm

S: The mass is the same.

<long pause>

S: The acceleration would get smaller, math-wise technically because you could switch the a and

the F so it would be m over F equals a. So that way if the force is larger, technically the acceleration should be smaller. I guess yeah that's true...math-wise.

I: Which do you have more confidence in? The equation force equals mass divided by

acceleration, or your idea that when you apply more force you get more acceleration?

S: I mean in my mind, I'm trying to think of a real life example where the more force you apply

the faster something would go, but when you look at the math equation it says the opposite of that, so I'm confused now. I don't know.

I: Sure. Is it possible you're remembering the equation wrong?

S: Yeah. Definitely possible I'm remembering the equation for force wrong. <long pause> Is it

mass times acceleration? It might just...it is mass times acceleration, ok.

I: mhm

S: So, ok. That makes sense now. That makes more sense than mass divided by acceleration. This episode highlights a potential pitfall in students’ use of our system as well as an example of what a synthetic tutoring system must aspire to. It highlights the former in the sense that we observe a student who is claiming that the SI tutor is just confirming what he knows, but in reality there is a problem with what he knows that the SI tutor could help him with, if he were open to it. The interviewer is successful in helping the student recognize this issue, but the interviewer doesn’t do it by providing declarative statements that are in contrast to the students understanding. The interviewer does it by asking the student questions that forces the student to reflect on what he thinks. In this way the interviewer is able to lead the student through a line of

66

reasoning that ultimately allows him to realize his error. This example likely illustrates one of the main reasons one-on-one tutoring is more effective than other means of instruction. Examples like this were not uncommon in the corpus of data, though this is the most striking because of the stark conflict between what the student believed and the accepted physical law and because of the success the interviewer had in guiding the student. Examples where the interviewer tried to do this and failed to help the student reach the correct conclusion were also present in the data. The SI tutor is not currently capable of leading a student through this type of sophisticated and nuanced reasoning chain. One student observed as much indicating that while asking the SI tutor questions might be helpful, the questions that the interviewer was asking her were actually more helpful in getting her to understand things. This is not surprising. In a real tutoring session the tutor doesn’t just answer questions she or he also asks the student questions.