Field Study

We conducted a study in course of the regular weekly meeting of a debate club. We decided to test the first prototype in a real world scenario. Our intention was to get feedback early in the design process both from people who are experts in debating, as well as from people who are less experienced but who are probably interested in learning how to debate as they did join a debate club of their own accord. Our main questions were whether speakers can understand and apply the information shown by the system and if or how much the feedback distracts from the actual task.

Method

We conducted the field study using a repeated measures design with two conditions: a base- line in which the speakers were not equipped with a tablet and a condition in which the feedback was displayed on a tablet.

Setup and Procedure

The study was conducted in a quiet room. The study took part on two dates with one week in between. Before each session, the experimenter gave a brief introduction to the study. The system was briefly introduced to the speakers and the juror. Then, the normal procedure of the debates began. In the beginning, the juror introduced the topic. On the first date, the topic was: This house would send the prodigal son back. On the second date the topic was: This house would forbid pornography. These are common debate topics and had been chosen by the debate club. Then, the speakers had 15 minutes to prepare their arguments.

8 Supporting Debates

During the debates, the feedback system was positioned at the speaker’s desk for the sessions in which the speaker was supported by the system (see Figure 8.2). The juror controlled the system by using the interface on the smartphone, additionally to taking notes about the performance of the speaker as jurors usually do.

After the debates, questionnaires using 5-point Likert scales (1 = strongly disagree, 5 = strongly agree) were handed out. Furthermore, a semi-structured interview was held with the whole group after both debates. Both sessions were audio and video recorded. All input from the jury was logged with timestamp and the name of the button that was pressed.

Participants

In the study, 12 voluntary participants took part (2 female, average age 24, range: 18 to 42 years). All of them were members of the same debate club. Two of the participants can be seen as experts (took part in over 100 debates), three of them already had substantive experience in debating (20-60 debates), while seven were classified as novices (less than 10 debates). Four of the participants and one juror took part in both sessions. The other eight participants (four in each session) were different people in both sessions. We gathered qualitative feedback about their experience with the system from the other participants that had used the system only once.

Results

We evaluated the study by measuring the gaze direction of the speakers using the video recordings and by evaluating the reported self-efficacy using the questionnaires (Bandura, 1977; Schunk, 1990). Qualitative results were gathered from the interviews.

Acceptance

In general, most participants regarded the feedback system as a valuable support due to the immediate feedback. Participants stated that the “real-time feedback allows for direct improvement.” (P4), or that“when all three parts are green, I receive an extra push, like a reward system”(P5).

We asked participants if they thought that the separation intoclaim,explanationandexample

was helpful for them. Figure 8.3 shows the answers from the questionnaires. One participant (10%) strongly agreed and five participants (50%) agreed that this was the case. Two of these participants were experts in debates (more than 100 debates) and four were novices (less than 10 debates). Three participants (30%) disagreed and one participant (10%) strongly disagreed. Two of these participants were novices. Both stated in the comments that they either did not get much feedback or did not pay attention to the feedback. The other two had substantive experience in debating (20-60 debates).

These results indicate that especially experts as well as novices who are open to being sup- ported by the system perceive the feedback about the arguments structure as helpful. The one novice speaker who did not receive much feedback did not structure arguments in the way the system suggests, which leads to a situation in which the feedback is less valuable.

10 30 50 10

Strongly Agree Strongly Disagree

Neutral

20 10 20 10

The separation into claim, explanation and example was useful.

The feedback system simplifies learning.

40

Figure 8.3: Results of the questionnaires. Answers to the questionnaires about the structure of arguments and how participants perceived that learning was supported. Numbers indicate the percentage of participants who answered with that score on the 5-Point Likert scale.

The speaker who did not pay attention to the feedback did not believe that he or she could have benefited from the feedback. In the comments, he or she stated: “I don’t believe that I could have changed my speech in real time.” (P11)

We, moreover, asked participants whether they thought that the feedback could simplify learning. Four participants (40%) were neutral regarding this aspect, one (10%) strongly agreed, two (20%) agreed, one disagreed (10%) and two (20%) strongly disagreed. This divergent opinions are also reflected in the interviews after the debate. One participants stated that“when you are a good debater you don’t need it and if you are a poor debater it doesn’t help you (...) but will confuse you.”(P5). Another participant on the contrary thought that it is“super for beginners, because it is the basis of debating to effectively convey your arguments”(P1).

Distraction To evaluate how distracted participants were due to the system, we counted the number of gazes to the system and compared it to the number of gazes to the desk or to the notes in the baseline. All participants used hand written notes that were lying on the table or which participants held in their hands. The average amount of gazes to the desk was 20 without the system and 21 with the system. This shows that the participants did not look at the system more often then they would look at their notes. Additionally, gazes mostly endured less than one second. However, the system was also perceived as“sometimes distracting”(P3) and“too inflexible”(P4).

Estimation of the Jurors In the interviews, we asked the jurors about their experiences with the system. They could take notes, follow the debate and provide feedback through the system without much problems. However, sometimes it was difficult for them to decide which feedback to provide in case a speaker did not follow the given structure at all.

Suggestions for Improvement Several proposals were made by the participants, both by experts as well as by novices and the jurors. Especially the jurors would have liked to have more flexibility in providing feedback. In this prototype, an argument part is either absent or present. However, it is often the case that speakers build arguments of the parts claim,

8 Supporting Debates

ommended to introduce a more fine-grained feedback, for instance, by using colors: yellow could indicate that the argument part has been mentioned, but still needs more elaboration to receive a green rating.

Furthermore, several participants proposed to include more feedback than only the one on the structure of an argument. The juror reported about a debate he had attended in which one person gave the speakers feedback by raising pictures showing symbols about the structure, loudness, pace etc. Several participants shared the estimation that more information than just the structure of an argument can effectively be perceived and included during a debate. Another suggestion for improvement was to include a display of the elapsed or remaining time into the system. In this study, debaters brought their own timer and positioned it next to the system.

Summary and Discussion

In this section, I described a prototype designed for supporting debaters in structuring their arguments. We take advantage of the fact that in debates as practiced in debate clubs, jury members are present who rate the performance of the speakers. Our system uses the ratings of the jury to provide real-time feedback to the speakers with the goal to allow debaters to improve their speech immediately.

We conducted a field study in the course of the debates of a debate club to get some pre- liminary feedback on our prototype both from experts and novices who are interested in debating. Results indicate that participants generally liked the feedback. Especially experts and novices valued the feedback. One assumption why the two experts of our study valued the feedback system might be that they see the use case of training novices with such a tool. Novices who believed that structuring arguments in the three parts (claim, explanation and

example) is valuable and who thought that the feedback could effectively support them in learning how to debate, also valued to system.

A number of areas for improvement were recommended, such as showing more information in a more flexible way. In the next section, I will present a system that is based on these suggestions and describe a study that investigates how much information speakers actually can process during a debate.