Capturing the state of the audience during a lecture balanced between intrusiveness and objectivity of the capturing method. Invasive approaches such as equipping students with eye- trackers (Rosengrant et al., 2012) were not scalable to the classes of 30+ students. We developed two types of questionnaires which we used inter-changeable on two student populations. Questionnaires were developed with the generous input from Roland Tormey and Jessica Dehler, who helped both to formulate the questions and the format of the questionnaire. We did not use previously developed systems for evaluating public speaking, such as PSCR - Public Speaking Competence Rubric (Schreiber et al., 2012). Even though such solutions are well adjusted for evaluation of the presenter, the main focus of the research is to evaluate the state of the audience as a reaction to the efforts of the speaker.
We also conducted a pilot-study using the clickers (Caldwell, 2007) instead of the paper-based questionnaire. In our experience, the clickers were much harder to setup and more disruptive to the class (students took longer to respond, and were more confused with several questions presented). Even with the prior explanation of the usage, students were confused when several questions were asked in a single fill-out period without visual guidance from the clicker devices (questions were presented separately, on the projector, but the device lacks clear identification which question is being answered). Also, each sampling time required more involvement from the teacher, who needed to incorporate the questions into the lecture presentation. The approach was abandoned after the initial attempt.
The main difference between two used questionnaire formats was the time when the student was expected to fill-out the data. Our first attempt was aimed at simulating strobe-sampling of student attention (similar to attempts of Lindquist and McLean (2011)). We will refer to this format as in-class questionnaire. In the second format, the paper was filled-out at the end of teaching period, and will be referred to as post-class questionnaire. Questionnaire designs are shown in Figure 4.3.
All values collected with the questionnaires are shown in Table 4.2. Over the period of two years, the format of the questionnaire evolved, with some questions taken out in the attempt to minimize the impact of data-collection on the lectures. This is one of the reasons for the uneven number of samples shown in Table 4.2. Second reason is that students refused to answer some of the questions (typically - classroom attention was usually left out, because students refused to evaluate their peers). The four measures listed as “perception” (attention, classroom attention, teacher’s energy and material importance) were present in all versions of the questionnaire.
With teachers’ cooperation, we also conducted the post-class knowledge test related only to the content shown during that lecture. The questions were prepared by the teachers and projected in front of the classroom, while the students filled out the answers on the back of the questionnaire sheets. The tests were not previously announced, were not collected as
4.3. Capturing changes in attention
a) b)
Figure 4.3 – The a) in-class and b) post-class questionnaire used during our experiments.
part of the grading, and took around 5 minutes to finish. The scoring of the tests used in our analysis was normalized to the scale of 0.0-1.0, to make tests with different number of points comparable.
4.3.1 In-class questionnaires
In-class questionnaire (shown in Figure 4.3a) was created with the goal of sampling the perceptions of students during the lecture. The entire lecture was represented with four samples, each reflecting roughly 10 minutes.
The procedure included distributing the questionnaires at the beginning of the class. At four equally-spaced moments during the lecture a sound signal was given, at which point the students filled out the appropriate questionnaire section (one of four), indicating their attention, class attention, etc of the previous 10 minutes. The filling out procedure was aimed to take less than one minute, including the time needed for the students to find the paper and set it back away.
The early versions of the questionnaire (used in Populations 1 and 2) also contained the activity questions with which we tried to sample the frequency of three activities positively associated
Question Format No. sessions No. samples Perceptions (in- and post-class)
Personal attention Likert scale (1-10) 9 1075
Classroom attention Likert scale (1-10) 9 1025
Teacher energy Likert scale (1-10) 9 1062
Material importance Likert scale (1-10) 9 1058
Pre-class questions (only post-class)
Prior interest Likert scale (1-10) 3 388
Prior knowledge Likert scale (1-10) 3 388
Post-class questions (only post-class)
Test of knowledge Open-ended questions 3 344
Activities (only in-class)
Listening Check-box 2 56
Taking notes Check-box 2 56
Repeating key ideas Check-box 2 56
Distracted thoughts Check-box 2 56
Interacting with others Check-box 2 56
Using laptop/phone Check-box 2 56
Table 4.2 – Parameters collected with the questionnaires, with the number of samples and classes captured. Brackets beside the parameter group name indicate the questionnaire format which was used for that group.
with learning (listening, taking notes and repeating key ides) and the three common activities negatively associated with learning (distracting thoughts, interacting with others without permission and using laptop for tasks unrelated to learning). The students were instructed to indicate yes/no answers for all activities they performed in the previous time segment. The question were later removed in the attempt to minimize the disruption of the classroom. Overall statistic of the periods captured by in-class questionnaire shows that the average time length of the period was 10.6 minutes, and the average break length used for filling out the questionnaire was 44 seconds. This questionnaire format was used in 6 recordings.
Even though the interruptions were kept to a minimum, this approach to sampling attention was considered intrusive on the learning experience. With the concern that the approach might bias the students in the positive direction (mentally “waking up” the students and making them conscientious about their learning) the post-class approach was developed.
4.3.2 Post-class questionnaires
Post-class questionnaire (shown in Figure 4.3b) has restructured in opinion sampling into 2 blocks: pre-class and post-class, leaving the lecture time intact.