2.3 Field Studies
2.3.2 Study II: Annotating with Pen and Paper vs Annotating with
After having analyzed which media students take notes on as well as the influences of contextual factors, the goal of a second study was to take a closer look on the annotations themselves. In order to assess the influence of the annotation medium on the annotations, we collected annotations made by postgraduate students on presentation slides and manually analyzed this user generated content.
3Review after class: Annotators: M
1 = 3.3 (SD1 = 1.1, N1 = 132) vs. Notetakers: M2 = 2.9 (SD2=1.0, N2=89), T= −2.797, d f=219, p=.006.
Review before the exam: Annotators: M1=2.5 (SD1=1.1, N1=127) vs. Notetakers: M2=1.9 (SD2=1.0, N2=88), T= −4.034, d f=213, p=.000.
Completion before the exam: Annotators: M1 = 3.6 (SD1 = 1.2, N1 = 109) vs. Notetakers: M2=3.1 (SD2=1.4, N2=82), T= −2.681, d f=189, p=.008.
4Review after class: Collaborative users: M
1=3.3 [SD=1.0, N=134] vs. Non-collaborative users: M2=2.7 [SD=1.1, N=92]), T= −4.142, d f=224, p=.000.
5Completion after class: Collaborative users: M
1 =2.4 [SD=1.0, N=129] vs. Non-collaborative users: M2=1.9 [SD=1.0, N=91]), T= −3.671, d f=218, p=.000.
2.3 Field Studies 29
Figure 2.7: Screenshot of the tool for textual annotations and highlightings
Method
A total of 24 postgraduate students participated to this study. These students at- tended an interdisciplinary seminar on technology-enhanced learning. Each semi- nar session consisted of a presentation held by a student and of a subsequent ple- nary discussion. All presenters used PowerPoint slides during their talks. The par- ticipants could annotate these slides. We collected and analyzed annotations from nine seminar sessions, each lasting about 90 minutes and containing an average of 24.7 slides (SD =18.3). We moreover conducted informal interviews after the ses- sions. Participation was voluntary and no compensation was given. We chose a between-subjects design for the study.
Pen and paper condition. One group of students used pen and paper to make per- sonal annotations on printouts of the presentation slides, which we distributed be- forehand. In order to guarantee a realistic setting, each printed A4 page contained four presentation slides. (Informal investigations had shown that this layout is used by most students for printing lecture slides.) Figure 2.10 on p. 33 gives examples of handwritten annotations. A handwritten annotation consists of a set of one or more pen strokes a user has made on a page.
Laptop condition. A second group of students used an annotation tool developed by our department (see Figure 2.7). This enabled the user to make typewritten anno-
tations and textmarker highlightings on the presentation slides using a laptop. This tool displayed the presentation slides in real-time and moreover enabled manual scrolling through the entire slide set. The tool was designed to make typewritten annotations as easy as possible. In order to make an annotation, the user marks a context area on a slide by drawing a rectangle and then enters the text of the annota- tion. The context area is then permanently visible. The text of the annotation can be viewed later on by hovering with the mouse on this area. Alternatively, if no text is entered, the context area is displayed as a yellow highlighting in a textmarker style. Individual notes can optionally be labeled with a semantic type (question, attention or problem). Moreover, the user can optionally mark an annotation as public. This annotation is then shared with other users over a network connection and immedi- ately displayed in their slide sets as well.
Students who disposed of an own laptop participated to the laptop group, while the remaining students participated to the pen and paper group. Since not all stu- dents attended all sessions, the size of the groups varied. In the pen and paper group, an average of 6 students (SD = 1.7) participated to the each session. An average of 4.3 students (SD=2.2) participated to the laptop group. Participants of both groups were instructed to make annotations the same way they normally do during the seminar.
Annotation segmentation. A methodological problem with the analysis of hand- written annotations is how to segment all pen strokes on a page into individual annotations. This segmentation cannot always be made unambiguously by persons other than the annotator, since the annotator typically does not explicitly mark the extent of an annotation. For counting the number of handwritten annotations, we had to manually segment them. In order to guarantee a more reliable segmentation than by following merely subjective classification criteria, we used the following rough guidelines for the segmentation procedure:
• An annotation must have a coherent content on a semantic level.
• The strokes forming one annotation must be spatially coherent, i.e. two clus- ters of strokes that have a large distance are considered as two separate anno- tations.
• Clusters of strokes that are visually separated in an explicit manner (e.g. by a line drawn by the annotator) are considered as separated annotations.
• Sketches are own annotations. However, if text can be attributed unambigu- ously to a sketch, both this text and the sketch is considered as one annotation. In contrast, the segmentation of typewritten annotations was not problematic, as a typewritten annotation consists of a text string. This text can be unambiguously attributed to this annotation.
2.3 Field Studies 31 30 40 50 60 o tation s per s e ssio n Laptop Paper 0 10 20 30 40 50 60 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 Av. num ber of an n o tation s per s e ssio n User Laptop Paper
Figure 2.8: Average number of annotations made by each participant per session. The horizontal lines indicate the average over all participants of each condition. Error lines indicate the 95 % confidence intervals of the means.
Results
We collected a total of 1409 typewritten and handwritten annotations.
A main finding was that the use of a keyboard seems to constrain the annotation process. We collected all typewritten and handwritten annotations, manually seg- mented handwritings into individual annotations which were spatially contiguous and seemed to express the same content and finally counted all annotations. On average, participants of the laptop group made significantly less annotations per participant and session (M = 8.9, SD = 6.0, N = 10) than those of the pen and paper group (M = 19.9, SD = 12.1, N = 14) (t = 12.1, d f = 22, p = .015). In- formal interviews with members of the laptop group indicate that the participants perceived typing annotations on a keyboard as rather inflexible and complicated. Moreover, making handwritten annotations was judged being faster than typing text on the keyboard. Two participants of the laptop group explicitly stated having observed that they frequently did not make annotations they would make with pen and paper because it took too much effort.
A manual analysis of the annotations showed that the annotation style is highly in- dividual, a finding which is consistent with studies of textbook annotations [Mar97, Mar98]. On the one hand, this concerns the number of annotations made. Figure 2.8 depicts the average number of annotations made by each participant per session. In the laptop group for instance, the number of annotations per participant and session varies from 4 to 24. In the paper condition, this number varies from 3 to 48. On the other hand, the style of individual annotations can highly vary. While some users prefer making their annotations within the slides, others obviously prefer notes at the outer zones within the margins. Some participants write short and abbrevi-
40 50 60 70 80 nno tati ons pe r us er 0 10 20 30 40 50 60 70 80 1 2 3 4 5 6 7 8 9 Av. num be r of anno tati ons pe r us er Session Laptop Paper
Figure 2.9: Average number of annotations per user of each session. Error lines indi- cate the 95 % confidence intervals of the means.
ated notes in a telegraphic style or do only underline important passages without making textual comments at all, whereas others tend to write more extensive notes. Figure 2.10 shows examples of different annotation styles.
Another finding was that the course context heavily influences the number of an- notations made. Figure 2.9 gives the average number of annotations per user for each session. It shows that there is a large variance between the individual sessions. A further relevant outcome was that the majority of the participants used spe- cific classification schemes to classify document contents for later retrieval. In or- der to do so, participants used symbols with a meaning which is generally well- understandable, such as question marks and exclamation marks (76 % of the par- ticipants). Moreover, 43 % of the participants used additional symbols which they have defined themselves and whose meaning would not be evident for others. Fig- ure 2.10 (lower right) shows an example of a user-defined symbol.
We manually analyzed 1097 handwritten annotations and identified types of an- notations and their repartitions. Figure 2.11 depicts the type of the annotations. 60.3 % are annotations with an implicit context area, i.e. without an explicit high- lighting of the area in the document the annotation applies to. 15.9 % of the anno- tations contain only highlightings (e.g. an underlining of a phrase or a circle or box around a printed passage). 11.5 % of the annotations contain both a highlighting of the context area and a textual content. Further, 10.8 % are specific annotations that contain only a symbol and act as semantic markers in the sense described above. For this purpose, the use of standard symbols (like question and exclamation marks is more common than using personal user-definded symbols. Finally, 1.5 % of the annotations are lines that are drawn to make a clear delimitation between two or more annotations written near each other.
2.3 Field Studies 33
Highlighting; 15,9% Compound (Annotation + Highlighting); 11,5% Annotation; 60,3% Highlighting; 15,9% Compound (Annotation + Highlighting); 11,5% Standard symbol; 7,8% User‐defined symbol; 3,0% Delimitation line; 1,5%
Figure 2.11: Types of handwritten annotations (N=1097)