Transformations between Printed and Digital Representations

A final issue, which concerns the entire system, is the question of how printouts and their corresponding digital views are coupled.

Scenario 5 (Interwoven Use of Printed and Digital Representations). After a lec- ture session, Sally usually briefly reviews the contents when she is back at home. She goes through the printed lecture scripts which includes the annotations she has made during the lecture. As she has difficulties understanding one particular slide, she wants to read the an- notations made by the instructor and by her student fellows. For this purpose, she displays the digital version of this slide in the CoScribe viewer. This includes the annotations of other persons.

Then she browses in the digital version and reads other slides, as she hopes finding further helpful annotations of other students. After some time, she comes across a very helpful annotation on another slide. She decides to add this annotation directly to her printed script. She therefore searches the sheet of paper containing the slide and writes the annotation on it. Users can quickly access a digital version of a printed page by tapping with the pen on a button which is printed on each page (Fig. 5.11 upper right). The ecological view can be accessed with a second button (Fig. 5.11 upper left). The thumbnail of this particular document page is then centered and highlighted.

For the reverse direction – accessing a printed representation –, the user can print a new copy of the piece of information. However, it is impractical, expensive and harmful to the environment to re-print the information at a frequent basis. Another way of transformation is to access an already existing printed representation of this

Figure 5.11: These buttons for accessing the page in the document viewer (right) and in the ecological view (left) are printed in the top right corner of each document page.

piece of information. This implies that the user must find this physical copy, which can be hard if the user disposes of a large number of printed representations. There- fore, the digital system should provide support for finding an equivalent physical representation.

CoScribe provides support on two levels:

Map-based indication. In a non-mobile setting, CoScribe can digitally track the po- sition of printed documents using ARTag [Fia04] marker-based location tracking. The location of a given document or document page is then highlighted on a photo of the desk.

Property-based indication. A drawback of map-based indication is that the techni- cal setup for tracking the location of physical object strongly restricts mobile use. A second approach does not indicate the concrete physical location but provides in- formation about properties of the representation, such as its visual appearance. The user can then utilize this information to find the physical representation. CoScribe uses the following properties:

• The visual appearance of the document (the thumbnail or document page view visually corresponds to the printed page). This is particularly helpful if the layout of different documents and pages is diverse. In this case, the size of the margins, the number of columns and the repartition of headings, paragraphs and illustrations can provide a good visual cue for finding a given document or page.

• If the document is contained within a folder (see p. 157), the system indicates this folder. As the number of folders is typically much smaller than the num- ber of documents, this simplifies finding the document page.

• Digital Paper Bookmarks are adhesive sticker that the user can attach to indi- vidual document pages in order to index them. These stickers are well visible

5.2 Integrating Printed and Digital Documents 129

because they jug out of the document. The arrangement of bookmarks on a docu- ment as well as their colors and labels provide good indicators for identifying a specific document at a glance. More details on Digital Paper Bookmarks and the corresponding digital representations are given in Section 5.5.1.

Up to now, we have discussed issues relevant to the entire system that equally concern all the activities of annotating, linking and tagging documents. In the next three sections, we discuss interaction techniques and visualizations that are more specific for one of these three activities.

5.3 Annotating Documents

The interaction techniques and visualizations presented in this chapter aim at sup- porting one or several users to make handwritten annotations on printed and digital documents, to access these in both printed and digital form and to share annotations with co-workers.

On page 2, we defined an annotation as an amendment to an existing document that is conceptually separate yet contextually related. We define a specific sub-type, handwritten annotations, as follows:

Definition 12(Handwritten Annotations). A handwritten annotation is an annotation that consists of a set of ink strokes which are spatially and temporally contiguous and made by one single user.

The contents of handwritten annotations consist for example of an underlining, one or several words, formulae, drawings or combinations of these.

In the remainder of this chapter, we will first discuss the general interaction con- cept. This includes how annotations are made on which media and how informa- tion can be transformed between printed and digital representations. We will then briefly discuss how the user interface for annotating documents can be adapted to the user. In a next step, we focus on collaborative issues, particularly on a paper- based mechanism for sharing annotations and on novel visualizations of shared annotations made by other users. Finally, we will assess the performance of hand- writing recognition and discuss to what extent handwriting recognition can support the interaction and visualization techniques. Table 5.2 provides an overview on the challenges which we address in this chapter.

5.3 Annotating Documents 131

Desideratum Approach Section Concept Function Innovation

w.r.t. related work Main re- quire- ments Flexible free-form annotations Unrestricted PPUI; printing documents with annotations 5.3.1 • • R1, R2, R4a User-adaptable PPUI

Various print lay- outs 5.3.2 • • R4 Free arrangements of several pages 5.3.2 • • • R4b Paper-based shar- ing and classifica- tion of annotations

Tagging with But- tons 5.3.3 • • • R1, R3, R4a Integrated ac- cess to shared annotations Integrated visu- alization using dynamic shrink- ing/expanding and repositioning of annotations 5.3.4 • • R1, R3, R4a Integrated multi- user visualization • • Integrated multi- user printouts • •

Use for presenta- tions Presentation inter- face 5.3.5 • • R1, R3, R4 Access to text of handwritten anno- tations Evaluation and use of handwriting recognition 5.3.6 • • R1, R3, R4

Table 5.2: Desiderata and our approaches concerning the annotation of printed and digital documents

Figure 5.13: Annotating a printed document