Introduction

Nowadays people interact with multiple types of touch surfaces in every- day life. Half of the world population uses a smartphone as an everyday tool; tablets are less ubiquitous, but still widespread; laptops are increas- ingly getting touchscreens and becoming more similar to tablets; and interactive tabletops and public displays are gaining attention.

We analyze performance and ergonomics of interaction with the above- mentioned touch surface types. These factors were already studied be-

1_{This section is based on the paper Performance and Ergonomics of Touch Surfaces: A}

fore, for example in [129,133,138,141], but in most cases only a single surface type and single factor (either performance or ergonomics) was in the focus: tabletop [131], vertical display [130], tilted display [132], or mobile devices [133]. Each of the existing studies was also different with respect to task, performance or ergonomics measure, participant popu- lation, devices used, etc., which makes it impossible to compare results or consolidate them into a single body of knowledge. As a result, there is a lack of knowledge of how different form-factors affect touch interac- tion, which would be necessary to create applications compatible across devices.

In this section we describe a user study directly comparing the touch surfaces in a target selection task in a within-subject design. In this way we can directly compare performance and ergonomics factors among the surfaces. As a result, we can identify strengths and weaknesses of each surface type and trade-offs in speed, accuracy, joint and muscle loads and loads sustained over time causing muscular fatigue. From the find- ings it is possible to derive recommendations for design of user interfaces satisfying both performance and ergonomics requirements. In this way new designs can avoid or minimize adverse effects, for example “touch thumb”, “gorilla arm”, or “smartphone neck”.

While interaction with touch surfaces involves the same basic princi- ple, namely “direct touch”, every surface involves different types of pos- tures, limbs, movements and accuracies. We assume that these differences are reflected in the human musculoskeletal system, and the differences in performance and ergonomics characteristics of each surface can be at- tributed to the underlying biomechanics. Touch surfaces are very flexible in terms of how they can be situated in space. Furthermore, they can be carried on the user. This flexibility involves an immense space of possible interaction postures. In this study we do not limit the participants’ pos- tures, allowing them to take whatever is preferred by them; based on this data we analyze the posture space and identify “typical” postures—the postures commonly used by participants. We analyze typical postures used in interaction with 5 surface types:

1. public display: large area, vertically positioned, used while stand- ing

2. tabletop: large area, horizontally positioned, used when seated 3. laptop: medium area, adjustable tilted position, used when seated

4. tablet: medium area, handheld

5. smartphone: small area, used with one or two hands

In the user study we apply the motion capture-based biomechanical simulation pipeline described and validated in this thesis. Unique to this study is the free choice of posture by the participants for each touch sur- face. The time series of postures of all participants are then grouped per surface type and clustered into equivalence classes: similar postures used by different users belong to one class. Further, to attribute the perfor- mance and ergonomics characteristics to the posture type, more detailed analysis is performed individually for each class. In multiple conditions the users were interacting with the surfaces in a sitting posture. As men- tioned in previous chapters, external force recording is necessary to make simulation in such cases more accurate; thus, we have instrumented a chair with multiple force sensors allowing measurement of main external forces.

While gesture-based input is becoming more popular, as an exper- imental task we have chosen aimed movements, as they are the most common type of input on touch surfaces, and in some cases even gesture- based tasks can be represented through a sequence of simple aimed move- ments. The task was a multidirectional Fitts pointing task, which is com- mon for HCI experiments and allows computation of throughput (bits/s) based on speed and accuracy of corresponding aimed movements [95]. The target setup is equivalent with respect to indices of difficulty and di- rections for each surface, which allows direct comparison of performance between surfaces.

Additionally, we have consolidated the data into a single dataset, TouchCorpus, and released it to the research community. This dataset includes the data from multiple processing levels of the pipeline: mo- tion capture, speed, accuracy, throughput and measure of quality of Fitts’ models, joint angles, joint moments, muscle forces and muscle activa- tions. The dataset is generated on both the frame level as well as the ag- gregated per-trial level. We hope that this shared corpus will contribute to the replicability of user studies, and allow comparison of findings from studies of individual surfaces in a single context.

Cameras

Force plates

Surface with

targets

Markers

Fig. 6.1: The experiment was carried out in a motion capture laboratory equipped with a special chair instrumented with force plates. Surfaces were emulated with cardboard. The targets were registered in 3D space and tracked during performance. Here a user is performing the task in the Tablet condition (seated). The inset shows an example of the multi-directional target setups used in the experiment.