We have described the state of the art of fields of HCI input method design, physical ergonomics, and biomechanical modeling and simula- tion in the aspects related to this thesis. Each field has own tasks, re-
search methods and models, but this work advances all fields by provid- ing bridges between them and advancing each field by methods from the others.
The largest is the contribution to HCI input method design: we adapt and validate biomechanical simulation for HCI goals, tasks and settings, which makes feasible detailed analyses of gestural and mid-air interfaces to accelerate their development and wide adoption. We show the ef- fectiveness of biomechanical simulation on analysis of touchscreen de- vices and the added knowledge compared to traditional methods. Our movement space summarization provides quick access to biomechanical properties of mid-air aimed movements based on movement location and orientation. The summarization can be used by input method designers without much prior knowledge in biomechanics, kinesiology or physiol- ogy, while knowledge of anatomy is still necessary to understand areas under load in the human body.
The contribution for the field of physical ergonomics is similar, but not as significant as for HCI, as ergonomists usually have better knowledge of human physiology and biomechanics, and while they were not able to perform such detailed analysis, the movement summarization may have less value for them.
The contribution to biomechanics is two-fold: we validate biomechani- cal simulation with an upper extremity model for new types of movement tasks and, by creating the simulation pipeline, we make biomechanical simulation more user-friendly and accessible to a wider range of users. In particular this aspect is important for doctors and practitioners who have knowledge of the human body, but lack technical knowledge to run biomechanical simulation, tune optimization parameters or adjust mod- els.
The HCI Biomechanics Pipeline
3.1
Introduction
Following the overview of modern ergonomics and biomechanics meth- ods in the previous chapter, we answer Research Question 1.1 in this chapter by adapting and framing the motion capture-based biomechani- cal simulation as a method suitable for the HCI field.
As already described, user performance and physical ergonomics are two key characteristics of input methods defined during the design pro- cess. A usable input method satisfies both these aspects by allowing high throughput (high words per minute for typing, fast target selection and menu navigation, etc.) and necessitating low ergonomics cost (postures closer to neutral, small joint and muscle loads, low energy expenditure and fatigue). For post-desktop input methods the problem of perfor- mance and ergonomics assessment becomes particularly hard due to such issues as lack of previous knowledge, large input space and its non- uniformity with respect to performance, and complexity of ergonomics analysis using traditional methods.
As presented in the overview of modern methods deployed in rele- vant fields, the most promising data collection method for our purposes is optical motion capture in combination with biomechanical simulation. While optical motion capture data is perfect for movement performance analysis (end-effector velocity, movement time, Fitts’ law, throughput, etc.), it also serves as input to biomechanical simulation, which pro- vides physical ergonomics variables for the corresponding movement. The computations are executed with a generic musculoskeletal model as a prior and generate joint angles, joint moments, muscle forces and activations.
The proposed analysis pipeline collecting and integrating both per- formance and ergonomics is shown in Figure 3.2. The pipeline is im- plemented in Matlab and includes the OpenSim biomechanical simula- tor [17] to generate biomechanical data. The experiments need to be performed in a motion capture laboratory and necessitate only slight adaptations to typical-for-HCI experimental setups and procedures. The preprocessing typical for motion capture data is applied to remove ar- tifacts. This data is used further to derive performance characteristics, and in parallel it serves as the main input to biomechanical simulation. The resulting performance and ergonomics indices are synchronized and registered back in 3D movement space together with representation of experimental setup, namely the 3D targets.
The final dataset broadly covers both performance and ergonomics with more than 400 variables (for the upper extremity musculoskeletal model). Any movement can be analyzed and compared against others with respect to various types of indices, and at different levels of granu- larity, from frame level, to aggregates per movement and per movement type. In contrast to traditional methods, the richness of the dataset makes it possible to define the scope of the analysis a posteriori, rather than before the experiment. This gives the researcher additional advantages and flexibility in the search for general trends and anomalies. Another big advantage of the proposed method is the possibility of joint analysis of performance and ergonomics. This is the first method which tightly integrates both measures and makes it possible to systematically assess trade-offs between them, which would be very valuable for post-desktop interface designers and researchers.
Creation of the dataset is not the last step of the proposed analysis pipeline. To support the practitioners in analysis of such multidimen- sional multi-factor data we, together with our collaborators, have devel- oped an interactive visualization tool. This tool makes it possible to ex- plore different facets of the data using individual, most intuitive visu- alizations for them; for example, it supports 3D trajectory visualization to analyze end-effector kinematics; muscle visualizations to analyze re- cruitment, loads and energy expenditure; or task-specific visualizations relating indices of the data with spatial characteristics of the task. The tool supports joint analysis of the data across all three facets: perfor- mance, ergonomics, and task-specific movement characteristics; for ex- ample, it is possible to select desired movements in 3D space, then an- alyze their properties in ergonomics space (joint angles and moments,
cam $ cam $ markers$ physical$targets$ end2effector$
Data Collection Setup Generalized human
virtual$ markers$ muscle$ ac7on$ lines$ virtual$ target$ muscle$ wrap$ surfaces$ joints$ bones$
Fig. 3.1: Terminology and setup for data collection. Markers on the human body are mapped via anatomical landmarks to a generalized model of the human. Physical tar- gets are also registered in the virtual 3D space.
muscle forces and activations), or performance space (speed, accuracy, throughput). The tool is suitable for typical HCI decision-making tasks: validation, exploration and planning.