Design Guidelines for Immersive Touch Workbenches

Through the implementation of the Interactive Slice WIM, we have arrived at a number of general guidelines that should be followed when developing techniques for use with Immersive Touch Workbench hardware. These guidelines are supported by the design decisions we arrived at through the implementation of the interactions that make up the Interactive Slice WIM tool.

Support New Complex Tasks via More Immediate Gestural Interfaces One of the most exciting opportunities provided by emerging touch technologies is the ability to explore styles of input that have not previously been possible (e.g., using many simultaneous touch points). In developing new interfaces, we should strive to go beyond replicating existing interfaces with touch wrappers and to instead enable new complex tasks with fluid and immediate gestures. For example, consider how different the inter- action described in section 6.1.1 would be to perform using 3D modeling software with a

mouse and keyboard. That process would take many iterations and multiple viewpoint adjustments to precisely create a curve. The immediate, real-time updates combined with physical action in the 3D touch interface are likely to dramatically decrease com- pletion time and accuracy. Similarly, the volumetric selection in section 6.1.2 allows not only quick 3D lassoing, but also fluid, immediate adjustment of the selection, qualities that are absent in current 3D visualization interfaces.

Use Smart Metaphors and Hardware to Avoid Hand-Occlusion Issues An important design challenge in immersive touch interfaces stems from the problem of breaking the stereo illusion when one’s hands block the view of virtual objects beneath. Some hardware-based solutions to this problem are possible (e.g., the multiscreen ap- proach of ITWs). Equally important is the use of smart interaction metaphors. For example, the slicing metaphors all use a shadow projection to ground the floating WIM to the horizontal table. Additionally, the visual elements and colors that depict the slice through the WIM are linked to the physical orientation of the screens, with red indicating a horizontal orientation and green a vertical orientation. The balloon tech- nique employed in section 6.1.1 is also a good example of a metaphor that helps to work around the problem of hands occluding stereo touches. By interacting with a “string” attached to a 3D object above the table, this interaction technique avoids the situation where the hands and object are collocated and has the advantage that the mapping between the hands and balloon is grounded to the physical touch surface.

Combine 2D and 3D Inputs and Visuals

One potential strength of immersive touch workbenches is their ability to easily mix and match 2D and 3D inputs and visuals. Too often we unnecessarily limit ourselves to one or the other when each presents its own strengths and weaknesses, leading us to advocate for a more holistic approach. For example, the Shadow Icon GUI system shows how traditional 2D GUI elements are adapted to the ITW workspace for effective usage. Similarly, the visuals employed in section 6.1.2 make use of the tabletop as a rich 2D organizational space that is linked to a detailed 3D view.

6.5

Conclusion

We have implemented and built upon the core Slice WIM navigation metaphor to de- velop a software tool we call Interactive Slice WIM. The Interactive Slice WIM tool introduces a number of new generalizable techniques for interrogating and querying volumetric data. These include techniques for:

• specifying curves and volumetric selections,

• creating persistent interactable 2D slices on the table surface,

• manipulating multiple 3D environments that exist above the tabletop, • performing radial measurements, and

• flying through volumes via animated camera paths.

These extensions offer many new possibilities for querying and exploring volume data, as demonstrated by the numerous specific scientific applications presented. We also in- troduced a new graphical user interface technique designed to work with the Interactive Slice WIM called Shadow Icons that allows for familiar software operations within this new hardware and software platform. Finally, we provided a discussion of the Interactive Slice WIM tool that includes expert evaluation and a set of design guidelines derived from the implementation. In the future, we are excited by the potential to expand upon these techniques and the new VR applications presented in this dissertation, specifically with the goal of providing not just views of data, but also new interactive engineering workflows.

Chapter 7

The Design by Dragging

Framework: Direct and Natural

Exploration of Simulation Design

Spaces

This chapter introduces an exploratory visualization system targeted toward overcoming the second limiting measure of big data: its many data instances. Specifically, we focus on simulation-based design problems through a system we call Design by Dragging. Paralleling the earlier description of the Slice WIM metaphor and implementation, this chapter describes an abstract overview of the Design by Dragging framework. The subsequent chapter provides a detailed technical description of our implementation of this framework, along with numerous examples and expert evaluations.

Simulations (e.g., computational fluid dynamics [CFD], finite element analysis [FEA]) are poised to play an increasingly important role in design processes within engineering, science, entertainment, and art. Unfortunately, despite their clear importance to many disciplines, current frameworks for working with simulations do not complement (and in many cases inhibit) the human process of creative design. In the medical device in- dustry, for example, the current consensus is that simulation-based engineering will be one of the most important keys to future innovation [136]. Nonetheless, working with

simulation tools often requires practicing engineers to adopt a completely different de- sign process. The major problems with today’s approaches to simulation-based design are twofold:

1. Simulation tools do not allow designers to understand, compare, and make deci- sions based upon alternatives in a large design space (i.e., a set of many related simulations).

2. Simulation tools require an often distracting focus on boundary conditions, param- eters, and other low-level details that in many cases are tangential to the designer’s goals and may be understood only by a trained expert in computational methods. Today, these limitations make simulation more useful as a method for detailed validation of a single design than for creative exploration of alternatives throughout the design process.

Outside of engineering, designers in other contexts (e.g., entertainment) face many of the same challenges in trying to appropriately fit physically based simulations into creative processes that rely upon human input. On one end of the spectrum of design tools, traditional paper and pencil and quick, sketch-inspired 3D modeling systems provide designers with the immediacy, expressiveness, and human engagement needed to enhance creative design. On the other end of the spectrum, detailed simulations, often run in batch mode, provide one or more sets of physically accurate data. There are few computational tools between these that can facilitate creative design along with new insight from data-driven simulation.

In this chapter, we argue that these challenges can in large part be solved through new computer graphics interfaces. Our work shares a common motivation with the early work in design galleries in setting simulation parameters for computer graphics and animation [67] and with a series of recent results that focus on visualizing large sets (ensembles) of simulation results (e.g., [68, 72, 137]).

Here we introduce a direct manipulation framework we call Design by Dragging. Our goal is to enable the designer to focus entirely on the design problem rather than the underlying details of simulations and to react naturally to visualizations of simulation results. For medical device engineers exploring the space of possible biopsy device designs, this means that they will sometimes want to drive the exploration in a specific

direction by adjusting a geometric parameter of the device, which we call forward design. In addition, they will also want to react to the visualization of simulation results, asking, for instance, “What causes this region of high stress, and what would it take to move this region to another location or remove it entirely?” We call this inverse design, since the path it implies through the design space is specified in terms of simulation outputs rather than inputs.

One of the defining features of this framework is that it supports direct manipula- tion for both forward and inverse design strategies within the same interactive computer graphics display. For forward design, we accomplish this through interfaces in the style of successful direct manipulation 3D modeling tools (e.g., [138]). For inverse design, we navigate and interpolate within a set of pre-computed simulations, displaying the best match based on the user’s input. To make the user interaction feel as direct as possi- ble, we introduce the idea of directly manipulating data visualizations, such as scalar stress fields output from simulations, in order to create a continuous descriptive visual query. The style of the direct manipulation is reminiscent of recent shape manipulation interfaces based on dragging or tugging metaphors [139, 140].

The major contribution of this chapter is a novel generalizable interaction framework for exploratory visualization of simulation-based design problems using direct manipu- lation, which includes the following:

• A new accessible approach to simulation-based design for tuning parameters of computationally intensive simulations and for visually evaluating large sets of results.

• An overview of our work on the Design by Dragging framework in terms of the pre-computation data structure and the interactive exploratory visualization loop. • A discussion of the design philosophy for systems implementing the Design by Dragging framework, including a set of design principles and guidelines to be followed when designing interactions in this framework.

• A description of the visual elements used in the framework and a specific example of a forward and of an inverse direct manipulation interaction.

7.1

Related Work

Our work builds upon recent results in direct manipulation interfaces, simulation-based design tools, and ensemble visualization.