Providing interactivity with touch gestures

Grounded cognition and embodied interaction research suggest that action can play significant roles in perception, acquisition and thought. Gibson (1979), for example claimed that action underlies perception and the ability to perceive evolved from a need to interact with the world. Barsalou et al. (2003) found that there is a correlation between one’s physical state and one’s mental state. Gestural interfaces in computing environment provide a more hands-on experience and therefore could support cognition and create meaningful learning experiences by using a more direct manipulation of objects (Segal, 2011). There is also a growing body of research based on embodied cognition theory which argues that physical manipulation of objects support thinking and learning (Bara et al., 2004; Glenberg et al., 2004; Siegle, 1996). Also, studies about technological devices and learning provide evidence that incorporating

4.4 Interactive problem-solving environment 88

the haptic channel yields better learning performance (Chan and Black, 2006; Hallman et al., 2009; Jang, 2010). These findings suggest that incorporating gestural actions in educational tasks may enhance teaching and learning efforts. A list of common gestures used in computing environments is presented in Figure 4.4.

Fig. 4.4 Basic touch gestures (Bank, 2015)

This research suggests that using multi-touch gestural interfaces can provide needed inter- activity in the assessment process. Multi-touch technologies present novel opportunity to increase the input bandwidth between humans and the computer that can aid capture of problem-solving steps. Advances in computing hardware technology following Moore’s law has led to higher processing speed, memory capacities, sensors and the number and size of pixels in digital cameras (Myhrvold, 2013). As a result, computers are getting smaller, cheaper and more capable of advanced functions. These developments are reflected in changes in the way humans interact with computers over the years. For instance the command line interface (CLI) was the primary way of interacting with the PC in the 1970s. The arrival of the graphical user interfaces (GUI) of the 1980s brought about the WIMP interfaces (windows, icons, menus and pointing devices) as an improved method of interac-

4.4 Interactive problem-solving environment 89

tion. In recent times the landscape is changing to natural user interfaces (NUI) multi-touch devices, voice control, gesture input and augmented reality (Wigdor and Wixon, 2011). van Dam (1997) described a Post-WIMP interface as one containing at least one interaction technique not dependent on classical 2D widgets such as menus and icons which ultimately will involve all senses in parallel, natural language communication and multiple users. The multi-touch technologies (handhelds, tablets, tabletops and whiteboards) are at the forefront of this revolution.

Multi-touch refers to the ability to simultaneously register two or more distinct positions of input touches on a touch screen or surface. The unique features and characteristics of multi-touch technologies present several advantages over traditional computers with mouse and keyboard. Some of these as highlighted by several studies are outlined below:

• Abstraction: Multi-touch removes abstraction from the interaction process. The direct manipulation of digital objects on the surface allows for working with virtual manipulatives directly instead of being mediated through another input device such as a mouse.

• Fun: The multi-touch environment provides a playful environment that engages users. Their interactive nature is engaging and appeals to a wide variety of learning styles (Heinrich, 2011). Skinner and Belmont (1993) argued that students who are engaged show sustained behavioural involvement in learning activities. Such students can exert intense effort and concentration in the implementing learning tasks.

• Parallelism: Multi-point interaction allows more parallel interaction. Multi-touch technology can capture multiple touches on a screen and convert these actions into events that can be interpreted by software. This permits performance of a complex task in a reduced time (Jiao et al., 2010). In the simplest case these might be mouse actions like mouse clicks and mouse drags, but because several touches can be combined into gestures it is possible for the user to give more information than just translations (that correspond to dragging the mouse) or (x,y) positions that correspond to clicking the mouse (Villar et al., 2009). For instance, with two fingers it is easily possible to rotate or scale or move several objects at once, each in a separate direction. With more fingers more degrees of freedom are available to input almost arbitrary transformations. This results in a reduction of task switching times.

• Collaboration: The split of concurrent touch points between multiple users makes it possible to create learning environments using large screens (for example multi-touch-

4.4 Interactive problem-solving environment 90

tables), which encourage collaborative learning and communication (Don and Smith, 2010; Mercier and Higgins, 2013).

• Easier and faster: Multi-touch provides a good illustration of Fitts’s Law (Fitts and Peterson, 1964). The law states that the time needed to move to a target area is a function of the distance to and the size of the target. In effect, it means that big icons are easier to hit than little ones, top-of-screen menus are easier to click on than top-of-window ones, and pop-up menus are faster than pull-downs

• Intuitive and natural: The interfaces support intuitive and natural interactions and allow for rough motor skills and imprecise manipulation. These, in combination with the visually appealing interface elements and graphics, lead to an increased overall user experience (Moscovich, 2007).

• Bi-manual interactions: The multi-touch favours two-handed interaction which is common in the physical world (Bailey and Garner, 2010). Don and Smith (2010) showed that it allows for bimanual text entry.

However, multi-touch technology does have some challenges. Several authors have pointed out these limitations. For instance Villar et al. (2009), noted that the playfulness of the interface encourages ephemeral interactions. Jacucci et al. (2010) illustrated the limitation to the implementation of full gestural language; he argued that virtual objects in the 3D space require six degrees of freedoms to be manipulated in full detail (i.e. translations and rotations in all three dimensions). In contrast, the multi-touch input is sampled from a 2D surface, giving only 3 degrees of freedom (translation in the x/y dimensions and rotation around the z-axis). Other authors have argued that the interfaces lack the precision usually afforded by indirect pointing devices (Hansen et al., 2009), are inconsistent across different manufacturers (Kammer et al., 2010) and can result in task complexity (Davies, 2010). Some activity theorists have argued that conscious learning and activity are completely interactive and interdependent (Jonassen, 2000). In this thesis, we argue that multi-touch gestures can be used to to perform arithmetic work intuitively. Numbers from the word problems (see Fig. 4.5) may be further utilised for the solution by using multi-touch gestures to indicate intentions. In primary schools’ operations and algebraic classes, for example, addition is taught as putting two numbers together and adding. Likewise, subtraction is often modelled as taking apart and taken from (Dolan et al., 2012). According to Resnick et al. (1998), it is vital that students have many varied experiences building number sentences (equations) through the use of concrete manipulatives. This may incorporate the tactile, visual, and abstract experiences and assist in developing conceptual understanding. A multi-touch computing environment provides varied opportunities for these. Figure 4.5 illustrates how

4.4 Interactive problem-solving environment 91

addition and subtraction may be modelled in using digital manipulatives and multi-touch gestures. Intuitive models of multiplication and division may also be conceptualised and implemented.

Fig. 4.5 Multi-touch gestures for arithmetic work

Because a multi-touch environment provides increased opportunities for expression through direct and intuitive touch gestures, it may be useful in computer-aided assessment envi- ronments, problem-solving steps can be made explicit and captured as they can enable the decomposition of problem-solving steps. Touch gestures

• Allow direct and intuitive expression

• Engage the student in problem solving

• Provide opportunities for decomposition of steps

• Allow for sequence detection

• Provide a non-intrusive method of obtaining assessment data

• May provide opportunities for collaboration in problem solving

4.4 Interactive problem-solving environment 92