Robotic User Interface for Interactive Telecommunication

This section reviews HCI/HRI studies on the user interfaces of computer-/robot-mediated communication. As this robot phone project focuses on RUIs, the review starts with an introduction to previous research on non-verbal interpersonal communication systems. The second subsection looks at the RUI controlling methods presented in the literature. Then the section closes with a brief discussion on how RUIs possibly contribute personalization.

3.2.1 Non-verbal Interpersonal Communication Over Distance

Computer-mediated communication tools have been introduced with many services, for examples, SMS (Short Message Service), email, IM (instant messaging), video call, blogs, and social networking applications (King & Forlizzi, 2007; Lottridge, Masson, & Mackay, 2009). While people mostly rely on verbal expressions to communicate semantic content, there have been non-verbal communication means, such as emoticons, emojis, and haptic vibrations, which were quite actively used or are in use. One of the more recent examples

1 More detailed review on telepresence systems is provided in Chapter 4.

of non-verbal communication tools is Apple’s animation messages that deliver touchscreen taps, sketches, and heartbeats (Apple Inc., 2016).

People use communication tools to exchange both informative contents and emotional feelings to each other. According to Tsetserukou et al. (2010), 20% of sentences in a text-based human conversation carry emotional expressions including joyful (68.8%), surprise, sadness, and interest feelings. Non-verbal tools may support similar functions. Li et al.’s study supports this idea from a RUI view: even a simple robot gesture are able to deliver emotional and semantic content, but situational context and facial expressions had much stronger impact (Li et al., 2009). Ogawa and Watanabe’s work developing an embodied communication system tackles similar issues (Ogawa & Watanabe, 2000). Their system, InterRobot, is a bidirectional mediating system that consists of a set of half human scale robots capable of motorized facial expressions and torso/arms gestures. My work of social robot mediator is along those lines: it regards verbal conversation as the primary means of interpersonal communication and uses physically embodied anthropomorphic movements to support the interactions.

A number of other approaches have been attempted in academia to build interpersonal telecommunication assistants that enhance emotional relationships between remote users e.g., couples in a long-distance relationship (King & Forlizzi, 2007; Lottridge et al., 2009;

Mueller et al., 2005; Werner et al., 2008). HCI and HRI researchers and designers have suggested expressive and tangible means of interpersonal communication including icons (Rivera et al., 1996), abstract graphics with animation (Fagerberg, Ståhl, & Höök, 2003, 2004), phonic signals (Shirazi et al., 2009), tactile vibrations (Werner et al., 2008), force feedback (Brave et al., 1998; Mueller et al., 2005), and RUI features (Sekiguchi et al., 2001) in zoomorphic (J. Li et al., 2009; Marti, 2005; Marti & Schmandt, 2005; Nabaztag, 2006), anthropomorphic (Sakamoto et al., 2007), or symbolic (J.-H. Lee & Nam, 2006; J.

Park, Park, & Nam, 2014; Y.-W. Park, Park, & Nam, 2015) forms. Studies have revealed that people are more engaged with a conversational process when they create messages with an interactive user interface (Sundström et al., 2005) and talked to a humanoid robot (Sakamoto et al., 2007).

Section 3.4 will continue this review on the paradigms of non-verbal human-human communication interfaces and provide more details of selected interface systems.

3.2.2 Robot Control in Socially Interactive Systems

Robot teleoperation provides a technical basis for interface systems that control remote RUIs. It has been extensively studied for applications such as space exploration, undersea exploration, and bomb disposal. Podnar et al. presented a telesupervisor workstation that consisted of vision/robot status monitoring displays and a set of controllers including a keyboard, mouse, and joysticks (Podnar et al., 2006). Goza et al. introduced a teleoperation system for an “arms on wheels” humanoid robot astronaut (Goza et al., 2004). Their system used a HMD (Head Mounted Display) for vision monitoring, 3D tracking sensors for arms control, optical gloves for hand/fingers operation, and foot switches for robot mobility control. Use of phones and tablet computers has been widely studied for remote robot control tasks since the late 2000s, mostly after 2010 (Gutierrez & Craighead, 2009; Chen et al., 2011; Panizzi & Vitulli, 2012; Lu et al., 2013; Parga et al., 2013; Oros & Krichmar, 2013; Sarmento et al., 2015).

Teleoperation paradigms are useful also for applications where a robot occupies a social role. A group of researchers in Japan frequently used Wizard-of-OZ methods to experiment the social aspects of robots, and examined how teleoperation is equipped for practical use of social robots (Sakamoto et al., 2007; Glas et al., 2008; Okuno et al., 2009). The pilot user interfaces of avatar-like telepresence robots once inherited the desktop workstation style design from teleoperation systems (O. Kwon et al., 2010; Kristoffersson et al., 2013) and now are compatible with touchscreen enabled handheld devices (Romo by Romotive, 2011; Double by Double Robotics, 2013; Beam Pro by Suitable Technologies, 2015) and motion tracking techniques (Nagendran, Steed, Kelly, & Pan, 2015; Tanaka et al., 2014).

Previous HCI and HRI studies have shown multimodal interface styles such as direct manipulation with/without kinetic memory (Frei et al., 2000; Raffle et al., 2004; Weller et al., 2008), audio-driven methods (Ogawa & Watanabe, 2000), and vision-based control (R.

Li et al., 2007) to support computer-/robot-mediated communication scenarios. The goals and requirements of robot control may be different between social intermediaries and space exploration robots. Ogawa et al. (2000) and Li et al. (2007) pointed that quick response and adequate accuracy to the user’s input are sometimes more important than precise estimation for avatar-like communication systems.

Section 3.3 provide a broader review of user interface techniques for animating human figures. Chapter 4 introduces the details of recently created telepresence robots. In Chapter 7, I will provide more comprehensive analysis of smartphone-based miniature telepresence robots.

3.2.3 Personalization

The term personalization is defined as a process that changes the functionality, interface, information content, or distinctiveness of a system to increase its personal relevance to an individual (Blom, 2000). Personalization has two aspects. On one hand, work-related personalization is elicited to enable access to information content, to accommodate work goals, and/or to accommodate individual differences. On the other hand, socially motivated personalization is initiated to elicit emotional responses and/or to express the identity of the user (Blom, 2000; Oulasvirta & Blom, 2008).

Personalization, also often described as customization, is initiated by either a system or a user, or sometimes both (Blom, 2000). An agent system deals with information to help work related personalization and lets the user confirm the change. A user may customize a work environment, use stickers, or change ringtones for different contact groups in his/her cell phone. Such activities not only give a product an identity and sense of lifelikeness, but also make a user feel an attachment to a product (Sung et al., 2009) or to a robot (Groom et al., 2009). Thus personalization activities may build long-term intimacy between the user and the product (Sung et al., 2009).

A RUI, if it is well designed, would be able to provide users with great opportunities for personalization. However, RUI has been less discussed than look-and-feel, functionalities, and screen-based pet applications (Tamagotchi, 1996) in the context of personalization in industry or in recent research. More discussions on personalization of social robots will be provided in Section 7.2.2.