Description of Data

Our surface database and hardware to create it were designed with two end goals in mind: first, to have a dataset spanning many common material categories and ranges of haptic properties, so that robots of varying haptic awareness could have a wealth of vicarious experience to use in evaluating haptic percepts from their environment; and second, to help develop and test machine learning algorithms to explore the relationship between multiple surface perception modalities. To fulfill these goals, the database will have to contain high- fidelity representations of its surfaces, as well as be easily accessible to researchers so they can integrate it with other systems or federate it with additional datasets. These criteria drove our apparatus design and surface search, and continue to motivate the presentation in this chapter of the data along with ways to parse and visualize it.

Not every mobile robot possesses a dedicated haptic sensor. Of those that do, there are many different types. Some robots may have force sensors as part of grippers, or over a region of “robotic skin,” while others could use flexible sensors like the SynTouch BioTac or the newer GelSight. Still others have only visual sensors, or other modalities that can be repurposed to provide some level of haptic sensing (e.g., a microphone or an accelerometer as part of anIMU). This diversity of sensors makes sharing of haptic data between different research groups less valuable than sharing the computer vision data discussed above. In envisioning our visuo-haptic database, we incorporated visual sensors and various types of

haptic sensors in order to collect diverse (yet matched and time-synchronized) data about each surface.

The surface recordings are episodic. For each surface, we continuously recorded data for about one minute with each end-effector, while several sensors were always on regardless of end-effector (see Sec. 5.1.2 on the next page for more details). All data points are times- tamped against the same clock, which means all the sensor readings can be played back in correct temporal relation to each other (see [71] for some discussion of the importance of timestamps, though we avoid needing to use their TICSync algorithm by using a single clock). Thus, in every episode we have a baseline measurement of the surface, matched to varying haptic end-effectors. In this chapter I will present the precise procedure we used to collect the data, describe the storage format, and provide sample code for researchers wishing to use the data.

5.1.1. Surface Criteria

A useful database will have a wide variety of surfaces included, measured both by material type (e.g., wood, plastic, metal) and variance across values of relevant haptic properties (such as hardness, roughness, friction coefficient, and thermal conductivity). In addition to this criterion of variety, we constrained our search space in several ways in order to keep the dataset coherent and for practical considerations. We sought flat, texturally homogeneous surfaces which would not damage, or be damaged by, our end-effectors. Surfaces that would not damage the end-effectors were dry and not so rough or sticky that they would abrade or tear off the skin of the BioTac (this being the most delicate of the three end-effectors). On the other hand, we avoided granular materials, like sand, which would change shape after being touched by an end-effector. A few surfaces were not scanned with all three end-effectors because they were either so delicate that the tooling ball would leave tracks, or too abrasive to scan with anything but the tooling ball.

Figure 5.1: An experimenter holds the Proton, equipped with the OptoForce end-effector, in contact with a desk surface (repeated from Fig. 2.4).

Some of the surfaces we scanned were fixed to the environment, such as floors and tables. Others were movable objects or fabric samples. We clamped down movable surfaces during data collection. In the case of thin fabrics or paper-like materials, securing them to a surface means that there was a hard backing, transferring some haptic properties through the surface. This backing may affect the relevance of our measurements of those materials when they are encountered in other contexts (e.g., clothing on a human body or an empty cardboard box).

5.1.2. Collection Procedure

Our database consists of recordings made by several human operators, all with the same instrument, the Proton. The surface episodes were recorded in batches: we performed a series of recordings with one end-effector, and then we went back and scanned each surface again (not always in the same order) with the other end-effectors. We made an effort to ensure that surfaces were observed under adequate lighting in a relatively quiet environment, but given that the surfaces were scanned in different places, inside and outside, over a period

of many months, environmental conditions such as light levels, temperature and humidity were not held constant.

In a typical data collection episode, the operator holds the Proton with two hands, one holding the base and the other holding the handle halfway between the base and the cameras, as shown inFig. 5.1on page90. The fiducial marker frame (seeFig. 4.10on page67) is placed on the surface under study (sometimes clamped or manually held down if the surface is not fixed in place). The end-effector tip is brought into contact with the surface and moved back and forth inside the marker frame, in both straight lines and curves, deliberately varying the speed and normal force. The operator will also tap the end-effector on the surface several times with varying impact speeds. Fig. 5.2on the following page shows the traced 3D position of the end-effector during a typical episode. Some of the sensor data streams from that same episode are shown inFig. 5.3 on page93.

To give an idea of the parameters of data collection,Figs. 5.4 and 5.5on page 94 show the distribution of normal forces, tangential speeds, and impact forces across the entire dataset. They are separated by end-effector, which is the most significant source of variation (the three end-effectors are made of different materials and therefore have quite different friction characteristics when interfacing with most surfaces). In particular, it is clear that lower impact forces were used with the BioTac end-effector, which is unsurprising as it is a soft, delicate and expensive instrument. Of course, these are averages and we expect the force and speed distributions to vary somewhat between operators, and surfaces (for example, operators will instinctively use slower speeds and lower forces on high-friction surfaces). However, since we measure these parameters, we can use them in analysis, either directly as features or as a normalization factor.

5.1.3. Surface Sources and Locations

We designed the Proton to be portable and self-contained so that it would be possible to collect data in the field from a variety of sources. Some previous surface datasets are

(a) 3D view (b) Top view

(c) Side view (d) Side view

Figure 5.2: Trace of end-effector position during a typical surface recording. The majority of the data is planar dragging, with several tapping motions above the surface. The color gradient shows the progression of time, from start of data collection (cyan) to end (pink).

0 10 20 30 40 50 60 70 Time (s) -50 0 50 100 150 200 250 Position (mm) X Y Z

(a) End-effector position

0 10 20 30 40 50 60 70 Time (s) -20 0 20 40 60 Force (N) X Y Z (b) End-effector force 0 10 20 30 40 50 60 70 Time (s) -15 -10 -5 0 5 10 15 Acceleration (m/s 2 ) X Y Z (c) End-effector vibration

Figure 5.3: Data gathered by the (a) onboard motion tracking, (b) ATI Mini40 six-axis force/torque sensor, and (c) high-bandwidth accelerometers during the same typical surface recording shown in Fig. 5.2on page 92.

0 10 20 30 40 50 60 Normal force (N) 100 101 102 103 104 105

Log bin count

(a) Tooling ballFN

0 10 20 30 40 50 60 Normal force (N) 100 101 102 103 104 105

Log bin count

(b) OptoForceFN 0 10 20 30 40 50 60 Normal force (N) 100 101 102 103 104 105

Log bin count

(c) BioTacFN 0 5 10 15 20 25 30 Tangential force (N) 100 101 102 103 104 105

Log bin count

(d) Tooling ballFT 0 5 10 15 20 25 30 Tangential force (N) 100 101 102 103 104 105

Log bin count

(e) OptoForceFT 0 5 10 15 20 25 30 Tangential force (N) 100 101 102 103 104 105

Log bin count

(f) BioTac FT 0 5 10 15 20 25 30 Tangential speed (cm/s) 100 101 102 103 104 105

Log bin count

(g) Tooling ballvT 0 5 10 15 20 25 30 Tangential speed (cm/s) 100 101 102 103 104 105

Log bin count

(h) OptoForcevT 0 5 10 15 20 25 30 Tangential speed (cm/s) 100 101 102 103 104 105

Log bin count

(i) BioTacvT

Figure 5.4: Histograms of end-effector force and speed, decomposed into normal and tangen- tial components, over the entire dataset. The first and second rows show the distribution of normal and tangential forces exerted throughout data collection, separated by end-effector, while the second row shows the tangential speed while dragging.

0 10 20 30 40 50 60 70 80 Impact force (N) 100 101 102 103

Log bin count

(a) Tooling ballFI

0 10 20 30 40 50 60 70 80 Impact force (N) 100 101 102 103

Log bin count

(b) OptoForceFI 0 20 40 60 80 Impact force (N) 100 101 102 103

Log bin count

(c) BioTacFI

associated with a set of material samples stored in one place, but this can limit the generality of its contents. For maximum diversity of surfaces and data collection conditions, we aim to collect data from surfaces in their natural environment. To be sure, many of the surfaces in our database are portable, but not all. In particular, we have drawn upon the materials in the Penn Haptic Texture Toolkit [48], the LMTmaterial collection [61],1

the installation of materials from Material ConneXion [72] at the Princeton University engineering library,2_and

a few flat-surfaced objects from the YCB object set [68]. Several other portable materials were purchased from the bookstore at the University of Pennsylvania (Philadelphia, PA, USA) or collected3_{at the Max Planck Institute for Intelligent Systems (Stuttgart, Germany).}

Finally, substantial data was collected from surfaces fixed in place at the University of Pennsylvania as well as several private homes in the greater Philadelphia area. Further details of the precise locations are given in Appendix Bon page 167.

5.1.4. Limitations

There are several limitations inherent in this dataset, mainly related to the amount of time required for data collection, as well as the difficulty of finding a wide variety of surfaces and the difficulty of controlling environmental factors while collecting data with a large number of sensors.

For about half of the surfaces, we do not have data with all three end-effectors. In some cases, the omission was deliberate because the surface was too rough to use with the comparatively delicate OptoForce and BioTac sensors, or conversely, so soft that the steel tooling ball would have damaged it. In other cases, this lack of coverage results from omitting data that was low quality due to poor lighting or sensor malfunction.

We further have some surfaces that are included in the dataset, but missing data from certain sensors because they were not working at various times during data collection: namely the microphone and the Structure Sensor. In particular, the manufacturer-provided Structure

1

The authors acknowledge Matti Strese for providing on-location access and an extended loan of materials.

2

The authors also acknowledge Willow Dressel for facilitating access to this collection.

Sensor driver was unreliable, such that it often stopped reading frames in the middle of data collection, and interfered with the OptoForce sensor.