Performance and Qualitative Evaluation

In a study we tested the performance of the presented system regarding overall recognition time and error rate. We further wanted to see whether participants understand the selection. We wanted to gain insights into the acceptance and usability of such systems. However, we did not compare

the system against existing solutions due to the increased selection time caused by the image transmission and analysis. In terms of performance, we present the outcomes only for selecting remote content. The interaction and manipulation with it served as basis for qualitative feedback. During our test, we showed 12 different images on the display (see figure 5.10). The visual repre- sentation of these images differed with at least 20%. This difference is measured by subtracting one image from another and subsequently building the ratio of different pixels to overall pixels. Pixels are considered different if they differ in terms of their grayscale value with more then 25 units (out of 255). Although the pictures showed different objects, they may be equal from the computer’s point of view in terms of structure and color. This increases the risk of detecting the wrong image, but also gives insights into the performance of our algorithm. Each picture was placed with its neighbors in a way that theoretically each item can be identified uniquely. We logged the pure processing time (i.e., the time needed for both image and result transmissions were excluded) as well as the error rates. Selection errors can have two characteristics: first, the system is not able to detect anything asking the user to select the item again. And second, the algorithm detects the wrong item requiring the user to go back and select the item again. We did not distinguish between these errors as they both lead to the same result of having the user to re-select the intended content.

Figure 5.10: A person using our prototype on a large display.

We recruited 28 participants (14 were female) ranging in age from 18 to 47 years (average age was 25.2). 25 participants own a mobile phone, but only 15 of them use the built-in camera. During the test, we asked participants to capture each picture in a randomized order. If the picture was the correct one, they could go back and select the next one. If the system was not able to detect a picture (or it detected the wrong one), we increased the error count and asked the participants to select it again. The processing times of failed selections were not further considered when the

5.3 A Proof-of-Concept Implementation 103

system was not able to identify anything. This is because a missed detection of an item could lead to extremely short detection times as the contour detection might have failed already. This would cause the system to stop even before any image comparison mechanisms (which we consider the bottleneck) would take place. Missed detections of the mentioned nature would result in an overall lower processing (and shorter selection) time. After correctly selecting each image, participants had to fill a post-questionnaire also stating their opinion about the system.

In terms of performance we found that our system was reasonably well suited for short interaction sessions such as selecting content. The average detection time of selected content was about 180 ms. The error rate was about 4% (i.e., only 4 out of 100 taken pictures resulted in a wrong detection). Most of the errors (about 90%) resulted in not detecting the item at all. This was mainly caused by the slight movement of the mobile device while selecting the content (i.e., pressing the ”select” button). The taken image was then affected by motion blur making the detection of contours complicated due to the missing contrast. Delivering wrong results (i.e., the system favors another item), on the other hand, was mainly caused by bad lighting conditions such as flickering occurring in fluorescent lamps. While having a noticeably low error rate, we acknowledge that increasing the number of pictures on a display (and potentially lowering the pair-wise difference of them) would result in both higher processing times and higher error rates. For numerous items as well as real-time interactions, the approach needs to be revised.

In post-questionnaires users ranked the overall use of the system as well as its ease-of-use. In addition, we encouraged them to give further feedback about our prototype. On a five-point Likert-scale (1 equals ”I dislike the system” and 5 equals ”I like the system”) the average rating was 4.1 indicating that the system was appreciated among our participants. When asking whether they would use such a system in public spaces, we found that 82% (exactly the same as partici- pants that liked the overall idea) would like to use it. During the test, the Bluetooth connection time was a large portion of the overall selection time (i.e., connecting, selecting, sending, and

receiving). The average time for a successful selection was 9.2 seconds although the actual con-

tent recognition only took 180 milliseconds. While we thought that this time would decrease the ratings, participants stated that a selection time of 10 seconds still meets the criteria for inter- acting during a short period of time. This is also in line with Rukzio et al. [RSH04]. The most interesting statement regarding our system was that participants felt comfortable in using the prototype regarding privacy. They stated that although everybody seesthata person is accessing information, no one can actually seewhichinformation has been captured. Our system and the underlying concept of interactingthrough the display may be a first step towards more private interactions on external displays by allowing users to select distant content on their local display.