8.3 User-based testing
8.3.5 SUS test results
I tested all six people. SUS score: 98 which corresponds to the adjective excel- lent, Learnability: 100, Usability: 97.5.
8.4 Results and discussion
The general acceptance of the application was very good. Users were able to find gestures described in the tasks. Further comments included:
”Nice image of the robot”, ”great idea of the application”, ”I may use it in my job”. The remote procedure testing worked very well. The sound was good and the users gladly participated in research. Only once I had a glitch with Skype connec- tion. One user turned out to be not fluent in English but in the end we were able to understand each other and she also gave positive feedback for the user interface. Each session took about an hour.
I think that with this test I proved that it is possible to implement the whole user- centered design cycle to the team of one person. Sometimes it was not easy to switch hats between creative work and coding, but introducing more strict scheduling helped. I also tried to put on different cloths to make it easier to switch between roles. SUS score went up, compared to the results for Gestunary for iOS score and Hi-Fi prototype test. I think that even though SUS is considered a good measure of usability, in my case the score shows very slight grades of change. Users liked the application, the robot picture. It seems that this also affected the judgment.
Chapter 9
Additional research
In this chapter I present results of gesture illustrations preference research. I also present a feasibility study of gesture search using machine learning methods.
9.1 Gesture illustrations preference research
In this section I present results of research, dedicated to checking user pref- erences for gesture illustrations. The choice of illustrations include drawing, color and black and white photos. All photos are presented in two ways: fully focused and blurred. Blurred photos have an important part of the gesture in focus. I also present pictures which are used in the survey.
Motivation
I wanted to check Epstein’s hypothesis [11] about the order of user picture prefer- ences for gesture illustration. The highest preference Epstein gives to drawing. The next preference is black and white photos. The least preferable are color photos. In his own work, Epstein uses sketches to illustrate gestures in An American Gestu- ary [10]. McNeill also considers drawings superior to video stills. At the same time, he mentions that drawings should be created with ”intelligence and sensitivity” [34]. An additional motivation was to test user preferences acquired during user-based testing of the Gestunary for iOS (3.3.4). Three people out of seven expressed neg- ative emotions towards blurred illustrations of gestures.
As the method for testing user preferences in gesture illustrations I choose sur- vey. Surveys are a very popular cheap research method to understand stakeholders and cover a wide audience. Surveys are good at getting shallow data, but bad at getting detailed information. According to Lazar [24], survey is a convenient method to quickly get many responses.
The goal
9.1. GESTURE ILLUSTRATIONS PREFERENCE RESEARCH
of gesture images. Gestures are represented by 5 different types of images:
• hand drawing by pencil • black and white photograph
– Untouched
– The face and most parts of the photograph is blurred. Parts of the body which participate in the gesture are in focus.
• color photograph
– Untouched
– The face and most parts of the photograph is blurred. Parts of the body which participate in the gesture are in focus.
Test gestures
The number of gestures to test the hypothesis was chosen to be 5. The number of questions was used to not overburden the user with questions and to ensure a higher response rate.
Illustrations
The drawn pictures were partly traced from photographs. Photos were used from the depositephotos1 photo bank and from the Gestunary photo library.
The target audience
The target audience is any person from 12 y.o, who has experience in communi- cations with different cultures, or who has traveled abroad. To filter out people, who do not have experience in communication with other cultures, I added a question on travel experience of the user [43]. I assumed that this question will tell us that the person had experience in communication with people from other cultures. I dis- tributed the survey among people who traveled a lot. I also published the survey on a few web forums23 which are attended by people from different countries, including
big countries, such as USA or Russia.
In Table 9.1 I present the contents of the survey with questions and illustrations.
1http://depositephotos.com 2http://puttylike.com 3
9.1. GESTURE ILLUSTRATIONS PREFERENCE RESEARCH
1. The pictures below illustrate a Spanish gesture which means ”small”, ”a little”. In your opinion, which one is conveying the gesture meaning in the best way?
1. 2. 3.
4. 5. 6.
2. The pictures below illustrate a German gesture which means ”crazy”. In your opinion, which one is conveying the gesture meaning in the best way?
1. 2. 3.
9.1. GESTURE ILLUSTRATIONS PREFERENCE RESEARCH
3. The pictures below illustrate a Japanese gesture which means ”OK”, good. In your opinion, which one is conveying the gesture meaning in the best way?
1. 2. 3.
4. 5. 6.
4. The pictures below illustrate an American gesture which means ”time- out”. In your opinion, which one is conveying the gesture meaning in the best way?
1. 2. 3.
9.1. GESTURE ILLUSTRATIONS PREFERENCE RESEARCH
5. The pictures below illustrate a Russian gesture which means ”silence”. In your opinion, which one is conveying the gesture meaning in the best way?
1. 2. 3.
4. 5. 6.
6. Have you traveled abroad or met people from other cultures? (YES/NO) 7. Please share any comments you may have.
Table 9.1: Gesture illustration survey