3. RESEARCH PROCESS AND METHODS
3.5 User testing method
The qualitative data is draw upon from evaluation process. A background questionnaire is given to the participants for their participation experience and preference of game. A set of tasks are prepared to ask the participates to perform within both the gamified service and non-gamified service. The service provides three types of interaction for participation: submitting feedback, answering survey, and interacting with others (only in the gamified version). All the three interactions are evaluated with given tasks. In addition, other features implemented such as personal profile and submitted history in the prototype are evaluated. There is also a task related to virtual market and planting simulation designed in the gamified prototype. The planned tasks are presented in Appendix G.
3.5.1 Comparative test
For a more intuitive insight and understanding of gamification effects in youth participation, the user testing is conducted as a comparative test between the gamified prototype and the non-gamified prototype. The control variable is gamification. The participants are given these two prototypes to evaluate their attractiveness and intention of use. The comparison performed in the process of conducting User Experience Questionnaire (UEQ) and interviewing. With UEQ, the pragmatic and hedonic quality can be compared with intuitive data. The process of interview aims to collect details of preference of two versions of prototypes. With a control testing object, it is expected that the participants would have direct comments of motivation and attractiveness from gamification.
3.5.2 User Experience Questionnaire (UEQ)
While observing the participants to perform the given tasks, the overall usability and user experience of designed prototype can be understood roughly. For a broad and measurable view of pragmatic and hedonic aspects of the gamified prototype, the User Experience Questionnaire (UEQ) developed by Schrepp et.al (2017) is conducted (see Appendix H).
UEQ aims to inspect UX of the interactive product with a benchmark quickly and directly. The questionnaire consists of 26 items grouped into 6 scales, and each item consists of a pair of opposite terms:
- Attractiveness (6 items): annoying / enjoyable, good / bad, unlikable / pleasing, unpleasant / pleasant, attractive / unattractive, friendly / unfriendly.
- Perspicuity (4 items): not understandable / understandable, easy to learn / difficult to learn, complicated / easy, clear / confusing.
- Efficiency (4 items): fast / slow, inefficient / efficient, impractical / practical, organized / cluttered.
- Dependability (4 items): unpredictable / predictable, obstructive / supportive, secure / not secure, meets expectations / does not meet expectations.
- Stimulation (4 items): valuable / inferior, boring / exciting, not interesting / interesting, motivating / demotivating.
- Novelty (4 items): creative / dull, inventive / conventional, usual / leading edge, conservative / innovative.
The scales influence the user’s impression of the product, and the effect from each scale is not assumed to be independent. Each item is rated on a 7-point Likert scale. The value is ranged from -3 (totally meets the negative term) to 3 (totally meets the positive term). The relationship between the 6 scales and product quality is presented in Figure 11.
Figure 11. Assumed scale structure of the UEQ (Schrepp et.al., 2017, p.41) Based on the data from previous UEQ evaluations, a benchmark is created to judge UX quality of the evaluated product in general. However, the benchmark has the limitation of non-distinguish types of evaluated product. The benchmark is explained as below (Schrepp et.al 2017):
- Excellent: The evaluated product is among the best 10% of results.
- Good: 10% of the results in the benchmark are better than the evaluated product, 75% of the results are worse.
- Above average: 25% of the results in the benchmark are better than the evaluated product, 50% of the results are worse.
- Below average: 50% of the results in the benchmark are better than the evaluated product, 25% of the results are worse.
- Bad: The evaluated product is among the worst 25% of results.
Since the experience perceived by the user is highly subjective, UEQ provides a simple and efficient method for the user to measure and indicate their feelings and impressions of the evaluated product. (Schrepp et.al 2017)
3.5.3 Gamification inspection
The semi-structured interviews are organized for explore the subjective feelings and thoughts of using the service with gamification elements from the participants. The participants will be asked about personal preference of gamified service or non-gamified service. General motivations of the participants are assessed with the open-ended questions, such as “What would motivate you to use the public service for participation?”. In addition, the participation will answer the question about the intention of continued use and voluntary participation.
To inspect the match between the specific motivation and game elements, 13 statements in post-questionnaire based on gamification components need the participants to identify. The questionnaire aims to validate the perceived motivation from gamification. The statements target to the game design elements used in gamified prototype, thus, it is a progress of validating the perceived motivation from gamification. The statements related to different game elements are described as below:
- Achievement:
1. I can get positive feedback with virtual rewards. 2. The points I got give me a sense of competence. 3. I feel satisfied when I earn the badges.
4. I like to the challenge from missions/quests/tasks. - Social:
1. I feel connected to peer in my community.
2. I like to explore others’ ideas and find the people with similar thoughts. 3. I need social interaction with others for communicating and supporting. - Immersion:
1. I like to choose the avatars and characters based on my preference. 2. I enjoy virtual world.
3. It is fun to purchase virtual goods with virtual currency. 4. I like to feed virtual pets or planting.
In addition, there is one statement related to contribution: “I would like to contribute in city environment improvement” and one related to general pleasurable impression: “Using the service is fun and pleasurable”.
3.5.4 Semi-structured interview
The semi-structured interview is the final step of user testing in this research, which is used for a qualitative analysis. Some open questions are prepared to collect the detailed opinions from the participants:
- Do you prefer the gamified version or non-gamified version? Why? - What game elements do you think attract you to use mostly? Why?
- Would you like to use the gamified service in the future? In what kind of situations? - What do you think is the most important factor that motivate you to participate in your
city improvement?
- Do you have any suggestions for improving the gamified service?
The questions focus on the preference of gamification, intention of use, and potential improvement. However, there is freedom of adapting questions in the actual interview process and allowance for the participants to bring up new ideas.