E xperi mental Design

The experimental design tests the learnability, efficiency and effectiveness for all features of the 5 map applications (Pan and Zoom, Positioning, Three Search Bars and Click Event on Symbols) and

aims to determine the degree of satisfaction (from “Strongly Disagree” to “Strongly Agree”) with

each kind of online map. The three main sections of the questionnaire include: completing the given tasks to test each map feature, filling in the System Usability Scale (SUS) form to illustrate the satisfaction of their evaluated map and answering some questions about text-free map improvements. Participants were asked to:

1. Completed the following 5 tasks:

a. Find the shortest route between two given landmarks (Text map interface)

b. Find the corresponding symbols of three given landmarks (Text feature: three search bars)

c. Get your current location (Test feature: Positioning).

d. Find three given symbols through using the “Pan and Zoom” (Test feature: Pan and Zoom).

e. Determine the place name of three given symbols. 2. Complete the System Usability Scale (SUS) form.

3. Answer some open-ended questions about their experience using the online map. The full questionnaire is show in Appendix J. Participants were recorded and timed while they were completing the survey.

Each part of the map is required to be tested more than once to allow for verification of the learnability, efficiency and effectiveness of the map (e.g. Using Pan and Zoom; finding three specific

symbols which means that “Pan and Zoom” are being tested three times) (Jeng, 2005; Joo, 2011). The

evaluation of Learnability and Efficiency of maps is based on the time taken to complete tasks. The Learnability test is for discovering the capability of users to learn how to use the online maps. Each feature needs to be tested three times. The time taken for the first testing of finishing the task can be

measured as “Learnability” (e.g. when testing the learnability of “Pan and Zoom”, if participants finish finding the first symbol through the “Pan and Zoom” quickly, which indicates that the

learnability of “Pan and Zoom” functions is effective) (Joo, 2011). The efficiency is used to measure the ability of participants to use the map well without time waste. The speed for finishing the second

and third testing for the same task can be measured as the “Efficiency” (Jeng, 2005). The

|

1/1/201

6

identifying whether the participants can finish the task accurately or not (Joo, 2011)). In addition, in order to learn something about participants for more comprehensive results analysis, a total of 6 pre- questions are designed to determine their age range, their occupation, the frequency of using maps and online maps and their level of English. The detailed questionnaire is shown in Appendix J (The Chinese version (Appendix O) of questionnaire was designed as there are 28 Chinese included in the 30 Non-English Speakers group. A further 2 Non-English Speakers come from Japan and Korea and the questionnaire was read out to them), and the questionnaire was read out to the participants in the illiterate people group. The corresponding Data Collection (Task Recording) is shown in Appendix N.

The experiment took place in Albany, Auckland, New Zealand. All participants were required to finish the consent form before the experiment started to ensure willingness to take part in the experiment. The content of the consent form was read out to the illiterate people, and a Chinese version was provided to non-English Chinese speakers. In order to ensure all maps were evaluated by all groups of participants (illiterate people, non-English Speakers, Native English Speakers), each group was divided into five sub-groups and each sub-group was required to evaluate one kind of maps. Each participate has their own ID (1 ~ 90) and is randomly allocated into a small group:

TAB LE 15:ALLOC AT ION FO R P AR T IC IP ANT S TO EV ALU ATE O N LINE M AP S

Group A (Illiterate People) Group B (Non-English Speakers) Group C (Native English Speakers)

Participant No

Map Category

Participant No Map Category Participant No Map Category

1~ 6 Text with Symbol Only

1~ 6 Text with Symbol Only

1~ 6 Text with Symbol Only 7~ 12 Text with

Audio Only

7~ 12 Text with Audio Only 7~ 12 Text with Audio Only 13~ 18 Text-free with Symbol Only 13~ 18 Text-free with Symbol Only 13~ 18 Text-free with Symbol Only 19~24 Text-free with Audio Only 19~24 Text-free with Audio Only 19~24 Text-free with Audio Only 25~30 Text-free with Symbol and Audio 25~30 Text-free with Symbol and Audio

25~30 Text-free with Symbol and

62

To ensure this research was executed with the highest ethical standards, the entire survey process was approved through the ethical principles of Massey University. This requires that research procedures be appropriate to the participants. Researchers have a responsibility to recognize their own cultural location and to inform themselves of and take steps necessary to respect the social and cultural sensitivity of all participants. Meeting language preferences of participants in the provision of information was particularly important (Massey University, 2016) for our survey, as was ensuring that illiterate participants were fully informed. The study followed the Massey University Code Of Ethical Conduct.