Task 3: Report water bubbling out of the ground

In the third task, participants were presented with the scenario: “You have just seen water bubbling out of the ground at 1 High Street. Please report this problem to Makana Municipality”.

Again, the task consisted of two steps:

1. First, participants were instructed to report the issue to Makana Municipality at the provided address (1 High Street). They were asked to also include a photo of the is­

sue being reported. To simulate participants discovering this issue, they were instructed to take a photo of a printed picture which depicted water bubbling out of the ground.

2. Once their report was successfully submitted, participants were instructed to return to the application’s Home screen.

Similar to Task 2, no results were presented to participants. After completion of task 3 parti­

cipants were asked to move on to the final task, Task 4.

7.5.4 Task 4: Change application language

Task 4 aimed to determine how easily participants were able to change their preferred language within MobiSAM and MobiSAM Report, and consisted of the following steps:

1. Starting at the Home screen, participants were asked to change the application language from English to isiXhosa.

2. Participants were then instructed to navigate back to the Home screen, now presented in isiXhosa.

7.6. TASK PERFORMANCE MEASURES 143 Following the Android design pattern, changes were applied automatically, therefore no ‘Save’

button was provided.

Once participants had completed all four tasks using both client applications, they were presented with the post-intervention questionnaire. Participants were asked to complete the questionnaire verbally with responses being written down verbatim by the researcher. As with previous studies, questionnaires were only available in English with the research assistant acting as translator when required. Participant responses were guaranteed to remain anonymous and would not be traced back to them.

Each evaluation took between 15 and 25 minutes to complete, depending on the speed of the participant and the length of the responses given. Responses were captured electronically using Google Forms and later analysed with Microsoft Excel and IBM SPSS (a statistical analysis software package).

7.6 Task performance measures

Several different performance measures were recorded in an attempt to determine which client application participants preferred. Measures included task completion time (Section 7.6.1); num­

ber of display tap events when performing a task (Section 7.6.2); as well as task usability rating (Section 7.6.3). These quantitative results were used in conjunction with qualitative feedback in the form of open-ended responses from participants.

7.6.1 Duration

Task duration is a standard performance measure used in many areas of HCI research (122], and was recorded for each participant in the study. Although these times do not necessarily indicate how easy or difficult participants found a particular task, combined with other performance measures they allowed the researcher to perform a comparison between MobiSAM and MobiSAM Report.

Task timing was triggered by the first display tap event and continued until the participant had completed the task, and navigated back to the Home screen. A Wilcoxon signed-rank test was performed on the timings in order to determine whether there was any significant difference (p<0.05) in task durations between each client application. As the geometric mean has been shown to have less error and bias when compared to the sample median (188], it was used when finding the mean participant task completion times.

7.7. RESULTS 144 7.6.2 Display tap events

The number of display tap events made by each participant when performing a task was noted.

All tap events, except those made when using the virtual keyboard, were recorded. As with task completion times, a Wilcoxon signed-rank test was performed on these values, indicating whether there was any statistically significant difference (p<0.05) between the number of tap events made when using MobiSAM and MobiSAM Report. The median was used when calculating the average number of tap events made by participants for each task. This decision was taken as Lewis and Sauro, in their paper titled Average Task Times in Usability Tests: What to Report?, only advocate using the geometric mean when calculating average task completion times, and not display tap events [188].

7.6.3 Usability ratings

As with the first usability evaluation, presented in Chapter 6, a value was assigned to each option in an attempt to quantify participant responses, with ‘Very easy’ representing a value of five, and ‘Very difficult’ representing a value of one. The overall usability rating for each task was then calculated by summing all participant responses for a participant task, then dividing the result by the total.

The resulting usability rating for each task was then compared across MobiSAM and MobiSAM Report in an attempt to determine which version participants preferred. As the summed Likert responses are considered interval data [14], the Wilcoxon signed-rank test could not be used to determine whether the differences in task usability ratings between the client applications was statistically significant.

7.7 Results

This section presents the results of the pre and post-intervention questionnaires, as well as the results of the four study tasks.

7.7.1 Demographics

As with the previously described studies (see Chapter 4 and Chapter 6), the first section of the pre-intervention questionnaire consisted of five questions and aimed to elicit basic demographic information from each participant.

7.7. RESULTS 145 Of the study sample, 60% of participants were female while 40%. were male. Referring to par­

ticipant age groups, the majority (70%.) indicated that they were between 18-30 years; one participant (10%) indicated that they were between 31-45 years; while two participants (20%) indicated that they were over 60 years. 40%. of participants indicated that they were Black African, while 60%. were White. Examining participants’ preferred language, 20%. of participants specified isiXhosa, while 80%. specified English. Furthermore, 20%. of participants had obtained their matrie, while 80%. had graduated from university.

7.7.2 Use of mobile technology

The second section of the pre-intervention questionnaire solicited feedback relating to parti­

cipants’ use of mobile technology and related services.

Of the participant group, 90%. owned a smartphone while a single participant owned a feature phone. 1(1% of participants had been using a mobile phone for 6-10 years, while the remaining 60%: indicated that it had been over 10 years. The most prevalent brand of handset amongst participants was Samsung, representing 60%: of the study sample. Nokia handsets were second most prevalent and represented 20%: of the study sample, while the remaining two handsets were an Apple and a Huawei branded respectively.

Figure 7.1: Frequency of use: data versus voice versus SMS (n = 10)

Participants were asked to rank the frequency in which they used the data, voice and SMS services available on their mobile phones. Responses arc illustrated in Figure 7.1, with 80%: of participants indicating that they used data most frequently; 60%: indicating that they used voice calls second most frequently; while the same amount (60%:) indicated that they used SMS least frequently. One participant never used data from their handset.

When asked to specify which messaging services participants used on their handsets, all parti­

cipants indicated SMS; 90%: of participants indicated WhatsApp; while a single participant used

7.7. RESULTS 146 BlackBerry Messenger (BBM). All participants indicated that they used these messaging services daily.

Figure 7.2: Use of online data services (n = 10)

Examining which online data services participants accessed from their handsets, 70% indicated both e-mail and Facebook; 10% indicated news; 20%. indicated Wikipedia; while 20%. indicated that they did not make use of any data services (see Figure 7.2). It should, however, be noted that of this 20% (n = 2), one participant (Participant_2B) did make use of WhatsApp, a messaging service which makes use of packet-data. This indicates that the participant possibly uses other non-specified data services which they do not realise are data services. Of the participants (80%.) who made use of online data services, all indicated that they did so on a daily basis.

7.7.3 Task 1: Log in

The mean task completion time and median number of display tap events made by participants when completing Task 1 are summarised in Table 7.1. As each participant performed the task using MobiSAM and MobiSAM Report, both sets of results are provided. 80%. of participants managed to log in to MobiSAM on their first attempt, while 20%. managed to do so on their second attempt. All participants managed to log in to MobiSAM Report on their first attempt.

MobiSAM MobiSAM Report Mean task completion time 18.08s 14.70s

Median tap events 2.5 2.5

Usability rating 94% 96%

Table 7.1: Duration, tap events and usability rating for Task 1

A Wilcoxon signed-rank test showed that there was no statistically significant difference (p <

7.7. RESULTS 147