I calculated a NASA-TLX adjusted workload rating for each participant by multiplying the rating given to each source of workload by the importance of the workload as indicated by each participant. Figure 7.2 displays the distribution of each workload.
It is clear in the figure that Mental Demand was perceived by partici- pants as the largest source of workload. After Mental Demand, Temporal Demand, Performance, and Effort were perceived as the second largest sources of workload. Physical Demand and Frustration were considered
7.3. PERCEPTION OF WORKLOAD 107 ● ● ● ● ● ● 0 100 200 300 400 Mental Demand Physical Demand Temporal Demand
Performance Effort Frustration Sources of Workload
Adjusted Rating
Figure 7.2: NASA-TLX adjusted workload ratings.
to have the least influence.
As Mental Demand was the highest perceived workload, using SpEx must require much thought on the part of the user. For instance, when pro- vided with a task, participants must either identify useful search queries to enter into the search facility (which may include synonyms and related topical words) and/or either browse or scan [10] Word Clouds for words that correspond to the task at hand. Such a process may lead to multiple candidate regions of the audio which may be relevant. Participants must then narrow their search to identify the most relevant regions to listen to.
Although searching is a non-trivial task, participants rated Frustration as contributing little to the tasks. Low frustration indicated that, although much thinking was required, participants did not have difficulty when performing their tasks. Low frustration is a good indicator that SpEx could be used effectively for audio retrieval tasks. Sufficient information was provided to participants, information was displayed clearly, and interac- tion controls were intuitive. Comments from participants attest to the ease of use of SpEx:
“Search feature, can find the right section really fast. Highlight- ing, makes it very easy to see the most likely place to look.” (P7)
“It’s easy to tell what parts of the talk are about which things. You can actually get an idea of the content of the sections with- out fully listening to them.” (P9)
With regard to performance, Section 7.2 describes the good perfor- mance participants were able to achieve. Median Section Selection pre- cision of 80.72%, Section Selection recall of 93.75%, Fact Finding accuracy of 100% (mean of 92.10%), and Summarisation quality of 75.00% indicates that participants were able to use SpEx successfully. But despite the re- sults, participants’ perception of performance was not rated low. During the user study, I observed participants frequently re-listening to portions of audio that were believed to contain the answer. Re-listening was used to double-check an answer before settling on the answer. But double- checking was not always performed. The extra time-cost of re-adjusting the audio play position back to the start of a sentence coupled with the inability to quickly perform an in-depth scan of the surrounding content (which is possible with text documents) was an issue. The issue con- tributed to participants settling on an answer before being fully comfort- able with the answer obtained. As an example, here is one such feedback regarding replaying a sentence that was just listened to:
“Something to show the start and end of each sentence. I.e. when sentence A meets sentence B.” (P8)
The amount of audio listened to by participants is presented in Figure 7.3a. As a comparison, the amount of audio listened to more than once is displayed in Figure 7.3b.
It is evident that while Parts A and B contained a median play dura- tion of 7.63 minutes and 6.54 minutes respectfully, the median amount of
7.3. PERCEPTION OF WORKLOAD 109 ● ● 1.6 0.13 1.82 5.68 0.42 4.28 7.63 1.01 6.54 9.1 1.49 8.87 12.17 2.49 14.09 0 5 10
Greek History Psychology of Food State of the Union
Tasks A udio Pla y ed in Min utes
(a) Audio Played
● ● ● ● ● ● 0 0 0 0 0 0.08 0.05 0 0.42 0.37 0.01 0.73 0.88 0.01 1.3 0.0 0.5 1.0 1.5 2.0
Greek History Psychology of Food State of the Union
Tasks A udio Repla y ed in Min utes (b) Audio Replayed
audio re-played for Parts A and B were 0.05 minutes and 0.42 minutes respectfully. Most participants replayed some audio which indicates that the need to replay audio is important for audio retrieval. Re-playing au- dio allows users to double-check what was said. Had SpEx incorporated mechanisms to support efficient audio re-play, participants may have been more confident of their results.
The desire to re-listen to what was just played coupled with searching for audio to listen to had lead to a notable amount of perceived effort on participants. As previously mentioned, searching for places in the audio to listen to is a non-trivial process. Participants frequently attempted mul- tiple search queries, compared multiple segments, and listened carefully to the playing audio. A high perceived effort may mean SpEx is not suit- able for use when multitasking. Users may need to pay full attention to the search task at hand. It seems like the inability to multitask may be an issue, but research has found that students like to mimic lecture settings and refrain from other activities when listening to lecture recordings [32].
Common issues by participants that may have increased the effort in- volved include not providing sufficient information about the transcript and accidentally deselecting all highlighted words. In particular, partici- pants mentioned that some context sentences were ambiguous and there was a desire to produce context sentences over Transcript Markers, not just over Word Cloud words. As a consequence, SpEx may have been more difficult to use. Accidentally deselecting all highlighted words oc- curred when participants would click on the background of SpEx with the intent to skip the audio to the selected position. Instead, all selected words would deselect and the participant would manually reselect all relevant words again. As one participant, of several, noted:
“I kept accidentally removing word markers when trying to change when I was listening to — would be better if clicks on blank space moved the time bar if audio was playing (only cleared the screen if paused).” (P15)