Qualitative Analysis of User Feedback

In order to analyse the written comments that were part of the post-test question- naires, the transcriptions of these written responses were broken into 3 documents: keyword feedback, visualisation feedback, and final thoughts. Each of these tran- scripts correspond to a section in the post-test questionnaire—questions 32-36 for the keyword feedback, questions 38-43 for the visualisation feedback, and questions 47-50 for the final thoughts.16 The transcripts can be found in Appendix D. Each question was treated as a codeable segment (as most responses were only a sentence long). In no situation does a coded segment span more than one question. The ac- tual coding of the documents leveraged the same open coding technique detailed in Chapter4. Additionally, if the participant made the same (or similar) comment in multiple questions (such as "I didn’t like x" or "I really enjoyed y"), it was only coded once so as to avoid artificial inflation of the codes.

The code coverage across all three documents was 44%. The codes primarily broke down into 2 main themes: positive and negative feedback. Each of these themes then broke down into 3 further categories:

• relevance - feedback regarding the relevance of the data returned via a search result

• clarity - feedback regarding how easily the participant understood the data • usability - feedback related to how usable the system was from a functional

standpoint

Other codes not part of these two themes were codes for tracking: • a preference for visualisation search over keyword search • a preference for keyword search over visualisation search

• cognitive overload (which relates to any comments by the user of feeling "over- whelmed" by the system or the data in their result sets)

15_{It should be noted that these tests were somewhat spurious from the outset as the calculations for} attractiveness, pragmatic, and hedonic scores in the UEQ workbook rely on means; however, since the questions themselves are based on a Likert scale, they should be leveraging the median rather than the mean, as it is ordinal rather than interval data. As the data in the workbook is already based on peer- reviewed and accepted work, the calculations in the workbook were not adjusted for this flaw, and thus these scores were imported as means and treated as continuous interval data for the additional statistical analysis in order to remain consistent with the accepted work.

5.3. Analysis of Results 143

• learning (which was used to track any references to new information the user gleamed as a direct result of the using the system)

Table5.12details the percentage of each code as a reflection of all coded segments, cross-tabulated by the transcription document in which they appear.

TABLE5.12: Cross-tabulation of the % of codes as a reflection of all coded segments across transcriptions.

Code Final Response Keyword Visualisation Total

Keyword Preference 2% 3% 1% 2% Visualisation Preference 6% 3% 3% 3% Cognitive Overload 1% 0% 2% 1% Learning 19% 20% 17% Positive (General) 5% 5% 7% 6% Relevance 43% 7% 1% 8% Clarity 27% 11% 8% 11% Usability 14% 12% 11% 12% Negative (General) 11% 3% 6% Relevance 8% 5% 2% 4% Clarity 6% 29% 17% Usability 1% 35% 22% 25%

Generally speaking, the qualitative analysis supports the findings from the quan- titative analysis.17 Most users found the data in the visualisation search more rele- vant to their questions but had a more difficult time understanding the data (i.e. it took more cognitive effort to parse the visualisation). This point supports the ideas in Chapter 2 that visualisations are treated as more cognitively demanding. The keyword search, however, was seen as much easier to understand but made it more difficult to find relevant information. Additionally, while the keyword search had slightly better positive usability comments (12% versus 11% for visualisation), it had considerably more negative comments. Many of these were related to not being able to order search results beyond the default relevance sort, and some wanted a more in-depth default search.18 In addition, there is a slight edge to what users stated they learned during the visualisation search as opposed to the keyword search (19% versus 14%). Again, this ties back to the comments in Chapter2, which discusses the importance of visual information and the advantage it provides in assisting in- dividuals with data extrapolation.

There also appeared to be a strong correlation between participant comments about their mode of thinking and their ultimate choice of their preferred method of search. The participants who specifically stated they preferred to start with the micro-analysis of the text through the keyword search before looking for larger

17_{A more in-depth discussion of this analysis will be conducted further on.}

18_{It should be noted that a basic default keyword search and advanced search were implemented} to mimic many of the common functions inherent in most DREs. Therefore, many of the comments related to a desire for more robust search are comments which should be considered beyond the scope of just this work and should be applicable to all DREs.

trends via the visualisation search tended to prefer keyword searches. Conversely, those who preferred to conduct a macro-analysis first and use that analysis to drill into more detailed results preferred the visualisation search. Additionally, numer- ous participants noted how well the two search mechanisms complement each other and that they are much better suited to the overall goals of the user when used in concert, as opposed to standing on their own. This is perhaps best summed up by participant 5c0a5fdc9e02299a92d16b52:

I started to identify the data around the key pieces of information I was looking for. For example, I wanted to use the visualisation tool to get a sense of when more wine was purchased. Then, subconsciously, I was using the keyword search tool to find out more information around men- tions of wine in the archive. That’s when I began to see other patterns and identify other questions, such as what about abstinence days? How can I incorporate the cheaper rates into my previous visualisations...It re- minded me that subconsciously I was using the two tools to address the same theme but my expectations of what answers I would get out of the results modified my research questions to meet the search/exploration tools. [310]

Finally, as part of the overarching research question also concerned itself with aspects of learning, the analysis of the data from both case studies also attempted to determine if there were any statistically significant effects on learning. While there were no direct evaluations of learning outcomes,19 there was data captured in the post-test questionnaire that specifically asked what the individual learned as part of a particular search method. If a user directly stated a fact or piece of data which was gleaned during a search in response to any of the questions in the post-test question- naire, this was recorded as an indication of learning. Each participant was assigned a value of either 0 or 1 for each search indicating whether they acquired new knowl- edge, thus two dichotomous variables were created to indicate that either the partic- ipant learned new information during either the keyword search or the visualsiation search, or they did not. If the user was vague in their response to the question—such as "I learned about diet" or "There were 88 results related to ’spice’"—these instances were discarded and coded as 0 (or no knowledge acquired) as the statement was too vague to indicate that there was any real knowledge gained. When adding these variables to the larger dataset and running both Fisher’s Exact Tests as well as bi- nomial regressions for preferred method, study type, primacy group, and training type, there appeared to be no statistical significance between these variables and whether the user obtained new knowledge during the visualisation search or the keyword search.

19_{The case study focused on the engagement aspects, due in part to the restrictions around the gath-} ering of participants and an inability to directly leverage the system within a structured classroom setting.

5.4. Discussion 145

5.4

Discussion: The Impact of the Data on the Research

"Data are just summaries of thousands of stories—tell a few of those stories to help make the data meaningful" [311, qtd. in Chapter 3 opening]. In the previous section, the data collected from the case studies was presented and the methodology behind the analysis was defined. But data needs a discussion in order to be born into rel- evance. Without contextualisation, the data is spurious at best and meaningless at worst, and so the question remains: what stories do the data tell?