Empirical part 2 Sopia case

My initial choice to analyze system usability scale as one of the potential UX evaluation methods came from my UX studies in Aalto where we learnt about the method. We were presented the method in line with other UX evaluation tools such as tree testing, heuristic evaluation and cognitive walkthrough. From my studies, I formed an initial assumption that all studied methods have a similar contribution in evaluation of UX issues. At that point of time, I perceived system usability scale to be as effective to reveal UX issues as other methods analyzed in this study. In addition, I thought that time to conduct system usability scale is relatively similar to the requirements for evaluation using other methods.

However, later, trying to understand how I can apply the method to evaluate Sopia website, I faced certain challenges. On the website of Norman Nielsen Group it is said that system usability scale is difficult to understand for novice UX practitioners (Norman Nielsen Group, 2018). The beginners in UX do not understand the aim of the tool, the process, the scoring system and its interpretation. For them, the method is perceived as complex and ambiguous. In the case of Sopia I shared the same sentiment. Personally, I partly grasped the process of system usability scale only after spending a significant amount of hours analyzing the research literature on the topic. When we were rushing to develop our prototype – the situation familiar for the majority of startups – our team did not have time to learn how to use system usability scale. I intentionally put the method aside. While comparing cognitive walkthrough and system usability scale, I decided to proceed with cognitive walkthrough and heuristic evaluation as these methods seemed to us to be more clear, easy to perform and flexible.

The system usability scale questions partly repeat the heuristic evaluation questions and cover the same topics. Thus, I considered that in the Sopia case the system usability scale evaluation method would not bring special value into the product development process.

64

Another reason why our team did not apply system usability scale evaluation to measure the usability of Sopia’s prototype, and later the MVP, is that system usability scale evaluation would provide us with only a general score for the system, without specification of particular usability problems as it was in the case of cognitive walkthrough and heuristic evaluation. However, in order to progress with our service, we needed a detailed understanding of where the usability problems are. For a startup, it is crucial to understand as early as possible what kind of usability mistakes the prototype has and to fix them before putting resources into coding. This I considered as a limitation of the method, because we could not address specific problems and improve them later, as in the cognitive walkthrough.

In addition, I did not find system usability scale to be useful when the product is in the early stages of development. When we started to make the evaluation, we had just two prototypes with a limited amount of features and many usability mistakes. System usability scale would not help us to reveal these mistakes and correct them.

6.3.4. Conclusion

Using my theory-based research framework, I summarized my findings of system usability scale evaluation method – Figure 13. The structure of the framework allows me to describe the holistic process of system usability scale from the preparation phase to the interpretation phase taking into account startup context.

65

Figure 13 – The description of system usability scale method using theory-based research framework to describe the UX evaluation method involving startup constraints

My multi-faceted research of system usability scale disclosed the difference between researchers and UX practitioners in understanding of the process and value of system usability scale in UX evaluation of early stage startups. Researchers claim that system usability scale enables the usability practitioners to quickly and easily evaluate the usability of a given system (Bangor et al., 2008). However, based on my empirical analysis and Sopia case study none of the practitioners used the method or had plans to do so. The observation that the majority of my respondents are not even familiar with the term might signal that in general in the industry the method is not very popular: “no one here uses system usability scale as a word. Or at least I never heard anyone mentioning it once” (Interviewee #3). In the case of Sopia, we also neglected the tool despite the fact that I knew the method. From both empirical analyses, I can conclude that the startup environment does not support the usage of system usability scale. The time constraints and the

66

necessity to put all the resources into MVP development do not allow for the dedication of the effort and time to apply system usability scale.

Empirical findings contradict the academics’ point of view. Researchers defined the method as a quick and easy way to measure usability. However, from empirical analysis and comparing with other UX evaluation methods that I investigated in this study, system usability scale is perceived to be difficult and time-consuming. First, the evaluator should have special training to be able to conduct the evaluation. Second, researchers state that the minimum sample for evaluation is eight users, which is a big investment in terms of time and money for startups. And, lastly, the method provides only a general score that could not be elaborated in a more detailed description of the UX problems. However, for startups the main reason for evaluation is to be able to track usability mistakes and correct them as soon as possible before the programming of the MVP.