All in all, the meCUE questionnaire results show very similar results for the conditions, probably as a result of the very subtle differences between the conditions. The meCUE questionnaire results do however show some differences in product qualities and emotion, but none are significant except for the ‘overall UX’. It is therefore not surprising that these results do not always account for the differences in preference. To see why participants prefer a certain visualization with corresponding animation, researchers might have to compare the conditions and specifically ask for their opinion about the difference. This suggestion is strengthened by the fact that participants do have an opinion about the visualizations and the differences, which they often give as qualitative feedback and in the form of their preference. They however seem to be unable to catch this opinion in the current statements of the meCUE questionnaire.
A possible solution to improve the sensitivity of this questionnaire is adding extra constructs to the product qualities of the CUE model. Similar to leaving out certain constructs, which is possible because of the modular nature of the evaluation method, certain constructs could be added. Identification and status were left out in this experiment, as they are not applicable to information visualization. Also, the questions about the emotions should be carefully reconsidered in further research, as these constructs frustrate participants; they had a hard time associating specific emotions with charts. As described in the introduction, animation is an aspect that mainly influences the aesthetic quality of an information visualization, but can also affect aspects as relational understanding (Heer & Robertson, 2007) and engagement (Bartram & Nakatani, 2010), which are not well reflected by the current questions of the meCUE questionnaire. Future research with the meCUE model should thus try to find all product qualities for a domain and note that these qualities are different for every application.
Also, it might be beneficial to make a less strong separation between emotions and product qualities. Aranyi & van Schaik (2016) for example suggest a direct relation in the CUE model from interaction characteristics to emotions, not necessarily passing the interpretation of product qualities first. This way ‘desirability’ and ‘credibility’ could be added as product qualities of UX, even though these qualities are intertwined with emotions. By not making this strong separation, self-reporting a UX might be easier for a user, as users might not always know what product quality triggers their emotions.
Another interesting observation is that the conditions with higher rated aesthetics are also rated higher on usability, even though the loading animation does, objectively speaking, not improve anything other than aesthetics. This finding is in line with the aesthetic-usability effect as described by Tractinsky; that if something is more beautiful it is also perceived as more usable (Tractinsky et al., 2000). This shows that the perceived usability differs from the objective usability, but simultaneously raises the question how good humans are at self-reporting their UX on the basis of different product qualities. Do users always know why they prefer a system or have a better experience with a system? Some research argues that when designing a UX, one should not listen to the user but rather observe them (Nielsen & Levy, 1994). As the meCUE method of measuring UX solely depends on the self-report of the user, this is an important question to consider in future research with the meCUE questionnaire. This same fact also suggests that objective behavioural or physiological measurements form a promising alternative as measurement of UX.
Aside from the limitations mentioned above, a limitation involves the property of UX that it can change over time, and that a positive initial experience is not guaranteed to motivate prolonged use. How UX evolves over time is not often researched and one could imagine that animations are fun for a while, but after a user has seen it a couple of times, it might get distracting. Future research should prove if this is indeed the case when a visualization is more often used. At a first encounter hedonics are important, but over time aspects as usability will become more important (Wilamowitz-Moellendorff et al., 2006). This should also be considered and researched for the animations described in this experiment. Giving users the ability to turn on or off animations, could help in a positive UX over a longer time.
As this research found that the meCUE method is not as useful for small differences between conditions, also bigger differences in conditions should be researched with the meCUE method, to see if larger differences can be measured more accurately. Even though the objectivity of the ratings of qualitative feedback is debatable as it was just rated by one person, it indicates that qualitative methods can strongly
63
outweigh quantitative methods of measuring UX in cases with small differences between conditions. Future research could benefit from specifying specific user groups; this research already shows how age has an influence on the preference for animated visualizations or non-animated visualizations. It is interesting to research how other user characteristics have an influence on UX.
65
6
66
6
Conclusion
The first research question was: “How can the UX of an information visualization be measured in a quantitative manner?”. An extensive literature review set out different definitions and models of UX and related these to the domain of data visualization. This literature review showed how the concept of UX is still vague with a wide variety of meanings. Different UX models have different areas of focus. There are many methods for measuring UX, both objective and subjective methods. As this experiment aimed to research the UX of an information visualization using it while having a goal, a model was chosen in which instrumental aspects such as the usability were well reflected. Since information visualizations are graphics based, also aesthetics plays an important role. The fact that information visualizations are not physical products made aspects as identification or status less important. From these criteria the CUE model seemed to match best with the domain of data visualization. This model was used in the experiments.
The second research question was: “What aspects of the CUE model and its measurement tool meCUE can be used for the domain of information visualization?”. The meCUE results show no significant differences between the conditions, but some small differences are observable. These small differences are not only a consequence of the subtleness of the differences between the conditions, they are also caused by the diverseness of opinions in the participants group. By averaging results over this group, contrasting opinions can cancel each other out. The only significant difference measured by the meCUE questionnaire was the ‘overall UX’, where the bouncy animation scored lower than the other two conditions. The preferences however show a clear preference for the calm animation, even though it’s measured UX is approximately the same as the non-animated condition.
First, the constructs of the product qualities usability, functionality and aesthetics do clearly not cover all aspects of the UX of information visualization. The needed constructs to measure the product qualities are different for each application. It is thus important that future research determines what other constructs have to be added to the meCUE questionnaire, to be able to apply it to data visualization. As discussed in the background, identification and status are examples of constructs that do not apply to data visualization but rather apply to physical products. Other models and theories indicate that promising candidates for added constructs could be engagement, interaction aesthetics, credibility and desirability. Future research should however show if these constructs overlap too much with other constructs or with each other, and find suitable questions to measure them.
Further, it is conceptually adequate to separate emotion from product qualities, but impractical as humans cannot always relate emotions to product qualities causing them. In this research for example, people had negative emotions associated with the bouncy animation, but this was not reflected in the aesthetic construct. It might be practical to not separate emotions and qualities as strong as the CUE model currently does, as users might understand their emotions better than the product qualities causing them. By including constructs as “Joy of use” or “desirability”, which are clearly qualities intertwined with user emotions, users might succeed better in self-reporting their UX.
Finally, compared to qualitative research, quantitative UX research using the meCUE method has a lot of disadvantages in the context of this research. First of all, the effect size is very small, requiring a very large number of participants to show significant differences. Second, qualitative research allows for finding more specific answers as to why a certain visualization is preferred if participants get to express their ideas freely. The third research question was: “How do loading animations and transition animations influence the UX of information visualizations?”. The different conditions in the experiment have clearly shown that animations can both hurt and improve the UX of information visualizations. For the loading animations, there was a big preference amongst the participants for the calm loading animation. The bouncy loading animation was an example of an animation that hurt the UX of the information visualization; participants generally found it distracting and unnecessary which resulted in a lower overall UX. Assuming that the preference for the calm loading animation is a consequence of a positive UX (as suggested by the CUE model), animation did in this case improve the UX. The measured UX with the meCUE was however approximately the same for the non- animated condition, suggesting that not all aspects of the UX were properly measured. For the transition
67
animations, there was a small preference for both animated transitions, but overall participants had very different opinions. Some participants really appreciated the animated transitions, whereas others found them distracting.
These different experiences emphasize the subjectivity of UX. For one participant the animated transitions improved the UX, for others the animated transitions were a distraction from the message of the graph. A one-size-fits-all UX design is therefore not favourable. By more specifically specifying a user group these deviations within the participants can be decreased. In addition to that, some participants indicated increased usability of functionality even if the animation only had aesthetic purpose. This shows raises questions about the tenability of self-reported UX and could be an argument for more objective measures of UX.
68
References
Ackoff, R. L. (1989). From data to wisdom. Journal of Applied Systems Analysis, 16, 3–9. https://doi.org/citeulike- article-id:6930744
Ahsanullaha, Suziah Sulaiman, Ahmad K B Mahmood, & Muzafar Khan. (2015). Understanding factors influencing User Experience of interactive systems: A literature review. ARPN Journal of Engineering and Applied Sciences, 10, 18175–18185.
Alben, L. (1996). Quality of experience. Interactions, 3, 11–15. https://doi.org/10.1145/235008.235010
Alves, R., Valente, P., & Nunes, N. J. (2014). The state of user experience evaluation practice. Proceedings of the
8th Nordic Conference on Human-Computer Interaction Fun, Fast, Foundational - NordiCHI ’14, 93–102.
https://doi.org/10.1145/2639189.2641208
Amar, R., Eagan, J., & Stasko, J. (2005). Low-level components of analytic activity in information visualization. In Proceedings - IEEE Symposium on Information Visualization, INFO VIS (pp. 111–117). https://doi.org/10.1109/INFVIS.2005.1532136
Anwyl-Irvine, A., Massonnié, J., Flitton, A., Kirkham, N., & Evershed, J. (2019). Gorilla in our Midst: An online behavioral experiment builder. BioRxiv, 438242. https://doi.org/10.1101/438242
Aranyi, G., & van Schaik, P. (2016). Testing a model of user-experience with news websites. Journal of the Association for Information Science and Technology, 66, 2471–2493. https://doi.org/10.1002/asi.23462 Bacigalupi, M. (1998). The craft of movement in interaction design. In Proceedings of the working conference on
Advanced visual interfaces (pp. 174–184). ACM. https://doi.org/10.1145/948496.948521
Bartram, L. (1997). Perceptual and interpretative properties of motion for information visualization. Workshop on New Paradigms in Information Visualization and Manipulation. https://doi.org/10.1145/275519.275520 Bartram, L. (1998). Enhancing Visualizations with Motion. In Proceedings of Information Visualization (pp. 3–7). Bartram, L., & Nakatani, A. (2010). What makes motion meaningful? Affective properties of abstract motion. In
Proceedings of the 2010 Fourth Pacific-Rim Symposium on Image and Video Technology (pp. 468–474). https://doi.org/10.1109/PSIVT.2010.85
Bederson, B. B., & Boltman, A. (2007). Does Animation Help Users Build Mental Maps of Spatial Information? In The Craft of Information Visualization. https://doi.org/10.1016/b978-155860915-0/50015-9
Bertin, J. (1983). Semiology of Graphics. University of Wisconsin Press. https://doi.org/10.1103/PhysRevLett.94.208902
Börner, K., Bueckle, A., & Ginda, M. (2019). Data visualization literacy: Definitions, conceptual frameworks, exercises, and assessments. Proceedings of the National Academy of Sciences, 116(6), 201807180. https://doi.org/10.1073/pnas.1807180116
Bradley, M. M., & Lang, P. J. (1994). Measuring emotion: The self-assessment manikin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry, 25(1), 49–59. https://doi.org/10.1016/0005-7916(94)90063-9
Brath, R., Peters, M., & Senior, R. (2005). Visualization for communication: The importance of aesthetic sizzle. In Proceedings of the International Conference on Information Visualisation (pp. 724–729). https://doi.org/10.1109/IV.2005.145
Card, S. K., Mackinlay, J. D., & Shneiderman, B. (1999). Readings in Information Visualization: Using Vision to Think. In Information Display. Morgan Kaufmann Publishers Inc.
Cawthon, N., & Moere, A. V. (2006). A Conceptual Model for Evaluating Aesthetic Effect within the User Experience of Information Visualization. Tenth International Conference on Information Visualisation (IV’06), 374–382. https://doi.org/10.1109/IV.2006.4
Cawthon, N., & Moere, A. Vande. (2007). The effect of aesthetic on the usability of data visualization. Proceedings
69
https://doi.org/10.1109/IV.2007.147
Chen, M., Ebert, D., Hagen, H., Laramee, R. S., Van Liere, R., Ma, K.-L., … Silver, D. (2009). Data, Information, and Knowledge in Visualization. Computer Graphics and Applications, IEEE, 29(1), 12–19. https://doi.org/10.1109/MCG.2009.6
Few, S. (2006). Information Dashboard Design. O’Reilly Media, Inc.
Few, S. (2009). Now You See It: Simple Visualization Techniques for Quantitative Analysis (1st ed.). Analytics Press. Few, S. (2017). Data Visualization Effectiveness Profile. Perceptual Edge.
Figueiras, A. (2015). Towards the Understanding of Interaction in Information Visualization. In 2015 19th International Conference on Information Visualisation (pp. 140–147). https://doi.org/10.1109/iV.2015.34 Forlizzi, J., & Battarbee, K. (2004). Understanding experience in interactive systems. Proceedings of the 5th
Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques, 261–268. https://doi.org/10.1145/1013115.1013152
Friedrich, C. (2002). Animation in relational information visualization. (Unpublished doctoral dissertation).
University of Sydney. Retrieved from
https://ses.library.usyd.edu.au/bitstream/2123/9288/1/FriedrichThesis.pdf
Ganglbauer, E., Schrammel, J., Deutsch, S., & Tscheligi, M. (2009). Applying psychophysiological methods for measuring user experience: Possibilities, challenges and feasibility. Interact’09 Conference. https://doi.org/10.1007/978-3-642-23768-3
Hassenzahl, M. (2003). The Thing and I: Understanding The Relationship Between User and Product. In Human– Computer Interaction Series, Funology (pp. 31–42). Springer. https://doi.org/10.1007/1-4020-2967-5_4 Hassenzahl, M. (2008a). Aesthetics in interactive products: Correlates and consequences of beauty. In Product
Experience (pp. 287–302). Elsevier. https://doi.org/10.1016/B978-008045089-6.50014-9
Hassenzahl, M. (2008b). User experience (UX): Towards an experiential perspective on product quality. Proceedings of the 20th International Conference of the Association Francophone d’Interaction Homme- Machine on - IHM ’08, 11–15. https://doi.org/10.1145/1512714.1512717
Hassenzahl, M., Kekez, R., & Burmester, M. (2002). The importance of a software’s pragmatic quality depends on usage modes. Proceedings of the 6th International Conference on Work With Display Units (WWDU 2002), 275–276.
Hassenzahl, M., & Monk, A. (2010). The inference of perceived usability from beauty. Human-Computer Interaction, 25(3), 235–260. https://doi.org/10.1080/07370024.2010.500139
Hassenzahl, M., Platz, A., Burmester, M., & Lehner, K. (2000). Hedonic and ergonomic quality aspects determine a software’s appeal. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems CHI 00, 201–208. https://doi.org/10.1145/332040.332432
Hassenzahl, M., & Tractinsky, N. (2006). User experience - A research agenda. Behaviour and Information Technology, 25(2), 91–97. https://doi.org/10.1080/01449290500330331
Heer, J., & Bostock, M. (2010). Crowdsourcing Graphical Perception: Using Mechanical Turk to Assess Visualization Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 203–212). ACM. https://doi.org/10.1145/1753326.1753357
Heer, J., & Robertson, G. G. (2007). Animated transitions in statistical data graphics. IEEE Transactions on Visualization and Computer Graphics, 13(6), 1240–1247. https://doi.org/10.1109/TVCG.2007.70539 Herne, H., & Huff, D. (2006). How to Lie with Statistics. Applied Statistics. https://doi.org/10.2307/2346789 Hong, W., Thong, J. Y. L., & Tam, K. Y. (2004). Does animation attract online users’ attention? The effects of flash
on information search performance and perceptions. Information Systems Research, 15(1), 1–108. https://doi.org/10.1287/isre.1040.0017
ISO 9241-11. (2017). Part 11: Usability: Definitions and concepts. Retrieved January 5, 2019, from https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed-2:v1:en
70
Jiang, Z. (Jack), Wang, W., Tan, B. C. Y., & Yu, J. (2016). The Determinants and Impacts of Aesthetics in Users’ First Interaction with Websites. Journal of Management Information Systems, 33(1), 229–259. https://doi.org/10.1080/07421222.2016.1172443
Karapanos, E. (2013). Modeling Users’ Experiences with Interactive Systems. Studies in Computational Intelligence 436. Berlin, Heidelberg: Springer. https://doi.org/10.1007/978-3-642-31000-3
Karapanos, E., Zimmerman, J., Forlizzi, J., & Martens, J. (2009). User experience over time: An initial framework. In Proceedings of the 27th SIGCHI Conference on Human Factors in Computing Systems (pp. 729–738). New York, NY, USA: ACM. https://doi.org/10.1145/1518701.1518814
Kirakowski, J., & Corbett, M. (1993). SUMI: the Software Usability Measurement Inventory. British Journal of Educational Technology, 24, 210–212. https://doi.org/10.1111/j.1467-8535.1993.tb00076.x
Kurosu, M., & Kashimura, K. (1995). Apparent usability vs. inherent usability: experimental analysis on the determinants of the apparent usability. In Conference Companion on Human Factors in Computing Systems (CHI ’95) (pp. 292–293). New York, NY, USA: ACM. https://doi.org/10.1145/223355.223680
Lachner, F., Naegelein, P., Kowalski, R., Spann, M., & Butz, A. (2016). Quantified UX: Towards a Common Organizational Understanding of User Experience. In Proceedings of the 9th Nordic Conference on Human-
Computer Interaction - NordiCHI ’16 (pp. 1–10). New York, NY, USA: ACM.
https://doi.org/10.1145/2971485.2971501
Lambdin, C., & Shaffer, V. A. (2009). Are Within-subjects designs transparent? Judgment and Decision Making, 4(7), 554–566. https://doi.org/10.1037/e722352011-194
Lavie, T., & Tractinsky, N. (2004). Assessing dimensions of perceived visual aesthetics of web sites. International Journal of Human Computer Studies, 60(3), 269–298. https://doi.org/10.1016/j.ijhcs.2003.09.002
Law, E. L.-C., Roto, V., Hassenzahl, M., Vermeeren, A. P. O. S., & Kort, J. (2009). Understanding, scoping and defining user experience. In Proceedings of the 27th international conference on Human factors in
computing systems - CHI 09 (pp. 719–728). New York, NY, USA: ACM.
https://doi.org/10.1145/1518701.1518813
Lemon, K., Allen, E. B., Carver, J. C., & Bradshaw, G. L. (2007). An empirical study of the effects of Gestalt principles on diagram understandability. In Proceedings - 1st International Symposium on Empirical Software Engineering and Measurement, ESEM 2007 (pp. 235–244). https://doi.org/10.1109/ESEM.2007.17 Mahlke, S. (2008). Visual aesthetics and the user experience. In Proceedings of The Study of VisualAesthetics in
Human-Computer Interaction (pp. 1862–4405).
Mahlke, S., & Thüring, M. (2007). Studying antecedents of emotional experiences in interactive contexts. In Proceedings of the SIGCHI conference on Human factors in computing systems - CHI ’07 (pp. 915–918). New York, NY, USA: ACM. https://doi.org/10.1145/1240624.1240762
Mao, J. Y., Vredenburg, K., Smith, P. W., & Carey, T. (2005). The state of user-centered design practice. Communications of the ACM, 48(3), 105–109. https://doi.org/10.1145/1047671.1047677
McNamara, N., & Kirakowski, J. (2006). Functionality, Usability, and User Experience: Three Areas of Concern. Interactions, 13(6), 26–28. https://doi.org/10.1145/1167948.1167972
Michailidou, E., Harper, S., & Bechhofer, S. (2008). Visual complexity and aesthetic perception of web pages. In