5.3 AutobahnVis : Visualizing Messages
5.3.3 User-centered Development: From Paper Mockups to an Integrated Tool
tegrated Tool
The development process we used for developing AutobahnVis is a very good example of how we in general proceeded in developing our tools in close collaboration with automotive engineers. By providing this representative example in more detail, I want to give the reader the opportunity to learn more about the steps we took, which kind of prototyping techniques we used, and how this influenced us in coming up with a final solution. All work on AutobahnVis was conducted
(a) Line Chart (b) Bar Chart (c) ThemeRiver
(d) Bubble View (e) Message View
Figure 5.7: Early visualization mockups in the AutobahnVis project.
in close collaboration with five trace analysts and three analysis tool developers (Carmen). Be- fore we developed the final version presented above, we involved them but also other engineers and external testers over a period of two years in a variety of studies with prototypes of differ- ent fidelity. Our approach is similar to the staged design and development process applied by
McLachlan et al. in their LiveRAC project [MMKN08] and we hope that our approach might
serve as another best practice example in this context. In general, our process could be divided into three major steps:
1. Paper Prototyping As a very first step, we6 used paper mockups in order to discuss possible solutions and features with our target users. We created a catalog of mockups which contained existing visualizations from the InfoVis domain, newly developed visualization con- cepts and adaptations of traditional automotive solutions. The catalog most importantly included:
• A traditionalList View(no figure).
• Traditional Viewssuch as line and bar charts (cf.Figure 5.7-a and b).
• A ThemeRiver [HHW+02] which we thought might be useful for showing trends in the
communication process, for instance, showing the number of messages sent over the vari- ous bus systems over time (cf.Figure 5.7-c).
6Early work on AutobahnVis has been published in [SKHB09]. Any use of “we” in this context therefore refers
charts) activity. - Finding transition states ThemeRiver 5.7-c Not
requested
High abstraction level with less level of de- tail was assessed to be not applicable to the in-car communication domain.
- Showing combined trends
Bubble View
5.7-d Optionally requested
Although this view had no direct use case mapping it was well liked in discussions be- cause of its innovative character.
No use cases found
Message View
5.7-e Strongly requested
This visualization reached a wide accep- tance by the expert users because it sup- ported a common mental model with a sim- ple and pleasant visual vocabulary.
- Finding Errors
- Monitoring Cyclic Traffic
- Monitoring the In-Car Communication - Getting familiar with the car network domain - Explore Cause-Effect relations
Table 5.1: Results of evaluating early design mockups in the AutobahnVis project.
• A design idea which we called the Bubble Viewwhich was based on scale-free networks
[BO04]. We suggested this with the idea to visualize different points of an in-car network, for instance, ECUs or functional blocks, and draw them as bubbles and directed graphs with arrows between them. The basic idea was to adapt the bubbles’ sizes, whenever the activity of its component increases in order to present a current state of the system and to show the “big players” at a specific time (cf.Figure 5.7-d).
• The initial design sketch of ourMessage Viewconcept (cf. Figure 5.7-e).
To evaluate our design ideas, we discussed printed versions (cf.Figure 5.7) with our eight domain collaborators. Their input narrowed the visualization concepts down very quickly and led to a short list of pragmatic design solutions. We asked them to assign use cases to the visualization concepts and to imagine and illustrate in each case an example how it might be used to visualize the underlying data, such as: “In my opinion the Message View could be used to show message bursts and to investigate frequencies in sending actions”. Based on this evaluation, we classified the visualization concepts into three categories: Strongly requested, optional, or not requested. The results of this classification as well as the reasons and the mapped use cases can be found in
Table 5.1.
High-fidelity Prototype In order to turn the basic concepts into a high-fidelity prototype, we took the most requested visualization concepts, namely the Message View and the List View and implemented a first stand-alone version that was not integrated into engineers’ analysis software (cf. Figure 5.8). We basically used this version as a proof of concept and as a test environment for doing several usability and utility studies—mostly think-aloud studies—both with engineers
Figure 5.8:Screenshot of the early AutobahnVis high-fidelity prototype with:(a)Message View;
(b)Signal View which is similar to the Message View but shows signals instead of messages;(c)a Message List;(d)a magnified excerpt from the Message View illustrating the concept of message stacking we used for representing messages that were sent very closely in time; (e)a POI that can be set by the user; and(f)the Sync Button.
.
and with outside testers. These early studies, provided us with many interesting insights which strongly influenced (a) us in continuing the project, (b) our stakeholders’ interests in our work, and (c) our design decisions of the final version of AutobahnVis. In the following, I briefly discuss design variations and the results of our earlier studies with the high-fidelity prototype.
The most interestingsummative result we derived from our early studies with domain experts was the indication of a novel mental model that appeared for thinking and discussing about trace data. In a think-aloud study with five domain experts, it was very interesting to see them starting to explain things directly by means of the novel Autobahn metaphor, such as: “As you can see, we have lots of traffic on this ECU’s lane”,“Oh, what does that burst of message rectangles [— we initially used rectangles instead of diamonds—] mean?”, or“On the road you can perfectly track cyclic messages”. Note, that we already found indications in these studies for the potential value of detecting message bursts and better understanding cyclic messaging (see above for more details). Another hint for the appearance of a novel mental model was an observation we made during a presentation with a live demo of our high-fidelity prototype in a meeting of analysis experts. During the live demo, the attendees suddenly started to discuss a known problem of a specific ECU by means of the Autobahn representation. The problem was about the ECU’s
“spamming”activities on the bus and the engineers started arguing:“Does anyone know why the LRR ECU sends message bursts in this compressed cycle?”, “In my opinion that has to be the reason for the bus spam.”,“No, the other ECUs seem to work normally”, etc.
On the other hand, we also got much formative feedback which was very helpful for re-
participants. Instead of representing multiple messages, it was interpreted as a length coding of information and guessed to be the messages’ byte length. Inquiring this in detail, it showed that coding the messages’ length would not be beneficial for the engineers at all. While after resolving the misconception, they had no further problems in understanding it, we took this initial misun- derstanding very seriously and refused message stacking in our final concept. Another point of criticism was the understanding of the coordination feature. In our early version of AutobahnVis we allowed the user to open an arbitrary number of Message, Signal and List Views. Each view held a synchronize icon placed in the upper right corner. By pressing, the user added this partic- ular view to a list of views that were synchronized in time. Two of our study subjects, however, wondered that they had to select more than one view to start the synchronization action. Their current mental model matched more an “all-or-none” synchronization feature and the synchro- nization of sub-groups of views was not self-explanatory. We addressed this aspect in our final tool and left out group synchronization.
Finally, we also presented our prototype to Carmen’s tool developers and outlined the potentials we found in our user studies. This helped us to convince them to allow us connecting our later approach closely to their software platform.
In general, however, there were two basic technical drawbacks of the first high-fidelity prototype that hindered our target engineers in productively using the tool and us in studying the tool under real condition. These aspects were the missing integration into daily working practices (cf. R-9 and R-8) and scalability restrictions as our tool could only work with traces up to five seconds.
Final Tool, Integration, Limitations We therefore discarded the first prototype and used the lessons learned to implement a second, closely integrated (R-9) system of AutobahnVis that I have described above. For our novel tool, we, on the one hand, slightly improved and adapted the design according to the results of our early user studies: We, for instance, used diamonds instead of rectangles for messages as they indicate an exact point in time, we introduced group hexagons for better scalability, we changed the Signal View to a line plot representation, provided more and better filter possibilities, and changed the coordination paradigm. On the other hand, we invested a significant amount of work in integrating the tool and in making it scalable to real traces. Both aspects required us to overcome a variety of technical, non InfoVis/VA specific challenges, such as handling the proprietary and restricted interfaces of Carmen we were allowed to use.
The current version of AutobahnVis is available as a block-module in Carmen’s visual test config- urator (cf. Figure 5.2) and, in doing so, can closely interact with several trace analysis modules, most importantly the Replay Module and diverse filter modules (R-4). Via the Replay Module an engineer can load a trace file and replay the file either in real time or up to 105times faster/slower.
and AutobahnVis the engineer can use common modules for filtering the data, for instance, in terms of relevant bus systems, ECUs and/or messages.
While we generally succeeded in making AutobahnVis available as a module in Carmen, there are still some minor restrictions based on Carmen’s connection technology for external modules we were allowed to use. Compared to the first prototype, the final version of AutobahnVis is re- stricted in showing messages solely separated by bus systems but not by ECUs. This functionality is not supported by the connection technology we used. For the same reason, we currently cannot distinguish between incoming and outgoing messages on bus systems which would—according to lead analysts’ statements—further improve the Message View.
5.3.4 Discussion and Adoption
AutobahnVis is a tool that we built to visually support current techniques of message-based browsing and analyzing raw data. Due to the fact that we incorporated AutobahnVis into Car- men, a widely-used and accepted automotive software analysis platform used in our company, we had the chance to evaluate our tool under real conditions. In general, the feedback we got in our studies was very positive such as stated by one of our lead analysts: “AutobahnVis heralds the future of trace analysis tools’ interfaces!”Following our initial design recommendation, we also could show that our design decisions did indeed support engineers with several of their analysis needs using AutobahnVis (most importantly, R-1–5, R-9 and R-10).
Even several months after conducting the studies, asking our participants revealed that they are all still using AutobahnVis on an occasional basis in a similar frequency as they used it during our studies (once or twice a week). One engineer stated: “I would use it even more often, but you know there are still these little technical restrictions [...] when these limitations will be overcome, I guess I would use it everyday!”. To make this possible, we recently transferred our final software to the Carmen tool developers who plan to overcome the very last technical hurdles and directly integrate our concept into the Carmen core tool and“not just as a plugin”.