Initial Beta Design User Testing

The usability testing protocol is presented in Appendix A. It consists of an introductory period where the system and testing procedure was explained. Instead of using the user-generated scenarios as

previously done, videos of real intrusions captured on camera by MnDOT were used. The volunteering supervisor would watch a video of an intrusion in a Minnesota work zone, and fill out the report using the beta system. The time to completion for reporting the intrusions in the first two videos was recorded to measure both intuitiveness and learnability of the beta. During the third video report, the supervisor was asked to verbalize his or her thoughts on the beta system, to assess preferences and mental processes as he or she used the system. Afterwards, three questionnaires were administered, including the System Usability Scale (Appendix B), the Rating Scale Mental Effort (Appendix C) and a researcher designed questionnaire titled “Intrusion Form Survey” about the beta system (Appendix D).

Finally, a series of free-response questions were asked (Appendix A). The testing procedure was iterative, therefore, after every testing session, the researchers would evaluate the results and update the design of the beta system accordingly.

For the videos, the researchers used camera footage supplied by MnDOT. The three videos used different work zone types (i.e., ramp closure, shoulder work), two on Interstate 35W South, and one on Highway 36. The first intrusion was minor, involving no risk to any crewmembers. The second intrusion was moderate in the risk to crewmembers, while the third was quite high regarding the risk level to the maintenance crew on site. Researchers tested four potential users of the intrusion reporting form. The supervisors reflected urban and suburban areas in Minnesota, and worked day and night shifts.

4.1.1.1 Initial Beta User Testing Results

The average time for completion of the first report was 6 minutes and 18 seconds. The average time for completion of the second report was 3 minutes and 9.3 seconds. The first report time reflects moderate to high intuitiveness of the beta interface. While ideally it would initially take under 5 minutes to complete, taking a little over six minutes reflects an acceptable time investment. Furthermore, the learnability of the beta system was high, as the second time to complete was considerably lower at approximately 3 minutes. Reporters easily figured out the system and rapidly filled out the report with the beta system on their second use.

The System Usability Scale (SUS) scores of the interface was fairly good, as scores between 60 and 70 represent average system usability, and the average SUS scores for the beta was 86.875 (SD = 13.288).

RSME scores measuring self-reported mental effort reflected an average of 36.25 (SD = 7.5), which indicates a degree of mental demand between “some”, and “a little”, which reflects satisfactorily low demand imposed by the beta.

The Intrusion Form Survey averages are reported in Table 4.1, with the first three questions scored on a 1 (Strongly Disagree) to 5 (Strongly Agree) scale, and the last question scored on a multiple choice (7-point) option of: Worst Imaginable, Awful, Poor, Ok, Good, Excellent, and Best Imaginable.

Table 4.1 Intrusion Form Survey Scores

Average Standard Deviation

Information Entered Accurately 3.75 0.50

Confidence While Using Form 4.50 0.58

Interface Has Annoying Features 2.25 0.50

Overall User-Friendliness 5.50 0.58

As reflected in Table 4.1, supervisors were moderately confident in the accuracy of the information entered in the form, and highly confident in their ability to complete the form. Most users reported relatively low annoyance with the form, although there may be room for improvement in later designs.

Finally, overall user-friendliness was highly rated, the average score somewhere between “Good” (score of 5) and “Excellent” (score of 6).

The most popular aspects of the interface were the drop-down menus and the comprehensiveness of the form. This allows for quick use of the form and the ability to enter in and report most useful points

of data about a work zone intrusion. The least popular aspects was whether it would be reasonable to use the form for minor intrusions that supervisors did not consider a risk to crew, especially if the supervisor could not fill out a form on site with a portable electronic device (e.g., iPad, tablet). Multiple supervisors noted their discomfort with reporting whether they thought the intruding vehicle was committing a speed limit violation, as they did not feel confident they would be able to report speed limit violations accurately. This issue may need to be revisited. Some supervisors also reported wanting a clear explanation of the rationale for the form when the final version is rolled out, to provide

motivation for filling out reports on work zone intrusions, especially minor ones.

4.1.1.2 Initial Beta Revisions

There were several changes and adjustments made to the beta from its original design. A number of these changes are described in the following Table 4.2.

Table 4.2 Minor Initial Revisions to Beta Design

Website

The initial revisions to the beta design were implemented prior to a significant revision. The layout of the initial revisions are presented in Figure 4.1 through Figure 4.6.

Figure 4.1 Revised Beta Interface, First Screen.

Figure 4.2 Revised Beta Interface, Second Screen.

Figure 4.3 Revised Beta Interface, Third Screen.

Figure 4.4 Revised Beta Interface, Fourth Screen.

Figure 4.5 Revised Beta Interface, Fifth Screen.

Figure 4.6 Revised Beta Interface, Sixth Screen.