Three-Day Sprints: Interviews, Generating Scenarios, and Prototype Testing

Once the initial strategic vision analysis and interview protocol are complete, the RMS intervention relies on a UCD approach to conduct the interviews, generate scenarios, and test a prototype. UCD is an approach to software development that started in the early 1980s (Ghaoui, 2005; Norman & Draper, 1986). The basic premise of UCD is that users are aware of their needs but rarely have adequate knowledge, skills, abilities, or understanding of the technical

possibilities to clearly translate their needs into technical requirements. Despite this, technical requirements are needed so that a developer knows how to develop the system. The craft of optimum UX, the end goal of UCD, is one of translation of system needs through design

UX methods) based on the assertion that collecting and analyzing data through direct

observations and interactions with users, and then testing and evaluating the outcomes, achieves the best translation. RMS relies on primary data to capture accurate user behavior rather than secondary sources of data. Primary sources come directly from observing, testing, and/or interacting with end users. Secondary sources are assumptions inferred from relevant facts that pertain to users. RMS assumes that systems that are designed predominantly from primary sources of information are more likely to meet user expectations and support user goals, thereby generating the least amount of risk for the hosting organization. The intervention part of RMS targets the acquisition of primary data sources through direct observations and interactions with users in the context of their system use.

The RMS method described here would best serve organizations with a more participatory orientation on the User Data Spectrum. As Chapter 3 explains, a participatory orientation seeks to design with the end user in designing and developing the technology, as opposed to a genius orientation that designs for the end user. Weaving the users’ words into the fabric of the technology design, as RMS does, is an inherently collaborative process. Thus, RMS seeks to have the users drive how the technology is designed.

RMS relies on direct observations and engagement with end users and the end user context of use. If the end users are in one location, then there will only be one site to visit; however, like most systems, if there are multiple site locations where end users will engage with the system, there will be multiple site visits. Sites should be selected according to which sites give the best representation of users. The number of sites is based on saturation; in other words, site engagements should continue until saturation is experienced, i.e., the data collected begins to appear redundant.

Each 3-day siteconsisted of six major steps: (a) preparation, (b) interviews, (c)

generating scenarios, (d) usability testing, (e) analysis, and (f) sharing findings with organization leadership and key stakeholders. It should be noted that a 3-day engagement to collect user requirements, create a prototype and test that prototype is unprecedented in common product development practice. The most popular model is a 2-week period for these activities with the most advanced companies. The activities of each step are outlined below.

Preparation. All on-site activities are planned and coordinated with the stakeholder at the site to be visited. If there are strategic artifacts that are specific to the site to be visited, those elements should be incorporated into the interview protocol. Then communication material is generated for the site stakeholders to educate them about the larger organization engagement and their role in that effort.

Interviews. One-hour, semi structured, face-to-face interviews are held with site stakeholders and end users. It is recommended to start with site leaders (formal and informal), and then move on to other end users and stakeholders. Most importantly, the end users engaged in the site visit must be representative of the final end users of the system.

Generating scenarios. After Day 1 of interviews, the UX team gathers notes taken during the interviews and generates the scenarios for that site. The team also examined the full organization list of scenarios and identifies whether the scenarios generated at other sites are applicable to the current site.

Usability testing. In 1-hr, face-to-face sessions, end users are asked to perform approximately five tasks with the current software and then five additional, similar tasks on a new prototype. Ideally, the same users who participated in the interviews participate in the

usability testing sessions; but this is not required if all users are representative of the final end users at that site.

Analysis. Usability testing data is processed and UX measures are compared with those from each site visited.

Sharing. A 1-hr meeting is held with the organization leaders and key stakeholders to share findings from the 3-day site visit.

Figure 6.1. RMS intervention process.

UCD emphasizes the use of prototypes as a design strategy. When requirements gathering remain only verbal, users often do not have an accurate account of their technology practices and create requirements that are either inaccurate or not grounded in behavior. Prototyping provides a tangible replica of a system, which the user may interact with, thereby providing more accurate and richer information for requirements gathering.

Given the short time window, which RMS is dedicated to supporting, it is important to select a prototyping approach that allows the UX team to interact with as many sites and end users as possible. The RMS prototyping approach combines Carroll’s scenarios-based design

method (Go & Carroll, 2004) with the rapid iteration testing and evaluation (RITE) method supporting rapid prototype iterations (Medlock, Wixon, Terrano, Romero, & Fulton, 2002). The combination of these two development methods involves taking user data, presented during user interviews, and quickly transforming it into scenarios, which supply design direction and

evaluation guidance. The combination of the RITE and scenario-based design methods aligns with the minimal time allotted for on-site visits and the short overall time for prototype development. This combination of the RITE method and scenario-based design method is the essence of this new strategy, called the Rapid Meaningful Scenarios (RMS) Method.

Scenarios depict users and their objective, motivation, challenge, and context (Carroll, 1995). Scenarios allow a system development effort to be scalable and best serve the users in the dynamic environment in which the system is to be deployed. After each interview, several scenarios are generated by the UX team. Each scenario contains at a minimum a primary actor (in the CIRTL case, for example, a graduate student), a task that the actor wants to accomplish (e.g., connecting with other graduate students), and an outcome (such as having a conversation). Applying RMS in other contexts would require the identification of different actors, tasks, and desired outcomes. The UX team members who create the scenarios in this use case followed the basic scenario creation principles with the three main elements of actor, task/goal, and outcome. However, the CIRTL experience led, upon deeper analysis, to additional considerations were incorporated into the creation of the scenarios. These additional considerations were uncovered when one member of the UX team interviewed another member of the UX team to assess the mental model used in the scenario generation process. The following two additional

First, when hearing an end user’s responses to the interview questions, the team considered the relationship that the end user had to the organization and the mission of that organization. Questions asked by interviewers during interviews, such as:

• Is this person an important hub in the network/part of the strategy?

§ What is his or her position in the local and larger organization? What is his or her role? Whom does he or she influence in the hierarchal structure and information organization structures? Whom does he or she interact with, and how often?

§ Which end user(s) can this person adequately represent? (The answer to this question will determine the actors who can be used in the scenarios created from the interview with this person.)

§ Is he or she the center of social hubs? Is he or she at the top of the hierarchy? Whom is he or she in charge of? (As a rule of thumb, for enterprise-wide technology solutions, scenarios created from those closer to the end users’ position in the

organization structure tend to be more valid than those created from employees who reside at the top of the organizational structure.)

• What are the user challenges? How can those challenges be turned into opportunities to show worth and promote technology penetration into the organization?

Once these initial questions are considered, the second consideration focuses on liberties taken need to be addressed. A liberty in the case of RMS is anything that the researcher creates or any assertion on behalf of the user that did not come directly from users. If the liberty taken influences the system functions, then that liberty should not be taken. If the liberty taken

influences the business strategy, then it should be taken. For example, in the case of CIRTL, one of the participants expressed the need to be able to find resources on the CIRTL network site that

would help her be a better educator in her class. This expressed need translates to a resource section on the site and is an example of a request for a system function. Therefore, liberties about what the user wants to do with the resources, how the resources should be structured, and the relationship of the resource-seeking behavior to other site behaviors should not be taken. On the other hand, one CIRTL leader expressed concern about needing to share the value of being a part of CIRTL with her fellow colleagues. Liberties about how to illustrate the value of CIRTL (e.g., creating promotional materials like data visualizations) should be taken. Decisions about system functions are within the skill set of the UX design team; therefore, the risk of taking liberties with user words is minimal. However, taking liberties about the content and setting the tone of the virtual presence of the organization lies beyond the skill set and domain of the UX team. It is critical that the organization establish the tone and create the content without injection of bias from the UX team. After all, the organization leadership sets the boundaries for the business strategy.

Scenarios are used to inform design. Knowing what users want to do, how they want to do it, what order they want to do it in, and what context they are working in allows the UX team to design, develop, and test iterations of a prototype.