How to Run an End User Feedback Session on Your Product

(1)

1

Teena Singh, Principal Product Manager, HCM User Experience Lulit Bezuayehu, User Experience Architect, HCM User Experience

(2)

2

2-Hour Us er E xperience Works hop

Hands-On Session

• Steps for running a user feedback session

• Example materials to help you run your own usability study

• Do’s and don’ts of moderating a feedback session Key Takeaways

• How to collect valuable user feedback on your application

(3)

3

(4)

4

UX Direct FREE Toolkit: Usability Tools

 How to conduct usability testing

 Usability Testing Fact sheet

 How to conduct Heuristic Evaluation

 Usability Rating Scales

(5)

5

(6)

6

(7)

7

Question 1: Usability Study Experience

What is your experience with usability testing?

A. Ran a usability study

B. Was a Participant for a usability study C. Observed a usability study

(8)

8

What is Usability Testing?

• Usability: the extent to which a product can be used by specified

users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use. (Source: ISO 9241-11) • Usability testing is a technique used to evaluate a product by

(9)

9

Definition of Usability testing

Essential features

• Measuring

a) Efficiency – rate/speed completing a task

“Efficiency is doing things right”

b) Effectiveness – ability to complete the task

“Effectiveness is doing the right things”

c) Satisfaction of users - delight

• Gives direct input from real users

(10)

10

Benefits of Usability testing

ORACLE CUSTOMER

Do it right the first time. Save time & cost of implementation Reduce costly errors Less customer & Oracle liability Understand user patterns & habits Users less likely to complain Improves the product Increase customer satisfaction.

End-user completes tasks faster Fewer errors- managers are happier. More Oracle product sold Increased customer satisfaction & loyalty Compliance w/ gov. regs, contracts,

unions-

Avoid fines & penalties Decrease customer support, training, &

docs

Cheaper total cost of ownership for customer

(11)

11

(12)

12

Discount Usability

• What is Discount Usability: An approach to usability

that seeks ways to optimize usability methods for

cost-effectiveness.

- User Feedback Session is an example of Discount Usability Data Collection Method

- We will focus on this methodology as an example and give you the tools and knowledge to run this kind of usability activity

(13)

13

What is a User Feedback Session?

• Task-based

method for evaluation of products at an

early stage of development

• Conducted with a small group of participants from a

target user group

• Used to validate designs

• Performance data is typically not gathered for this kind

of evaluation

(14)

14

What resources do you need?

• Required:

- A computer connected to the website, prototype (even paper-based) or code you are testing

- A well thought out script of tasks and scenarios that you will ask the user to complete

- A Moderator to ask questions and request tasks

- MOST IMPORTANTLY: An end user participant to provide feedback

• Recommended:

- A note taker to take notes

- Resources to either record audio or video of the test to record responses and reactions and analyze website navigation

(15)

15

What resources do you need?

• Specify key characteristics of user types when you recruit

• Recommended: 8-10

• Usability Study for HCM: Time and Labor:

• 5 employees

• 3 managers

• Provide an incentive

(16)

16

Non-Lab User Feedback Sessions

• The method is flexible with respect to:

- Where the study is conducted, (in the lab, at a customer location, at a trade show, or through remote collaboration tools).

(17)

17

Question 2: User Feedback Session

Which resource is NOT REQUIRED to run a user feedback session? A. Well thought out script of tasks and scenarios

B. Computer connected to the website or prototype you are testing C. Moderator

D. Participant E. Note taker

(18)

18

(19)

19

Planning Your User Feedback Session

- Step 1: Determine Design Focus

- Step 2: Assemble your Moderator Script - Step 3: Create Click Path for Prototype

- Step 4: Review Moderator Script with your Project Team

(20)

Step 1: Determine Design Focus

• 1: Identify key scenarios and tasks that you want to validate with customer

• 2: Select prototype to show customer based on task frequency, importance, priority, etc.

• 3: Determine areas of the prototype that you have design questions about or feature/functionality that you need to verify

• 4: Translate the design-focused questions/areas into tasks and questions for participants.

(21)

Step 1: Determine Design Focus

• Important features to test are typically

– New

– Used often

– Problematic, based on customer’s feedback

– To gain consensus on design direction

– Likely to cause the product to fail if used incorrectly

– Commonly used job tasks

(22)

Step 1: Determine Design Focus

• Identify key scenarios (or high level use cases)

– Talk through the key scenario that you will describe to your customers. – Make it simple. A quick story that you can always refer to when showing

your prototype.

• List core tasks that your prototype supports

– Determine which tasks you want to validate! • You won’t be able to do all of them.

– Which tasks are the most important or frequently completed by users?

(23)

23

S tep 2: As s emble your Moderator S cript

• What is the Moderator Script?

- A guide that helps to steer the conversation

• A cheat sheet for the discussion with the customer

- Outlines the test protocol

- Includes the Participant scenario and task list

- Includes all follow up questions you want to ask each participant

- Helps you ask all customers the same questions so that you have a basis for comparison.

(24)

24

Step 2: Assemble your Moderator Script

Guide format

• Section 1: Introduction/Introductory Script - Test Objective

- Outlines the process for the Participant

• Section 2: Describe the overall background scenario

• Section 3: First task - Task Scenario

- Task-specific questions

• Section 4: Second task

• Section 5: Third task, etc.

• Section 6: Any overall or final questions

(25)

Step 2: Assemble your Moderator Script

• The introduction to the session should contain the following elements:

– Introduce yourself, your role, and any additional team members attending

– Have participants introduce themselves and their roles

– Set expectations by describing the format of the meeting and what will be expected of participants

• Let participants know that rather than just show them a demo we will have them “test drive” the design by describing how they would attempt to complete the tasks

• Let participants know that you will be asking questions after each task about what they think of what they are seeing

(26)

– Emphasize that we are looking for ways to improve the design by having customer reviews and we want their honest feedback

– If recording the session, informed consent is required. You must ask their permission to record the session

– You must read or show the Safe Harbor Statement to participants – This is fully scripted. See template.

Step 2: Assemble your Moderator Script

(27)

• Why set up the scenario?

– To provide context for what the participants are about to see – Allows you to refer back to the scenario throughout the session

• Initial scenario set-up

– Verify user profile(s) and scenario with customer

– Describe the scenario that lead the end-user up to the first step of the flow

– State any assumptions or dependencies relevant to this flow

Step 2: Assemble your Moderator Script

(28)

• Overall scenario should focus on the user role

– Write the scenario in a way that reflects test participants’ real life work situations (e.g. employee name, employer name, location)

• A good scenario is

– Clear (all participants will understand it)

– Short

– Include relevant information

– Gives background and context to the whole script (and not the first task only)

Step 2: Assemble your Moderator Script

(29)

29

Sample Background Scenario: You are Albert Katz, a Quality

Assurance (QA) Engineer with six years of experience. You aren’t happy with your current position, so you are thinking about

changing jobs. You have heard great things about Vision

Corporation, and have navigated to the Vision Corporation careers page from a link on the company’s website.

Step 2: Assemble your Moderator Script

(30)

• Tasks need to be focused on one specific feature or a

cluster of features

– Reasonable

– Represent a realistic situation

• Write the task in a way that is related to the test participant’s work

– Specific

• Avoid using terms that exist on the interface (don’t lead)

– Sequential (maintain appropriate flow)

Step 2: Assemble your Moderator Script

(31)

31

S ample Tas k S cenario: You want to see if Vision Corporation is currently hiring QA Engineers. You are willing to relocate, so you want to find out about QA Engineer openings in San Francisco.

- S ample Tas k 1: Describe how you would find QA Engineer jobs at Vision Corporation in San Francisco.

Step 2: Assemble your Moderator Script

(32)

UI Layout

If you want to know… Ask…

If participants correctly interpret the purpose of the page based on the UI layout

What is the primary task to be accomplished on this page? What is the first thing you would do on this page?

If users can find features on the page If you were to do [task], where would you go?

Step 2: Assemble your Moderator Script

(33)

Is important information missing?

if the design provides all the information users would need to complete typical tasks

Would you have all the information you need to complete [task]? If not, what else would you need?

Are there any analytics or metrics you would like to see on this screen?

Validation of new features

how a new feature would be received by customers

Would this feature be useful to you? Would you use this [new feature]? How so?

Step 2: Assemble your Moderator Script

(34)

Information Architecture

if data is categorized in logical groupings Do these groupings [point to a set of items] make sense to you? If not, how would you want them to be grouped? What’s missing?

Navigation

do participants understand the navigation structure

How would you expect to get from here to the next page? How would you launch [this

feature] from here? if participants have the navigation options they

would need

What other tasks or parts of the application would you want to have quick access to from this page? Are there related tasks that you would go to from here?

Step 2: Assemble your Moderator Script

(35)

Terminology

if the terminology used makes sense and is intuitive

What is the typical wording you would use for [concept]?

Mental Model

if the design matches the user’s mental model What would you expect to see next after [taking an action]?

Step 2: Assemble your Moderator Script

(36)

Expectations of how UI elements and widgets work

if users understand how to interact with UI elements/widgets

What would you expect this <UI widget> to do?

Step 2: Assemble your Moderator Script

(37)

• Ask any questions overall or final questions that you

might have

- Ask each participant the same questions

- “What did you think of the Prototype overall?”

- “Can you tell me your top likes and dislikes of the designs you saw today?”

- “How does this compare to what you are currently using?”

Step 2: Assemble your Moderator Script

(38)

• Is there anything we missed that you want to talk about

• About what you saw today

• Or the product overall

• Thank the Participant

- Give them a gift/incentive for their time

Step 2: Assemble your Moderator Script

(39)

• Background Scenario: You are Paul

McCartney and often to go Amazon.com to purchase household products.

• Task Scenario: He recently purchased a new

digital camera on Amazon.com and you are disgusted with it.

- Task 1: Visit the uniform resource locator for Amazon and see if there is a link for “Returns are Easy.” Hint: The link is towards the bottom of the page.

- Task 2: Determine the number of total products that Amazon sells.

Quiz

(40)

• The Scenario and tasks were not written using the “you” perspective

• Name should have no bias

• Strong phrasing like “disgusted” detracts from the task

• “Uniform resource locator” is using technical jargon/language that an end user may not understand

• Providing too much information

• Link name “Returns are easy” gives away answer to user

• False scenarios not recommended

Discussion Points (Results)

(41)

• Create a “Click Path” document

– Identify the actions the user should take to successfully

complete the tasks and what the state of the prototype should be at each point along the way.

– Share the click path with the Note Taker. It is a good tool to help the Note Taker stay oriented as the participant works through the tasks.

(42)

- Actions to successfully complete the tasks

- Documents the state of the prototype/application as the user clicks through it

- Share the click path with the Note Taker

- Helps the Note Taker stay oriented

(43)

• Solicit feedback on Moderator Script with project team.

– Ask your Project Team if they want to add any task-based or overall questions

– Finalize Moderator Script

(44)

• Pilot the session once or twice with an internal Oracle

employee, not involved in the task creation process to

get an idea of the session length and clarity of the tasks

• Based on Pilot Test finalize task and questions in your

Moderator Script

(45)

45

(46)

46

Conducting User Feedback Session

• Set up testing area

- Have the prototype ready

- Make sure your recording device is ready

- Have the Participant Tasks and Scenarios printed out and ready

• Prepare yourself mentally for the test

- Try and get a good night of sleep

- Review Moderator Script one last time

- Remember to: Be ready to expect the unexpected

(47)

47

• Meet the participant

- Relax, introduce yourself, make small talk and help the Participant feel at ease

- Make eye contact, thank them for coming - Offer the Participant a refreshment

• Transition to the testing station

- Explain how the station is setup

- Position yourself and the Participant

Preliminary Details

(48)

48

• Allows you to go systematically through details

- Introductions

- Outlines the structure of the test and general protocol

• Test Objective

• Procedure

- Explain and demonstrate “thinking aloud” - Advise the Participant to be Honest

- We are not testing the participant but rather the design - Ask them if they have any questions

- Confirm if you have permission to record

Go through the Introductory Script

(49)

49

Conducting User Feedback Session

Introductory Script: Explain Thinking Aloud Concept

• As you work with the product, we would like you to

‘think aloud’ about what you are experiencing,

what you like and dislike about it, what your

expectations are, and what you are trying to do

with the prototype. Please continue working as

you think aloud however; don’t stop working to

provide comments. I will give you the opportunity

to provide more detailed comments at the end of

each task.

(50)

50

Conducting User Feedback Session

Thinking Aloud Concept: Advantages

• Preference information in real time

• Improves focus and concentration for participants

• Clues about why something is confusing

(51)

51

Conducting User Feedback Session

• Not a common behaviour

• Increases mindfulness

(52)

52

Conducting User Feedback Session

• Double check to see if your Note taker is ready

• Ask Participant to Read Background Scenario aloud

• Ask participant to read first task aloud

• Confirm with the participant that they understand the

task

- “Could you go ahead and summarize the task please?” - “Do you have any questions about what the task is asking

you to do?”

(53)

53

Conducting User Feedback Session

• Remind the Participant to Think Aloud

- “You may go ahead and start the task.” - “Remember to think aloud”

• Ask the Participant to tell you when they feel they have

completed the task

- “Do you believe you have completed the task?”

• Moderate only when Necessary

- “That is helpful feedback”

• Ask Task specific Follow-up Questions

(54)

54

Conducting User Feedback Session

• Ask any questions overall or final questions that you

might have

- Ask each participant the same questions

- “What did you think of the Prototype overall?”

- “Can you tell me your top likes and dislikes of the designs you saw today?”

- “How does this compare to what you are currently using?”

• Thank the Participant

- Give them a gift/incentive for their time

(55)

55

Conducting User Feedback Session

• Tests should be scheduled so there is at least a half

hour break between sessions

• Get away from the test area, if only briefly

• Reset the prototype or application

• Save the recording and notes

• Prepare for your next participant and test

(56)

56

Question 4: Usability Study Experience

Poll 4: The think aloud methodology enables the Participant to give you real time preference data.

True or False

(57)

57

Do’s and Don’ts of Moderating

a User Feedback Session

(58)

58

Guidelines for Moderator:

Be Unbiased/Impartial

• Give the same feedback for negative and positive

comments on the product

- “That is helpful feedback. Thank you.”

• You don’t have to answer every question

• React to “mistakes” in exactly the same way as

(59)

59

Guidelines for Moderator:

Let the Participant Speak

• Minimize interruptions

- Participants should be speaking 80% of the time - When in doubt, wait before saying anything

• Saying nothing is still communicating

• Your response should be unbiased if the

Participant makes a comment on the application

- “That is helpful feedback.”

(60)

60

How to Deal with:

Participant Frustration

• Since frustration is usually related to a critical deficiency of the product, if you move on too quickly you can miss an opportunity to understand how the Participant learns to use the product

• The key is to find just that point where the Participant is frustrated but still willing to try

• Encourage the Participant to verbalize what is happening. Often, if the Participant is allowed to vent while performing the task, their frustration can be minimized

“I can see you are getting frustrated, but would you please try a bit longer.”

(61)

61

Guidelines for Moderator:

Be aware of the effects of your voice and body language

• Never indicate through your speech or mannerisms that you approve or disapprove of any actions or comments offered by a particular

Participant

- Moving closer to someone indicates acceptance of what the person is saying, moving further away indicates rejection

- Raising the pitch of your voice usually signifies agreement, while lowering it communicates the opposite

• Record your pilot test so you can observe/listen to your behavior Remember: Its good to see Participants have problems with the application during testing situation. It provides and opportunity to identify and correct issues in the design.

(62)

62

Guidelines for Moderator:

• Very often, especially if a Participant is unsure of a task, there is a moment of indecision after completing the task when a Participant is not sure if he or she has completed the task correctly

• If you jump in too soon, because the Participant is finished, you are confirming that they completed the task correctly and might be

cutting into their moment of indecision

• If you are sitting close to the Participant be careful not to mark the task as completed which can be an inadvertent signal to them

• The best way to prevent this problem all together is to have the Participant signal when they are finished as part of the protocol