Assignment # 4 Summing up the Course

(1)

Assignment # 4 Summing up the Course

Usability form an Industrial Prospective (DV1301)

Ch. M Nadeem Faisal Muhammad Asim Farrukh Sahar Rana Asif Sattar

790102-P373 811115-P373 800102-P373 840715-P177

[email protected] [email protected] [email protected] [email protected]

Ms Computer Science Students

Blekinge Institute of Technology, Ronneby, Sweden

ABSTRACT

This is a report over our study of “summing up the course”. We started by doing what we have learnt from the previous course elements. Then we compared the testing methods in sense of similarities/differences and identified the stakeholders’ needs to drive the usability work. The report also discuss the usability test as a concrete example, where we used students as test subjects in an experiment to evaluated the usability of “Gmail web email service”. The findings of the testing are presented in suitable form according to the concerning stakeholders.

1.0 Introduction

According to ISO 9241-11:1998, Guidance on usability: “Usability is the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use”. Usability evaluation is the usability analysis of a prototype or system. Its objective is to provide response in an iterative software development process. According to Nielsen, the most fundamental usability method, which is in some sense irreplaceable, is usability testing. This is so since it provides direct information regarding people’s use of computers, and of their problems with the system performance that are tested [1]. Usability tests can be useful in a competitive marketplace. They can be performed on competitor’s products that are already on the market, either in isolation or in comparison to a version of your own product, in order to understand their strengths and weaknesses, and to help your product meet user’s needs better than the competitor does. They can also be used to make design decisions, by testing different proposed design solutions [1].

Section 2.0 described the concise reflection of our learning

from the previous course elements about concepts and definition, methods and usability testing. Section 3.0 compared testing methods in sense of similarities/differences and selected a method which was suitable according to testing requirements. Section 4.0 described the stakeholders’ needs for the findings to drive the usability work. Section 5.0 described concrete example, as a usability testing in which we used students as test

subjects in an experiment to evaluated the usability of “Gmail web email service”.

2.0 Reflection of Previous Study

We learnt from the previous course elements a lot but due to the words limit constraints a concise reflection of learning is being presented.

Usability is basically a measure for how easily users can use the product to achieve their goals by performing some particular set of tasks. Usability concepts set the measures against which we measure the usability of any product in relevant industrial area. There are many industrial concepts in the area of usability like memorability, testability, observability, familiarity, understandability, reliability and many more. We consider the usability concepts according to industrial area and goals which we need to achieve. For example, to measure the usability of “Gmail web email service” we used effectiveness, efficiency and satisfaction usability concepts. Usability test is a part of larger research series that involve preparation and analysis. Different methods are used to achieve the research series [2]. Methods include recruiting and interviewing, user profiles, contextual inquiry, task analysis, focus group, competitive research, usability tests, surveys etc. Multiple methods can be used in usability evaluation of any system. For example, to select the candidate for testing you may need recruiting, interviewing, user profiles methods.

Usability testing is technique to ensure the product is easy to learn, satisfying to use and contains the functionality that a user wants. The usability testing can be divided into three categories: inquiry, inspection and formal of usability testing [2]. Usability testing involves different testing methods according to goals and needs. Testing methods includes coaching method, co-discovery method; think-aloud method, remote testing, teaching method etc. Detail descriptions about the usability methods are given below. Test results are presented in suitable form to the concerning stakeholders.

3.0 Usability Testing Methods

In usability testing users are systematically observed as they perform tasks [1]. Usability experts can observe the users directly while they perform their tasks. Battleson, B.,

(2)

Booth, A., and Weintrop, J. et al. claim that usability testing is the most effective way to asses a website's usability [3]. They proposed three categories of usability testing: inquiry, inspection, and formal. Various methods of usability testing have been proposed in literature.

1. Coaching Method 2. Co-discovery Learning 3. Performance Measurement 4. Question-asking Protocol 5. Remote Testing 6. Retrospective Testing 7. Shadowing Method 8. Teaching Method 9. Thinking Aloud Protocol

Web industry mostly uses five usability testing methods to evaluate the web site.

1. Remote Usability Testing 2. Coaching Method 3. Co-discovery Learning 4. Performance Measurement 5. Think Aloud Protocol

Testing Methods Similarities/Differences [4]

U.T Methods

Usability issues covered Conduct Remotely

Quantitative data Effectiveness Efficiency Satisfaction

1 Yes Yes Yes Yes Yes

2 Yes No Yes No No

3 Yes No Yes No No

4 Yes Yes No No Yes

5 Yes No Yes No No

Remote Usability Testing

The idea of conducting RUT emerged ten years ago [5]. This method is very different method with other methods because in this method user is not physically present to perform the test tasks.

Advantages: According to [6], RUT is comparatively

cheaper and wider reach.

Disadvantages: According to [6], RUT has security issue,

performance issue and limited visual feedback.

Coaching Method (CM)

In CM, usability expert works as a coach. In this method users are encouraged to ask questions from the usability expert [7]. Usability expert gives the instructions to user when s/he feel problem or answer any query while performing the test task.

Advantages: According to [7], CM builds the user’s

involvement in the evaluation and users learn more quickly.

Disadvantages: According to [7], CM relatively time

consuming and coach has to do dual job i.e. coaching and evaluations.

Co discovery Method (C-DM)

In this method, two participants work together and express their thoughts while exploring a product’s interface. This method discovers how participants communicate with each other to complete test tasks. [8]

Advantages: According to [8], C-DM is more structured

and explorative technique and good for learning aspects of usability.

Disadvantages: According to [8], C-DM is required careful

candidate screening and verbalization of users can distract exploration.

Performance Measurement (PM)

In this method, usability expert gets the quantitative data of the participant’s performance when s/he performs the test tasks. Quantitative data is very useful in doing comparative testing, or testing against predefined benchmarks. [9]

Advantages: According to [9], PM provides quantitative

data; Due to quantitative data the prediction is comparatively easy as compare to other UEM’s

Disadvantages: According to [9], users are needed to act

naturally in unnatural environment. Interaction between user and tester is prohibited.

Think Aloud Protocol (TAP)

In TAP, users are asked to speak their thoughts as they perform a task [7]. The user is instructed to speak aloud their thoughts, assessment that they make, any misunderstanding, their reactions while interacting with the product. Usability expert understand how the user approaches the interface and what considerations the user keeps in mind when using the interface [10].

Advantages:

Leads to direct design solutions. [7]

Verbalization gives the answer of both what and why problem arises with interfaces. [7]

Rapid, high-quality, qualitative user feedback (e.g. as compared with questionnaires). [11]

High degree of flexibility. [11]

The presence of two people allows meaningful, direct dialogue. [11]

Disadvantages:

Participant has to perform two tasks. i.e. doing test and verbalizing what they are doing. [7]

Verbalization can interrupt the task that is being performed by users. [7]

Usability is basically a measurement for how easily users can use the product to achieve their goals by performing some particular set of tasks.

(3)

4.0 Stakeholders Requirements

Information need of CEO: [12]

Information need to develop the overall vision of the project.

By using Information, CEO can look into the future and predict changes as well as project solutions.

Information regarding the changes in environment and how various functions can organize to deal with these foreseen changes.

Information used to set performance standards for the company

Information needs to determine what standards are required to maintain competitive advantages in the industry.

Information needs of interaction designer:

Information regarding user needs and customer goals to produce innovative, engaging and effective designs [13].

The information requires in designing the information on a web page, tasks flows and user strategies for complex applications such as making a Gmail account [13].

Information regarding company’s clients and their competitors such as “yahoo mail.com” and “Hotmail.com”, to gain competitive advantages [14].

Information about the new standards, technologies and trends, to formulate strategies as well as plans for the future[14].

5.0 Usability Testing

After the literature review, authors planed a usability test to evaluate the “Gmail Web email services” and “Performance Measurement” method is found most suitable and selected to perform the usability test. [9]

Purpose

The purpose of usability test is to determine how effective, efficient, and easy to remember the “Gmail Web email services” are in terms of “Registration Gmail”, “Composing a mail”,” label” and “Auto responder”. [9]

Problem and Test Objects

We are performing the usability test on the “Gmail Web email services” to find the results for “Chief Executive Officer” (C.E.O) and “Interaction Designer” by testing the different objects.

Registration Gmail Composing a mail Check the mail Delete the mail Contacts

Sorting mails in Inbox Labels

Auto mail responder Mail attachment

The focus of this testing is the user response against the tasks. The test team decided to narrow down the list of issues. The tests were conduct to find the quantifiable and verifiable data. The pre and post questionnaires as well as scenario are given to the novice and advance users. The tests were performed in a full scale usability lab. The novice users were given 22 minutes to complete the two task and 41 minutes given to the advance user to complete the all four tasks. We proposed this time for tasks on the basis of experienced testers.

Task Advance User Beginners Importance

Registration x x x Composing a new mail x x x Labels x Auto Responder x

Material

The material we use in usability testing. Set-up sheet

Task lists

Observation sheets Questionnaires

User profile and Recruitment

Authors selected 6 students of BTH randomly according to their convenience. They were familiar with the different functionalities of “Gmail Web email services”. We divide the students in two groups the advance or the expert users and the beginners or the novice users.

Location

The whole usability testing processes will be performed in our room. There should be some schedule for the both advance or the beginner users to participate during the testing processes. [15]

(4)

Equipment

Pentium 4 computer with Microsoft windows professional

Some other material like the lists, notepad and check list for recording the user’s comments.

Minimal the Risk

Before starting the usability test minimize the number of risk that face by the participants in the processes of usability testing. So they perform the test with better intention.

Usability Test Approach

We are testing the performance of Gmail so we select usability laboratories or our room to perform all the tasks, this laboratory consisting of e.g. a living room or office-like area connected to a monitoring area with a one-way mirror whenever the participant perform the test the evaluator [16].

Scenario Diagram [17]

Test designed for Novice

We selected important feature for testing scenario

a. Scenario one: Make the registration for the new users

b. Scenario two: Compose the new mail

Novice Scenarios / Task List

Time allotted 12 minutes Scenario one: Registration

Task: during this task the novice have to signup on the

Gmail website to create the new account for the email services and the online help or feedback is provide on the page that help them who to create the new account.

Time allotted 10 minutes

Scenario three: Compose the new mail.

Task: during this scenario the users has to compose the

new email message and send it to the some on person.

Test design for advance users

a. Scenario one: Make the registration for the new users

b. Scenario two: Compose the new mail c. Scenario three: Create Labels d. Scenario four: Auto Responder

Advance User Scenarios/Task List

Time allotted 12 minutes Scenario One: Registration

Task: during this task the novice have to signup on the

Gmail website to create the new account for the email services and the online help or feedback is provide on the page that help them who to create the new account.

Time allotted 10 minutes

Scenario Two: Compose the new mail.

Task: during this scenario the users has to compose the

new email message and send it to the some on person. User also has to attach the file.

Time allotted 5 minutes Scenario Three: create Label

Task: During this task the advance user has to label mail

with specific name to find them again and easily see in specific area means to sort the mail by label label.

Time allotted 14 minutes Scenario Four: auto responder

Task: During in this task the advance user has to create a

auto response message in setting page regarding the new email. Auto responders are feature that automatically acknowledge the message to the email sender. When the user receives the mail form a person the auto responder automatically send them an acknowledgement mail.

Novice and advance user pre test questioner

There are some basic questions that we are going to ask to the basic and advance users. These questions helped us in the evaluation processes.

Does there Gmail member use the internet. What is their opinion about the use of internet

(5)

Specially what they think about the Gmail previously

Impression about the speed of Gmail Their recommendations for the other users

Novice and advance user post test questioner

There are some post questions for the users. Such as what kind of difficulties they face during the test, their requirements and suggestions for improvement.

Suggestion about the updating

Recommendation for other users because it is free service

What tasks you can easy to complete

What did they found most difficult to perform What are the three things you like best about the email service during the whole processes

Evaluation

Our whole evaluation is based on the qualitative and qualitative, that we observe and make during the testing processes about the “Gmail Web email services”.

Quantitative Data

Most of our observation is based on the quantitative data. That data we collected during the observation on the “Gmail Web email services”.

Total time for each task, each task have the constant time for standard as we conclude it by many time making the experiments on the Gmail, but still there is no standard due to system, speed problem.

Most of the participants complete the task within time but the novice users takes more time then the advance users.

participants complete the tasks with extra time because of some problems within the system error or some other reasons

2 participants face problem during the whole testing processes

Total number of errors

Total number of problems on local system or any other problems

Task that are not performed by the novice users

Qualitative Data

There expression when they are using the Gmail mail services

Verbal comments during the testing processes by the participant

Interview before the test (pre and post test questionnaires or interviews)

Interviews after the test Other activities

Understand ability about the interface

Learnability

Simplicity of the interface while interaction Structure also has effect on the Gmail usability because it is overall sketch of the Gmail site.

Report for C.E.O

Description of the usability study in the whole processes we discuss and make practice in the Gmail email services. Our goal is to make a performance base test on Gmail. To find out the problems that users are facing while using Gmail web services for the sake of improvements. In this regards we select some important feature of the Gmail for our test the tasks and objects that we have tested:

Registration Gmail Composing a new mail Labels

Auto responder

Description

We did the performance measurement and other methods that will be supportive to enhance the standards of “Gmail Web email services”. The outcomes of this study that we have found during the whole usability process gave a suitable stuff for future predictions and suggestions for enhancements.

Information about the changes in user preferences when they are using different “Web email services”.

The reliability and satisfaction that the users found while using “Gmail Web email services”.

The different usability attributes such as effectiveness, efficiency etc, that are used to maintain the competitive advantages in the “Web email services” industry.

Quantitative Results of Performance Measurement

Total time is = 22 minutes Average time = 19 minutes

(6)

Average time = 20 minutes

Report for Interaction Designer

Information regarding user preferences while designing labels, registration forms and compose email feature.

The information that requires in designing the information on a web page, such as designing the registration form , compose email tasks flows and strategies for complex applications such as making a Gmail account.

Information regarding company’s clients and their competitors such as “yahoo mail.com” and “Hotmail.com”, to gain competitive advantages. Information about the new web email services standards, technologies and design trends, to formulate strategies as well as plans for the future.

Quantitative Results of Performance Measurement

Average time for each Task that Performed by Novices

Average time for each Task that performed by Advance users

Checking criteria

We checked the simplicity and consistency, by observing that is the registration form is simple for clear understanding to the participants, is all the parts of the form are consistent, doest it provide the feedback to the users, is it easy to navigate in all parts, and the interface is interactive.

REFERENCES

[1] Usability Literature Review, ORC Macro And QRC Division of Macro International Inc, 2000, available from Internet

sbh.orcmacro.com/Revised-Usability%20Literature%20Review.pdf, November 2007

[3] Battleson, B., Booth, A., & Weintrop, J. Usability testing of an academic library Web site: A case study. Journal of Academic Librarianship, 27, 3, 2001, 188-198. [4] Jakob Nielse's website http://www.usabilityhome.com

last visited 22 may, 2008

[5] M. Andresen , H. Nielsen , S. Schrøder, J.Stage, What happened to remote usability testing?: an empirical study of three methods,” Conference on Human Factors in

Computing Systems, Proceedings of the SIGCHI

conference on Human factors in computing systems, ACM Press, 2007

[6] internotredici.com, available form Internet,

http://www.internotredici.com/article/remoteusabilitytesting , 13 November 2007

[7] usabilityfirt.com, available from Internet

http://www.usabilityfirst.com/glossary/term_512.txl, 12 November 2007.

[8] P.W.Jordan, An introduction to Usability, Taylor & Francis Ltd, 1998.

[9] http://usabilityhome.com/, available from Internet, 14th November 2007

[10] J. Nielsen "Usability Engineering", pp 195-198, Academic Press, 1993.

[11] http://www.psy.gla.ac.uk, available from Internet, http://www.psy.gla.ac.uk/~steve/HCI/cscln/trail1/Lecture5. html

[12] S.McClayland, CEO Compensation website http://academic.emporia.edu Latest Visited 21st may, 08

(7)

[13] Interaction designer (Austin) http://austin.craigslist.org , visit date 21st may 08

[14] Hesketh.com, Covergence of usability, technology and creativity http://www.hesketh.com , visit date 21st may 08

[15] I.T. Toolbox website http://blogs.ittoolbox.com last visited 8 May 2008

[16] Anne Kaikkone,” Mobile Applications: A Comparison between Laboratory and Field Testing”, Usability Testing of Issue 1, Vol. 1, November 2005, pp. 4-

16

[17] Jakob Nielse's website http://www.useit.com last visited 7 May 2008.