Vol.5, No.1, 2015
Evaluating and Testing User Interfaces for E-Learning System:
Blackboard Usability Testing
Abdulhameed Alelaiwi Mohammod Shamim Hossain
Department of Software Engineering, College of Computer & Information Sciences,King Saud University, P.O. Box 51178, Riyadh 11543, Kingdom of Saudi Arabia
[email protected];[email protected] ABSTRACT
E-learning usability studies require the involvement of real end-users. Different users with varying background of Human Computer Interaction (HCI) knowledge behave differently when using any e-learning tools. To study user behaviour in the e-learning context, an empirical usability study on a specific e-learning tool is conducted. The study is performed by using usability evaluation questionnaires collected from two different groups of real users of the tool; one group with HCI knowledge and the other without HCI knowledge. It is found that end users without HCI knowledge are more satisfied than the end-users with HCI knowledge, as they have more expectations concerning the tool. Categories and Subject Descriptors: H.5.2 [Information Interfaces and Presentation (e.g., HCI)]: User Interfaces-User-centered design, Evaluation/methodology; H 5.3 [Information Interfaces and Presentation (e.g., HCI)]: Group and Organization Interfaces-Evaluation/methodology,
Web-based interaction; K.3.1 [Computers and Education]: Computer Uses in Education-Collaborative learning
Keywords: E-learning usability heuristics, Blackboard, e-learning tool, Usability evaluation, HCI. I. INTRODUCTION
In this technologically rich information and communication age, many companies and educational institutions are adopting e-learning platforms as their delivery tool. Unusable and inaccessible interfaces cost these companies and institutions billions of dollars in lost revenue, as significant number of learners are unable to use their e-learning tools [Anthony 2001]. In this paper, e-learning and online learning have been used synonymously.
Nowadays most of the e-learning courses have weak web design and poor usability. Significant amounts of money are being spent on e-learning through the world, but most of the producers of e-learning are not checking the usability of the provided e-learning courses [Quigley 2011]. The reasons behind these problems are as follows [Feldstein 2002]: (a) a vast majority of learners or consumers are unaware of usability evaluations or do not check whether the course is usable, (b) testing methods are sometimes time-consuming and expensive, and (c) the budget of e-learning production does not support it.
Sound usability in e-learning tools ensures a successful and an enjoyable learning. If usability issues are not properly considered in an e-learning environment, then it will become hurdle for learners. Usability and the art of instructional design are crucial for the designing and development of successful e-learning tools [Squires 2009]. Indeed learners cannot take advantage of only a good instructional design because sometimes content in online learning is difficult to find, course tools do not work and navigation is inconsistent. So usability testing and evaluation is critical in the design and implementation of e-learning environments
Numerous articles about online course delivery systems have been published that evaluate factors other than usability issues which are discussed in only very few papers [Quigley 2002; ECC 2002; Reeves et al 2001]. Unfortunately, there are no defined criteria or standards for the evaluation of the usability of an e-learning system.
In this study, literatures related to e-learning usability are reviewed and an empirical study on a popular e-learning tool: “Blackboard” [Blackboard 2010] is conducted by using usability questionnaire from real end users in a classroom environment. The Blackboard is chosen among many other e-learning tools such as FirstClass, WebCt etc., because students and faculty members of our institution are mostly using this tool. The users were selected based on their current experience of the said e-learning tool (Blackboard) and their background knowledge in Human Computer Interaction (HCI). End users with HCI knowledge are hard to satisfy, as they expect more and can find malfunctions more easily. However, both groups are satisfied with online assignment submission tool. One of the challenges for conducting usability studies is to collect data from real user in classroom environment because of ethical issues.
The rest of the paper is organized as follows: Section 2 gives a background of this study, Section 3 describes the methodology of the empirical study of the e-learning tool-Blackboard, Section 4 presents results and analysis from this study, Section 5 gives a brief discussion and the paper concludes with section 6.
2. BACKGROUND OF THIS STUDY 2.1 Usability and its Attributes
There has been many studies conducted on the user experience and how to design that pleasantly meet user expectation and diversity [M. Gualtieri 2009] [J. Forsslund, 2003] [B. Buxton 2007].
Usability in e- learning is about the way content is presented and about the content itself [Feldstein 2002]. The definition of usability in the ISO 9241 standard is: “The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use”. This definition can be expanded to include additional characteristics such as engaging, easy to learn and error tolerant [Feldstein 2002]. According to Whitney [Whitney 2001], usability relies on user-feedback through evaluation rather than simply trusting the experience and expertise of the designer. Usability evaluation involves watching real-end-users use a product and applying what is learned to improve the product. He also points out that usable interface design should consider three factors: content presentation, user approaches (interaction techniques), and technology.
Investigation of usability attributes is important in online courses, as these usability attributes are considered for the designing of learner centered interface in online environments [Zaharias et al. 2002]. “Usability is a measurable attribute of a product” [Notess 2001]. These attributes include efficiency, user satisfaction, error-tolerance [Nielsen 1993]. Learnability is also one of the important measures of usability in e-learning. Learnability of e-learning ensures: learner centeredness, iterative design and frequent testing [Quigley 2002]. To measure the usability of e-learning tools, the following characteristics should also be considered: learner profile, learner satisfaction with the learning content, perception, enjoyment and engagement [Feldstein 2002].
Different usability evaluation techniques such as learner observations, inspections, interviews, questionnaires and expert reviews can be applied or adapted to evaluate the usability attribute of e-learning products.
2.2 Importance of Usability in E-learning
Worldwide revenues in the corporate e-learning market will surpass $62 billion by 2014 [Pastore 2011] Of these revenues, North America has projected 15.3 billion and the European e-learning market has projected $4 billion. However, poor usability will create unsatisfactory results for the following reasons:
Lack of accessibility as a significant number of learners due to their visible, learning, cognitive, auditory and physical disabilities will not be using e-learning products (8% in US and 11% in Europe) Failure to develop ongoing learner loyalty
Inability to stimulate learners
Besides these, according to Nielsen [Nielson 2001], if e-learning courses are offered with poor usability, bad user experience, and lack of engagement and inefficient ways (e.g. slow, awkward to use, etc.), then the learner will not come back for a second course.
2.3 Usability Evaluation in E-learning
Usability testing has a direct application within instructional design when it is used as a formative evaluation technique for e-learning products. Usability testing is the evaluation of instructional tools/software for its ease of use to the frequent users within an instructional design practice [Hughes 1999]. Hughes and Burke [Hughes and Burke 2001] also define usability testing as the observation of typical users performing tasks with a product, conducted for the purpose of determining what changes need to be made to the content, presentation or user-interface for that product.
There are no established techniques relating to distance learning and/or e-learning usability evaluation or testing [Heines 2000]. Some researchers [Squires 2009; Nielson 2001; Dringus 1995b; Notess 2001] recommend that the usability heuristics defined by Nielson, can be applied to evaluate or design online learning interfaces with some pedagogical guidelines, since many guidelines for e-commerce users are not directly applicable to online learners. Squire and Preece [Squires and Preece 2009] argued that these heuristics failed to adopt accepted socio-constructivist view of learning, which cope with the integration of usability and learning issues; they proposed an approach that adapts the idea of Nielsen’s heuristics as a part of a usability evaluation exercise.
Reeves et al. [Reeves et al. 2011] list twenty e-learning usability heuristics, some of which are based upon Nielson’s heuristics [Nielsen] and the others are related to instructional design. They [Reeves et al. 2011] also recommend evaluators to consider the following background information before starting the evaluation of e-learning: Target audience and learner characteristics; Instructional goals and objectives; Typical context for using e-learning programs; Instructional design strategies used in the program; and the status of the program’s development and possibilities for change. According to them, the above e-learning heuristics are complete. However, evaluators may add or delete any heuristics based on the specific type of e-learning program.
Vol.5, No.1, 2015
Anthony [Anthony 2001] discusses general e-learning usability issues which should also be considered during the development of e-learning interfaces. These issues include: motivation problems; repeat sales and ongoing customer loyalty; internationalization; long transaction times; relationship between learning simulations and real world experiences. If usability and accessibility are followed during e-learning product development, then time and money may be saved in the long run. As a result it will get widest possible target learners. Real end users should be involved in usability testing during the development of e-learning product that will result in a more usable e-learning application. After releasing of the product, it is always more costly and time consuming to redesign an interface in order to make e-learning product usable or accessible.
The usability questionnaire that we used in this study was designed based on the usability characteristics and questionnaire from [Feldstein 2002; Neilson 1993; Storey et al.]. Unlike other research studies we chose in the presented empirical study two types of users; one with HCI knowledge and another without HCI knowledge. In the following sections, the discussions of the methodology and the achieved results from this study will be presented.
2.4 Usability Study of Blackboard
The success of any e-learning product depends on how easily and quickly students can use the tools in order to accomplish specific tasks. Hayes [Hayes 2000] argues that specific usability attribute such as ease of learning, should be a criteria for the evaluation of online course delivery systems. He conducts a usability evaluation of two online course delivery applications: BLACKBOARD and Web-Course-in-a-BOX.
Piguet and Peraya [Piguet and Peraya 2000] give an analysis of online learning tools such as BLACKBOARD in respect to usability. In their study, they focus on two factors: satisfaction and control within the user interface interaction. Chalk [Chalk P. D. 2002] used BLACKBOARD to evaluate the usability of virtual learning systems. He argues that BLACKBOARD can be applied to any virtual learning environment with facilities for on-line instructional materials.
Tselios et al. [Tselios et al 2000] conducted usability evaluation experiments of distance learning systems. They used two e-learning software tools, BLACKBOARD and IDLE, in order to (a) measure the usability and effectiveness and (b) investigate the impact of usability on student performance. The usability evaluation was based on usability questionnaire.
Web-based learning is different from traditional web design. In designing online learning tools, the designer should consider the concepts of instructional design strategies and learner experiences. Learners’ experience is important in designing effective e-learning systems. Evaluating e-learning usability is one of the difficult tasks. One of the evaluations is the survey evaluation which quantitatively measure how much users learn from the e-learning course through questionnaire [ECC 2002]. In the following sections, the discussions of the methodology used and the achieved results from this study will be presented. Apparently, a developer should utilize the user experiences with system development lifecycle to develop e-learning system to study the user interface and build interactive application [H. J. Kim 2011] [J. J. Lee 2009] [D. Nam 2010].
3. METHODOLOGY 3.1 Participants
One of the first considerations for conducting usability study is to understand target participants, as it is an important tenet for evaluating usability [Nielsen 1993]. For the described study we selected two groups from two different courses consisting each of 30 students: a third year software engineering course and a first year computer science course. Both courses were given at our university. The first group to which we refer as group “A” had experience about Human Computer Interaction (HCI); the second group (referred as Group B) did not have any experience about software usability.
3.2 Choosing the E-learning Tool: Blackboard
In this study, Blackboard was chosen as the e-learning delivery tool to be used. Since most of the students at the University of Ottawa have been using it as a supplementary course delivery for conventional learning. It is also widely used by a good number of educational institutions worldwide as an e-learning platform and/or as supplementary tool for traditional teaching and learning. Students at our university use Blackboard mainly for online assignment submission in courses for which they are registered. Other used features are: Course Content and Management tools including Syllabi, My Grade, Lectures, Glossary, References and Presentations; Assessment tools including Online Quizzes, Assignments Submission, Self-test etc.; Communication tools including Mail, Chat, Bulletin Board, Discussion etc. Faculties and teaching assistants also use Blackboard for organizing, importing and constructing course materials, as well as other course related purposes.
3.3 Procedures
a five point Likert scale answers1 and open-ended questions (part 6-8). After collecting the questionnaire from each of the group, the completed responses were converted into numbers in such a way that low numbers indicate a positive rating of the tool.
Participants were asked to use Blackboard for online assignment submissions and to record required time for submission, after that they were given the questionnaire. They were asked to submit the questionnaire at their convenience.
The first part of the questionnaire was designed to gather information about the participant’s experience with Blackboard course tools. The second part was aimed at evaluating the user interface. The third part was aimed at evaluating the effectiveness, enjoyment, satisfaction and interaction with the content tools (course content). The fourth part was designed to gather information concerning ease of use and effectiveness with the communication tools. The fifth part was designed to collect data about the time required to accomplish a task and users’ satisfaction with the assessment tools, in particular online assignment submissions. The sixth part (open-ended questions) was only given to Group A’ users and was aimed to collect information concerning the redesign of Blackboard (e.g. would you like to redesign the usability of the above features and how much usability redesign do you think is needed?). The seventh part had open-ended questions related to full online course delivery without classroom lectures (e.g. would you like to extend your course to a full online course delivery without classroom lectures?). The last part was aimed at collecting anonymous comments and suggestions from participants concerning their views on e-learning usability.
4. RESULT AND ANALYSIS
Here t-test analysis tools were used. A t-test is used to determine whether two samples are likely to have come from the same two underlying populations that have the same mean. P is the mathematical probability which can range from 0 to 1. The value should be less than 0.1 before accepting the difference as being statistically significant. The alpha level is of significance level related to the probability. Alpha is the confidence level which must be between 0 and 1 for the test.
4.1. User Profile
As learners’ experience is important to design an effective e-learning system, we have to understand the target real end users. As mentioned earlier, two groups of participants took part in this study: Group A with HCI knowledge and Group B without HCI knowledge. 80% of the participants of Group A and 29% of the participants of Group B had used Blackboard extensively while 20% of the participants of Group A and 71% of the participants of Group B had used Blackboard few times. This is illustrated in Fig. 2.
Fig. 2: User s’ experience with Blackboard 4.2 User Interface (UI) Characteristics
No significant differences were found between Group A (Mean=2.60) and Group B (Mean=2.73) for participants’ ratings concerning the ease of logging and ease of learning, t(29)=.5914, p=.5914. Significant differences were found between Group A (Mean =3.26) and Group B (Mean=2.58) for the participants’ ratings concerning the
1
The answers ranges from (1) “Extremely easy to use” or “Very satisfied” or “Very quickly” to (5) “Extremely hard to use” or “Very unsatisfied” or “Extremely slow”.
0 10 20 30 40 50 60 70 80 Group A Group B 80 29 20 71 p e rc e n ta g e o f u s e r
Experience with WebCT
Used extensively Used a few times
Vol.5, No.1, 2015
ease of use. According to participants of Group B, the Blackboard was easier to navigate than the participants of Group A, t(29)=3.36, p=.0049. The mean value regarding satisfaction with user interface of Blackboard was 3.46 for Group A while it was 2.60 for Group B. By performing a t-test t(29)=3.71, p=.0005, alpha =.05, we deducted a negative attitude towards the Blackboard. Participants of Group B were more unsatisfied with the Blackboard UI than those of Group A. Help and error recovery were not analysed.
According to the majority of the participants (70%), Blackboard did not have consistent navigation format; at times the navigation was confusing, at times it showed inconsistent results, e.g., while clicking back button multiple times.
4.3 Course Content Tools
The elements of a course content tools help students to prepare for the next classes and strengthen their understanding about the respective courses. At the University of Ottawa, course content posted in Blackboard is used as a supplementary part of a classroom teaching and learning. This feature is essential for delivering any e-learning courses.
There were no significant differences found between Group A (Mean =3.0) and Group B (Mean=2.90) for participants’ ratings regarding organization and presentation of information, t(29)=.486, p=.628. There were also no significant differences found between Group A (Mean=2.76) and Group B (Mean=2.60) for participants’ ratings regarding effectiveness, enjoyment and interactions with course contents, t(29)=.926, p=.443. There were also no significant differences found between Group A (Mean =2.80) and Group B (Mean=2.70) for participants’ ratings concerning ease of finding information of the course contents, t(29)=.593, p=.630.
Most of the participants (80%) found Blackboard course tools effective in helping them to study. However, only 68% of the participants were satisfied with the course contents.
4.4 Communication Tools
Communications tools such as discussion forums, mail, chat, and whiteboard allow students a lively communication with other learners as well as instructor. There were significant differences found between Group A (Mean =1.8) and Group B (Mean=2.50) for participants’ rating regarding effectiveness and ease of use of the communication tools, t(29)=3.36, p=.00023. According to most of the participants (87%), communication tools are effective, helpful and satisfactory.
4.5 Assessment Tools-Assignment Submission
Online Assignment Submission allows participants to submit course assignments and receive the feedback from the instructor or teaching assistants. Here how users’ performance regarding online assignment submission affects usability was studied. The presence of a correlation between the required time to submit assignment and their perceived ease of use for every user was studied. Correlation describes the strength of the relationship between two variables. Correlations are valid only when the data has been taken in matched pairs. The rvalue is the Pearson Coefficient of correlation and it can range from 0 to 1. If the value of r is between 0.7 and 1.0, then the strength of correlation is strong. R square is the square of r and is an indication of how much the variance in one variable is affected by variance in the other.
There were no significant differences found between Group A (Mean =2.46) and Group B (Mean=2.50) for participants’ rating regarding satisfaction of the online assignment submission, t(29)=.27, p=.90. There was a strong correlation between required time to submit assignments and their perceived ease of use. The value of Pearson coefficient was r=. 8520 for Group A, but this value was r=. 8353 for Group B. Both values were considered high, indicating strong correlation between required time and ease of use concerning assignment submission. This correlation can be clearly depicted in Fig. 3 (a) and (b) respectively.
4.6 User Interface Redesign
The participants of Group A (Mean =3.1) were less satisfied than those of Group B (M=2.6) concerning overall satisfaction with Blackboard, t(29)=1.63, p=.0702. As the participants of group A were not completely satisfied with the usability of Blackboard, they gave feedback concerning the redesign of Blackboard’s User Interface. 20% of them suggested a complete redesign, 27% suggested a major redesign, 45% suggested an average redesign and 13% of them suggested minor changes. The above statistics could be based on the fact that group A had a more experience with usability.
4.7 Participants’ Recommendation about Open-Ended Questions
As mentioned before the last two parts1 of the evaluation questionnaire consisted of open-ended questions asking our subjects about full online course delivery without classroom lecture, and general comments about e-learning usability
The majority of the students (98%) did not want full e-learning course delivery without classroom lecture; they wanted e-learning course delivery as a supplementary part of classroom teaching. Instructional design issues should be considered for full e-learning course without classroom.
In the following we summarize some anonymous comments from both groups: Not satisfied with user interface
Easy to learn, easy to use, enjoyable Frustration and confusion with navigation Error message after pressing back button Slow, disappointing
Insecure but it has already solved Supportive of classroom lecture
Consider usability feature for future design specifically in UI Make them easier to keep up to date information about courses Accessibility issues should be considered
Difficult to download all kinds of file format
Universality in access and usability should be considered. Some of the participants’ comments are directly quoted:
“I am really frustrated with the navigation facility of Blackboard, very disappointing. An advice to have more usability studies and apply the findings from these studies to future design”
“Slow, user interface is junky. Try to improve speed and user interface”
“Instructional design guidelines as well as usability should be considered during designing and constructing course contents”
“I am very satisfied with online assignment submission tools” 5. DISCUSSION
Participants of both user groups pointed out drawbacks concerning navigation difficulties in user interface of Blackboard. They were not satisfied with this feature. Group A (users with HCI knowledge) were less satisfied than those of Group B.
Real end users’ experience is very important to the design and development of a successful usable e-learning tool. In addition, it is essential to involve a good number of end-users with HCI-knowledge to find malfunctions during development process. The results achieved from the presented study may not vary the way they are now, if usability tests are taken into consideration during the design and development process. The results might again be different if faculties and teaching assistants were included in the study. From the test, it may be concluded that users with HCI knowledge are more careful regarding user interface and therefore they are able to find malfunctions more easily. However, users without HCI knowledge are easily satisfied because of fewer expectations and less HCI knowledge.
In both groups there was a strong correlation between required time and ease of use in submitting online assignment. Most of the usability ratings from the two groups were statistically significant and there was a positive impact on the e-learning tool used except in the navigation part which negatively impacted towards the studied e-learning tool.
We have conducted small scale usability study to evaluate the system. This study consists of 15 volunteers. At first they were briefed about the system and then the users were asked to use the system to view desired learning content. Based on their usage experience, they were asked to answer a number of questions on a Likert five-point scale. The following Figure 4 shows the users’ responses in percentage scale. From the figure, we found that around 63% of users are agree with satisfaction rate in viewing learning content.
1
Vol.5, No.1, 2015
Figure 4. User response percentage on the evaluation of the proposed framework
The overall scores received from the users while using the system are summarized in Table 1. The lower mean value of ease of use and visual control represents poor acceptance by the user. The higher mean value of users satisfaction and usefulness means strong acceptance of the proposed system.
Table 1. User acceptance based on user studies
Visual control Usefulness Ease of use Satisfaction
Mean 1.76 3.12 2.10 3.85
Std Dev 0.34 0.41 0.39 0.47
6. CONCLUSION
As e-learning environments continue to increase, it is important to understand the usability constraints associated with them. There is no doubt that these constraints impact the student learning experience. Blackboard is becoming increasingly popular, so it is essential that we assess the usability of its user-interface. This is particularly pertinent for users with no or little HCI knowledge. As such, this paper could provide an important contribution to the field and will gain importance in the future because this type of learning becomes more accepted. It addresses a critical aspect of e-learning in terms of potential usability barriers. Thus, we have initiated a study on the usability of Blackboard as an e-learning platform that could benefit a large group of users. As for the work, we may consider to examine grades in both courses and to correlate these with questionnaire results
ACKNOWLEDGEMENTS
The authors wish to thank the Research Center in the College of Computer of Computer and Information Sciences, King Saud University for funding and support of this work.
REFERENCES
ANTHONY, Q. 2001. Frontend Usability InfoCentre:Why People Can’t use eLearning, Frontend.com Usability Info Centre, Dublin 4, Ireland,http://infocentre.frontend.com/servlet/Infocentre
Chalk P. D. 2002. Evaluating the use of a virtual learning environment for teaching aspects of HCI”, in Proc. ACM ITiCSE 2002, Aarhus, Denmark, 2002 (June).
DRINGUS, L., 1995b. Interface Issues Associated with Using the Internet as a Link to Online Courses. Journal of Interactive Instruction Developmentfall 1995, 16-20.
DRINGUS, L. 1995a An Iterative Usability Evaluation Procedure for Interactive Online Courses. Journal of Interactive Instruction Developmentspring 2001, 10-14.
H. J. Kim, “Interactive Social U-Learning Community Design”, The J. of IWIT, vol.11(5), pp.193-201 (2011). J. J. Lee, “A Study of User Interface Design Method on Ubiquitous Computing Environments for the Aged”, The
Journal of IWIT, vol. 9(6), pp. 245-251 (2009).
D. Nam, “ Design of Adaptive User Interface (AUI) for Bus Information Terminal”, The Journal of IWIT, vol. 11(2), pp. 89-94 (2010).
ECC 2002, Usability Evaluation in E-Learning Development, Feel Issue 04-Nov 02 , E-learning Competency Center,
Singapore.http://www.ecc.org.sg/cocoon/ecc/website/services/newsletter/feel04/article10.newsletter M. Gualtieri, “Best Practices in User Experience (UX) Design”, pp.1-19 (2009).
J. Forsslund, “Three Themes of User Experience in Haptic Application Design”, pp.1-4 (2003).
B. Buxton, “Sketching User Experiences: Getting the Design Right and the Right Design”, Morgan Kaufmann, (2007).
FELDSTEIN, M. 2002 What is "usable" e-learning?” eLearn, ACM Press 2002, 9.
Products. The Internet and Higher education2, 2-3,119-134.
HEINES, J. M. 2000. Evaluating the effect of a course web site on student performance. Journal of Computing in Higher Education 12, 1, 57-83.
HUGHES, M. 1999, Rigor in Usability Testing. Technical Communication 46, 4, 488-494
HUGHES, M. AND BURKE, L. 2001, Usability testing of web-based training, in Web based training B.Khan
Ed., Englewood Cliffs, NJ: Educational Technology Publications
L-C, E., L. Virpi, R. Hassenzahl, “Understanding, Scoping and Defining User eXperience: A Survey Approach”, ACM CHI’09 Conf., Boston, MA, USA, pp. 719-728 (2009). M. Gualtieri, “Best Practices in User Experience (UX) Design”, pp.1-19 (2009).
NIELSEN, J. 1993.Usability Engineering, Boston, MA: Acade mic Press.
NIELSON, J. 2001. Jacob Nielson on e-learning,
http://www.elearningpost.com/features/archives/001015.asp
NIELSEN, J. Ten usability Heuristics. http://www.useit.com/papers/heuristic/heuristic_list.html
NOTESS, M. 2001 Usability, User Experience, and Learner Experence. eLearn, ACM Press 2001, 8.
PASTORE, M. 2011, Companies, Universities Moving Toward E-Learning,
http://cyberatlas.internet.com/markets/education/article/0,,5951_737341,00.html
PIGUET, A. AND PERAYA, D. 2000. Creating Web-integrated Learning Environments: An Analysis of
BLACKBOARD Authoring Tools in respect to Usability. Australian Journal of Educational
Technology 16, 3, 302-314.
QUIGLEY, A. 2011. Usability-Tested E-learning? Not Until the Market requires it. eLearn, ACM Press, 2. REEVES, T. ET AL. 2011. Heuristic Evaluation Instrument and Protocol for E-Learning Program. The
University of Georgia, USA http://it.coe.uga.edu/~treeves/edit8350/HEIPEP.html
SQUIRES, D. 2009. Usability and Educational Software Design: Special Issue of Interacting with Computers. Interacting with Computers.11, 5, 463-466
SQUIRES, D. AND PREECE, J. 2009. Predicting quality in educational software: Evaluating for learning, usability and the synergy between them. Interacting with Computers 11, 5, 467-483.
STOREY, M.A., PHILLIPS, B., MACZEWSKI, M. AND WANG, M. 2002. Evaluating the Usability of Web-based Learning Tools. Educational Technology & Society, 5. 3.
S. Abduljalil, D. Kang, “Analysis of human factors in software application design for effective user experience”, Advanced Communication Technology (ICACT), pp.1446-451 (2011).
TERRELL, S. R. AND DRINGUS, L., An Investigation of the Effect of Learning Style on Student Success in Online Learning Environment. Journal of Education Technology System 28, 3, 231-238.
TSELIOS, N., AVOURIS, N., DIMITRACOPOULOU, A., AND DASKALAKI, S. 2001. Evaluation of Distance-learning Environments: Impact of Usability on Student Performance, International Journal of Educational Telecommunications (IJET) 7,. 4, 355-378.
Blackboard 2010. http//www.Blackboard.com
WHITNEY, Q. 2001. What does usability Mean: Looking beyond ‘Ease of Use’,. Cognetics Corporation, Princeton Junction, NJ 08550,USA. http://www.cognetics.com/papers/whitney/index.html
ZAHARIAS, P., VASSILOPOULOU, K., AND POULYMENAKOU A. 2002. On-line Learning Courses: A review and usability attributes in Proc. E-Learn 2002, Montreal, Canada, 15-19
