Qualitative data was collected and transcribed from videos taken during the experiment into an excel spreadsheet. Qualitative data included the participants’ opinions and feedback about the advantages and disadvantages of the two versions of the e-learning sites. Certain contents in the statements were counted for the Arabic and English versions of the e-learning site. On average, the Arabic version received 100 positive comments and 122 negative comments. This is 2 positive comments and 2.44 negative comments per participant. However, the English version received 91 positive comments and 84 negative comments. This is 1.82 positive comments and 1.68 negative comments.
Each of these comments were then coded and classified under a theme as summarised in the table below. Each of themes were then counted and summed up. The positive feedback for the Arabic version resulted in a total of nine themes. The Arabic version was praised for its clear structure (18.62% of the themes), ease of use (14.70% of the themes), appealing design (13.72% of the themes), good use of colours (11.76% of the themes), ease of navigation (11.76% of the themes), simplicity (9.80% of the themes), good organisation (9.80% of the themes), easy to access (5.88% of the themes), and ease of transferring information (2% of the themes). The positive feedback for the
English version resulted in a total of 7 themes. The English version was praised for its clear structure (32.97% of the themes), appealing design (18.68% of the themes), ease of navigation (18.68% of the themes), ease of access (15.38% of the themes), simplicity (9.89% of the themes), interesting design (2.20% of the themes), and fast download times (2.20% of the themes).
The negative feedback for the Arabic version resulted in a total of 8 themes. The Arabic version was criticised for the inappropriate pictures for the audience of the site (31.15% of the themes), the weak quality of the translated content on the site (25.40% of the themes), the inappropriate contents and words used within the website (18.85% of the themes) which clashes with the cultural values of the participants, the small font size which made it difficult to read the text on the site (16.40%), the absence of sound in the site (3.27% of the themes), bad use of colours (1.63%), the lack of images within the site (1.63%), and difficulty to navigate through the site (1.63%). However, the negative feedback for the English version resulted in a total of 5 themes. Participants criticised the use of the inappropriate pictures deemed unsuitable for the audience of the site (32.14% of the themes), the inappropriate contents and words on the site (28.57%), the lack of pictures (9.52% of the themes), and the lack of interesting colours (5.95% of the themes).
Table 5-11: Results of Content Analysis of the Qualitative Data for the Arabic and English Versions of the E-learning Site.
Negative Feedback
Positive Feedback
Arabic version English version English
version
Arabic version
1. Announcement: website
contains inappropriate
words/content 23
occurrences (18.85%)
Announcement: website contains
inappropriate words / content
24 occurrences (28.57%) Clear structure: 30 occurrences (32.97%) Clear structure: 19 occurrences (18.62%) 2. Translation:
Some content does not reflect
the correct meaning
31 occurrences (25.40%)
Lack of pictures: 8 occurrences (9.52%) Appealing design: 17 occurrences (18.68%) Good use of colour: 12 occurrences (11.76%)
3. Pictures: Website contains
inappropriate pictures
38 occurrences (31.15%)
Pictures: website contains
inappropriate pictures 27 occurrences (32.14%) Easy to access: 14 occurrences (15.38%) Appealing design: 14 occurrences (13.72%) 4. Font size:
Small font size
20 occurrences (16.40%)
Font size:
Small font size
20 occurrences (23.80%) Easy to navigate: 17 occurrences (18.68%) Easy to use: 15 occurrences (14.70%)
5. No sound: 4 occurrences Lack of interesting colours: 5
occurrences (5.95%) Simple: 9 occurrences (9.89%) Easy to navigate: 12 occurrences (11.76%)
6. Bad use of colour: 2
occurrences Interesting: 2 occurrences Simple: 10 occurrences (9.80%)
7. Need more pictures: 2
occurrences Quick download: 2 occurrences Good organisation: 10 occurrences
(9.80%) 8. Hard to navigate: 2 occurrences Easy to access: 6 occurrences (5.88%) 9. Easy to transfer: 2 occurrences
The following table shows the hypotheses and their data analysis Table 5-12: Hypotheses and data analysis
Hypothesis Test Results
H01: There is no difference between the effect of images on usability for English and Arabic e-learning websites
Tested through Questionnaire and survey
H02: There is no difference between the effect of colour on usability for English and Arabic e-learning websites.
Tested through Questionnaire and survey
H03: There is no difference between the effect of font on usability for English and Arabic e-learning websites.
Tested through Questionnaire and survey
H04: There is no difference between the effect of Learnability on usability for English and Arabic e-learning websites.
Tested through Questionnaire, survey and T-test
H05: There is no difference between the effect of language on usability for English and Arabic e-learning websites.
Tested through Questionnaire and survey
H06: There is no difference between the effect of satisfaction on usability for English and Arabic e-learning websites.
Tested through Questionnaire, survey and T-test
H07: There is no difference between the effect of content on usability for English and Arabic e-learning websites.
Tested through Questionnaire, survey and T-test
H08: There is no difference between the effect of efficiency on usability for English and Arabic e-learning websites.
Tested through Questionnaire, survey and T-test
H09: There is no difference between the effect of Aesthetics on usability for English and Arabic e-learning websites.
Tested through Questionnaire, survey and T-test
5.8 Summary
This chapter reports on the key findings from the experiment that was undertaken by a total of 50 Arabic participants. The experiment was created to analyse an e-learning websites, which was implemented in Arabic and in English. Participants were instructed to accomplish seven tasks and to give qualitative feedback about both versions.
Analysis of the performance data demonstrated that the average completion time for tasks four, five and seven differed significantly between the two versions, with participants taking longer to complete the tasks in the Arabic version. These differences may be attributed to a lack of understanding of the Arabic version, as a result of the poorly-translated content. This meant that participants spent longer in interpreting the content of the Arabic version. However, no statistical differences were found with
respect to the average number of clicks between the two versions. Similarly, the average task correctness for each version was 100%.
The results of the evaluation questionnaire showed that the majority of the participants reported negative feedback on the Arabic version, which was perceived to contradict their culture and religion. The majority of participants agreed that blue is the most preferred colour for Arab users followed by red. They agreed that black and yellow were the least preferred colours for Arab users. Participants preferred using the Arabic language rather than the English language. They agreed that the direction of the writing (i.e. right to left) in the Arabic version affected their judgement of the e-learning website. Moreover, the Arab users indicated that the translation of the content into Arabic was very poor and delivered incorrect meanings which affected the usability of the Arabic e-learning version. Analysis of the data showed that the majority of participants preferred font size 12 or 13 for the Arabic version and size 12 for the English. Moreover, the participants preferred the Traditional Arabic font type for the Arabic version and Times New Roman font for the English version.
The analysis of the usability questionnaire data demonstrated that the usability of Arabic and English versions differed statistically (using Paired-samples T-tests) with the Arabic version receiving lower usability scores. The differences related to three criteria: efficiency (question three), learnability (questions two, six and seven), and satisfaction (questions five, twelve, fourteen and fifteen). This indicates that satisfaction and learnability are the most affected criteria when the two versions are compared. Clearly, the Arabic e-learning version failed to fulfil the needs of the Arab users. Results showed no statistical difference in respect to the aesthetics of the Arabic and English versions. Finally, content analysis of the qualitative data resulted in 100
positive comments and 122 negative comments for the Arabic version, against 91 positive comments and 84 negative comments for the English version. The analysis showed that participants complained mostly about inappropriate content in the two versions in the form of text and images, and about the poor quality of the translation in the Arabic version. Both versions were complemented for their clear structure and appealing design.
Following the discussion of the experiment in this chapter, the next chapter (chapter six) presents a case study to validate the key results from the experiment.
6 Chapter Six: Case Study
6.1 Introduction
Chapter six presents the results of a case study. A case study is a research strategy used to investigate a particular phenomenon within a real life context and is used to provide answers for questions (what, why, and how). This case study focuses on examining two different e-learning websites for two universities to validate the findings of the experiment. The two selected e-learning websites belong to two universities ranked amongst the top Arabic universities in the Shanghai University Ranking (2013). King Saud University ranks 195 (Saudi Arabia) and Cairo University ranks 435 (Egypt). The case study examines four criteria: images, colour, language and font and how these criteria are influenced by Arabic culture.