PROJECT DOCUMENTATION WEBMAIL EVALUATION SUMMARY. Clients Replacement Project

(1)

PROJECT DOCUMENTATION

WEBMAIL EVALUATION SUMMARY

Email Clients Replacement Project

Release: Final 1.0 Date: 23rd August 2006

(2)

0.1

0.2 Purpose of Document

This document provides a summary of the various evaluations of two candidate webmail systems for the replacement of Silkymail.

0.3 Acknowledgments

Thanks are due to Lisa Gault, who made extensive enquiries to other universities about their use of IMP and SquirrelMail and accessibility; Gilles Couzin, who did a very thorough evaluation of accessibility; Richard Hopkins, who looked at security aspects; Sarah Agarwal and the Portal team for advice and feedback. And last, but not least, all the users who contributed to the survey.

(3)

1 Executive Summary

The aim of this project stage is to replace the Webmail client (Silkymail). Silkymail is no longer being developed or supported. Its interface is old-fashioned and it is not popular with its users. It is not compatible with the Portal and the Portal project requires a new webmail client that it can use. There is a risk that a security ‘exploit’ may appear at any time which could force us to suspend the service immediately.

A number of candidate webmail packages were evaluated against a set of technical requirements and two were short-listed:

• Horde IMP

• SquirrelMail

Independently, the Portal Project Team evaluated the packages for suitability for use with the Portal and arrived at the same shortlist of two, though their clear preference was for IMP. The short-listed packages were installed and then subjected to the following:

• user evaluation

• accessibility evaluation

• performance evaluation

The user survey showed that the two candidate packages are both acceptable replacements for Silkymail, with a preference for the IMP user interface and for the speed of SquirrelMail.

The performance evaluation suggests that SquirrelMail is likely to be the faster of the two and it is our impression that it is easier to set up and administer.

The accessibility evaluation showed that neither package currently meets the University’s Web Accessibility Policy. However, if we reject both then we are left with Silkymail, which also contravenes the Policy.

The proposed solution is to implement SquirrelMail for the following reasons:

• We believe it will be easier to ensure the performance of SquirrelMail is satisfactory

• IMP uses JavaScript for some features that would be inaccessible to users of screen readers – there is no easy solution to this

• The next release of SquirrelMail will have some changes that will aid accessibility (eg the removal of frames)

• We can make a number of changes to SquirrelMail to improve accessibility (and this would be more difficult to do for IMP)

• Reservations about the user interface can be reduced by using the improved skins from

(4)

2 Introduction

The provision of the central email service is done using open standards. Email servers using IMAP (Internet Message Access Protocol) communicate with email client software on the user’s desktop. Users are able to use any IMAP client software they wish, so a number of different packages are in use around the University. Information Services currently recommend and support Mulberry as the desktop client and provide a webmail service using Silkymail.

Both Silkymail and Mulberry are no longer commercially supported. Silkymail is not compatible with the student Portal currently under development, and it is recognized that email capabilities are crucial to the success of the Portal.

The priority is to replace the Webmail client (Silkymail). Silkymail is no longer being developed or supported. Its interface is old-fashioned and it is not popular with its users. It is not compatible with the Portal and the Portal project requires a new webmail client that it can use. There is a risk that a security ‘exploit’ may appear at any time which could force us to suspend the service immediately.

A number of candidate webmail packages were evaluated against a set of technical requirements and two were short-listed

(http://www.bris.ac.uk/ict-projects/emailclients/docs/docs/webtechassess.pdf):

• Horde IMP – www.horde.org

• SquirrelMail – www.squirrelmail.org

Independently, the Portal Project Team evaluated the packages for suitability for use with the Portal and arrived at the same shortlist of two, though their preference was for IMP

(http://www.bris.ac.uk/ict-projects/emailclients/docs/docs/portaleval.pdf). The short-listed packages were installed and then subjected to the following:

• user evaluation

• accessibility evaluation

• performance evaluation

• security evaluation

(5)

3 User Evaluation

User evaluation was carried out by means of an on-line survey that ran from 14 June to 28 June. Over 220 people were invited to take part by trying out the two webmail packages and then giving their opinions on the following attributes:

• appearance of user interface

• speed of user interface

• listing messages in your Inbox and other folders

• reading messages

• composing messages

• using the Address Book

• handling attachments

They were also asked to rate Silkymail on the same attributes to give a baseline for comparison. Initially, there were four mail servers (central Student and Staff, IS staff Shark, and ILRT)

accessible from the trial software. The Geography server was added by the 15 June. However, for various technical reasons, the Social Medicine server was not added until there were less than 24 hours of the survey to run. The end date of the survey was extended by a week to give users of that server time to give their views, but this document is based on the results received by the morning of 29 June.

• 140 people completed the survey, of which 70 were staff and 81 were students (some can be both).

• 68% of the students were undergraduates.

• 79% of the total claim to access their email seven days a week.

• 39% use Silkymail, and 44% use Mulberry, as their main email client.

• 91% use Silkymail at one time or another

(6)

Responses from Academic Staff 2 11 2 1 10 7 0 2 4 6 8 10 12 Arts: Engineering: Medical & Veterinary Sciences: Medicine & Dentistry: Science: Social Sciences and Law:

Responses from Support Staff

9 1 24 3 0 5 10 15 20 25 30 Academic Department: Estates: Information Services: Student Services:

Responses from Students

10 21 6 13 13 18 0 5 10 15 20 25 Arts: Engineering: Medical & Veterinary Sciences: Medicine & Dentistry: Science: Social Sciences and Law:

The main results are shown in Appendix A. From this, it can be seen that:

• Both IMP and SquirrelMail were considered an improvement on Silkymail on all attributes

• IMP was preferred to SquirrelMail on ‘appearance of user interface’ and ‘listing messages …’

• SquirrelMail was preferred to IMP on ‘speed of user interface’

(7)

The opinion of the IS trainers was that it would be easier to give courses on SquirrelMail.

However, less than 4% of respondents thought training courses would be necessary. One evaluator made the suggestion that a technical session for support staff would be beneficial.

There is undoubtedly more value to be gleaned from the survey results, but it is clear that either candidate would be considered an improvement on Silkymail, with a small preference expressed for the appearance of IMP’s user interface. It should be borne in mind, though, that the appearance of both interfaces can be changed; for example, see www.NutsMail.com for examples of

replacement ‘skins’ for SquirrelMail.

The poor performance of IMP was cited by a number of users. This is considered further in the next section.

(8)

4 Performance Evaluation

Measuring the speed of web applications can be extremely difficult. The setup can take several days and the results are, at best, merely an indication. Changing the size of the server, altering configuration options, and tuning parameters, can make dramatic differences in speed or, sometimes, achieve practically nothing.

We decided that, instead of investing a lot of effort in setting up performance tests, we would do two things:

• Survey universities who are existing users of IMP and SquirrelMail to enquire about performance issues

• Measure the amount of network traffic generated by following a simple sequence of commands

4.1 Survey of Universities

We made enquiries to a number of other universities running SquirrelMail and IMP and received replies from four SquirrelMail sites and five IMP sites.

The four SquirrelMail sites are running 1-2 servers each, mostly with 2 or 4 GB of memory per server, though one site has 12 GB and claims up to 2000 concurrent users. When asked how satisfied they were with the performance, the sites gave one vote each to Poor, Acceptable, Good and Excellent.

The five IMP sites were mostly running two servers each with between 2 and 4 GB of memory per server. One site claimed to be able to run 25,000 concurrent users, which seems rather high. One site regarded performance as Poor; three as Acceptable; and one as Good.

All we can conclude from this is that there are university sites that run these packages with acceptable performance.

4.2 Measuring Network Traffic

Many users of webmail are accessing their email from locations away from the University campus. Some of these locations will be on low bandwidth connections and so it is important that the chosen webmail package performs well for the hypothetical user in Timbuktu1.

In order to measure the potential load of each package, a set of operations were performed using a large inbox (over 2,000 messages). These operations were:

1. Login

2. Read a message 3. Send a message

4. Go to the last page of the inbox

1

Apparently the Most Remote Internet Café in the world is the Télé Centre Polyvalent (TCP), Timbuktu, Mali, (Yahoo! Mail Internet Café Awards, 2004)

(9)

5. Logout

The elapsed time for each operation, and the number of kilobytes sent and received were measured and averaged over three runs. The PC used was directly connected to the University network. The browser cache was cleared before each run.

IMP SquirrelMail

Elapsed Kilobytes transferred Elapsed Kilobytes transferred Time (s) From server To server Time (s) From server To server Inbox folder 31 103 40 18 51 11 Read message 10 14 5 6 13 1 Send message 31 32 14 22 44 7 Go to last page 10 13 5 4 24 2 Logout 6 12 4 4 2 1 Total 88 174 67 54 134 22

These numbers back up the impression that IMP is slow when accessing large inboxes, taking 60% longer to complete the run. The most striking thing about these results is that IMP sends 30% more data to the user’s browser than does SquirrelMail, and the browser sends back 200% more when using IMP. Although these amounts of data are irrelevant in speed terms on a fast network, they are significant on a slow connection: 174KB would take about 35 seconds on a 56K modem.

(10)

5 Accessibility Evaluation

The University’s Web Accessibility Policy states:

“It is the policy of the University to make reasonable adjustments in order to make all web-based information, services and learning resources on the University's Web site accessible to all users regardless of disability. The University requires that all new Web content conforms to

W3C/WAI's Web Content Accessibility Guidelines 1.0, Conformance Level "Double-A" with immediate effect, and that existing Web content be gradually brought into line, as they are periodically updated; it is the aim of the University that most of its websites will be in the

process of being made compliant with the guidelines by July 2004.”

Gilles Couzin evaluated both webmail packages for conformance to the W3C/WAI Web Content Accessibility Guidelines. The main conclusions were:

• Neither package conforms to W3C Web Content Accessibility Guidelines 1.0, Conformance Level "Double-A" in its present state. Although both have some accessibility features, they also lack some essential ones.

• Most issues in both packages could be addressed through the integration of accessibility features in the HTML mark-up and better use of CSS.

• Both applications use nested layout tables which should be replaced by CSS wherever possible.

• Horde IMP relies heavily on JavaScript to perform essential tasks. The fact that these functions are not accessible without JavaScript is the most serious issue.

Some improvements could undoubtedly be made to both packages, but at the cost of implementation and maintenance effort. Future releases may also bring improvements; for example, the next release of SquirrelMail is expected to dispense with the use of frames. Lisa Gault made enquiries to a number of UK universities running SquirrelMail and IMP about accessibility issues.

Aberdeen found SquirrelMail to be acceptable. Leeds and Aston have made some modifications to make SquirrelMail more accessible.

There was little feedback on IMP. Bath was non-committal. Kings College London said that a study had advised against IMP and apparently preferred SquirrelMail. KCL points out that there is now MIMP (Mobile-IMP) which is a cut down version for mobile devices that may be more accessible.

(11)

6 Security Evaluation

IMP can do PGP and S/MIME (read and write) encryption. There are currently problems with it recognising valid certificate issuing authorities (S/MIME) but these can almost certainly be overcome. There are problems with it communicating with its known PGP key servers.

There is a concern about the traceability of messages sent using IMP. Anyone can set their "From" address to anything they like and there is nothing inserted into the message headers which would allow us to identify the real (authenticated) sender. It may be that some information is logged that would enable us to identify real senders. Lack of traceability would be considered a show stopper. SquirrelMail also does PGP (read and write) encryption which is good, but seems to have the same problems communicating with its known PGP key servers. It only has a "read" S/MIME plug-in so it is not possible to send S/MIME (signed/encrypted) messages with it. This is disappointing, but it could be argued that users who need to encrypt email could use PGP, or use a desktop client. Unlike IMP, it does insert a message header identifying the real (i.e. authenticated) sender of a message.

(12)

7 Conclusions

The initial technical evaluations produced the two candidate packages, SquirrelMail and IMP. The Portal Project team expressed a preference for the IMP product.

The user survey showed that the two candidate packages are both acceptable replacements for Silkymail, with a preference for the IMP user interface. Neither package would enable us to dispense with the need for a desktop client at this time; it is expected that packages will appear over the next year or two that may have this potential.

The performance evaluation suggests that SquirrelMail is likely to be the faster of the two and it is our impression that it is easier to set up and administer.

The accessibility evaluation showed that neither package currently meets the University’s Web Accessibility Policy. This is obviously a major problem and puts us in a difficult position. If we implement one of the packages then we contravene the Policy. If we do not, then we are left with Silkymail, which also contravenes the Policy.

However, we believe that some changes can be made to SquirrelMail that will bring us very close to meeting the Policy and this would be significantly more difficult to do for IMP.

The proposed solution is to implement SquirrelMail for the following reasons:

• We believe it will be more difficult to ensure the performance of IMP is satisfactory

• IMP uses JavaScript for some features that would be inaccessible to users of screen readers – there is no easy solution to this

• The next release of SquirrelMail will have some changes that will aid accessibility (eg the removal of frames)

• We can make a number of changes to SquirrelMail to improve accessibility

• Reservations about the user interface can be reduced by using the improved skins from

(13)

A

Appendix: User Survey Results

Appearance of user interface

0% 10% 20% 30% 40% 50%

Very poor Poor Acceptable Good Excellent Silkymail IMP SquirrelMail

Speed of user interface

0% 10% 20% 30% 40% 50%

Listing messages in your Inbox and other folders

20% 30% 40% 50% Reading messages 30% 40% 50% 60%

(14)

Composing messages 0% 10% 20% 30% 40% 50% 60%

Using the Address Book

0% 10% 20% 30% 40% 50%

Handling attachments 0% 10% 20% 30% 40% 50% 60%

PROJECT DOCUMENTATION WEBMAIL EVALUATION SUMMARY. Clients Replacement Project