A Performance Study on Internet Server Provider Mail Servers
Jun Wang
Computer Science & Engineering Department
University of Nebraska Lincoln
Lincoln, NE 68588
e-mail:
{wang}@cse.unl.edu
Yiming Hu
Department of Electrical & Computer
Engineering and Computer Science
University of Cincinnati
Cincinnati, OH 45221-0030
email:
{yhu}@ececs.uc.edu
Abstract
This paper presents a comprehensive performance study on Internet Service Provider (ISP) mail server, which plays an important role in Internet-based distributed computing. By feeding the SPECmail2001 benchmark into an ISP mail server testbed set up by a commercial mail server soft-ware system–MDaemon 5.0.1, we study both networking and I/O performance by varying its user population from 200 to 10,000. The benchmark utilities are adopted for net-working analysis while file system traces are collected for I/O measurement. Based on the benchmark study and of-fline trace analysis, we arrive at several important conclu-sions for ISP mail servers and give corresponding technical suggestions to improve the performance. First we observed that, in SMTP and POP sessions, the initial network con-nection setup step usually takes a very long time. Second, I/O latencies typically contribute to 40–55% of the total data transfer time in e-mail requests, especially in a server with large user population support. Third, A group of e-mail messages will easily make a remote recipient server become overloaded.
1. Introduction
Mail servers play an important role in information soci-ety today. As Internet user population continues to grow ex-ponentially, there is an increasing demand for high perfor-mance mail servers. Internet Service Providers (ISP) pro-vide a majority of such mail services. Unfortunately, few results have been published on studying and analyzing mail servers. One possible reason may be that, it is difficult and impossible to explore real life mail service systems for pri-vacy and security concerns.
In order to understand the impact of modern computer systems architecture on the ISP mail server performance, we set up an experimental framework and conducted a per-formance evaluation on a state-of-the-art ISP mail server, specifically in networking and I/O. During the experiments,
we chose SPECmail2001 [8] as a standard mail service benchmark and made some modifications to emulate to-day’s real-life e-mail service environment. The enterprise version of MDaemon 5.0.1 [1] is installed as the ISP mail server under test. E-mail systems can be evaluated in dif-ferent dimensions such as performance and scalability (in terms of users and per user data storage limits), data persis-tence and fault-tolerance, etc. In this paper we emphasize the aspect of performance.
This paper makes the following contributions. First, we developed an experimental framework to study the ISP mail server performance. A detailed picture of network behavior and I/O performance of ISP mail servers is clarified. Sec-ond, we conducted a comprehensive performance analysis on ISP mail server and gave technical suggestions on how to improve the performance. Specifically, we have arrived at some major observations and propose some schemes for performance optimizations.
2. Evaluation Methodology
2.1. The Mail Server and Benchmarks
There are many types of mail server systems from both academia and industry on the market. We chose one of the most popular commercial products, MDaemon 5.0.1 from Alt-N Technologies Inc. [1], as the major ISP mail server under test. MDaemon product provides both ISP and en-terprise e-mail services. It is a versatile mail server which supports POP3 [6], SMTP [4] and IMAP [5] protocols. Its friendly configuration interface allows us to flexibly vary the system input parameters like the maximum number of concurrent users. As the first stage in the experiments, we installed MDaemon 5.0.1 as an ISP mail server in the Win-dows 2000 Operating Systems.
Currently there are very few standard mail server bench-marks. We chose SPECmail2001 as the major mail server benchmark, which is developed by one of the lead-ing benchmark software vendor–SPEC organization. SPECmail 2001 is a standard mail server benchmark
de-signed to measure a system’s ability to act as a mail server serving e-mail requests, based on the Internet stan-dard protocols SMTP and POP3. This benchmark char-acterizes throughput and response time of a mail server system under test based on realistic network connec-tions, disk storage, and client workloads. The benchmark design focuses on ISP mail servers rather than enter-prise class of mail servers, with an overall user count in the range of approximately 100 to 1,000,000 users. The goal of SPECmail2001 is to enable objective com-parisons among different kinds of mail server systems. We study only POP3 and SMTP protocols in this pa-per since SPECmail2001 benchmark does not incorporate IMAP workload at present. To emulate the real life to-day’s e-mail service in ISP mail servers, we modify the default values set in SPECmail2001 based on empiri-cal data. For example, messages sent per user, per day is increased to 10 messages from original 2 and messages re-ceived per user, per day is increased to 10 messages from original 2. One reason to increase the number of mes-sages comes from spam/junk mesmes-sages. The percentage of users using 56 K bit modems to establish Internet connec-tion is reduced from 90% to 70%. The detailed workload characteristics are shown in Table 1.
Parameter Workgroup Estimate
messages sent per user, per day 10 average recipients per message 10 messages received per user, per day 4
mailbox checks per user, per day 10
% of mailbox checks which 75%
don’t download messages
average message size 25 KB
% of users using 70%
modems (56 Kbit)
% of mail to/from 70 %
remote addresses
Table 1. The Message Traffic Characteristics
Filemon/EE (i.e, Enterprise Edition), a commercial soft-ware developed by System Internal Inc., is chosen as a file system trace collector that can monitor the file system ac-tivity in Windows NT/2000 OS [9]. It runs as a log process working in the background without affecting normal sys-tem behavior. Every file syssys-tem event is recorded in a log buffer and written to the disk later soon.
2.2. Experimental Framework
We set up an experiment framework in a switched Fast Ethernet (100 Mbps) Local Area Network. Three machines from a same small-scale cluster are used to emulate e-mail
SMTP Sink Server
POP3 Retrieval, 100Mb/s
Benchmark Manager
2 GB RAM, Windows 2000Intel Pentium IV 2.0 GHz, Mdaemon Mail Server 5.0 512 MB RAM, RedHat Linux 6.2
Filemon/EE Tracer SMTP Relay, 100Mb/s SMTP Store, 100Mb/s
Intel Pentium III 800 MHz, Load Generator
Figure 1. Experiment Framework
servers and users in experiments. The isolated local area network with a firewall protection could prevent the dis-turbance from the external network traffic. One Dell 4500 PC, with a Pentium IV (2 GHz) CPU and 1 GB SDRAM main memory running Windows 2000, acted as the ISP mail server under test. Other two PCs with Pentium III (800 MHz) CPUs and 512 MB SDRAM memory func-tioned as both local clients and remote clients. In addition, one of them worked as a remote mail server (also called sink server).
The experimental system architecture is shown as in Fig-ure 1. The MDaemon ISP mail server system is running to provide mail service all the time during experiments. Two load machines run separate load generator processes and initialize clients to communicate with the ISP mail server. The benchmark management process, which synchronizes two load generators based on pre-configured parameters, is running on either of two load machines before the exper-iments start. Here are the major operations in the experi-ments:
• POP Session (Uplink). Clients retrieve e-mails from
the mail server.
• SMTP Store (Downlink). Both internal users (i.e, from
local area network users) and external users (i.e, from remote mail servers) send e-mails to the ISP mail server. These requests are generated by two load gen-erators. In SPECmail2001, we assume 90% of e-mail messages are sent from external users.
• SMTP Forward (SMTP Relay). The sink server
re-ceives e-mails from the MDaemon ISP mail server. In SPECmail2001, we assume that 90% of the messages to send are going to external users (i.e. are forwarded to remote mail servers).
The maximum number of users (user population) a mail server can support is the best metric to study scalability of a mail server. How much load the SPECmail2001 benchmark and its transactions generate are determined by this num-ber. During the experiments, we varied the user population
from 200 to 10,000 and studied both network and I/O per-formance of ISP mail server with different user population. For performance study, we modified some parts of SPEC-mail2001 source codes to produce a set of statistical results on networking process. Since Windows 2000 does not pro-vide tools to measure I/O storage subsystem performance, we used a commercial software called Filemon/EE to col-lect file system traces on the mail server side. We can study I/O performance of the mail server in detail by analyzing the file system traces collected.
To guarantee experiments run correctly and traces are collected successfully, we carefully validate all benchmark results and file system traces we collected, which satisfy all the factors with respect to the quality of service, deliv-ery time and remote delivdeliv-ery defined in SPECmail 2001. In addition, we installed and ran Filemon/EE in the mail server system to collect file system traces for I/O perfor-mance analysis.
3. Performance Evaluation and Analysis
In this section, we analyze networking and I/O perfor-mance based on benchmark results and analysis of file sys-tem traces.
3.1. Networking Performance
Networking performance is a very important factor in ISP mail server. To study the network implications on ISP mail server performance, we analyze networking traffic in two directions, between ISP mail server and users, and be-tween a local mail server and remote mail servers.
3.1.1. Effects of The Number of Users ISP mail server performance is very sensitive to the number of users. To study the service capacity of ISP mail server system, we varied the number of users and collected the experimental results in every benchmark run. Table 2 shows the detailed results.
In Table 2, we include results for two different POP ses-sions: those download mails from the server and those do not. The latter one is a common case in which mail clients contact the mail server to check for new mails but do not find any, and derive from a POP Retry session.
We can see that network traffic (the number of attempts) in ISP mail server becomes heavier as the number of users is increasing. Also, the average transaction latency is increas-ing. Because the maximum concurrent number of threads in SMTP (or POP) outbound (or inbound) sessions in MDae-mon is limited to 500, ISP mail server would most likely meet with a network congestion problem when it serves more users at the same time, especially exceeding a thresh-old (2,000 users in this paper).
3.1.2. The Breakdown of SMTP sessions Because SPECmail2001 only tells part of breakdown results in SMTP and POP sessions (mean, minimum and maxi-mum), we modified and added a few lines of Java source
codes (around 100 lines) in SPECmail2001 to collect tim-ing issues for each network transaction, which allows us conduct an in-depth analysis.
Based on our own tools and benchmark utilities, we col-lected latency results on every phase of SMTP store session. SMTP rely session has similar results and is not included here. Figure 2 shows the latency result in each phase as well as its percentage of the total time in a SMTP session. The “SMTP Data” phase, in which the email body text is trans-ferred to the server, is the most time-consuming part. This phase takes almost 30% of the total latency. Surprisingly, the initial network connections, including SMTP Connect and SMTP Hello, are also very expensive. Both steps to-gether take more than 30% of the total session time. The remaining steps, including “SMTP Mail From” and “SMTP Quit”, have much shorter latencies than that in other phases. A long “SMTP Hello” phase may be related to the ex-pensive TCP/IP protocol, handling TCP SYN and TCP ACK signals. The expensive “SMTP Connect” phase de-rives from a fact that e-mails involve a more complicated protocol (with more round trips) than the simple Web ser-vice (i.e., HTTP protocol). Hence SMTP connections stay around with a longer time than that of simple data trans-fers. One example is that, each e-mail message may get sent to one or more recipients. SPECmail2001 configures 86% messages sent to single recipient while others sent to up to 30 recipients. The “SMTP-sender” and “SMTP-receiver” would even negotiate with several recipients if some mes-sages get rejected. An intricate SMTP protocol results in more overhead in network communications. We suggest that the initial network overhead of the mail server can be reduced by grouping and reorganizing techniques. Since the e-mails are not “real time”, we can adaptively schedule the sending sequence for e-mail messages within a time win-dow frame based on their destinations and priority rates. Messages from the same user or sent to the same remote server (even not the same user) can be packed up together into a “super message”. We do not delay messages that are in “high priority” status. When we need to send multiple small messages to a same remote server, a single packed su-per message will be formed and sent only once to the re-mote server rather than in many separate times. This strat-egy saves many unnecessary initial overhead even for dif-ferent users. The implementation issue is also easy. We can either allow SMTP protocol to incorporate a pair of mes-sage pack and unpack operations, or develop an application program to encapsulate and decapsulate messages in a pre-defined format.
During the experiments, we found a few error messages like “the SMTP Sink Server is busy, try again later” when running the peak load benchmark. The reason is that, if all outbound SMTP e-mails are directed to a same sink server (simulating a remote server) within a short period,
User SMTP Session POP Session with Downloads POP Session without Downloads
Number Attempts Mean Mean Attempts Mean Mean Attempts Mean
Time(ms) Size(KB) Time(ms) Size(KB) Time(ms)
200 236 152 39 183 164 11 61 95 400 480 165 26 344 174 34 114 105 1,000 1022 176 29 888 189 26 296 115 2,000 2,266 187 27 1,714 209 17 572 123 5,000 4,606 201 25 3,553 231 30 1,185 134 10,000 9,280 224 16 7,325 253 28 2,441 142
Table 2. The Statistics of Network Transactions.
(a) SMTP Session Breakdown
0 50 100 150 200 250 200 users 400 users 1000 users 2000 users 5000 users 10000 users Latency(ms) SMTP Connect SMTP Hello SMTP Mail From SMTP Rcpt to SMTP Data SMTP Quit
(b) The Breakdown Percent of SMTP sessions
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 200 users 400 users 1000 users 2000 users 5000 users 10000 users Percent
SMTP Connect SMTP Hello SMTP Mail From SMTP Rcpt to SMTP Data SMTP Quit
Figure 2. The Breakdown of an SMTP session the sink server easily becomes overloaded. This
phenom-ena is quite common in real life e-mail services. Most of outbound SMTP e-mails from an ISP mail server will go to remote mail servers rather than to individual PCs, and therefore making a remote recipient server become over-loaded. It may be a possible solution to solve this problem by introducing a load negotiation and balancing protocol among mail servers in a wide-area network. This protocol helps reorder the sending sequence of outgoing e-mails in one or multiple mail servers if excessive messages will go to a same remote mail server in a short time. For example, since the e-mail can be delayed in a reasonable time (This does not include priority e-mails.), we could arrange those e-mail senders to send their e-mail messages to the remote mail server based on a round robin rule in case the sender detects that its one-time outbound traffic is too heavy. 3.1.3. The Breakdown of POP sessions Figure 3 and Figure 4 show the breakdown results for POP sessions.
In POP sessions with download, the “POP Retrieve” phase takes the longest time (about 30%) to transfer email body text to users. “POP Delete”, the second most expen-sive phase, takes about 20–23% of the total time. This is
be-cause message deletions are very expensive file I/Os in na-tive file systems. Other phases have relana-tively short laten-cies. In POP sessions without download, the latencies of all phases are similar. The reason is that, POP session is a much simpler mail retrieval protocol compared to SMTP proto-col. It uses the ASCII standard as the message transfer for-mat without any marshal/unmarshal operations at all.
3.2. I/O Performance
Since e-mail servers are I/O intensive applications, it is important to understand in what role I/O plays for ISP mail server systems. A typical ISP mail server has frequent file I/Os involved to maintain a large number of e-mail mes-sages. Incoming new e-mails result in file creations while e-mail retrieval requests generate file reads and file invali-dations. Outbound (sending/forwarding) e-mails lead to file reads and file creations for saving message copies on disk. It is expected that a large user population may easily result in more I/O traffic.
In experiments, MDaemon 5.0.1 server adopts NTFS as its mail store. It creates a specific directory for every user and a unique file to store each e-mail. SPECmail2001 gen-erates e-mail workloads that are initialized by the number of
(a) POP Sessions Breakdown without Downloads 0 20 40 60 80 100 120 140 160 200 users 400 users 1000 users 2000 users 5000 users 10000 users Latency(ms)
POP Connect POP User ID POP Password POP Status POP Retrieve POP Delete POP Quit
(b) The Breakdown Percent of POP Sessions without Downloads 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 200 users 400 users 1000 users 2000 users 5000 users 10000 users Percent
POP Connect POP User ID POP Password POP Status POP Retrieve POP Delete POP Quit
Figure 3. The Breakdown of POP session without mail downloads
(a) POP Session Breakdown with Downloads
0 50 100 150 200 250 300 200 users 400 users 1000 users 2000 users 5000 users 10000 users Latency(ms)
POP Connect POP User ID POP Password POP Status POP Retrieve POP Delete POP Quit
(b) The Breakdown Percent of POP Sessions with Downloads 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 200 users 400 users 1000 users 2000 users 5000 users 10000 users Percent
POP Connect POP User ID POP Password POP Status POP Retrieve POP Delete POP Quit
Figure 4. The Breakdown of POP session with mail downloads users. Such workloads obey a distribution reflecting an
age message count of 3.43 messages per mailbox. The aver-age messaver-age count per mailbox is a base figure for estimat-ing the size of the mail store. In a mail server experiment with ten thousand users support, the total message count in the mail store (average over time) is 34,300 and the over-all raw message data volume is around 860 MB.
For every user population from 200 to 10,000, we col-lected a corresponding file system trace during the bench-mark run. Total I/O traffic observed in ISP mail server scales proportionally to the number of users. In addition, mail server has both file read and file write intensive workloads, with the rate around 3:1. Small e-mail messages result in many small file write operations.
3.2.1. The Breakdown of Mail Server Response Time In order to compare the roles that networking and I/O (i.e., file I/O in this paper) play in ISP mail server, we explored the breakdown of total response time for every e-mail re-quest. To figure out the average file I/O response time, in
both SMTP and POP sessions, we went through file sys-tem traces, and calculated the latency of file operations by checking their corresponding file events, such as file open, file write, file read, file close etc.
Figure 5 presents the results of the percentage of I/O la-tency among data transfer portions of SMTP and POP ses-sions, including “SMTP Data”, “POP Retrieve” and “POP Delete”. Other phases such as “SMTP Connect”, “SMTP Mail From”, “POP Connect”, etc. are not included because there are no I/O operations involved. We can see that I/O part places a more impact on server performance as the user population is growing. When the user population exceeds 5,000, I/O becomes the major performance bottleneck. For an ISP mail server to support 10,000 users, the I/O latency takes 40–55% of data transfer time in both SMTP sessions and POP sessions with download. For POP sessions without download, since there is no email file to be read or deleted, the I/O overhead remains relatively small.
(a) Percentage of I/O Overhead in SMTP Data Phase 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 200 users 400 users 1000 users 2000 users 5000 users 10000 users Percent
Network + CPU Latency(ms) I/O Latency(ms)
(b) Percentage of I/O Overhead in POP Retrieve and POP Delete Phases with Downloads
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 200 users 400 users 1000 users 2000 users 5000 users 10000 users Percent
Network + CPU Latency(ms) I/O Latency(ms)
(c) Percentage of I/O Overhead in POP Retrieve and POP Delete Phases without Downloads
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 200 users 400 users 1000 users 2000 users 5000 users 10000 users Percent
Network + CPU Latency(ms) I/O Latency(ms)
Figure 5. The Breakdown of Mail Server Work Sessions
4. Related Work
There are a limited number of research papers published on mail servers. Recently Behren et al. described a working-in-progress project called NinjaMail, a high-performance clustered, distributed e-mail system [10]. It used a collec-tion of clusters distributed through a wide area to provide thousands of users with highly available and scalable ser-vices. Saito et al. described the motivation, design and per-formance of Porcupine, a scalable mail server [7]. The goal of Porcupine was to provide a highly available and scalable electronic mail service using a large cluster of commodity PCs. Their focus was on dynamic load balancing, automatic configuration, and graceful degradation in the presence of failures. They used duplicated services that were distributed homogeneously and dynamically across nodes in a clus-ter. Christenson et al. presented a highly scalable electronic mail service using Open Systems in EarthLink [2]. They used the Network Appliance family of servers [3] as the network file storage to achieve good I/O performance, high reliability and easy maintenance. The above papers all fo-cused the design and implementation of the distributed mail servers. To the best of our knowledge, we are unaware of published results on performance evaluation of mail servers.
5. Conclusions
In this paper, we study the performance of ISP mail server with a wide range of user populations support, es-pecially in networking and I/O fields. We used SPEC-mail2001 as a standard benchmark to evaluate a medium-scale ISP mail server performance. We also collected file system traces via a Filemon/EE tracer for I/O analysis. By analyzing benchmark results and file system traces, we ar-rive at several important conclusions: 1) In SMTP and POP sessions, the initial network connection setup and the first phase (e.g., SMTP Hello), usually takes a very long time. 2) I/O latencies contribute up to 40–55% of the data transfer time in e-mail requests, especially in a server with large user population support. Such a high I/O overhead would
seri-ously limit the scalability of ISP mail servers. 3) A group of e-mail messages may easily make a remote recipient server become overloaded.
References
[1] Alt-N Technologies, Inc. Mdaemon 5.0.1 versatile email
server, 2001. http://mdaemon.deerfield.com/.
[2] Nick Christenson, Tim Bosserman, and David Beckemeyer. A highly scalable electronic mail service using open systems. In Proceedings of the USENIX Symposium on Internet Tech-nologies and Systems (ITS-97), pages 37–48, Berkeley, De-cember 8–11 1997. USENIX Association.
[3] D. Hitz, J. Lau, and Malcolm. File system design for an nfs file server appliance. In Proceedings of the USENIX 1994 Winter Technical Conference, pages 235–246, San Fran-cisco, CA, Jan. 1994.
[4] J.Postel. Simple Mail Transfer Protocol, Internet RFC 821, Aug 1982.
[5] M.Crispin. Internet Message Access Protocol, Internet RFC 2060, Dec. 1996.
[6] J. Myers and M.Rose. Post Office Protocol – Version 3, In-ternet RFC 1939, May 1996.
[7] Yasushi Saito, Brian N. Bershad, and Henry M. Levy. Man-ageability, availability, and performance in Porcupine: a highly scalable, cluster-based mail service. ACM Transac-tions on Computer Systems, 18(3):298–298, 2000.
[8] Standard Performance Evaluation Corporation.
Spec-mail2001, Jan. 2001. http://www.spec.org/osg/mail2001/.
[9] System Internal Inc. Filemon/EE Manual, Sep. 1999.
http://www.sysinternals.com.
[10] J.R. von Behren, S. Czerwinski, A. D. Joseph, E. A. Brewer,
and J. Kubiatowicz. Ninjamail: The design of a
high-performance clustered, distributed e-mail system. In P. Sa-dayappan, editor, Proceedings of ”International Workshops on Parallel Processing 2000, volume pp 151-158, Toronto, Canada., August 21-24 2000.
