The second service I investigated operated on port 110, which is the port for POP3, the “post- office protocol” version 3. POP3 is used by email applications to fetch email from the email server. Of course, just because a service runs on port 110 does not necessarily mean that it is a POP3 server; however, this conclusion makes sense for other reasons, which we will see later. Figure 5.12 shows the anomaly score per day for the server. There was an ongoingcritical performance issue starting on January 8, 2009, and ending on February 3. Figure 5.13 shows that the issue was first detected at around 3:00 p.m. on January 8, and it reached critical status by midnight. Also, the issue began to wane around the same time on February 3, and the anomaly ended around 3:00 a.m. on February 4.
Figure 5.14 shows the distributions of response times comprising the anomaly, at both the beginning and end of the anomaly. The anomalous distributions are relatively typical for most of the distribution, differing only in last few bins, where they jump to about three seconds. In
Figure 5.13: Anomaly score heat-map with a rolling-day time-scale for the POP3 server. 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.5 1 1.5 2 2.5 3 3.5 cumulative probability response time (s) 2009-01-08 2009-01-09 2009-01-10 2009-02-01 2009-02-02
5000 5500 6000 6500 7000 7500 8000 8500 9000 9500 12/06 12/20 01/03 01/17 01/31 02/14 02/28 03/14
number of SYN records
day (MM/DD) (a) SYN records
2000 3000 4000 5000 6000 7000 8000 9000 12/06 12/20 01/03 01/17 01/31 02/14 02/28 03/14
number of SEQ records
day (MM/DD) (b) SEQ records
Figure 5.15: The number of SYN (SEQ) records, or connection attempts (connection establish- ments), reported by adudump per day for the POP3 server. The color of the X indicates the anomaly classification of that day. Days without an X are from the training set. Breaks in the line represent outages.
general, such a consistent value for the last QF bin is indicative of a timeout mechanism on the server, which is an upper bound of the time to generate a response. I discuss the significance of three seconds later in this section.
The demand on the server in terms of the number of connection attempts per day is roughly the same during the anomaly as it is during typical operation, as shown in Figure 5.15(a). However, the number of connection establishments per day (Figure 5.15(b)) drops significantly as soon as the anomaly is over. The timing is suspicious, and an obvious question arises: is the drop in connection establishments somehow related to resolution of the performance issue? I will conjecture in the rest of this section that, indeed, the two are related.
Because the number of accepted connections drops after the anomaly, it would make sense if the number of requests dropped also. If that were not the case, then the nature of the traffic has changed in that the number of requests per connection has increased. But Figure 5.16 shows that, in fact, the number of requests drops commensurately with the number of accepted connections.
Because the number of connection attempts is roughly the same but the number of connec- tion establishments decreases on February 4, the number of rejected connection attempts must commensurately increase. Indeed, Figure 5.17(a) makes this observation explicit. However, the number of rejected clients does not increase, which shows that there are a small number
0 5000 10000 15000 20000 25000 30000 35000 40000 45000 50000 12/06 12/20 01/03 01/17 01/31 02/14 02/28 03/14
number of request ADUs
day (MM/DD)
Figure 5.16: Number of request ADUs per day for the POP3 server.
0 500 1000 1500 2000 2500 3000 3500 4000 4500 12/06 12/20 01/03 01/17 01/31 02/14 02/28 03/14
number of rejected connections
day (MM/DD) (a) total 0 10 20 30 40 50 12/06 12/20 01/03 01/17 01/31 02/14 02/28 03/14
unique rejected client addresses
day (MM/DD)
(b) unique rejected client addresses
Figure 5.17: Number of rejected connection attempts (a), and the number of unique client addresses rejected (b), per day, for the POP3 server.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.01 0.1 1 10 100 1000 10000 100000 1e+06 Cumulative probability
Connection duration, seconds Anomalous slice
Normal slice Anomalous slice w/o host H
Figure 5.18: CDFs of connection durations in the normal period, anomalous period, and anoma- lous period with host H removed. Connection durations are taken by subtracting the timestamp of the SYNrecord from the timestamp of the ENDrecord reported by adudump.
of clients being rejected over and over again. In fact, a manual analysis of the adudump data shows that a single client is being rejected about 2,800 times per day, or about twice per minute. However, this client, which I label R (for Rejected), did not contact the POP3 server prior to February 4 at 4:35 p.m., so this does not explain why the number of connection establishments decreased as well. Curiously, another client (with a different IP address in a different network) successfully initiated a connection to the POP3 server about every thirty seconds, and this client, which I label H for heavy-hitter, ceased to contact the POP3 server on February 5 at 6:51 a.m. Both of these clients contacted the server far more frequently than any other client, and they strongly influence the performance and traffic characteristics, as we will see.
As in the previous section, I select a multi-day period of normal traffic and a multi-day period of anomalous traffic to compare. This time, I can line up the periods by week, so that the weekdays represented in each period are the same. The normal period includes all traffic from Sunday, January 4 to Wednesday, January 7, 2009, and the anomalous period is one week later, from January 11 to January 14. I deliberately choose a period before the anomaly occurs because I want to simulate the diagnosis process that the network manager might have gone through using real-time data displays.
Figure 5.18 shows the connection durations during each period. There is a mode at 3.2
0 20 40 60 80 100 0 5 10 15 20 25
% of conns having at least X exchanges
X = number of exchanges
Anomalous slice Normal slice difference (norm - anom)
Figure 5.19: Percentage of connections having at least X exchanges for the POP3 server.
seconds in the anomalous period accounting for about 14% of the connections. Host H initiated 98% of the connections in this mode (which I selected as durations between 3.2 and 3.3 seconds), and the figure shows that removing host H also removes the mode.
So far, I have only considered contextual information about connections and network traffic provided by adudump. Now, I will begin to consider a richer sort of information that adudump provides: data about the structure of connections. Figure 5.19 shows the percentage of connec- tions that have at least X exchanges. The anomalous period has about twenty percent fewer connections lasting beyond two exchanges, yet both periods have about the same number of connections lasting beyond four exchanges. However, all but about three percent of this dif- ference is solely because of host H. Host H typically has the vast majority of its connections last for at least four exchanges (and most of them for exactly four), but, starting on January 7, about half of its connections only last for two exchanges.
There are more differences between the normal and anomalous periods in the connection structure. Specifically, I will now use ordinal analysis, which I introduced in Section 4.10. Figure 5.20 shows the ordinal response times. It is the second response time that is most to blame for the anomaly, because it is much longer in the anomalous period and still accounts for a significant proportion of response times (unlikee.g. the twelfth response time).
0.0001 0.001 0.01 0.1 1 10 0 5 10 15 20 25
Median resp. time, with 5
th and 95 th %-ile
Ordinal index Anomalous slice
Normal slice
Figure 5.20: Median response time by ordinal for the normal and anomalous periods, with 5% and 95% error bars, for the POP3 server. For example, the point (and bars) at X= 2 represents the distribution of response times that occur for the second request/response exchange within their respective connection.
1 10 100 1000
0 5 10 15 20 25
Median request size, with 5
th and 95 th %-ile
Ordinal index Anomalous slice
Normal slice
(a) request size
1 10 100 1000 10000 100000 0 5 10 15 20 25
Median resp. size, with 5
th and 95 th %-ile Ordinal index Anomalous slice Normal slice (b) response size
Figure 5.21: Median request and response size by ordinal for the normal and anomalous periods, with 5% and 95% error bars, for the POP3 server.
bigger in terms of the median, but generally, the requests are similar in both periods. Also, the requests are all small in an absolute sense. The only significant difference between the response sizes is that the third and fourth responses are bigger during the anomaly.
Recall that the server runs on port 110, which is the port for POP3. POP3 is described in RFC 1939, which gives the following example session:
S: <wait for connection on TCP port 110> C: <open connection>
S: +OK POP3 server ready <[email protected]> C: APOP mrose c4c9334bac560ecc979e58001b3e22fb
S: +OK mrose’s maildrop has 2 messages (320 octets) C: STAT
S: +OK 2 320 C: LIST
S: +OK 2 messages (320 octets) S: 1 120
S: 2 200 S: . C: RETR 1
S: +OK 120 octets
S: <the POP3 server sends message 1> S: .
C: DELE 1
S: +OK message 1 deleted C: RETR 2
S: +OK 200 octets
S: <the POP3 server sends message 2> S: .
C: DELE 2
C: QUIT
S: +OK dewey POP3 server signing off (maildrop empty) C: <close connection>
S: <wait for next connection>
The APOP command does user authentication in one step, avoiding sending clear-text pass- words and at the same time avoiding replay attacks. The key by which the user’s password is encrypted is given in the “+OK POP3 server ready” command. However, not all servers sup- port the APOP command, and a standardUSER and PASS two-step authentication is provided as well. Here is an example of a connection using the two-step authentication:
S: <wait for connection on TCP port 110> C: <open connection>
S: +OK POP3 server ready C: USER mrose
S: +OK User accepted C: PASS mrosepass S: +OK Pass accepted C: <dialog continues...>
The POP3 server we have been considering in this section does not use the one-step APOP authentication because, as Figure 5.21(a) shows, the first request is too small. Therefore, I conclude that the server uses the two-step authentication. This explains why almost all connec- tions (in the normal period, at least) have at least four exchanges: the two-step authentication, plus one to check for new mail, and one more to quit if there are no new messages. This also offers an explanation for why a connection might end after two exchanges: the authentication failed.
I will now offer a hypothesis for a correct diagnosis of the performance issue based on the data we have seen so far. This is the sort of hypothesis that the adudump data supports. If I were the administrator of this server (or if I had the time to talk to him or her), I would offer this hypothesis in the expectation that it could be verified as explaining the cause this anomaly. The
hypothesis is as follows. Many Linux distributions include pluggable authentication modules, or PAM, which handles authentication for a variety of services. The default configuration of PAM typically includes a three second waiting period upon a failed authentication attempt. This is designed to reduce the feasibility of an attacker flooding the server with password guesses and eventually guessing the right one. Recall that, as we saw in Figure 5.14, the anomaly manifests as response time distributions with a relatively large number of response times around three seconds. My hypothesis is that this anomaly is entirely due to the authentication failures of the heavy-hitter client H.
A major contribution of this case study is to demonstrate not only the diagnostic power of adudump data, but specifically the diagnostic power that information about theconnection structure provides.