This section provides some mitigation techniques for the SSL renegotiation based DoS attack. Further, the effectiveness of these techniques is also discussed.
4.3.1 Server control
The server might have been configured to serve single connection with a single thread. Such threads might have been allowed to use a specific amount of the CPU and the memory. Such configuration is not uncommon [42]. In such configuration, the renegotiation based compu- tational DoS can overload the single thread pertaining to the renegotiating connection and making it slower. This cannot affect the other threads serving to the legitimate users. Hence, such threading mechanism cannot be completely affected by the renegotiation based compu- tational DoS. On the other hand, the traditional computational DoS attack can effectively attack such servers because it can create a large number of connections with the server and can produce the computational overhead. The server can deny the legitimate users due to the probability of the lack of the threads to serve the connections.
Under the traditional computational (D)DoS attack, it is very hard to block the sources of such attack as the attacking machines could be distributed over a wide area of the network. Blocking some part of the network sources introduces the risk of blocking the legitimate users. Whereas, in the case of renegotiation based DoS attack, the attacker machines are likely to be limited in numbers (since they are likely to produce the same effect with lesser machines). However, the machines can also be distributed over the large network area making it equally hard to block such attacking resources like the traditional distributed attacking mechanism. The renegotiation is carried out in the cipher text mode. Therefore, it is not easy to detect DoS attacks based on the renegotiation. Traditional IDS/IPS systems, including anti-DOS mechanisms and firewall configuration cannot easily detect these attacks. In the renegotiation based attack, the packets are encrypted and hence it is not easy to understand the contents inside the packets. On the similar lines, IDS/IPS cannot be really configured to detect such attack unless and until the server is already overloaded.
4.3.2 Disable renegotiation
The simplest way to mitigate renegotiation based DoS attack is to disable the renegotiation feature on the server side. Certainly, this is not the best way to mitigate any attack (The authentication gap and the Computational DoS attack described earlier in this chapter). The SSL renegotiation is one of the important features of the SSL protocol. The applications of the SSL renegotiation are already mentioned in this chapter. The administrator of the server should consider the requirement of his/her server and take appropriate action about disabling the SSL renegotiation. SSL renegotiation can be initiated both by the clients as well as the servers. The better choice would be to disable the client initiated renegotiation. However, this will not be the best thing to do since some of the application where the client initiated renegotiation is required cannot work. Finally, as previously said, the choice of disabling the SSL renegotiation depends on the application requirements.
4.3.3 Heuristic based SSL Handshake Rate Limitation
The SSL renegotiation is carried out in the encrypted mode. Therefore, it is usually not possible to detect such packets with the help of usual packet filtering mechanism. However, it is possible to infer the existence of such packets by monitoring the format of the encrypted SSL messages.
4.3 Mitigation techniques and their effectiveness 39
Figure 4.2: Wireshark Screen-shot: SSL renegotiation packet is flagged with "PUSH".
By looking at the packet, it cannot be decided if the packet is encrypted or not. Therefore, some kind of structural aspect of the SSL handshake needs to be considered. Following list gives some structural peculiarities of the renegotiation message.
• If the handshake protocol message type is unknown, then such packet can be considered as the SSL renegotiation message. Since if the record type is displayed to be the hand- shake protocol message, it is confirmed that the record payload contains the handshake protocol message. Next step is to observe the handshake message type. There are 10 types of handshake messages. If the handshake message type is none from this standard list of handshake types, then such record can be considered as the SSL renegotiation message.
• The renegotiation messages are usually encapsulated in a TCP packet flagged with "PUSH" [7].
Vincent bernat [7] has written a simple iptable entry for the packet filtering detecting a possible encrypted SSL renegotiation message following the logic given above.
Figure 4.2 shows the output from packet capturing software wireshark. Observe that the Encrypted handshake message has TCP PUSH flag set. When TCP payload is observed, the first five bytes denote that this record consist of SSL handshake message (Handshake message identifier: 16). The TLS protocol version 1 is used (03 01). These three bytes are followed by two other bytes denoting the length of the TCP payload (00 e0). This confirms that this is a handshake message. Please note that the SSL record header is not encrypted. But rest of the record data is encrypted. The next byte after the length bytes of the record header denotes the handshake type. The handshake type is found to be 51. The number 51 is not a known handshake type. Therefore, heuristically it can be assumed that this packet is an encrypted packet. In addition, this packet consist of the record header of valid handshake type. Therefore, it can be concluded that this TCP packet consist of encrypted SSL renegotiation message.
The iptable entry written by Vincent Bernat [7] is given in Listing 4.1.
1 p a y l o a d=" 0 >> 22 & 0 x3C @ 12 >> 26 & 0 x3C @" 2 $IPTABLES −A LIMIT_RENEGOCIATION \
3 −p t c p \
4 −−t c p−f l a g s SYN , FIN , RST , PSH PSH \ 5 −m u32 \
6 −−u32 " $ p a y l o a d 0 >> 8 = 0 x 1 6 0 3 0 0 : 0 x 1 6 0 3 0 3 && $ p a y l o a d 2 & 0 xFF =
3 : 1 0 , 1 7 : 1 9 , 2 1 : 2 5 5 " \
7 −m h a s h l i m i t \
8 −−h a s h l i m i t−a b o v e 5/ m i n u t e −−h a s h l i m i t−b u r s t 3 \ 9 −−h a s h l i m i t−mode s r c i p −−h a s h l i m i t−name s s l−r e n e g \ 10 −j DROP
Listing 4.1: iptable Entry: Heuristic based Rate Limiting SSL handshakes (source:[7]).
This packet filter works if there is no TCP fragmentation. The payload on Line 1 is defined to access the TCP payload inside the IP protocol. The protocol selected is TCP protocol and only packet with PUSH flag set is selected. Line 6 is important entry in this iptable entry. This line is responsible for checking the record type residing inside TCP payload. It is checked whether the record has valid header with correct indication of the SSL protocol version. (Either SSLv3.0, TLSv1.0, TLSv1.1 or TLSv1.2). This is followed by the checking if 6th byte (Record header is 5 byte long and 6th byte (first byte of handshake message header) consist of handshake type) is not a valid handshake message type. If these conditions are satisfied then the message inside the encrypted TCP packet is assumed to be the SSL renegotiation message. Further hashlimit match extension is used to limit the rate of these packets. The entries in Listing 4.1 are explained in the Appendix B in detail.
In this way, the renegotiation requests can be kept limited using a simple heuristic tech- nique to mitigate the computation DoS using the SSL renegotiation. However, this technique is likely to produce number of false positive result. Therefore, this technique cannot become the de-facto standard to completely mitigate the renegotiation based computational DoS.