It is possible to use IKE over TCP, but this demands a TCP connection to be always open; the open session reserves the socket on the Gateway, taking up valuable system resources. The more reasonable solution is to keep open the port on the NATing device by sending UDP “keep alive” packets to the Gateway, and then performing IKE phase II in the usual way. However, there is still a need to shorten the UDP packets to prevent possible fragmentation.
Small IKE Phase II Proposals
Both Gateway and remote peer start the IKE negotiation by proposing a small number of methods for encryption and integrity. The more common methods are included in the small proposals.
If proposals match between the remote client and the Gateway, the proposed methods are used; if no match is found, a greater number of proposals are made. Usually a match is found with the small proposals, and fragmentation is no longer an issue. However, there are cases where a match is not found, and a larger number of proposals need to be made. (This will most likely happen in instances where the remote Gateway uses AES-128 for encryption, and AES-128 is not included in the small proposals.)
A greater number of proposals can result in larger UDP packets. These larger packets are once again fragmented at the Data Link Layer of the TCP/IP stack on the client, and then discarded by the hide NAT device that does not support fragmentation. In the case of AES-128, this method of encryption can be included in the small proposals by defining AES-128 as the preferred method.
During IPSec
NAT traversal (UDP Encapsulation for Firewalls and Proxies)
Having successfully negotiated IKE phases I and II, we move into the IPSec stage. Data payloads encrypted with (for example) 3DES and hashed (for integrity) with MD5, are placed within an IPSec packet. However, this IPSec packet no longer contains a TCP or UDP header. A hide NAT device needs to translate the port information inside the header. The TCP/UDP header has been encrypted along with the data payload and can no longer be read by the NATing device.
A port number needs to be added; UDP Encapsulation is a process that adds a special UDP header that contains readable port information to the IPSec packet:
FIGURE 11-2 UDP Encapsulation:
The new port information is not the same as the original. The port number 2746 is included in both the source and destination ports. The NAT device uses the source port for the hide operation but the destination address and port number remains the same. When the peer Gateway sees 2746 as the port number in the destination address, the Gateway calls a routine to decapsulate the packet.
IPSec Path Maximum Transmission Units
IPSec Path MTU is a way of dealing with IPSec packet fragmentation. The Data Link layer imposes an upper limit on the size of the packets that can be sent across the physical network, the Maximum Transmission Unit, or MTU. Before sending a packet, the TCP/IP stack of the operating system queries the local interface to obtain its MTU. The IP layer of the TCP/IP stack compares the MTU of the local interface with the size of the packet and fragments the packet if necessary.
When a remote client is communicating across multiple routers with a Gateway, it is the smallest MTU of all the routers that is important; this is the path MTU (PMTU), and for remote access clients there is a special IPSec PMTU discovery mechanism to prevent the OS of the client from fragmenting the IPSec packet if the IPSec packet is too large.
However, the PMTU between the remote client and the Gateway will not remain constant, since routing across the Internet is dynamic. The route from Gateway to client may not be the same in both directions, hence each direction may have its own PMTU. VPN-1 handles this in two ways:
During IPSec
Active IPSec PMTU
After IKE phase II but before the IPSec stage, the remote access client sends special discovery IPSec packets of various sizes to the Gateway. The DF (do not fragment) bit on the packet is set. If a packet is longer than any router’s MTU, the router drops the packet and sends an ICMP error message to the remote client. From the largest packet not fragmented, the remote client resolves an appropriate PMTU. This PMTU is not conveyed directly to the OS. Unknown to the operating system, during the TCP three-way handshake, the Maximum Segment Size (MSS) on the SYN and SYN-ACK packets are changed to reflect the PMTU. This is known as Active IPSec PMTU.
FIGURE 11-3 IPSec discover packets
Passive IPSec PMTU
Passive IPSec PMTU solves the problem of dynamic Internet routing. Passive IPSec PTMU is a process that occurs when either side receives an ICMP error message resulting from a change in the routing path. Since routes change dynamically on the Internet, if a different router needs to fragment the packet that has the DF bit set, the router discards the packet and generates an ICMP “cannot fragment” error message. The error message is sent to the VPN peer that sent the packet. When the peer receives this error message, the peer decreases the PMTU and retransmits.
Note - From the system administrator’s perspective, there is nothing to configure for PMTU; the IPSec PMTU discovery mechanism, both active and passive, runs automatically.