In the data center context, there are advantages of designing the network on layer 2 protocols rather than layer 3. In spite of the difficulties of using a bridge to perform the network role of a router, many vendors, customers, and service providers are attracted to the idea of using Ethernet in as many parts of their networks as possible. The benefits of selecting a layer 2 design are:
Ethernet frames contain all the essentials for networking. These include, but are not limited to, globally unique source addresses, globally unique destination addresses, and error control.
Ethernet frames can carry any kind of packet. Networking at layer 2 is independent of the layer 3 protocol.
More layers added to the Ethernet frame only slow the networking process down. This is known as 'nodal processing delay'.
Adjunct networking features, for example class of service (CoS) or multicasting, can be added to Ethernet as readily as IP networks.
VLANs are an easy mechanism for isolating networks.
Most information starts and ends inside Ethernet frames. Today this applies to data, voice (for example, VoIP) and video (for example, web cameras). The concept is that, if more of the
end-to-end transfer of information from a source to a destination can be done in the form of Ethernet frames, more of the benefits of Ethernet can be realized on the network. Though it is not a substitute for IP networking, networking at layer 2 can be a powerful adjunct to IP networking.
The basic reasoning behind using layer 2 Ethernet over layer 3 IP networks is the speed, the reduced overhead of the IP hierarchy, and the lack of requirement to keep track of IP
address configuration as systems are moved around. Whereas the simplicity of layer 2 protocols might work well in a data center with hundreds of physical machines, cloud data centers have the additional burden of needing to keep track of all virtual machine addresses and networks. In these data centers, it is not uncommon for one physical node to support 30-40 instances.
Important Note: Networking at the frame level says nothing about the presence or absence of IP addresses at the packet level. Almost all ports, links, and devices on a network of LAN switches still have IP addresses, as do all the source and destination hosts. There are many reasons for the continued need for IP addressing. The largest one is the need to manage the network. A device or link without an IP address is usually invisible to most management applications. Utilities including remote access for diagnostics, file transfer of configurations and software, and similar applications cannot run without IP addresses as well as MAC addresses.
Layer 2 Architecture Limitations
Outside of the traditional data center the limitations of layer 2 network architectures become more obvious.
Number of VLANs is limited to 4096.
The number of MACs stored in switch tables is limited.
The need to maintain a set of layer 4 devices to handle traffic control must be accommodated.
MLAG, often used for switch redundancy, is a proprietary solution that does not scale beyond two devices and forces vendor lock-in.
It can be difficult to troubleshoot a network without IP addresses and ICMP.
Configuring ARP is considered complicated on large layer 2 networks.
All network devices need to be aware of all MACs, even instance MACs, so there is constant churn in MAC tables and network state changes as instances are started or stopped.
Migrating MACs (instance migration) to different physical locations are a potential problem if ARP table timeouts are not set properly.
It is important to know that layer 2 has a very limited set of network management tools. It is very difficult to control traffic, as it does not have mechanisms to manage the network or shape the traffic, and network troubleshooting is very difficult. One reason for this
difficulty is network devices have no IP addresses. As a result, there is no reasonable way to check network delay in a layer 2 network.
On large layer 2 networks, configuring ARP learning can also be complicated. The setting for the MAC address timer on switches is critical and, if set incorrectly, can cause significant performance problems. As an example, the Cisco default MAC address timer is extremely long. Migrating MACs to different physical locations to support instance migration can be a significant problem. In this case, the network information maintained in the switches could be out of sync with the new location of the instance.
In a layer 2 network, all devices are aware of all MACs, even those that belong to instances.
The network state information in the backbone changes whenever an instance is started or stopped. As a result there is far too much churn in the MAC tables on the backbone
switches.
Layer 3 Architecture Advantages
In the layer 3 case, there is no churn in the routing tables due to instances starting and
stopping. The only time there would be a routing state change would be in the case of a Top of Rack (ToR) switch failure or a link failure in the backbone itself. Other advantages of using a layer 3 architecture include:
layer 3 networks provide the same level of resiliency and scalability as the Internet.
Controlling traffic with routing metrics is straightforward.
layer 3 can be configured to use BGP confederation for scalability so core routers have state proportional to number of racks, not to the number of servers or instances.
Routing ensures that instance MAC and IP addresses out of the network core reducing state churn. Routing state changes only occur in the case of a ToR switch failure or backbone link failure.
There are a variety of well tested tools, for example ICMP, to monitor and manage traffic.
layer 3 architectures allow for the use of Quality of Service (QoS) to manage network performance.
Layer 3 Architecture Limitations
The main limitation of layer 3 is that there is no built-in isolation mechanism comparable to the VLANs in layer 2 networks. Furthermore, the hierarchical nature of IP addresses means that an instance will also be on the same subnet as its physical host. This means that it
cannot be migrated outside of the subnet easily. For these reasons, network virtualization needs to use IP encapsulation and software at the end hosts for both isolation, as well as for separation of the addressing in the virtual layer from addressing in the physical layer. Other potential disadvantages of layer 3 include the need to design an IP addressing scheme
rather than relying on the switches to automatically keep track of the MAC addresses and to configure the interior gateway routing protocol in the switches.