Brocade Data Center Fabric Architectures

(1)

WHITE PAPER

Brocade Data Center Fabric Architectures

Building the foundation for a cloud-optimized data center.

Based on the principles of the New IP, Brocade is building on the proven

success of the Brocade

®

_VDX

®

_{platform by expanding the Brocade}

cloud-optimized network and network virtualization architectures and delivering

new automation innovations to meet customer demand for higher levels

of scale, agility, and operational efficiency.

The scalable and highly automated Brocade data center fabric

architectures described in this white paper make it easy for infrastructure

planners to architect, automate, and integrate with current and future data

center technologies while they transition to their own cloud-optimized data

center on their own time and terms.

This paper helps network architects, virtualization architects, and network engineers to make informed design, architecture, and deployment decisions that best meet their technical and business objectives. The following topics are covered in detail:

•

Network architecture options for scaling from tens to hundreds of thousands of servers

•

Network virtualization solutions that include integration with leading controller-based and controller-less industry solutions

•

Data Center Interconnect (DCI) options

•

Server-based, open, and programmable

turnkey automation tools for rapid

Evolution of Data Center

Architectures

Data center networking architectures have evolved with the changing require-ments of the modern data center and cloud environments.

Traditional data center networks were a derivative of the 3-tier architecture, prevalent in enterprise campus environments. (See Figure 1.) The tiers are defined as Access, Aggregation, and Core. The 3-tier topology was architected with the requirements of an enterprise campus in mind. A typical network access layer requirement of an enterprise campus is to provide connectivity to workstations. These

When compared to an enterprise campus network, the traffic patterns in a data center network are changing rapidly from north-south to east-west. Cloud applications are often multitiered and hosted at different endpoints connected to the network. The communication between these application tiers is a major contributor to the overall traffic in a data center. In fact, some of the very large data centers report that more than 90 percent of their overall traffic occurs between the application tiers. This traffic pattern is commonly referred to as east-west traffic. Traffic patterns are the primary reasons that data center networks need to evolve into out architectures. These scale-out architectures are built to maximize the throughput for east-west traffic. (See Figure 2.) In addition to providing high east-west throughput, scale-out architectures provide a mechanism to add capacity to the network horizontally, without reducing the provisioned capacity between the existing endpoints. An example of scale-out architectures is a leaf-spine topology, which is described in detail in a later section of this paper. In recent years, with the changing economics of application delivery, a shift towards the cloud has occurred. Enterprises have looked to consolidate and host private cloud services. Meanwhile, application cloud services, as well as public service provider clouds, have grown at a rapid pace. With this increasing shift to the cloud, the scale of the network deployment has increased drastically. Advanced

scale-Core

Agg

Access

Figure 1: Three-Tier Architecture: Ideal for North-South Traffic Patterns Commonly Found in Client-Server Compute Models.

Figure 2: Scale-Out Architecture: Ideal for East-West Traffic Patterns Commonly Found with Web-Based or Cloud-Web-Based Application Designs.

Leaf / Spine

Scale Out

(3)

out architectures allow networks to be deployed at many multiples of the scale of a leaf-spine topology (see Figure 3). In addition to traffic patterns, as server virtualization has become mainstream, newer requirements of the networking infrastructure are emerging. Because physical servers can now host several virtual machines (VM), the scale requirement for the control and data planes for MAC addresses, IP addresses, and Address Resolution Protocol (ARP) tables have multiplied. Also, large numbers of physical and virtualized endpoints must support much higher throughput than a traditional enterprise environment, leading to an evolution in Ethernet standards of 10 Gigabit Ethernet (GbE), 40 GbE, 100 GbE, and beyond. In addition, the need to extend Layer 2 domains across the infrastructure and across sites to support VM mobility is creating new challenges for network architects.

10GbE

DC PoD N Edge Services PoD

Super-Spine Border Leaf WAN Edge Internet DC PoD 1 Spine Leaf DCI

Figure 3: Example of an Advanced Scale-out Architecture Commonly Used in Today’s Large-Scale Data Centers.

For multitenant cloud environments, providing traffic isolation at the networking layers, enforcing security and

traffic policies for the cloud tenants and applications is a priority. Cloud scale deployments also require the networking infrastructure to be agile in provisioning new capacity, tenants, and features, as well as making modifications and managing the lifecycle of the infrastructure. The remainder of this white paper describes data center networking architectures that meet the requirements for building cloud-optimized networks that address current and future needs for enterprises and service provider clouds. More specifically, this paper describes:

•

Example topologies and deployment models demonstrating Brocade VDX switches in Brocade VCS fabric or Brocade IP fabric architectures

•

Network virtualization solutions that

include controller-based virtualization such as VMware NSX and

controller-less virtualization using the Brocade Border Gateway Protocol Ethernet Virtual Private Network (BGP-EVPN)

•

DCI solutions for interconnecting

multiple data center sites

•

Open and programmable turnkey automation and orchestration tools that can simplify the provisioning of network services

Data Center Networks:

Building Blocks

This section discusses the building blocks that are used to build the appropriate network and virtualization architecture for a data center site. These building blocks consist of the various elements that fit into an overall data center site deployment. The goal is to build fairly independent elements that can be assembled together, depending on the scale requirements of the networking infrastructure.

(4)

Networking Endpoints

The first building blocks are the networking endpoints that connect to the networking infrastructure. These endpoints include the compute servers and storage devices, as well as network service appliances such as firewalls and load balancers.

Figure 4 shows the different types of racks used in a data center infrastructure. as described below:

•

Infrastructure and Management Racks: These racks host the management infrastructure, which includes any management appliances or software used to manage the infrastructure. Examples of this are server virtualization management software like VMware vCenter or Microsoft SCVMM, orchestration software like OpenStack or VMware vRealize Automation, network controllers like the Brocade SDN Controller or VMware NSX, and network management and automation tools like Brocade Network Advisor. Examples of infrastructure racks are IP physical or virtual storage appliances.

•

Compute racks: Compute racks host

the workloads for the data centers. These workloads can be physical servers, or they can be virtualized

servers when the workload is made up of Virtual Machines (VMs). The compute endpoints can be single or can be multihomed to the network.

•

Edge racks: The network services

connected to the network are consolidated in edge racks. The role of the edge racks is to host the edge services, which can be physical appliances or VMs.

These definitions of infrastructure/ management, compute racks, and edge racks are used throughout this white paper.

Single-Tier Topology

The second building block is a single- tier network topology to connect

endpoints to the network. Because of the existence of only one tier, all endpoints connect to this tier of the network. An example of a single-tier topology is shown in Figure 5. The single-tier switches are shown as a virtual Link Aggregation Group (vLAG) pair.

The topology in Figure 5 shows the management/infrastructure, compute racks, and edge racks connected to a pair of switches participating in multiswitch port channeling. This pair of switches is called a vLAG pair.

The single-tier topology scales the least among all the topologies described in this paper, but it provides the best choice for smaller deployments, as it reduces the Capital Expenditure (CapEx) costs for the network in terms of the size of the infrastructure deployed. It also reduces the optics and cabling costs for the networking infrastructure.

vLAG Pair

Servers/Blades IP Storage Servers/Blades

Management/Infrastructure Racks Compute Racks Edge Racks

Figure 5: Ports on Demand with a Single Networking Tier.

Servers/Blades IP Storage Servers/Blades

Management/Infrastructure Racks Compute Racks Edge Racks

(5)

Design Considerations for a Single-Tier Topology

The design considerations for deploying a single-tier topology are summarized in this section.

Oversubscription Ratios

It is important for network architects to understand the expected traffic patterns in the network. To this effect, the oversubscription ratios at the vLAG pair should be well understood and planned for.

The north-south oversubscription at the vLAG pair is described as the ratio of the aggregate bandwidth of all the downlinks from the vLAG pair that are connected to the endpoints to the aggregate bandwidth of all the uplinks that are connected to the edge/core router (described in a later section). The north-south oversubscription dictates the proportion of traffic between the endpoints versus the traffic entering and exiting the data center site.

It is also important to understand the bandwidth requirements for the inter-rack traffic. This is especially true for all north-south communication through the services hosted in the edge racks. All such traffic flows through the vLAG pair to the edge racks and, if the traffic needs to exit, it flows back to the vLAG switches. Thus, the aggregate ratio of bandwidth connecting the compute racks to the aggregate ratio of bandwidth connecting the edge racks is an important consideration.

Another consideration is the bandwidth of the link that interconnects the vLAG pair. In case of multihomed endpoints and no failure, this link should not be used for data plane forwarding. However, if there are link failures in the network, then this link may be used for data plane forwarding. The bandwidth requirement

for this link depends on the redundancy design for link failures. For example, a design to tolerate up to two 10 GbE link failures has a 20 GbE interconnection between the Top of Rack/End of Row (ToR/EoR) switches.

Port Density and Speeds for Uplinks and Downlinks

In a single-tier topology, the uplink and downlink port density of the vLAG pair determines the number of endpoints that can be connected to the network, as well as the north-south oversubscription ratios. Another key consideration for single-tier topologies is the choice of port speeds for the uplink and downlink interfaces. Brocade VDX Series switches support 10 GbE, 40 GbE, and 100 GbE interfaces, which can be used for uplinks and downlinks. The choice of platform for the vLAG pair depends on the interface speed and density requirements.

Scale and Future Growth

A design consideration for single-tier topologies is the need to plan for more capacity in the existing infrastructure and more endpoints in the future.

Adding more capacity between existing endpoints and vLAG switches can be done by adding new links between them. Also, any future expansion in the number

of endpoints connected to the single-tier topology should be accounted for during the network design, as this requires additional ports in the vLAG switches. Another key consideration is whether to connect the vLAG switches to external networks through core/edge routers and whether to add a networking tier for higher scale. These designs require additional ports at the ToR/EoR. Multitier designs are described in a later section of this paper.

Ports on Demand Licensing

Ports on Demand licensing allows you to expand your capacity at your own pace, in that you can invest in a higher port density platform, yet license only a subset of the available ports on the Brocade VDX switch, the ports that you are using for current needs. This allows for an extensible and future-proof network architecture without the additional upfront cost for unused ports on the switches. You pay only for the ports that you plan to use.

Leaf-Spine Topology (Two-Tier)

The two-tier leaf-spine topology has become the de facto standard for networking topologies when building medium-scale data center infrastructures. An example of leaf-spine topology is shown in Figure 6.

Leaf

L2 Links

Spine

(6)

The leaf-spine topology is adapted from Clos telecommunications networks. This topology is also known as the “3-stage folded Clos,” with the ingress and egress stages proposed in the original Clos architecture folding together at the spine to form the leaves.

The role of the leaf is to provide connectivity to the endpoints in the network. These endpoints include compute servers and storage devices, as well as other networking devices like routers and switches, load balancers, firewalls, or any other networking endpoint—physical or virtual. As all endpoints connect only to the leaves, policy enforcement including security, traffic path selection, Quality of Service (QoS) markings, traffic scheduling, policing, shaping, and traffic redirection are implemented at the leaves.

The role of the spine is to provide interconnectivity between the leaves. Network endpoints do not connect to the spines. As most policy implementation is performed at the leaves, the major role of the spine is to participate in the control plane and data plane operations for traffic forwarding between the leaves.

As a design principle, the following requirements apply to the leaf-spine topology:

•

Each leaf connects to all the spines in the network.

•

The spines are not interconnected with each other.

•

The leaves are not interconnected with each other for data plane purposes. (The leaves may be interconnected for control plane operations such as forming a server-facing vLAG.)

These are some of the key benefits of a leaf-spine topology:

•

Because each leaf is connected to every spine, there are multiple redundant paths available for traffic between any pair of leaves. Link failures cause other paths in the network to be used.

•

Because of the existence of multiple paths, Equal-Cost Multipathing (ECMP) can be leveraged for flows traversing between pairs of leaves. With ECMP, each leaf has equal-cost routes, to reach destinations in other leaves, equal to the number of spines in the network.

•

The leaf-spine topology provides a basis for a scale-out architecture. New leaves can be added to the network without affecting the provisioned east-west capacity for the existing infrastructure.

•

The role of each tier in the network is

well defined (as discussed previously), providing modularity in the networking functions and reducing architectural and deployment complexities.

•

The leaf-spine topology provides granular control over subscription ratios for traffic flowing within a rack, traffic flowing between racks, and traffic flowing outside the leaf-spine topology.

Design Considerations for a Leaf-Spine Topology

There are several design considerations for deploying a leaf-spine topology. This section summarizes the key considerations.

It is important for network architects to understand the expected traffic patterns in the network. To this effect, the oversubscription ratios at each layer should be well understood and planned for.

For a leaf switch, the ports connecting to the endpoints are defined as downlink ports, and the ports connecting to the spines are defined as uplink ports. The oversubscription ratio at the leaves is the ratio of the aggregate bandwidth for the downlink ports and the aggregate bandwidth for the uplink ports. For a spine switch in a leaf-spine topology, the east-west oversubscription ratio is defined per pair of leaf switches connecting to the spine switch. For a given pair of leaf switches connecting to the spine switch, the oversubscription ratio is the ratio of aggregate bandwidth of the links connecting to each leaf switch. In a majority of deployments, this ratio is 1:1, making the east-west oversubscription ratio at the spine nonblocking.

Exceptions to the nonblocking east-west oversubscriptions should be well understood and depend on the traffic patterns of the endpoints that are connected to the respective leaves. The oversubscription ratios described here govern the ratio of traffic bandwidth between endpoints connected to the same leaf switch and the traffic bandwidth between endpoints connected to different leaf switches. As an

example, if the oversubscription ratio is 3:1 at the leaf and 1:1 at the spine, then the bandwidth of traffic between endpoints connected to the same leaf switch should be three times the bandwidth between endpoints connected to different leaves. From a network endpoint perspective, the network oversubscriptions should be planned so that the endpoints connected to the network have the required bandwidth for communications. Specifically, endpoints that are expected to use higher bandwidth

(7)

should be localized to the same leaf switch (or same leaf switch pair—when endpoints are multihomed).

The ratio of the aggregate bandwidth of all the spine downlinks connected to the leaves to the aggregate bandwidth of all the downlinks connected to the border leaves (described in the edge services and border switch section) defines the north-south oversubscription at the spine. The north-south oversubscription dictates the traffic destined to the services that are connected to the border leaf switches and that exit the data center site.

Leaf and Spine Scale

Because the endpoints in the network connect only to the leaf switches, the number of leaf switches in the network depends on the number of interfaces required to connect all the endpoints. The port count requirement should also account for multihomed endpoints. Because each leaf switch connects to all the spines, the port density on the spine switch determines the maximum number of leaf switches in the topology. A higher oversubscription ratio at the leaves reduces the leaf scale requirements, as well.

The number of spine switches in the network is governed by a combination of the throughput required between the leaf switches, the number of redundant/ ECMP paths between the leaves, and the port density in the spine switches. Higher throughput in the uplinks from the leaf switches to the spine switches can be achieved by increasing the number of spine switches or bundling the uplinks together in port channel interfaces between the leaves and the spines.

Port Speeds for Uplinks and Downlinks

Another consideration for leaf-spine topologies is the choice of port speeds for the uplink and downlink interfaces. Brocade VDX switches support 10 GbE, 40 GbE, and 100 GbE interfaces, which can be used for uplinks and downlinks. The choice of platform for the leaf and spine depends on the interface speed and density requirements.

Scale and Future Growth

Another design consideration for leaf-spine topologies is the need to plan for more capacity in the existing infrastructure and to plan for more endpoints in the future.

Adding more capacity between existing leaf and spine switches can be done by adding spine switches or adding new interfaces between existing leaf and spine switches. In either case, the port density requirements for the leaf and the spine switches should be accounted for during the network design process.

If new leaf switches need to be added to accommodate new endpoints in the network, then ports at the spine switches are required to connect the new leaf switches.

In addition, you must decide whether to connect the leaf-spine topology to external networks through border leaf switches and also whether to add an additional networking tier for higher scale. Such designs require additional ports at the spine. These designs are described in another section of this paper.

Ports on Demand Licensing

Remember that Ports on Demand licensing allows you to expand your capacity at your own pace in that you can invest in a higher port density platform, yet license only the ports on the Brocade VDX switch that you are using for current needs. This allows for an extensible and future-proof network architecture without additional cost.

Deployment Model

The links between the leaf and spine can be either Layer 2 or Layer 3 links. If the links between the leaf and spine are Layer 2 links, the deployment is known as a Layer 2 (L2) leaf-spine deployment or a Layer 2 Clos deployment. You can deploy Brocade VDX switches in a Layer

2 deployment by using Brocade VCS®

Fabric technology. With Brocade VCS Fabric technology, the switches in the leaf-spine topology cluster together and form a fabric that provides a single point for management, distributed control plane, embedded automation, and multipathing capabilities from Layers 1 to 3. The benefits of deploying a VCS fabric are described later in this paper.

If the links between the leaf and spine are Layer 3 links, the deployment is known as a Layer 3 (L3) leaf-spine deployment or a Layer 3 Clos deployment. You can deploy Brocade VDX switches in a Layer 3 deployment by using Brocade IP fabrics. Brocade IP fabrics provide a highly scalable, programmable, standards-based, and interoperable networking infrastructure. The benefits of Brocade IP fabrics are described later in this paper.

(8)

Data Center Points of Delivery

Figure 7 shows a building block for a data center site. This building block is called a data center point of delivery (PoD). The data center PoD consists of the networking infrastructure in a leaf-spine topology along with the endpoints grouped together in management/ infrastructure and compute racks. The idea of a PoD is to create a simple, repeatable, and scalable unit for building a data center site at scale.

Optimized 5-Stage Folded Clos

Topology (Three Tiers)

Multiple leaf-spine topologies can be aggregated together for higher scale in an optimized 5-stage folded Clos topology. This topology adds a new tier to the network, known as the super-spine. The role of the super-spine is to provide connectivity between the spine switches across multiple data center PoDs. Figure 8 on the following

page shows four super-spine switches connecting the spine switches across multiple data center PoDs.

The connection between the spines and the super-spines follow the Clos principles:

•

Each spine connects to all the super-spines in the network.

•

Neither the spines nor the super-spines are interconnected with each other. Similarly, all the benefits of a leaf-spine topology—namely, multiple redundant paths, ECMP, scale-out architecture and control over traffic patterns—are realized in the optimized 5-stage folded Clos topology as well.

With an optimized 5-stage Clos topology, a PoD is a simple and replicable unit. Each PoD can be managed independently, including firmware versions and network configurations. This topology also

allows the data center site capacity to scale up by adding new PoDs or scale down by removing existing PoDs without affecting the existing infrastructure— providing elasticity in scale and isolation of failure domains.

This topology also provides a basis for interoperation of different deployment models of Brocade VCS fabrics and IP fabrics. This is described later in this paper.

Design Considerations for Optimized 5-Stage Clos Topology

The design considerations of

oversubscription ratios, port speeds and density, spine and super-spine scale, planning for future growth, and Brocade Ports on Demand licensing, which were described for the leaf-spine topology, apply to the optimized 5-stage folded Clos topology as well. Some key considerations are highlighted below.

Figure 7: A Data Center PoD.

IP Storage

Spine

Leaf

Servers/Blades 10 GbE Servers/Blades 10 GbE Servers/Blades 10 GbE

Compute Racks

Controller Management SW 10 GbE

Management/Infrastructure Racks

(9)

Because the spine switches now have uplinks connecting to the super-spine switches, the north-south oversubscription ratios for the spine switches dictate the ratio of aggregate bandwidth of traffic switched east-west within a data center PoD to the aggregate bandwidth of traffic exiting the data center PoD. This is a key consideration from the perspective of network infrastructure and services placement, application tiers, and (in the case of service providers) tenant placement. In cases of north-south oversubscription at the spines, endpoints should be placed to optimize traffic within a data center PoD.

At the super-spine switch, the east-west oversubscription defines the ratio of bandwidth of the downlink connections for a pair of data center PoDs. In most cases, this ratio is 1:1.

The ratio of the aggregate bandwidth of all the super-spine downlinks connected to the spines to the aggregate bandwidth of all the downlinks connected to the border leaves (described in the section of this paper on edge services and border switches) defines the north-south oversubscription at the super-spine. The north-south oversubscription dictates the traffic destined to the services connected to the border leaf switches and exiting the data center site.

Deployment Model

Because of the existence of the Layer 3 boundary either at the leaf or at the spine (depending on the Layer 2 or Layer 3 deployment model in the leaf-spine topology of the data center PoD), the links between the spines and super-spines are Layer 3 links. The routing and overlay protocols are described later in this paper.

Layer 2 connections between the spines and super-spines is an option for smaller scale deployments, due to the inherent scale limitations of Layer 2 networks. These Layer 2 connections would be IEEE 802.1q based optionally over Link Aggregation Control Protocol (LACP) aggregated links. However, this design is not discussed in this paper.

Edge Services and

Border Switches

For two-tier and three-tier data center topologies, the role of the border switches in the network is to provide external connectivity to the data center site. In addition, as all traffic enters and exits the data center through the border leaf switches, they present the ideal location in the network to connect network services like firewalls, load-balancers, and edge VPN routers.

Figure 8: An Optimized 5-Stage Folded Clos with Data Center PoDs.

10 GbE 10 GbE 10 GbE 10 GbE

DC PoD N

Spine

Leaf

Compute and Infrastructure/Management Racks Super-Spine

10 GbE 10 GbE 1 0bEG 10 GbE

DC PoD 1

Spine

Leaf

(10)

The topology for interconnecting the border switches depends on the number of network services that need to be attached, as well as the oversubscription ratio at the border switches. Figure 9 shows a simple topology for border switches, where the service endpoints connect directly to the border switches. Border switches in this simple topology are referred to as “border leaf switches” because the service endpoints connect to them directly.

More scalable border switch topologies are possible, if a greater number of service endpoints need to be connected. These topologies include a leaf-spine topology for the border switches with “border spines” and “border leaves.” This white paper demonstrates only the border leaf variant for the border switch topologies, but this is easily expanded to a leaf-spine topology for the border switches. The border switches with the edge racks together form the edge services PoD.

Figure 9: Edge Services PoD.

Design Considerations for Border Switches

The following section describes the design considerations for border switches.

The border leaf switches have uplink connections to spines in the leaf-spine topology and to super-spines in the 3-tier topology. They also have uplink connections to the data center core/Wide-Area Network (WAN) edge routers as described in the next section. These data center site topologies are discussed in detail later in this paper.

The ratio of the aggregate bandwidth of the uplinks connecting to the spines/ super-spines to the aggregate bandwidth of the uplink connecting to the core/edge routers determines the oversubscription ratio for traffic exiting the data center site. The north-south oversubscription ratios for the services connected to the border leaves is another consideration.

Because many of the services connected to the border leaves may have public interfaces facing external entities like core/edge routers and internal interfaces facing the internal network, the north-south oversubscription for each of these connections is an important design consideration.

Data Center Core/WAN Edge Handoff

The uplinks to the data center core/WAN edge routers from the border leaves carry the traffic entering and exiting the data center site. The data center core/WAN edge handoff can be Layer 2 and/or Layer 3 in combination with overlay protocols.

The handoff between the border leaves and the data center core/WAN edge may provide domain isolation for the control and data plane protocols running in the internal network and built using one-tier, two-one-tier, or three-tier topologies. This helps in providing independent

Border Leaf

Servers/Blades 10 GbE

Edge Racks

Load Balancer 10 GbE Firewall SW Router SW VPN SW Firewall

(11)

administrative, fault isolation, and control plane domains for isolation, scale, and security between the different domains of a data center site. The handoff between the data center core/WAN edge and border leaves is explored in brief elsewhere in this paper.

Data Center Core and

WAN Edge Routers

The border leaf switches connect to the data center core/WAN edge devices in the network to provide external connectivity to the data center site. Figure 10 shows an example of the connectivity between border leaves, a collapsed data center core/WAN edge tier, and external networks for Internet and DCI options. The data center core routers might provide the interconnection between data center PoDs built as single-tier, leaf-spine, or optimized 5-stage Clos deployments

Figure 10: Collapsed Data Center Core and WAN Edge Routers Connecting Internet and DCI Fabric to the Border Leaf in the Data Center Site.

Data Center Core / WAN Edge

Internet

Border Leaf

DCI

within a data center site. For enterprises, the core router might also provide connections to the enterprise campus networks through campus core routers. The data center core might also connect to WAN edge devices for WAN and interconnect connections. Note that border leaves connecting to the data center core provide the Layer 2 or Layer 3 handoff, along with any overlay control and data planes.

The WAN edge devices provide the interfaces to the Internet and DCI solutions. Specifically for DCI, these devices function as the Provider Edge (PE) routers, enabling connections to other data center sites through WAN technologies like Multiprotocol Label Switching (MPLS) VPN, Virtual Private LAN Services (VPLS), Provider Backbone Bridges (PBB), Dense Wavelength

Division Multiplexing (DWDM), and so forth. These DCI solutions are described in a later section..

Building Data Center Sites

with Brocade VCS Fabric

Technology

Brocade VCS fabrics are Ethernet fabrics built for modern data center infrastructure needs. With Brocade VCS Fabric technology, up to 48 Brocade VDX switches can participate in a VCS fabric. The data plane of the VCS fabric is based on the Transparent Interconnection of Lots of Links (TRILL) standard, supported by Layer 2 routing protocols that propagate topology information within the fabrics. This ensures that there are no loops in the fabrics, and there is no need to run Spanning Tree Protocol (STP). Also, none of the links are blocked. Brocade

(12)

VCS Fabric technology provides a compelling solution for deploying a Layer 2 Clos topology.

Brocade VCS Fabric technology provides these benefits:

•

Single point of management: With all the switches in a VCS fabric participating in a logical chassis, the entire topology can be managed as a single switch chassis. This drastically reduces the management complexity of the solution.

•

Distributed control plane: Control plane and data plane state information is shared across devices in the VCS fabric, which enables fabric-wide MAC address learning, multiswitch port channels (vLAG), Distributed Spanning Tree (DiST), and gateway redundancy protocols like Virtual Router Redundancy Protocol–Extended (VRRP-E) and Fabric Virtual Gateway (FVG), among others. These enable the VCS fabric to function like a single switch to interface with other entities in the infrastructure.

•

TRILL-based Ethernet fabric: Brocade VCS Fabric technology, which is based on the TRILL standard, ensures that no links are blocked in the Layer 2 network. Because of the existence of a Layer 2 routing protocol, STP is not required.

•

Multipathing from Layers 1 to 3: Brocade VCS Fabric technology provides efficiency and resiliency through the use of multipathing from Layers 1 to 3:

- At Layer 1, Brocade trunking (BTRUNK) enables frame-based load balancing between a pair of switches that are part of the VCS fabric. This ensures that thick, or “elephant” flows do not congest an Inter-Switch Link (ISL).

- Because of the existence of a Layer 2 routing protocol, Layer 2 ECMP is performed between multiple next hops. This is critical in a Clos topology, where all the spines are ECMP next hops for a leaf that sends traffic to an endpoint connected to another leaf. The same applies for ECMP traffic from the spines that have the super-spines as the next hops.

- Layer 3 ECMP using Layer 3 routing protocols ensures that traffic is load balanced between Layer 3 next hops.

•

Embedded automation: Brocade VCS

Fabric technology provides embedded turnkey automation built into Brocade Network OS. These automation features enable zero-touch provisioning of new switches into an existing fabric. Brocade VDX switches also provide multiple management methods, including the Command Line Interface (CLI), Simple Network Management Protocol (SNMP), REST, and Network Configuration Protocol (NETCONF) interfaces.

•

Multitenancy at Layers 2 and 3: With

Brocade VCS Fabric technology, multitenancy features at Layers 2 and 3 enable traffic isolation and segmentation across the fabric. Brocade VCS Fabric technology allows an extended range of up to 8000 Layer 2 domains within the fabric, while isolating overlapping IEEE

802.1q-based tenant networks into separate Layer 2 domains. Layer 3 multitenancy using Virtual Routing and Forwarding (VRF) protocols, multi-VRF routing protocols, as well as BGP-EVPN, enables large-scale Layer 3 multitenancy.

•

Ecosystem integration and

virtualization features: Brocade VCS Fabric technology integrates with leading industry solutions and products like OpenStack, VMware products like vSphere, NSX, and vRealize, common infrastructure programming tools like Python, and Brocade tools like Brocade Network Advisor. Brocade VCS Fabric technology is virtualization-aware and helps dramatically reduce administrative tasks and enable seamless VM

migration with features like Automatic Migration of Port Profiles (AMPP), which automatically adjusts port profile information as a VM moves from one server to another.

•

Advanced storage features: Brocade VDX switches provide rich storage protocols and features like Fibre Channel over Ethernet (FCoE), Data Center Bridging (DCB), Monitoring and Alerting Policy Suite (MAPS), and AutoNAS (Network Attached Storage), among others, to enable advanced storage networking.

The benefits and features listed simplify Layer 2 Clos deployment by using Brocade VDX switches and Brocade VCS Fabric technology. The next section describes data center site designs that use Layer 2 Clos built with Brocade VCS Fabric technology.

(13)

Data Center Site with

Leaf-Spine Topology

Figure 11 shows a data center site built using a leaf-spine topology deployed using Brocade VCS Fabric technology. The data center PoD shown here was built using a VCS fabric, and the border leaves in the edge services PoD was built using a separate VCS fabric. The border leaves are connected to the spine switches in the data center PoD and also to the data center core/WAN edge routers. These links can be either Layer 2 or Layer 3 links, depending on the requirements of the deployment and the handoff required to the data center core/WAN edge routers. There can be more than one edge services PoD in the network, depending

on the service needs and the bandwidth requirement for connecting to the data center core/WAN edge routers.

As an alternative to the topology shown in Figure 11, the border leaf switches in the edge services PoD and the data center PoD can be part of the same VCS fabric, to extend the fabric benefits to the entire data center site.

Scale

Table 1 summarizes scale numbers with key combinations of Brocade VDX platforms at the Places in the Network (PINs) for 10 GbE edge ports and racks for a leaf-spine topology. The following assumptions are made:

•

48 switches in a VCS fabric with 4 spines in a leaf-spine topology

•

2 border leaves used in the topology

•

40 GbE links between the leaves and

the spines: 4 × 40 GbE uplink ports on each leaf that connect to each of the 4 spines

•

40 GbE links between the border leaves and the spines: 4 × 40 GbE uplink ports on each border leaf that connect to each of the 4 spines

•

40 GbE interface breakout to 4 × 10 GbE interfaces where available on the Brocade VDX 6740 Switch and Brocade VDX 6940 Switch platforms for endpoint connections

•

Brocade VDX 8770 Switch platforms that use 27 × 40 GbE line cards with 40 GbE interfaces

Figure 11: Data Center Site Built with a Leaf-Spine Topology and Brocade VCS Fabric Technology.

Spine

Leaf

Compute and Infrastructure/Management Racks Edge Racks

10 GbE 10 GbE

Border Leaf

Internet DCI

Data Center Core/ WAN Edge

DC PoD Edge Services PoD

(14)

Table 1: Scale Numbers for a Data Center Site with a Leaf-Spine Topology Implemented with Brocade VCS Fabric Technology.

Leaf Switch Leaf Count Spine Switch for Data Center Site10 GbE Port Count

6740 34 6940-36Q 1632 6740 44 8770-4 2112 6740 44 8770-8 2112 6940-144S 34 6940-36Q 4352 6940-144S 44 8770-4 5632 6940-144S 44 8770-8 5632 6940-36Q 34 6940-36Q 4352 6940-36Q 44 8770-4 5632 6940-36Q 44 8770-8 5632 8770-4 34 6940-36Q 14144 8770-4 44 8770-4 18304 8770-4 44 8770-8 18304 8770-8 34 6940-36Q 28832 8770-8 44 8770-4 37312 8770-8 44 8770-8 37312

Scaling the Data Center Site

with an Optimized 5-Stage

Folded Clos

If multiple VCS fabrics are needed at a data center site, then the optimized 5-stage Clos topology is used to increase scale by interconnecting the data center PoDs built using leaf-spine topology with Brocade VCS Fabric technology. This deployment architecture is referred to as a multifabric topology using VCS fabrics. An example topology is shown in Figure 12.

In a multifabric topology using VCS fabrics, individual data center PoDs resemble a leaf-spine topology deployed using Brocade VCS Fabric technology. However, the new super-spine tier is used to interconnect the spine switches in the data center PoD. In addition, the border leaf switches are also connected to the spine switches. Note that the super-spines do not participate in a VCS fabric, and the links between the super-spines, spine, and border leaves are Layer 3 links.

Figure 12: Data Center Site Built with a Optimized 5-Stage Folded Clos Topology and Brocade VCS Fabric Technology.

Border Leaf Spine

Leaf _{10 GbE} 10 GbE

DC PoD N

Edge Services PoD Super-Spine

Internet DCI

DC PoD 1

Compute and Infrastructure/Management Racks L2 Links

(15)

Figure 12 shows only one edge services PoD, but there can be multiple such PoDs depending on the edge service endpoint requirements, the oversubscription for traffic that is exchanged with the data center core/WAN edge, and the related handoff mechanisms.

Scale

Table 2 summarizes scale numbers with key combinations of Brocade VDX platforms at the PINs for 10 GbE edge ports and racks for an optimized 5-stage Clos topology. The following assumptions are made:

Table 2: Scale Numbers for a Data Center Site Built as a Multifabric Topology Using Brocade VCS Fabric Technology.

Leaf Switch Spine Switch Super-Spine Switch Leaf Count per Data Center PoD Number of Data Center PoDs 10 GbE Port Count for Data Center Site

6740 6940-36Q 6940-36Q 32 8 12288 6940-144S 6940-36Q 6940-36Q 32 8 32768 6940-36Q 6940-36Q 6940-36Q 32 8 32768 8770-4 6940-36Q 6940-36Q 32 8 106496 8770-8 6940-36Q 6940-36Q 32 8 217088 6740 8770-4 6940-36Q 44 8 16896 6940-144S 8770-4 6940-36Q 44 8 45056 6940-36Q 8770-4 6940-36Q 44 8 45056 8770-4 8770-4 6940-36Q 44 8 146432 8770-8 8770-4 6940-36Q 44 8 298496 6740 6940-36Q 8770-4 32 26 39936 6940-144S 6940-36Q 8770-4 32 26 106496 6940-36Q 6940-36Q 8770-4 32 26 106496 8770-4 6940-36Q 8770-4 32 26 346112 8770-8 6940-36Q 8770-4 32 26 705536 6740 8770-4 8770-4 44 26 54912 6940-144S 8770-4 8770-4 44 26 146432 6940-36Q 8770-4 8770-4 44 26 146432 8770-4 8770-4 8770-4 44 26 475904 8770-8 8770-4 8770-4 44 26 970112

•

48 switches in each data center PoD in a leaf-spine topology

•

4 super-spines and 2 border leaves used in the topology

•

40 GbE links between the leaves and the spines: 4 × 40 GbE uplink ports on each leaf that connect to each of the 4 spines: 4 × 40 GbE interfaces used as uplinks on each leaf

•

40 GbE links between the spines and the super-spines: 4 × 40 GbE uplink ports on each spine that connect to each of the 4 super-spines

•

40 GbE links between the border leaves and the super-spines: 4 × 40 GbE uplink ports on each border leaf that connect to each of the 4 super-spines

•

40 GbE interface breakout to

4 × 10 GbE interfaces where available on the Brocade 6740 and 6940 platforms for endpoint connections

•

Brocade 8770 platforms that use 27 × 40 GbE line cards with 40 GbE interfaces

(16)

Building Data Center Sites

with Brocade IP Fabric

The Brocade IP fabric provides a Layer 3 Clos deployment architecture for data center sites. With Brocade IP fabric, all the links in the Clos topology are Layer 3 links. The Brocade IP fabric includes the networking architecture, the protocols used to build the network, turnkey automation features used to provision, manage, and monitor the networking infrastructure and the hardware

differentiation with Brocade VDX switches. The following sections describe these aspects of building data center sites with Brocade IP fabrics.

Because the infrastructure is built on IP, advantages like loop-free communication using industry-standard routing

protocols, ECMP, very high solution scale, and standards-based

interoperablility are leveraged.

These are some of the key benefits of deploying a data center site with Brocade IP fabrics:

•

Highly scalable infrastructure: Because the Clos topology is built using IP protocols, the scale of the

infrastructure is very high. These port and rack scales are documented with descriptions of the Brocade IP fabric deployment topologies.

•

Standards-based and interoperable protocols: The Brocade IP fabric is built using industry-standard protocols like the Border Gateway Protocol (BGP) and Open Shortest Path First (OSPF). These protocols are well understood and provide a solid foundation for a highly scalable solution. In addition, industry-standard overlay control and data plane protocols like BGP-EVPN and Virtual Extensible Local Area Network (VXLAN) are used to extend Layer 2 domain and extend tenancy domains by enabling Layer 2 communications

•

Active-active vLAG pairs: By

supporting vLAG pairs on leaf switches, dual-homing of the networking

endpoints are supported. This provides higher redundancy. Also, because the links are active-active, vLAG pairs provide higher throughput to the endpoints. vLAG pairs are supported for all 10 GbE, 40 GbE, and 100 GbE interface speeds, and up to 32 links can participate in a vLAG.

•

Layer 2 extensions: In order to enable Layer 2 domain extension across the Layer 3 infrastructure, VXLAN protocol is leveraged. The use of VXLAN provides a very large number of Layer 2 domains to support large-scale multitenancy over the infrastructure. In addition, Brocade BGP-EVPN network virtualization provides the control plane for the VXLAN, providing enhancements to the VXLAN standard by reducing the Broadcast, Unknown unicast, Multicast (BUM) traffic in the network through mechanisms like MAC address reachability information and ARP suppression.

•

Multitenancy at Layers 2 and 3: Brocade IP fabric provides multitenancy at Layers 2 and 3, enabling traffic isolation and segmentation across the fabric. Layer 2 multitenancy allows an extended range of up to 8000 Layer 2 domains to exist at each ToR switch, while isolating overlapping 802.1q tenant networks into separate Layer 2 domains. Layer 3 multitenancy using VRFs, multi-VRF routing protocols, and BGP-EVPN allows large-scale Layer 3 multitenancy. Specifically, Brocade BGP-EVPN Network Virtualization leverages BGP-EVPN to provide a control plane for MAC address learning and VRF routing for tenant prefixes and host routes, which reduces BUM traffic and optimizes the traffic patterns in the network.

•

Support for unnumbered interfaces: Using Brocade Network OS support for IP unnumbered interfaces, only one IP address per switch is required to configure the routing protocol peering. This significantly reduces the planning and use of IP addresses and simplifies operations.

•

Turnkey automation: Brocade automated provisioning dramatically reduces the deployment time of network devices and network virtualization. Prepackaged, server-based automation scripts provision Brocade IP fabric devices for service with minimal effort.

•

Programmable automation: Brocade

server-based automation provides support for common industry automation tools such as Python Ansible, Puppet, and YANG model-based REST and NETCONF APIs. Prepackaged PyNOS scripting library and editable automation scripts execute predefined provisioning tasks, while allowing customization for addressing unique requirements to meet technical or business objectives when the enterprise is ready.

•

Ecosystem integration: The Brocade IP fabric integrates with leading industry solutions and products like VMware vSphere, NSX, and vRealize. Cloud orchestration and control are provided through OpenStack and OpenDaylight-based Brocade SDN Controller support.

Data Center Site with

Leaf-Spine Topology

A data center PoD built with IP fabrics supports dual-homing of network endpoints using multiswitch port channel interfaces formed between a pair of switches participating in a vLAG. This pair of leaf switches is called a vLAG pair. See Figure 13 on the following page.

(17)

The switches in a vLAG pair have a link between them for control plane purposes, to create and manage the multiswitch port channel interfaces. These links also carry switched traffic in case of downlink failures. In most cases these links are

not configured to carry any routed traffic upstream, however, the vLAG pairs can peer using a routing protocol if upstream traffic needs to be carried over the link, in cases of uplink failures on a vLAG switch. Oversubscription of the vLAG

link is an important consideration for failure scenarios.

Figure 14 shows a data center site deployed using a leaf-spine topology and IP fabric. Here the network endpoints

Figure 13: An IP Fabric Data Center PoD Built with Leaf-Spine Topology and a vLAG Pair for Dual-Homed Network Endpoints.

IP Storage

Spine

Leaf

Servers/Blades 10 GbE Servers/Blades 10 GbE Servers/Blades 10 GbE

Compute Racks

Controller Management SW 10 GbE

Management/Infrastructure Racks

L3 Links

Spine Leaf

10 GbE 10 GbE

Border Leaf

Internet DCI

Edge Services PoD L3 Links

(18)

are illustrated as single-homed, but dual homing is enabled through vLAG pairs where required.

The links between the leaves, spines, and border leaves are all Layer 3 links. The border leaves are connected to the spine switches in the data center PoD and also to the data center core/WAN edge routers. The uplinks from the border leaf to the data center core/WAN edge can be either Layer 2 or Layer 3, depending on the requirements of the deployment and the handoff required to the data center core/ WAN edge routers.

There can be more than one edge services PoD in the network, depending on service needs and the bandwidth requirement for connecting to the data center core/WAN edge routers.

Scale

Table 3 summarizes scale numbers with key combinations of Brocade VDX platforms at the PINs for 10 GbE edge ports and racks for a leaf-spine topology. The following assumptions are made:

•

4 spines in the data center PoD

•

2 border leaves used in the topology

•

40 GbE links between the leaves and

the spines: 4 × 40 GbE uplink ports on each leaf that connects to each of the 4 spines

•

40 GbE links between the border leaves and the spines: 4 × 40 GbE uplink ports on each border leaf that connects to each of the 4 spines

•

40 GbE interface breakout to 4 × 10 GbE interfaces where available on the Brocade 6740 and 6940 platforms for endpoint connections

•

Brocade 8770 platforms that use

27 × 40 GbE line cards with 40 GbE interfaces for connections to the border leaves and the leaves when used as a spine switch

•

4 × 10 GbE breakouts that are used with the 27 × 40 GbE line cards when the Brocade 8770 is used as a leaf switch

Scaling the Data Center Site with an Optimized 5-Stage Folded Clos

If a higher scale is required, then the optimized 5-stage Clos topology is used to interconnect the data center PoDs built using Layer 3 leaf-spine topology. An example topology is shown in Figure 15 on the following page.

Figure 15 shows only one edge services PoD, but there can be multiple such PoDs, depending on the edge service endpoint requirements, the amount of oversubscription for traffic exchanged with the data center core/WAN edge, and the related handoff mechanisms.

Scale

Table 4 summarizes scale numbers with key combinations of Brocade VDX platforms at the PINs for 10 GbE edge ports and racks for an optimized 5-stage Clos topology. The following assumptions are made:

•

4 super-spines and 2 border leaves used in the topology

•

40 GbE links between the leaves and the spines: 4 × 40 GbE uplink ports on each leaf that connects to each of the 4 spines; 4 × 40 GbE interfaces are used as uplinks on each leaf

•

40 GbE links between the spines and the super-spines: 4 × 40 GbE uplink ports on each spine that connect to each of the 4 super-spines

•

40 GbE links between the border leaves and the super-spines: 4 × 40 GbE uplink ports on each border leaf that connect to each of the 4 super-spines

•

40 GbE interface breakout to 4 × 10 GbE interfaces where available on the Brocade 6740 and 6940 platforms for endpoint connections

•

Brocade 8770 platforms that use 27 × 40 GbE line cards with 40 GbE interfaces

Table 3. Scale Numbers for a Leaf-Spine Topology with Brocade IP Fabrics in a Data Center Site

Leaf Switch Leaf Count Spine Switch for Data Center Site10 GbE Port Count

6740 34 6940-36Q 1632 6740 106 8770-4 5088 6940-144S 34 6940-36Q 4352 6940-144S 106 8770-4 13568 6940-36Q 34 6940-36Q 4352 6940-36Q 106 8770-4 13568 8770-4 34 6940-36Q 14144 8770-4 106 8770-4 44096 8770-8 34 6940-36Q 28832 8770-8 106 8770-4 89888

(19)

Table 4: Scale Numbers for an Optimized 5-Stage Folded Clos Topology Built with Brocade IP Fabrics in a Data Center Site.

Leaf Switch Spine Switch Super-Spine Switch Leaf Count per Data

Center PoD Number of Data Center PoDs 10 GbE Port Count for Data Center Site

6740 6940-36Q 6940-36Q 32 8 12288 6940-144S 6940-36Q 6940-36Q 32 8 32768 6940-36Q 6940-36Q 6940-36Q 32 8 32768 8770-4 6940-36Q 6940-36Q 32 8 106496 8770-8 6940-36Q 6940-36Q 32 8 217088 6740 8770-4 6940-36Q 104 8 39936 6940-144S 8770-4 6940-36Q 104 8 106496 6940-36Q 8770-4 6940-36Q 104 8 106496 8770-4 8770-4 6940-36Q 104 8 346112 8770-8 8770-4 6940-36Q 104 8 705536 6740 6940-36Q 8770-4 32 26 39936 6940-144S 6940-36Q 8770-4 32 26 106496 6940-36Q 6940-36Q 8770-4 32 26 106496 8770-4 6940-36Q 8770-4 32 26 346112 8770-8 6940-36Q 8770-4 32 26 705536 6740 8770-4 8770-4 104 26 129792 6940-144S 8770-4 8770-4 104 26 346112 6940-36Q 8770-4 8770-4 104 26 346112 Edge Racks Super-Spine Border Leaf WAN Edge Internet DCI 10 GbE 10 GbE 10 GbE 10 GbE 10 GbE 10 GbE

DC PoD N

SPINE

LEAF

Compute and Infrastructure/Management Racks

Edge Services PoD

DC PoD 1

Spine Leaf

L3 Links

(20)

Building Data Center Sites

with Layer 2 and Layer 3

Fabrics

A data center site can be built using Layer 2 and Layer 3 Clos that uses Brocade VCS fabrics and Brocade IP fabrics simultaneously in the same topology. This topology is applicable when a particular deployment is more suited for a given application or use case. Figure 16 shows a deployment with both Brocade VCS based data center PoDs based on VCS fabrics and data center PoDs based on IP fabrics, interconnected in an optimized 5-stage Clos topology.

In this topology, the links between the spines, super-spines, and border leaves are Layer 3. This provides a consistent interface between the data center PoDs and enables full communication between endpoints in any PoD.

Scaling a Data Center Site

with a Data Center Core

A very large data center site can use multiple different deployment topologies. Figure 17 on the following page shows a data center site with multiple 5-stage Clos deployments that are interconnected with each other by using a data center core. The role of the data center core is to provide the interface between the different Clos deployments. Note that the border leaves or leaf switches from each of the Clos deployments connect into the data center core routers. The handoff from the border leaves/leaves to the data center core router can be Layer 2 and/ or Layer 3, with overlay protocols like VXLAN and BGP-EVPN, depending on the requirements.

The number of Clos topologies that can be connected to the data center core depends on the port density and throughput of the data center core devices. Each deployment connecting into the data center core can be a

single-tier, leaf-spine, or optimized 5-stage Clos design deployed using an IP fabric architecture or a multifabric topology using VCS fabrics.

Also shown in Figure 17 is a centralized edge services PoD that provides network services for the entire site. There can be one or more of the edge services PoDs with the border leaves in the edge services PoD, providing the handoff to the data center core. The WAN edge routers also connect to the edge services PoDs and provide connectivity to the external network.

Figure 16: Data Center Site Built Using VCS Fabric and IP Fabric PoDs.

10 GbE 10 GbE

DC PoD N Spine

Leaf

Compute and Infrastructure/Management Racks Edge Racks Edge Services PoD Super-Spine

Internet DCI

DC PoD 1

L3 Clos

L2 Links L3 Links

Border Leaf

(21)

Figure 17: Data Center Site Built with Optimized 5-stage Clos Topologies Interconnected with a Data Center Core. DC PoD 1 DC PoD 2 Super-Spine DC PoD N DC PoD 1 Spine Leaf Spine Leaf Spine Leaf Spine Leaf DC PoD 2 Super-Spine DC PoD N Data Center Core Internet DCI WAN Edge Edge Racks

Edge Services PoD

Control Plane and Hardware

Scale Considerations

The maximum size of the network deployment depends on the scale of the control plane protocols, as well as the scale of hardware Application-Specific Integrated Circuit (ASIC) tables. The control plane for a VCS fabric includes these:

•

A Layer 2 routing protocol called Fabric Shortest Path First (FSPF)

•

VCS fabric messaging services for protocol messaging and state exchange

•

Ethernet Name Server (ENS) for MAC

address learning

•

Protocols for VCS formation: - Brocade Link Discovery Protocol

(BLDP)

- Join and Merge Protocol (JMP)

•

State maintenance and distributed protocols:

- Distributed Spanning Tree Protocol (dSTP)

The maximum scale of the VCS fabric deployment is a function of the number of nodes, topology of the nodes, link reliability, distance between the nodes, features deployed in the fabric, and the scale of the deployed features. A maximum of 48 nodes are supported in a VCS fabric.

In a Brocade IP fabric, the control plane is based on routing protocols like BGP and OSPF. In addition, a control plane is provided for formation of vLAG pairs. In the case of virtualization with VXLAN overlays, BGP-EVPN provides the control plane. The maximum scale of the topology depends on the scalability of these protocols.

For both Brocade VCS fabrics and IP fabrics, it is important to understand the hardware table scale and the related control plane scales. These tables include:

•

MAC address table

•

Host route tables for IP host route lookup

•

Longest Prefix Match (LPM) tables for IP prefix matching

•

Tertiary Content Addressable Memory (TCAM) tables for packet matching

•

Address Resolution Protocol/Neighbor

Discovery (ARP/ND) tables

These tables are programmed into the switching ASICs based on the information learned through configuration, the data plane, or the control plane protocols. This also means that it is important to consider the control plane scale for carrying information for these tables when determining the maximum size of the network deployment.

(22)

Choosing an Architecture for

Your Data Center

Because of the ongoing and rapidly evolving transition towards the cloud and the need across IT to quickly improve operational agility and efficiency, the best choice is an architecture based on Brocade data center fabrics. However, the process of choosing an architecture that best meets your needs today while leaving you flexibility to change can be paralyzing. Brocade recognizes how difficult it is for customers to make long-term technology and infrastructure

investments, knowing they will have to live for years with those choices. For this reason, Brocade provides solutions that help you build cloud-optimized networks with confidence, knowing that your investments have value today—and will continue to have value well into the future.

High-Level Comparison Table

Table 5 provides information about which Brocade data center fabric best meets your needs. The IP fabric columns represent all deployment topologies for IP fabric, including the leaf-spine and optimized 5-stage Clos topologies.

Deployment Scale Considerations

The scalability of a solution is an

important consideration for deployment. Depending on whether the topology is a leaf-spine or optimized 5-stage Clos topology, deployments based on Brocade VCS Fabric technology and Brocade IP fabrics scale differently. The port scales for each of these deployments are documented in previous sections of this white paper.

In addition, the deployment scale also depends on the control plane as well as on the hardware tables of the platform.

Table 5: Data Center Fabric Support Comparison Table.

Customer Requirement VCS Fabric Multifabric VCS with VXLAN IP Fabric EVPN-Based VXLANIP Fabric with

BGP-Virtual LAN (VLAN) extension Yes Yes Yes

VM mobility across racks Yes Yes Yes

Embedded turnkey provisioning and automation

Yes Yes,

in each data center PoD Embedded centralized fabric

management

Yes Yes,

in each data center PoD Data center PoDs optimized for

Layer 2 scale-out

Yes Yes

vLAG support Yes,

up to 8 devices Yes, up to 8 devices Yes, up to 2 devices Yes, up to 2 devices Gateway redundancy Yes,

VRRP/VRRP-E/FVG Yes, VRRP/VRRP-E/FVG Yes, VRRP-E Yes, Static Anycast Gateway Controller-based network virtualization

(for example, VMware NSX)

Yes Yes Yes Yes

DevOps tool-based automation Yes Yes Yes Yes

Multipathing and ECMP Yes Yes Yes Yes

Layer 3 scale-out between PoDs Yes Yes Yes

Turnkey off-box provisioning and automation

Planned Yes Yes

Data center PoDs optimized for Layer 3 scale-out

Yes Yes

Controller-less network virtualization (Brocade BGP-EVPN network virtualization)

(23)

Table 6 provides an example of the scale considerations for parameters in a leaf-spine topology with Brocade VCS fabric and IP fabric deployments. The table illustrates how scale requirements for the parameters vary between a VCS fabric and an IP fabric for the same environment. The following assumptions are made:

•

There are 20 compute racks in the leaf-spine topology.

•

4 spines and 20 leaves are deployed at ToR switches. Physical servers are single-homed.

•

The Layer 3 boundary is at the spine of the VCS fabric deployment and at the leaf in IP fabric deployments.

•

Each peering between leaves and spines uses a separate subnet.

•

Brocade IP fabric with BGP-EVPN extends all Virtual VLANs across all 20 racks.

•

40 1 Rack Unit (RU) servers per rack (a standard rack has 42 RUs).

•

2 CPU sockets per physical server × 1 Quad-core CPU per socket = 8 CPU cores per physical server.

•

5 VMs per CPU core × 8 CPU cores per physical server = 40 VMs per physical server.

•

There is a single virtual Network Interface Card (vNIC) for each VM.

•

There are 40 VLANs per rack.

Table 6: Scale Considerations for Brocade VCS Fabric and IP Fabric Deployments.

Brocade VCS Fabric Brocade IP Fabric Brocade IP Fabric with BGP-EVPN Based VXLAN

Leaf Spine Leaf Spine Leaf Spine

MAC Adresses 40 VMs/server × 40 servers/rack × 20 racks = 32,000 MAC addresses 40 VMs/server × 40 servers/rack × 20 racks = 32,000 MAC addresses 40 VMs/server × 40 servers/rack = 1600 MAC addresses Small number of MAC addresses needed for peering 40 VMs/server × 40 servers/rack × 20 racks = 32,000 MAC addresses Small number of MAC addresses needed for peering VLANs 40 VLANs/rack × 20 racks = 800 VLANs 40 VLANs/rack × 20 racks = 800 VLANs 40 VLANs No VLANs at spine 40 VLANs/rack extended to all 20 racks = 800 VLANs No VLANs at spine

ARP Entries None 40 VMs/server × 40 servers/rack × 20 racks = 32,000 ARP entries 40 VMs/server × 40 servers/rack = 1600 ARP entries Small number of ARP entries for peers 40 VMs/server × 40 servers/ rack × 20 racks = 32,000 ARP entries Small number of ARP entries for peers

L3 Routes (Host + LPM)

None Default gateway for 800 VLANs = 800 L3 routes 40 default gateways + 40 remote subnets × 19 racks + 80 peering subnets = 880 L3 routes 40 subnets × 20 racks + 80 peering subnets = 880 L3 routes 3200 host routes/rack × 20 racks + 80 peering subnets = 64080 L3 routes Small number of L3 routes for peering Layer 3 Default Gateways None 40 VLANs/rack × 20 racks = 800 default gateways 40 VLANs/ rack = 40 default gateways None 40 VLANs/rack × 20 racks = 800 default gateways None

(24)

Fabric Architecture

Another way to determine which Brocade data center fabric provides the best solution for your needs is to compare the architectures side-by-side (see Figure 18). Figure 18 provides a side-by-side comparison of the two Brocade data center fabric architectures. The blue text shows how each Brocade data center fabric is implemented. For example, a VCS fabric is topology-agnostic and uses TRILL as its transport mechanism, whereas the topology for an IP fabric is a Clos that uses IP for transport.

It is important to note that the same Brocade VDX switch platform, Brocade Network OS software, and licenses are used for either deployment. So, when you are making long-term infrastructure purchase decisions, be reassured to know that you need only one switching platform.

Recommendations

Of course, each organization’s choices are based on its own unique requirements, culture, and business and technical objectives. Yet by and large, the scalability and seamless server mobility of a Layer 2 scale-out VCS fabric provides the ideal starting point for most enterprise and cloud providers. Like IP fabrics, VCS fabrics provide open interfaces and software extensibility, if you decide to extend the already capable and proven embedded automation of Brocade VCS Fabric technology.

For organizations looking for a Layer 3 optimized scale-out approach, Brocade IP fabrics is the best architecture to deploy. And if controller-less network virtualization using Internet-proven technologies such as BGP-EVPN is the goal, Brocade IP fabric is the best underlay.

Brocade architectures also provide the flexibility of combining both of these deployment topologies in an optimized 5-stage Clos architecture, as illustrated in Figure 18. This provides flexibility of choice in choosing a different deployment model per data center PoD.

Most importantly, if you find your infrastructure technology investment decisions challenging, you can be confident that an investment in the Brocade VDX switch platform will continue to prove its value over time. With the versatility of the Brocade VDX platform and its support for both Brocade data center fabric architectures, your infrastructure needs will be fully met today and into the future.

Figure 18: Data Center Fabric Architecture Comparison.

L2 ISL Layer 3 Boundary L3 ECMP Layer 3 Boundary Topology: Clos Transport: IP Provisioning: Componentized Scale: 100s of Switches Topology: Agnostic Transport: TRILL Provisioning: Embedded Scale: 48 Switches

Brocade Data Center Fabric Architectures

WHITE PAPER