White Paper
Executive Overview
This white paper describes a blueprint for network consolidation in the enterprise data center leveraging Ethernet as the unified fabric. To date, the deployment of Ethernet as the converged fabric to consolidate networking, storage and clustering traffic has largely not occurred because IT managers have had legitimate concerns about the proper isolation of the different traffic types, inability to leverage their existing SAN investments and increased operational complexities of a shared infrastructure. This white paper describes how emerging storage and networking technologies are providing the missing pieces to enable a true unified fabric that addresses the aforementioned concerns. The resulting converged fabric solutions will address data center IT managers’ requirements for increased flexibility, consolidation, power-reduction and increasing needs for storage and virtualized environments. This white paper also provides a perspective on how the new unified fabric fits into the storage industry’s roadmap for Fibre Channel and iSCSI technologies.
FCoE Enabled Network Consolidation
in the Enterprise Data Center
History of Unified Ethernet
The phrase “unified Ethernet” first emerged in the early part of this decade. The notion of using Ethernet to consolidate disparate networking, storage and clustering fabrics quickly caught the industry’s attention because of the promise of cost-savings and infrastructure simplification. At the time, the concept was fueled by the transition from Fast Ethernet to Gigabit Ethernet and the emergence of the iSCSI-storage and iWARP-RDMA1 technologies.
Three fundamental issues impeded the deployment of unified Ethernet:
The traditional Ethernet fabric could not provide
adequate isolation and protection between the traffic types in the data center network—IT managers were
concerned that LAN traffic will jeopardize both the security and reliability of SAN traffic on the same fabric given the more unpredictable and more open nature of LAN traffic. Ethernet performs poorly under congestion and the resultant packet drops adversely affect storage traffic. As a result, the majority of iSCSI deployments to date have utilized a dedicated iSCSI “SAN” which precludes any cost savings from unification.
iSCSI does not leverage data center customers’
existing investment in storage SANs—Fibre Channel
SANs are broadly deployed in large enterprise data centers. IT managers had deployed sophisticated, Fibre Channel specific tools to efficiently manage storage assets. From the perspective of these customers, iSCSI was a very different storage technology that required an incremental, non-trivial investment in hardware, software and training to deploy in the data center. The combination of lack of cost/simplification benefits and the inability to effectively leverage existing investments had significantly limited iSCSI penetration in the enterprise data centers. However, iSCSI has found success in segments such as remote offices of large enterprises, small and medium enterprises and department-level data centers where customers were migrating from Direct-Attach-Storage to SANs.
Operational complexities of converging different
fabrics—In larger data centers, the storage and
networking infrastructures are typically managed by different IT staff. Convergence onto a single infrastructure introduces operational complexities such as resource allocation, coordinating down-times, firmware upgrades, etc.
Figure 1—Historical vision for unified Ethernet.
XB2B8PaaPhb ^a52PaaPhbeXPXB2B8 52VPcTfPh F0= 4cWTa]TcUPQaXR BTaeTab BX]V[TRPQ[T XB2B8 2^]eTaVTS PSP_cTa 2^]eTaVTS PSP_cTa 4cWTa]TcbfXcRWTb 4cWTa]TcbfXcRWTb XF0A? C2? 8?
Now well known, iSCSI technology is designed to transport SCSI storage traffic over a TCP/IP (typically Ethernet-based) network and its specification was ratified by IETF IP Storage Working Group in February 2003. Similarly, with the objective of providing a low-latency protocol for applications such as clustering, the iWARP specification was ratified by the IETF in 2002. The stage was now set for Unified Ethernet deployment in the data center using TCPIP for networking, iSCSI for storage and iWARP for clustering. But it didn’t happen.
White paper
Emerging Technologies
The Institute of Electrical and Electronic Engineers (IEEE) developed two protocols to provide traffic isolation and service differentiation, IEEE 802.1Q Virtual Local Area Network (VLAN) and 802.1P Quality of Service (QoS). While they provide an ability to have multiple logical networks or VLANs on a single physical network, and to prioritize traffic within these VLANs, respectively, the protocols did not provide mechanisms to prevent an application from congesting the network and negatively affecting other network traffic. In such a scenario, traffic is discarded or dropped, increasing latency and causing frame loss. Members of the IEEE recognized an opportunity for Ethernet as a consolidated Layer 2 solution in high speed, short range networks, such as on a blade chassis backplane and within a data center with the addition of congestion management. Efforts within the Data Center Bridging workgroup are well underway to provide the necessary congestion management for unified fabrics within a data center. Four key protocols are being defined or enhanced as shown in Table 1.
Table 1—Ethernet enhancements for congestion management.
Priority Flow Control (PFC) Enhanced Transmission Selection (ETS) Data Center Bridging Capabilities Exchange Protocol (DCBCXP) Congestion Management (CM)
Provides ability to manage bursty, single traffic source on a multi-protocol link
Bandwidth management between traffic types for multi-protocol links Allows auto exchange of Ethernet parameters between peers (switch to NIC, switch to switch)
Addresses problem of sustained congestion, driving corrective action to the edge
P802.1Qbb
P802.1Qaz
802.1AB
P802.1Qau
The enhancements supported by P802.1Qbb, P802.1Qaz, 802.1AB provide adequate functionality for implementing lossless characteristics at the access network such as the connectivity from the server to the top-of-rack (ToR) switch, The P802.1Qau enhancements address packet loss and provide necessary congestion management to allow segregated lanes of traffic with true isolation in the core network. When complete, the P802.1Qau will enable larger deployments that enable an end-to-end lossless Ethernet fabric. The P802.1Qau based congestion management is much like the use of freeway metering lights. Congestion is pushed from the core of the network to the edge, and then rate limiters (freeway metering lights) are used to meter the traffic back into the network (freeway). Simulations of standards-based techniques under development show fairness between traffic flows and much higher overall throughput than without congestion management. Several protocols exist to run storage on Ethernet in the data center, most notably iSCSI and iFCP. They both require the use of TCP/IP, adding unnecessary overhead in a data center environment. iSCSI also doesn’t address a fundamental issue to Data Center storage professionals— that iSCSI has a different SCSI encapsulation, and therefore uses different host software and management tools than Fibre Channel. This prevents iSCSI from efficiently leveraging data center customers’ existing investment in storage SANs. The INCITS T11 committee has developed a standard for the encapsulation of Fibre Channel over Ethernet (FCoE) – T11-BB-5. This provides an efficient storage protocol for use on a lossless Ethernet network within the data center as outlined in Table 2.
Table 2—Comparison of iSCSI, iFCP and FCoE.
Parameter
TCP/IP required Performance Framing overhead FC security model FC host drivers & firmware FC management model iSCSI Yes Low High No No No iFCP Yes Low High Yes Yes Yes FCoE No High Low Yes Yes Yes
Network Consolidation Solutions Based on the
Converged Ethernet Fabric
Figure 4 illustrates a network consolidation solution utilizing a Converged Ethernet Fabric.
FCoE fully leverages the lossless characteristics supported by the Ethernet enhancements to carry storage traffic over Ethernet. FCoE encapsulates complete Fibre Channel frames onto enhanced Ethernet frames, as shown in Figure 2, without the need for an intermediate and processing intensive transport protocol.
Figure 3 summarizes how the emerging technologies described above supplement existing Ethernet technologies to create a blueprint for a Converged Ethernet Fabric.
The following section describes how this collection of technologies combines to offer compelling converged fabric solutions for the data center.
This solution has the following components:
Converged Network Adapter (CNA)—A multi-protocol
adapter that supports Fibre Channel storage and networking/clustering protocols. Via a Fibre Channel driver, the CNA presents the functionality of a “traditional” Fibre Channel HBA to the host server’s operating system. Via NIC/clustering drivers, it presents the functionality of a “traditional” networking/clustering device to the host server’s operating system. Fibre Channel traffic would be encapsulated into FCoE frames—per the emerging standards described above— and converged with networking/clustering traffic utilizing separate traffic classes for each.
Figure 2—FCoE encapsulates FC frames directly into the Ethernet payload.
4cWWTPSTa 52^4 52 2A2
Figure 4—A consolidated network solution using the converged Ethernet fabric. F0= 644 6Q4 # '652 710 =82SaXeTa R[dbcTaX]V SaXeTab 52SaXeTa 52bfXcRW 52bfXcRW 44bfXcRW 52^4 52VPcTfPh 44bfXcRW 4gXbcX]VB0=T]eXa^]\T]c BTaeTafXcW
caPSXcX^]P[P__a^PRW R^]eTaVTSPSP_cTaBTaeTafXcW
2^]eTaVTS 4cWTa]TcUPQaXR =82 2^]eTaVTSPSP_cTa =82SaXeTa R[dbcTaX]V SaXeTab 52SaXeTa BX]V[T RPQ[T
Figure 3—Converged Ethernet Fabric: Key emerging technologies.
4]WP]RTS4cWTa]Tc
A^QdbcXb^[PcX^]^UcaPUUXRR[PbbTb' ! PdTcR ;PhTa P]S!_WhbXRP[P]SSPcP[X]Z ;PhTa"P]SPQ^eT]Tcf^aZcaP]b_^acTcR
=Tcf^aZX]V Bc^aPVT 2[dbcTaX]V
C2? 8? D3? TcR 52^4 ]Tf0=B8 _a^_^bP[ XF0A?
White paper
FCoE capable Ethernet switch—An Ethernet switch
supporting at a minimum Priority Flow Control, Enhanced Transmission Selection and Data Center Bridging
Capabilities Exchange Protocol (DCBCXP). The switch (and resulting fabric) would be capable of isolating up to eight different traffic classes presented to it by the end-points. One or more of these switches must be configured to support FCoE forwarding function (see below) along with support for Fibre Channel fabric services.
FCoE forwarder—A function within a switch that
translates the FCoE frames between the Enhanced Ethernet fabric and a Fibre Channel SAN. On the Ethernet side, this device is connected to or is integrated into an Enhanced Ethernet switch; FCoE frames from multiple FCoE end-points are sent to one or more devices. On the Fibre Channel SAN side, native Fibre Channel ports connect to a traditional Fibre Channel switch. From the perspective of the Fibre Channel switch, the Fibre Channel port can be made to look like either a Fibre Channel switch1 or a multi-N-port initiator.2
Benefits
From the perspective of an IT manager, the converged fabric solution described above provides the following benefits:
Cost, simplification and flexibility—As described
above, Enhanced Ethernet addresses IT concerns on proper isolation of traffic classes which enables a true converged fabric. An IT manager may view this as a LAN fabric (a mandatory investment) that can safely and optionally carry storage and clustering traffic. This solution delivers the cost and simplification benefits of a single set of adapters, cables and switches—and the flexibility to add services as needs change.
Leverages and seamlessly integrates with a data
center’s existing SAN—The converged fabric solution
does not replace a data center’s existing Fibre Channel SAN. On the SAN side of the gateway in Figure 4, all hardware (switches, arrays, etc), software (replication, backup, management, storage resource management) and operational practices remain intact. On the server/ converged adapter side, the host operating systems use the same legacy Fibre Channel drivers as HBAs and continue to be managed as Fibre Channel “end-points” (e.g. WWNs, etc).
Minimizes operational overlaps between networking
and storage IT staff—In addition to leveraging a data
center’s SAN, critical storage and networking functions continue to be managed separately as shown in Figure 5. This minimizes the need to unify or dramatically overhaul the operational procedures used by the storage and networking IT staff within a data center. The need for common operational practices is limited to the “common management domain” shown—where the network and storage IT administrators must collaborate on managing the storage traffic class (and associated resources) within the converged fabric. An appropriate metaphor for this common domain is one of an “easement” that the network IT manager creates for the storage IT manager. Figure 5—The converged fabric solution separates storage vs. networking IT management concerns.
Bc^aPVT \P]PVT\T]c S^\PX]
3aXeTabSTeXRT
Ud]RcX^] PSP_cTa7F2^]eTaVTS
=Tcf^aZ \P]PVT\T]c S^\PX] 2^\\^] \P]PVT\T]c S^\PX] 4]S _^X]cb 2^]eTaVTS UPQaXR B0=^aF0= 8?S^\PX] =82SaXeTab cTP\X]VBF <02PSSaTbbTb 52SaXeTab \d[cX _PcWX]VBF FF=bTcR 52bfXcRWTb 52PaaPhb <V\cBF i^]X]VTcR 4]WP]RTS 4cWTa]TcbfXcRWTb P]SRPQ[Tb CaPUUXRR[PbbP[[^RPcX^] F0=8?a^dcTab
The Converged Ethernet Fabric solution above addresses the three primary impediments to the historical vision for Unified Ethernet (described earlier in this paper) and provides a compelling converged fabric for the data center. The converged Ethernet fabric does this by providing a robust, lossless fabric with traffic isolation and protection. It also preserves Fibre Channel investments and preserves the SAN management domain maintained within consolidated network.
1 Using E-Port or F-Port Configuration 2 Using N_Port ID Virtualization (NPIV)
The Converged Ethernet Fabric Complements
Other Data Center Trends
Network consolidation solutions based on this converged Ethernet fabric support other dominant trends occurring in the enterprise data center.
Adoption of server virtualization and blades—With
similar IT motivations for improvements in infrastructure efficiency and flexibility, the adoption of server
virtualization and server blades has taken hold over the last several years and is expected to continue to ramp at a healthy pace. The adoption of server virtualization and blades drive the need for more I/O bandwidth and share a common attribute in that they favor the use of shared external storage3 and are creating a pent up
demand for SAN storage. This is true even where server virtualization and blades are being used for front-tier and mid-tier applications that have traditionally not been SAN-attached. The converged Ethernet fabric provides an ideal network consolidation solution—that leverages Fibre Channel SANs (already in place to support back-end applications)—to extback-end Fibre Channel storage over Ethernet to front and mid-tier apps that are deployed on the growing infrastructure of virtualized servers and blade servers.
Adoption of 10G networking for data center
consolidation—At the time of writing, the deployment
of 10G networking in the data center has been limited by the end-to-end costs of 10G products (adapters, cables, switch-ports). Nevertheless, both the industry and IT managers are forecasting these costs to decrease over time—similar to the transition from Fast Ethernet to Gigabit Ethernet—which is estimated to enable volume deployment for data center consolidation starting in the 2009 timeframe. With the ability to consolidate not just networking fabrics but networking, storage and clustering fabrics, 10G network consolidation solutions based on the converged Ethernet fabrics are expected to become the key driver of deployment of 10G for the data center and replace the deployment of “vanilla” 10G technologies. These converged 10G solutions are expected to benefit from economies of scale and offer cost benefits as converged networks transition to volume deployments.
The greening of the data center—As power and cooling
has become a dominant portion of the costs of operating a data center, IT managers are seeking every possible opportunity to reign in power consumption. While IT managers have traditionally focused the primary server “culprits”, their attention has increasingly turned towards the storage and networking infrastructure.4 With the
ability to converge multiple fabrics, converged fabric solutions based on the converged Ethernet fabric are expected to become an obvious data center initiative for power reduction.
Industry Impact of the Converged
Ethernet Fabric
Based on rough estimates of how long it will take the IEEE and ANSI working groups to agree on the standards described earlier, converged solutions based on these standards are expected to arrive in the 2009 and beyond timeframe. When these products arrive, Figure 6 illustrates how the landscape for storage-networking technologies will be reshaped.
Traditionally, only high-end/back-end servers within the data center have been attached to external high performance storage via a dedicated Fibre Channel SAN. These environments are dominated by high availability, reliability and performance thus dictating a strong need for dedicated fabric solutions. In these environments, the use of separate networking and storage fabrics is expected to continue. Data networking in this segment is expected to be optimized around 10G Ethernet and storage around 8G Fibre Channel with a roadmap to 16G.
Network consolidation solutions—with their ability to extend Fibre Channel storage via FCoE—are expected to increase the percentage of data center servers that are Fibre Channel attached (beyond the high-end/back-end servers) by providing Fibre Channel storage to front, mid-tier and department-level servers. This addresses growing need for SAN storage in these tiers that is being driven by blade server deployments and by server virtualization deployments.
With its ability to save cost via consolidation, the use of converged Ethernet fabrics is also expected to penetrate into the remote-office and small-medium-enterprise segments over time. In these segments, we would
3 Reference: Why blade server virtualization and Fibre Channel SAN connectivity are rapidly being adopted (www.emulex.com/white/hba/virtServerWhyAdopted.pdf) 4 Datacom Equipment Power Trends and Cooling Applications, 2005, ASHRAE
White paper
expect to see a mix of FCoE and iSCSI depending on a given customers preference and comfort-level with Fibre Channel vs. IP-based technologies. As these segments are generally expected to adopt 10 Gigabit Ethernet after it is adopted in enterprise data centers, we expect the corresponding switch products supporting enhanced Ethernet and FCoE to follow the availability of similar products for the data center.
FCoE to Enable SAN Market Expansion
Blade server deployments and server virtualization initiatives are increasing the demand for external SAN storage. FCoE addresses this demand for storage
connectivity through the converged Ethernet network while leveraging the robustness and maturity of Fibre Channel. These projects will drive the deployment of converged networks in the data center, making the deployment a strategic decision driven by the storage requirements. Customers will look to suppliers with a history of meeting these requirements in the data center for solutions. Emulex is working with other industry leaders to develop the solutions and standards required for a converged Ethernet fabric. Many suppliers are supporting the development of Enhanced Ethernet and Fibre Channel over Ethernet technologies. Emulex brings a wealth of experience in delivering reliable, robust storage connectivity products to market, and is well positioned to address the market needs for converged Ethernet fabrics in next generation data centers.
Phases of Unified Fabric Deployment
The deployment of Unified Fabrics in Data Centers is expected to be a methodical process over a number of years. Four primary phases are expected to take place over the next five years.
Phase I: Pilot Deployment Phase
This initial phase began in mid-2008 with the initial general availability of FCoE products. The bulk of customer installations during this phase are expected to be pilot deployments, primarily due to lack of OEM and OSM qualifications, and standards maturity. Concurrent with this early availability the FCoE standard is being completed in INCITS and the necessary Standards for the prerequisite Ethernet enhancements are being completed in the Ethernet Alliance.
These early market deployments are primarily focused on server I/O consolidation. With this focus, server administrators will be working through the transition from managing distinct NICs and HBAs to managing Converged Network Adapters. With the deployment of unified Top-of-Rack switches, role-based administration will support the management of SAN resources by the storage
administrator, and LAN resources by the LAN administrator. Figure 6—The converged Ethernet fabric reshapes the storage-networking landscape: 2009 and beyond.
4cWTa]Tc) ³³ 6 2^]eTaVTS4cWTa]TcUPQaXR 6 2^]eTaVTS4cWTa]TcUPQaXR ³³ 6 5XQaT2WP]]T[ '³³ %6 52 52^4 =0B 52XB2B8 =0B30B 4cWTa]Tc) 6 C^ cP[\PaZ TcU ^abc ^a PV T ]Tcf ^aZX]V_a ^SdRc b 1PRZ T]S)^]V^X]V bca^]V]TTSU^a STSXRPcTSUPQaXRb 5a^]c\XS cXTa) Va^fX]V]TTSU^a bc^aPVTR[dbcTaX]VeXP R^]eTaVTSUPQaXRb B^\TXB2B8_aTUTaT]RT SdTc^UP\X[XPaXchfXcW 8?cTRW]^[^VXTb AT\^cT ^UUXRT B<4 4]cTa_aXbT SPcTRT]cTa B<1 BTaeTaUPQaXRb)PSP_cTabbfXcRWTb
White paper
www.emulex.com
This document refers to various companies and products by their trade names. In most, if not all cases, their respective companies claim these designations as trademarks or registered trademarks. This information is provided for reference only. Although this information is believed to be accurate and reliable at the time of publication, Emulex assumes no responsibility for errors or omissions. Emulex reserves the right to make changes or corrections without notice. This report is the property of Emulex and may not be duplicated without permission from the Company.
World Headquarters 3333 Susan Street, Costa Mesa, CA 92626 +1 714 662 5600 Wokingham, UK +44 (0) 118 977 2929 | Munich, Germany +49 (0) 89 97007 177 Paris, France +33 (0) 158 580 022 | Beijing, China +86 10 68499547
09-329 · 10/08
Phase II: Volume Deployment Phase
This phase is expected for new data center deployments in late 2009 and early 2010. Organizational dynamics will likely keep deployments focused on server I/O consolidation. Blade server deployments will begin as single-chip CNA mezzanine cards become available. Low cost interconnect will be continue to be based on SFP+ Direct Attach and Top-of-Rack or Middle-of-Row switching architectures. As third generation 10GBASE-T transceivers become available, this will enable end of row and other more centralized designs where low cost reach beyond 7 meters is enabled.
Phase III: End-to-End Convergence Phase
This phase is expected for new data center deployments in the 2011 and beyond timeframe. As organizations evolve, the processes will be developed to optimize the management of unified fabric, such as consolidated change control, fault isolation of virtual Fibre Channel resources and traffic engineering. A unified fabric also requires additional protocol support, including end to end congestion management being developed in the IEEE 802.1Qau project, and upgrades to spanning tree being developed in the IETF with the TRILL project.
In this phase, storage devices natively running FCoE are expected to make their presence. Organizational dynamics will likely keep deployments focused on server I/O
consolidation.
Conclusion
The overarching need for improvements in infrastructure efficiency and flexibility is driving organizations to deploy blade servers and server virtualization solutions. The rapidly growing demand for SAN, driven by these server initiatives, is making IT managers look for network convergence solutions that will improve the overall cost structure and operational efficiency of the data center. This technology blueprint presents a comprehensive solution to network consolidation through the use of unified Ethernet fabric. The unified Ethernet is made possible through the Ethernet enhancements and through the evolutionary FCoE technology. FCoE enabled unified fabric simplifies network infrastructure, protects existing investments in Fibre Channel SAN and retains the existing SAN management tools and processes. Emulex has been actively working with its partners to deliver compelling unified fabric solutions.
This blueprint further equips the IT managers with a roadmap for phased adoption of convergence solutions so to enable their organizations gain early advantage with converged network deployments.
Figure 7—FCoE enabled network convergence enables organizations to stage the deployments of converged network into multiple phases.
