Performance
September 16, 2011
Abstract Abstract Abstract Abstract
This paper provides a preliminary analysis of the performance capabilities of the Server Message Block (SMB) 2.2 file sharing protocol with 10 gigabit Ethernet interfaces. The Multi-Channel feature, introduced with SMB 2.2 in Windows
Developer Preview, enables the use of multiple physical network interfaces in an SMB 2.2 client and server. This paper assumes the reader is familiar with the basics of SMB file sharing, networking technologies, and file system performance measurement with the Iometer tool.
This information applies to the following operating systems:
Windows Developer Preview Windows Server Developer Preview
The current version of this paper is maintained on the Web at:
Windows 8 SMB 2.2 File Sharing Performance
Disclaimer Disclaimer Disclaimer
Disclaimer: This document is provided “as-is”. Information and views expressed in this document, including
URL and other Internet website references, may change without notice. Some information relates to pre- released product which may be substantially modified before it’s commercially released. Microsoft makes no warranties, express or implied, with respect to the information provided here. You bear the risk of using it.
This document does not provide you with any legal rights to any intellectual property in any Microsoft product. You may copy and use this document for your internal, reference purposes.
© 2011 Microsoft. All rights reserved.
Contents
Introduction... 3
Hardware Shifts and Trends...4
Targeted Workloads...4
SMB Connection Scaling...5
Connection Resiliency...6
Network Utilization...6
Load Balancing... 7
Transport Flexibility... 7
Multi-Channel Behaviors... 7
Multi-Channel Performance (Preliminary)...9
Test Environment...9
Hardware Configuration...9
Iometer Configuration...10
Non-cached I/O... 11
Server Baseline... 11
Server Read Performance...11
Server Write Performance...12
SMB 2.2 Client Performance...12
Server vs. SMB 2.2 Client... 13
Throughput Comparison... 13
Operations Per Second and CPU Consumption Comparison... 14
SMB 2.2 Client Network Interface Scaling... 15
Summary... 17
Introduction
With versions 2.0 and 2.1 of the Server Message Block (SMB) protocol, the SMB client associates a session with a single TCP connection. The session represents a single authentication context between the SMB client and the server. In these versions of SMB, the association of an SMB session to a single TCP connection is unduly limiting for resource utilization and connection resiliency.
The one-to-one mapping of a session to a connection is sufficient for single-user SMB clients that host simple applications with limited demands on networking and system resources. After the use cases are expanded to include large multiuser or multiserver application environments, the one-to-one session to connection limitation represents a bottleneck for CPU, network, and storage utilization.
Issues of resiliency are also present when a single TCP connection is used. Failure of a network interface or intervening network route can result in a loss of connectivity for the SMB client and its applications.
By introducing more than one connection per session, the SMB client has many advantages, such as the following:
• If a network interface or network route failure occurs, the SMB client can be resilient to recover connectivity to the server.
• The SMB client can be more adaptable to server hardware configuration and provide flexibility to an ever changing network infrastructure.
• The SMB client and server can scale proportionately with gains in CPU and networking capabilities.
This white paper will focus on the scaling capability that can be used by SMB 2.2 clients and servers.
The feature of using multiple TCP or Remote DMA (RDMA) connections over one or more physical network interfaces is called SMB 2.2 Multi-Channel. The Multi-Channel feature is just one portion of a larger scope of new capabilities that are provided in Windows Developer Preview. SMB 2.2 integrates the capabilities of RDMA
networking technologies in a way that provides greater performance scale and reduced CPU utilization. These RDMA-based networking technologies include the following:
• InfiniBand
• Internet Wide Area RDMA Protocol (iWARP)
• RDMA over Converged Ethernet (RoCE)
To fully utilize the resiliency capabilities of the Multi-Channel feature, the SMB 2.2
protocol introduces resiliency mechanisms in the SMB 2.2 client and server. These
mechanisms allow the client and server to fully recover from network connection
faults and server failures.
Note:
Note:
Note:
Note: The results in this paper are based on the performance in Windows Developer Preview. Therefore, the results in this paper are preliminary and do not represent any future capabilities.
Hardware Shifts and Trends
There are many system hardware capabilities that are well established and continue to trend positively. The SMB protocol that is used in client and server
implementations have adapted to a degree. However, there is room for improvement for full utilization of system features and performance.
For example, CPU socket and core counts continue their trend upward. The
combination of this CPU capability with virtualization is adding to the requirements for ease of management and greater performance capabilities of the overall storage system. Therefore, the SMB client and server have to offer an adaptable platform for the full range of system capability.
Delivering to the capability of scaling "up" must be accomplished while guaranteeing that smaller systems are not burdened with the accommodation of larger systems.
For example, the SMB client and server have to account for the full range of networking interfaces, such as the following interface types:
• WLANs
• WWANs
• MANs
• Standard Ethernet (100 bits per second or greater)
• Hyper-V virtual interfaces
• Receive-side Scaling (RSS) capable interfaces
• Link Aggregated interfaces, such as load balancing and failover (LBFO)
• RDMA-based interfaces (iWARP, InfiniBand, RoCE).
At the SMB file server, individual disks can be aggregated into a single logical unit number (LUN) or volume. This practice has long been established with either hardware-based or software-based RAID solutions. These aggregated LUNs provide the best throughput for handling a reasonable queue of outstanding I/O requests.
For a workload like SQL Server, the queued I/O count is best at two per physical device. With the introduction of flash-based storage devices, such as discrete PCI or solid-state drives (SSDs), file server storage characteristics are changing. For example, flash-based storage devices may have several internal memory channels and exhibit the best throughput for handling concurrent or larger I/O requests. Storage sub- systems are being delivered to the market that automatically and transparently tier data between SSD and traditional spinning disks. These tiered solutions deliver good performance for both random and sequential workloads while delivering the capacity of traditional storage subsystems.
Targeted Workloads
As noted, the SMB client must be effective at handling the wide range of network
environments and general system types (client and server). The same is true of
application workloads. The focus on one workload or class of workloads must not be done at the expense of others. For Windows Server Developer Preview, one major focus area has been server application workloads, such as SQL Server workloads. In terms of storage I/O requests or transactions per second, the SMB client and server must be able to deliver comparable capabilities as local storage.
Virtualization via Hyper-V and its use of SMB 2.2 storage provides another major conduit of server application workloads, such as the following:
• SQL Server running across a set of virtual machines (VMs).
• Virtual desktop environments that are deployed and executed in large numbers on a single system.
By hosting this variety of applications, the SMB 2.2 client will, by definition, service a variety of workload types, ranging from small, random I/O requests to large
sequential I/O requests. Within a virtualized environment, the SMB 2.2 client will not only service a variety of workloads, but will do so simultaneously.
While the I/O patterns will vary, there are stringent requirements for storage
resiliency in this environment. Server applications require resilient access to storage.
As a result, the SMB 2.2 client must adapt to that requirement along with the new workload patterns.
SMB Connection Scaling
Given the capabilities of hardware, changes in storage workloads, and resiliency requirements, the SMB 2.2 client must move beyond its history and adapt to more effective use of network connections.
An SMB 2.1 session is an authenticated context established between a client and a server over a single TCP connection for a specific security principal. This is depicted in Figure 1.
S e ssio n 1 C onnection 1
S e ssio n 2 C onnection 2
Figure Figure Figure
Figure 1 111:::: SMB SMB SMB SMB 2.1 2.1 2.1 2.1 Session Session Session Session to to to to Connection Connection Connection Connection Association Association Association Association
The one-to-one association between a session and TCP connection works reasonably well when there is a limited need for bandwidth scaling. A single TCP connection is very effective for a 1 gigabit Ethernet (GbE) network interface or a reasonable portion of a 10 GbE network interface. However, a single TCP connection can be limiting because of CPU loading by other workloads on the system that compete for resources that are needed for TCP processing.
If multiple security principals are present, multiple TCP connections can be used to
provide reasonable network scaling. Unfortunately, many server application
workloads use one or a limited set of security principals. Therefore, the scaling for
throughput and I/O request handling is limited in an environment where each session
is mapped to a single TCP connection.
If many TCP connections are present, RSS can be used effectively to distribute the workload across available CPU resources. With a single TCP connection, CPU scaling is unavailable. With single connections, there is limited capability in being resilient to network interface or network path failures. With the LBFO feature in Windows Server Developer Preview, a portion of the failure modes are addressed. However, not all potential failure modes are recoverable with LBFO alone. For example, interior network path failures may not be observable at the endpoints.
To provide for effective scaling and resiliency, the SMB 2.2 protocol in Windows Server Developer Preview adapts by allowing for a many-to-many association between SMB 2.2 session and network connection as depicted in Figure 2.
S e ssio n 1 C onnection 1
S e ssio n 2 C onnection 2
Figure Figure Figure
Figure 2 222:::: SMB SMB SMB SMB 2.2 2.2 2.2 2.2 Session Session Session Session to to to to Connection Connection Connection Connection Association Association Association Association
With this model, not only can multiple sessions use a single connection to communicate with the server but multiple connections can be used for a single session. This association is created dynamically and allows for an increase and decrease in session and connection counts. This also allows for a shift from a less capable network interface (such as WLAN interfaces) to a network interface that may more closely match that of the workload (such as 1 GbE interfaces).
Connection Resiliency
With the ability to associate multiple connections with a single session, the SMB 2.2 client can be resilient to connection failures. These are most likely associated with failures of a network interface or of an intermediate network component (such as a router or switch). Technologies exist that deal with both types of failures, such as LBFO for network interface failures. The user may choose not to deploy these technologies.
As an alternative to these technologies, another approach to connection resiliency would be to provide multiple network paths between SMB 2.2 client and server. If the paths consist of similar capability, as described later, the SMB 2.2 client will actively manage those paths for resiliency. The client does this by creating multiple
connections that span all paths between client and server. If a connection fails or becomes unresponsive, the client will choose the path that remains available.
Network Utilization
Using a single TCP connection, the SMB 2.2 client and server are not assured that they will be unable to fully use the available bandwidth of a 10 GbE network. If multiple connections are used over a network interface that supports RSS, the SMB client and server can easily use all of the bandwidth of a 10 GbE network link.
The ability to scale by using multiple connections is important to both client and
server. For the client, significant networked data movement comes in the form of
SMB read operations. For the server, the incoming data movement of SMB write operations benefits from multiple connections. The load can be effectively distributed across the available CPU cores.
Load Balancing
Enabling the SMB 2.2 client to adapt its load across a set of TCP connections is important to the overall capabilities of the client. For example, shared network paths may experience uneven loading from other clients at the server interface or by routers or switches that experience congestion.
Other conditions that could lead to uneven resource capability are server-side CPU loading or server storage hot-spots. Most of these can be overcome or minimized by allowing the client to schedule requests dynamically across a set of connections.
Transport Flexibility
Supporting the association of one session to multiple TCP connections allows for greater flexibility. For example, if a client and server share connectivity with 1 GbE and 10 GbE network interfaces, the client can start its server interaction over 1 GbE.
In order to handle requested workloads, the client can move the requests to a second or multiple connections associated with the 10 GbE interface. This type of connection adaptation applies as well to shifting requests from an IP over InfiniBand (IPoIB) connection to a standard InfiniBand/RDMA connection. By design, the transport types (such as TCP or RDMA) that are used for connections do not need to be homogenous.
This type of transport flexibility allows for gradual and effective deployment of an SMB 2.2 multiconnection or multi-channel—capable solution. For example, physical network interfaces may be present in a client and server but not initially enabled. The user may choose to delay deployment of network connectivity because of switch utilization or cost, and can add connectivity at a later time. When those network interfaces become active, the SMB 2.2 client and server will dynamically adapt to their presence and use them if appropriate.
Multi-Channel Behaviors
The SMB 2.2 Multi-Channel feature accomplishes its design goals by combining the following behaviors:
• Grouping or aggregating similar kinds of physical interfaces and associate
connections with each physical interface. This provides for resiliency in the event of hardware or path failures.
• Establishing multiple connections (such as TCP) for a single physical interface. This allows for effective utilization of scalable hardware configurations.
The SMB 2.2 client can use these behaviors individually or in combinations. The client decides how to apply these behaviors based on the attributes of the available
network interfaces.
The SMB 2.2 client is responsible for the policy decision of selecting the network interfaces and number of connections. The SMB 2.2 server only needs to identify available network interfaces to the client along with associated attributes. Upon initial contact to the server, the client will use a new SMB 2.2 operation to enumerate the server’s network interfaces and their attributes.
If multiple interfaces exist at the client or server (or both) and the network interfaces represent a valid network path between them, the interfaces are placed in an available list. From the available list, the client will group network interfaces of similar kind (such as 1 GbE, 10 GbE, and RDMA). From these groupings, the client selects which interfaces are to be actively used. It does this by ordering the group type, and then choosing from the highest priority group which contains available interfaces.
The priority ranking of interfaces is ordered as follows:
1. RDMA-capable network interfaces, such as InfiniBand, iWARP, or RoCE.
2. RSS-capable interfaces. The RSS capability allows the client to identify conditions where connection scaling may improve throughput or responsiveness.
3. LBFO or aggregate interfaces that represent the collection of two or more physical interfaces.
4. Standard interfaces and Hyper-V virtual interfaces are next in priority.
Note:
Note:
Note:
Note: The interface types for this and higher priorities are considered capable of multi-channel operations.
5. Wireless interfaces, which are not considered capable of multi-channel operations.
As mentioned earlier, the SMB client will group interfaces into like-kinds. For example, InfiniBand-based interfaces will be placed into the same grouping and will be used together. If the SMB client and server each have access to two InfiniBand interfaces, the client will use both interfaces for SMB 2.2. If the client and server have single 10 GbE interfaces each of which are RSS capable, the client will create multiple TCP connections for the single interface.
The client could have a mismatch for interface speed. For example, the client could have access to four 1 GbE interfaces, while the server has access to a single 10 GbE interface. In this case, the client could create a TCP connection for each of the 1 GbE interfaces. This achieves better performance because the server’s overall network capability exceeds that of the client.
The interface enumeration will occur dynamically. If an interface is added or removed after the client has begun interaction with the server, the change in interface
availability will be dynamically used by the client. For example, if a user starts an SMB
2.2 file copy over a wireless interface and subsequently plugs in or enables a 1 GbE
interface, the SMB 2.2 client will move its requests to the newly available interface
because it has a higher priority in the ranking that was described earlier.
Multi-Channel Performance (Preliminary)
All data that is presented in this paper is based on the SMB 2.2 implementation in pre-release versions of Windows Developer Preview and Windows Server Developer Preview. By definition, these same results will change before the release of Windows Server Developer Preview.
In the following sections, the performance of the SMB 2.2 Multi-Channel feature is presented. To allow for comparison, the same hardware configuration was used for the SMB 2.2 client and server. Details of their configuration are provided in the following sections. If you use similar hardware for your evaluations, a comparison of local storage access and remote access with SMB 2.2 can be made.
Both client and server are installed with Windows Server Developer Preview.
Measurements are taken by using the Iometer tool. The configuration of the Iometer for the data collection is also described in the following sections.
Test Environment
This section discusses the various configurations that were used in the test environment.
Hardware Configuration
This section describes the hardware configuration used to collect the reported data.
As mentioned earlier, the SMB 2.2 client and SMB 2.2 server have the same hardware configuration.
Table Table Table
Table 1. 1. 1. 1. Hardware Hardware Hardware Hardware configuration configuration configuration configuration Hardware
Hardware Hardware
Hardware component component component component Client Client Client Client Server Server Server Server CPU type 2 sockets with 6 cores each at
2.66 GHz (12 cores total)
2 sockets with 6 cores each at 2.66 GHz (12 cores total)
Memory 48 GB 48 GB
Network type 2 network interface adapter cards, Each card has two 10 GbE interfaces.
2 network interface adapter cards. Each card has two 10 GbE interfaces.
Number of network links
4 x 10 GbE 4 x 10 GbE
Storage adapter N/A 2 RAID Host Bus Adapters –
6 Gbps SAS connectivity All network and storage adapters were installed in PCIe 2.0 multi-lane x8 slots. As mentioned earlier, all network connectivity in the test-bed was 10 GbE. Each system had two network interface adapters, and each adapter has two network interfaces for a total of 4 x 10 GbE network interfaces.
The 10 GbE network switch was configured with a single VLAN. Both client and server
had 48 GB of memory. The server had two storage host bus adapters (HBAs). Each
HBA was attached to a single JBOD enclosure for a total of two enclosures. Each JBOD
enclosure had two SAS expanders. Each HBA was dual attached to a single enclosure
with 4 lane, 6 Gbit SAS cables for a total of 8 lanes of 6 Gbit SAS. Each enclosure
housed 12 SSDs.
Two LUNs were presented to the server (one LUN from each HBA/JBOD). The SSDs were configured into a single RAID 0 virtual disk via RAID HBA, with a stripe size of 64 KB. The RAID HBA was configured so that the HBA cache was not used (no read- aheads and writes were write-through). Each LUN had a formatted capacity of 1.7 terabytes.
The following figure shows the configuration of the test hardware.
Se rv e r Clie nt
1 0 G b E
RA ID 0 – 1 2 S S D s RA ID 0 – 1 2 S S D s