High Throughput File Servers with SMB Direct,
Using the 3 Flavors of RDMA network adapters
Jose Barreto
Principal Program Manager
Microsoft Corporation
Abstract
In Windows Server 2012, we introduce the “SMB Direct”
protocol, which allows file servers to use high throughput/low
latency RDMA network interfaces.
However, there are three distinct flavors of RDMA, each with
their own specific requirements and advantages, their own pros
and cons.
In this session, we'll look into iWARP, InfiniBand and RoCE, outline
the differences between them. We'll also list the specific vendors
that offer each technology and provide step-by-step instructions
for anyone planning to deploy them.
The talk will also include an update on RDMA performance and a
customer case study.
Summary
• Overview of SMB Direct (SMB over RDMA)
• Three flavors of RDMA
• Setting up SMB Direct
• SMB Direct Performance
• SMB Direct Case Study
SMB Direct (SMB over RDMA)
•
New class of SMB file storage for the Enterprise
– Minimal CPU utilization for file storage
processing
– Low latency and ability to leverage high speed
NICs
– Fibre Channel-equivalent solution at a lower cost
•
Traditional advantages of SMB file storage
– Easy to provision, manage and migrate
– Leverages converged network
– No application change or administrator
configuration
•
Required hardware
– RDMA-capable network interface (R-NIC)
– Support for iWARP, InfiniBand and RoCE
•
Uses SMB Multichannel for Load
Balancing/Failover
File Client File Server
SMB Server SMB Client User Kernel Application Disk R-NIC Network w/ RDMA support NTFS SCSI Network w/ RDMA support R-NIC
What is RDMA?
• Remote Direct Memory Access Protocol
– Accelerated IO delivery model which works
by allowing application software to bypass
most layers of software and communicate
directly with the hardware
• RDMA benefits
– Low latency
– High throughput
– Zero copy capability
– OS / Stack bypass
• RDMA Hardware Technologies
– Infiniband
– iWARP: RDMA over TCP/IP
– RoCE: RDMA over Converged Ethernet
File
Server
SMB Direct
Client
RDMA
NIC
SMB Direct
Ethernet or
InfiniBand
SMB
Server
SMB Client
Memory
Memory
NDKPI
NDKPI
RDMA
NIC
RDMA
File Server
SMB Direct
1. Application (Hyper-V,
SQL Server) does not
need to change.
2. SMB client makes the
decision to use SMB
Direct at run time
3. NDKPI provides a much
thinner layer than
TCP/IP
4. Remote Direct Memory
Access performed by
the network interfaces.
SMB over TCP and RDMA
Client
Application
NIC
RDMA
NIC
TCP/ IP
User
Kernel
SMB Direct
Ethernet and/or
InfiniBand
TCP/ IP
Unchanged APISMB Server
SMB Client
Memory
Memory
NDKPI
NDKPI
RDMA
NIC
NIC
RDMA 1 2 3 4 1 2 3 4Type (Cards*) Pros Cons
Non-RDMA Ethernet (wide variety of NICs)
• TCP/IP-based protocol
• Works with any Ethernet switch • Wide variety of vendors and models • Support for in-box NIC teaming (LBFO)
• Currently limited to 10Gbps per NIC port • High CPU Utilization under load
• High latency iWARP (Intel NE020*, Chelsio T4)
Lo
w
CP
U
Utiliz
atio
n
un
der
lo
ad
Lo
w
la
tenc
y
• TCP/IP-based protocol• Works with any 10GbE switch • RDMA traffic routable
• Currently limited to 10Gbps per NIC port*
RoCE
(Mellanox ConnectX-2, Mellanox ConnectX-3*)
• Ethernet-based protocol
• Works with high-end 10GbE/40GbE switches • Offers up to 40Gbps per NIC port today*
• RDMA traffic not routable via existing IP infrastructure • Requires DCB switch with Priority Flow Control (PFC)
InfiniBand
(Mellanox ConnectX-2, Mellanox ConnectX-3*)
• Offers up to 54Gbps per NIC port today* • Switches typically less expensive per port than
10GbE switches*
• Switches offer 10GbE or 40GbE uplinks • Commonly used in HPC environments
• Not an Ethernet-based protocol
• RDMA traffic not routable via existing IP infrastructure • Requires InfiniBand switches
• Requires a subnet manager (on the switch or the host)
