Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces

(1)

Reference Architecture:

Microsoft Scale-Out File Server

using Storage Spaces

Microsoft Software Defined

Storage using Lenovo

rack-based servers

Designed for Enterprises,

MSPs/CSPs, and HPC

Affordable, high performance,

scalable, and highly available

Low footprint and high density

to achieve cost savings on

space, power and cooling

David Ye

Last update: 02 July 2015

(2)

ii _{Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces}

1 Introduction ... 1

2 Business problem and business value ... 2

2.1 Business problem ... 2

2.2 Business value ... 2

3 Requirements ... 4

3.1 Functional requirements... 4

3.2 Non-functional requirements ... 5

4 Architectural overview ... 6

5 Component model ... 9

6 Operational model ... 12

6.1 Hardware components ... 12

6.1.1 Lenovo System x3550 M5 ... 12 6.1.2 Lenovo System x3650 M5 ... 12

6.1.3 10 GbE RDMA network adapters... 13

6.1.4 Lenovo RackSwitch G8124E ... 13

6.1.5 Lenovo RackSwitch G8264 ... 14

6.1.6 NetApp DE6600 ... 15

6.2 Storage nodes and disk enclosures ... 16

6.3 Networking ... 20

6.4 Systems management ... 21

6.5 Windows Server 2012 R2 ... 22

6.6 Deployment example ... 23

7 Deployment considerations ... 24

7.1 Systems Management ... 24

7.2 Storage sizing and tuning ... 24

7.2.1 Storage sizing ... 24

(3)

iii _{Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces}

7.2.3 Storage write-back cache ... 25

7.2.4 Storage column and interleave sizing ... 25

7.2.5 MPIO load balancing policy ... 25

7.2.6 Allocation unit size ... 25

7.2.7 Network adapters and switches RDMA configurations ... 25

7.2.8 Lenovo G8264 OS application guide ... 26

7.3 Hardware certifications... 26

7.4 Deployment best practices ... 26

8 Appendix: Lenovo Bill of materials... 27

8.1 Recommended configurations ... 27

8.2 Storage Node BOMs ... 27

8.3 Networking BOMs ... 30

(4)

1 _{Lenovo Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces}

1 Introduction

This document describes the Lenovo Reference Architecture for Microsoft Scale-Out File Server (SOFS) that usesStorage Spaces. Lenovo reference architecture offerings create virtually turnkey solutions that are built around the latest Lenovo servers, networking, and storage, which takes the complexity out of the solution. This Lenovo reference architecture combines Microsoft software, consolidated guidance, and validated

configurations for compute, network, and storage.

Software-defined storage (SDS) is an evolving concept for computer data storage that manages policy-based provisioning and data storage independent of hardware. SDS definitions typically include a form of storage virtualization to separate the storage hardware from the software that manages the storage infrastructure. The software that enables an SDS environment might also provide policy management for feature options, such as deduplication, replication, thin provisioning, snapshots, and backup. The key benefits of SDS over traditional storage are increased flexibility, automated management, and cost efficiency. One approach to SDS that is gaining traction is Microsoft SOFS.

Microsoft SOFS is a high-performance, scalable, and continuously available storage solution for Microsoft Hyper-V and SQL servers that offer an alternative to traditional Storage Area Network (SAN) solutions with the same or better performance.

This Lenovo solution for Microsoft SOFS combines the Storage Spaces and Failover Cluster features of Windows 2012 R2 with Lenovo industry standard x86 servers, Lenovo RackSwitch network switches, and NetApp high density disk enclosures to provide turnkey solutions for enterprises. The architecture that is described here was validated by Lenovo and certified for Microsoft Storage Spaces.

The intended audience of this document is IT professionals, technical architects, sales engineers, and consultants to assist in planning, designing, and best practices for implementing the Lenovo Reference Architecture for Microsoft SOFS that uses Storage Spaces.

(5)

2 Business problem and business value

This section describes the challenges organizations face and how this Reference Architecture for Microsoft Scale-Out File Server (SOFS) with Storage Spaces can help meet those challenges.

2.1 Business problem

The cloud and mobile innovations in the last few years present tremendous amount of growth opportunity for those enterprises that are equipped with proper IT infrastructure. However, companies are discovering that their IT infrastructure is not always up to the task at hand; finding that budgetary constraints or outdated architectures are hindering their ability to compete. Enterprises that use proprietary systems are finding that they are locked into expensive maintenance contracts and obligations that force them to continue buying expensive proprietary technologies.

With digital data growing rapidly annually in the enterprises, companies who deployed traditional proprietary SAN storage are seeing a significant amount of their budget being allocated for storage purchases. This factor is one of the inhibiting factors that limit companies’ growth and competitiveness because of a lack of

investments in other key areas, such as new applications.

2.2 Business value

The Lenovo solution for Microsoft SOFS combines the skills and technologies of the leading enterprise software and hardware vendors to create a non-proprietary storage solution that lowers overall storage costs, increases storage reliability, and frees you from expensive maintenance and service contracts. Gaining access to near-zero downtime with exceptional fault tolerance, dynamic pooling, enhanced virtualization resources, end-to-end architectural and deployment guidance, predefined, out-of-box solutions and much more gives you the tools to compete today and into the future.

(6)

Table 1: Comparison of SAN and Microsoft SOFS capabilities

FC/iSCSI SAN Microsoft SOFS with Storage Spaces

RAID resiliency Two/three way mirroring, single/dual parity Disk pooling Storage Spaces disk pooling

Data deduplication Data deduplication

High availability (controller failover) Continuous availability (transparent failover) Storage tiering Storage tiering

Snapshots Snapshots

Persistent write back cache Persistent write back cache Copy offload SMB copy offload

FC, FCoE, iSCSI 10 GbE, 40 GbE, 56 Gb IB SMB Multichannel, SMB Direct (RDMA) The basic building block of this Lenovo solution can scale 2 – 4 storage nodes and up to 240 disks. In some cases, it can scale up to 360 disks. This 2 – 4 storage nodes building block model can scale horizontally and linearly as much as your compute infrastructure requires. It also provides high performance and continuous availability. All of these features can be achieved by using standard Lenovo 2-socket x86 server hardware and ultimately lower total cost of ownership.

(7)

3 Requirements

The functional and non-functional requirements for this reference architecture are described in this section.

3.1 Functional requirements

The following functional requirements are featured:

 Integrate with Windows 2012 R2 Storage features:

o Create storage pools with just a bunch of disks (JBOD)

o Create RAID equivalent virtual disks or logical unit numbers (LUNs):  Enable virtual disk for storage tiering

 Allocate Write-back cache for virtual disks from SSD tier o_{Create Cluster Shared Volumes}

o Support Multipathing

o Support SMB 3.0 and 3.02 for storage access protocol:  Create Continuous Availability (CA) Files Shares

 Create multiple SMB connections for each network adapter on-demand  Detect and utilize RDMA-capable network adapters

 Encrypt storage traffic between hosts and storage nodes  Provide transparent failover capability

o_{Support File Management:}

 Enable/Disable deduplication on per volume basis  Configure as DFS Namespace folder target server  Enable/Disable Folder Redirection

 Enable/Disable Offline Files (client side caching)  Support Roaming User Profiles

 Support Home Directories

o_{Support IO intensive application workloads:}  Microsoft Hyper-V Servers

 Microsoft SQL Servers

 Integrate with Windows Failover Cluster for high availability (HA): o Create Windows Failover Cluster:

(8)

 Create Cluster Communication Network  Create Client/Server Network for Storage  Create Client Access Point for CA Shares o Single Management User Interface:

 Manage all Microsoft SOFS storage functionalities  Provide wizard driven tools for storage related tasks

o Enterprise Management: Support integration with Microsoft System Center

3.2 Non-functional requirements

Table 2 lists the non-functional requirements for a server-based storage solution.

Table 2: Non-functional requirements

Requirement Description

Scalability Scale out linearly with building block approach

Load balancing SMB storage traffic is distributed across storage nodes Fault tolerance Storage nodes and JBOD enclosures are redundant

Physical footprint Low profile servers and high density JBOD enclosures reduce rack space Ease of installation Standard server hardware and software with cluster setup wizard

Ease of management/operations Managed through Windows Failover Cluster Manager

Flexibility Different choices for various hardware components and SSD and HDD drives mixed

Security Encryption of storage traffic

High performance High speed low latency Remote Direct Memory Access over 10GbE Certification Hardware should be certified for Microsoft Windows 2012 R2 and Storage

(9)

4 Architectural overview

Microsoft SOFS is considered an SDS solution. It is not just using standard x86 servers and disk enclosures to build a storage solution to lower cost; it also provides continuous availability storage for applications. Two primary use cases for this solution are Microsoft SQL Server and Hyper-V Server.

Figure 1 shows the Microsoft SQL Server and Hyper-V clusters that are connected to Microsoft SOFS as storage backend through Server Message Blocks (SMB 3.02) protocol over high speed Ethernet networks.

(10)

Figure 2 shows more details of the software and hardware layers.

Figure 2: Microsoft Hyper-V and SQL Servers with Microsoft SOFS Storage Logical View

The following software and hardware layers are featured: Server Message

Block (SMB) protocol

Server Message Block protocol is the Windows file sharing protocol in which Microsoft Hyper-V or SQL Servers use to communicate with the back-end Microsoft SOFS storage.

SOFS Microsoft SOFS is a Windows operating system feature that provides scale-out file shares that are continuously available for file-based server application storage. Microsoft SOFS also provides the ability to share the same folder from multiple nodes of the same cluster.

Cluster Shared Volumes (CSV)

A CSV is a shared disk that contains a volume that is made accessible for read and write operations by all nodes within a Microsoft Windows SOFS cluster. Microsoft SQL and Hyper-V data are stored in CSV.

Storage Space Storage Space is a Windows feature with which you can create a virtual disk from a storage pool and present it a volume or LUN to Microsoft SOFS cluster nodes. Virtual disk can be created with resiliency options, such as 2-way mirror, 3-way mirror, single parity, dual parity, and no parity.

(11)

Storage Pool A storage pool is a grouping of industry-standard disks, such as solid-state disks (SSDs) or hard disk drives (HDDs) to provide the aggregate performance of all disks in the pool.

Storage Node A storage node is an industry-standard x86 server that is running Windows 2012 R2 to provide storage controller functionalities.

Shared JBOD A shared JBOD is a physical disk enclosure that houses the physical disks, such as SSDs and HDDs.

Microsoft SOFS hosts SQL Server’s databases files and Hyper-V’s VHDX files. When one of the Microsoft SOFS storage nodes fails, the SMB client that is running on the SQL or Hyper-V server automatically selects the next best available SOFS storage node to connect to and resumes IO operations on behalf of the SQL or Hyper-V. From SQL and Hyper-V servers’ perspective, the IO appears stalled for a short amount of time.

(12)

5 Component model

Windows Server 2012 R2 provides a rich set of storage features with which you can use lower-cost,

industry-standard hardware without compromising performance or availability. Figure 3 shows an overview of the Microsoft software storage and virtualized components in the Microsoft SOFS storage solution.

Figure 3: Windows Server 2012 R2 Storage Stack Diagram

The Microsoft SOFS storage solution consists of the following components:

Microsoft Directory Service

 Microsoft Active Directory (AD), Domain Name Servers (DNS), and Dynamic Host Configuration Protocol (DHCP) servers provide user authentications, domain name service, and dynamic IP service infrastructure for Hyper-V and SOFS storage nodes.

(13)

Microsoft Virtualization Stack

 Hyper-V Storage NUMA I/O

Windows Server 2012 R2 supports large virtual machines, up to 64 virtual processors. Any large virtual machine configuration typically also needs scalability in terms of I/O throughput. Windows Server 2012 R2 provides a Hyper-V storage NUMA I/O capability to create several communication channels between the guest devices and host storage stack with a specified dedicated set of VPs for the storage IO processing. Hyper-V storage NUMA I/O offers a more efficient I/O completion

mechanism that interrupts distribution amongst the virtual processors to avoid expensive inter-processor interruptions. With those improvements, the Hyper-V storage stack can provide scalability improvements in terms of I/O throughput to support the needs of large virtual machine configuration with data intensive workloads like Exchange, SQL, and SharePoint.

 Storage Quality of Service (QoS) and Hyper-V IO Balancer

Storage QoS is a new feature that is provided in Windows Server 2012 R2. It offers the capability to set certain QoS parameters for storage on the virtual machines. Storage QoS and Hyper-V IO balancer provide the ability to specify maximum I/O operations per second (IOPS) values for an individual virtual hard disk. Windows Server 2012 R2 Hyper-V with Storage QoS can throttle storage that is assigned to VHD/VHDX in the same volume to prevent a single virtual machine from using all I/O bandwidth and help to control the balance of VM storage demand and storage performance capacity.

 VHDX

VHDX is a new virtual hard disk format with which you can create resilient high-performance virtual disks up to 64 TBs. Microsoft recommends the use of VHDX as the default virtual hard disk format for VMs. VHDX provides more protection against data corruption during power failures by logging updates to the VHDX metadata structures and the ability to store custom metadata. The VHDX format also provides support for the TRIM command, which results in smaller file size and allows the underlying physical storage device to reclaim unused space. The support for 4KB logical sector virtual disk and the larger block sizes for dynamic and differential disks allows for increased performance.

Microsoft SMB 3.0

 SMB Multichannel

SMB multichannel provides the capability to automatically detect multiple networks for SMB connections. It offers resilience against path failures and transparent failover with recovery without application service disruption with much improved throughput by aggregating network bandwidth from multiple network interfaces. Server applications then can use all available network bandwidth, which makes them more resistant to network failure.

 SMB Direct

SMB Direct (SMB over RDMA) makes available Remote Direct Memory Access (RDMA) hardware support for SMB to provide high-performance storage capabilities. SMB Direct is intended to lower CPU usage and latency on the client and server while delivering high IOPS and bandwidth utilization. It can deliver enterprise-class performance without relying on expensive Fibre Channel SAN. With the

(14)

CPU offloading and the ability to read and write directly against the memory of the remote storage node, RDMA network adapters can achieve extremely high performance with low latency. SMB Direct is also compatible with SMB Multichannel to achieve load balancing and automatic failover.

SOFS Cluster

 Failover Clustering

A failover cluster is a group of independent computers that work together to increase the availability and scalability of clustered roles. If one or more of the cluster nodes fail, the service automatically fails over to other node without disruption of service.

 SOFS role

The SOFS role provides a continuously available SMB service and a mechanism for clustered file servers in an active-active configuration to aggregate bandwidth across the cluster nodes.

 Cluster Shared Volumes (CSV)

CSVs in a Windows Server 2012 R2 failover cluster allow multiple nodes in the cluster to

simultaneously access shared storage with a consistent and distributed namespace. To each of the cluster nodes in the cluster, the CSV provides a consistent single file namespace. Therefore, CSVs simplify the management of many LUNs in a failover cluster.

 Continuous Availability (CA) File Share

A feature in Windows Failover Clustering, CA tracks file operations on highly available file shares so that storage clients (such as Hyper-V or SQL servers) can fail over to another node of the cluster without interruption.

 Failover Cluster Manager

Failover Cluster Manager is a user interface with which you can manage all aspects of Failover Cluster management. By using this feature, you can run tasks, such as create Failover Cluster, Storage Pools, CSVs, and CA File Share server.

Microsoft Storage Stack

 Storage Spaces

Storage Spaces is a storage virtualization feature in Windows Server 2012 R2 provides many features including write-back cache, storage tiering, thin provisioning, mirroring, and parity with just a bunch of disks (JBOD). It gives you the ability to consolidate all of your Serial Attached SCSI (SAS) and Serial ATA (SATA) connected disks into Storage Pools. Storage Spaces are compatible with other Windows Server 2012 R2 storage features, including SMB Direct and Failover Clustering.

 Microsoft Multipath I/O (MPIO)

MPIO is a Microsoft-provided framework that allows storage providers to develop multipath solutions to optimize connectivity and create redundant hardware paths to storage systems. In this solution, we use Microsoft Device Specific Module (DSM) working with Microsoft Storage Spaces to provide fault tolerant connectivity to the disk enclosures or JBOD.

(15)

6 Operational model

It is useful to view this solution in the following layers:

 Physical Layer: This layer includes the physical hardware, such as rack, servers, switches, and disk enclosures units (JBODs).

 Software Layer: This layer includes the Windows operating system and its built-in software features that enable the physical hardware to provide storage functionality.

6.1 Hardware components

The following section describes the components in a Microsoft SOFS deployment.

6.1.1 Lenovo System x3550 M5

At the core of this reference architecture, the Lenovo System x3550 M5 server delivers the performance and reliability that is required for mission-critical storage solutions, such as SOFS. Lenovo System x3550 M5 servers can be equipped with up to two 18-core E5-2600 v3 series processors and up to 1.5 TB of TruDDR4 memory. Up to three PCIe 3.0 expansion slots, four integrated 1 GbE network ports, and an optional embedded dual-port 10/40 GbE network adapter provides ports for your data and storage connections.

The Lenovo System x3550 M5 includes an on-board RAID controller and the choice of spinning hot swap SAS or SATA disks and small form factor (SFF) hot swap SSDs. The x3550 M5 supports a maximum of 24 TB of internal storage.

The x3550 M5 supports the following components:

 Up to 10 front and two rear SFF HDDs or SSDs

 Up to four 3.5-inch HDDs

The x3550 M5 also supports remote management via the Lenovo Integrated Management Module (IMM), which enables continuous management capabilities. All of these key features (including many that are not listed here) help solidify the dependability Lenovo customers are accustomed to with System x servers. Figure 4 shows the Lenovo x3550 M5 server.

Figure 4: Lenovo System x3550 M5

For more information, see this website: lenovopress.com/tips1194.html

6.1.2 Lenovo System x3650 M5

The Lenovo System x3650 M5 server (as shown in Figure 5) is an enterprise class 2U two-socket versatile server that incorporates outstanding reliability, availability, and serviceability (RAS), security, and high efficiency for business-critical applications and cloud deployments. It offers a flexible, scalable design and simple upgrade path to 14 3.5-inch HDDs, with doubled data transfer rate via 12 Gbps SAS internal storage connectivity and up to 1.5TB of TruDDR4 Memory. Its onboard Ethernet solution provides four standard

(16)

embedded Gigabit Ethernet ports and two optional embedded 10 Gigabit Ethernet ports without occupying PCIe slots.

Figure 5: Lenovo System x3650 M5

Combined with the Intel® Xeon® processor E5-2600 v3 product family, the Lenovo x3650 M5 server offers a high density of workloads and performance that is targeted to lower the total cost of ownership (TCO) per virtual machine. Its flexible, pay-as-you-grow design and great expansion capabilities solidify dependability for any kind of virtualized workload with minimal downtime.

The Lenovo x3650 M5 server provides internal storage density of up to 84 TB (with up to 14 x 3.5-inch drives) in a 2U form factor with its impressive array of workload-optimized storage configurations. The x3650 M5 offers easy management and saves floor space and power consumption for the most demanding storage

virtualization use cases by consolidating the storage and server into one system. For more information, see the following websites:

 System x3650 M5 – Overview

 System x3650 M5 Product Guide

6.1.3 10 GbE RDMA network adapters

This Lenovo solution for Microsoft SOFS offers the following choices of 10 GbE RDMA network adapters:

 Emulex VFA5

 Melllanox CX3

These adapters provide high throughput and low latency capabilities for a SOFS solution. For more information, see the following websites:

 lenovopress.com/tips1142.html  lenovopress.com/tips0897.html

6.1.4 Lenovo RackSwitch G8124E

The Lenovo RackSwitch™ G8124E is a 10 Gb Ethernet switch that is specifically designed for the data center and provides a virtualized, cooler, and easier network solution. The G8124E offers 24 10 GbE ports in a high-density, 1U footprint.

Designed with ultra-low latency and top performance in mind, the RackSwitch G8124E provides line-rate, high-bandwidth switching, filtering, and traffic queuing without delaying data. Large data center grade buffers keep traffic moving. The G8124E also supports Converged Enhanced Ethernet (CEE) and Data Center Bridging for support of FCoE and can be used for NAS or iSCSI.

(17)

The G8124E is virtualized and supports VMready® technology, which is an innovative, standards-based solution to manage virtual machines in small to large-scale data center and cloud environments. VMready works with all leading VM providers. The G8124E also supports Virtual Fabric, which allows for the carving up of a physical NIC into 2 - 8 vNICs for improved performance, availability, and security while reducing cost and complexity.

The G8124E is easier to use and manage, with server-oriented provisioning via point-and-click management interfaces. Its industry-standard command-line interface (CLI) and easy interoperability simplifies configuration for those users who are familiar with Cisco environments.

Figure 6 shows the Lenovo RackSwitch G8124E.

Figure 6: Lenovo RackSwitch G8124E

The RackSwitch G8124E includes the following benefits:

 A total of 24 SFP+ ports that operate at 10 Gb or 1 Gb Ethernet speeds

 Optimal for high-performance computing and applications that require high bandwidth and low latency

 All ports are nonblocking, 10 Gb Ethernet with deterministic latency of 570 nanoseconds

 VMready® helps reduce configuration complexity and improves security levels in virtualized environments

 Variable-speed fans automatically adjust as needed, which helps to reduce energy consumption

 Easy, standards-based integration into Cisco and other networks helps reduce downtime and learning curve

For more information, see this website: lenovopress.com/tips0787.html

6.1.5 Lenovo RackSwitch G8264

Designed with top performance in mind, Lenovo RackSwitch G8264 (as shown in Figure 7) is ideal for today’s big data, cloud, and optimized workloads.

Figure 7: Lenovo RackSwitch G8264

The G8264 switch offers up to 64 10 Gb SFP+ ports in a 1U form factor and can be used in the future with four 40 Gb QSFP+ ports. It is an enterprise-class and full-featured data center switch that delivers line-rate,

high-bandwidth switching, filtering, and traffic queuing without delaying data. Large data center grade buffers keep traffic moving. Redundant power and fans and numerous HA features equip the switches for

(18)

The G8264 switch is ideal for latency-sensitive applications, such as client virtualization. It supports Virtual Fabric to help clients reduce the number of I/O adapters to a single dual-port 10 Gb adapter, which helps reduce cost and complexity. The G8264 switch supports the newest protocols, including Data Center Bridging/Converged Enhanced Ethernet (DCB/CEE) for support of FCoE, iSCSI, and NAS.

For more information, see the Lenovo RackSwitch G8264ttttttttttttt Product Guide.

6.1.6 NetApp DE6600

NetApp DE6600 JBOD is a low-profile, high-density external disk enclosure, as shown in Figure 8.

Figure 8: NetApp DE6600 Disk Enclosure

This enclosure can provide cost savings for your data center with space saving, power and cooling, and its low footprint can house up to 60 HDDs or a mix of SDDs and HDDs, in a 4U form factor footprint. It features dual SAS modules that can provide redundancy for fault tolerant and dual paths to support storage multipathing capability. It can also scale vertically by cascading multiple enclosures together to provide higher storage performance and with capacity over 1 PB.

NetApp DE6600 is certified for Microsoft Windows 2012 R2 Storage Spaces. It is easy to deploy and manage with Windows 2012 R2 and is an ideal platform for the following applications:

 Managed Service Providers (MSP) and Cloud Service Providers (CSP) with low cost and high density requirements

 High Performance Computing (HPC) deployments that require high density storage to lower cost on space, power and cooling

 High throughput applications, such as oil and gas, manufacturing, video surveillance, and call center

 Enterprises with low-cost, high-density, and performance storage requirements For more information, see this website:

(19)

6.2 Storage nodes and disk enclosures

This solution can use the Lenovo System x3550 M5 or Lenovo System x3650 M5 server as storage nodes. Each server features the following components:

 Two E5-2670 v3 Intel Xeon processors

 128 GB of memory

 Two 2.5-inch HDDs in a RAID1 pair

 Four on-board 1 Gbps Ethernet ports

 One dual-port 10 Gbps Ethernet adapter with RDMA capability

 Two dual-port SAS HBAs

For more information, see “Storage Node BOMs” on page 27

In addition to the storage nodes, physical disk enclosures and disks are based on JBOD and SSD plus

7.2k rpm Near-Line SAS (NL-SAS) disks, respectively. Each storage node has dual connections or paths to the JBOD enclosures for fault tolerant and performance. SSD provides high performance storage tier for daily work set for various application workloads. NL-SAS disks provide high storage capacity for data that are not used daily.

The basic building block for the Microsoft SOFS storage solution is four storage nodes and four JBODs (4-Node/4-JBOD). However, business customers can start with a minimum of 2-Node/1-JBOD and add storage nodes or JBODs as needed. Customers can scale the Microsoft SOFS storage solution linearly with 4-Node/4-JBOD increment.

Depending on the capacity and performance requirements, the storage nodes are connected to up to 4 NetApp DE6600 JBOD disk enclosures and can have up to 240 SSDs or HDDs in any combination. All disk enclosures are directly connected with SAS cables and shared among all storage nodes. All connections from each SOFS storage node to the shared JBODs are redundant for HA. There are no SAS switches involved in this solution to reduce complexity and costs. The storage nodes are clustered to provide continuous availability back-end storage for Microsoft Hyper-V or SQL servers.

Microsoft Storage Spaces is used to pool all the disk resources in the enclosures. Cluster virtual disks are created from the pool to present them to SOFS storage nodes as logical unit numbers (LUNs). Storage Spaces provides RAID resiliency options, such as 2-way mirror or 3-way mirror during the creation of virtual disks. Single or dual parity is also supported and is suitable for backup, archiving, or sequential workloads. SSDs can be used for storage tiering or as a write-back cache for virtual disks. Virtual disks are formatted by using the Microsoft Clustered Shared Volume File System (CSVFS). CSV volumes are accessed by all storage nodes in the SOFS cluster. Server Message Block (SMB) 3.02 file shares are created by using CSV volumes on the SOFS cluster.

Each Hyper-V host and storage node is equipped with a dual-port 10 Gbps Ethernet adapter. The Ethernet adapter is RDMA-capable, which is used by the SMB-Direct feature of the SMB 3.02 protocol to provide high speed and low latency network access with low CPU utilization. These RDMA adapters are dedicated for SMB file share storage traffic.

(20)

With the file shares in place, Hyper-V or SQL server hosts can use thefiles shares as back-end storage to store VHDX VM or SQL server databases files. If there is a storage node failure while a Hyper-V or SQL host is accessing the storage, the host’s SMB and witness client is notified immediately by the SOFS cluster witness service and then Hyper-V or SQL server automatically switches to the next best available storage node, re-establishes the session, and continues IO operations from the point of failure. This process is transparent to Hyper-V or SQL server and it appears that the IO operation was stalled for a short time only. This transparent failover capability or continuous availability is what makes SOFS a highly available storage solution. Table 3 lists the three most common deployment configurations.

Table 3: Microsoft SOFS Sizing

Configurations Number of storage nodes Number of JBODs Maximum disk counts

2-Node/1-JBOD 2 1 60

2-Node/2-JBOD 2 2 120

4-Node/4-JBOD 4 4 240

The SAS cabling diagrams for the three configurations are shown in Figure 9, Figure 10, and Figure 11.

(21)

(22)

(23)

6.3 Networking

Depending on the network scalability requirements, this solution uses two Lenovo RackSwitch G8124E or G8264 rack switches for HA and performance capabilities and is ideal for I/O-intensive storage software applications, such as Microsoft SOFS. These Lenovo top-of-the-rack switches support the latest CEE standard, which is required for network adapters that have RDMA capability. Network adapters that have RDMA can function at full speed with low latency while using little CPU. For workloads (such as Hyper-V or Microsoft SQL Server), this feature enables SOFS storage to resemble local direct-attached block storage.

As shown in Figure 12 and Figure 13, each storage node has one independent 10 Gbps Ethernet connection to each Lenovo rack switches.

Figure 12: Logical SOFS Networking view

There are two 10 Gbps connections total for each storage node. There is no network adapter teaming requirement for the storage node. Microsoft SOFS load balances I/O traffic across both connections, which means there is no requirement for inter-switch links (ISL) between two switches in this solution.

This solution focused on the hardware components for Microsoft SOFS. It is assumed that there is a network infrastructure that supports client connections, Active Directory, DNS, and DHCP servers that is in place at the customer premise.

For best practices, the switches that are described in this solution can be dedicated for storage traffic between Hyper-V or SQL servers and storage nodes. This configuration ensures the best throughput and low I/O latency for storage. For more information, see “Networking BOMs” on page 30.

(24)

Figure 13: Physical Layer Rack and Networking view

6.4 Systems management

Lenovo XClarity™ Administrator is a centralized resource management solution that reduces complexity, speeds up response, and enhances the availability of Lenovo® server systems and solutions.

The Lenovo XClarity Administrator provides agent-free hardware management for Lenovo’s System x® rack servers and Flex System™ compute nodes and components, including the Chassis Management Module (CMM) and Flex System I/O modules. Figure 14 shows the Lenovo XClarity administrator interface, where Flex System components and rack servers are managed and are seen on the dashboard. Lenovo XClarity

(25)

Figure 14: XClarity Administrator interface

6.5 Windows Server 2012 R2

The Microsoft SOFS solution requires the Windows Failover Cluster feature from Windows Server 2012 R2 Standard or DataCenter edition.

(26)

6.6 Deployment example

Although the scope of this paper focuses on SOFS storage, it is beneficial to see how it is deployed in the enterprise production environment. Figure 15 shows an example of the complete solution in a production environment. The customer has an initial capacity requirement of 120 hard drives with mixed of SSDs and HDDs. The configuration for this deployment is two storage nodes and two JBODs.

Figure 15: End-to-end SOFS solution

In this example, Microsoft SOFS storage is used in a Hyper-V private cloud or Microsoft SQL server environment. As shown in Figure 15, it is assumed that the enterprise customer has an infrastructure that includes Windows Active Directory domain, client computers, network switches and router, Hyper-V, and SQL server hosts. When SOFS storage is deployed, it is providing storage resources for Hyper-V and SQL server hosts that are connected via the dedicated dual 10 GbE RDMA network interfaces (R-NIC) and 10 GbE switches (switch 5 and 6). The two other network adapter ports in each SOFS storage node are connected to the switch 3 and 4 for Active Directory authentication, cluster management, and cluster private network communication.

(27)

7 Deployment considerations

This section describes noteworthy deployment considerations.

7.1 Systems Management

Systems management for this solution uses the built-in Windows Failover Cluster Manager Tool or Windows Powershell cmdlets. Optionally, this solution can be managed by Microsoft System Center if it is deployed in the environment. Lenovo System x servers provide System Center Management Packs so that you can manage the entire solution from a single console.

7.2 Storage sizing and tuning

SOFS storage sizing and tuning is an important element of providing optimal storage performance for Microsoft Hyper-V and SQL servers. The following outlines some of the key aspects of storage sizing and tuning for SOFS that uses Storage Spaces.

7.2.1 Storage sizing

There are two considerations when it comes to storage sizing: IO requirements and usable capacity.

The ideal use case for Microsoft SOFS storage solution is that the daily application working set is stored in the SSD storage tier for performance and the infrequently used data is stored in the slower NL-SAS storage tier for capacity.

The best approach is to monitor the production environment for some time and analyze how much data is used daily. Each environment is unique; however, as general best practice and recommendation, the

SSD-to-NL-SAS disk capacity ratio is usually 5 - 10%. For example, if the NL-SAS storage tier is 100TB, the SSD tier should be 5 - 10 TB.

7.2.2 Storage resiliency

Similar to RAID capabilities in RAID adapters or SAN storage, Storage Spaces in SOFS provides storage resiliency. Table 4 lists the supported resiliency types and general deployment recommendations or use cases.

Table 4: SOFS Resiliency Type and Target Workloads

Resiliency Type Number of Data Copies Maintained Deployment Recommendations Mirror 2 (two-way mirror)

3 (three-way mirror)

All Workloads

Parity 2 (single parity) 3 (dual parity)

Sequential workloads such as archival or backup (not

recommended for random workloads)

Simple 1 Workloads that do not need resiliency or provide alternative resiliency mechanism, such as data replication that is built into the application.

(28)

7.2.3 Storage write-back cache

Storage Spaces can use SSDs for storage tiering and a write-back cache (WBC). WBC can improve

performance by handling bursty small random write I/O operations to virtual disks. By default, Storage Spaces allocates 1 GB from SSD tier and you can increase this size to improve write performance. For more

information, see the following websites:

 technet.microsoft.com/en-us/library/dn387076.aspx#bkmk_wbc  technet.microsoft.com/en-us/library/hh848643(v=wps.630).aspx

7.2.4 Storage column and interleave sizing

Similar to the RAID Group concept, the number of columns in Storage Spaces specifies how many physical disks data is striped across at once. More columns provide better performance because of parallelism. Interleave is stripe size or the amount of data written at once to a column. You can adjust the interleave parameter to fit the average I/O size of the application to avoid split I/O and improve performance. For more information, see the following websites:

 social.technet.microsoft.com/wiki/contents/articles/11382.storage-spaces-frequently-asked-quest ions-faq.aspx?PageIndex=3#What_are_columns_and_how_does_Storage_Spaces_decide_how _many_to_use

 social.technet.microsoft.com/wiki/contents/articles/15200.storage-spaces-designing-for-performa nce.aspx

7.2.5 MPIO load balancing policy

Adjusting the MPIO load balancing policy other than the default can improve performance. For more information, see this website: technet.microsoft.com/en-us/library/jj649803(v=wps.630).aspx

7.2.6 Allocation unit size

As a best practice, format the CSV volumes with 64 K allocation unit size. This size provides better performance for Hyper-V, SQL, and Exchange Servers.

7.2.7 Network adapters and switches RDMA configurations

RDMA tuning and configurations are essential to the performance of the SOFS solution. For more information about RDMA setup, see these websites:

 implementerslab.com/artifacts/2C2C78A8-24D3-4440-B2F3-970EEB1D73E9/elx_sis_all_smbdir ect_oce14000_windows2012_appnote_microsoftNew.pdf

 blogs.technet.com/b/josebda/archive/2012/07/31/deploying-windows-server-2012-with-smb-direc t-smb-over-rdma-and-the-mellanox-connectx-3-using-10gbe-40gbe-roce-step-by-step.aspx

(29)

7.2.8 Lenovo G8264 OS application guide

For configurations of G8264 RackSwitch to support RDMA capability, read Part 4 (section on CEE) of the G8264 Application Guide. The following most common features are used for supporting network adapters that have RDMA capability:

 Priority-Based Flow Control (PFC) (IEEE 802.1Qbb) extends 802.3x standard flow control to allow the switch to pause traffic that is based on the 802.1p priority value in each packet’s VLAN tag.

 Enhanced Transmission Selection (ETS) (IEEE 802.1Qaz) provides a method for allocating link bandwidth that is based on the 802.1p priority value in each packet’s VLAN tag.

 Data Center Bridging Capability Exchange Protocol (DCBX) (IEEE 802.1AB) allows neighboring network devices to exchange information about their capabilities.

For more information, see this website: ibm.com/support/docview.wss?uid=isg3T7000679

7.3 Hardware certifications

This entire solution is certified for Windows and Storage Spaces and the Bill of Materials in section 8 reflects the Lenovo certified hardware. When newer hardware is released from Lenovo, you can replace the hardware in this solution with the newer hardware if the hardware is Windows and Storage Spaces certified. For more information about Microsoft certification, see this website: windowsservercatalog.com/

7.4 Deployment best practices

To be successful in deploying Microsoft SOFS, adhere to the following general deployment best practices:

 Verify purchased hardware

Verify the disks and enclosures to make sure that the disk performance of the same disk model is consistent. This verification can be useful in isolating issues, such as hardware components that might have different firmware levels because of different manufacturing and purchase dates. For more information, see the following websites:

o gallery.technet.microsoft.com/scriptcenter/Storage-Spaces-Physical-7ca9f304 o technet.microsoft.com/en-us/library/jj134244.aspx

 Deploy and configure Microsoft SOFS.

o How-To Deploy Clustered Storage Spaces:

http://technet.microsoft.com/en-us/library/jj822937.aspx

o Step-By-Step for Tiered Storage Spaces:

http://blogs.technet.com/b/josebda/archive/2013/08/28/step-by-step-for-storage-spaces -ti ring-in-windows-server-2012-r2.aspx

 Validate Microsoft SOFS performance: Use an I/O performance tool to verify the SOFS performance.

 Test workloads against Microsoft SOFS: Run your application workloads against SOFS to ensure SOFS meets your I/O requirements.

(30)

8 Appendix: Lenovo Bill of materials

This appendix features the Bill of Materials (BOMs) for different configurations of hardware for Microsoft SOFS deployments. There are sections for storage servers and networking.

The BOM lists in this appendix are not meant to be exhaustive and must always be confirmed with the configuration tools. Any description of pricing, support, and maintenance options is outside the scope of this document.

8.1 Recommended configurations

Table 5 lists the three recommended configurations with the number and type of SAS cables that are needed for each configuration.

Table 5: Recommended configurations of servers and JBOD enclosures

Configurations x3550 M5 or x3650 M5 Server Nodes JBOD Enclosures 6 Gb SAS cables P/N 39r6529 12 Gb SAS cables (mSAS-HD to mSAS) P/N 00MJ166 2-Node/1-JBOD 2 1 0 4 2-Node/2-JBOD 2 2 2 4 4-Node/4-JBOD 4 4 4 16

8.2 Storage Node BOMs

The Bill of Materials for the storage nodes are listed in Table 6, Table 7, and Table 8.

Table 6: Netapp JBOD

Code Description Quantity

00D1653 NetApp DE6600 Disk Enclosure 1 46W0472 3.5" drive carrier for DE6600 48 49Y6210 4TB 7.2K 6Gbps NL SAS 3.5" G2HS HDD 48 00D0826 2.5" driver carrier for DE6600 (SSD) 12 00W1311 400GB SAS 3.5" MLC HS Enterprise SSD 12 46W0474 DE6600 Rear Rack Mount Bracket (L&R) 1

(31)

Table 7: System x3550 M5

5463AC1 Lenovo System x3550 M5 1 A59F System Documentation and Software-US English 1 A5AY System x 750W High Efficiency Platinum AC Power Supply 1 A5B7 16GB TruDDR4 Memory (2Rx4, 1.2V) PC4-17000 CL15 2133MHz LP RDIMM 8 A5AY System x 750W High Efficiency Platinum AC Power Supply 1 A59V System x3550 M5 Planar 1 5978 Select Storage devices - configured RAID 1 A2K7 Primary Array - RAID 1 (2 drives required) 1 ASBF 600GB 10K 12Gbps SAS 2.5" G3HS 512e HDD 2 A45W ServeRAID M1215 SAS/SATA Controller 1 A5BK Intel Xeon Processor E5-2670 v3 12C 2.3GHz 30MB Cache 2133MHz 120W 1 A5BZ Addl Intel Xeon Processor E5-2670 v3 12C 2.3GHz 30MB 2133MHz 120W 1 A1ML Integrated Management Module Advanced Upgrade 1 A5AL System x Enterprise 1U Cable Management Arm (CMA) 1 A59W System x3550 M5 4x 2.5" HS HDD Kit 1 A5AK System x3550 M5 Slide Kit G4 1 A5AB System x Advanced LCD Light Path Kit 1 6311 2.8m, 10A/100-250V, C13 to IEC 320-C14 Rack Power Cable 2 A5AF System x3550 M5 PCIe Riser 2, 1-2 CPU (LP x16 CPU1 + LP x16 CPU0) 1 A5M0 N2225 SAS/SATA HBA 2 A5AH System x3550 M5 PCIe Riser 1 (1x ML2 x16 CPU0) 1 A40Q Emulex VFA5 ML2 Dual Port 10GbE SFP+ Adapter * 1 A413 2U Bracket for Emulex VFA5 ML2 Dual Port 10GbE SFP+ Adapter 1 A5Q6 2U bracket for N2225 SAS/SATA HBA 1 A1PJ 3m Passive DAC SFP+ Cable 2

(32)

Table 8: System x3650 M5

5462AC1 Lenovo System x3650 M5 1 A5FR System x3650 M5 PCIe Riser 1 (1 x16 FH/FL + 1 x8 ML2 Slots) 1 A5FY System x3650 M5 2.5" ODD/LCD Light Path Bay 1 A5FX System x Enterprise 2U Cable Management Arm (CMA) 1 A5R5 System x3650 M5 PCIe Riser 2 (1 x16 FH/FL + 1 x8 FH/HL Slots) 1 A5FD System x3650 M5 2.5" Base without Power Supply 1 A5EU System x 750W High Efficiency Platinum AC Power Supply 2 A5EN Addl Intel Xeon Processor E5-2670 v3 12C 2.3GHz 30MB 2133MHz 120W 1 A5B7 16GB TruDDR4 Memory (2Rx4, 1.2V) PC4-17000 CL15 2133MHz LP RDIMM 8 A5EY System Documentation and Software-US English 1 A1ML Integrated Management Module Advanced Upgrade 1 A5GV Intel Xeon Processor E5-2670 v3 12C 2.3GHz 30MB Cache 2133MHz 120W 1 A40Q Emulex VFA5 ML2 Dual Port 10GbE SFP+ Adapter 1 A5G3 System x3650 M5 2.5" ODD Bezel with LCD Light Path 1 A5M0 N2225 SAS/SATA HBA 2 A45W ServeRAID M1215 SAS/SATA Controller 1 A5G6 x3650 M5 8x 2.5" HS HDD Assembly Kit (Single RAID) 1 ASBF 600GB 10K 12Gbps SAS 2.5" G3HS 512e HDD 2 6400 2.8m, 13A/125-10A/250V, C13 to IEC 320-C14 Rack Power Cable 2 A5FV System x Enterprise Slides Kit 1 ASFE Notice for Advanced Format 512e Hard Disk Drives 1 A4EL HDD Filler ASM GEN 3 Quad Filler 2 A5G5 System x3650 M5 Riser Bracket 2 A5FT System x3650 M5 Power Paddle Card 1 A5G4 System x3650 M5 ODD Filler 1 A4C2 HDD Filler ASM GEN 3 6 A5G1 System x3650 M5 EIA Plate 1 A5V4 System x3650 M5 Right EIA 1 A4VH Lightpath LCD Op Panel 1 A5EA System x3650 M5 Planar 1

(33)

8.3 Networking BOMs

The Bill of Materials for the networking switches are listed in Table 9 and Table 10.

Table 9: RackSwitch G8124E

7159BR6 Lenovo System Networking RackSwitch G8124E (Rear to Front) 1 6201 1.5m, 10A/100-250V, C13 to IEC 320-C14 Rack Power Cable 2 3802 1.5m Blue Cat5e Cable 1 A1DK Lenovo 19" Flexible 4 Post Rail Kit 1

Table 10: RackSwitch G8264

7159G64 Lenovo System Networking RackSwitch G8264 (Rear to Front) 1 6201 1.5m, 10A/100-250V, C13 to IEC 320-C14 Rack Power Cable 2 A3KP Lenovo System Networking Adjustable 19" 4 Post Rail Kit 1 5053 SFP+ SR Transceiver 2 A1DP 1m QSFP+-to-QSFP+ cable 1 A1DM 3m QSFP+ DAC Break Out Cable 0

(34)

Resources

For more information about the topics that are described in this document, see the following resources:

 Storage Spaces Frequently Asked Questions (FAQ):

social.technet.microsoft.com/wiki/contents/articles/11382.storage-spaces-frequently-asked-question s-faq.aspx

 Control Scale-Out File Server Cluster Access Point Networks:

jeffgraves.me/2014/10/27/control-scale-out-file-server-cluster-access-point-networks/

 The basics of SMB Multichannel, a feature of Windows Server 2012 and SMB 3.0:

blogs.technet.com/b/josebda/archive/2012/05/13/the -basics-of-smb-multichannel-a-feature-of-windo ws-server-2012-and-smb-3-0.aspx

 SMB Transparent Failover – making file shares continuously available:

blogs.technet.com/b/clausjor/archive/2012/06/07/smb-transparent-failover-making-file-shares-contin uously-available.aspx

 Storage Spaces - Designing for Performance:

social.technet.microsoft.com/wiki/contents/articles/15200.storage-spaces-designing-for-performance .aspx

 Cluster Shared Volume (CSV) Inside Out:

(35)

Trademarks and special notices

References in this document to Lenovo products or services do not imply that Lenovo intends to make them available in every country.

Lenovo, the Lenovo logo, ThinkCentre, ThinkVision, ThinkVantage, ThinkPlus and Rescue and Recovery are trademarks of Lenovo.

IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

Intel, Intel Inside (logos), MMX, and Pentium are trademarks of Intel Corporation in the United States, other countries, or both.

Other company, product, or service names may be trademarks or service marks of others. Information is provided "AS IS" without warranty of any kind.

All customer examples described are presented as illustrations of how those customers have used Lenovo products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer.

Information concerning non-Lenovo products was obtained from a supplier of these products, published announcement material, or other publicly available sources and does not constitute an endorsement of such products by Lenovo. Sources for non-Lenovo list prices and performance numbers are taken from publicly available information, including vendor announcements and vendor worldwide homepages. Lenovo has not tested these products and cannot confirm the accuracy of performance, capability, or any other claims related to non-Lenovo products. Questions on the capability of non-Lenovo products should be addressed to the supplier of those products.

All statements regarding Lenovo future direction and intent are subject to change or withdrawal without notice, and represent goals and objectives only. Contact your local Lenovo office or Lenovo authorized reseller for the full text of the specific Statement of Direction.

Some information addresses anticipated future capabilities. Such information is not intended as a definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any future products. Such commitments are only made in Lenovo product announcements. The information is presented here to communicate Lenovo’s current investment and development activities as a good faith effort to help with our customers' future planning.

Performance is based on measurements and projections using standard Lenovo benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve throughput or performance improvements equivalent to the ratios stated here.

Photographs shown are of engineering prototypes. Changes may be incorporated in production models. Any references in this information to non-Lenovo websites are provided for convenience only and do not in any manner serve as an endorsement of those websites. The materials at those websites are not part of the materials for this Lenovo product and use of those websites is at your own risk

Reference Architecture: Microsoft Scale-Out File Server using Storage Spaces