Brocade VCS Fabric Technology with the NetApp 3170 NAS Storage Array

(1)

53-1003927-02

November 2015

Brocade VCS Fabric

Technology with the

NetApp 3170 NAS Storage

Array

Validation Test Report

Supporting Network OS 6.0.1 and 4.1.2

(2)

ADX, Brocade, Brocade Assurance, the B-wing symbol, DCX, Fabric OS, HyperEdge, ICX, MLX, MyBrocade, OpenScript, The Effortless Network, VCS, VDX, Vplane, and Vyatta are registered trademarks, and Fabric Vision and vADX are trademarks of Brocade

Communications Systems, Inc., in the United States and/or in other countries. Other brands, products, or service names mentioned may be trademarks of others.

Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning any equipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this document at any time, without notice, and assumes no responsibility for its use. This informational document describes features that may not be currently available. Contact a Brocade sales office for information on feature and product availability. Export of technical data contained in this document may require an export license from the United States government.

The authors and Brocade Communications Systems, Inc. assume no liability or responsibility to any person or entity with respect to the accuracy of this document or any loss, cost, liability, or damages arising from the information contained herein or the computer programs that accompany it.

The product described by this document may contain open source software covered by the GNU General Public License or other open source license agreements. To find out which open source software is included in Brocade products, view the licensing terms applicable to the open source software, and obtain a copy of the programming source code, please visit http://www.brocade.com/support/oscd.

(3)

Preface...4

Overview... 4

Purpose of This Document...4

Audience... 4 Objectives... 5 Document History...5 Related Documents...5 About Brocade... 5 About NetApp...6

Configure DUT and Test Equipment...7

Task 1. Brocade VDX Switch Configuration—Logical Chassis Mode...7

Task 2. NetApp 3170 Configuration... 7

Task 3. Windows Host Configuration... 20

Task 4. Red Hat Linux Host Configuration...26

Task 5. Medusa I/O Generation Configuration...26

NOS 6.0.1 Test Report... 28

Test Plan... 28 Scope... 28 Test Configuration...29 DUT Descriptions... 30 DUT Specifications...31 DUT Configuration... 32 Test Cases... 34 1.1 Base Functionality...36 1.2 Advanced Functionality... 37

1.3 Stress and Error Recovery...40

1.4 Brocade Network Advisor...45

Test Conclusions...46

NOS 4.1.2 Test Report... 47

Test Plan... 47 Scope... 47 Test Configuration...48 DUT Descriptions... 49 DUT Specifications...50 DUT Configuration... 52 Test Cases... 54

1.1 Ethernet Storage - Base Functionality... 55

1.2 Ethernet Storage - Advanced Functionality...58

1.3 Stress and Error Recovery...85

Test Conclusions...93

(4)

Preface

● Overview... 4

● Purpose of This Document...4

● Audience... 4 ● Objectives... 5 ● Document History...5 ● Related Documents...5 ● About Brocade... 5 ● About NetApp...6

Overview

Many data centers use network attached storage (NAS) for a variety of applications and workloads. As more data is stored on NAS, efficient network designs are under evaluation to minimize latency, to maximize use of interswitch links between switches, and to automate network configuration changes. For these reasons, Brocade VCS Fabric technology with Brocade VDX switches is increasingly being deployed when new NAS arrays are added to the storage pool.

Purpose of This Document

The goal of this document is to validate Brocade VCS Fabric technology with the NetApp 3170 NAS storage array. The testing includes cases that demonstrate VCS Fabric feature support, high availability and resiliency, and automatic network reconfiguration when links are added or removed. This document provides validation test results for typical use cases of NAS with Brocade VCS Fabric technology and NetApp 3170 NAS storage arrays.

Audience

This document is written for a technical audience, including solution architects, system engineers, and technical development representatives.

(5)

Objectives

1. Validate interoperability with redundant and nonredundant network topologies with Brocade VCS Fabric technology and the NetApp 3170.

2. Identify any unique configuration details necessary to establish end-to-end connectivity. 3. Simulate real-world customer environment perturbation and validate end-to-end behavior and

recovery.

Document History

Date Part Number Description

June 2015 53-1003927-01 Initial Release

November 2015 53-1003927-02 Additional Test Configuration and Test Case details added.

About Brocade

Brocade networking solutions empower the world's leading organizations to transition smoothly to a world where applications and information reside anywhere. This vision is realized through the Brocade One™ strategy, which is designed to deliver key business benefits such as unmatched simplicity, non-stop networking, application optimization, and investment protection.

Innovative Ethernet and storage networking solutions for data center, campus, and service provider networks help reduce complexity and cost while enabling virtualization and cloud computing to increase business agility.

To help ensure a complete solution, Brocade partners with world-class IT companies and provides comprehensive education, support, and professional services offerings.

To learn more, visit ( www.brocade.com)

(6)

About NetApp

NetApp creates innovative products-storage systems and software that help customers around the world store, manage, protect, and retain one of their most precious corporate assets: their data. We are recognized throughout the industry for continually pushing the limits of today's technology so that our customers never have to choose between saving money and acquiring the capabilities they need to be successful.

(7)

Configure DUT and Test Equipment

● Task 1. Brocade VDX Switch Configuration—Logical Chassis Mode...7

● Task 2. NetApp 3170 Configuration... 7

● Task 3. Windows Host Configuration... 20

● Task 4. Red Hat Linux Host Configuration...26

● Task 5. Medusa I/O Generation Configuration...26

This section describes the initial configuration of devices under test (DUTs)and test equipment. In some test cases, additional configuration changes are made.

Task 1. Brocade VDX Switch Configuration—Logical Chassis Mode

On the VCS cluster, configure VLAN interfaces on all cluster members. Configure the VLAG interfaces connecting to Windows and Red Hat hosts as well as the NetApp 3170 storage array.

NOTE

Please see DUT Configuration section of each test report below for applicable configuration commands.

Task 2. NetApp 3170 Configuration

This section describes the NetApp 3170 NAS and share configurations. 1. Physical Interface Configuration

a. For single interface configuration, open the NetApp OnCommand System Manager and navigate to Configuration -> Network -> Network Interfaces. Select the desired physical interface and click Edit. Enter the desired interface IP address in the IPv4 Address / Netmask field. Finally, click Save and Close.

(8)

FIGURE 1 NetApp 3170 Network Interface Configuration - Screen 1

(9)

FIGURE 2 NetApp 3170 Network Interface Configuration - Screen 2

b. For LACP interface configuration, open the NetApp OnCommand System Manager and navigate to Configuration -> Network -> Network Interfaces. Then, click on Create VIF to start the VIF Wizard. Supply a name for the Virtual Interface (VIF) Name, select the desired physical interfaces to aggregate, and select LACP as the Trunk Mode.

(10)

FIGURE 3 NetApp 3170 VIF Wizard - Screen 1

c. Choose IP based for the Load balancing type.

(11)

d. Choose This interface will not be part of another virtual interface.

(12)

e. FIGURE 5 NetApp 3170 VIF Wizard - Screen 3

f. Enter the desired interface IP address in the IPv4 Address / Netmask field.

(13)

(14)

2. Storage Volume Configuration

a. Open the NetApp OnCommand System Manager and navigate to Storage -> Volumes. Then, click on Create to start the Create Volume Wizard. Supply a volume name for the Name, select the volume size for Total Size, and set Snapshot Reserve to 0.

(15)

FIGURE 8 NetApp 3170 Storage Volume Configuration - Screen 1

b. (Optional) After the volume is created, return to Storage -> Volumes, highlight the newly-created volume, and click Snapshot Copies -> Configure. Then, uncheck Make Snapshot directory

(.snapshot) visible and uncheck Enable scheduled Snapshot Copies. This option reduces disk

usage at the expense of data backup.

(16)

(17)

3. CIFS Share Configuration

a. Open the NetApp OnCommand System Manager and navigate to Storage -> Shares. Then, click on Create to create a new share. Next, click on Browse and select the full path to the desired volume previously created. The Selected Path will autofill with the appropriate path.

(18)

FIGURE 11 NetApp 3170 CIFS Share Configuration - Screen 1

b. Click Create to complete the share creation.

(19)

4. NFS Export Configuration

NetApp 3170 automatically creates NFS exports for any volumes created, so the exports designated for the Windows servers can be deleted if desired. Open the NetApp OnCommand System Manager and navigate to Storage -> Exports. Then, click on the NFS export you wish to delete and click

Delete.

For this configuration, anonymous access should be granted to all hosts. Open the NetApp

OnCommand System Manager and navigate to Storage -> Exports. Then, click on the NFS export you wish to delete and click on the Unix folder entry and click Edit.

Within the Edit Export Rule window, select Grant root access to all hosts under the Anonymous Access header.

(20)

FIGURE 14 NetApp 3170 NFS Export Configuration

Task 3. Windows Host Configuration

Configuring and mapping windows hosts for the VCS cluster. 1. Configure Windows 2012 Server Hosts for Ethernet Teaming.

a. Configure Windows 2012 Server NIC teaming interface for connectivity to VCS cluster.

(21)

FIGURE 15 Windows 2012 NIC Teaming

2. Configure Windows 2008 Server Hosts for Ethernet Teaming

a. Configure Windows 2008 Server NIC teaming interface for connectivity to VCS cluster using the Brocade Host Connectivity Manager.

(22)

FIGURE 16 Windows 2008 NIC Team using Brocade Host Connectivity Manager

3. Map NetApp 3170 Network Shares on Windows 2012 Server a. Map network drives on Windows 2012 Server hosts.

(23)

FIGURE 17 Windows 2012 NIC Team Configuration - Screen 1

(24)

FIGURE 18 Windows 2012 NIC Team Configuration - Screen 2

4. Map Network Drives on Windows 2008 Server

a. Map network drives on Windows 2008 Server hosts.

(25)

FIGURE 19 Windows 2008 Network Drive Mapping - Screen 1

FIGURE 20 Windows 2008 Network Drive Mapping - Screen 2

(26)

Task 4. Red Hat Linux Host Configuration

This task describes configuring NIC bonding and mounting shares on Red Hat Linux. 1. Configure NIC Bonding

The following are the entries within /etc/sysconfig/network-scripts/ on Red Hat hosts to enable Ethernet bonding, or NIC bonding.

< =========== > ifcfg-eth4 DEVICE=eth4 BOOTPROTO=none ONBOOT=yes MASTER=bond0 SLAVE=yes USERCTL=no NM_CONTROLLED=no ifcfg-eth5 DEVICE=eth5 BOOTPROTO=none ONBOOT=yes MASTER=bond0 SLAVE=yes USERCTL=no NM_CONTROLLED=no ifcfg-bond0 DEVICE=bond0 IPADDR=9.79.100.21 NETMASK=255.255.0.0 ONBOOT=yes BOOTPROTO=none USERCTL=no NM_CONTROLLED=no

BONDING_OPTS="mode=802.3ad miimon=100 downdelay=5000 updelay=5000" < =========== >

2. Mount NFS Disk Shares

The following commands mount NetApp 3170 NFS shares on hosts hosts running Red Hat 6.5 or 6.3.

< =========== >

mount -t nfs 9.79.20.1:/REDHAT_66.109 /mnt/netapp3170 mount -t nfs 9.79.20.1:/REDHAT_79.141 /mnt/netapp3170 < =========== >

Task 5. Medusa I/O Generation Configuration

The following shows the configuration of the Medusa I/O data patterns used in the test cases. 1. Standard I/O with 64KB Blocks

For test scenarios using standard I/O (64-KB testing size), configure Medusa for the following command line result:

< =========== >

pain 2GB -b64KB -t8 -v2 -H45 -l35 -Y1 –M45 < =========== >

2. Maximum I/O with 1-MB Blocks

(27)

For test scenarios using maximum bandwidth I/O (1-MB testing size), configure Medusa for the following command line result:

< =========== >

maim 2GB -b1MB -Q8 -n -u -q5 -w -o -Y1 –M45 --perf-mode < =========== >

3. Microsoft Exchange Server I/O

For test scenarios using Microsoft Exchange Server Simulation (Medusa preconfigured Storage Simulation test), configure Medusa for the following command line result:

< =========== >

maim 2GB -%r50@4KB -%w25@64KB -%w25@4KB -Q1 -n -u -q5 -o -l69 -Y1 -M45 < =========== >

4. File Server I/O

For test scenarios using File Server Simulation (Medusa preconfigured Storage Simulation test), configure Medusa for the following command line result:

< =========== >

maim 2GB %10:r80@512B,w20@512B %5:r80@1KB,w20@1KB %5:r80@2KB,w20@2KB %60:r80@4KB,w20@4KB %2:r50@8KB,w50@8KB %4:r75@16KB,w25@16KB

-%4:r75@32KB,w25@32KB -%10:r80@64KB,w20@64KB -Q1 -n -u -o -l69 -Y1 -M45 < =========== >

(28)

NOS 6.0.1 Test Report

● Test Plan...28

● Test Cases...34

● Test Conclusions...46

Test Plan

The storage array is connected to one VDX Ethernet fabric with multiple Windows and Red Hat server hosts to drive I/O.

Scope

Testing will be performed with Brocade's Network Operating System (NOS) in a heterogeneous environment. Test beds will include Brocade switches configured in routed and non-routed fabric configurations.

Testing is centered on interoperability and optimal configuration. Performance is observed within the context of best practice fabric configuration; however absolute maximum benchmark reporting of storage performance is beyond the scope of this test.

(29)

Test Configuration

FIGURE 21 Test Topology

The following diagram shows Brocade VCS Fabric switch port connections and vLAG (port-channel) ports for each server and the NetApp 3170 array. For example, “1/0/1” means the device port connects to RBridge# = 1, Card# = 0, and Port# = 1. For fixed top-of-rack switches such as the Brocade VDX 6720, 6730, and 6740, there are no port cards, which is indicated by “0” for the Card#.

(30)

FIGURE 22 Device to VCS Fabric Switch Port Connections

DUT Descriptions

The following tables provide details about the devices under test (DUTs). Storage Array

TABLE 1

Model Vendor Description

FAS3170 NetApp Network attached storage (NAS) array

Switches

TABLE 2

(31)

Switches (Continued)

TABLE 2

BR-VDX6740 Brocade The Brocade VDX 6740 and 6740T are Ethernet fabric top-of-rack (ToR) switches featuring 10-GbE ports with 40-GbE uplinks. The Brocade VDX 6740T-1G Switch offers dual-speed functionality. It can be deployed with1000BASE-T for existing 1-GbE server connectivity and upgraded via software to 10GBASE-T for future bandwidth growth. Together with Brocade VCS Fabric technology, these switches deliver the high performance and low latency needed to support demanding virtualized data center environments.

DUT Specifications

NAS Array Version

NetApp 3170 7-Mode 8.2.2

NetApp OnCommand System Manager Version 3.1 (3.1.2RC2)

Brocade Switches Device ID Version

VDX 6740 CASTOR-VCS20-1 Network OS 6.0.1

Adapters Version

Brocade 1020 3.2.5.1 (Red Hat Enterprise Linux 6.6) Brocade 1860 3.2.5.0 (Windows Server 2008) Brocade 1860 3.2.3.0 (ESXi 5.5u2)

Emulex OCe14102 10.4.255.25 (Windows Server 2012)

Servers Memory Processor

IBM x3650 M3 8 GB E5506 @ 2.13 GHz

IBM x3650 M3 32 GB E5506 @ 2.13 GHz

HP ProLiant DL380 G7 8 GB E5506 @ 2.13 GHz

(32)

HP ProLiant DL380 G7 8 GB E5506 @ 2.13 GHz Cisco UCS B200 M3 (x4) 64 GB E5-2600 @ 2.10 GHz (x2)

Operating Systems Version

ESXi 5.5u2 (x2)

ESXi Guests Windows Server 2012 R2 (x2), Red Hat Enterprise Linux 6.6, Red Hat Enterprise Linux 7.0, SUSE Linux Enterprise Server 11.3

Red Hat Enterprise Linux Server 6.6, 7.0 SUSE Linux Enterprise Server 11.3

Windows Server Windows Server 2008 R2, Windows Server 2012 R2 (x2)

Test Equipment Version

Medusa Labs Test Tools 7.0.0.157942

Device Interface IP Address

NetApp 3170 node4, e4a 9.79.20.1

NetApp 3170 node3, VIF NOS_VCS20 9.79.20.2

Windows 2012 Server NIC Team 9.79.100.1

Red Hat 6.6 Server bond0 9.79.100.21

Windows 2012 Server Cisco VIC Ethernet 2 9.79.100.61

Red Hat 7.0 Server Cisco VIC enp8s0 9.79.100.62

SUSE Linux Server Cisco VIC eth2 9.79.100.63

ESXi 5.5u2 Server Cisco VIC vSwitch2 9.79.100.64

ESXi 5.5u2 Server vSwitch1 9.79.100.74

DUT Configuration

< ========== > interface Vlan 979 ! rbridge-id 1 interface Ve 979 ip address 9.79.1.1/16

DUT Configuration

(33)

no shutdown ! ! rbridge-id 2 interface Ve 979 ip address 9.79.1.2/16 no shutdown ! ! rbridge-id 11 interface Ve 979 ip address 9.79.1.11/16 no shutdown ! ! rbridge-id 12 interface Ve 979 ip address 9.79.1.12/16 no shutdown ! ! interface TenGigabitEthernet 1/0/1 no fabric isl enable

no fabric trunk enable

channel-group 101 mode active type standard lacp timeout long

no shutdown !

interface TenGigabitEthernet 1/0/2 no fabric isl enable

no shutdown !

interface TenGigabitEthernet 11/1/3

(34)

no fabric isl enable no fabric trunk enable

no shutdown !

interface TenGigabitEthernet 11/1/4 cee default

no fabric isl enable no fabric trunk enable switchport

switchport mode access switchport access vlan 979 spanning-tree shutdown no shutdown

!

no shutdown ! interface Port-channel 21 cee default vlag ignore-split switchport

switchport mode access switchport access vlan 979 spanning-tree shutdown no shutdown ! interface Port-channel 101 vlag ignore-split switchport

!

< ========== >

Test Cases

The following test cases are designed to verify basic and advanced functionality features between the Brocade VCS cluster and host and storage devices, as well as to stress all devices and confirm successful error recovery.

(35)

1.1 Base Functionality

1.1.1 Physical Connection and Speed Negotiation

1.1.2 Storage Connectivity

1.1.3 Virtual LAG Connectivity

1.2 Advanced Functionality

1.2.1 MTU Jumbo Frame

1.2.2 Maximum Bandwidth

1.2.3 Congested Fabric with Auto-NAS 1.2.4 Jammer CRC Corruption on an ISL Port 1.2.5 Jammer Packet Corruption on an ISL Port 1.2.6 Jammer Packet Loss on an ISL Port 1.2.7 Jammer CRC Corruption on a Storage Port 1.2.8 Jammer Packet Corruption on a Storage Port 1.2.9 Jammer Packet Loss on a Storage Port

1.2.10 ISL Congestion and Interface Buffer Queue Length

1.3 Stress and Error Recovery

1.3.1 Congested Fabric with Auto-NAS Extended Run

1.3.2 Manual Cable Pull

1.3.3 ISL Port Relocation

1.3.4 Device Port Toggle Extended Run 1.3.5 ISL Port Toggle Extended Run

1.3.6 Split Brain Recovery

1.3.7 Brocade VDX 8770 Line-Card Maintenance Extended Run

1.3.8 Switch Offline

1.3.9 Workload Simulation—Microsoft Exchange Server 1.3.10 Workload Simulation—File Server

1.3.11 Host-side Interface Port Toggle

(36)

1.3.12 Storage-side Interface Port Toggle

1.3.13 Storage Failover

1.4 Brocade Network Advisor

1.4.1 Ethernet Fabric Discovery

1.4.2 Ethernet Fabric Addition and Removal

PASS. All I/O completes error-free. All host and storage ports show bandwidth under maximum

capacity.

(38)

1.2.3 Congested Fabric with Auto-NAS

Test Objective

Test host-to-storage traffic using an I/O generator configured for maximum throughput while configured with Brocade VCS Auto-NAS enabled and additional Layer 3 FCoE and Layer 2 Ethernet traffic to congest ISL links.

Validation

drops decrease.

1.3 Stress and Error Recovery

1.3.1 Congested Fabric with Auto-NAS Extended Run

and reconnections.

1.3.4 Device Port Toggle Extended Run

1.3.7 Brocade VDX 8770 Line-Card Maintenance Extended Run

Test Objective

Test that host-to-storage traffic can be maintained during line-card power-off and power-on commands sequentially on each Brocade VDX 8770 line card in the VCS cluster.

(43)

Validation

All storage I/O runs error-free during line-card power-off and power-on actions for a minimum of 12 hours.

Results

N/A. The Brocade VDX 8770 is not present in the network configuration.

1.3.8 Switch Offline

Test Objective

Test that host-to-storage traffic can be maintained during chassis disable and enable commands, switch reloads, and chassis power cycles sequentially on each Brocade VDX in the VCS cluster.

Validation

All storage I/O runs error-free during chassis disable and enable commands, switch reloads, and chassis power cycles.

Results

Chassis Disable/Enable: PASS. I/O successfully resumes after the chassis is enabled. Reboot: PASS. I/O successfully resumes after the chassis is rebooted.

Power Cycle: PASS. I/O successfully resumes after the chassis is power-cycled.

1.3.9 Workload Simulation—Microsoft Exchange Server

Test Objective

Test host-to-storage traffic using the Medusa Microsoft Exchange Server workload simulation suite.

Validation

All storage I/O runs error-free for a minimum of 30 minutes.

Results

PASS. All I/O completes error-free.

1.3.10 Workload Simulation—File Server

does not allow individual interfaces to be manipulated if they are part of a trunk group. Testing only validated that traffic recovered after the trunk group was disabled and enabled.

1.3.13 Storage Failover

Test Objective

Test that storage node redundancy failover occurs within the expected storage node feature set.

(45)

Validation

All storage I/O runs error-free (if lossless failover is supported) during the storage failover action, or traffic successfully resumes after the storage failover action is complete (if lossless failover is not supported).

Results

PASS. I/O successfully resumes after the node is rebooted.

1.4 Brocade Network Advisor

1.4.1 Ethernet Fabric Discovery

Test Objective

Test the Brocade Network Advisor Ethernet fabric discovery.

Validation

Brocade Network Advisor successfully discovers the VCS cluster.

Results

PASS. The VCS cluster is successfully discovered.

1.4.2 Ethernet Fabric Addition and Removal

Test Objective

Test the Brocade Network Advisor Ethernet fabric node addition and removal.

Validation

Brocade Network Advisor successfully tracks both VCS node addition and removal.

Results

PASS. Brocade Network Advisor accurately reports the VCS cluster changes during both removal and

addition.

1.4.3 Ethernet Fabric Configuration Backup and Restoration

Test Objective

Test the Brocade Network Advisor Ethernet fabric configuration backup and restoration.

(46)

Validation

The Brocade Network Advisor successfully performs configuration backup and restoration.

Results

PASS. Brocade Network Advisor successfully performs both configuration backup and restoration.

1.4.4 Ethernet Fabric Firmware Download

Test Objective

Test the Brocade Network Advisor Ethernet fabric firmware download.

Validation

Brocade Network Advisor successfully downloads firmware to all VCS cluster nodes.

Results

PASS. Brocade Network Advisor successfully performs firmware download to Network OS 6.0.1.

Test Conclusions

1. Brocade VCS Fabric technology and the NetApp 3170 NAS array smoothly interoperate with both redundant and nonredundant configurations. No exceptions were noted.

2. The configuration procedures followed Brocade VCS Fabric and NetApp 3170 standard documentation guides and found no issues with these procedures.

3. All simulated changes in network and storage (add, remove, fail over) resulted in expected behavior and I/O recovery.

(47)

NOS 4.1.2 Test Report

● Test Plan... 47

● Test Cases... 54

● Test Conclusions...93

Test Plan

The storage array is connected to one VDX Ethernet fabric with multiple Windows and Red Hat server hosts to drive I/O.

Scope

Testing will be performed with Brocade's Network Operating System (NOS) in a heterogeneous environment. Test beds will include Brocade switches configured in routed and non-routed fabric configurations.

Testing is centered on interoperability and optimal configuration. Performance is observed within the context of best practice fabric configuration; however absolute maximum benchmark reporting of storage performance is beyond the scope of this test.

(48)

Test Configuration

FIGURE 23 Test Topology

The following diagram shows Brocade VCS Fabric switch port connections and vLAG (port-channel) ports for each server and the NetApp 3170 array. For example, “1/0/1” means the device port connects to RBridge# = 1, Card# = 0, and Port# = 1. For fixed top-of-rack switches such as the Brocade VDX 6720, 6730, and 6740, there are no port cards, which is indicated by “0” for the Card#.

(49)

FIGURE 24 Device to VCS Fabric Switch Port Connections

DUT Descriptions

The following tables provide details about the devices under test (DUTs). Storage Array

TABLE 3

FAS3170 NetApp Network attached storage (NAS) array

Switches

TABLE 4

(50)

Switches (Continued)

TABLE 4

BR-VDX6740-24 Brocade The Brocade VDX 6720 is a high-performance, ultra-low latency wire-speed 10 Gigabit Ethernet (GbE) fixed configuration switch. The VDX 6720 has been superseded by the VDX 6740 which is the latest generation of ToR switch in the VDX Switch family offering more bandwidth (40 GE) and options for 10BASE-T connectivity.

BR-VDX6730-32 Brocade The Brocade VDX 6730 Switch is a 10 Gigabit Ethernet (GbE) fixed configuration switch with LAN and native Fibre Channel ports. It supports multiple connectivity options, Ethernet storage connectivity for Fibre Channel over Ethernet (FCoE), iSCSI, NAS, and bridges to Fibre Channel Storage Area Networks (SANs).

BR-VDX6740 Brocade The Brocade VDX 6740 and 6740T are Ethernet fabric Top of Rack (ToR) switches featuring 10 GbE ports with 40 GbE uplinks. The new Brocade VDX 6740T-1G Switch offers dual-speed functionality. It can be deployed with1000BASET for existing 1 GbE server connectivity and upgraded via software to 10GBASE-T for future bandwidth growth. Together with Brocade VCS Fabric technology, these switches deliver the high performance and low latency needed to support demanding virtualized data center environments.

BR-VDX8770-4 Brocade The Brocade VDX 8770 Switch is a highly scalable, low-latency, 1/10/40/100 Gigabit Ethernet (GbE) modular switch. Designed to easily scale out Brocade VCS fabrics, the Brocade VDX 8770 Switch brings new levels of performance to VCS fabric deployments.

DUT Specifications

NAS Array Version

NetApp 3170 NetApp Release 8.2.1 7-Mode

NetApp OnCommand System Manager Version 3.1 (3.1.1RC1)

QLogic CNA 8152 Firmware Revision 5.8.0

Brocade Switches Version

VDX 6720 Network OS 4.1.2

(51)

Adapters Version

Brocade 1020 3.2.4.0 (Windows), 3.0.0.0 (Red Hat) Brocade 1860 3.2.4.0 (Windows)

Emulex OCe14102 8.3.5.68.5p (Red Hat)

Servers Memory Processor

IBM x3650 M3 8 GB E5506 @ 2.13 GHz (x4)

HP ProLiant DL380 G7 8 GB E5506 @ 2.13 GHz

HP ProLiant DL380 G7 8 GB E5506 @ 2.13 GHz (x4)

Operating Systems Version

Red Hat Enterprise Linux Server 6.5, 6.3

Windows Server 2008 R2, 2012 R2

Test Equipment Version

Medusa Labs Test Tools 6.0.1.148039

Device Interface IP Address

NetApp 3170 node4, e4a 9.79.20.1

NetApp 3170 node3, VIF NOS_VCS20 9.79.20.2

(52)

DUT Configuration

< ========== > interface Vlan 979 ! rbridge-id 1 interface Ve 979 ip address 9.79.1.1/16 no shutdown ! ! rbridge-id 2 interface Ve 979 ip address 9.79.1.2/16 no shutdown ! ! rbridge-id 11 interface Ve 979 ip address 9.79.1.11/16 no shutdown ! ! rbridge-id 12 interface Ve 979 ip address 9.79.1.12/16 no shutdown ! ! interface TenGigabitEthernet 1/0/1 no fabric isl enable

no shutdown !

channel-group 105 mode active type standard no fabric isl enable

no fabric trunk enable lacp timeout long no shutdown !

interface TenGigabitEthernet 1/0/7 channel-group 107 mode on type standard no fabric isl enable

no fabric trunk enable no shutdown

!

no shutdown

(53)

!

no shutdown !

interface TenGigabitEthernet 2/0/7 channel-group 107 mode on type standard no fabric isl enable

no fabric trunk enable no shutdown

!

no shutdown !

interface TenGigabitEthernet 11/0/4 cee default

no fabric isl enable no fabric trunk enable switchport

!

no shutdown ! interface Port-channel 21 cee default vlag ignore-split switchport

!

interface Port-channel 102 vlag ignore-split

(54)

switchport

switchport mode trunk

switchport trunk allowed vlan add 979 switchport trunk tag native-vlan spanning-tree shutdown no shutdown ! interface Port-channel 106 vlag ignore-split switchport

switchport trunk allowed vlan add 979 switchport trunk tag native-vlan spanning-tree shutdown no shutdown ! interface Port-channel 107 no vlag ignore-split switchport

switchport trunk allowed vlan add 979 switchport trunk tag native-vlan spanning-tree shutdown

no shutdown !

< ========== >

Test Cases

The following test cases are designed to verify basic and advanced functionality features between the Brocade VCS cluster and host and storage devices, as well as to stress all devices and confirm successful error recovery.

1.1 Ethernet Storage - Base Functionality

1.1.1 Storage Device - Physical Connection and Speed Negotiation

1.1.2 NAS Connectivity

1.1.3 vLAG Connectivity

1.2 Ethernet Storage - Advanced Functionality

1.2.1 Storage Device - Jumbo Frame/MTU Size Validation

1.2.2 NAS Bandwidth Validation

(55)

1.2.3 Storage Device - w/Congested Fabric

1.2.4 Storage Device - NAS Protocol Jammer CRC Corruption on ISL Port 1.2.5 Storage Device - NAS Protocol Jammer Packet Corruption on ISL Port 1.2.6 Storage Device - NAS Protocol Jammer Packet Loss on ISL Port 1.2.7 Storage Device - NAS Protocol Jammer CRC Corruption on Storage Port 1.2.8 Storage Device - NAS Protocol Jammer Packet Corruption on Storage Port 1.2.9 Storage Device - NAS Protocol Jammer Packet Loss on Storage Port

1.3 Stress and Error Recovery

1.3.1 Storage Device Fabric IO Integrity - Congested Fabric

1.3.2 Storage Device Integrity - Device Recovery from Port Toggle - Manual Cable 1.3.3 Storage Device Integrity - Device Recovery from ISL Port Relocation 1.3.4 Storage Device Stress Device Recovery from Device Port Toggle

-Extended Run

1.3.5 Storage Device Recovery - ISL Port Toggle - Extended Run 1.3.6 Storage Device Recovery - All-ISL Port Toggle

1.3.7 Storage Device Recovery - VDX 8770 Line Card Maintenance 1.3.8 Storage Device Recovery - Switch Offline

1.3.9 Storage Device Recovery - Switch Firmware Download ISSU on VDX8770 1.3.10 Workload Simulation Test Suite - Microsoft Exchange Server

1.3.11 Workload Simulation Test Suite - File Server Simulation

Test Case Descriptions

Lines marked with ' < == ' indicate important output used to validate the test results.

1.1 Ethernet Storage - Base Functionality

1.1.1 Storage Device - Physical Connection and Speed Negotiation

Test Objective

Verify device connectivity to VDX switch with all supported speed settings. Configure storage port for iSCSI connectivity. Validate base connectivity.

(56)

Results

PASS. All interfaces are up and operational at the expected speed settings. Brocade VDX show interface commands reflect the expected interface speed.

< =========== >

CASTOR-VCS20-12# show interface tengigabitethernet 11/1/3

TenGigabitEthernet 11/1/3 is up, line protocol is up (connected) < == Hardware is Ethernet, address is 0027.f81c.71d0

Current address is 0027.f81c.71d0 Pluggable media not present

Interface index (ifindex) is 47651586051 MTU 2500 bytes

LineSpeed Actual : 10000 Mbit < == LineSpeed Configured : Auto, Duplex: Full ...

TenGigabitEthernet 11/1/4 is up, line protocol is up (connected) < == Hardware is Ethernet, address is 0027.f81c.71d1

TenGigabitEthernet 12/0/3 is up, line protocol is up (connected) < == Hardware is Ethernet, address is 0005.3365.2967

Current address is 0005.3365.2967 Pluggable media not present

< =========== >

1.1.2 NAS Connectivity

Test Objective

Verify host to File Share connectivity with CIFS and NFS with multiple simultaneous connections.

Results

PASS. All network drives on all Windows and Red Hat hosts are successfully mapped.

< =========== >

PS C:\Users\Administrator> net use New connections will be remembered.

Status Local Remote Network

---OK J: \\9.79.20.1\WINDOWS_68114 Microsoft Windows Network < == OK K: \\9.79.20.2\WINDOWS_68114 Microsoft Windows Network < == \\TSCLIENT\C Microsoft Terminal Services The command completed successfully.

C:\Users\Administrator>net use New connections will be remembered.

Status Local Remote Network

(57)

OK J: \\9.79.20.1\WINDOWS_68128 Microsoft Windows Network < == OK K: \\9.79.20.2\WINDOWS_68128 Microsoft Windows Network < == \\TSCLIENT\C Microsoft Terminal Services The command completed successfully.

[root@ilo65205 ~]# df

Filesystem 1K-blocks Used Available Use% Mounted on ...

9.79.20.1:/vol/REDHAT_79141 20971520 16816768 4154752 81% /mnt/netapp3170 < == [root@v-e-vmw066109 ~]# df

Filesystem 1K-blocks Used Available Use% Mounted on ... 9.79.20.1:/vol/REDHAT_66109 < == 20971520 16816704 4154816 81% /mnt/netapp3170 < == < =========== >

1.1.3 vLAG Connectivity

Test Objective

Configure vLAG connectivity from Storage Ports to 2 separate VDX switches. Verify data integrity through vLAG.

Results

PASS. All port-channel interfaces are operational and in sync.

< =========== >

CASTOR-VCS20-12# show port-channel 21 LACP Aggregator: Po 21 (vLAG) Aggregator type: Standard Ignore-split is enabled Member rbridges: rbridge-id: 11 (1) rbridge-id: 12 (1)

Admin Key: 0021 - Oper Key 0021

Partner System ID - 0x0001,02-a0-98-12-39-00 Partner Oper Key 0001

Member ports on rbridge-id 11:

Link: Te 11/1/3 (0xB18418003) sync: 1 * < == Member ports on rbridge-id 12:

Link: Te 12/0/3 (0xC18018002) sync: 1 < == CASTOR-VCS20-12# show port-channel 101

LACP Aggregator: Po 101 (vLAG) Aggregator type: Standard Ignore-split is enabled Member rbridges: rbridge-id: 1 (1) rbridge-id: 2 (1)

Partner System ID - 0x0000,00-90-fa-6c-01-ae Partner Oper Key 0000

Link: Te 1/0/1 (0x118008000) sync: 1 * < == Member ports on rbridge-id 2:

Link: Te 2/0/1 (0x218008000) sync: 1 < == CASTOR-VCS20-12# show port-channel 102

Partner System ID - 0xffff,8c-7c-ff-10-84-02

(58)

Partner Oper Key 0017 Member ports on rbridge-id 1:

Link: Te 1/0/2 (0x118010001) sync: 1 * < == Member ports on rbridge-id 2:

Link: Te 2/0/2 (0x218010001) sync: 1 < == CASTOR-VCS20-12# show port-channel 103

Partner System ID - 0xffff,00-05-33-48-4c-12 Partner Oper Key 0033

Link: Te 1/0/3 (0x118018002) sync: 1 < == Member ports on rbridge-id 2:

Link: Te 2/0/3 (0x218018002) sync: 1 * < == CASTOR-VCS20-12# show port-channel 104

Partner System ID - 0xffff,00-00-c9-b1-e5-c2 Partner Oper Key 0033

Link: Te 1/0/4 (0x118020003) sync: 1 < == Member ports on rbridge-id 2:

Link: Te 2/0/4 (0x218020003) sync: 1 * < == < =========== >

1.2 Ethernet Storage - Advanced Functionality

1.2.1 Storage Device - Jumbo Frame/MTU Size Validation

Test Objective

Perform I/O validation testing while incrementing MTU size from minimum to maximum with reasonable increments. Include Jumbo frame sizes as well as maximum negotiated/supported between device and switch.

Test Configuration

1. Open the NetApp OnCommand System Manager and navigate to Configuration -> Network ->

Network Interfaces. Then, highlight the desired network interface and click Edit. Click on the Advanced tab and change the entry for MTU Size (in bytes) to the desired value. Finally, click Save and Close.

(59)

FIGURE 25 NetApp 3170 MTU Configuration

2. For Red Hat hosts, add "MTU=9000" to /etc/sysconfig/network-scripts/ifcfg-bond0 and then execute

service network restart.

< =========== >

[root@ilo65205 ~]# more /etc/sysconfig/network-scripts/ifcfg-bond0 DEVICE=bond0 IPADDR=9.79.100.21 NETMASK=255.255.0.0 ONBOOT=yes BOOTPROTO=none MTU=9000 USERCTL=no NM_CONTROLLED=no

BONDING_OPTS="mode=802.3ad miimon=100 downdelay=5000 updelay=5000" [root@ilo65205 ~]#

< =========== >

3. For Windows 2012 Server host, reconfigure individual NIC team members for increased MTU size. Perform Disable and Enable on NIC Team interface within Windows to apply packet size change.

(60)

FIGURE 26 Windows 2012 Server Emulex NIC MTU Configuration

4. For Windows 2008 Server host, reconfigure NIC team interface.

(61)

FIGURE 27 Windows 2008 Server Brocade NIC MTU Configuration

5. On VDX devices, configure as follows:

< =========== > rbridge-id 1 interface ve 979 ip mtu 9018 rbridge-id 2 interface ve 979 ip mtu 9018 rbridge-id 11 interface ve 979 ip mtu 9018 rbridge-id 12 interface ve 979 ip mtu 9018 interface TenGigabitEthernet 11/1/4 mtu 9216 interface Port-Channel 21 mtu 9216 interface Port-Channel 101 mtu 9216 interface Port-Channel 102 mtu 9216 interface Port-Channel 103 mtu 9216 interface Port-Channel 104 mtu 9216 < =========== >

(62)

Results

PASS. Non-fragmented ping packets were successfully responded to and all jumbo frame storage

traffic was error-free.

< =========== >

PS C:\Users\Administrator> ping 9.79.20.1 -l 8972 -f Pinging 9.79.20.1 with 8972 bytes of data:

Reply from 9.79.20.1: bytes=8972 time<1ms TTL=255 Reply from 9.79.20.1: bytes=8972 time<1ms TTL=255 Reply from 9.79.20.1: bytes=8972 time<1ms TTL=255 Reply from 9.79.20.1: bytes=8972 time<1ms TTL=255 Ping statistics for 9.79.20.1:

Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), < == Approximate round trip times in milli-seconds:

Minimum = 0ms, Maximum = 0ms, Average = 0ms C:\Users\Administrator>ping 9.79.20.1 -l 8972 -f Pinging 9.79.20.1 with 8972 bytes of data:

Reply from 9.79.20.1: bytes=8972 time<1ms TTL=255 Reply from 9.79.20.1: bytes=8972 time<1ms TTL=255 Reply from 9.79.20.1: bytes=8972 time<1ms TTL=255 Reply from 9.79.20.1: bytes=8972 time<1ms TTL=255 Ping statistics for 9.79.20.1:

Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), < == Approximate round trip times in milli-seconds:

Minimum = 0ms, Maximum = 0ms, Average = 0ms [root@ilo65205 ~]# ping -c 4 -M do -s 8972 9.79.20.1 PING 9.79.20.1 (9.79.20.1) 8972(9000) bytes of data.

8980 bytes from 9.79.20.1: icmp_seq=1 ttl=255 time=0.161 ms 8980 bytes from 9.79.20.1: icmp_seq=2 ttl=255 time=0.160 ms 8980 bytes from 9.79.20.1: icmp_seq=3 ttl=255 time=0.166 ms 8980 bytes from 9.79.20.1: icmp_seq=4 ttl=255 time=0.160 ms 9.79.20.1 ping statistics

---4 packets transmitted, ---4 received, 0% packet loss, time 2999ms < == rtt min/avg/max/mdev = 0.160/0.161/0.166/0.015 ms

[root@v-e-vmw066109 ~]# ping -c 4 -M do -s 8972 9.79.20.1 PING 9.79.20.1 (9.79.20.1) 8972(9000) bytes of data.

8980 bytes from 9.79.20.1: icmp_seq=1 ttl=255 time=0.730 ms 8980 bytes from 9.79.20.1: icmp_seq=2 ttl=255 time=0.224 ms 8980 bytes from 9.79.20.1: icmp_seq=3 ttl=255 time=0.225 ms 8980 bytes from 9.79.20.1: icmp_seq=4 ttl=255 time=0.232 ms 9.79.20.1 ping statistics

---4 packets transmitted, ---4 received, 0% packet loss, time 3000ms < == rtt min/avg/max/mdev = 0.224/0.352/0.730/0.219 ms

< =========== >

(63)

< =========== >

CASTOR-VCS20-12# show interface port-channel 101 Port-channel 101 is up, line protocol is up Hardware is AGGREGATE, address is 0005.336b.a6ca Current address is 0005.336b.a6ca

Interface index (ifindex) is 671088741

Minimum number of links to bring Port-channel up is 1 MTU 9216 bytes

LineSpeed Actual : 20000 Mbit Allowed Member Speed : 10000 Mbit Priority Tag disable

IPv6 RA Guard disable

Last clearing of show interface counters: 00:15:14 Queueing strategy: fifo

Receive Statistics:

5523041 packets, 34803532143 bytes

Unicasts: 5518149, Multicasts: 4810, Broadcasts: 82

64-byte pkts: 714591, Over 64-byte pkts: 690914, Over 127-byte pkts: 42459 Over 255-byte pkts: 1468, Over 511-byte pkts: 5830, Over 1023-byte pkts: 2474 Over 1518-byte pkts(Jumbo): 4065305 < ==

Runts: 0, Jabbers: 0, CRC: 0, Overruns: 0 Errors: 0, Discards: 0

Transmit Statistics:

6238372 packets, 25262286680 bytes

Unicasts: 6235769, Multicasts: 1946, Broadcasts: 657 Underruns: 0

Errors: 0, Discards: 0

Rate info (interval 299 seconds):

Input 2.872640 Mbits/sec, 3491 packets/sec, 0.01% of line-rate Output 618.673012 Mbits/sec, 9358 packets/sec, 3.09% of line-rate Time since last interface status change: 00:22:28

CASTOR-VCS20-12# show interface port-channel 102 Port-channel 102 is up, line protocol is up Hardware is AGGREGATE, address is 0005.336b.a6c8 Current address is 0005.336b.a6c8

Receive Statistics:

5474289 packets, 34801617484 bytes

6328123 packets, 25934260554 bytes

LineSpeed Actual : 20000 Mbit Allowed Member Speed : 10000 Mbit

(64)

Priority Tag disable IPv6 RA Guard disable

Receive Statistics:

11401661 packets, 73071221978 bytes

14038276 packets, 69848610349 bytes

Receive Statistics:

11161421 packets, 63728202020 bytes

11233391 packets, 52506438558 bytes

TenGigabitEthernet 11/1/4 is up, line protocol is up (connected) Hardware is Ethernet, address is 0027.f81c.71d1

LineSpeed Actual : 10000 Mbit

LineSpeed Configured : Auto, Duplex: Full Priority Tag disable

Receive Statistics:

37860349 packets, 173659550086 bytes

64-byte pkts: 4341558, Over 64-byte pkts: 10638197, Over 127-byte pkts: 2055092 Over 255-byte pkts: 22, Over 511-byte pkts: 1, Over 1023-byte pkts: 1

Over 1518-byte pkts(Jumbo): 20825478 < == Runts: 0, Jabbers: 0, CRC: 0, Overruns: 0 Errors: 0, Discards: 0

33570288 packets, 206550446466 bytes

(65)

Input 1258.197448 Mbits/sec, 35288 packets/sec, 12.58% of line-rate Output 1694.879580 Mbits/sec, 31493 packets/sec, 16.95% of line-rate Time since last interface status change: 1d18h17m

1.2.2 NAS Bandwidth Validation

Test Objective

Validate maximum sustained bandwidth to storage port via CIFS/NFS. After 15 minutes, verify I/O completes error-free.

Results

PASS.All IO completes error-free.All host and storage ports show bandwidth under maximum capacity.

< =========== >

Receive Statistics:

36026612 packets, 53691133663 bytes

Over 1518-byte pkts(Jumbo): 0

19775385 packets, 1336954677 bytes

Input 352.092120 Mbits/sec, 29514 packets/sec, 1.76% of line-rate < == Output 8.860276 Mbits/sec, 16379 packets/sec, 0.04% of line-rate Time since last interface status change: 4d04h17m

Receive Statistics:

40826443 packets, 60899431647 bytes

Over 1518-byte pkts(Jumbo): 0

Runts: 0, Jabbers: 0, CRC: 0, Overruns: 0

Brocade VCS Fabric Technology with the NetApp 3170 NAS Storage Array

53-1003927-02

November 2015