Most medium and large sized IT organiza ons have deployed several genera on of virtualized servers, becoming more comfortable with the performance and reliability with each deployment. As IT organiza ons started to increase VM density, they hit the limits of Hyper-V so ware and server memory, CPU, and I/O. A new VM Engine is now available and this documents describes how it can help IT organiza ons maximize use of their servers running Hyper-V in Windows Server 2012.
Best Prac ces in Hyper-V…………...………..………...………..…3
Power Needed for More VMs………..……….4
A New VM Engine………...………..………..…………...5
A New VM Chassis ………..………....6
Virtualized I/O………....……….………...………..7
Racing Exhaust Systems: 16Gb FC, 10GbE & Converged Networking………..………….…..…..8
New Hyper-V Performance ……….………..………..……….9
Performance with 16Gb Fibre Channel………..………...………….………...10
Live VM Migra on…..……….……….…………..……….11
Live Storage Migra on……….…………..……….………..……….………..12
Lowering the Cost of VM I/O………...………..………13
Low-Latency Connec vity……….……...…14
Scalability with 16Gb Fibre Channel……….……….15
Accelera ng App Performance………..………...…16
The Bo om Line……….………..……….….17
Resources……….………..……….………18
Harnessing the Power
In a survey conducted by IT Brand, IT professionals said the
average number of VMs per server would almost double in the
next 24 months.
VMs Per
Server
Best Prac ces in Hyper-V
No hardware resource is more important to overall performance
than memory. Plan to ensure each VM has the memory it needs,
but without was ng memory in the process.
Memory
Start with Planning
When planning a Hyper-V installa on, it is important to take into account the new capabili es of Hyper-V in Windows Server 2012. Windows Server 2012 has added significantly to the scalability of Hyper-V. For datacenters virtualizing Tier-1 applica ons, the cri cal scalability enhancement is the ability to have up to 1TB of memory and 64 virtual CPU cores per VM. This will ensure almost all Tier-1 applica ons should perform well in a Microso Hyper-V environment.
However, these new capabili es bring new complexi es, and with them the need to plan new datacenter architectures. This not only includes planning the deployment for today’s needs, but also thoroughly inves ga ng evolu on strategies for applica ons before bol ng down racks and filling them with servers.
Planning which applica ons are going to run on your virtualized servers is the first step in understanding your needs.
From there, it is cri cal to define server integra on points with exis ng resources (likely core switching and storage resources), and how these will be affected by the evolu on of exis ng resources.
A er that, planning your approach to Live Migra on and capacity growth over the life me of your new infrastructure will help you scope internal I/O requirements appropriately.
Finally, determining whether to u lize converged networks or not, and what I/O performance you need, will enable you to intelligently discuss your I/O and networking op ons with your SAN/LAN equipment providers. These steps will help you ensure success when virtualizing your Tier-1 applica ons.
To fully op mize virtualized data centers, servers need maximum I/O capacity to support high input output opera on rates and high bandwidth applica ons. Increased bandwidth is also needed for server
virtualiza on, which aggregates I/O from mul ple virtual machines (VMs) to the host’s data path. This next-genera on combina on takes full advantage of new features that are described in detail in this planning guide. Read on to discover how QLogic can increase your infrastructure ROI and overall compe veness.
The Great VM Migra on
With many server admins working on their 3rd and 4th genera on of virtualized servers, the focus has changed from interoperability and learning the behavior of Hyper-V, to increasing VM density (#VMs/Physical Host Server). With the availability of servers based on Intel’s E5 processors (mul -core, 768GB of RAM, PCI Express Gen3) and the
combina on of new features within Hyper-V, a new, game-changing compute pla orm was introduced. This new pla orm allows for new levels of VM density and for the first me Tier-1 applica ons that previously required dedicated server hardware can now run on virtual servers, achieving improved performance, scalability and efficiency.
While Hyper-V and E5-based servers are seeing significant deployments in many enterprise
datacenters, the I/O and network infrastructure to support these new technologies lags far behind. In a survey conducted by IT Brand Pulse, IT professionals said the average number of VMs per server would almost double in the next 24 months. Approximately 25% of IT professionals surveyed also said what they need most to increase the VM density is more I/O bandwidth. The purpose of this industry brief is to provide a planning guide to help enterprises deploy Tier-1 applica ons with adequate bandwidth in a dense Microso Hyper-V Server 2012 virtualiza on environment.
Power Needed for more VMs
Approximately 25% of IT professionals surveyed said what they
need most to increase the density of VMs per server is more I/O
bandwidth.
VM
Density
IT Brand Pulse
The average number of VMs per server in
my environment:
What I need most to increase the density of
VMs per physical servers is more:
A New VM Engine
2 more cores, 8MB more of cache, 6 more DIMMs of faster
DDR3-1600 memory increasing to 768GB, double the I/O bandwidth with
PCIe 3.0, and more Intel QuickPath links between processors.
Xeon E5
Xeon E5 offers double the I/O bandwidth to 10GbE NICs, 10GbE/16Gb FC CNAs and 16Gb FC HBAs.
The Intel Xeon E5 Pla orm
The introduc on of the Intel® Xeon® E5 family of processors responds to the call for more virtual server re-sources with 2 more cores, 8MB more of cache, 6 more DIMMs of faster DDR3-1600 memory. Increasing the total to 8 cores, 768GB of RAM, and doubling the I/O bandwidth with PCIe 3.0.
Intel’s new Xeon E5 promises a significant increase in server I/O by enabling full- bandwidth, four-port 10GbE server adapters as well as dual-port 16Gb FC server adapter support, addressing the VM density issue with a substan al increase in I/O bandwidth that host servers require.
Windows Server 2012 Hyper-V
The newest release of Windows Server 2012 Hyper-V delivers a high-performance VM chassis harnessing the new I/O
capabili es of 16Gb Fibre Channel (FC) Storage Networking and
10GbE Data Networking. Several new features of Windows Server 2012 Hyper-V are highlighted below: vCPU—Virtual machines can now have up to 64 virtual CPUs (vCPUs) and 1TB of virtual RAM (vRAM) allowing
Tier-1 applica on to be virtualized and new levels of VM density to be reached.
Virtual Fibre Channel (Virtual FC)—Hyper-V now enables VM workloads to access FC SANs by provisioning virtual FC ports with a standard Worldwide Name (WWN) within the guest OS.
Live Migra on—Virtual FC also enables Live Migra on of VMs across Hyper-V hosts while maintaining FC connec vity. Two WWNs are configured and maintained for each virtual FC adapter.
Live Storage Migra on—A VM’s Virtual Hard Disk (VHDX) storage can now be migrated without shu ng down the VM. The opera on copies data from source storage device to a target via a FC or similar interconnect.
Mul path I/O (MPIO)— Hyper-V now extends MPIO capability to VMs ensuring fault-tolerant FC connec vity for delivering High Availability and Resiliency to virtualized workloads
16Gb Fibre Channel—To help maximize efficiency of Live Migra on and Storage Mo on, Hyper-V includes support for 16Gb Fibre Channel, the fastest storage interconnect available today.
10GbE and SR-IOV— Allows 10GbE NIC to appear as mul ple virtual devices that can op mize I/O performance by providing direct I/O for individual virtual machines.
From a storage planning perspec ve, when comparing Windows Server 2012 to previous version, there are two specifica ons which stand-out: the amount of memory/VM (1TB), and the amount of ac ve VMs per machine (1,024).
Using today’s storage usage, a petabyte of storage could be needed to support 1024 VMs. While this scenario is unlikely for at least a few years, running 100 VMs with 512GB of virtual memory each on a single server (which would require 52TB of storage for the memory contents alone) is very foreseeable.
The ability to provide high-performance storage is cri cal for a high-density or Tier-1 virtualiza on strategy. The new storage tools in Hyper-V that we will cover later in this paper (virtual Fibre Channel, offloaded data transfer, and the new virtual hard disk format) can posi vely impact performance in these environments.
A New VM Chassis
In Hyper-V, Virtual Fibre Channel SAN connec vity on a
per VM basis.
Virtual
Picking the Right I/O Pieces, and Making Them Work Together
Tier-1 applica ons are uniquely demanding in many dimensions. Their needs with respect to CPU power, memory footprint, high availability/failover, resiliency and responsiveness to outside s muli is typically unmatched within the enterprise. Moreover, Tier-1 applica ons also tend to be ghtly integrated with other applica ons and resources within the enterprise. Because of this, virtualizing a Tier-1 applica on requires rigorous planning of the I/O strategy. There are five steps to this:
Iden fy the I/O fabrics that the Tier-1 applica ons will use (it may very well be “all of them”).
Quan fy the data flows for each fabric when the applica on was opera ng on a standalone system.
Es mate Live Migra on I/O needs for failovers and evolu on. Note that most Live Migra on traffic will be storage I/O; if the data stays within one external array during the Live Migra on, Microso ’s ODX capability can significantly reduce the I/O traffic.
Determine your primary and secondary I/O paths for mul -pathing on all of your networks.
Determine QoS levels for the Tier-1 apps.
One simplifying op on available is to u lize converged networking adapters that can func on on both FC and Ethernet/ FCoE networks. The QLogic QLE2672 is an example of such an adapter; it can be reconfigured in the field to operate on 16Gb FC or 10Gb FCoE/ Ethernet networks.
Virtualized I/O
Business Cri cal Applica ons such as ERP, CRM, eCommerce
and Email need high-performance and high availability I/O
infrastructure to meet business SLAs.
Tier-1 Apps
QLogic Server Adapters
Windows Server 2012 Hyper-V and Xeon E5 streamline the process of moving VMs and their associated storage with Live Migra on and Live Storage Migra on and support low-latency networking traffic with SR-IOV. Moving terabytes of data across the virtual machines and migra on virtual servers requires low-latency, high performance I/O adapters. QLogic offers a family of server adapters for 16Gb Fibre Channel , 10GbE or converged network connec vity provide the bandwidth for increased Virtual Machine (VM) scalability and to power Tier-1 applica on workloads.
Racing Exhaust Systems
The latest genera on of CNAs from QLogic support Ethernet LANs, NAS,
iSCSI SANs and FCoE SANs, as well as na ve Fibre Channel SANs
CNA
QLogic Server
Adapters
2600
Series
3400
Series
8400
Series
Server Adapter Descrip on Speed 16Gbps 10Gbps 10GbpsProtocols FC TCP/IP LAN and NAS, iSCSI SAN
TCP/IP LAN and NAS, iSCSI SAN, FCoE SAN
Use in virtualized server
Highest performance Fibre Channel SAN connec vity for storage intensive applica ons
Consolidate mul ple 1GbE server connec ons to LAN and NAS on one high-speed Ethernet wire
Consolidate server connec ons to LAN and SAN on one Ethernet wire
The size of a dynamically expanding VHD is as large as the data that is wri en to it. As more data is wri en to a dynamically expanding VHD, the file increases to a maximum size. A differencing VHD is similar to a dynamically expanding VHD, but it contains only the modified disk blocks of the associated parent VHD. Dynamically expanding VHDs are useful for tes ng environments because there is less impact if you have to rebuild the VHD. For example, some of the tests performed for this report used mul ple dynamically expanding VHDs, each with a different Windows image. Fixed VHDs are recommended for produc on.
New Hyper-V Performance
VHDX
A new Hyper-V virtual hard disk (VHD) format introduced with Windows
Server 2012 which increases storage capacity from 2TB to 64 TB
25% More Throughput with Windows Server 2012 Hyper-V
One of the most important features in Microso Windows 2012 Hyper-V for I/O performance is the VHDX virtual hard disk format, which provides storage for the guest OS. Tes ng by Microso shows that VHDX delivers nearly 25% be er write throughput than VHD for both dynamically expanding and differencing disks.
Tes ng with Iometer showed the performance of compe ve products was iden cal at real world 4k, 8k and 16k block sizes. However, the QLogic QLE2672 used up to 23% less CPU processing power to do the same work.
Performance with 16Gb FC
QLE2672
Tes ng by QLogic shows the QLE2672 16Gb Fibre Channel adapter
delivers high IOPS with Microso Windows Server 2012 with
signi
fi
cantly less CPU u liza on than compe
ve products.
Real World Performance and CPU E
ffi
ciency
When used with a high-efficiency 16Gb Fibre Channel adapter, Hyper-V with VHDX can provide even larger performance advantages. In tes ng performed by QLogic, the QLogic QLE2672 16Gb FC adapter delivered the same IOPS performance using real-world 4KB & 8KB block sizes for dual-port adapters as the nearest compe tor—with 23% less CPU u liza on. This frees CPU cycles for virtual machines and their workloads which is cri cal for Tier-1 applica ons or in dense VM environments.
CPU % - Dual Port, 100% Reads
IOPS - Dual Port, 100% Reads
ODX Data Copy Model O
ffl
oads Server
Microso ’s Live Migra on offers the ability to move a live virtual machine from one physical server to another. Included are some interes ng storage-oriented capabili es that provide added value if you use SANs (especially Fibre Channel SANs). The first is the ODX Data Copy Model. On compa ble external storage arrays, ODX provides the ability to move data between LUNs without involving the server. For large data movements, this results in a huge
performance improvement.
Improved Live Migra on with vFC
Hyper-V’s Virtual Fibre Channel (vFC) is a new capability that augments Live Migra on for those end users with Fibre Channel SANs. By crea ng virtual FC HBAs na vely within Hyper-V, Microso simplifies migra ons by moving the adapter with the virtual machine. This eliminates the need to re-configure network switches a er a Live Migra on.
Microso Hyper-V’s powerful migra on
capabili es can provide even more u lity if likely failover and evolu on paths in the private cloud are planned into the framework. This is
especially true for migra ons to resolve
hardware failures, which tend to be done under considerable stress. Planning failover
migra ons decreases the likelihood of nega ve performance impacts that may ul mately have to be undone later.
Live VM Migra on
Windows Offloaded Data Transfer (ODX) in Windows Server 2012 directly
transfers data within or between compa ble storage devices, bypassing
the host computer.
ODX
This diagram shows a Live Migra on u lizing vFC. The 2-port virtual HBA ping-pongs from the first port to the second port, avoiding traffic disrup on during the live migra on.
Tradi onal Data Copy Model ODX Data Copy Model 1. A user copies or moves a file or this occurs as part of a
virtual machine migra on.
2. Windows Server 2012 translates this transfer request into an ODX and creates a token represen ng the data. 3. The token is copied between the source server and
des-na on server.
4. The token is delivered to the storage array.
5. The storage array internally performs the copy or move and provides status informa on to the user.
16Gb Fibre Channel Helps Close the Storage Migra on Window
A process known as Live Storage Migra on allows for a non-disrup ve migra on of a running VM disk files between to different physical storage devices. This process allows for the virtual machine to remain running with no need to take its workload offline to move the VM’s files to a different physical storage device. Addi onal use cases for Live Storage Migra on include migra on of data to new storage arrays or larger capacity, be er performing LUNs. NPIV zoning and LUN masking must be properly configured to ensure the VM and host server con nue to have access to the storage a er the migra on is completed. Live Storage Migra on across a 16Gb Fibre Channel link can finish in half the me it takes a 8Gb Fibre Channel link. All paths related to Live Storage Migra on should be supported by high performance networks in order to reduce the me it takes to evacuate storage safely to a new des na on and resume normal opera ons. Addi onally, 10GbE links can replace 1GbE links to ensure proper bandwidth exists for Live Storage Migra on in Ethernet environments.
Live Storage Migra on
A single port QLE2670 16Gb Fibre Channel HBA doubles throughput for a live storage migra on.
Storage Live Migra on at 16Gbps
This bandwidth intensive opera on now enables virtual
machines and associated VHDX
fi
les to be migrated between
clusters that do not have a common set of storage.
Live Storage
Migra on
QLE2670 16Gb Fibre Channel HBAs QLE3240 10Gb Intelligent Ethernet Adapters8400 Series CNAs Enable Convergence at the VM Edge
QLogic Converged Network Adapter solutions leverage core technologies and expertise including the most established and proven driver stack in the industry. These adapters are
designed for next-generation, virtualized, and unified data centers with powerful multiprocessor, multicore servers. Optimized to handle large numbers of virtual machines and support for VM aware network services with support for concurrent NIC, FCoE, and iSCSI traffic.
One 8400 series CNA can be configured for connectivity to an
Ethernet network and to deliver storage networking via Fibre Channel over Ethernet simultaneously. Powerful iSCSI and FCoE hardware offloads improve system performance and advanced virtualization
technologies are supported through secure SR-IOV or switch and OS agnostic NIC Partitioning (NPAR). VXLAN and NVGRE tunneling offloads break through virtual LAN scaling limitations while securing multitenant connectivity across disparate data centers
Lowering the Cost of VM I/O
The Fibre Channel over Ethernet (FCoE) protocol allows Fibre Channel
traffic to run over Data Center Ethernet (DCE) for LAN and SAN
convergence on one wire.
FCoE
For organiza ons maintaining a parallel LAN and SAN architecture all the way to the server adapter, QLogic offers the QLE8400 Series of adapters suppor ng 10GbE LAN, NAS and iSCSI SAN traffic, as well as Fibre Channel traffic.
Network Convergence at the VM Server
Adapter & Fabric Convergence Adapter Convergence, Separate Fabrics
Both ports used for LAN, NAS and SAN traffic over Ethernet
Ethernet ToR Switch Ethernet ToR Switch FCoE ToR Switch
1 port used as FCoE CNA
8400 Series CNAs O
ffl
oad the VM Kernel from Switching Virtual NICs
Single Root I/O Virtualiza on is a standard that allows one PCI Express (PCIe) adapter to be presented as mul ple separate logical devices to virtual machines for par oning adapter bandwidth. The hypervisor manages the Physical Func on (PF) while the Virtual Func ons (VFs) are exposed to the virtual machines. In the hypervisor, SR-IOV capable network devices offer the benefits of direct I/O, which includes reduced latency and reduced host CPU u liza on. With SR-IOV, pass through func onality can be provided from a single adapter to mul ple virtual machines through Virtual Func ons. To deploy SR-IOV today, an
organiza on needs to ensure a minimum level of infrastructure (server hardware and OS) support for SR-IOV. In contrast, QLogic NPAR technology can similarly be used today without the minimum levels of
dependencies of SR-IOV.
Low-Latency Connec vity
Latency is the me between the start and comple on of one ac on
measured in microseconds (µs) .
Latency
With SR-IOV enabled on a 10GbE NIC, pass through func onality can be provided from a single adapter to mul ple virtual machines through Virtual Func ons (VFs).
Implemen ng Pass-Through Func ons with SR-IOV
8400 Series Converged Network Adapters
In Transac on Intensive and Bandwidth Intensive Environments
For virtualized environments, the most cri cal measure of performance is the ability to scale as the number of VMs and applica on workloads increase. In tes ng conducted by QLogic, the QLogic QLE2672 delivered three mes the transac ons and double the bandwidth of 8Gb Fibre Channel Adapters. The QLE2672 also demonstrated a 50% advantage over compe ve products for read-only performance and 25% be er mixed-read-write performance. The superior performance of QLogic 16Gb Fibre Channel Adapters translates to support for higher VM density and support for more demanding Tier-1 applica ons.
QLogic achieves superior performance by leveraging the advanced 16Gb Fibre Channel and PCIe® Gen3 specifica ons—while maintaining backwards compa bility with exis ng Fibre Channel networks. The unique port-isola on architecture of the QLogic FlexSuite adapters ensures data integrity, security and determinis c scalable performance to drive storage traffic at line rate across all ports. QoS enables IT teams to control and priori ze traffic. And paired with adapter par oning technology the QLE2672 can deliver capacity on
demand and mul tenant feature requirements of highly virtualized environments.
More Virtual CPUs for scaling Tier-1 Apps
If you’re concerned about hos ng er-1 apps on VMs, the argument about virtualizing er-1 apps is over. Even flagship enterprise applica ons such as
Microso SQL Server 2012 and Exchange 2010 have adopted server virtualiza on as a best prac ce. In fact the CPU, memory, storage and networking requirements are well documented by
In the example on the right, a mission-cri cal OLTP Workload running on a single SQL Server 2012 VM demonstrates linearly increasing transac onal
performance and reduced transac on response mes as the number of virtual CPUs assigned to the
workload are increased to the maximum of 64 now supported on Hyper-V.
Scalability with 16Gb FC
Throughput
Bandwidth refers to the maximum poten al volume. Throughput
is the actual volume. Both are measured as the amount of data
transferred in a given me or megabytes per second (MBps).
The number of transac ons processed per second and the average response me were monitored as virtual CPUs were increased from 4-64. The OLTP workload and concurrent user counts remained constant.
Hyper-V Virtual CPU Scalability
With OLTP Workloads
10000 Series FabricCache Adapters Cache Hot VM Data
The 10000 Series is the industry's first caching SAN adapter. This new class of server-based PCIe SSD/Fibre Channel HBAs uses the Fibre Channel network to cache and share SAN metadata. Adding large caches to servers places the cache closest to the applica on and in a posi on where it is insensi ve to conges on.
An advantage to this approach is that PCIe flash based caching can be shared and replicated in different servers for high-availability and for cache coherency across
migra ng servers in a virtual machine cluster. With the FabricCache architecture, the new genera on of PCIe SSDs provide redundancy and fail-over for a new level of enterprise-class availability.
Accelera ng App Performance
A QLogic architecture for sharing and replica ng cache on a PCIe
SSD adapter in a SAN.
FabricCache
The lightning fast SSD SLC flash from the 10000 Series FabricCache adapters is used to cache hot data stored on a FC SAN array. For high availability, the cache LUNs from a FabricCache adapter in one server can fail-over to a FabricCache adapter in another server and can also be used for cache coherency across migra ng servers in a virtual machine cluster.
The Bo om Line
The improvement factor for Memory per VM for Windows Server 2012
Hyper-V — addressing the biggest issue in scaling VMs.
16X
More VMs with Hyper-V, Xeon E5 and QLogic Server Adapters
Fabric-based networks are a fundamental requirement in suppor ng highly virtualized data centers. Fibre Channel SANs are the nucleus of the next-genera on Windows Server 2012 data center. If your goal is to increase VM density, Windows Server 2012 Hyper-V, combined with the latest genera on of servers based on Intel Xeon E5 processors, and QLogic server adapters allow you to more than double the number of VMs per server while enjoying the same level of performance. Virtualiza on features like Microso vFC and Fibre Channel QoS from QLogic combine to deliver reliability, performance, and the flexibility necessary to manage the complexity and risks associated with virtualiza on projects. Choosing to virtualize er-1 data center applica ons or increase virtualiza on densi es with QLogic and Hyper-V will enable your businesses to leverage the built-in architecture of both products to increase availability, improve agility, and overcome scalability and performance concerns.
Hyper-V delivers improvements on all key virtualiza on metrics—making I/O performance cri cal.
Windows Server
2008 R2 Hyper-V 2012 Hyper-V
Factor
Host
HW Logical Processors
64 LPs
320 LPs
5x
Physical Memory
1 TB
4 TB
4x
Virtual CPUs per Host
512
2048
4x
VM
Virtual CPUs per VM
4
64
16x
Memory per VM
64GB
1TB
16x
Ac ve VMs per Host
384
1024
2.7x
Guest NUMA
No
Yes
-
Cluster
Max Nodes
16
64
4x
Related Links
About the Authors
Rahul Shah, Director, IT Brand Pulse Labs
Rahul Shah has over 20 years of experience in senior engineering and product management posi ons with semiconductor, storage networking and IP networking manufacturers including QLogic and Lantronics. At IT Brand Pulse, Rahul is responsible for managing the delivery of technical services ranging from hands-on tes ng to product launch collateral. You can contact
Rahul at
Tim Lus g, Director of Corporate Marke ng, QLogic Corpora on
With over 18 years of experience in the storage networking industry, Lus g has authored numerous papers and ar cles on all aspects of IT storage, and has been a featured speaker at many industry conferences on a global basis. As the Director of Corporate Marke ng at QLogic, Lus g is responsible for corporate communica ons, , third party tes ng/valida on, outbound marke ng ac vi es and strategic product marke ng direc ves of QLogic. His responsibili es include customer research, evalua on of market condi ons, press and Media rela ons, social media and technical wri ng.
