How Virtualization Breaks Traditional Storage

How Virtualization Breaks Traditional Storage

Virtual computing has dramatically affected data center infrastructures. It enables companies to move away from

inflexible, inefficient infrastructure silos that are complex and brittle. Furthermore, according to Gartner, “the total cost

of ownership has become oppressive.” Virtualization is at the core of a new paradigm —the modern data center built

on a cloud architecture. The Hybrid Cloud is a set of shared resource pools of compute, network and storage that are

allocated on-demand to virtual machines through software. In effect, the infrastructure itself becomes programmable,

flexible and efficient, leading to greater agility, simpler management and significantly lower costs. Gartner believes these

benefits are so significant that as much as 50% of companies will adopt “Webscale IT” by 2017 forecasts.

To deliver the full benefits of the Hybrid Cloud, virtual infrastructure is generally deployed using networked shared storage (i.e. SAN, NAS). Valuable administrative utilities like live migration, failover and workload balancing require it. These solutions are based around storage architec-tures that were developed decades ago when the computing infrastructure was dominated by a one-to-one relationship between physical servers and their associated storage. When used in virtual computing infrastructure, these traditional solutions do not perform well.

Administrators unfamiliar with the characteristics of virtual environments are often surprised at the storage performance they get, and begin to add storage to improve performance. As storage is added, costs increase significantly. It is ironic that much of the savings from server consolidation are lost to the increase in storage costs necessary to create virtual computing solutions that meet production performance requirements.

Why does this happen in virtual environments? Basically, there is a fundamental mismatch between traditional storage architectures and what virtual infrastructure requires for optimum storage performance and efficiency. Virtual infrastructure essentially shifts the one-to-one architecture of traditional, physical computing to a many-to-one architecture that maps potentially hundreds of virtual machines onto a single storage controller, presenting a single LUN to the hypervisor. Traditional storage architectures can operate in these environ-ments, but suffer significant performance degradation—on the order of 40% to 60% and as much as 90% as the ratio of VMs to a LUN increases.

This poor performance is due to a phenomenon known as the “ I/O blender effect”. As admin-istrators add more storage to overcome the I/O blender effect to get back to desired perfor-mance levels, storage costs increase. Ultimately, administrators are often left with a choice between storage performance that does not meet specified requirements or unexpectedly high costs with storage configurations that are significantly over-provisioned in terms of capacity.

Virtual computing not only poses performance and cost issues for traditional storage, but also demands a management granularity that virtualization just can’t provide. Storage is typically managed per-LUN for things like snapshot, replication etc. With the many-to-one relationship between VMs and a LUN, a snapshot taken for a LUN may be a snapshot for hundreds of VMs—regardless if they wanted the snapshot or not—and if they did, is it the right time to take one? The lack of management granularity in traditional storage architec-tures makes these environments significantly more complex and error-prone to manage, driving operating costs even higher.

The I/O Blender Effect

Most server operating systems were written under the assumption that they would have direct access to dedicated storage devices. To maximize storage performance, these operat-ing systems optimize their I/O stream to write to disk sequentially. As compared with a very random I/O workload, a conventional spinning disk handling a sequential I/O workload will be able to perform up to 10x more IOPS, thereby driving much higher performance.

On the storage side, it could also optimize for a specific I/O pattern. It could coalesce, cache, or pre-fetch data according to a prevailing pattern.

Very Random I/O Patterns

In virtual environments, a single physical server may be hosting tens or hundreds of VMs, each of which is generating its own optimized I/O stream. These VMs no longer interact directly with the storage, they interact with the hypervisor, which interacts with the storage. The hypervisor multiplexes all the individual sequential I/O streams into an extremely random pattern, which it then writes to disk. The multiplexing of the I/O streams completely undoes any optimization performed by the operating systems of the individual VMs, resulting in an extremely random I/O pattern. And this random I/O pattern from many VMs makes it impossible for the storage controller to optimize for any specific pattern.

Random I/O can be significantly slower than sequential I/O. This is because of the additional time it takes to position the read/write heads to a new location to service the I/O request when the relevant blocks are not very close to each other on the disk platters. With a sequen-tial workload, the I/O is ordered in such a way as to minimize the rotational latencies and seek times for each I/O (i.e. the “next” I/O is relatively close to the “last” I/O). In random workloads, the rotational latency and seek time associated with each I/O is increased as the read/write heads hop back and forth across the platters to service the I/Os.

Very Write-Intensive I/O Patterns

I/O tends to be more write-intensive in virtual environments. This is due to the nature of virtual I/O as well as the use of various hypervisor utilities—like thin provisioning, snapshots and clones—that often require multiple writes to the storage for each write requested by an application. In physical server environments, a workload that exhibited 20% writes and 80% reads was considered write-intensive. In virtual server environments, workloads can easily be comprised of 40% to 50% writes, and in virtual desktop environments, writes can comprise as much as 90% of a steady state workload.

Traditional Storage

Optimized 1:1

Optimized I/O

Figure 1

Optimized I/O from OS

OS

APP

Capacity

Controller

Most types of storage handle writes more slowly than they do reads. With spinning disks, this is primarily due to head settling times which are not required on reads, only on writes. With flash memory-based storage like solid state disk (SSD), higher write latencies are due to additional administrative overhead that is not required for reads. A quick review of the technical specifica-tions for most storage devices will show this to be the case for enterprise-class storage.

The I/O Blender Effect Causes an IOPS Problem

This very random, very write-intensive I/O pattern is referred to as the “I/O blender effect” and it essentially increases the input/output operations per second (IOPS) required to service a given set of VM workloads by 40% to 60%. It is a general characteristic of how server hypervi-sors operate across all major platforms (Citrix, Microsoft, VMware). It means that it takes that much more raw IOPS in the storage to deliver a given level of performance from a VM’s point of view.

For individual spinning disks, IOPS are a function of the interface (e.g. FC, SAS, SATA) and the rotational speed (e.g. 7200 RPM, 10K RPM, 15K RPM). A typical SAS 15K RPM drive will generally be capable of delivering 180 IOPS across typical commercial workloads. Disk array vendors generally provide a maximum IOPS rating for their products, assuming a fully loaded array. Assuming a minimum array configuration, an enterprise would add spinning disks to an array to increase its performance (the amount of IOPS it can deliver).

If a given workload requires 15,000 IOPS to meet VM performance requirements in a virtual computing environment, then assuming only a 50% performance decrease due to the I/O blender effect an enterprise will need to provide an array with 30,000 IOPS. Assuming a price of $1,500 per SAS 15K RPM disk (a very aggressive street price for enterprise storage), it would take an additional 167 disk spindles to make up the 15,000 IOPS shortfall at a cost of $250,500 (assuming no additional cabinets need to be purchased). This is over and above the initial cost of the array to get it to 15,000 IOPS, and makes no assumptions about the additional energy and secondary storage costs required to support this larger storage configuration.

Note that adding disk spindles not only increases performance, but it also increases capacity. By the time a traditional storage configuration has been expanded to meet IOPS require-ments, it has way more storage capacity than is needed. In essence, storage capacity that was not needed had to be purchased just to meet performance requirements.

Virtual Computing Requires a VM-Centric

Management Model

Traditional Storage Uses a LUN-Centric Management Model

Traditionally, storage has been managed at the storage array level. Virtual storage objects were created and managed through array management GUIs that operated at the logical unit number (LUN1 _{level. Storage operations like snapshots, clones, replication and others}

were performed at the LUN level using the storage array management GUI. When an array was used by a single server, this management model worked fine.

The introduction of networked storage where a single array was shared between multiple servers began to break this model. Administrators generally want to perform storage operations on a given server. As soon as a LUN includes data from more than one server, a potential conflict arises. For example, for disaster recovery purposes an administrator may want to replicate Server A to a remote site, but not Server B. If Server A and Server B are both using LUN 1, there is no way to replicate Server A’s data without also replicating Server B’s. Managing at the LUN level does not provide sufficient management granularity to just replicate Server A’s data. The result was multiple LUNs had to be created so there was a 1:1 relationship between Server A - LUN A and Server B – LUN B. Now each could be managed cohesively from the storage arrays data services.

Virtual computing compounds this problem exponentially in two ways. First, storage administrators have generally wanted to limit the number of LUNs they configure in storage. This is because it is harder and more complex to manage a storage configuration with more LUNs and because storage arrays support a limited number of LUNs. In virtual computing environments with large numbers of VMs, an administrator would often not have enough LUNs to assign one LUN to each VM.

Second and more importantly, much of the flexibility gained with virtualization comes from the clustered file system within the hypervisor. VMs are simply represented as files within this file system. The clustered file system is accessible by all hypervisor servers where each can access all files within the clustered file system. This capability allows critical features such as HA or the ability to live-migrate a VM to a different physical server to load balance or for better performance. To make this work, all physical hosts must access the same LUN, and each VM across the virtualized data center lives here as a file on that LUN. A given LUN can easily be supporting thousands of VMs running across multiple hosts.

In virtual computing environments, administrators would prefer to still be able to perform storage data services at the VM level, but LUN level management does not come close to meeting this need. It in fact leads to very inefficient use of both storage capacity and network bandwidth. To demonstrate this, let’s take a look at creating a snapshot that an administrator wants to replicate.

Snapshots are often used to create point-in-time copies for recovery purposes. These point-in-time copies could be used to revert a VM back to a known good state after a patch update caused unexpected problems, they could be used to offload backup to a

VM.1 VM.2 VM.3 VM.n Hypervisor I/O Blender Server Storage Array VM.1 VM.2 VM.3 LUN

Traditional Storage Architecture

Control Plane Data Services Data Plane (I/O) Capacity

Storage Control Per LUN

Figure 3 Many VMs to one LUN

VM.1 VM.2 VM.3 VM.n

Server

Hypervisor

VM-Centric View: Manage snapshots, replication per VM

LUN-Centric View: Manage snapshots, replication per LUN (all VMs)

One-to-many data services

Clustered file system

Storage Array VM.1 VM.2 VM.3

VM.1 VM.2 VM.3 VM.n VM.1 VM.2 VM.3

Traditional Storage Architecture

Control Plane

Data Services (Snapshot, Replicate...) Data Plane (I/O)

LUN-1

LUN

Storage Control Per LUN

Figure 4

LUN-centric management model

VM.1 VM.2 VM.3 VM.n VM.1 VM.2 VM.3

dedicated backup appliance, and they could be used to provide recovery options to meet high availability and disaster recovery requirements. An administrator will not necessarily always want to perform snapshots on the same groups of VMs. What they need is the management granularity to perform snapshots on individual VMs or dynamically created groups of VMs that will vary over time.

A LUN could easily be hosting data for ten, twenty or hundreds of VMs. If the administrator specifically just wants to snapshot VM 5, LUN-level storage operations could not do this. It could only snapshot the LUN that VM 5 is on, along with all the other VMs on that LUN. Storing that snapshot would require additional capacity to store VMs that the administrator does not care about just to store that one VM. If the administrator then wanted to replicate that snapshot to a remote location, they would have to replicate all the VMs, not just the one VM. This would take up additional network bandwidth which for many enterprises is precious. What the administrator needs is a VM-centric management model for storage operations. They need to be able to select individual VMs and perform various storage operations on them without consuming storage capacity or network bandwidth to also perform those operations on other unwanted VMs. A VM-centric management model is much more intuitive than a LUN-centric model in virtual computing environments where the data from many VMs resides on each LUN.

VM-Centric LUN-Centric

■

■ Easier, more intuititive ■

■ Manages at VM level (more efficient) ■

■ Centralized management ■

■ Works with any storage

■

■ Harder to manage ■

■ Manges at LUN level (less efficient) ■

■ Fragmented management ■

■ Storage array-specific Figure 5 A VM-centric management model is much better suited to virtual computing environments than a LUN-centric one.

Fragmented Management Model

Management granularity is not the only problem traditional storage architectures impose on virtual computing environments. In Microsoft environments, all aspects of a VM can be man-aged through Microsoft System Center tools like Hyper-V Manager, Virtual Machine Manager and others. This provides an intuitive, easy-to-use management model that is centralized and allows storage operations like thin provisioning, snapshots, clones, replication, and failover to be defined at the level of individual VMs.

LUN-level storage operations are, however, managed through the array’s management GUI. So if an administrator does decide to use array-based storage management in virtual environ-ments, they will be introducing a second management GUI that fragments the management model, making it less intuitive and more difficult to administer.

High Availability and Disaster Recovery

Most commercial production workloads require high availability and disaster recovery, and this is true whether they are running virtual servers or virtual desktops. In the physical world, if a server failed, only its workload was affected. In the virtual world, a physical host may support tens or hundreds of VMs, all of which fail when it fails. For this reason, providing for fast, reliable recovery is generally a part of all virtual computing projects.

High availability is much easier and more cost-effective to configure in virtual infrastructures. Each VM is basically just a file which can potentially be run on any host. In the physical world, failover often required that two hosts—a primary and a standby—be configured exactly the same. In the virtual world, these constraints are significantly relaxed, creating far less configuration risk and cost when deploying high availability configurations.

Shared storage configurations keep a VM’s data in a location where it can potentially be accessed by a number of different hosts (although not at the same time). If a given host fails, its VMs can be re-started on other hosts, re-connected to their data on the shared storage, and their applications can be up and running with no data loss very quickly. Most server hypervisors provide native utilities to detect failures and manage recoveries automat-ically without human intervention. On Microsoft Hyper-V, failover clusters provide this capability.

When creating storage configurations in virtual environments, high availability is a strategic consideration which must be considered along with performance (IOPS) and capacity re-quirements. Meeting performance requirements is primarily an IOPS problem, and there are host-based storage options like flash memory cards and PCIe SSDs that provide a significant amount of IOPS. Why couldn’t an administrator just purchase one of these devices for each host to provide the IOPS applications need?

If there is no requirement for high availability, then administrators potentially can take this approach to address storage performance in these environments. But if any kind of failover is required, data on these devices is not accessible when their hosts are down, and will not be accessible until their host can be repaired and brought back online. An application could be re-started on another node, but it would not have access to its data (which resides in a failed host and is not accessible). Failover products are designed to detect failures within seconds, and bring applications back up, along with their most recent data, within just a minute or two in virtual environments. But to do so, they must have access to an application’s data.

Before a storage solution can be chosen, an administrator must understand not only an envi-ronment’s performance and capacity requirements, but also its high availability requirements. This is why local storage options are often not viable in virtual infrastructure. The real trick is to cost-effectively meet both performance and capacity requirements, while at the same time providing for high availability.

Clustered File System on Traditional Storage Architecture

Figure 6 Share storage required for HA

VM.n Server 2

Hypervisor

Clustered file system

VM.1 VM.2 VM.3

LUN-1 VM.n

Server 1

Hypervisor

Clustered file system

Business Implications

It should be clear by now that virtual computing environments have significantly greater IOPS requirements than most physical computing environments, and that traditional storage architectures provide a management model that does play well in the many-to-one architec-ture around which virtual computing is built. The heightened high availability requirements of production virtual infrastructure introduce additional constraints that need to be taken into account. Because storage performance is generally 40% to 60% lower in these environments, there is a significant shortfall that has to be addressed, but it has to be addressed with storage that can support features like live migration, failover and workload balancing. It should also be clear why we stated early on that most storage solutions in virtual environ-ments suffer from some combination of poor performance, unexpectedly high costs, and complex storage management. Sizing a storage configuration to meet performance require-ments based on experience in physical server environrequire-ments will result in poor performance. As additional disk spindles are added to increase IOPS, administrators are also adding capacity along with cost. By the time the IOPS shortfall has been addressed, configurations have way more physical capacity than is required. As that capacity is not used because it is not needed, it is in effect wasted.

If high availability requirements exist, local SSDs or other host-based flash products may not be able to entirely address the IOPS shortfall. Host-based storage could be used to improve read performance, but virtual I/O patterns tend to be very write-intensive. SSDs could be purchased for use in the SAN but because the workloads are very write-intensive, these devices must be deployed using a write-back (not a write-through) caching scheme to provide meaningful performance improvements while still supporting failover without data loss. Caching schemes targeted at primarily improving read performance will probably not meet performance requirements for virtual computing’s very write-intensive workloads.

Lowering the VM density on each host could be used to minimize the I/O blender effect, but this will require additional hosts, which will increase both capital and operating costs, which drives up the cost per virtual server.

Summary

Virtualization has changed everything—and it has broken the traditional storage model. The traditional storage model designed almost two decades before virtualization is broken in the context of the virtualized environment.

What made compute virtualization so successful was its transparent insertion into the infrastructure stack. Inserting the hypervisor above the physical hardware and below operat-ing systems meant that applications, OS’s and hardware did not realize anythoperat-ing changed. The success of virtualization depended on this transparent insertion into the current infra-structure stack. In effect, neither the hardware or operating systems or applications knew anything changed. Compute resources were instantly turned into software giving a virtual machine complete freedom of movement from one physical machine to another.

Poor Storage Performance

Increasing Storage Cost

t US 855.786.7065 US 650.316.5515 UK +44(0)20 3553 3662 e [email protected] www.gridstore.com

FOLLOW US /gridstore /company/gridstore /user/GridstoreInc

© 2014 Gridstore. All rights reserved. Gridstore, the Gridstore logo, AutoPilot, Direct I/O, FlashGrid, GridControl, GridProtect, GridScale, Server-side Virtual Controller (SVC), TrueQoS, vController, vLUN, vmOptimized, vPool, and vStore are registered, trademarks or pending trademarks of Gridstore in the U.S. and other countries. All other trademarks are the property of their respective owners. Information regarding products, services and offerings may be superseded by subsequent documents and are subject to change without notice. For the latest information and specifications regard ing Gridstore and any of its offerings or services, please visit www.gridstore.com. 081814

Server virtualization of compute resources has created enormous economies of scale, efficiency and flexibility in operating compute infrastructures. While the magic of virtualization is the appearance to operating systems and applications that nothing is changed, the reality is that everything has changed around it—and this is the Achilles’ heel of virtualization. Server virtualization has in effect changed the infrastructure stack and as a result placed enormous strain on other parts of the infrastructure—namely network and storage. This paper outlined the impact of this architectural mismatch between virtual compute and traditional storage. To address this architectural mismatch, both the network and storage need to be virtualized in a way that matches the architecture of server virtualization. The full promise and value of virtualization will not be realized until both the network and storage undergo the same transformation and the entire infrastructure stack is aligned. Until these architectures align, the savings and efficiency created with server virtualization will simply transfer into increased storage spending and increased operating costs due to the management complexity of a fragmented infrastructure model.

Server hypervisors effectively deliver “software-defined compute”, allowing CPU and memory resources to be defined and allocated as needed without being subject to any physical constraints. Software-defined storage would do the same for storage resources, allowing them to be defined, allocated and optimized on a VM by VM basis.

And this is where the real challenges start. There is one fundamental difference between compute and storage—compute is transient and storage is persistent. In fact, the physical server that was virtualized now physically exists in storage.

Because data is physical and storage is effectively the base of the infrastructure stack, it is also the most difficult to change. Retrofitting an existing storage stack is almost impossible. Simply adding layers of complexity on top of what is already a complex environment just makes the problem worse and more fragile. To make the transition to software-defined storage—storage that aligns directly to the virtualized infrastructure stack—storage needs to be designed from the ground up for this new architecture.