Coping with SAN Storage Frustration Caused by
Server Virtualization
Advanced Storage Products Group
Page 1 of 8
Table of Contents
1 – Introduction
2 – Server Virtualization & Networked Storage Issues 6 – NEC’s D-Series Storage 8 – Conclusion
Introduction
Server virtualization is clearly one of the breakout technologies in the first decade of the 21st century. The unambiguous value of virtualizing servers comes from the measurable increase in hardware utilization and superior application deployment flexibility, which in turn facilitate highly cost effective server consolidation. The results are both operationally and
economically compelling, however, they are not without their pitfalls as server virtualization — unless properly managed — can result in difficulties with SAN storage.
Server virtualization provides highly desirable availability capabilities such as transparent live migration of applications without disruption, live migration of storage volumes without disruption, near instantaneous recovery of down machines at a local or remote site, on-demand allocation of hardware resources based on QoS policies per virtualized guest, and non-stop availability in the event of a hardware failure. These capabilities along with even basic hypervisor functionality necessitate networked storage for hypervisor functionality and ease of management. However, this is where the frustration begins.
Server virtualization is incredibly easy: creating virtual server guests is as simple as point, click, and create. Networked storage for virtualized servers however is anything but easy for most systems. Key challenges for storage admins include:
• Dealing with excessive storage network oversubscription;
• Effectively managing virtual server performance;
• Reducing virtual server scheduled downtime as a result of provisioning and volume expansion;
• Deciding convoluted storage tradeoffs in reliability, availability, performance, and cost for virtualized mission critical application data; and
• The deadly effects of silent data corruption.
Server Virtualization & Networked Storage Issues
Excessive storage network oversubscription
An odd thing occurs far too frequently when IT organizations move their applications from physical servers to virtual ones: the virtualized application performance degrades considerably and sometimes comes to a screeching halt. This phenomenon is perplexing and confusing especially after a pilot performed without a hiccup during testing.
Troubleshooting the root cause of the performance problem can be an extensive exercise in frustration. The reason is a bit complex and after much head scratching, can often be traced to excessive oversubscription.
Oversubscription is the assignment of more potential utilization demand than resources can possibly handle if all that potential utilization demand were to take place simultaneously. The underlying assumption for most storage networks is that this possibility is exceedingly remote. When this assumption is true, oversubscription makes sound financial sense because it demonstrably increases resource utilization. Increased resource utilization means significantly lower capital (less hardware and software) and operating (less maintenance, real estate, power, cooling, and management) expenditures.
However, when server virtualization is implemented on a wider scale this assumption unexpectedly becomes invalid and bad things happen when there is excessive
oversubscription. Excessive oversubscription means that it becomes probable for utilization demand to exceed the available resources. When that happens, the resource in question cannot respond quickly enough — or perhaps not at all — to utilization requests. (As an analogy, a commonplace example of oversubscription is when you dial a telephone number and receive a “fast busy” signal.) When network oversubscription level is greater than the current resources can handle, an inability to handle the request results in the equivalent of a storage network “fast busy” signal.
Oversubscription is the value principle behind server virtualization’s much increased server hardware utilization. The performance problem occurs when the oversubscribed virtualized servers are overlaid on top of oversubscribed networked storage. Instead of being additive, the oversubscription problem is multiplicative. Here is a simple example to more clearly illustrate this concept:
Figure 1: 40:1 Oversubscription Example
Another aspect of virtualized server oversubscription that tends to get overlooked — even though it occurs far too frequently – is LUN (or volume) oversubscription. Let’s explain how this occurs: the server virtualization hypervisor has a storage virtualization capability, and this capability allows the hypervisor to take physical volumes assigned by the storage admin, and carve them into virtual volumes for virtual machine guests. Problems occur when multiple virtual applications start pounding on those virtual volumes simultaneously. Each virtual application is requesting I/O from what it believes are its own unique, dedicated disks when in reality, all virtual applications are being serviced by the same shared disks. This form of oversubscription can get very ugly very fast. I/O requests are queued by both the storage system and the disks. As queues fill — which happens much faster with SATA than SAS or Fibre Channel drives because of their much smaller queue depth — response times degrade. When queues become full, virtual application I/O requests do not get serviced at all (the equivalent of a “fast busy” in the telephone analogy). Once again the SCSI protocol will very likely time out and crash the virtual application.
Too much virtual application scheduled downtime
When any storage process is application disruptive or requires an outage for maintenance, it has to be scheduled to occur when the application downtime is minimized. Storage
provisioning and volume expansion are traditionally application disruptive, and their requirements usually means scheduling such transactions in the wee hours of the morning, on weekends, or on holidays to minimize impact on business operations.
It is easy to see why this scheduling requirement might become frustrating to virtualized server users. Whereas creating virtual server guests is easy — point, click, create, and go — provisioning SAN storage is typically anything but easy. Most storage admins find storage provisioning today extremely convoluted. Provisioning is a considerable storage admin burden and time sink that continues to expand, and users commonly have to wait days, weeks, months, or sometimes longer, to get new storage capacity made available to them. Why is this the case? It’s because storage provisioning tasks are many, detailed,
each decision takes time to implement. As each hypervisor virtual application is provisioned, it has no access to its storage and remains offline until the storage provisioning process is complete. When there are human errors, the storage provisioning process takes longer. The longer the provisioning takes, the greater the virtual application disruption. And with few server admins possessing storage knowledge or provisioning skills, they have to rely on the storage administrator to get their storage provisioned.
After volumes are provisioned, expanding them is also problematic. Expanding a volume consists of taking the volume offline, adding drives, possibly restriping the RAID group, then informing the attached operating system and application that additional capacity is now available. While this is obviously painful for traditional physical servers, it is far more painful for virtual servers because so many more applications are affected each time a volume is taken offline to be scaled.
Convoluted storage tradeoffs
Tradeoffs are endemic with storage networking for virtualized servers: if you increase reliability and availability, performance takes a hit. Optimize performance levels instead, and your data may not be safe. Try to optimize for all three — performance, reliability AND availability — and cost goes through the roof.
Application performance takes a noticeable hit (as much as 50%) with common RAID arrays during a drive rebuild. Drive rebuilds can be tolerable if the rebuild is scheduled when application activity is low. However, drive rebuilds extend the timeframe in which there is an increased possibility of an unrecoverable read error during the rebuild, and an unrecoverable read error in a RAID5 drive rebuild will have the catastrophic result of permanently lost data — a much more common problem than you might at first suspect.
The lower-cost, high capacity SATA drives have a 10^-14 bit error rate (BER). When there are five drives in the RAID5 set (a common RAID5 drive ratio), the chance of an
unrecoverable read error becomes 32%. Higher cost SAS or Fibre Channel drives with their lower BER of 10^-15, reduce that probability to 3.2%. Even so, the risk of data loss with RAID5 protection levels is most likely unacceptable for mission critical applications, which usually means deploying RAID6 is the next option to consider.
RAID6 protects against the loss of data if there is an unrecoverable read error or other RAID set drive failure during a drive rebuild. Regrettably, the performance of the storage array during a 2nd drive rebuild is often halved again, which is most likely also unacceptable for a mission critical application.
data loss in the event of a 2nd drive failure, and that the risk of a 2nd drive failure is an order of magnitude lower than RAID5. This level of risk will still be unacceptable to many virtual mission critical applications, leaving admins in a quandary.
These issues are again far more severe in a virtualized server environment because the total number of virtualized mission critical applications running on each virtual server multiplies the problem. If data is corrupted or lost, it is not one mission critical application admin who will be calling, it will be many.
The deadly effect of silent data corruption
SATA drives have been a boon to the explosive growth of data because of their low cost, high capacities, and reasonable duty cycles. Numerous organizations are finding SATA very attractive in virtualized server environments for these very reasons and because of the way hypervisors virtualize attached storage. Because hypervisor storage virtualization capability carves physical volumes into virtual ones, the large capacity and low cost of SATA drives is quite appealing. Many vendors are now recommending SATA for virtualized server environments.
However, SATA nirvana is not reality: early adopters have discovered a disturbing problem unique to SATA hard disk drives called “silent data corruption.” Silent data corruption occurs when a read failure is not identified or resolved by the storage array. Despite high levels of data integrity built into the system, when silent data corruption occurs, corrupt data is passed by the storage array to the application without any notification or warning. This situation is very disconcerting because it can lead to serious incorrect application behavior and results. Silent data corruption can occur from misdirected writes, partial writes, and data path corruption. This phenomenon is exacerbated by parity pollution, which is when RAID parity is calculated using corrupt data making the original data irretrievable and the data corruption undetectable until it is too late.
Unfortunately, silent data corruption has received little press coverage even though it has been cropping up more and more frequently and causing more data problems. The problem has become large enough, however, that a number of Federal agencies have mandated that no storage system purchase can be made unless it eliminates or at least meaningfully protects against silent data corruption. For most organizations this means a big hit to the storage budget: instead of purchasing low-cost, high-capacity SATA drives, they have to implement more medium-to-high-cost (and lower capacity) SAS or FC drives.
Even more unfortunate is that SAS and FC drives are not safe from silent data corruption either. While the drives suffer data corruption with greater infrequency, the impact of data corruption on the high-cost drives can be more significant because typically those drives are used for more important data and applications.
It is easy to understand why so many organizations are frustrated with their storage and virtualized servers and are wondering if there is a better way: thankfully, there is.
NEC’s D-Series Storage Arrays
NEC thoroughly understands the issues arising from virtualized server environments and has developed the D-Series storage arrays to resolve all of them.
Simplified performance tuning for virtual servers
Simplifying storage-based application tuning helps alleviate the performance degradation surprise when moving from physical to virtual servers. The D-Series is designed from the ground up to provide the flexibility required to deal with the issues of oversubscription with its extraordinary dynamic pooling capacity, its ability to increase port count up to 64 Fibre Channel ports, and its PerformanceOptimizer software to do just that.
To reduce the impact of storage oversubscription by virtual servers, NEC turned to a positive form of oversubscription: Thin Provisioning, which is capacity oversubscription. Thin
provisioning allows multiple volumes to be created and allocated to virtual servers even when there is not enough physical capacity to hold the data if all of the volumes were full. Slices of physical capacity are allocated to the volume when the data is written, and not before. That means less downtime and management time spent provisioning storage.
The D-Series PerformanceOptimizer software automates virtual application performance tuning by identifying disk pool hotspots and allowing the administrator to non-disruptively move logical disks from that hotspot to a less utilized set of disks.
Eliminate convoluted tradeoffs in storage performance, reliability, availability,
and cost for the virtualized mission critical applications
As previously discussed, these tradeoffs present a prickly problem. NEC has come up with powerful solutions that can’t help but make other vendors think: “why didn’t we think of that!” For the vast majority of applications that require dual drive fault protection, NEC’s D-Series SAN Storage has implemented silicon-based RAID6, which, unlike software based RAID6, minimizes performance degradation during drive rebuilds.
And for those virtualized mission critical applications that require double fault protection and zero tradeoffs in performance, NEC has developed two unique hardware based RAID sets: the first is Triple Mirror (or TM) and the second is RAID3 double parity (or RAID-3DP).
RAID-TM provides the very high speed of a RAID1 set while providing the very high reliability of the RAID6 set accomplished by writing data simultaneously to three separate drives. Even if there are two drive faults or unrecoverable read errors in the same mirror, the application still has access to its data with no degradation in performance even as the drives are rebuilt.
Figure 3: RAID-TM
Page 8 of 8
adding the double parity of RAID6. This combination allows up to 2 disk failures or unrecoverable read errors to occur without any loss or data and nominal performance loss while drives are rebuilt.
Eliminates deadly effect of undetected, unreported, silent data corruption
This reliability area is where the D-Series differentiates itself from all other vendors with its “Extended Data Integrity Feature” (EDIF). EDIF is a standard feature that protects against SATA-based misdirected writes, partial writes, data path corruption, and parity pollution. More detailed information can be found in the white paper “Silent data corruption in SATA arrays: a solution” (http://www.necam.com/storage/Contacts/?ItemID=145&wp=7). EDIF is another clever NEC invention elegant in its simplicity. First the D-Series controller calculates a Data Integrity Field (DIF) for each sector before the data is written to disk. In essence, the DIF is providing a super-checksum that is stored on disk with the data, and the DIF is checked on every read and/or write of every sector. This process identifies corrupted data and enables corrupted data to be fixed.Few other storage systems have an EDIF or EDIF equivalent for either physical or virtual servers. What EDIF means is that lower cost, higher capacity SATA drives can now be used for applications where lower performance is acceptable, but where unrecoverable data corruption is not. EDIF makes SATA drives enterprise-capable.
Furthermore, the D-Series is one of a few arrays that implement the T10-DIF standard, which detects and addresses silent data corruption in SAS drives. As a result, storage
administrators can rely on their storage array to keep mission critical data integrity at its maximum.
Conclusion
There are a number of storage issues that frustrate server virtualization users including: 1. Complicated storage based application tuning when moving from physical to virtual
servers;
2. Application disruptive storage provisioning and volume scaling;
3. Unacceptable storage performance, reliability, availability, and cost tradeoffs for virtualized server mission critical applications;
4. And the deadly effects of undetected, unreported, silent data corruption. NEC eliminates these frustrations with its highly innovative D-Series. Any organization implementing virtualized servers or even just looking to add SAN storage would be well-served to take a hard look at the NEC D-Series.
NEC CORPORATIONOF AMERICA
© 2009 NEC Corporation of America. All rights reserved. Specifications are subject to change without notice. NEC is a registered trademark and Empowered by Innovation is a trademark of NEC Corporation. All other trademarks are the property of their respective owners. WP114-1_0209
