89 Fifth Avenue, 7th Floor New York, NY 10003 www.TheEdison.com
212.367.7400
White Paper
Comparing Management Costs for
Midrange Storage Arrays from EMC,
HP, and NetApp
Printed in the United States of America.
Copyright 2011 Edison Group, Inc. New York. Edison Group offers no warranty either expressed or implied on the information contained herein and shall be held harmless for errors resulting from its use. All products are trademarks of their respective owners.
First Publication: May 2011
Produced by: Kalicharan Rakam, Senior Analyst, Barry Cohen, Senior Analyst, Editor-in-Chief; Craig Norris, Editor
Table of Contents
Executive Summary ... 1
About This White Paper ... 3
Who Should Read This White Paper ... 3
Methodology Overview... 3
Contents of this White Paper ... 3
Storage Management Costs in the Enterprise ... 4
Overview of the Management Consoles ... 4
Results Summary ... 10
Comparative Management Cost Savings ... 11
Results Details ... 14
Provisioning ... 14
Data Protection ... 17
Maintenance and Configuration Changes ... 21
Monitoring and Troubleshooting ... 23
Methodology for This Study ... 25
CMCS Methodology... 25
Laboratory Configuration ... 25
Test Procedures ... 27
GUI vs. CLI ... 29
Other Considerations ... 29
How to Use These Results ... 30
Conclusions and Recommendations ... 31
Appendices ... 32
Appendix I ... 32
Executive Summary
Exponential growth in storage capacity requirements by organizations of all sizes has become the default discussion on the data storage industry. Accordingly, the Storage Networking Industry Association (SNIA) is collaborating with CompTIA to create a certification program to support the need for more storage administrators1. A growing pool of trained and certified storage administrators is a good thing, but controlling the costs of managing an ever-growing storage infrastructure requires more than
certification tests.
Medium sized businesses are especially challenged by storage capacity requirement growth. These organizations have many of the same needs as the largest organizations: maintaining performance in the face of increased application demand, and regulatory and other requirements affecting data retention policies within a context of resources limited by economic conditions. The need to acquire additional physical storage capacity is a capital expense that is not being significantly ameliorated by the growth in disk capacity from a few hundred megabytes to two or more terabytes per disk. Since CAPEX growth for storage cannot be stopped, but merely slowed, one of the few ways of
controlling the costs related to data storage is by lowering OPEX; for instance, lowering the costs of managing storage systems.
Edison has been researching the effects of how better, easier to use, management tools can help lower the costs of administering IT systems for several years. The present study is a refresh of earlier research, utilizing an improved methodology that compares the relative complexity and ease of use of management consoles for three storage platforms targeted at mid-sized organizations. These include arrays from EMC, HP, and NetApp. The products of the three vendors under consideration have evolved since the original studies, updating their management console software to address new feature sets and to improve usability. Of the three vendors, EMC has made the greatest number of changes, including a major upgrade of its Navisphere management software and the replacement of the CLARiiON product family with the new VNX family. NetApp has updated its management software, including improved management consoles that augment its embedded Data ONTAP and CLI tools. HP has revitalized its product line, renaming the family to P6000 Enterprise Virtual Arrays and improving the features and usability of its P6000 Command View software.
1 Press release: CompTIA and SNIT to Produce “CompTIA Storage+ Powered by SNIA” Certification: http://www.snia.org/home/images/01_04_20110_CompTIA_SNIA_FINAL.pdf
The results of Edison’s research show that administering HP arrays using P6000 Command View is 23 percent less time consuming, and over 28 percent less complex, than managing a NetApp storage system using Data ONTAP. Even with the significant update in the management software by EMC, managing a P6000 EVA with P6000 Command View is 20 percent less time consuming, and 32 percent less complex, than the EMC array and Unisphere software.
Using Edison-developed storage administrator workload weighting factors, these time savings can be evaluated within the context of OPEX savings. This demonstrates an improvement in HP of over 17 percent in storage administrator efficiency over NetApp and over 15 percent over EMC. Monetizing these efficiencies results in cost savings of about $13 thousand per year for HP over NetApp and a savings of about $12 thousand per year for HP over EMC.
About This White Paper
This white paper is based on a combination of hands-on evaluation, review of relevant published documents on storage cost of ownership, and reports on data center and storage administrator salaries.
Who Should Read This White Paper
This white paper should be read by anyone responsible for choosing storage solutions for their organizations or for making administrative personnel decisions in the data center.
Methodology Overview
Edison performed the series of tasks described in this white paper on storage systems from EMC, HP, and NetApp. The methodology and specifications for the hardware and software evaluated are listed within the document and in the appendices.
Contents of this White Paper
Executive Summary – a summary of the context for and the results of Edison’s research.
Storage Management Costs in the Enterprise – this section provides an overview of the factors affecting storage management costs and a discussion of the changes in the products tested in the current study.
Result Summary – this section discusses the results for each task category evaluated. Methodology for This Study – this section provides an overview of the
methodology Edison used in performing the hands-on research.
Conclusions – this section summarizes the results of Edison’s research.
Appendices – this section includes a more detailed discussion of the evaluation methodology and a terminology glossary.
Storage Management Costs in the Enterprise
Several years ago, Edison Group performed a hands-on research study comparing how the relative ease of use in administrative consoles affected the ownership costs of storage arrays. The study evaluated this particular factor for mid-range storage arrays from EMC, HP, and NetApp. The trends and developments driving storage costs then — growth, virtualization, facilities costs, and administration — are still affecting the acquisition decision process today.
The continuing exponential growth in capacity requirements is considered so ubiquitous as to be no longer worthy of comment except by financial analysts estimating the future fortunes of drive manufacturers. According to executives at VMware, the largest vendor of server virtualization on the x86/x64 platform, virtualization of servers has grown to such an extent that there are now more servers running under VMware than there are physical servers. Storage virtualization has now become a standard configuration approach for storage arrays of all sizes, from the Drobo SOHO arrays from Data
Robotics, Inc. to the largest enterprise systems from EMC, HP, IBM, and NetApp. Even with the ameliorating factors due to increased drive capacities, growing capacity requirements still translates into growing facilities’ needs.
All of these factors are inexorable and all are driving the need to control the cost of managing storage systems as the only area where an OPEX cost control lever exists. In the years since Edison’s first report on managing mid-range systems, many storage vendors have made an effort to address the costs of management complexity by
updating their management software. The assertions of ease-of-use improvements by its competitors have moved HP to ask Edison to compare once again their management software with that of NetApp and EMC.
Overview of the Management Consoles
EMC has delivered the most significant upgrade to the management software for their CLARiiON storage arrays. The original embedded software, Navisphere, has been superseded by Unisphere.2 Navisphere software was run directly on the storage
controller and accessed via a web browser. Each storage system had its one Navisphere instance; managing multiple CLARiiON arrays required accessing multiple Navisphere instances or the purchase of centralized management console software. The new
2 Navisphere software is still running on the array. When Unisphere is installed, the Navisphere APIs are integrated into the new management instance.
Unisphere software runs on a Windows server, allowing the monitoring and management of multiple EMC CLARiiON, Celera, and VNX storage arrays.
In addition to enabling the management of multiple arrays from a single console, EMC Unisphere presents an attractive and informative monitoring dashboard as the default view. The screenshot in Figure 1 below shows the dashboard view.3
Figure 1: EMC Unisphere Dashboard
Dashboards provide a status overview for each function on the managed arrays. The performance of actual management tasks utilizes the same interfaces as was used by Navisphere (Figure 2, page 6). This means that while the overall management environment has been simplified, in many cases the same procedures and their concomitant complexity are required for managing the array.
Recently, EMC announced the replacement of the current CLARiiON and Celerra product lines with new models under the VNX brand. These new storage arrays combine many of the features of CLARiiON and Celerra and include several hardware changes, including an update to the Intel processor in the controller of the current Intel
3 Our test environment contained one new storage array, so the amount of information presented is limited to that array.
Xeon x5600 family of multi-core processors. In addition, EMC joins the emerging industry transition from 3.5” FC drives to 2.5” SAS drives as the new standard for high performance enterprise class rotating disks.
Edison tested Unisphere before the new VNX hardware was readily available. While Unisphere has been updated to support the new hardware, the software features remain unchanged. Since the software has not changed, Edison believes that the results of this study reflect the manageability of the new VNX arrays as well.
Figure 2: EMC Unisphere - Create LUN Interface
NetApp has also recognized the need to address manageability challenges. Unlike EMC, however, NetApp has chosen to utilize multiple management tools. In addition to the embedded Data ONTAP software, accessible through FilerView and evaluated here, System Manager provides a server-based client that offers the same management tools and approach as FilerView with the additional capability of managing multiple filers. NetApp also provides, under separate license, NetApp Operations Manager (Ops
Manager) for viewing reports on system configuration, status, and utilization. In addition, NetApp offers Data Fabric Manager (DFM), which is also a server-based program that provides a dashboard overview similar in appearance to that offered by EMC Unisphere. DFM enables the management of multiple arrays, and if NetApp vFilers are installed, it enables the management of data across the virtualized devices.
Figure 3: NetApp Management Consoles: Top Row, Left to Right: Data ONTAP FilerView, OPS Manager. Second Row, Left to Right: DFM, System Manager
NetApp DFM allows for the configuration of NetApp filers, but it uses a different approach than that provided by Data ONTAP. In addition, some filer features such as deduplication, which cannot be configured with Data ONTAP or can only be configured via command line, can be configured and managed with DFM.
Finally, there is also a command line interface (CLI) accessible through a Java-based window within FilerView or a standard telnet or SSH terminal client.
This means that management of a NetApp filer can include several separate and entirely different management consoles.
An embedded browser-based management console, Data ONTAP FilerView A server-based management console, System Manager
A CLI (command line interface) accessible via a window in Data ONTAP or by telnet or SSH consoles
OPS Manager, a server-based monitoring and reporting tool accessible via web browser
Distributed Fabric Manager, a server-based management console accessed via Microsoft Management Console plug-ins.
This plethora of choices is not only confusing on its face, but as each console differs in what system features can be managed, the choices are also operationally confusing. The four screen shots in Figure 3 on page 7 show the home pages for each of the GUI
management consoles. 4
HP has not rested on its laurels, releasing a new family of mid-range arrays and
renaming the entire family to the P6000 Enterprise Virtual Arrays, or P6000 EVA. When Edison last compared these three platforms, P6000 Command View offered a
measurably simpler management interface than either Data ONTAP or Navisphere. HP has not changed the management approach used by P6000 Command View. Instead, it has augmented the interface by greatly improving the presentation of various system status metrics. In addition, the incorporation of new technologies and features has been all but transparent. For example, the new SSD drives are simply presented with a new icon linking to additional relevant product details within the familiar drive shelf and disk group navigation trees.
HP’s accomplishment is incorporating new capabilities without adding to interface complexity. Figure 4 on page 9 shows the home system properties view of an initialized storage system. When compared to the same view from the version of P6000 Command View evaluated in 2007 (Figure 5, page 9) the new interface offers additional
information, including a graphic view of capacity utilization, without disrupting the usability of the interface.
It will be seen in the Results Summary section that follows that, within the context of this study, P6000 Command View still delivers a more efficient and thus cost-effective storage management system.
Figure 4: P6000 Command View - Storage System Properties
Results Summary
The results of Edison’s testing (Table 1 below) show that the HP array demonstrates significant advantages over its competitors from EMC and NetApp in the area of management ease and, therefore, in administrative costs.
Our objective findings show that, overall, managing an HP array with P6000 Command View is 23 percent less time-consuming and over 28 percent less complex over a set of common tasks than managing a NetApp filer with Data OnTAP. Even with the significant update in the management software by EMC, managing a P6000 EVA with P6000 Command View is 20 percent less time consuming and 32 percent less complex than the EMC array and Unisphere software.
Table 1: HP CMCS Summary of Results
HP Array NetApp Array HP Array EMC Array
Time Steps Time Steps Time Steps Time Steps
Average Time or
Steps Per Platform 92.75 3.75 120.75 5.25 92.75 3.75 116.5 5.5 HP Advantage (HP - Competitor) -28 -1.5 -23.75 -1.75 HP Advantage % (HP/Competitor-1) -23.19% -28.57% -20.39% -31.82% Average Workday Savings -17.16% -15.09% $ savings (Average Workday Savings * Salary) $13,237 $11,637
For most IT executives, the best way to appreciate the importance of management cost savings is by examining the findings within the context of OPEX cost savings.
The time savings are a sum of the savings for the tasks evaluated. In general, Edison believes that these raw numbers do not accurately reflect the actual storage
administrators’ workloads experience. In order to provide insight into that workload experience, Edison performed a survey of storage administrators from organizations of all sizes, using storage arrays from several vendors in order to develop a set of workload weighting factors that could be applied to our testing.
The Edison white paper, Comparative Management Cost Survey: Workload Weighting for Mid-Range Storage Array Administrators, describes the research leading to the identification of the weighting factors. It can be found in the Published Reports Library on the Edison web site at http://www.theedison.com. The factors revealed in that paper cover a wider range of tasks than those evaluated in this current study. This is due to several factors, mostly related to the need to match measured tasks across the three platforms. The factors included in this study are:
Weighting Factors
Provisioning 14%
Data Protection 16%
Maintenance 27%
Monitoring and Troubleshooting 17%
As will be seen in the detailed results sections that follow, when these weighting factors are applied to the raw time savings test results, P6000 Command View demonstrated more than a 17 percent advantage over the NetApp array and more than a 15 percent advantage over the EMC array.
Comparative Management Cost Savings
The Edison CMCS methodology assigns a complexity metric and measures and
compares the time required to perform different tasks. The savings in time delivered by a less complex management platform can be given a financial value that can often provide a useful analytic context. In most real-world business environments,
management costs far outweigh licensing and vendor charged support costs throughout the life of the product. With this in mind, we estimated the annual cost savings a
business might expect due to storage administrator time savings that result from one product being easier to administer and operate than the other.
In order to compute cost savings, Edison used Storage Administrator salary figures gathered from a query of salaries made at Simplyhired.com. 5 Edison searched for the salaries in seven metropolitan areas of various sizes and costs of living. Though different choices would affect the results, Edison believes the derived average salary is
reasonable.
Table 2: Average Storage Administrator Salaries
Average Storage Administrator Salaries
Storage Administrator in NY, NY $87,000
Storage Administrator in Boston, MA $84,000 Storage Administrator in Chicago, IL $76,000 Storage Administrator in Memphis, TN $64,000 Storage Administrator in Houston, TX $71,000 Storage Administrator in San Francisco, CA $93,000 Storage Administrator in Birmingham, AL $65,000
Average $77,143
The average of the results of this search (Table 2 above), when multiplied by the weighted time savings, show cost savings:
HP array versus NetApp array
Median Storage Administrator Salary * (Storage Administrator Time Savings = $77,143 * 17.16% = $13,237.29
HP array versus EMC array
Median Storage Administrator Salary * (Storage Administrator Time Savings = $77,143 * 15.09% = $11,637.67
By substituting its own costs, an organization can determine its own potential savings. Edison recognizes that these savings are not going to be realized directly. Instead, they need to be interpreted within the context of overall storage administrator duties and storage group or IT budgets. Within this framework, savings in time of over 17 or 15 percent can be equated as lessening the need for new hires when additional storage arrays are acquired or by enabling a better allocation of resources to mission-critical tasks.
It is not practical to attach direct financial costs to complexity. However, it should be apparent that more complex management tools require more user training, can cause user error simply due to that complexity, and otherwise affect productivity.
These objective differences do not tell the whole story. As is seen elsewhere in this white paper, in most respects performing management tasks with P6000 Command View generally provides a better user experience than the competitors. Edison finds it striking that, within the context of this study, managing the P6000 EVA remains more efficient despite the ease-of-use improvements delivered by the competing vendors.
Results Details
Edison compared the performance of four categories of tasks: Provisioning, Data Protection, Maintenance, and Monitoring. Since the architecture and management approach of each vendor differs, not all the tasks that comprise a tested category could be compared: where no equivalent function or task existed for all three platforms, the time and complexity ratings for that task may have been measured, but the results were not included in the summarized results. Similarly, some tasks, such as replacing a disk, could not be performed for all of the platforms, as the test systems lacked sufficient additional disks.
The following sections each contain a description of the task categories, a summary of test results for that category, a listing of the tasks performed and compared, and a subjective appraisal of the user experience.
Provisioning
For purposes of this study, provisioning covers tasks performed after the storage array has been installed and initialized. Installation-related tasks were not included for two main reasons. First, a system is only installed once. Second, for many organizations, installation is performed by vendor technicians. There are organizations where storage systems are being added to the data center at a frequency sufficient to be called daily, but in such cases, the installation teams generally comprise staff dedicated to that task or vendor-provided technicians.
The tasks evaluated here include those related to preparing connectivity and the storage devices for use by hosts. The following table (Table 3: Provisioning Tasks Comparison) shows the specific provisioning tasks, using each vendor’s nomenclature, that Edison evaluated. (Only the tasks in bold and italics were included in the summary
Table 3: Provisioning Tasks Comparison
HP Array NetApp Array EMC Array
Creating Disk Groups Creating Aggregate Creating Raid Group or
Storage Pools (Default Raid 5)
Adding Hosts (iSCSI) Creating Volume/Qtree Create Raid Group or Storage
Pool (1/0) Creating Vdisk (Raid 6 -
Default)
Creating LUNs Create LUN (Raid 5 Default)
Creating Vdisk (Raid 5/1/0) Creating Initiator group FC/iSCSI
Create LUN (Raid 0/1 Default)
Presenting Vdisk to Host Mapping LUNs/Adding
LUNs to Initiator Group
Creating Storage Groups
Creating Container LUN Assignment to Storage
Group
The Summary of Results for Provisioning in Table 4 below shows that, for the compared tasks, HP array was 50 percent less time consuming and 50 percent less complex to manage than the array from NetApp. When compared to the EMC Array, the HP array was 36 percent less time-consuming and 43 percent less complex to manage.
Table 4: Provisioning Summary of Results
HP Array NetApp Array HP Array EMC Array Task Details Time Steps Time Steps Time Steps Time Steps Provisioning
Test Results for
Area 103 4 207 8 103 4 162 7 HP Advantage (HP - Competitor) -104 -4 -59 -3 HP Advantage % (HP/Competitor-1) -50% -50% -36% -43% Workday Savings (Difference* Weighting Factor) -7.03% -5.10%
When the weighting factor is applied, the advantages are smaller, but would probably be noticeable to most administrators.
The two charts below provide a visual indication of the differences in provisioning time and complexity for the three platforms.
Figure 6: Provisioning: Time and Complexity Comparisons
Thin Provisioning
One of the most talked about storage management features over the past few years is thin provisioning. With thin provisioning, storage blocks are allocated on demand to meet host requirements enabling storage utilization efficiency to approach 100 percent. Without thin provisioning, systems required large amounts of storage to be physically pre-allocated, often resulting in extreme resource underutilization.
All three vendors now offer thin provisioning as a standard feature for the arrays being evaluated. It was Edison’s intention to include the thin provisioning configuration as part of the evaluation process. Two of the vendors, HP and EMC, made this comparison irrelevant. Setting up thin provisioning on these platforms merely requires the clicking of a single option during LUN configuration. A single click has no effect on
configuration effort. (For example, clicking the check box labeled Thin in the Create LUN tool shown in Figure 2 on page 6 is all that is required in order to create a new LUN with thin provisioning enabled.
With P6000 Command View, enabling thin provisioning is just as straightforward. As can be seen in Figure 7: P6000 Command View - Vdisk Creation Advanced Settings below, selecting Enabled or Disabled from the pull-down menu is all that is required.
Time Comparison
Provisioning
HP Netapp EMCComplexity
Comparison
Provisioning
HP Netapp EMCFigure 7: P6000 Command View - Vdisk Creation Advanced Settings
Thin provisioning on NetApp with Data ONTAP is a different story: it is not explicitly provisioned within the user interface. Instead, there is a LUN volume level
configuration option, available with a license for SnapDrive for Windows, which allows LUNs to be created with space reservations off. Changing defaults requires that several configuration options be changed or set via the command line. As an alternative, NetApp recommends that LUNs be provisioned to the size anticipated and expanded through growing a LUN, as space is needed. This manual process requires extensive monitoring of capacity utilization on the array and on the host so that there is adequate space on the LUN.
Therefore, though configuring thin provisioning has no effect on the management ease of either EMC Unisphere or P6000 Command View, it can add considerable complexity to the management of a NetApp filer.
Data Protection
Without good data protection capabilities, there is almost no reason to invest in an enterprise storage array. Data protection includes such features as multiple controllers, multiple I/O paths, remote replication, and the creation of snapshots and clones. Multiple controllers and I/O paths are dependent upon the physical attributes of the appliance and are generally configured as part of installation. Management of the
physical controllers and I/O paths was not evaluated in this study. Remote replication was not tested because the Edison lab only had one appliance from each vendor. That leaves the creation of clones and snapshots.
A clone is defined the same way on the three platforms — a writable copy of existing data on the appliance managed in the same manner as the source data. Clones can be used as a backup, especially when replicated to a second local or remote storage system. When used in this manner they are usually called Mirror Clones, meaning that the data in each cloned copy is regularly updated and synchronized through replication. Clones are also often used in development environments where it is desirable for developers to work with real data, but where using live data would be too risky to allow. In these scenarios, the data in the cloned copy maintains its state at the time of cloning.
A snapshot is defined as a point-in-time read-only copy of data on the array. Compared to cloning, snapshots require much less additional storage space on an array, typically ten to twenty percent of the original size rather than the one hundred percent capacity required by a clone.
On a NetApp array, snapshots are created at the volume level. On the EMC and HP arrays, snapshots are created at the LUN or Vdisk level, respectively. This means that since a NetApp volume can contain multiple LUNs attached to multiple hosts, planning how data is stored requires consideration of how snapshots are to be used. For EMC and HP, each LUN can have its own snapshot policies more easily matching the business needs.
On the NetApp array, snapshots are enabled and scheduled by default at volume
creation. For comparison purposes, Edison manually created a snapshot, which is a very straightforward process: merely requiring navigation to the Add Snapshot page (under Volumes), selecting the volume upon which the snapshot will be made, giving the snapshot a name, and clicking Add. For both HP and EMC, creating snapshots is an entirely manual process. Neither HP nor EMC systems offer scheduling of subsequent snapshots in the GUI management system.
Table 5 on the following page shows the specific data protection tasks, using each vendor’s nomenclature that Edison evaluated. (Only the tasks in bold and italics were included in the summary calculations.)
Table 5: Data Protection Tasks Comparison
HP Array NetApp Array EMC Array
Creating a Snapshot (Demand Allocation/Full Allocation)
Creating Snapshots (Auto/Manual)
Creating a Snapshot (SnapView)
Creating Snap Clone Creating FlexClones Adding a Snapshot to
Storage Group
Creating Mirror Clone Creating Snap Mirror Create Reserved LUN Pool
Mirror Fracturing Start Snapshot Session
Resyncing Mirror Activating Snapshots
Detaching Mirror Restoring Snapshot Copies
Configure Clone Settings Creating Snap Clone/Mirror As can be seen in Table 6 below, the NetApp array has a significant advantage over the HP array in snapshot creation, even with manual snapshot creation. When the HP array’s required manual process is compared to the automatic snapshot creation and configuration of the FAS system, the NetApp is forty percent faster, as it requires no administrative effort. This advantage is even greater when snapshot scheduling is utilized. A NetApp administrator never has to perform a manual snapshot.
When the HP array is compared to the EMC array; however, it is 61 percent less time-consuming and 67 percent less complex to manage.
Table 6: Data Protection Summary of Results
HP Array NetApp Array HP Array EMC Array Task Details Time Steps Time Steps Time Steps Time Steps Data Projection
Test Results for
Area 58 3 50 2 58 3 147 9 HP Advantage (HP - Competitor) 8 1 -89 -6 HP Advantage % (HP/Competitor-1) 16% 50% -61% -67% Workday Savings (Difference* Weighting Factor) 2.56% -9.69%
Figure 8: Data Protection: Time and Complexity Comparison
The two charts in Figure 8 above show these differences even more dramatically. It is important to note that with automatic snapshot scheduling, NetApp’s advantage is even more pronounced as shown in the chart in Figure 9 below.
Figure 9: Data Protection: Time Comparison with NetApp Automatic Configuration
Time Comparison
Data Protection
HP NetApp EMCComplexity
Comparison
Data Protection
HP NetApp EMCTime Comparison
Data Protection
HP NetApp EMCMaintenance and Configuration Changes
Maintaining a storage system includes a wide range of activities, including software updates, replacement of failed drives, and so forth. Configuration changes include additional capacity realized by adding drive shelves and drives, changing usage configurations and the deletion of snapshots and clones for which there is no further use.
The Maintenance and Configuration Changes tasks evaluated in this study consist of such activities as deleting outdated snapshots, managing and expanding LUNs, destroying inactive LUNs, and reconfiguring or decommissioning storage pools or aggregates. Other tasks that fit under the topic but that were not evaluated include removing and replacing failed drives, adding additional drives, and so forth. Table 7 below shows the specific data protection tasks, using each vendor’s
nomenclature that Edison evaluated. (Only the tasks in bold and italics were included in the summary calculations.) Performing tasks may require additional prerequisite tasks. For example destroying a clone that is attached to a host requires that the clone be detached from the host or taken offline before it can be destroyed. The number of different possible scenarios would add unnecessary complexity to the analysis. Table 7: Maintenance and Configuration Changes Tasks Comparison
HP Array NetApp Array EMC Array
Adding Disks to Raid Groups/Pools
Adding Disks to the Aggregate
Adding Disks to Raid Groups/Pools
Expanding LUNs Using Striping, Concatenation, Mirroring
Modifying Aggregates Expanding LUNs Using Striping, Concatenation, Mirroring
Destroying LUNs Destroying Aggregates Destroying LUNs
Destroying RAID Groups/Storage Pool
Destroying Volumes Destroying RAID Groups/Storage Pool Destroying/Deactivating
Snapshots
Destroying LUNs Destroying/Deactivating
Snapshots
Destroying Clones Destroying Snapshots Destroying Clones
Destroying Mirrors Destroying FlexClones Destroying Mirrors Destroying Snap Mirror
Managing LUN Properties Expanding LUNs
Managing Snapshots Schedules
The results in Table 8 below show that in performing these tasks with P6000 Command View, the HP array is 24 percent faster and 17 percent less complex than in performing the equivalent tasks with Data ONTAP. Performing the equivalent tasks with Unisphere was both faster and less complex than what was required for P6000 Command View or for Data ONTAP.
Table 8: Maintenance and Configuration Changes: Summary of Results
HP Array NetApp Array HP Array EMC Array Task Details Time Steps Time Steps Time Steps Time Steps Data Projection
Test Results for
Area 94 32 124 3 94 32 69 4 HP Advantage (HP - Competitor) -30 29 25 28 HP Advantage % (HP/Competitor-1) -24% -17% 36% 25% Workday Savings (Difference* Weighting Factor) -6.53% 9.78%
The following two charts provide a graphic illustration of the performance differences.
Figure 10: Maintenance and Configuration Changes Comparison
Time Comparison
Maintenance And
Configuration Changes
HP NetApp EMCComplexity
Comparison
Maintenance and
Configuration Changes
HP NetApp EMCMonitoring and Troubleshooting
During the interviews Edison conducted in the development of the workload weighting factors, many of the respondents mentioned that their most important activity was monitoring system status. By its nature, monitoring does not require significant effort, but properly configuring monitoring tools both enables timely preventive maintenance and simplifies troubleshooting. Performing troubleshooting tasks was not included in this study, mostly because these activities themselves often occur outside of the management interfaces.
The monitoring capabilities of the three storage systems Edison evaluated in this study vary considerably. Not only do the capabilities of the systems vary, the location of the monitoring tools themselves also varies. Some tools are included in the management consoles Edison evaluated, but other tools require installation of additional management consoles or the use of management server monitoring tools such as Windows
Performance Manager. For this study, Edison only compared the management tools provided in the three management consoles, P6000 Command View, Data ONTAP, and Unisphere. Thus, monitoring tools that required the use of external software were not included. In addition, since by definition monitoring is a passive process, we did not evaluate the use of the monitoring tools themselves.
Therefore, the monitoring tasks we compared consisted of the configuration of monitoring on the three systems. Table 9 below shows the tasks compared and measured.
Table 9: Monitoring Tasks Comparison
HP NetApp EMC
Configuring Email Notification
Configuring SNMP Traps Configuring EMC Event Monitoring - Email/SNMP Configuring Event
Notification (SNMP)
Adding SNMP Trap Configuring EMC Event Monitoring - SNMP Configuring Host Notification Configuring Email Notification Configuring Host Notification Configuring Auto Support Configuring Real Time Status
(Health, Perf, Filer Glance)
Configuring Dial Home
Configuring Auto Support
It is important to note that though Edison performed all the tasks for each platform, not all of the tasks were performed separately on each. For example, with Data ONTAP, e-mail notification is configured as part of the SNMP configuration tasks. Similarly, with Unisphere, all of the monitoring configuration tasks are performed within a single
interface. With P6000 Command View, auto support is preconfigured: the functionality is enabled by default.
Table 10: Monitoring: Summary of Results
HP Array NetApp Array HP Array EMC Array Task Details Time Steps Time Steps Time Steps Time Steps Data Projection
Test Results for
Area 116 2 102 3 116 2 88 2 HP Advantage (HP - Competitor) 14 -1 28 0 HP Advantage % (HP/Competitor-1) 14% -33% 32% 0% Workday Savings (Difference* Weighting Factor) 2.33% 5.41%
The effect of task consolidation is evident in the results in Table 10 above. Unlike most of the other comparisons in this study, the time differences do not correlate with the
complexity differences. The charts in Figure 11 below show this more clearly than the numbers alone do.
Figure 11: Monitoring Comparison
Time Comparison
Monitoring
HP NetApp EMCComplexity
Comparison
Monitoring
HP NetApp EMCMethodology for This Study
Edison deployed storage arrays from EMC, HP, and NetApp in a lab within its New York City facility, and then performed a series of management tasks, capturing the steps and time required using the Edison Comparative Management Costs Study (CMCS) methodology. Details about the methodology, as well as the lab configuration and the testing process, are described below.
CMCS Methodology
The CMCS methodology was created by Edison to enable an objective comparison of the manageability ease of use for enterprise technology products. Edison devised the
methodology in order to provide an objective metric for comparing Information
Technology management platforms. More subjective metrics, such as those derived from interviews with groups of testers in typical user experience testing, are excellent for gaining an understanding of the preferences of the testers at the time of testing, but the results can vary significantly for different groups of testers and different test conditions. Objective metrics based on mouse clicks cannot adequately address issues such as the difference between Wizard-based versus non-Wizard based interfaces. The Edison methodology attempts to address these concerns by establishing a set of rules that can account for complexity and different interface design philosophies. A full description of the methodology can be found in the appendix of this report.
Laboratory Configuration
Edison's test lab was configured with arrays from EMC, HP, and NetApp. Table 11 on the following page lists the storage array models, hardware features, and the licensed software installed in the lab.
Table 11: Lab Configuration EMC
Hardware
EMC CX4-120 Dual Controller with one drive shell 5 1TB SATA Vault drives6
5 1TB SATA drives for data
Licensed Software
EMC Unisphere (Navisphere license included. Unisphere replaces Navisphere at installation),
SnapView, Thin Provisioning
HP Hardware
P6300 EVA dual controller with 2 drive shelves 10 300 GB FC drives per shelf
2 400 GB FC drives per shelf
Licensed Software
P6000 Command View
Thin Provisioning, Continuous Access, Business Copy
NetApp Hardware
Filer Model FAS2040 as a single enclosure and drive shelf 12 500 GB SATA drives
Licensed Software
Data ONTAP
NFS, iSCSI, ASIS (Dedupe) NearStore, HTTP, SyncMirror, OPS- DFM Management Server, SnapDrive for Windows, SnapRestore, SnapVault ONTAP, FlexClone, MultiStore, NearStore. Additional available software was neither licensed nor used.
6 In the past, Edison used the words Disks and Drives interchangeably. With the emergence of solid state drives (SSD) without rotating media, Edison has decided to use the term Drives where a generic usage is appropriate. Disks or Disk Drives are used for rotating media and SSD or SSD Drives will be used for solid state media.
The storage systems were connected to hosts using an HP Fibre Channel switch that was installed in the same rack as the HP P6000 EVA Array. P6000 Command View was installed on an HP DL360 G5 Server. EMC Navisphere and NetApp DFM and OPS management servers were installed on an HP DL580 Server.
Test Procedures
This study focuses on regularly performed administrator tasks. Edison divided the tasks into categories that relate to the tasks typically performed by storage administrators on a daily basis: 7
Provisioning – Provisioning includes all the tasks or operations performed after installation in order to attach an array to servers and store data. These include global tasks such as creating a storage pool (EMC), disk group (HP) or aggregate (NetApp), configuring ports for connectivity to hosts, and the creation LUNs or Vdisks so the array can actually be put to use. 8
Data Protection – Data protection covers such features as the creation of snapshots and clones.
Maintenance and Configuration Changes – Maintenance and Configuration Changes include performing maintenance tasks such as adding additional capacity, cleaning up unneeded snapshots, clones, and so forth 7.
Monitoring and Troubleshooting – Monitoring an array consists of identifying performance and operational parameters that affect uptime and performance, setting alerts, and generating reports on status and system condition. Troubleshooting consists of identifying the causes of issues identified through monitoring or from other sources such as server administrators or end users concerned about
performance or data availability. Only enabling and configuration of monitoring were evaluated in this study.
Overhead or Other – these tasks include such tasks as budget or planning meetings, training, and other activities not directly related to any specific storage array. These activities were not evaluated in this study.
7 The tasks listed for the categories are illustrative, not inclusive. Not all of the tasks listed were evaluated nor are all possible tasks for a category listed. In addition, these categories are not exclusive: tasks listed for one category may be justifiably perceived as belonging to a different category.
8 See the appendix for a table comparing the equivalent nomenclature for the features and functionality used by each of the vendors.
Edison identified tasks under each category and matched them between the platforms compared in this study. Since the vendors use either different nomenclature for the same feature or the same nomenclature for different features, this was a very important first step. 8
Administrators rarely perform specific tasks in isolation. Often, the administrator has a goal that requires the performance of a combination of several tasks or operations. To emulate this approach, Edison further categorized the task list to reflect a goal-oriented approach. This also simplified comparisons where the architectural differences between the two platforms would make task-to-task comparisons difficult or even impossible. Table 12 below shows the list of tasks and goals, organized by category.
Table 12: Task Categories 7
Provisioning
Baseline Provisioning: Tasks that need to be performed prior to creating usable storage on any storage system
Preparing Storage
This series of tasks includes the creation of a storage pool (EMC), disk group (HP), or aggregate (NetApp) from the drives in the array.
Preparing Connectivity
For this study, this series of tasks includes all the operations required for setting up access to FC Hosts on the array.
Configuration of other network protocols was not evaluated in this study.
Configuring Usable Storage
This category includes the creation of storage objects that can be attached to a host or server
Data Protection
Creating Snapshots These tasks include the creation and scheduling of snapshots.
Creating Clones These tasks include the creation of clones.
Replication These are tasks related to replicating data onto remote storage arrays. Replication tasks were not performed during this study. Maintenance and Configuration Changes
Removal of Unused Storage Objects
This includes the deletion of old snapshots or clones, deletion of LUNs, etc.
Monitoring and Troubleshooting Monitoring and
Troubleshooting
This includes the performance of a series of tasks related to the identification of problems on the arrays. For this study, the comparison focuses on the configuration of monitoring tools.
GUI vs. CLI
Edison believes that well designed Graphical User Interfaces (GUIs) can greatly improve the quality of system management as compared to the exclusive use of a command line interface (CLI) or scripting. This is especially true for the less skilled staff usually tasked with day-to-day management.
Edison does not oppose the use the CLI or scripts, but we believe that for the majority of day-to-day tasks, a Graphical User Interface can lower training and task-oriented
support costs and prevent operator error: both significant factors in administrative efficiency, and thus cost. In fact, a good GUI should streamline the use of scripts by providing an interface for running and creating scripts through learning or similar capabilities. By enabling the use of scripts for the performance of repetitive tasks while providing access through a GUI, the best features of both approaches can be utilized. Dick Benton, principle consultant at GlassHouse Technologies Inc., wrote an article published by SearchStorage on storage staffing. The theme of the article is that a traditional measure of storage administrator staffing needs — Terabytes per Full Time Employee — is almost meaningless. This is because the metric can only be calculated within the context of each individual organization. As a practical matter, TB/FTE can only be used after acquisition and implementation, and then only for analysis of staffing requirements for future expansion of the existing infrastructure. Changing the
infrastructure to a different platform changes the basis for TB/FTE calculations.
The article lists three things to consider when calculating storage management staffing requirements. The first factor is called “technology factors” which are the technologies (hardware and software) that require significant training or experience for use. The other two factors, “transaction factors” (the day-to-day tasks performed by storage administrators) and “complexity factors” (“factors that impact a storage administrator's need for skills and the time needed to execute a task”) have a major influence on the technology factor; the easier a technology is to use, the less training or experience is required for its use. Edison employs these same criteria in comparing storage administrator costs in this study.
Other Considerations
As mentioned above, this study only considers the effect of greater administrative efficiency on ownership costs. Other factors, such as acquisition costs, space and power utilization, third-party installation, decommissioning, and so forth are not considered. There is one other ownership cost factor that is not being directly analyzed but which has a bearing on this study: the importance and nature of planning in implementing complex IT infrastructures.
Planning for a SAN implementation requires an analysis not only of how much storage is required (including projected expansion), but also of the nature of what is being stored. Such considerations include the existence of Microsoft Exchange, SQL Server, or Oracle databases, how much capacity they require, and what data protection policies are needed for operational and regulatory reasons. At a minimum, SAN planning requires the identification of the hosts to be attached to the system, operating systems and application data to be stored, and any data retention and protection policies that are in place. Installing any of the arrays Edison evaluated requires these minimum planning steps.
In the modern organization, there is one constant for storage utilization: the storage capacity required and the hosts and applications to which it is attached will change rapidly and in often unanticipated ways. If a storage system offers insufficient flexibility to accommodate the rapidly changing environment into which it is placed, the costs of administration can grow very quickly. A system that requires adherence to an overly structured planning process and storage schema can prove very difficult to adapt to changing circumstances. A highly flexible system will enable rapid IT responsiveness to business changes and lower the costs accrued from those changes.
How to Use These Results
Whitepapers such as this one are intended to help readers make product acquisition decisions. For midmarket storage arrays, the purchase decision must include many factors. Mid-size organizations often have relatively smaller IT organizations, with administrators wearing many hats. The more efficient the administrator’s toolset, the more productive the administrative staff can be.
When reviewing these results, consider your organization’s file storage and application integration requirements. Review your staffing policies, as well, including training programs and other factors. You should also evaluate the skill levels of your existing staff. If your team has great skills and deep knowledge of storage management issues, the choice of storage array management tools may seem moot; but a more efficient platform can enable your team to make better use of their time.
Conclusions and Recommendations
The inexorable growth in storage capacity requirements along with the continued emergence of server virtualization and other factors has continued to drive
organizations to acquire additional storage systems or expand the systems they have. The ongoing CAPEX costs are not going away in the foreseeable future. Therefore, organizations seeking to control costs must look at the OPEX costs associated with managing their ever-growing storage hardware deployments.
P6000 Command View delivers a compelling savings in OPEX through its highly efficient management capabilities. Edison testing has shown savings of over 23 percent in administrator time over performing the same set of tasks with NetApp Data ONTAP, and over 28 percent less complexity. When compared to EMC Unisphere, P6000
Command View demonstrated a time reduction of more than 20 percent, and was over 32 percent less complex to manage.
If workload-weighting factors are applied to these results, the savings can be monetized in administrator time savings at over $13,000 per year for P6000 Command View versus NetApp Data ONTAP. Similar savings worth almost $12,000 versus EMC Unisphere were demonstrated.
Edison believes that for organizations in need of a mature, stable storage platform for mid-range enterprise applications, the P6000 EVA family of arrays provides a superior combination of capabilities at a lower OPEX cost than competing systems from EMC and NetApp.
Appendices
Appendix I - CMCS Methodology Description
The Methodology Defined
Edison Comparative Manageability Cost Studies methodology is a product
manageability cost evaluation process, whereby the products in question are compared against a set of task-oriented objective and subjective metrics in order to derive an accurate set of analytical results. The outcome of this study determines the Comparative Management Cost (CMC) incurred by managing and operating the products in a
production environment. The methodology employed to conduct this comparison consists of the following elements.
The Study
The study is the baseline checklist of standard administration tasks routinely performed, quantitatively and qualitatively compared in order to determine, on a task-by-task basis, which product is superior. This is measured primarily in terms of ease of administration and secondarily (for certain tasks only) in terms of system speed of execution — the wall clock time it takes for the system in question to complete a job once it has been
submitted by an administrator. The function of this study is to apply a set of quantitative metrics, developed by Edison Group, to a list of tasks typically regarded as qualitative in nature, in order to derive a meaningful set of CMCS statistics that can reveal the real difference in management costs for the two products in question.
Tasks
Edison defines a task as a complete logical activity, composed of one or more steps, all of which effect a significant alteration on the state of the device or software program that accomplishes a specific work goal. Each task is measured for time and complexity. Time and complexity, as used in the study, are defined as follows:
Time
Defined as the amount of time it takes to perform a given task. For certain
(asynchronous) tasks, when a job can be run in the background so that the administrator can use the time for accomplishing other tasks, time is measured strictly in terms of the time it takes the administrator to perform the steps to configure, initiate, and submit a given task.
For other (synchronous) tasks in the study that demand the administrator’s full attention and prevent the accomplishment of other tasks (as in performing a hot recovery
operation on a live database), time is measured to include both the time it takes for an administrator to configure/execute the task in question as well as the time it takes the system to complete the task. All time metrics are measured in wall clock time.
Complexity
Complexity is measured using a proprietary metric devised by Edison Group: it is the number of system-affecting steps it takes to complete a given task, where a step is defined as a task component that effects a change of state to the system under test. Because not all steps have the same inherent complexity, each step is further broken down into increments to account for the difference. An increment is a decision point that the user must make to complete a step. Increments are technically defined as a part of a step that will have a measurable effect on the state or execution path of that step in the task process, but which in and of itself does not effect a change upon the underlying system state until the step being executed is complete. For example, selecting Basic vs. Advanced Install with the installation wizard is an increment and not a step.
Complexity is then measured in terms of number of steps, but taking into account the following factors:
The number of increments it takes to complete each step.
Whether or not instrumentation for a given step is GUI-based or requires the use of a command line/scripting interface.
Whether or not the task requires a context switch between multiple interfaces in order to be completed. If a context switch exists, then additional steps will be added to the total step count for a given task.
The above factors affect the complexity calculation as follows:
The primary measure is steps. If a step has many increments, it is considered several steps. The metric allows each step five increments, and thereafter we add steps for each additional five increments rounded up. So if a step has between 0–5 increments, it remains unchanged; if it has between 6–10 increments, it is increased by one; between 11–15 increments, it is increased by two; and so on. We decided to do this because, while increments are secondary to steps in determining complexity, they do modify the
relative complexity of a given step in the course of completing a task. In other words, steps with a low number of increments are simple, and steps with a high number of increments are complex.
The other modifiers (instrumentation and context switching) may occur very
infrequently in the products under review, but are a significant enough factor that we needed to account for them in some meaningful way in order to generate a measure of complexity that accurately reflects our experience of using the products.
Regarding instrumentation, if an operation could be executed entirely within a GUI interface, then the complexity/step value for that task would remain unmodified. If, on the other hand, a step required the use of a command line interface, this would increase the step count. For a simple single-line command operation, the step count was
increased by one, whereas if the operation required the user to write a script, the step value was increased by two or more, depending on how much work was required to write the script in question.
Lastly, we come to the matter of context switching. If a context switch was encountered during the course of completing a given task, then two or more steps were added to the step count for that task. The possible addition of more than two steps was allowed for as a judgment call on the part of the subject matter expert performing the task under consideration. The reason tasks containing context switches may be penalized is that we regard it as inherently more complex to understand the dependencies of relating and performing a single operation in two different environments, as opposed to performing a similarly complex task in a well-integrated environment where all the operations can be accomplished in one place.
The following is the complexity formula utilized throughout these studies.
Complexity Calculation Formula
Complexity is defined as the number of computed steps it takes to complete a given task. The formula Edison typically uses to compute complexity for each of the tasks in this study is as follows:
For every five increments contained in a step, we increase the step value by one. For example:
If a step has 0–5 increments, step value = step + 0, If a step has 6–10 increments, step value = step + 1 If a steps has 11–15 increments, step value = step + 2 … and so on.
The type of instrumentation offered to perform a given task modifies the task’s complexity.
If a task requires the use of a command line interface, then step count is modified as follows:
If the command line operation consists of a single-line command, then step count = step count + 1.
If the command line operation requires writing a script, then step count = step count + 2 or more steps, depending on a subjective assessment of the complexity of the script. If a task requires a context switch between different environments, then step count = step count + 2 or more steps, depending on a subjective assessment of the complexity of the context switch.
Appendix II - Terminology Glossary
This table lists the equivalent feature nomenclature for the operations and tasks evaluated in this white paper. The list covers the three vendors with a brief description of the core functionality and descriptions for each vendor’s implementation of that functionality. The cell will be blank if the vendor lacks an equivalent feature.
HP
EMC
NetApp
Description
Disk Group
A virtualized storage pool automatically created on system initialization. By default it is comprised of all of the disks in the system, but customization is possible if required.
Storage Pool
Storage Pool is means of storage allocation from any type of disk and can incorporate multiple Tiers (Flash, FC, and SATA) in the same pool. Storage pools are defined by the administrator before creating any LUNs. LUNs can be created at pool level and these pool LUNs are no longer bound to a single storage tier and can be spread to different storage tiers within the same pool.
Aggregate
A collection of disks in the system, defined by the administrator on initial configuration. The aggregate will contain the virtualized storage space after the performance of additional configuration tasks.
Some or all of disks in a system, collected into a unit of or for virtual space. (The state of the collections varies with vendor).
HP
EMC
NetApp
Description
RAID Group
A collection of disks, defined by the administrator at initial
configuration. The RAID group will contain the virtualized storage space after the performance of additional configuration tasks.
Volume / QTree
NetApp currently has two types of volumes. The traditional volume (on which a file system sits that contains data) can be created on top of one or more RAID groups and is directly related to the underlying disk drives. With Data ONTAP 7G, NetApp introduced the concept of Aggregates (see above) on which can be created something called flexible volumes or FlexVols. Because a FlexVol volume is abstracted from the underlying disk, you can create a volume to meet your capacity needs without regard for physical layout. A NetApp Volume or FlexVol can be directly accessed by hosts in a NAS environment. For FC and iSCSI SAN environments, a LUN must be created.
In traditional storage management nomenclature, a volume is an identifiable unit of data storage that is sometimes (but not always) physically removable from the computer or storage system. The usage of the term varies somewhat with the computer operating system in use. When used in the context of network storage, the term can be used to define a storage space set aside for a specific host’s access. This space is often directly related to a set of physical devices though it can also refer to a virtualized storage space. For this study, the term relates to a specific usage, unique to NetApp.
HP
EMC
NetApp
Description
Vdisk
The Vdisk is defined and assigned to a host during configuration from the virtual space created in the Disk Group.
LUN
A LUN is defined during configuration, but additional operations must be performed before the LUN can be assigned to a host.
LUN
The LUN is a storage attribute assigned to a virtual space created on a NetApp filer specifically for use by iSCSI and FC SANS.
The Virtual Disk drive or Logical Unit to which a host (usually a server) is attached in a storage area network. In addition to the
terminology differences between the storage vendors, different operating systems also use different terminology to describe the storage thus made available.
Host Group
The Host Group is automatically discovered, though manual identification is possible and occasionally required.
Storage Group
Storage Groups are created during configuration from the collection of discovered or manually identified hosts.
Initiator Group
The Initiator Group is created from the collection of discovered or manually identified hosts. The term initiator label is based on iSCSI terminology.
The collection of hosts identified (in a SAN) by WWN (for Fibre
Channel) or IQN (for iSCSI) designate which can access the storage system. All three vendors are similar in functionality, with slight variations appropriate to their platforms.