EMC Solutions Group
Abstract
This technical note highlights the value of the partnership between EMC® and Ecologic Analytics. It includes technical reference architectures and test results for the EMC optimized infrastructure supporting Ecologic Analytics’ meter data management system (MDMS) for 500,000 meters.
September 2011
EMC OPTIMIZED INFRASTRUCTURE FOR
ECOLOGIC ANALYTICS’ METER DATA
MANAGEMENT SYSTEM
Enabled by EMC CLARiiON, RecoverPoint, NetWorker, and Data
Domain
A powerful and affordable combination for mid-market utilities
Best practice storage configuration with FAST VP
Copyright © 2011 EMC Corporation. All Rights Reserved.
EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.
The information in this publication is provided “as is.” EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of
merchantability or fitness for a particular purpose.
Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.
For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.
All trademarks used herein are the property of their respective owners. Part Number H8228.1
Table of contents
Executive summary ... 5
Business case ... 5 Solution overview ... 5 Key results ... 6Introduction ... 7
Overview ... 7 Purpose ... 8 Goals ... 8 Scope ... 9 Results ... 9 Audience ... 9 EMC ... 9 Ecologic Analytics ... 9 Terminology ... 10Technology overview ... 11
Introduction to components ... 11 Ecologic MDMS application ... 11EMC CLARiiON CX4-480 storage system ... 12
EMC FAST Suite ... 12
EMC FAST VP ... 12
EMC FAST Cache ... 13
EMC FAST VP with FAST Cache ... 13
EMC PowerPath ... 14
EMC RecoverPoint ... 14
EMC NetWorker ... 14
EMC Data Domain ... 14
Configuration ... 15
Overview ... 15 Physical environment ... 15 Hardware... 16 Solution software ... 17 Oracle configuration... 17 MDMS configuration ... 18 Backup environment ... 19Testing and validation ... 21
Testing methodology ... 21
Testing elements ... 21
Testing scenarios ... 21
Fibre Channel Baseline storage configuration ... 22
FC and FAST Cache ... 23
FAST VP configurations ... 24
Test results ... 25
Overview ... 25 Performance ... 25 Price ... 26 Price performance ... 27I/Os per second (IOPS) ... 28
Additional drill down IOPS results ... 29
Backup and Restore– EMC NetWorker and EMC Data Domain ... 29
Disaster recovery with EMC RecoverPoint ... 29
Conclusion ... 31
Executive summary
Utility companies are preparing for an explosion in new data because of Advanced Metering Infrastructure (AMI) projects. Many utilities will be moving from monthly meter reads to hourly and even 15-minute intervals.
AMI can produce a staggering 3,000-fold increase in data. In addition, AMI is just one of many new data-intensive applications. The smart grid will lead to the
implementation of distribution automation, wide area measurement systems, demand response, and electric vehicles—all fueled by data.
Utilities are asking, “Where will we store all of this data and how will we manage it? How will we back it up? How will we protect it?” The challenge is not just one of data throughput, but more importantly, one of management. Utilities will need a new architecture.
EMC® and Ecologic AnalyticsTM together provide an integrated meter data management system (MDMS) solutionfrom application to database to backend infrastructure.
Ecologic Analytics provides a world class, scalable MDMS, which collects, validates, and manages hundreds of millions of meter reads every single day. EMC provides the scalable, reliable, and high-performance infrastructure to store, protect, optimize, and secure the mission-critical and revenue-generating data assets.
This combined solution demonstrates an optimized architecture for a 500,000 dual-channel meter load, jointly tested and benchmarked by both organizations and includes the following components.
The Ecologic MDMS includes:
Oracle 10g Release 2 Enterprise Edition Relational Database Management System (RDBMS)
Oracle Enterprise Linux (version 5) operating system
EMC CX4-480 block storage system, leveraging the following five configurations in separate tests:
All Fibre Channel (FC) disks on dedicated logical units numbers (LUNs) FC plus FAST Cache
FAST VP (virtual pools) with three storage tiers (Flash, FC, and SATA) FAST VP with two storage tiers (FC and SATA) and FAST Cache
FAST VP with three storage tiers (Flash, FC, and SATA) and FAST Cache Disaster recovery using EMC RecoverPoint appliances
Backup and restore of the Ecologic MDMS database using EMC NetWorker® and Oracle RMAN to EMC Data Domain® backup appliances.
Business case
EMC Global Solutions tested the 500,000-dual-channel-meter load with the Ecologic MDMS solution.
The tests revealed that with various storage configurations, EMC FAST VP with three storage tiers (Flash, FC, and SATA) optimizes the application thread processing offered by the Ecologic MDMS. The storage input/output service times improved and used only one-half the disk capacity of the Fibre Channel baseline.
EMC has successfully built and demonstrated the backup architecture using EMC NetWorker with Oracle RMAN to back up the Ecologic MDMS database to Data Domain.
This solution easily meets disaster recovery requirements by using EMC RecoverPoint, which provides an RPO of 2.5 minutes and an RTO of 2.5 minutes.
Recovery time objective (RTO): Four times faster than the original 10-minute goal.
Recovery point objective (RPO): Twenty-four times faster than the original one-hour goal.
Introduction
Utility companies are preparing for an explosion in new data because of Smart Metering or AMI. As utilities move from monthly meter reads to 15-minute intervals, the amount of data will increase by almost 3,000 times. Figure 1 shows the growth in meter reads because of AMI.
Figure 1. Storage requirements for a generic AMI single-channel deployment This exponential increase in data requires a fundamentally new approach to meter data management. As Jesse Berst from SmartGridNews.com says, “start with the foundation.” In his article, Mr. Berst comments that architects would not build a mansion on a foundation intended for a small two-bedroom.1 The same is true for smart metering and meter data management.
The Electric Power Research Institute (EPRI) agrees. It reported that the goal of the smart grid is to enable multiple applications, but utilities must “first build the right foundation,”2 including security, network management, and data management. Figure 2 shows EPRI's vision of the smart grid, including the three critical foundations in the middle.
1 Berst, Jesse. “The meter data management pitfall utilities are overlooking.” SmartGridNews.com. May 31, 2011. 2 Ibrahim, Erfan. Ph. D. “EPRI’s Smart Grid Vision & AMI/HAN Research Overview.” Electric Power Research Institute.
Presented at EEI T&D Conference. April 7, 2009. Overview
Figure 2. Electric Power Research Institute (EPRI) smart grid conceptual diagram EMC and Ecologic Analytics have combined forces to provide the data management foundation as shown in the EPRI diagram in Figure 2. Customers are no longer asking for pieces of technology in silos; customers are asking their vendors to come together to offer complete solutions that include the MDMS, the storage configuration tuned for optimal performance, and the backup and recovery systems.
EMC and Ecologic Analytics worked together at EMC’s test facility to benchmark an optimized MDMS and storage infrastructure. The test case focused on a 500,000-dual-channel, 15-minute interval meter implementation loaded with 13 months of historical data to represent a fully loaded production system.
The purpose of this solution is to determine the optimal price/performance storage configuration for the Ecologic MDMS. The MDMS contains meter data from 500,000 dual-channel register and interval meters with three percent of the register reads missing, and one percent of the interval meter reads estimated. The MDMS used eight application-processing threads, which is optimal for a mid-market utility.
EMC and Ecologic have committed to providing a real-world test scenario. The application deployment and technology configurations chosen are representative of existing customers in production.
EMC and Ecologic Analytics sought to improve MDMS application per-thread
performance for core MDMS processing with the use of advanced storage technology. The core functions include:
Interval meter data consumption
Validating, estimating, and editing (VEE) processing Billing determinant creation
Purpose
This solution involves building an EcoLogic MDMS environment on EMC systems to demonstrate optimal configurations for storage and data protection.
This solution does not provide a comprehensive guide to every aspect of an Ecologic solution nor does it seek to demonstrate the maximum performance of an EMC storage system for all workloads. This whitepaper provides a reference architecture for achieving optimal performance on the storage system for the particular Ecologic MDMS workload.
Industry experience confirms that the test results reflect real-world production customer MDMS operations. The use of application threaded processing within the MDMS enables the Ecologic Analytics’ solution to be configured to scale down to mid-market and scale up to large-mid-market requirements by simply configuring additional application processing threads. Production experience and test results indicate that Ecologic MDMS provides consistent performance that can be measurably improved using advanced storage technology, as shown in Table 1.
Table 1. Ecologic MDMS processing times
MDMS process Meter reads per second per thread
Register and interval data VEE 5,088 Register and interval data consumption 2,215
This paper is for anyone currently engaged in or evaluating Advanced Metering Infrastructure or Meter Data Management System projects. It is important for IT personnel to right-size the IT systems necessary to support these mission-critical applications and utility processing objectives. It is also important for the business to understand the IT implications of such projects.
EMC Corporation is the world's leading developer and provider of information infrastructure technology and solutions that enable organizations of all sizes to transform the way they compete and create value from their information. EMC has served investor-owned utilities, municipalities, and cooperatives for over 25 years with solutions that address their information management challenges. You can find more information about EMC products and services at www.EMC.com/smartgrid.
Ecologic Analytics transforms AMI data into accurate, timely, and actionable information for electric, natural gas, and water utilities. Ecologic Analytics’ Flagship software product, Ecologic Meter Data Management System, is the most widely deployed MDMS in North America. It accommodates millions of AMI meters relaying billions of consumption reads yearly. Ecologic Analytics was founded in 2000 in Bloomington, Minnesota and counts Pacific Gas & Electric, Oncor, Austin Energy and Puget Sound Energy among its growing list of customers. More information can be found at www.EcologicAnalytics.com. Scope Results Audience EMC Ecologic Analytics
This paper includes the following terminology. Table 2. Utility industry terminology
Term Definition
SCADA Supervisory Control and Data Acquisition CIS Customer Information System
WMS Workforce Management System GIS Geographic Information System AMS Asset Management System OMS Outage Management System Terminology
Technology overview
This section details the components that make up the solution described in this white paper, which includes:
Ecologic MDMS application
EMC CLARiiON CX4-480 storage platform EMC FAST Suite
EMC PowerPath EMC RecoverPoint EMC NetWorker EMC Data Domain
The Ecologic MDMS consolidates many data streams into a manageable information flow, allowing all stakeholders secure access to the wealth of validated and stored AMI data offered in formats usable by other enterprise solutions.
The Ecologic MDMS technical architecture was deployed on the following software components:
Oracle 10g Release 2 Enterprise Edition RDBMS Oracle Enterprise Linux (version 5)
Introduction to components
Ecologic MDMS application
The MDMS architecture is divided into three tiers as represented in Figure 3 below.
Figure 3. Ecologic MDMS architecture
The EMC CLARiiON® CX4-480 storage system combines five 9s (99.999%) availability with innovative technologies like Fully Automated Storage Tiering (FAST), Flash drives, Virtual Provisioning™, 64-bit operating system, and multi-core processors. The CX4-480 scales up to 939 terabytes (TB) to handle future growth and features UltraFlex™ technology, providing multiple protocol options and online-expandable connectivity.
EMC FAST Suite is a software bundle offered on the VNX and CLARiiON series systems and is composed of two storage efficiency technologies—FAST VP and FAST Cache. Both technologies are aimed at reducing the number of Flash drives required to meet a given performance requirement, allowing business applications to take full
advantage of the investments made in Flash drives, delivering optimal application service time and improving the TCO of the solution.
EMC FAST VP
EMC FAST VP automates tiered storage strategies in virtual provisioning environments by easily moving workloads between storage tiers, since performance characteristics change over time. FAST VP performs data movements—improving performance and reducing costs—all while maintaining vital service levels.
FAST VP provides performance improvements by using automated sub-LUN level tiering and allows a single LUN to leverage the advantages of Flash, Fibre Channel, EMC CLARiiON
CX4-480 storage system
distribution of the tiers of storage to optimize both capacity utilization and performance delivered from the storage investment.
EMC FAST Cache
EMC FAST Cache technology introduces an extra layer of cache using EMC Flash drives between Dynamic Random Access memory (DRAM) cache and rotating spindles, which increases the I/O service responsiveness. The system automatically identifies both the hot (highly used, frequently accessed) and cold data and places the “hot” data into cache memory of this newly created cache layer on Flash drives. This technology provides very low latencies to frequently accessed data, thus improving the overall application response times and significantly reducing the investment in Flash drives.
EMC FAST VP with FAST Cache
FAST VP and FAST Cache technology provide very low latencies to frequently accessed data, thus improving overall application response times. By using Flash drives for the most frequently accessed data, the investment made in Flash tier/FAST Cache is leveraged to the optimal extent in order to deliver a magnitude of application level service improvement. As the focus of business changes, the data that becomes the most important and most frequently used is either automatically migrated based on a relocation schedule to the Flash tier or cached in Flash drives. Therefore, business applications can always take full advantage of the performance benefits of Flash drives. The performance improvement gained by using the FAST Suite provides quantifiable operational cost savings over time as illustrated in Figure 4.
Figure 4. EMC’s FLASH strategy leverages the FAST Suite
For more details on FAST VP and FAST Cache, contact your EMC representative or refer to the following white papers on the EMC Powerlink® website:
EMC FAST VP for Unified Storage Systems—A Detailed Review
EMC PowerPath® automates path failover and recovery and optimizes load balancing so that you can get the most out of your data center environment. It automatically tunes your storage area network (SAN) and selects alternate paths for your data if necessary. Residing on the server, PowerPath Multipathing enhances SAN
performance and application availability.
EMC RecoverPoint provides cost-effective data recovery for local and remote protection. Continuous data protection (CDP) and continuous remote replication (CRR) enable any-point-in-time recovery to protect you against data loss or corruption. RecoverPoint supports application bookmarks, instantaneous recovery, and bi-directional local and remote replication. Its clustered architecture provides linear scalability to support the most demanding environments. RecoverPoint CRR also provides synchronous replication and continuous asynchronous bi-directional replication at the block level with no distance limitation.
EMC NetWorker provides efficient and simple backup and recovery across
heterogeneous environments. It helps accelerate next-generation backup by allowing utilities to bridge traditional and next-generation environments. NetWorker offers seamless integration to the industry-leading deduplication platform, EMC Data Domain®.
The EMC Data Domain appliance delivers high-speed, inline deduplication to reduce the amount of disk storage needed to retain and protect data by ratios by 10 to 30 times and greater, making disk a cost-effective alternative to tape. This allows you to keep your data online and onsite for longer retention periods, and data restores become fast and reliable.
EMC PowerPath
EMC RecoverPoint
EMC NetWorker
Configuration
This solution includes a single server node on the front-end running both the Ecologic MDMS application and the Oracle database. The server connects to the SAN through redundant FC host bus adapters for multipath I/O for high performance and
availability.
The backend includes an EMC CLARiiON CX4-480 connected to the switches for production storage I/O and to an Ethernet network to a remote CX4-480 for disaster recovery. For de-duplicated backup, the system communicates with an EMC Data Domain system using an Ethernet connection.
Figure 5 displays the physical hardware architecture in this solution.
Figure 5. Environment overview Overview
Physical environment
Hardware
Table 3 lists the hardware used in this solution. Table 3. Solution hardware
Hardware Quantity Configuration
Storage array 2 EMC CLARiiON CX4-480 running FLARE 04.30.000.5.511
For details on the disk drives, see the section entitled “Testing scenarios” on page 21.
Network n/a SAN: 8 GB FC switch director DR SAN: 4 GB FC switch director LAN/WAN: 1 Gb Ethernet switch Database Server 1 Two quad-core Intel Xeon 2.9 GHz
processors (8 cores total) 96 GB physical memory
2 Dual-Port QLogic QLE2562 8Gb Oracle Enterprise Linux 5.5 64-bit ext3 file system
Backup n/a Data Domain 690 DDOS 4.9
40 TB of usable capacity; 5 trays containing 16 x 500 GB disks NetWorker server
16 GB of memory Two 2 GHz CPUs
One Ethernet connection NetWorker storage node
16 GB of memory Two 2.83 GHz CPUs Two Ethernet connection Disaster Recovery 4 RecoverPoint appliances Gen 4
Solution software
Table 4 describes the software used in this solution. Table 4. Solution software
Software Version
Ecologic Analytics MDMS 2.8 Database 10g R2 Enterprise Edition (64 bit)
with Partitioning option 10.2.0.5
EMC PowerPath 5.5
EMC RecoverPoint 3.4
EMC NetWorker (server and storage node) Windows 2008 64 bit operating system NetWorker 7.6 SP1
EMC Data Domain (DD) Boost 2.2.1
Table 5 shows the Oracle configuration parameters used in this solution. Table 5. Oracle database initialization parameters
Parameter Setting
db_cache_size 8 G
pga_aggregate_target 4 G shared_pool_size 1 G pga_aggregate_target 2 G Database block size 16 K
Table 6 shows the Oracle configuration settings. Table 6. Oracle configuration settings
Configuration Setting
Oracle Flashback On Oracle Archive log On
Redo logs 6 x 16 GB
Total database size 5.246 TB
Oracle version 10.2.0.5 Enterprise Edition -64bit with Partitioning Option
Oracle configuration
During the tests, the application and database tiers ran on one server, which is consistent with many like-sized utility customers in production.
Table 7 describes the key MDMS processing operations. Table 7. MDMS processing operations
Function Description
Master Data Synchronization Provisioning of meters, accounts, locations, and routes. Inbound interfaces from CIS, outbound to AMI.
Register Data Consumption Consumption of register (anchor) reads from AMI
Interval Data Consumption Consumption of interval load profile (LP) reads from AMI
Register Data VEE (validation, estimation, editing)
Validation, estimation, and editing of register data
Interval Data VEE Validation, estimation, and editing of interval data
Billing Determinant Creation Outbound interface to CIS containing billing determinant information by billing cycle Meter Reading Analytics Calculates usage baselines for a single
location or similar customers as defined by rate, geography, or other variables
Reporting Daily report generation MDMS
Figure 6 shows the backup environment used in this solution.
Figure 6. Backup environment
EMC NetWorker provides tight integration with Oracle RMAN, which relieves the burden of backup from the database administrator while allowing the administrator to retain control of the restore process.
The role of NetWorker server is to communicate with the NetWorker client on the Oracle server, initiate the RMAN backup script, and manage and monitor activity on the storage node and client.
The NetWorker storage node writes the data to the Data Domain DD690 during backup activity and reads data from the Data Domain DD690 during a restore. The storage node also sends the tracking information back to the NetWorker server. The Data Domain DD 690 stores the data that is being backed up and restored. Four Data Domain devices are configured to align with four RMAN channels defined in the RMAN script.
Backup environment
The RecoverPoint configuration includes two local and two remote RecoverPoint appliances in a four-node cluster. This solution uses a single consistency group configured for continuous remote replication of seven LUNs representing
approximately 9.3 terabytes of capacity. The LUNs include all the file systems needed to bring Oracle up on the remote server, both the binaries and the data. This solution does not replicate the database flashback or backup LUNs.
Figure 7 shows the EMC RecoverPoint environment.
Figure 7. EMC RecoverPoint environment Disaster Recovery
Testing and validation
EMC ran the same operations that a utility would carry out on a daily basis as described in the section entitled “MDMS configuration” on page 18.
Application performance was gauged by the total duration of the operations; a shorter duration indicates better performance. Log files identifying the start and stop times of each operation and Oracle database Automatic Workload Repository (AWR) reports were created during the tests. Performance statistics were also collected from the storage system during the tests. These statistics were subsequently correlated with the operation execution times so that an analysis of I/O characteristics could be performed. This resulted in a great deal of insight into the application’s I/O profile. We ran these operations for three consecutive real-world calendar days.
EMC and Ecologic Analytics ensured that the testing environment was as realistic as possible by:
Enabling Oracle Flashback Turning on Oracle Archive Log Using eight server CPU cores
Using eight MDMS application processing threads
Storing all data on the same platform (rather than placing data files on one array and redo logs on another)
Applying three percent missing register data; one percent missing interval data
In this solution, the EMC solutions team tested five different storage configurations: FC Baseline storage configuration—The baseline storage configuration
consisted of traditional all Fibre Channel drives following best practices for Oracle databases.
FC and FAST Cache—The second configuration adds FAST Cache on top of the all-FC baseline.
FAST VP pools with three storage tiers (EFDs, FC, and SATA)—The third storage configuration leverages FAST virtual pools (VP) with three storage tiers. FAST VP pools with two storage tiers (FC and SATA) and FAST Cache—The
fourth storage configuration leverages both FAST VP and FAST Cache. Within FAST VP, this configuration uses only two storage tiers—FC and SATA. FAST VP pools with three storage tiers (EFDs, FC, and SATA) and FAST
Cache—The fifth storage configuration also leveraged both FAST VP and FAST Cache, but with three storage tiers, adding an additional tier of Flash. Testing
methodology
Testing elements
Figure 8 shows the baseline storage configuration for the tests performed. As the name suggests, this configuration is the comparison point for the four subsequent system designs.
Figure 8. FC Baseline storage configuration
Each volume has of two or more disks, depending on the protection scheme and purpose of the volume. Database volumes U15 and U16 contain the database files, which are placed on RAID 5 and optimized for read performance for the given capacity. According to EMC’s best practices,3 RAID 5 has excellent random read performance and has the best ratio of available capacity for parity-protected RAID groups.
Oracle redo log volumes on U13 and U17 and the Oracle flash recovery volume
(flashback) on U18 are built specifically for write performance by leveraging RAID 1/0.
Fibre Channel
Baseline storage configuration
EMC does not recommend placing Oracle data files and log files in the same disk group because of4,5:
Reliability. The logs play a pivotal role in Oracle database recovery. In the event of data file corruption, the database administrator can go back to an older copy of the data file and apply the logs. Similarly, if logs are lost, the Oracle database can guarantee zero or minimal data loss if online redo logs are located on a different set of spindles.
I/O type and size. The I/O profile of log files tends to be highly sequential. By mixing log files with data files, the sustained write bandwidth of a drive drops as the spindle begins to seek more often.
Performance. Redo log writes are synchronous and are required to complete in the least amount of time. By placing them on separate storage devices, the commit writes do not have to share the LUN I/O queue with large asynchronous buffer cache checkpoint I/Os. Having the logs on their own devices makes it possible to use one RAID protection type for data files (RAID 5) and another for logs (RAID 1/0).
The second configuration adds FAST Cache on top of the all-FC baseline. It uses the same FC configuration as the baseline with the addition of four Flash drives, serving as a second-level cache. The team used EMC internal performance analysis tools to determine the size of the FAST Cache.
The benefits of FAST Cache are realized by the random read profile of the database files. The I/O patterns for Oracle logs, both online and archive, tend to be highly sequential in nature. The write cache of the storage system can easily coalesce these small writes into bigger back-end I/O stripes, allowing the more cost-effective
spinning hard disks to handle that load and freeing the higher-performing flash drives for large database files.
Flash drives in FAST Cache must be provisioned in pairs and must be assigned RAID 1 protection according to EMC’s best practices.6 FAST Cache uses RAID 1 (mirroring) to provide both read and write caching.
4 “Deploying Oracle Database on EMC VNX Unified Storage.” May 2011. Page 24.
5 Implementing FAST VP and Storage Tiering for Oracle Database 11g and EMC Symmetrix VMAX. April 2011. Page 26. 6 “EMC Unified Storage Best Practices for Performance nad Availability Common Platform and Block Storage 31.0.”
Applied Best Practices. Revised: 06/23/2011. Page 45-46. FC and FAST Cache
Figure 9 shows the meter data management disk pool layouts used for CLARiiON FAST Auto-Tiering and FAST Cache testing. All three configurations with FAST VP use three disk pools, one for the database, one for the redo logs, and one for the archive data. A pool is a collection of disks that can share the storage load, and the distribution of that load depends on the nature of the data. As in this scenario, multiple disk technologies, or tiers (Flash, FC, and SATA) can exist in the same pool.
In this diagram, the redo and archive pools are presented only once, since they are identical for all three configurations. The diagram also presents disk speeds, RAID protection, and a view from the perspective of the database file system.
During FAST Cache testing, 400 GB of additional Flash Cache was assigned to the database pool (four 200 GB, RAID 1 protected disks).
FAST VP configurations
Test results
EMC FAST VP with three tiers of disks proved to be the best configuration when evaluating both price and performance. This section describes the results and provides more context behind the recommendations in this paper.
There are different ways to view the testing results of this solution. This section includes the results and describes the various ways to view them, including by:
Performance Price
Price/performance I/Os per second
Additional drill down into IOPS results
Figure 10 shows the performance of each configuration tested. A lower duration is better performance. The three FAST Suite configurations all show improved
performance over the FC baseline. The FAST three-tier with FAST Cache configuration yielded the best performance at 21 percent better than the FC baseline. This
demonstrates that FAST Cache was able to optimize additional portions of the workload with its real-time caching algorithms.
Figure 10. Performance results Overview
Often when performance numbers are stated, the next question that comes up is price. It is usually not too difficult to architect a system that performs well—if money is no object. Therefore, another comparison of the configurations is price. Figure 11 shows a normalized price comparison of the storage system with the FC baseline configuration as the benchmark. What stands out in this chart is the FAST three-tier with FAST Cache configuration, which is 20 percent more than the FC baseline. However, the other two configurations cost less and provide better performance than the FC baseline.
Figure 11. Normalized price Price
Price and performance are typically used as product measures, and the term “price performance” is often used to combine these two. From a mathematical standpoint, these two can be combined by multiplying them together. Since lower is better for each measurement, a lower product of the two would indicate a better choice, as shown in Figure 12. This chart shows that the FAST three-tier configuration provides the best price performance.
Figure 12. Price performance Price performance
The results discussed thus far focus on performance (measured by duration of the jobs) and the price of the various configurations. Another performance metric often used with database applications is IOPS, or I/O per second. Figure 13 and Figure 14 below show the average and maximum IOPS for each of the configurations. The FAST Suite configurations always surpass the FC baseline. Note that these numbers are not the maximum that the storage system is capable of; they are simply what we
recorded during the tests.
Figure 13. Average IOPS I/Os per second
Another area in which the lab team drilled down was the IOPS achieved when running each job. Figure 15 shows an example of these results. Again, the FAST three-tier configuration out-performed the FC baseline (yet costs less).
Figure 15. Additional IOPS results
The focus of this effort was to show the backup and recovery functionality of the MDM database using EMC NetWorker and Data Domain. The total amount of data that was backed up was approximately 5.3 TB for each test run.
The results of the backup and recovery tests validate the functionality of the EMC NetWorker to Data Domain solution within the 500,000-meter Ecologic Analytics environment.
In a standard RecoverPoint installation, the customer requirements for data
availability are used to calculate the amount of data that can be acceptable lost due to an outage (RPO). The same requirements are used to determine the maximum amount of time an application can be unavailable before adversely affecting the business function (RTO).
Using these calculations, we developed a schedule and procedure for “application consistent recovery,” where an application is quiesced and a RecoverPoint snapshot (bookmark) is created. This provides a known “good point in time” to from which to restore the data. Next, we develop a procedure to fail over the application to a secondary site and continue with normal business operations.
Additional drill down IOPS results
Backup and Restore– EMC NetWorker and EMC Data Domain
Disaster recovery with EMC
By default, RecoverPoint creates a bookmark, approximately every minute, that can be used to perform “crash-consistent recovery,” where you recover to the point in time closest to an outage then check to see if the data is still valid. Using this method, you can recover data changes that occurred after an application-consistent bookmark but before an outage.
When testing backup and recovery, the solutions team performed a controlled shutdown using the following procedure:
1. Shut down the database.
2. Unmount the file systems from the production server.
3. Fail the consistency group over to the disaster recovery (DR) server with RecoverPoint.
4. Mount the replicated file systems on the DR server. 5. Start the database on the DR server.
In this test scenario, it took two minutes and thirty seconds to shut down the primary database, fail over the RecoverPoint volumes to the DR site, and start the secondary database server.
Conclusion
The team drew the following conclusion from the solution testing:
When built on EMC storage systems with FAST technology, the Ecologic MDMS application can consume 2,215 register and interval meter reads per second, per thread and 5,088 register and interval VEE per second, per thread. EMC and EcoLogic Analytics provide a cost-effective platform for mid-market utilities deploying smart meters today.
Interval data consumption, validation, estimation, and editing consume most of the processing activity within the MDMS application, so improving the performance and cost efficiency of these operations provides the maximum return on investment (ROI).
Using Ecologic MDMS application-threaded processing with EMC FAST VP as the storage configuration provides the best performance for the cost.
While all of the FAST and FAST Cache configurations improve performance over the FC baseline, the configuration with FAST VP with three storage tiers
provides the best price-performance. This configuration improves performance by six percent over the FC baseline configuration and, at the same time, is 15 percent more cost effective.
When performance is the top priority, FAST VP with three storage tiers and FAST Cache is the best choice.
Both FAST VP and FAST Cache help the Ecologic MDMS because a large percentage of the “working set,” or data responsible for a majority of the I/O, can be serviced by a reasonably priced Flash storage tier. A great benefit of the advanced FAST VP and FAST Cache technologies is that the storage system automatically places the most active data on the highest storage tier, while placing the least frequently accessed data on the lowest tier.
EMC has successfully built and demonstrated a backup architecture that uses EMC NetWorker with Oracle RMAN to back up the Ecologic MDMS database to Data Domain.
This solution easily meets disaster recovery requirements by using EMC RecoverPoint, which provides an RPO of 2.5 minutes and an RTO of 2.5 minutes.
RTO: Four times faster than the original 10-minute goal. RPO: Twenty-four times faster than the original one-hour goal.
References
For additional information, see the white papers listed below (available on EMC Powerlink). If you do not have the required Powerlink access, contact your EMC representative.
EMC CLARiiON Best Practices for Performance and Availability: Release 30.0 Firmware Update—Applied Best Practices (Part Number: h5773.4)
Leveraging EMC CLARiiON CX4-480 with Enterprise Flash Drives for Oracle Database Deployments—Applied Technology (Part Number: h5967)
Leveraging EMC FAST Cache with Oracle OLTP Database Applications—Applied Technology (Part Number: h8018)
Implementing FAST V P and Storage Tiering for Oracle Database 11g and EMC Symmetrix VMAX (Part Number: h8131)
Deploying Oracle Database on EMC VNX Unified Storage—Best practices for provisioning storage and leveraging storage efficiency features (Part Number: h8242)
EMC Unified Storage Best Practices for Performance and Availability Common Platform and Block Storage 31.0—Applied Best Practices (Part Number: h8268) Leveraging EMC Fully Automated Storage Tiering (FAST) and FAST Cache for SQL
