EMC VSPEX FOR VIRTUALIZED ORACLE DATABASE 11 g OLTP
Enabled by EMC Next-Generation VNX and EMC Backup
EMC VSPEX
Abstract
This Implementation Guide describes the high-level steps and best practices required to implement the EMC® VSPEX® Proven Infrastructure for Virtualized Oracle
Databases on a VSPEX Private Cloud with VMware vSphere enabled by EMC Next- Generation VNX® and E MC Backup. It describes implementation for the FAST™ Cache and FAST VP® features and explains their usage in VNX storage systems. Usage guidelines and major customer benefits are also included.
October 2013
Copyright © 2013 EMC Corporation. All rights reserved. Published in the USA.
Published October 2013
EMC believes the information in this publication is accurate as of its publication date.
The information is subject to change without notice.
The information in this publication is provided “as is”. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.
EMC2, EMC, and the EMC logo are registered trademarks or trademarks of EMC Corporation in the United States and other countries. All other trademarks used herein are the property of their respective owners.
For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.
EMC VSPEX for Virtualized Oracle Database 11g OLTP Enabled by EMC Next- Generation VNX Implementation Guide
Part Number H12064.1
Contents
Chapter 1 Introduction 9
Purpose of this guide ... 10
Business value ... 10
Scope ... 11
Audience ... 11
Terminology... 12
Chapter 2 Before You Start 13 Overview ... 14
Pre-deployment tasks ... 14
Deployment workflow ... 14
Deployment prerequisites ... 15
Planning and sizing Oracle Database 11g R2 ... 17
Essential Reading ... 17
Design guide... 17
Solution overviews ... 17
VSPEX Proven Infrastructure ... 17
VMware documentation ... 17
Backup and recovery ... 17
Chapter 3 Solution Overview 19 Overview ... 20
Solution architecture ... 20
Key components ... 21
Introduction ... 21
EMC VSPEX ... 22
Oracle Database 11g ... 24
VMware vSphere 5.1 ... 25
VMware vSphere HA ... 25
VMware vSphere Distributed Resource Scheduler ... 25
VMware vSphere PowerCLI ... 25
EMC Next-Generation VNX ... 25
VNX performance ... 27
Virtualization management ... 28
Red Hat Enterprise Linux 6.3 ... 30
EMC backup and recovery solutions ... 30
Chapter 4 Solution Implementation 31 Overview ... 32
Setting up the physical components ... 32
Implementing the network ... 33
Implementing the storage array ... 34
Overview ... 34
Setting up initial VNX configuration ... 35
Provisioning storage for VMFS example ... 35
Provisioning storage for NFS example ... 36
Configuring FAST Cache (optional) ... 40
Configuring FAST VP (optional) ... 41
Example storage layout ... 43
Implementing ESXi and vCenter ... 44
Preparing the Oracle Database 11g R2 virtual machine template ... 45
Overview ... 45
Creating the Oracle Database 11g R2 virtual machine ... 45
Completing the Oracle install pre-requisites ... 46
Installing the Oracle Database 11g R2 binaries and patches ... 48
Preparing the Oracle Database 11g R2 virtual machine template ... 50
Customizing the VMware template ... 50
Deploying the Oracle Database 11g R2 virtual machine ... 51
Overview ... 51
Deploying new virtual machines ... 51
Cloning the Oracle binaries ona new virtual machine ... 52
Determining the database prerequisites ... 53
Configuring the database file layout for NFS ... 54
Configuring the database file layout for Block ... 56
Configuring the initialization parameters ... 57
Enabling the HugePages setting ... 58
Creating the database using DBCA ... 58
Performing the database post creation activities ... 59
Implementing backup and recovery ... 61
Overview ... 61
Chapter 5 Solution Verification 63 Verifying the baseline infrastructure ... 64
Overview ... 64
Verifying the ESXi functionality ... 64
Verifying the solution component redundancy ... 64
Verifying the Oracle Database configuration ... 65
Using the health monitoring tools ... 65
Verifying the Oracle Database... 66
Overview ... 66
Understanding the key metrics ... 66
Reviewing the Oracle dataset ... 68
Backup and recovery verification methodology ... 68
Chapter 6 Reference Documentation 69 EMC documentation ... 70
EMC ... 70
Oracle ... 70
VMware ... 70
Other documentation ... 71
Links ... 71
Appendix A Configuration Worksheet 72 Pre-configuration worksheet for VSPEX ... 73
Appendix B: Example Virtual Disk Layout 77 Example Virtual Disk Layout of Oracle Database on the VMFS Datastore ... 78
Figures
Figure 1. Architecture of the validated infrastructure ... 21
Figure 2. VSPEX Proven Infrastructure ... 22
Figure 3. Next-Generation VNX with multicore optimization... 27
Figure 4. Active/active processors increase performance, resiliency, and efficiency ... 28
Figure 5. New Unisphere Management Suite ... 29
Figure 6. Redundant 10 GbE network configuration ... 34
Figure 7. Redundant Fibre Channel storage network... 34
Figure 8. Oracle Database 11g R2 storage elements ... 35
Figure 9. Example storage layout of Oracle database on VMware ... 36
Figure 10. Configuring EMC Unified Storage Management in vSphere Client ... 37
Figure 11. Configuring NFS Storage on VNX for Oracle VMs ... 37
Figure 12. Configuring VNX storage pools ... 38
Figure 13. Unisphere Create File System Wizard ... 38
Figure 14. EMC Unisphere Create NFS Export Wizard ... 39
Figure 15. Enabling FAST Cache in the Storage Pool Properties dialog ... 40
Figure 16. FAST VP creating storage pool... 41
Figure 17. Tier status before data movement ... 42
Figure 18. FAST VP in a steady state ... 42
Figure 19. Example virtualized oracle storage layout for VSPEX ... 43
Figure 20. Oracle Universal Installer – Install choose database software only ... 48
Figure 21. Enable Oracle Direct NFS client ... 49
Figure 22. Example of script root_clone.sh ... 49
Figure 23. Example of script cloneHome.sh ... 49
Figure 24. Virtual machine template example ... 50
Figure 25. vSphere custom specification example ... 50
Figure 26. Example PowerCLI script to deploy a virtual machine from template .... 52
Figure 27. Example PowerCLI script to clone and reattach the ORACLE_HOME ... 53
Figure 28. Oracle Net Configuration Assistant utility... 53
Figure 29. Example /etc/mtab NFS mount options ... 54
Figure 30. Oranfstab example ... 55
Figure 31. Example of /etc/fstab ... 57
Figure 32. Oracle Database Configuration Assistant setup screen ... 59
Figure 33. Oracle startup/shutdown script /etc/init.d/dbora ... 60
Figure 34. Swingbench interface load panel screenshot ... 67
Figure 35. Swingbench order entry screen for workload simulation ... 67
Tables
Table 1. Terminology... 12
Table 2. Tasks for pre-deployment ... 14
Table 3. Virtualized oracle deployment workflow for VSPEX ... 14
Table 4. Deployment prerequisites checklist ... 16
Table 5. Reference virtual machine characteristics ... 23
Table 6. Mapping the Oracle sizing model to the VSPEX Reference Virtual Machine ... 23
Table 7. Tasks for physical setup ... 32
Table 8. Tasks for switch and network configuration ... 33
Table 9. Tasks for storage configuration ... 34
Table 10. Steps for installing and configuring ESXi and vCenter ... 44
Table 11. Virtual machine template specification ... 45
Table 12. Oracle install prerequisites ... 46
Table 13. RHEL 6 RPM packages for Oracle Database 11g R2 ... 47
Table 14. Installing the Oracle Database binaries and patches ... 48
Table 15. Oracle Database 11g R2 reference virtual machine model ... 51
Table 16. Database file layout for NFS ... 54
Table 17. Tasks for verifying the VSPEX Proven Infrastructure installation ... 64
Table 18. Tools for monitoring the solution ... 65
Table 19. General operations ... 66
Table 20. Schema tables and indexes ... 68
Table 21. Common server information ... 73
Table 22. ESXi server information ... 73
Table 23. Array information ... 74
Table 24. Network infrastructure information ... 74
Table 25. VLAN information ... 74
Table 26. Service accounts ... 75
Table 27. Example Virtual Disk Layout ... 78
Chapter 1 Introduction
This chapter presents the following topics:
Purpose of this guide ... 10
Business value ... 10
Scope ... 11
Audience ... 11
Terminology ... 12
Purpose of this guide
EMC® VSPEX® Proven Infrastructures are optimized for virtualizing business-critical applications. VSPEX provides modular solutions built with technologies that enable faster deployment, more simplicity, greater choice, higher efficiency, and lower risk.
VSPEX provides partners with the ability to design and implement the virtual assets required to support a fully integrated virtualized solution for Oracle relational database management systems on a VSPEX private cloud infrastructure.
The VSPEX for virtualized Oracle infrastructure provides customers with a modern system, capable of hosting a virtualized database solution that is scalable and delivers a constant performance level. This solution uses VMware vSphere to provide virtualization, with EMC Next-Generation VNX® providing storage, and EMC Avamar® and EMC Data Domain® for backup. The compute and network components, while vendor-definable, are designed to provide redundancy and sufficient power to handle the processing and data needs of the virtual machine environment.
This Implementation Guide describes how to implement, using best practices, the VSPEX Proven Infrastructure for virtualized Oracle Databases on a VSPEX Private Cloud with VMware vSphere.
Business value
Database management system software continues to be the dominant software used to manage data in nearly all commercial segments. This growth is expected to
continue despite the increasing market share of other data management tools. This growth is expected to accelerate as customers continue to diversify their
infrastructures and supporting technologies and drive towards more hardware and software appliances and configurations.
This VSPEX Proven Infrastructure is focused on helping EMC partners understand the value that the VNX series, EMC backup and recovery systems, and Oracle bring to customers who often have growing, isolated IT environments running server-centric applications and who face increasing Oracle backup and recovery issues.
This VSPEX solution is designed to meet the customer’s Oracle database challenges while enabling customers to grow in performance, scalability, reliability, and
automation. By consolidating their database applications on VNX, they can consolidate onto a single centralized storage platform that enables them to more effectively manage the exploding growth in data that is challenging businesses today.
This solution has been sized and proven to:
Deploy faster, saving time and effort with Proven solutions
Increase performance and scalability out of the box
Reduce the customer’s backup storage requirements and costs
Accommodate backup windows
Enable fast disk-based recovery
Scope
This Implementation Guide describes the high-level steps required to deploy the VSPEX solution for Oracle RDBMS and Online Transaction Processing (OLTP) systems.
This VSPEX application solution is based on the current VSPEX private cloud with VMware solution and has been modified to support Oracle Database requirements.
This solution uses the VSPEX standard or “reference virtual machine” building blocks of vCPU, memory, and storage and I/O configurations, and describes the
implementation based on three (small, medium, and large) different database workload thresholds.
This Implementation Guide assumes that a VSPEX Proven Infrastructure already exists in the customer’s environment, which includes VMware vSphere,VNX, and EMC backup and recovery systems.
Audience
This guide is intended for internal EMC personnel and qualified EMC VSPEX partners.
This guide assumes that the VSPEX partners who intend to deploy this solution are:
Qualified by EMC to sell, install, and configure the EMC VNX family of storage systems
Qualified to sell, install, and configure the network and server products required for VSPEX Proven Infrastructures
Certified for selling VSPEX Proven Infrastructures
Partners implementing this solution should also have the necessary technical training and background to install and configure:
VMware vSphere 5.1
Red Hat Enterprises Linux 6.3
Oracle Database 11g R2 or above
EMC next-generation backup, which includes EMC Avamar and EMC Data Domain
This document provides external references where applicable. EMC recommends that readers are familiar with these documents. For details, refer to the section entitled EMC documentation.
Terminology
Table 1 lists the terminology used in the guide.
Table 1. Terminology
Term Definition
AWR Automatic Workload Repository DNFS Oracle Direct NFS client
DNS Domain name system
eMLC Enterprise multilevel cell
FAST VP Fully Automated Storage Tiering for Virtual Pools FQDN Fully Qualified Domain Name
IOPS Input/output operations per second FRA Fast Recovery Area (Oracle)
NFS Network File System
NL-SAS Near-line serial-attached SCSI OLTP Online transaction processing Oracle EE Oracle Enterprise Edition Oracle SE Oracle Standard Edition
PowerCLI A Windows PowerShell interface to the VMware vSphere and vCloud APIs
Reference virtual machine
Represents a unit of measure for a single virtual machine to qualify the compute resources in a VSPEX Proven Infrastructure
SGA System global area
Statspack Oracle database monitoring and reporting utilities TPS Transactions per second
VDM Virtual Data Mover
VMDK VMware virtual machine disk VMFS VMware virtual machine file system
Chapter 2 Before You Start
This chapter presents the following topics:
Overview ... 14
Pre-deployment tasks ... 14
Deployment workflow ... 14
Deployment prerequisites ... 15
Planning and sizing Oracle Database 11g R2 ... 17
Essential Reading ... 17
Overview
Before you layer the Oracle OLTP on a VSPEX private cloud, EMC recommends that you complete the pre-deployment tasks listed in Table 2.
Pre-deployment tasks
Pre-deployment tasks include procedures that do not directly relate to environment installation and configuration, but whose results are needed during installation.
Examples of pre-deployment tasks include the collection of hostnames, IP addresses, VLAN IDs, license keys, installation media, and so on. Perform these tasks before visiting the customer to decrease the time required on site.
Table 2. Tasks for pre-deployment
Task Description Reference
Gather documents Gather the related documents listed in Essential Reading. These are used throughout the text of this document to provide details on setup procedures and deployment best practices for the various components of the solution.
Essential Reading
Gather tools Gather the required and optional tools for the deployment. Use Table 4 to confirm that all equipment, software, and appropriate licenses are available before the
deployment process.
Deployment prerequisites
Gather data Collect the customer-specific configuration data for networking, naming, and required accounts. Enter this information into the Customer Oracle Database 11g R2
configuration worksheet for reference during the deployment process.
Appendix A Configuration Worksheet
Deployment workflow
EMC recommends that you follow the process flow in Table 3 to design and
implement your VSPEX Proven Infrastructure for virtualized Oracle Database 11g R2 solution.
Table 3. Virtualized oracle deployment workflow for VSPEX Step Action
1 Use the VSPEX qualification worksheet to collect user requirements. The qualification worksheet is in Appendix A of the companion ”EMC VSPEX FOR VIRTUALIZED ORACLE DATABASE 11g OLTP Design Guide”
Step Action
2 Use the VSPEX Sizing Tool to determine the recommended VSPEX Proven Infrastructure for a virtualized Oracle Database 11g R2 solution.
For more information about the VSPEX Sizing Tool, refer to the VSPEX Sizing Tool on the EMC Business Value Portal.
Note You need to register the first time you access the tool. If the VSPEX Sizing Tool is not available, you can manually size the application using the sizing guidelines in Appendix B of the companion Design Guide.
3 To determine the final design for the VSPEX Proven Infrastructure for virtualized Oracle Database 11g, refer to the companion Design Guide.
Note Ensure that all application requirements are considered, not just this application.
4 To select and order the right solution, refer to the appropriate VSPEX Proven Infrastructure document in Essential Reading.
5 To deploy and test your VSPEX solution, refer to this guide.
Deployment prerequisites
Table 4 itemizes the hardware and software requirements to configure the solution.
For additional information, refer to the hardware and software tables in the relevant document in the Essential Reading section.
Table 4. Deployment prerequisites checklist
Require-
ment Description Version Reference notes
Hardware
Physical servers: sufficient physical server capacity to host the required number of virtual machines as recommended by the companion Design Guide and VSPEX Sizing Tool.
Reference Architecture:
EMC VSPEX Private Cloud: VMware vSphere 5.1 for up to 1000 Virtual Machines
EMC Avamar 6.1 hardware and licenses EMC Data Domain hardware and licenses VMware vSphere 5.1 servers to the host virtual
infrastructure servers. This requirement may be covered in the existing infrastructure.
Networking: switch port capacity and capabilities as required by the virtual server infrastructure.
EMC VNX: multiprotocol storage array with the required disk layout.
EMC Backup: Avamar 6.1, Data Domain
Software
VNX OE for file Release
8.0.10000.3 4653
EMC online support
VNX OE for block Release
05.33.000.3 .653 EMC VSI for VMware vSphere: Unified Storage
Management 5.4
EMC VSI for VMware vSphere: Storage Viewer 5.4
VMware ESXi installation media 5.1
VMware vCenter Server installation media 5.1
EMC Avamar 6.1 in server
and client versions
Red Hat Enterprise Linux 6.3 Red Hat
Enterprise Linux 6.3 media
www.redhat.com
Licenses
VMware vSphere license keys. This may be
covered in the VSPEX Proven Infrastructure. 5.1 www.vmware.com
Red Hat Enterprise Linux 6.3 www.redhat.com
Oracle Database 11g R2 11.2.0.3.6 www.oracle.com
Planning and sizing Oracle Database 11g R2
To plan and size your Oracle Database 11g R2 deployment on the VSPEX
infrastructure, you should follow the recommendations in the companion Design Guide. Size the storage accordingly to information obtained from the VSPEX for Virtualized Oracle 11g Qualification worksheet as detailed in Appendix A and B of the companion Design Guide.
Essential Reading
Before implementing the solution described in this document, EMC recommends that you read the following documents, available from the VSPEX space in the EMC Community Network or from EMC.com or the VSPEX Proven Infrastructure partner portal.
Design Guide: EMC VSPEX for Virtualized Oracle Database 11g OLTP
EMC VSPEX Server Virtualization for Midmarket Businesses
EMC VSPEX Server Virtualization for Small and Medium Businesses
Reference Architecture: EMC VSPEX Private Cloud: VMware vSphere for up to 1,000 Virtual Machines
White Paper: EMC Avamar Backup for Oracle Environments
White Paper: EMC Avamar Backup with Data Domain VMware VSphere Product Documentation
Refer to the following backup and recovery papers:
White Paper: EMC Avamar Backup for Oracle Environments
White Paper: EMC Avamar Backup with Data Domain
White Paper: EMC Backup and Recovery Options for VSPEX for Virtualized Oracle 11gR2 Design and Implementation Guide i
Design guide Solution overviews
VSPEX Proven Infrastructure
VMware documentation Backup and recovery
Chapter 3 Solution Overview
This chapter presents the following topics:
Overview ... 20 Solution architecture ... 20 Key components ... 21
Overview
This chapter provides an overview of the VSPEX Proven Infrastructure for Oracle Database 11g and the key technologies used in this solution. The solution described in this Implementation Guide includes servers, storage, network components, and Oracle Database 11g components.
The solution enables customers to quickly and consistently deploy a virtualized Oracle Database 11g in the VSPEX Proven Infrastructure. The reference architecture will consume the reference virtual machine resources, based on the sizing guidance in the VSPEX Proven Infrastructure, and combine with additional storage for Oracle Database 11g application data.
This Implementation Guide can help EMC personnel and qualified EMC VSPEX
Partners to deploy a simple, effective, and flexible Oracle Database 11g solution on a VSPEX Proven Infrastructure for their customers.
Solution architecture
Figure 1 shows the architecture that characterizes the infrastructure validated for an Oracle Database 11g overlay on a VSPEX infrastructure. To validate this solution, we1:
Deployed all Oracle Database 11g servers as virtual machines on VMware vSphere 5.1.
Used the VSPEX sizing tool for Oracle Database 11g to determine the number of, and the detailed compute resources for, each Oracle Database 11g
database. Figure 1 displays an example with three Oracle sizing options (small, medium, and large). Use the sizing tools provided with this solution to size your customer’s environment and choose the options that best suit your customer.
Determined the recommended storage layout for Oracle Database 11g and the virtual infrastructure pool in the VNX series storage arrays (using the VSPEX sizing tool).
Note: The minimum Oracle version for this solution is 11.2.0.3. We refer to this as 11gR2 throughout this document.
1 In this paper, "we" refers to the EMC Solutions engineering team that validated the solution.
Figure 1. Architecture of the validated infrastructure
Key components
This section provides an overview of the key technologies used in this solution:
EMC VSPEX
Oracle Database 11g
VMware vSphere 5.1
VMware vSphere HA
vSphere Distributed Resources Scheduler
VMware vSphere PowerCLI
EMC Next-Generation VNX®
Red Hat Enterprise Linux 6.3
EMC Avamar
EMC Data Domain Introduction
EMC has joined forces with the industry’s leading providers of IT infrastructure to create a complete virtualization solution that accelerates the deployment of private cloud technologies. Built with best-of-breed technologies, VSPEX enables faster deployment, more simplicity, greater choice, higher efficiency, and lower risk.
VSPEX Proven Infrastructure, as shown in Figure 2, is a modular, virtualized system validated by EMC and delivered by EMC partners. VSPEX includes a virtualization layer, server, network, and storage, designed by EMC to deliver reliable and predictable performance.
Figure 2. VSPEX Proven Infrastructure
VSPEX provides the flexibility to choose the best network, server, and virtualization technologies that fit a customer’s environment to create a complete virtualization solution.
VSPEX provides a virtual infrastructure for customers looking to gain the simplicity of a truly converged infrastructure, while gaining flexibility in individual components of the stack. VSPEX solutions, proven by EMC, are packaged and sold exclusively by EMC channel partners. VSPEX provides channel partners with more opportunity, a faster sales cycle, and end-to-end enablement. By working even more closely together, EMC and its channel partners can now deliver an infrastructure that accelerates the journey to the cloud for even more customers.
EMC VSPEX
Reference virtual machine
To simplify the virtual infrastructure discussion, the VSPEX solution has defined a typical customer workload (described in this section) as a reference virtual machine.
For VSPEX solutions, we define the reference virtual machine as a measure unit of a single virtual machine to qualify the compute resources in the VSPEX virtual
infrastructure. Table 5 lists the characteristics of this virtual machine.
Table 5. Reference virtual machine characteristics
Characteristic Value
Virtual processors per virtual machine 1
RAM per virtual machine 2 GB
Available storage capacity per virtual machine 100 GB I/O operations per second (IOPS)
per virtual machine
25
I/O pattern Random
I/O read/write ratio 2:1
VSPEX for virtualized Oracle sizing model
Scale-up testing formed part of the validation process. We used a standard compute- sizing model for Oracle, which simplified and standardized the validation testing. It also enabled us to identify the configuration required to run a TCP-C like OLTP
database workload with a 60:40 read/write ratio, yielding acceptable response times.
Table 6 shows how we mapped the Oracle sizing model to the VSPEX reference virtual machine.
Table 6. Mapping the Oracle sizing model to the VSPEX Reference Virtual Machine
Oracle model Resources
Equivalent reference virtual machine Small—virtual
machine for up to 150 users
Compute requirements:
2 vCPU
8 GB memory
Storage requirements (OS & Oracle binaries):
100 GB
25 IOPS
4
Medium—virtual machine for up to 250 users
Compute requirements:
4 vCPU
16 GB of memory
Storage requirements (OS & Oracle binaries):
100 GB
25 IOPS
8
Oracle model Resources
Equivalent reference virtual machine Large—virtual
machine for more than 250 Users
Compute requirements:
8 vCPU
32 GB of memory
Storage requirements (OS & Oracle binaries):
100 GB
25 IOPS
16
We calculated the database storage I/O thresholds and capacity separately from those required by the VSPEX reference virtual machine.
Oracle Database 11g is available in a variety of editions tailored to meet the business and IT needs of an organization. In this solution we will be considering
Oracle Database 11g Release 2 Standard Edition (SE)
Oracle Database 11g Release 2 Enterprise Edition (EE)
Oracle Database 11g SE is an affordable, full-featured data management solution that is ideal for all companies. It is available on single or clustered servers and can be licensed on a maximum capacity of four processor sockets, regardless of core count.
The SE license includes Oracle Real Application Clusters (RAC) as a standard feature with no additional cost.
Oracle Database 11g EE delivers industry-leading performance, scalability, security, and reliability on a choice of clustered or single servers running Windows, Linux, or UNIX. It supports advanced features, either included or as extra-cost options, that are not available with Oracle Database 11g SE. These include security features such as Virtual Private Database and data warehousing options such as partitioning and advanced analytics. Oracle Database 11g Release 2 EE extends the processor- licensing model for multi-core processors and is priced using the following formula:
(number of Processors) x (number of cores) x (Oracle Processor Core Factor) For example, two 10-core Intel Xeon Processor E7-2870s (with an Oracle Processor Core Factor of 0.5) are licensed as follows
Oracle Database 11g Release 2 SE: 2 processor socket SE licenses
Oracle Database 11g Release 2 EE: 2 x 10 x 0.5 = 10 EE licenses
The Oracle Database 11g R2 edition can affect the licensing cost and the size and number of VMware ESXi clusters you can configure. This affects how you place and manage the virtual machines. Refer to the document entitled Design Guide for EMC VSPEX for Virtualized Oracle Database 11g OLTP, in the DRS Host Affinity and Oracle processor licensing section, for more information on virtualization and Oracle processor licensing.
Oracle Database 11g
VMware vSphere 5.1 abstracts applications and information from the complexity of underlying infrastructure through comprehensive virtualization of server, storage, and networking hardware. This transformation creates fully functional virtual machines that run isolated and encapsulated operating systems and applications just like physical computers. This virtualization of hardware resources enables efficiencies through consolidation of multiple applications on fewer physical servers.
VMware vSphere High Availability (HA) provides easy-to-use, cost effective high availability for applications running in virtual machines. In the event of a physical server failure, affected virtual machines automatically restart on other production servers with spare capacity.
HA enables you to create a cluster out of multiple ESXi servers, enabling you to protect virtual machines. If one of the hosts in the cluster fails, the impacted virtual machines automatically restart on other ESXi hosts within that same VMware vSphere cluster.
VMware vSphere Distributed Resource Scheduler (DRS) is an infrastructure service run by VMware vCenter Server (vCenter). DRS aggregates ESXi host resources into clusters and automatically distributes these resources to virtual machines by monitoring utilization and continuously optimizing virtual machine distribution across ESXi hosts. DRS can also use vMotion® and Storage vMotion® to ensure that the virtual machines have access by rebalancing resource capacity to make room for larger virtual machines. VMware recommends enabling DRS to achieve higher consolidation ratios.
VMware vSphere PowerCLI provides a Windows PowerShell interface for the users of vSphere 5.1 and above and VMware Infrastructure 4.x and above. VMware vSphere PowerCLI is a powerful command-line tool that lets you automate all aspects of vSphere management, including network, storage, VM, guest OS and more. PowerCLI is distributed as a Windows PowerShell snap-in, and includes 330 PowerShell cmdlets for managing and automating vSphere and vCloud, along with
documentation and samples.
The EMC VNX flash-optimized unified storage platform delivers innovation and enterprise capabilities for file, block, and object storage in a single, scalable, and easy-to-use solution. Ideal for mixed workloads in physical or virtual environments, VNX combines powerful and flexible hardware with advanced efficiency,
management, and protection software to meet the demanding needs of today’s virtualized application environments.
VNX includes many features and enhancements designed and built upon the first generation’s success. These features and enhancements include:
More capacity with multicore optimization with Multicore Cache, Multicore RAID, and Multicore FAST Cache (MCx)
Greater efficiency with a Flash-optimized hybrid array
Better protection by increasing application availability with active/active VMware vSphere
5.1
VMware vSphere HA
VMware vSphere Distributed Resource Scheduler
VMware vSphere PowerCLI
EMC Next- Generation VNX
Easier administration and deployment by increasing productivity with new Unisphere Management Suite
VSPEX is built with the next generation of VNX to deliver even greater efficiency, performance, and scale than ever before.
Flash-optimized hybrid array
VNX is a flash-optimized hybrid array that provides automated tiering to deliver the best performance to your critical data, while intelligently moving less frequently accessed data to lower-cost disks.
In this hybrid approach, a small percentage of flash drives in the overall system can provide a high percentage of the overall IOPS. Flash-optimized VNX takes full advantage of the low latency of flash to deliver cost-saving optimization and high performance scalability. The EMC Fully Automated Storage Tiering Suite (FAST Cache and FAST VP) tiers both block and file data across heterogeneous drives and boosts the most active data to the flash, ensuring that customers never have to make concessions for cost or performance.
Data is generally accessed most frequently at the time it is created, therefore new data is first stored on flash drives to provide the best performance and latency. As that data ages and becomes less active over time, FAST VP tiers the data from high- performance to high-capacity drives automatically, based on customer-defined policies. This functionality has been enhanced with four times better granularity and with new FAST VP solid-state disks (SSDs) based on enterprise multi-level cell (eMLC) technology to lower the cost per gigabyte. FAST Cache dynamically absorbs
unpredicted spikes in system workloads. All VSPEX use cases benefit from the increased efficiency.
VSPEX Proven Infrastructures deliver private cloud, end-user computing, and virtualized application solutions. With VNX, customers can realize an even greater return on their investment. VNX provides out-of-band, block-based deduplication that can dramatically lower the costs of the Flash tier.
VNX Intel MCx Code Path Optimization
The advent of Flash technology has been a catalyst in totally changing the
requirements of midrange storage systems. EMC redesigned the midrange storage platform to efficiently optimize multicore CPUs to provide the highest performing storage system at the lowest cost in the market.
MCx distributes all VNX data services across all cores—up to 32, as shown in Figure 3.
The VNX series with MCx has dramatically improved the file performance for
transactional applications like databases or virtual machines over network-attached storage (NAS).
Figure 3. Next-Generation VNX with multicore optimization
Multicore Cache
The cache is the most valuable asset in the storage subsystem; its efficient use is key to the overall efficiency of the platform in handling variable and changing workloads.
The cache engine has been modularized to take advantage of all the cores available in the system.
Multicore RAID
Another important part of the MCx redesign is the handling of I/O to the permanent back-end storage—hard disk drives (HDDs) and SSDs. Greatly increased performance improvements in VNX come from the modularization of the back-end data
management processing, which enables MCx to seamlessly scale across all processors.
VNX storage, enabled with the MCx architecture, is optimized for FLASH 1st and provides unprecedented overall performance, optimizing for transaction performance (cost per IOPS), bandwidth performance (cost per GB/s) with low latency, and
providing optimal capacity efficiency (cost per GB).
VNX provides the following performance improvements:
Up to four times more file transactions when compared with dual controller arrays
Increased file performance for transactional applications (for example, Microsoft Exchange on VMware over NFS) by up to three times with a 60 percent better response time
Up to four times more Oracle and Microsoft SQL Server OLTP transactions
Up to six times more virtual machines VNX performance
Active/active array service processors
The new VNX architecture provides active/active array service processors, as shown in Figure 4, which eliminate application timeouts during path failover since both paths are actively serving I/O.
Note: The active/active processors are only available for classic LUNs not for pool LUNs.
Load balancing is also improved and applications can achieve an up to two times improvement in performance. Active/active for block is ideal for applications that require the highest levels of availability and performance, but do not require tiering or efficiency services like compression, deduplication, or snapshot.
With this VNX release, VSPEX customers can use virtual Data Movers (VDMs) and VNX Replicator to perform automated and high-speed file system migrations between systems. This process migrates all snaps and settings automatically, and enables the clients to continue operation during the migration.
Figure 4. Active/active processors increase performance, resiliency, and efficiency
VMware Virtual Storage Integrator
Virtual Storage Integrator (VSI) is a no-charge VMware vCenter plug-in available to all VMware users with EMC storage. VSPEX customers can use VSI to simplify
management of virtualized storage. VMware administrators can gain visibility into their VNX storage using the same familiar vCenter interface to which they are accustomed.
With VSI, IT administrators can do more work in less time. VSI offers unmatched access control that enables you to efficiently manage and delegate storage tasks with confidence. Perform daily management tasks with up 90 percent fewer clicks and up to 10 times higher productivity.
VMware vStorage APIs for Array Integration
VMware vStorage APIs for Array Integration (VAAI) offloads VMware storage-related functions from the server to the storage system, enabling more efficient use of server and network resources for increased performance and consolidation.
Virtualization management
VMware vStorage APIs for Storage Awareness
VMware vStorage APIs for Storage Awareness (VASA) is a VMware-defined API that displays storage information through vCenter. Integration between VASA technology and VNX makes storage management in a virtualized environment a seamless experience.
EMC Storage Integrator
EMC Storage Integrator (ESI) is targeted towards the Windows and Application administrator. ESI is easy to use, delivers end-to end monitoring, and is hypervisor agnostic. Administrators can provision in both virtual and physical environments for a Windows platform, and troubleshoot by viewing the topology of an application from the underlying hypervisor to the storage.
Unisphere Management Suite
EMC Unisphere is the central management platform for the VNX series, providing a single, combined view of file and block systems, with all features and functions available through a common interface. Unisphere is optimized for virtual applications and provides industry-leading VMware integration, automatically discovering virtual machines and ESX servers and providing end-to-end, virtual-to-physical mapping.
Unisphere also simplifies configuration of FAST Cache and FAST VP on VNX platforms.
The new Unisphere Management Suite extends Unisphere’s easy-to-use, interface to include VNX Monitoring and Reporting for validating performance and anticipating capacity requirements. As shown in Figure 5, the suite also includes Unisphere Remote for centrally managing up to thousands of VNX and VNXe systems with new support for XtremSW Cache.
Figure 5. New Unisphere Management Suite
Red Hat Enterprise Linux is a versatile platform for x86 and x86-64 that can be deployed on physical systems, as a guest on the major hypervisors, or in the cloud. It supports all leading hardware architectures with compatibility across releases. Red Hat Enterprise Linux 6.3 includes enhancements and new capabilities that provide rich functionality, especially the developer tools, virtualization features, security, scalability, file systems, and storage.
EMC Avamar and EMC Data Domain deliver the protection confidence needed to accelerate deployment of virtualized Oracle. Optimized for virtualized application environments, EMC backup and recovery reduces backup times by 90 percent and increases recovery speeds by 30 times,even offering instant virtual machine access,for worry-free protection.
EMC backup also delivers big savings. Our deduplication solutions reduce backup storage by 10 to 30 times, backup management time by 81 percent, and bandwidth by 99 percent for efficient offsite replication, delivering a seven month payback on average.
Furthermore, EMC backup offers a solution with Data Domain systems and DD Boost software that allows for full DBA control of Oracle backup, recovery, and replication while the backup team maintains control of the infrastructure. This eliminates the occurrence of protection silos, increasing efficiency and lowering risk.
Red Hat Enterprise Linux 6.3
EMC backup and recovery solutions
Chapter 4 Solution Implementation
This chapter presents the following topics:
Overview ... 32 Setting up the physical components ... 32 Implementing the network ... 33 Implementing the storage array ... 34 Implementing ESXi and vCenter ... 44 Preparing the Oracle Database 11g R2 virtual machine template ... 45 Deploying the Oracle Database 11g R2 virtual machine ... 51 Implementing backup and recovery ... 61
Overview
This chapter discusses the steps required to install, configure, and implement the VSPEX Proven Infrastructure environment, which involves:
Preparing the components and their physical setup.
Implementing the network, including the configuration steps for physical and virtual switches in line with vendor guidelines and The vSphere Networking Guide.
Configuring the VNX storage array and optional FAST Cache or FAST VP feature.
Building the VMware vSphere environment with ESXi server and vCenter implementation.
Preparing a virtual machine template for rapid deployment of Oracle Database 11g R2.
Quickly and simply deploying an Oracle Database 11g R2 virtual machine using PowerCLI.
Implementing backup and recovery.
Setting up the physical components
This section includes information about how to prepare the solution’s physical components. After you complete the steps listed in Table 7, the new hardware components will be racked, cabled, powered, and ready for network connection.
Note If you already have a VSPEX Proven Infrastructure environment, you can skip this section.
Table 7. Tasks for physical setup
Task Description Reference
Prepare network switches
Install switches in the rack and connect them to power.
Your vendor’s installation guide
Prepare servers Install the servers in the rack
and connect them to power. Your vendor’s installation guide
Prepare VNX Install the VNX in the rack and
connect it to power. EMC VNX System Installation Guide
For details of the physical setup, refer to Reference Architecture: EMC VSPEX Private Cloud: VMware vSphere 5.1 for up to 1,000 Virtual Machines
Implementing the network
This section documents requirements for the network infrastructure needed to support this architecture. Table 8 provides a summary of the tasks for switch and network configuration and references for further information.
Note: If you already have a VSPEX Proven Infrastructure environment, you can skip this section.
Table 8. Tasks for switch and network configuration
Task Description Reference
Configure network
Configure storage network and host infrastructure networking as specified in the solution reference architecture.
Refer to the appropriate document in Essential Reading.
Complete network cabling
Connect:
Switch interconnect ports VNX ports
ESXi NIC/FC ports
N/A
Configure VLAN Configure private and public VLANs
as required. Vendor’s switch
configuration guide Configure
VMare vSphere networking
Create virtual switches for separation of application,
management, and storage networks.
vSphere Networking Guide
For details of network implementation, refer to the appropriate document in Essential Reading.
Figure 6 and Figure 7 show the configured vSphere network for this implementation.
As highlighted, redundant 10GbE network adapters or FC Host adapters have been dedicated for storage network traffic.
Figure 6. Redundant 10 GbE network configuration
Figure 7. Redundant Fibre Channel storage network
Implementing the storage array
This section describes how to configure the VNX storage array. In this solution, the VNX provides Network File System (NFS) or Virtual Machine File System (VMFS) data storage for the VMware host.
Table 9. Tasks for storage configuration
Task Description Reference
Set up initial VNX
configuration Configure the IP address information and other key parameters on the VNX.
VNX System Installation Guide
VNX Series Configuration Worksheet Provision storage Create VMFS datastores or
create NFS file systems that will be mounted on hosts Overview
Figure 8 depicts the high-level architecture of this solution.
Figure 8. Oracle Database 11g R2 storage elements
Ensure that network interfaces, IP address information, and other key parameters such as DNS and NTP are configured on the VNX before provisioning the storage.
For more information on how to configure the VNX platform, refer to the appropriate document in Essential Reading.
Before you provision the storage for a VMFS datastore, follow the recommendations and VSPEX Sizing Tool proposals introduced in thecompanion Design Guide. Before you start, refer to the Reference Architecture titled EMC VSPEX Private Cloud:
VMware vSphere 5.1 for up to 1000 Virtual Machines to provision storage for virtual machine operating system on VNX.
Figure 9 shows an example of how to provision a VMFS datastore for the Oracle Database 11g R2 data in VNX. For more information about the storage layout recommendations and design, refer to the Design Guide.
Setting up initial VNX configuration
Provisioning storage for VMFS example
Figure 9. Example storage layout of Oracle database on VMware
For more information on how to lay out virtual disks for the Oracle Database 11g R2 data on the VMFS datastore, refer to the example in Table 27 on page 78.
Present the NFS filesystem for the virtual machine datastore to all ESXi hosts, using either EMC Unisphere or the VSI plugin for VMware vSphere client. Figure 11 displays how how to mount the NFS file systems to ESXi hosts that are used to store virtual servers on the VNX array using VSI.
The EMC Unified Storage Management plugin for VMware vSphere provides an interface for creating and managing NFS file systems for VMware ESXi host directly from the vSphere Client interface. Figure 10 shows the configuration screen accessed through Home >Solutions and applications >EMC from the vSphere Client, which is accessible after running the installer on your management host.
Provisioning storage for NFS example
Figure 10. Configuring EMC Unified Storage Management in vSphere Client
Unified Storage Management enables the vSphere administrator to mount NFS file systems directly to the ESXi cluster nodes from vSphere Client, as shown in Figure 11.
Figure 11. Configuring NFS Storage on VNX for Oracle VMs
To configure NFS network settings, storage pools for file, file systems, and NFS exports on the VNX array in EMC Unisphere, perform the following steps:
1. In Unisphere, select the VNX array for this solution.
2. Select Settings Network Settings for File
3. Configure the IP address for network ports used for NFS. For detailed steps refer to EMC Procedure Generator for VNX.
4. Select Storage Storage Configuration
Figure 12. Configuring VNX storage pools
5. Select the Storage Pools for File tab and create the additional storage pools in the VNX for Oracle Database files, as shown in Figure 12. Refer to Table 27 on page 78 for detailed information.
6. Create the required file systems. Figure 13 shows the Create File System Wizard from Unisphere.
Figure 13. Unisphere Create File System Wizard
7. Create the NFS exports for the database components that the Oracle NFS client will mount on the database virtual machines. Figure 14 shows the Unisphere Create NFS Export Wizard. Root and Access permissions are granted to the NIC.
Figure 14. EMC Unisphere Create NFS Export Wizard
