Oracle E-Business Suite on Vblock Systems Backup and Recovery with EMC Avamar Solution Architecture

(1)

Table of Contents

Oracle E-Business Suite on Vblock™ Systems

Backup and Recovery with EMC Avamar

Solution Architecture

Version 1.0

May 2013

(2)

2

VCE believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED "AS IS." VCE MAKES NO

REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OR

(3)

3

Introduction ... 6

About this document ... 6

Solution overview ... 6

Objectives ... 6

Scope ... 7

Audience... 7

Feedback ... 7

Technology overview ... 8

Solution components ... 8

Architecture overview ... 9

Logical layout ... 9

Physical layout ...10

Hardware and software components ...11

Design considerations ...13

Compute configuration ...13

Network configuration ...14

Storage design ...15

Virtualization configuration ...16

Application configuration ...17

Oracle EBS ...18

EMC Avamar ...18

VCE Data Protection Overview ...19

Solution validation ...20

Backup and recovery test category summaries ...20

Category 1: Hot online backup and recovery ...20

Category 2: Cold offline backup and recovery ...21

Category 3: Point-in-time restore/recovery (PITR) ...21

Category 4: Incremental backup/recovery (optional) ...22

Validation objectives ...22

Test environment design ...22

Tests and test procedures ...23

Test acceptance criteria ...23

Test 1 Hot online backup and recovery ...24

Test 1 Environment ...24

Test 1 Procedures ...27

(4)

4

Test 2 Cold database backup for Oracle EBS with Avamar ...30

Test 2 Environment ...30

Test 2 Procedures ...30

Test 2 Results ...33

Test 3 Point-in-time restore and database recovery for Oracle EBS with Avamar ...34

Test 3 Environment ...34

Test 3 Procedures ...34

Procedure 1-Test 3: Database recovery from online hot Avamar backup ...35

Test results ...36

Procedure 2-Test 3: Application tier node recovery from cold Avamar backup ...37

Test results ...39

Procedure 3-Test 3: Application tier node recovery from hot Avamar backup ...39

Test results ...41

Procedure 4-Test 3: Full database recovery from cold Avamar database backup ...41

Test results ...41

Procedure 5-Test 3: Point-in-time recovery from Avamar hot online backup ...41

Test results ...44

Test 4 Oracle backup and recovery with VMware Data Protection 5.1 ...45

Test 4 Environment ...45

Test 4 Procedures ...45

Test 4 Results ...50

Test 5 Backup and recovery with VDP for Oracle EBS App Tier ...51

Test 5 Environment ...51

Test 5 Procedures ...52

Test 5 Results ...58

Summary of validation results ...59

Test 1 Results for hot online backup and recovery ...59

Test 2 Results for cold database backup for Oracle EBS with Avamar ...59

Test 3 Results for point-in-time restore and database recovery for Oracle EBS with Avamar

...59

Test 4 Results for Oracle EBS backup and recovery with VMware Data Protection 5.1 ...60

Test 5 Results for performing backup and recovery with VDP for Oracle EBS App Tier ...61

Combined summary ...61

Conclusion ...62

Next steps ...63

APPENDIX – Description of the solution components ...64

Vblock

™

Systems ...64

(5)

5

Vblock System 320 ...64

Compute components ...65

Network components ...65

Storage components ...65

Virtualization components ...65

Oracle Database 11g Release 2 Enterprise Edition ...65

Oracle Real Application Clusters ...66

Oracle E-Business Suite Release 12 ...66

EMC Avamar ...66

EMC Avamar Data Store ...66

(6)

6

Introduction

About this document

Integrated solutions for backup and recovery are critical for businesses. Exponential data growth, regulatory requirements, and the demand for continuous availability make it difficult to successfully complete backups within shrinking backup windows.

VCE addresses this need for Oracle customers with a new integrated backup and recovery solution architecture: Oracle E-Business Suite (EBS) on Vblock™ Systems Backup and Recovery with EMC Avamar. Also included in the solution is the VMware vSphere Data Protection Suite.

Solution overview

The solution architecture describes the key features and benefits of backup and recovery in the virtualized Oracle EBS on Vblock Systems with EMC Avamar.

VCE designs and delivers VblockSystems, the world’s most advanced converged infrastructure—it optimizes infrastructure, lowers costs, secures the environment, speeds deployment, and promotes innovation. And with best-in-class components, one architecture spanning the whole portfolio,

advanced operations management for converged infrastructure, a single point of contact from initiation through support, and the industry's most robust range of configurations and models, Vblock Systems address the unique needs and requirements of practically any size organization, a variety of price points, and a multitude of specific verticals and business drivers.

If you would like to use the Vblock Systems solution additionally as a cold site, see the previous VCE Oracle EBS-related papers: Enhanced Operational Agility and Scalability for Oracle E-Business Suite R12 and RAC on VMware vSphere 5 and Zero Downtime Bi-Directional Physical to Virtual Migration for Oracle® EBS R12 with RAC on Vblock™ Infrastructure Platforms. These papers also demonstrate the ease with which Vblock Systems can be integrated into existing investments.

Objectives

This Solution Architecture paper provides guidance on how to backup and recover a virtualized Oracle EBS deployment on Vblock Systems with Avamar and the VMware vSphere Data Protection Suite. Objectives for this solution:

 Provide a technology overview of the backup and recovery solution and supporting architecture that includes an explanation of the setup, configuration, and operational best practices for deploying the backup and recovery solution in the Oracle EBS environment on Vblock Systems  Define how data is protected end-to-end.

(7)

7

 Deployment for Oracle EBS in a typical Avamar environment (for example, this 2-4TB of the Oracle environment is a portion of the overall mixed application workload environment)  Description of the prototype lab environment in which VMware vSphere data protection

capability can be used as a utility

 Description of Vblock data protection solutions including EMC Avamar only, and Avamar and Data Domain, including the sizing and scaling

The validation performed while using this methodology includes the following objectives:

 Measure backup throughput in terms of TB/hour and GB/minute, measure latency, and calculate daily backup volume

 Verify near zero long-running or failed backups

 Demonstrate backups have been performed successfully without loss or corruption  Demonstrate how Avamar contributes to decreasing RTO/RPO in the event of a failure

Scope

The above objectives describe the scope for this project.

Audience

The primary audience of this solution architecture paper is technical practitioners, such as data center managers, database architects, and database administrators (DBAs). In addition, IT management teams would also be interested in reviewing this document as part of a product evaluation.

Feedback

To suggest documentation changes and provide feedback on this paper, send email

(8)

8

Technology overview

This section presents a summary of the technologies used in the VCE solution.

Solution components

The solution uses the following hardware and software components and technologies. See the Appendix for a description of each of these products and technologies.

 Vblock Systems

 Oracle Database 11g Release 2 Enterprise Edition  Oracle Real Application Clusters

 Oracle E-Business Suite Release 12  EMC Avamar

(9)

9

Architecture overview

Logical layout

Figure 1 shows the overall logical architecture of the environment.

Figure 1 Logical architecture: Oracle EBS Backup and Recovery with Avamar

The logical architecture diagram shows:

 There is a single-node Avamar solution backing up data.

 There are Avamar clients and proxies running on Vblock Systems. During nightly backups the Avamar agents copy the changed variable-length blocks from the Avamar grid and create a virtual full backup from only the changed variable-length blocks.

(10)

10

Figure 2, below, also describes backup and recovery with Avamar.

Figure 2: Oracle E-Business Backup Solution Overview with Avamar

Physical layout

Figure 3 shows the physical architecture deployed for Avamar with Oracle R12 EBS.

(11)

11

The physical architecture diagram shows:

1. The Avamar client connects to the database tier for Oracle R12 EBS to perform the backup and restore operations.

2. The Application Tier is installed on the database, and the application data is stored in the database.

3. The Avamar grid interacts to backup from and restore data to the database

4. Vblock Systems: For the compute, storage and network layers, the Oracle R12 EBS application is deployed across multiple virtual machines to process Oracle Financial operations key to the business.

5. Nexus 5548 (which is integral to the Vblock Systems) is directly connected to the Nexus 7000 for all network services.

6. Downstream from the Nexus 5548 everything is connected as per standard Vblock Systems. All the hosts and VMs are inside Vblock Systems and get network service from the Nexus 7000.

Hardware and software components

Table 1 shows the hardware resources used in this solution.

Table 1. Description of the hardware resources used in this solution

Hardware Description

Vblock 300 with VNX 5300 1

Compute Cisco Unified Computing System (UCS):

 3 x Cisco UCS B200 M3 blade server packs (4-core, 128 GB memory)

- 2 used for application server - 1 used for database ser ver Management (Mini-AMP):

 2 x Cisco UCS C220 M2 servers  Cisco Catalyst 3560

Network  2 x Cisco Nexus 5548 switch

 2 x Cisco MDS 9148 aggregation SAN switch

Storage  1 x EMC VNX 5300

Virtualization  1 x VMware vSphere 5.1

 1 x VMware vCenter Server 5.1

(12)

12

Table 2. Description of the software used to validate this solution

Software Quantity Version Purpose

Oracle Enterprise Linux (OEL) 3 Oracle Linux 6.2 Server operating system

Oracle R12 EBS 1 12.1.2 ERP Database Software

(13)

13

Design considerations

Oracle backup and recovery requires both instance and database recovery to resolve data and system failures. As such, copies of the database files, archive log files, and operating system binary files must be backed up to Avamar on a regular basis and be available for a restore and recovery operation. For details regarding Oracle backup and recovery procedures, see Oracle® Database Backup and Recovery User's Guide 11g Release 1 (11.1)

at: http://docs.oracle.com/cd/B28359_01/backup.111/b28270/toc.htm

EMC Avamar provides a feature-rich suite of tools to perform backup and recovery for the Oracle database and the Oracle EBS by using a variety of software based plug-ins. For example, to backup and restore the Oracle EBS application tier, since it is file-based binaries, the Linux operating system plug-in is used to backup and restore the application tier servers within the Oracle R12 EBS

environment.

To backup and restore the database tier servers for Oracle R12 EBS, Avamar provides a plug-in that interfaces directly with the Oracle Recovery Manager (RMAN) database layer. In addition to core backup and recovery features, Avamar also provides advanced backup and recovery options by using a policy-based system. Filters can be setup and applied based on time, date, retention period, and nature of the backup and restore operation for Oracle. These features within Avamar remediate the limitations present within the native operating system and Oracle database utilities. Furthermore, pre- and post-custom scripts can also be deployed to simplify the backup and recovery operations for a given customer Oracle environment.

Compute configuration

(14)

14

The Summary tab lists the general technical information and resources used for the configuration.

Network configuration

(15)

15

Storage design

This solution uses the VNX based storage arrays provided in the Vblock Systems.

The EMC VNX series is a dedicated network server optimized for file and block access that delivers high-end features in a scalable and easy-to-use package.

The VNX series delivers a single-box block and file solution that offers a centralized point of management for distributed environments. This makes it possible to dynamically grow, share, and cost-effectively manage, multiprotocol file systems and to provide multiprotocol block access. Administrators can take advantage of simultaneous support for NFS and CIFS protocols by enabling Windows and Linux/UNIX clients to share files by using the sophisticated file-locking mechanisms of VNX for File and VNX for Block for high-bandwidth or for latency-sensitive applications.

Following, are the storage data tables.

Table 3 presents the VM storage information.

VM IP Subnet GW VLAN vCPU

Mem-ory (GB) OS(G B) DB LUN and Log Size (TB) OracleR12EBS _AppTier 1 172.23.23.10 255.255.255.0 172.23.23.1 2323 8 16 200 - OracleR12EBS _AppTier 2 172.23.23.11 255.255.255.0 172.23.23.1 2323 8 16 200 - OracleR12EBS _DBTier 1 172.23.23.20 255.255.255.0 172.23.23.1 2323 8 32 20 2TB

Table 4 presents the RAID Group information.

RAID Group Number

Number of Disks Disk Type Disk Size RPM 34 1 SAS 600GB 10 K Hot Spare Hot Spare 35 1 SAS 600GB 10 K Hot Spare Hot Spare

Table 5 presents the Storage Pool information.

Pool Name Number of disks

Disk type

Disk size

RPM Raid type Purpose

OracleAvamar_AppTier 8 SAS 600 GB 10 K RAID I/O Oracle App Tier

OracleAvamar_DBTier 14 SAS 600 GB 10 K RAID 5 Oracle DB Tier

vb319 5 SAS 600 GB 10 K RAID 5 VM Bootswap and

(16)

16

Virtualization configuration

VMware vSphere 5.1 was deployed as the hypervisor virtualization component for Oracle and Avamar on the Vblock 300. A total of three virtual machines (VM) were created within VMware vCenter and mapped to three ESXi hosts. The following screenshot shows a sample configuration for each virtual machine for the database and application server configuration.

Since the Oracle EBS application tier houses lightweight application server processes, total

requirements for application processing are less than that used for the database tier server. Multiple application server node VMs were used to load balance and boost overall throughput as

(17)

17

Application configuration

After the infrastructure deployment for Oracle and Avamar was completed, the Oracle R12 EBS was installed and configured with the Oracle Linux operating system according to best practices and guidelines from Oracle, EMC and VMware. Two application tier servers were deployed to optimize performance and availability for concurrent batch processing and OLTP based workloads. In addition, best practices for the configuration of the concurrent managers and batch processing were optimized for the environment. A single Oracle database tier was deployed to complete the

configuration. The following shows the overall architecture for the deployed Oracle EBS environment. First, we have the application tier nodes running under the Oracle Linux 6 platform:

The second application tier node has the same configuration as shown above. The database tier has the following configuration:

(18)

18

Archive log mode was enabled as this is a requirement for hot backups with Oracle and Avamar.

Oracle EBS

To demonstrate a realistic workload, the Oracle Financial batch processes shown in the following screenshot were executed on the live environment during and after backup and recovery operations.

EMC Avamar

(19)

19

VCE Data Protection Overview

With Vblock™ Systems Data Protection products, VCE provides end-to-end solutions that are designed to meet their unique infrastructure needs. Customers can now add new levels of data protection to Vblock Systems, or they can convert their existing non-EMC, or non-VCE, data protection solutions to a VCE solution.

The VCE Data Protection solutions include:

EMC Avamar - Deduplication backup system and software that provides highly efficient, variable

length deduplication, with tight integration to EMC purpose-built, backup appliances. Avamar enables fast, complete daily backups of virtualized environments including the entire IT infrastructure,

enterprise applications, and data. Deduplication on the client side reduces ESXi contention and network traffic thus shortening backup times.

EMC Avamar plus Data Domain - Dual options: Avamar targets the Avamar Data Store in your

solution. Or, for larger, high-change-rate workloads, the target can be a Data Domain system that provides high speed, variable-length, inline deduplication. The result is a backup data storage footprint that on average is 10x-30x smaller than a traditional backup footprint.

EMC VPLEX - VPLEX enables administrators to share, protect, or load-balance data across multiple

(20)

20

Solution validation

The validation information is provided for backup and recovery categories, followed by a table of objectives for the project, and a table that correlates the objectives to the described actual tests performed.

Backup and recovery test category summaries

Oracle backup and recovery is a complex topic. To explain the objectives, process, and test plans, below are detailed summaries on the major types of backup and recovery to be performed for the solution. The backup and recovery types comprise 4 categories. The 4 categories explained below are reflected in the actual tests executed.

For the solution, backup and restore service (BRS), operational agility was tested from a hot online backup from Avamar, a cold offline backup, and a point-in-time recovery for Oracle R12 EBS. Also tested was an optional incremental backup and recovery.

Category 1: Hot online backup and recovery

Table 6 presents the features of hot online backup and recovery.

Category 1: Hot online backup and recovery

Description Oracle hot backup and recovery provides the highest level of RTO in comparison to cold backups since the database and application is able to remain online and available during the backup process. There is a slight maintenance window of downtime however during a restore and recovery process due to the fact that the database logs must be applied to resolve the data failure.

Benefit High level RTO and RPO

Tradeoff Some performance impact and additional maintenance involved with storage requirements for archive redo log files and read/write operations impact from Avamar with Oracle.

(21)

21

Category 2: Cold offline backup and recovery

Table 7 presents the features of cold offline backup and recovery.

Category 2: Cold offline backup and recovery

Description Oracle cold backups are beneficial to ensure against data loss thus ensuring a high level of RPO. However since the application and database must be taken offline during the backup and restore operation, there is some downtime involved for the application to complete a cold offline backup and restore.

Benefit: High RPO All committed data in place before the database and application is shutdown that has been backed up is able to be restored completely given that the backup image is not corrupted. Thus, a cold backup has the highest level of RPO for Oracle backups.

Tradeoff: Higher RTO Some downtime is involved since the application and database services must be taken offline to perform the cold backup for Oracle EBS.

Backup window 4 hours estimated (actual to be determined via testing process). Retention period 10 days (varies by business requirements).

Backup schedule Weekly.

Category 3: Point-in-time restore/recovery (PITR)

Table 8 presents the features of point-in-time restore/recovery.

Category 3: Point-in-time restore/recovery (PITR)

Description Recover to point in time before data failure occurred.

Benefit Best RTO for BRS with Oracle and Avamar.

Tradeoff Provides restore only to point-in-time, versus a complete recovery from a site-wide disaster. PITR is best suited to supplement a weekly full backup in the event that, for example, a user accidentally drops a table from a schema. Backup window Estimated 10 minutes to 30 minutes based on recovery amount of redo to be

applied for PITR restore.

Retention period 10 days (varies by business requirements).

(22)

22

Category 4: Incremental backup/recovery (optional)

Table 9 presents the features of incremental backup/recovery (optional).

Category 4: Incremental backup/recovery (optional)

Description Incremental backup and recovery supplements a traditional cold or hot full backup by capturing only delta changes in blocks during application and database processing. For large bulk processing warehouses that load different amounts of massive volume transactions which change on a frequent basis, an incremental backup may prove useful to save on time (RTO) to restore an environment.

Benefit Provides partial backup and recovery for point-in-time scenarios and uses less storage for capture only changes made since last full backup operation. Tradeoff Less than a full restore is not possible. Must use a full recovery as well. Backup window Varies from 10 minutes to few hours (to be tested and determined). Retention period 10 days (varies per business requirements).

Backup schedule Weekly.

Validation objectives

This validation is focused on how to back up the virtual Oracle E-Business environment. The following objectives are related to showing how to restore data when there is a corruption within E-Business (which can happen within the Oracle E-Business integrated environment or supporting components), or when there is a failure with VM due to power outage, or there are failed processes, or other failures. The objectives are:

1. Deploy Oracle EBS R12 on Vblock Systems 300 with EMC VNX 5300 and perform functional testing with VMware vSphere 5.1 Data Protection.

2. Measure baseline backup performance and ensure best performance on Vblock Systems 300 for Oracle.

3. Simulate application tier node failure and perform data recovery with no data loss for Oracle EBS.

4. Simulate database tier failure and show that data recovery with Avamar produces no data loss. 5. Implement point in time recovery (PITR) from Avamar for the Oracle R12 EBS.

Test environment design

(23)

23

Tests and test procedures

The test scenarios cover all of the key goals for the objectives, and reflect the stated categories for Oracle backup and recovery.

There are 5 different test sections:

 Test 1: Hot online backup and recovery

 Test 2: Cold offline database backup for Oracle EBS with Avamar

 Test 3: Test 3 Point-in-time restore and database recovery for Oracle EBS with Avamar  Test 4: Test 4 Oracle Backup and Recovery with VMware Data Protection 5.1

 Test 5: Perform Backup and Recovery with VDP for Oracle EBS App Tier

Test acceptance criteria

Successful test results will meet the following test acceptance criteria. 1. Backup performance:

- Measure data throughput (TB/Hour) for back-up tasks and calculate daily backup volume. - Optimize speed as needed.

2. Recovery performance metrics and test results:

- Recovery Time Objective (RTO), the duration of time and a service level within which a business process must be restored after a disaster (or disruption) must be met.

- Recovery Point Objective (RPO), the maximum tolerable period in which data might be lost from an IT service due to a major incident typical for database applications, must be met. - Avamar helps contribute to RTO and RPO objectives.

(24)

24

Test 1 Hot online backup and recovery

This test is to measure hot online backup and recovery processes for Oracle with Avamar. Test 1 objectives are to show that:

1. Application and database recovery minimizes data loss.

2. Applications and database all function correctly after recovery is performed. 3. Recovery time (RTO) is shortened due to online recovery functionality.

Test 1 Environment

Most customers automate the process for backup and recovery during low activity maintenance windows so that users are not impacted during business operation hours. For the solution, a typical customer standard backup window and schedule of 30-45 days was implemented. Avamar uses the concept of a dataset to pre-configure a standard set of backup operations including backup type, schedule, and retention period.

Procedure:

1. Configure Avamar for Oracle RMAN backup.

2. Install Avamar client software for Oracle RMAN and Oracle Linux client per the EMC Avamar 6.1 documentation.

3. Configure the schedule and backup policy for Oracle EBS data sets.

(25)

25

(26)

26

6. Click Tools and Manage All Datasets, or click Tools and Manage All Schedules.

7. The corresponding window opens (either Manage All Datasets or Manage All Schedules):

8. From the Avamar main window, above, click Tools and then click Manage Retention. The

Manage All Retention Policies window appears. Here, the settings can be reviewed or edited.

9. Click OK to exit.

(27)

27

Test 1 Procedures

The task is to perform a full Oracle RMAN online hot database backup. Oracle hot database online backups allow you to make a database backup while the database and application remain online thus minimizing impact to the availability of the Oracle EBS environment

1. Perform an on demand backup for Oracle with Avamar.

Avamar uses the Oracle RMAN plug-in software to perform the database backups for Oracle EBS. The retention period was set to 30 days for the database backup policy. The initial hot backup taken was a hot online database backup that included a control file and archive log full backup. Incremental (level 0/1) backups were taken after the initial full online database backup to save on disk space required to capture deltas in the application backup environment.

(28)

28

(29)

29

Test 1 Results

Multiple RMAN channels were implemented to improve database backup performance for the solution. For this solution, the default retention period for the database backups was set to 30 days. Oracle Avamar online database backup performance data results for the Oracle EBS and Avamar hot online backup and recovery are shown in Table 10.

Table 10 shows Oracle EBS and Avamar hot online backup and recovery performance data results.

Hostname Application Backup Type Total Runtime Backup Size

ebsdb01 Database Tier Hot online full 50m:06s 159.4Gb

ebsdb01 Database Tier Level 0

Incremental online

15m:09s 159.3Gb

Incremental online

1m:06s 20.5Mb

ebsapp01 Application tier Full online file

system

8m:33s 90.5Gb

system

11m:57s 89.4Gb

Both the database tier and the application tier backups were performed. The total runtime and backup size are also provided for each test run.

As expected, the results show:

 There was no downtime during the backup  There was a high level RTO

(30)

30

Test 2 Cold database backup for Oracle EBS with Avamar

This test is to run and measure a cold offline database backup for Oracle EBS with Avamar. Test 2 objectives:

1. Show that full application and database recovery fulfills requirements for minimal to zero data loss (RPO).

2. Show that performance for backup and recovery is optimal for a full database backup.

Test 2 Environment

The test environment was as described in the above sections, except that for this test the Oracle cold database offline backups are performed when the Oracle EBS environment has been quiesced and shut down for maintenance purposes.

Test 2 Procedures

First, the Oracle application and database tiers must be cleanly shutdown and taken offline, and only then can the backup for Oracle with EBS be set up. Both procedures are described below.

1. The following tasks were performed to prepare Oracle EBS for Avamar cold offline backup: [oracle@ebsdb01 ~]$ echo $ORACLE_HOME

/d01/oracle/VIS/db/tech_st/11.1.0 [oracle@ebsdb01 ~]$ echo $ORACLE_SID VIS

[oracle@ebsdb01 ~]$ sqlplus '/as sysdba'

2. Connected to:

Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options: SQL> shutdown immediate;

Database closed. Database dismounted.

(31)

31

SQL> exit

Disconnected from Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production

With the Partitioning, OLAP, Data Mining and Real Application Testing options

[oracle@ebsdb01 ~]$

Next, a file level backup was performed from Avamar with Oracle RMAN plug-in.

1. In the Select Backup tab, the file level backup option, along with the Oracle Linux Avamar plug-in, were selected as shown below.

2. An on demand cold backup was performed for the solution.

(32)

32

(33)

33

5. Click OK.

6. The offline cold backup was performed.

7. Verification showed that the cold Oracle Avamar backup completed successfully.

Test 2 Results

(34)

34

Test 3 Point-in-time restore and database recovery for Oracle

EBS with Avamar

The point-in-time restore and recovery testing involves using the systems and results from Test 1 and Test 2.

Test 3 objectives:

The objectives are the five procedures to be performed and discussed in Test 3 for the point-in-time restore and recovery.

Table 11 Five procedures comprising Test 3

Procedure number – Test 3 Procedure/test objective name

1 Full database recovery from online hot Avamar backup

2 Application tier node recovery from cold Avamar backup

3 Application tier node recovery from hot Avamar backup

4 Full database recovery from cold Avamar backup

5 Point-in-time recovery from Avamar hot online backup

Note: To conduct the point-in-time restore and database recovery (PITR) for Oracle EBS, Avamar interacts with the Oracle RMAN utility to perform full and incremental database restore and recovery operations using the previously taken backups.

Test 3 Environment

In the test environment, the interaction of the Oracle RMAN utility was utilized.

Test 3 Procedures

(35)

35

Following are 5 sub-sections that present the 5 different procedures comprising Test 3.

Procedure 1-Test 3: Database recovery from online hot Avamar backup

To perform a full online database recovery from Avamar for Oracle EBS:

1. The backup image was retrieved from the Avamar environment.

2. In the Avamar Administrator Backup & Restore window, click on the Select for Restore tab. 3. Select the recent full online database backup to restore

(36)

36

5. Click under More Options to set the database instance and backup account details. 6. Run the operation.

Test results

1. The operation was successfully completed in less time than a traditional backup and recovery operation by using Avamar instead of native BRS tools.

2. Zero data loss was completed from the hot recovery operation with Avamar for Oracle. 3. Downtime was kept to minimal level by high performance results of Avamar with Oracle native

(37)

37

Procedure 2-Test 3: Application tier node recovery from cold Avamar backup

First the application tier node was quiesced to stop all processing and avoid impacting the

environment. Then the cold recovery process from the application tier node was run from the Avamar Administrator. Results showed zero data loss with a total time of less than 10 minutes.

Following are the steps for the cold recovery of the application tier for Oracle EBS: 1. The application tier nodes were shut down with Oracle EBS scripts:

[apps@ebsapp01 scripts]$ adstpall.sh apps/apps You are running adstpall.sh version 120.10 The logfile for this session is located at

/d01/oracle/VIS/inst/apps/VIS_ebsapp01/logs/appl/admin/log/adstpall .log

Executing service control script:

/d01/oracle/VIS/inst/apps/VIS_ebsapp01/admin/scripts/jtffmctl.sh stop

script returned:

**************************************************** You are running jtffmctl.sh version 120.3

(38)

38

2. The recent cold backup from Avamar for the Oracle EBS application tier was selected.

3. The image to restore was selected.

(39)

39

5. Next, the application tier node services were restarted:

[apps@ebsapp01 scripts]$ adstrtal.sh apps/apps You are running adstrtal.sh version 120.15 The logfile for this session is located at

/d01/oracle/VIS/inst/apps/VIS_ebsapp01/logs/appl/admin/log/adstrtal .log

Executing service control script:

/d01/oracle/VIS/inst/apps/VIS_ebsapp01/admin/scripts/adopmnctl.sh start

script returned:

**************************************************** You are running adopmnctl.sh version 120.6

Starting Oracle Process Manager (OPMN) ... opmnctl: opmn started.

adopmnctl.sh: exiting with status 0 adopmnctl.sh: check the logfile

/d01/oracle/VIS/inst/apps/VIS_ebsapp01/logs/appl/admin/log/adopmnct l.txt for more information ...

6. Application functionality was verified with Oracle R12 EBS

Test results

The cold offline recovery test completed successfully and met requirements for RPO of zero data loss. As mentioned in the following Procedure 3-Test 3 section Test Results, the hot online application tier recovery resulted in a shorter recovery time process (RTO) window than did the cold offline

application recovery. Both test results met requirements for RPO of zero data loss.

Procedure 3-Test 3: Application tier node recovery from hot Avamar backup

1. The application tier server was kept online and available during the hot restore process. [apps@ebsapp01 scripts]$ adapcctl.sh status

You are running adapcctl.sh version 120.7.12010000.2 Checking status of OPMN managed Oracle HTTP Server (OHS) instance ...

Processes in Instance: VIS_ebsapp01.ebsapp01.mordor.vce

(40)

40

2. From Avamar the backup image was selected to perform the in-place online recovery and restore for the application tier node.

3. Restore options were selected.

(41)

41

Test results

As expected, the application tier node recovery was performed from Avamar and completed successfully in less than 10 minutes with zero data loss.

The actual results obtained show that it completed in less than 8 minutes with zero data loss.

Since the backup was completed as a hot online application tier backup, no downtime was incurred as a result during the recovery process and all database processing continued without incident. This resulted in a shorter RTO window than did the cold offline recovery. Both test results met

requirements for RPO of zero data loss.

Procedure 4-Test 3: Full database recovery from cold Avamar database backup

1. Preview the cold backup image from Avamar, as shown below, before performing the restore and recovery from the backup.

2. Make sure that the Oracle database is shutdown and that no background instance processes are running for the database.

Test results

The cold database recovery required a short maintenance window in which the database and application tier nodes were taken offline to perform the recovery process. After the recovery was completed, the database and application tier nodes and services were brought back online successfully.

Procedure 5-Test 3: Point-in-time recovery from Avamar hot online backup

(42)

42

1. Shutdown and start nomount mode.

[oracle@ebsdb01 ~]$ sqlplus '/as sysdba'

SQL*Plus: Release 11.1.0.7.0 - Production on Mon Apr 1 14:10:55 2013

SQL> startup nomount; ORACLE instance started.

Total System Global Area 1068937216 bytes Fixed Size 2166536 bytes Variable Size 427819256 bytes Database Buffers 624951296 bytes Redo Buffers 14000128 byte

This procedure is performed similarly to the earlier full online database restore and recovery (as specified in the previous section).

The key difference in this restore and recovery is to specify the advanced options for the point-in-time recovery (PITR) from the online backup.

(43)

43

2. The Restore Command Line Options window opens. Click Point-in-Time (PIT). Also, click

Show Advanced Options and set them.

(44)

44

4. The Advanced Options provide recovery from block corruption issues as well as debugging messaging capability.

5. Next, it was verified that the restore was completed successfully from Avamar and Oracle EBS.

SQL> select instance_name, status from v$instance; INSTANCE_NAME STATUS

--- --- VIS OPEN

Test results

The restore from Avamar and Oracle EBS was successfully completed. The following results were obtained from the point in time restore:

1. Zero data loss for application and database with Avamar for the Oracle R12 EBS.

(45)

45

Test 4 Oracle backup and recovery with VMware Data

Protection 5.1

VMware provides a backup and recovery (BRS) product in the vSphere 5.1 web based Data Protection Suite (VDP) for backup and recovery of virtual machines. It complements enterprise BRS products for the BRS solution for Oracle on the Vblock System.

In this use case, VDP will be deployed to back up and restore the Oracle VMs in case of data failure. The advantage of using VDP versus a full BRS product such as Avamar is to take advantage of shorter RTO windows when only a single VM must be restored versus a full enterprise recovery operation which could take hours with Avamar. In addition, by using VDP with Oracle virtualization on the Vblock platform, archival tasks can be simplified by data retention of individual application and database VMs versus a full database backup. Full details regarding the installation and configuration for VDP are provided in the VMware document titled vSphere Data Protection Administration Guide - vSphere Data Protection 5.1 EN-000846-00 available online from VMware for reference.

Test 4 Environment

The same test environment was used.

Test 4 Procedures

(46)

46

The VDP configuration is displayed.

(47)

47

1. The backup was performed from VDP using the following link: https://si-vc128.mordor.vce:9443/vsphere-client

Login as appuser/Vce12345

The VMware Data Protection application is available as shown below.

(48)

48

4. In the Schedule page, the solution daily VMware Data Protection (VDP) backups were scheduled in order to simulate a real customer environment.

5. In the Retention Policy page, the backup retention period was set to 60 days.

6. Next the standard name is provided to the new backup job for VDP.

(49)

49

8. The backup is reviewed and completed, as shown below.

9. Next, the backup is enabled and started to kick off the backup job for Oracle.

10. Select the Oracle VDP Weekly Backup job and Backup Now. The dropdown box opens. Click

(50)

50

11. A confirmation Info appears. Click OK.

12. Next, on the Reports tab verify run time and completion for the Oracle VDP Backup.

Test 4 Results

Oracle and VDP backup:

The use case simulated a virtual machine (VM) failure for the Oracle EBS R12 application tier node with a full VM recovery using VDP.

(51)

51

Test 5 Backup and recovery with VDP for Oracle EBS App

Tier

This test is to run backup and recovery with VDP for the EBS application tier. Test objectives:

1. Demonstrate for the RPO objective that failure of virtual machines (VMs) for the Oracle application tier can be resolved quickly with zero data loss.

2. Show that of the application tier for Oracle R12 EBS has low to no impact to current operations. 3. Demonstrate that recovery meets time objectives (RTO) to restore the application in less than 1

hour.

Test 5 Environment

For use in the Oracle and VDP restore, VDP provides a wizard interface to perform backup and restore operations as shown below.

(52)

52

Test 5 Procedures

The following test was performed with VDP and Oracle: Application VM failure recovery testing for application tier:

The application tier VM is restored as follows from the VDP Restore tab.

Note: Due to a size limitation within VMware, virtual machines larger than 2TB in size cannot be backed up with VDP. Details on this limitation are provided in the VMware knowledge base document. Creating a snapshot for a virtual machine fails with the error: file is larger than maximum files size supported (1012384: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&ext ernalId=1012384 The use case for using VDP 5.1 with backup and recovery for Oracle environments is best suited to

(53)

53

For the solution a new backup was taken for the Oracle VMs as shown below: 1. In the Create a new backup job screen, choose the VM back up.

2. The backup schedule is set to daily for the solution testing. 3. The retention period is set to 30 days.

(54)

54

5. Restore and Recovery of Oracle Application VM after failure screen appears.

6. Select the recent backup(s) to restore the Oracle VMs.

(55)

55

8. In the solution, the restores for the Oracle VMs were performed to the original location after the disk and storage failure was resolved.

To avoid impact on the overall Oracle R12 EBS environment, the database tier was cleanly shutdown to ensure the entire ERP environment is restored in a consistent state.

9. From VDP Restore Options, select:

(56)

56

10. Select the most recent VDP backup image:

11. Click Next.

(57)

57

12. Click Finish. 13. Begin the restore.

14. Monitor the restore operations.

15. Verify that the VM has been restored, then power on to check status of the application VM:

(58)

58

Test 5 Results

(59)

59

Summary of validation results

This section contains the results from each test for comparative purposes, along with a final summary.

Test 1 Results for hot online backup and recovery

Online database backup and application backup performance data results for the Oracle EBS and Avamar hot online backup and recovery are shown in Table 12.

Table 12 Restore and recovery times

Hostname Application Backup Type Total Runtime Backup Size

ebsdb01 Database Tier Hot online full 50m:06s 159.4Gb

Incremental online

15m:09s 159.3Gb

Incremental online

1m:06s 20.5Mb

system

8m:33s 90.5Gb

system

11m:57s 89.4Gb

As expected, the results show:  No downtime during the backup  High level RTO

 Short total runtime was obtained, the total runtime achieved was only 86 min: 51 seconds

Test 2 Results for cold database backup for Oracle EBS with Avamar

The successful cold offline Oracle EBS and Avamar backup process resulted in a non-corrupted backup image, as expected.

(60)

60

Table 13 Results for Test 3 procedures

Procedure number – Test 3

Procedure/test objective name Results

1 Full database recovery from online

hot Avamar backup

Results showed the operation was successfully completed in less time than a traditional backup and recovery operation by using Avamar instead of native BRS tools.

There was zero data loss.

Downtime was kept to minimal level by high performance results of Avamar with Oracle native recovery tools.

2 Application tier node recovery from

cold Avamar backup

The cold offline recovery test completed successfully and met requirements for RPO (Recovery Point Objective) of zero data loss

3 Application tier node recovery from

hot Avamar backup

The hot online application tier recovery resulted in a shorter recovery time process (RTO) window than did the cold offline application recovery (above). Both test results met requirements for RPO (Recovery Point Objective) of zero data loss.

4 Full database recovery from cold

Avamar backup

The cold, full database recovery requires a short maintenance window in which the database and application tier nodes were taken offline to perform the recovery process.

Results: after the recovery was completed, the database and application tier nodes and services were successfully brought back online.

5 Point-in-time recovery from Avamar

hot online backup (uses the previously taken backups).

The following results were obtained from the successful point-in-time restore:

 Zero data loss for application and database with Avamar for Oracle R12 EBS.

 Recovery time objectives of less than 60 minutes were successfully met. This accomplished RTO and RPO goals for a typical customer Oracle EBS environment.

Test 4 Results for Oracle EBS backup and recovery with VMware Data Protection 5.1

VDP was deployed to backup and restore the Oracle VMs in case of data failure. The advantage of using VDP versus a full BRS product such as Avamar is to take advantage of shorter RTO windows when only a single VM must be restored versus a full enterprise recovery operation which could take hours with Avamar. And, by using VDP with Oracle virtualization on the Vblock platform, archival tasks can be simplified by data retention of individual application and database VMs versus a full database backup.

The test simulated a virtual machine (VM) failure for the Oracle EBS R12 application tier node with a full VM recovery using VDP

(61)

61

Test 5 Results for performing backup and recovery with VDP for Oracle EBS App Tier

The Oracle EBS application tier VM was restored with zero data loss in less than 5 minutes.

Combined summary

Hot and cold backups were successful.

The backup validations measured backup throughput in terms of TB/hour and GB/min, measured latency, and calculated daily backup volume. Test results:

 Showed no long running or failed backups. All backups were performed successfully without loss or corruption.

 Demonstrated how Avamar contributes to decreasing RTO/RPO in the event of a failure, showing that hot online backup and recovery operations for Oracle with Avamar provide the best RTO and RPO overall in terms of performance and availability.

Restore and recovery tests were successful, and each showed zero data loss.

Table 14 provides the times for the various restore and recovery tests in the solution, in order of performance.

Table 14 Restore times for each restore type tested

Environment Restore Type Restore Time Results

Database Tier Hot full restore 10m:11sec Zero data loss

Database Tier Cold restore 14m:28 sec Zero data loss

Database Tier PITR recovery 9m:4 sec Zero data loss

Application Tier Hot restore 7m:20 sec Zero data loss

Application Tier Cold restore 8m:21 sec Zero data loss

Application Tier VDP restore 5m:22 sec Zero data loss

The shortest recovery time occurred with the application tier node restore process from VMware Data Protection and the longest time required for the restore and recovery window took place with the cold full recovery process for the Oracle database tier with Avamar and Oracle EBS.

(62)

62

Conclusion

Customers will benefit from this new VCE integrated solution for fast backup and recovery of their business-critical applications and data - at lower cost and less risk.

All of the objectives for the Oracle EBS on Vblock Systems Backup and Recovery with EMC Avamar solution architecture have been met.

Guidance has been provided on how to backup and recover a virtualized Oracle EBS deployment on Vblock Systems with Avamar and the VMware vSphere Data Protection Suite, including a technology overview, architecture, explanations of setup, configuration and best practices for deploying the backup and recovery solution in the Oracle EBS environment on Vblock Systems. The paper also described the backup methodology and defined end-to-end data protection through Vblock data protection solutions with EMC Avamar only and with Avamar and Data Domain, including sizing and scaling information.

The conceptual and hands-on best practice aspects of this solution have included a description of the deployment for Oracle EBS in a typical Avamar environment with a prototype lab environment in which the VMware vSphere data protection capability can be used as a utility.

The solution validation performed while using this methodology met the test objectives:

 It measured backup throughput in terms of TB/hour and GB/minute, measured latency, and calculated daily backup volume

 Test results showed no long running or failed backups. All backups were performed successfully without loss or corruption.

 Results demonstrated how Avamar contributes to decreasing RTO/RPO in the event of a failure, showing that hot online backup and recovery operations for Oracle with Avamar provide the best RTO and RPO overall in terms of performance and availability.

 The test objectives have been met, as follows:

- Zero data loss occurred after the restore and recovery tasks were performed for the stated use cases with the solution for Oracle EBS and Avamar.

- The shortest recovery time occurred with the application tier node restore process from VMware Data Protection.

- The best protection against data loss for a high level of protection (RPO) is validated by the results from the cold backup and recovery process with Avamar for Oracle.

(63)

63

Next steps

(64)

64

APPENDIX – Description of the solution components

The solution uses the following hardware and software components and technologies.

 Vblock Systems

 Oracle Database 11g Release 2 Enterprise Edition  Oracle Real Application Clusters

 Oracle EBS Release 12  EMC Avamar

 EMC Avamar Data Store

Vblock

™

Systems

The Vblock System from VCE is the world's most advanced converged infrastructure—one that optimizes infrastructure, lowers costs, secures the environment, simplifies management, speeds deployment, and promotes innovation. The Vblock System is designed as one architecture that spans the entire portfolio, includes best-in-class components, offers a single point of contact from initiation through support, and provides the industry's most robust range of configurations.

Vblock System 720

The Vblock System 720 is an enterprise, service provider class mission-critical system in the Vblock System 700 family, for the most demanding IT environments—supporting enterprise workloads and SLAs that run thousands of virtual machines and virtual desktops. It is architecturally designed to be modular, providing flexibility and choice of configurations based on demanding workloads. These workloads include business-critical enterprise resource planning (ERP), customer relationship management (CRM), and database, messaging, and collaboration services. The Vblock System 720 leverages the industry’s best director-class fabric switch, the most advanced fabric based blade server, and the most trusted storage platform. The Vblock System 720 delivers greater configuration choices, 2X performance and scale from prior generations, flexible storage options, denser compute, five 9s of availability, and converged network and support for a new virtualization platform that accelerates time to service and reduces operations costs.

Vblock System 320

(65)

65

For more information, go to www.vce.com.

Compute components

The compute components in Vblock Systems are built on the Cisco Unified Computing System (UCS) line of products. Components include one or more blade server chassis, compute blades, I/O

modules, and fabric interconnects that connect the unified fabric to the rest of the environment.

Network components

The network components in Vblock Systems comprise various models of Cisco Nexus and MDS storage switches. Included are the Cisco Nexus 7000 Series, Cisco Nexus 5000 Series, Cisco Nexus 1000V, Cisco Catalyst 3000 Series, and Cisco MDS 9000 Series switches.

Storage components

The Vblock 300 used in this solution contains VNX 5300 storage arrays. The VNX5300 is a high-end, enterprise storage array that supports multiple drive technologies, including Flash, serial-attached SCSI, SAS, and nearline SAS (NL-SAS) drives, and a full range of RAID types. Intel Xeon processors power the VNX series for intelligent storage that automatically and efficiently scales in performance, while ensuring data integrity, security, and high availability for mission-critical SQL Server

environments.

Virtualization components

Virtualization components include VMware vSphere, VMware vCenter Server, and VMware ESXi. This solution uses VMware vSphere 5.1.

The VMware vCenter Server provides a scalable and extensible platform that forms the foundation for virtualization management. It centrally manages VMware vSphere environments, allowing IT

administrators control over the virtual environment.

For more information on the VMware vCenter Server, go to http://www.vmware.com/products/vcenter-server/overview.html.

Oracle Database 11g Release 2 Enterprise Edition

Oracle Database 11g Release 2 provides industry-leading performance, scalability, security, and reliability on a choice of clustered or single servers with a wide range of options to meet user needs. It provides the foundation for IT to successfully deliver more information with higher quality of service, reduce the risk of change within IT, and make efficient use of IT budgets.

(66)

66

Oracle Real Application Clusters

Oracle RAC supports the transparent deployment of a single database across a cluster of servers, providing fault tolerance from hardware failures or planned outages. Available as an option with Oracle Database 11g, Oracle RAC is a cluster database with a shared cache architecture that overcomes the limitations of traditional shared-nothing and shared-disk approaches to provide a highly scalable and available database solution.

Oracle E-Business Suite Release 12

The Oracle EBS architecture is a framework for multi-tiered distributed computing. And, in the Oracle EBS model, services are distributed among multiple levels or tiers. A service is a process or a group of processes that exercises some business logic and provides a particular functionality. A tier is a grouping of services that can potentially span physical machines. In other words, a tier is a logical grouping of services that is not limited by physical nodes or machines. Each tier can consist of one or more nodes, and each node can accommodate more than one tier. For instance, the database server can reside on the same node as one or more servers in the application tier.

Note: For more details, refer to Oracle Applications Concepts [B10642-01].

EMC Avamar

EMC Avamar is a deduplication backup-and-recovery software and system solution. It performs deduplication on the client, which eliminates data from backup streams before it crosses the network. Client deduplication realizes savings on bandwidth, storage, and backup windows. VMware’s vStorage API (in vSphere as deployed in Vblock Systems) also provides the ESXi hypervisor architecture.

EMC Avamar Data Store

(67)

67

References

EMC

 EMC Avamar 6.1 for Oracle User Guide, EMC 2013

VCE

 Enhanced Operational Agility and Scalability for Oracle E-Business Suite R12 and RAC on VMware vSphere 5

 Oracle E-Business Suite and RAC Performance and Elasticity on Vblock™ Platforms with vSphere 4.1

 Zero Downtime Bi-Directional Physical to Virtual Migration for Oracle® EBS R12 with RAC on Vblock™ Infrastructure Platforms

VMware

 vSphere Data Protection Administration Guide: vSphere Data Protection 5.1

Oracle

 Oracle® Database Backup and Recovery User's Guide 11g Release 1 (11.1)  Oracle® Database Backup and Recovery User's Guide 11g Release 2 (11.2)

(68)

ABOUT VCE

VCE, formed by Cisco and EMC with investments from VMware and Intel, accelerates the adoption of converged infrastructure and cloud-based computing models that dramatically reduce the cost of IT while improving time to market for our customers. VCE, through the Vblock Systems, delivers the industry's only fully integrated and fully virtualized cloud infrastructure system. VCE solutions are available through an extensive partner network, and cover horizontal applications, vertical industry offerings, and application development environments, allowing customers to focus on business innovation instead of integrating, validating, and managing IT infrastructure.

For more information, go to http://www.vce.com.