VMware vsphere 5: Best Practices for Oracle RAC Virtualization

(1)

VMware vSphere 5:

Best Practices for

Oracle RAC Virtualization

Jeff Browning, Webmaster

Oracle Solutions Enablement – EMC Corporation

IOUG – November 8, 2012

(2)

Agenda

• Why RAC on vSphere?

• Support

• Licensing

• Storage

• Performance

• Version 5 new features

(3)

(4)

Virtualization is on the rise

Source:

Toward a Smarter Information Foundation

: 2010 IOUG Enterprise Platform Decision Survey

Oracle Magazine

July / August 2010

(5)

Source: 2010 IOUG Enterprise Platform Decision Survey:

Toward a Smarter Information Foundation

*Source: Paul Mauritz keynote, VMware vSphere 5 launch

Oracle Magazine

July / August 2010

(6)

Business challenge

Infrastructure silos lead to higher

costs, lower ROI on assets, and

increased environment complexity

• Lower server and storage

utilization

• Difficulty meeting service levels

–

Maintaining and managing separate

environments

–

Provisioning inefficiencies

–

Inconsistent data protection strategies

Infrastructure silos

Oracle11g

(7)

VMware vSphere with Oracle RAC

benefits

Match native performance even in consolidation

scenarios



95%+ Oracle instances match native performance on VMware



VMware facilitates migration to current hardware – a major

performance benefit

Performance

Reduce hardware and software license costs



Consolidate servers



Reduce CPU burn by improving utilization



$/TPS declines dramatically as utilization improves

DB consolidation

Provision databases on demand



Minutes to provision a new VM from template



Seconds to provision database copy using storage semantics



Rapidly provision, use and destroy test / dev and other utility servers

DB on demand

Increase application quality of service



Scale dynamically



Built-in high availability and simple disaster recovery



Dynamically load balance RAC nodes across physical servers

Quality of service

(8)

Current Oracle RAC /

(9)

(10)

My Oracle Support note 249212.1

Purpose

Explain to customers how Oracle supports our products when running on VMware Scope & Application

For Customers running Oracle products on VMware virtualized environments. No limitation on use or distribution.

Support Status for VMware Virtualized Environments ---

Oracle has not certified any of its products on VMware virtualized environments. Oracle Support will assist customers running Oracle products on VMware in the following manner: Oracle will only provide support for issues that either are known to occur on the native OS, or can be demonstrated not to be as a result of running on VMware.

If a problem is a known Oracle issue, Oracle support will recommend the appropriate solution on the native OS. If that solution does not work in the VMware virtualized environment, the customer will be referred to VMware for support. When the customer can demonstrate that the Oracle solution does not work when running on the native OS, Oracle will resume support, including logging a bug with Oracle Development for investigation if required.

If the problem is determined not to be a known Oracle issue, we will refer the customer to VMware for support. When the customer can demonstrate that the issue occurs when running on the native OS, Oracle will resume support, including logging a bug with Oracle Development for investigation if required.

NOTE: Oracle has not certified any of its products on VMware. For Oracle RAC, Oracle will only accept Service Requests as described in this note on Oracle RAC 11.2.0.2 and later releases.

Source:

My Oracle Support website

Purpose

Oracle has not certified any of its products on VMware virtualized environments. Oracle Support will assist customers running Oracle products on VMware in the following manner: Oracle will only provide support for issues that either are known to occur on the native OS, or can be demonstrated not to be as a result of running on VMware.

NOTE: Oracle has not certified any of its products on VMware. For Oracle RAC, Oracle will only accept Service Requests as described in this note on Oracle RAC 11.2.0.2 and later releases. Purpose

Oracle has not certified any of its products on VMware virtualized environments. Oracle Support will assist customers running Oracle products on VMware in the following manner: Oracle will only provide support for issues that either are known to occur on the native OS, or can be demonstrated not to be as a result of running on VMware.

Support Status for VMware Virtualized Environments ---

If a problem is a known Oracle issue, Oracle support will recommend the appropriate solution on the native OS. If that solution does not work in the VMware virtualized environment, the customer will be referred to VMware for support. When the customer can demonstrate that the Oracle solution does not work when running on the native OS, Oracle will resume support, including logging a bug with Oracle Development for investigation if required.

NOTE: Oracle has not certified any of its products on VMware. For Oracle RAC, Oracle will only accept Service Requests as described in this note on Oracle RAC 11.2.0.2 and later releases.

(11)

Range of possible issues

Oracle centric

Prop

ens

it

y

to

require

phy

sical

Hardware / OS

centric

Statement

execution

Oracle

ASM

(12)

VMware / EMC support

for Oracle RAC on vSphere

• E-Labs tests and

validates Oracle RAC

11g clustering on

vSphere

• EMC Global Solutions

provides comprehensive

testing of Oracle RAC

11g on vSphere

(13)

(14)

Oracle licensing is on the processor core

Almost all Oracle software

products are licensed on the

physical CPU.

For most products, a processor

core factor (PCF) is applied to

the number of processor cores.

The higher the PCF, the higher

the core count, and the higher the

cost for Oracle licensing.

Source:

Oracle Technology

Global Price List

(May 12, 2011)

Oracle Processor Core Factor

Table

(March 16, 2009)

Int el and A MD are the m ost com m on

(15)

Transaction cost vs. utilization

Cost per TPS for a four-node Oracle RAC 11g cluster running EE

Software license cost: around $2,200,000

(16)

Oracle must be licensed on all processor

cores in a VMware ESX cluster

VMware ESX Server 1

Oracle License Applies

VMware ESX Server 2

Oracle License Does Not Apply

VMotion

VMware ESX Server 2

Oracle License Now Applies

• As soon as an Oracle VM hits an ESX server, the license

and maintenance costs for Oracle apply

• There is no vCPU licensing of Oracle software

• These costs are high, and reducing them is very

(17)

Virtualized vs. physical

licensing compared

Oracle RAC 11g EE running on physical hardware

4 Node RAC:

4 * 4 * quad processors:

4

3

_{cores = 64}

_{* PCF}

(0.5)

32 licensed cores

Oracle Database 11g EE license per

core:

$47,500

* 32

$1,520,000

Oracle RAC upcharge per core:

$23,000

* 32

$736,000

Total:

$2,256,000

Annual maintenance on 11g EE per

core:

$10,450

* 32

$334,400 / year

Annual maintenance on RAC:

$5060 per year

* 32

$161,920 / year

Total:

$496,320 / year

Source:

Oracle Technology Global Price List

(May 12, 2011)

OS

CRS

(18)

Compatible workloads

for consolidation

2 Identical 4-node RAC clusters

Workload RAC1: OLTP Workload RAC2: Batch-oriented DSS

RAC1 peaks during the day, RAC2 at night

(19)

Virtualized vs. physical

licensing compared

Oracle RAC 11g EE running on VMware vSphere

2 4 Node RAC clusters consisting of 4 VMs running on a cluster of 4 ESX servers (each VM runs

on 8 vCPUs):

4 * 4 * quad processors:

4

3

_{cores = 64}

_{* PCF}

(0.5)

32 licensed cores

Oracle Database 11g EE license per

core:

$47,500

* 32

$1,520,000

Oracle RAC upcharge per core:

$23,000

* 32

$736,000

Total:

$2,256,000

Annual maintenance on 11g EE per

core:

$10,450

* 32

$334,400 / year

Annual maintenance on RAC:

$5060 per year

* 32

$161,920 / year

Total:

$496,320 / year

(20)

Oracle DRS / HA cluster

Microsoft DRS / HA cluster

Well managed licensing on

VMware DRS / HA cluster

(21)

ROI on vSphere software with

well-managed VMware DRS / HA cluster

Assumes 2X compression

Higher compression would result in higher savings

3 year Oracle physical software costs

$3,744,960

3 year Oracle virtualized software costs

$1,872,480

Savings on Oracle software costs (physical vs. virtual)

$1,872,480

3 year vSphere software costs

$92,384

(22)

DRS / HA cluster

Poorly managed licensing on

VMware DRS / HA cluster

(23)

Negative ROI on vSphere software with

poorly-managed VMware DRS / HA

cluster

Assumes Oracle VMs are 50% of the cluster

Negative ROI increases as the percentage of Oracle VMs decreases

3 year Oracle physical software costs

$1,872,480

3 year Oracle virtualized software costs

$3,744,960

Wasted money on Oracle software costs (physical vs. virtual)

($1,872,480)

3 year VMware vSphere software costs

$184,768

(24)

Licensing summary

• Correctly managed,

investment in vSphere

software yields

phenomenal ROI for

Oracle RAC

environments

• Incorrectly managed,

investment in vSphere

can be spectacularly

wasteful

(25)

(26)

Possible storage configurations

F

ile

Bloc

k

Physical

Virtual

Config 1

Config 2

Config 3

Config 4

①

Config 1: Block / SAN storage

w/ RDM LUNs mounted

directly on VMs then used to

make ASM diskgroups

②

Config 2: Block / SAN storage

w/ LUNs mounted onto ESX,

used to make VMFS volumes;

.vmdk files used to create

ASM diskgroups

③

Config 3: File / NFS mounted

directly on VMs managed by

Oracle using Direct NFS Client

④

Config 4: File / NFS mounted

on ESX as datastores; .vmdk

files used to create ASM

diskgroups

(27)

Storage config 1: Block / RDM -> VM /

ASM:

Layer diagram

LUNs

ESX

VM

Oracle

_•

_{ESX is bypassed almost}

completely, and LUNs

are mounted on VMs as

RDM

• ―SCSI‖ disks within VMs

are used to create ASM

diskgroups

• Extremely large LUNs

and I/O can be supported

• Minimal overhead vs.

(28)

Storage config 1:

Block / RDM -> VM / ASM

:

Network diagram

FC Switch

10 GbE Switch

ESX Server

Physical RAC Nodes

Virtual RAC Nodes

SAN Network

R

AC

I

nterc

o

nne

c

t

SAN Storage Array

• Virtual and physical

nodes can co-exist in the

same RAC cluster

• Can also use storage

tools which integrate with

Oracle (EMC Replication

Manager, RecoverPoint)

• VMotion is not supported

so no DRS cluster (but

HA cluster still works)

• SCSI bus must be set to

(29)

Storage config 2: FC / VMFS / .vmdk /

ASM:

Layer diagram

LUNs

ESX

VM

Oracle

• LUNs are mounted onto

ESX and formatted as

VMFS volumes (must be

eager zero thick)

• Shared write bit is set in

.vmdk files to allow RAC

clustering / shared

storage (KB Article:

1034165)

• .vmdk files are then used

to create ASM

diskgroups

• Maximum RAC nodes

supported: 8

• No VMware snaps or

linked clones (same as

FT)

(30)

Storage config 2:

FC / VMFS / .vmdk / ASM

:

Network diagram

FC Switch

10 GbE Switch

HA / DRS Cluster

Virtual RAC Nodes

SAN Network

R

AC

I

nterc

o

nne

c

t

SAN Storage Array

• Because Oracle datafiles

are stored on VMFS, no

physical nodes are

allowed

• Not possible to use

storage tools that

integrate with Oracle

(EMC Replication

Manager, RecoverPoint)

–

But storage replication

(VMware layer) can still

be performed if follow

best practices

.vmd k .vmd k .vmd k .vmd k

(31)

Storage config 3: IP / NFS -> VM / dNFS:

Layer diagram

NAS Array

ESX

VM

Oracle

• VM OS mounts NFS

mounts using normal

Linux / UNIX semantics;

ESX is bypassed entirely

• Oracle then manages the

NFS mount points using

Oracle Direct NFS Client

(dNFS)

• Very large databases /

I/O can be supported

• Minimal overhead vs.

(32)

Storage config 3:

IP / NFS -> VM / dNFS

:

Network diagram

10 GbE Switch

Physical RAC Nodes

Virtual RAC Nodes

NAS Storage Array

• Virtual and physical

nodes can co-exist in the

same RAC cluster

• NFS file systems are

mounted on both virtual

and physical nodes

• Can also use storage

tools which integrate with

Oracle (EMC Replication

Manager)

• DRS / HA cluster works

fine

IP Storage Network

R

AC

I

nterc

o

nne

c

t

HA / DRS Cluster

(33)

Storage configs compared

Config

Networ

k

Storag

e

Laye

r

Physic

al

Compa

t

Storag

e

Tools

VMware

Value

Add

Scaling

VMotion

Config 1:

FC / RDM -> VM / ASM

FC

Block

VM

Yes

No

No limit

No*

Config 2:

FC / VMFS / .vmdk /

ASM

FC

Block

ESX

No

No**

Yes***

8 nodes

Yes

Config 3:

IP/ NFS -> VM / dNFS

IP

File

VM

Yes

No

No limit

Yes

*Therefore, no DRS Cluster with RDM (HA Cluster still works fine)

**Replication Manager / RecoverPoint replication possible at VM level, but without Oracle

coordination

(34)

Comparison between VMware RDM and

VMFS for Oracle 11g Databases

• OLTP Transactions per

minute

• ESX server CPU utilization

• Database transactions

response times

LUN Number Size (GB) Datafile Type

1 LUN 1 50 Contains DATA device 1 2 LUN 2 50 Contains DATA device 2 3 LUN 3 50 Contains DATA device 3 4 LUN 4 50 Contains DATA device 4 5 LUN5 50 Contains DATA device 5 6 LUN6 50 Contains DATA device 6 7 LUN7 50 Contains DATA device 7 8 LUN8 50 Contains DATA device 8 9 LUN9 50 Contains DATA device 9 10 LUN10 50 Contains DATA device 10 11 LUN11 50 Contains DATA device 11 12 LUN12 50 Contains DATA device 12 13 LUN13 50 Contains FRA device 1

Datastore Type Size (GB) Description

1 Data Datastore 588 Contains all database DATA devices 2 FRA Datastore 98 Contains all database FRA devices 3 REDO

Datastore 98 Contains all database REDO devices 4 TEMP

Datastore 98

Contains all database EMP/UNDO devices

Disk Performance

VMFS RDM 40,000 35,000 30,000 25,000 20,000 15,000 10,000 5,000 0 KBps 5 10 15 20 25 30 35 40 45 50 55 Minutes

(35)

VMFS / NFS Oracle RAC:

Best practices for file placement

Small VMFS 1 / Small NFS 1

Small VMFS 2 / Small NFS 2

(36)

FAST VP vs. LUN migration

T

ier

0:

F

las

h

F

AST

VP

Pool

T

ier

2:

N

L

-SAS

Non FAST VP

Storage Array

FAST VP

Storage Array

T

ier

1:

SAS

(37)

FAST VP performance comparison

• 17% improvement in

response times after

FAST policy applied

• FAST VP moved most

active data to EFDs, so

ensuring improved

response times for entire

storage group

• FAST VP also moved

30% of total allocated

storage for storage group

to SATA with no negative

impact on response times

(38)

FAST Cache

①

Page requests satisfied from

DRAM if available

②

If not, FAST Cache

driver checks

map to determine where page is

located

③

Page request satisfied from disk

drive if not in FAST Cache

④

Policy Engine copies the page to

FAST Cache if it is being used

frequently

⑤

Subsequent requests for this

page satisfied from FAST Cache

⑥

Dirty pages are copied back to

disk drives as background activity

Page Map

Policy

Engine

Driver

DRAM

(39)

dNFS (storage config 3)

FAST Cache performance comparison

• Difference between

FAST Cache and

non-FAST Cache was more

than 2X

• Difference between

virtualized and physical

was 11%, at most

(40)

FAST Virtual Pool

FAST Cache

FAST Suite—FAST VP + FAST Cache

FAST VP

Tiers across drives in pool



Optimizes drive utilization



Relative ranking over time



1 GB slices ideal for

deterministic workload



Periodic rebalance

FAST Cache

Copies hottest data to

Flash



Optimizes Flash utilization



Dynamic movement in near

real-time



64 KB sub-slices ideal for

bursty workload or where

working set varies a lot

(41)

Will FAST Suite benefit my application

• How much improvement

is expected from FAST

Suite?

• How to find if an

application benefits?

–

Monitor application

performance data

–

Analyze it for I/O size,

randomness & locality

Flash

Performance

HDD

Performance

0 % Data set on flash

100

All flash performance

All HDD performance

(42)

Multi-pathing and failover for

virtualized environments

• Scale virtual machines on

SAN storage over FC, iSCSI,

FCoE

• Dynamically balance I/O

requests

• Automatically detect and

recover from I/O path failures

EMC PowerPath VE

Highly recommended for databases

PowerPath

VE

VMs

ESX

Cluster

IP /SAN

Switch

Storage

Array

(43)

PVSCSI Storage Performance white paper

Storage

Array

ESX

VM

PVSCSI

Driver

• For high I/O workloads

–

2,000 IOPS or greater

• Improves CPU cost for:

–

Fibre Channel

–

iSCSI

• Works with:

–

RDM

–

VMFS

• With VMware ESX 4.1

can boot with PVSCSI

from:

–

Windows Server 2008

–

Windows Server 2003

–

Red Hat Enterprise Linux

5

• Free

OS

PVSCSI

Driver

App

(44)

(45)

Physical & virtual Oracle RAC

comparison

• OLTP database workload

• SAN / RDM (storage config 1)

• Virtual environment delivered performance consistently within 4% of

physical environment

(46)

Performance assistance for virtualization

Hardware Assist Virtualization

Linux / Windows

Offers Significant

Performance

Improvements

HBA / Network Cards

Other Devices

Off loads and increases

efficacy of Hypervisor

device management

CPU multi-generation

Support & Security

Compatibility support for

multi-generations of the

same CPU family within a

cluster / compatibility pool

Secure execution of VMs

using boot profiles to

prevent attacks

Linux Huge Pages

Windows Large Pages

Intel VT or

AMD V

AMD – RVI or

Intel EPT

Interrupt

Virtualization

Directed

I/O

CPU

Compatibility

Pools

Secure

Execution

(47)

Memory virtualization:

Non-accelerated

Guest operating system

(inside the virtual

machine)

Virtual memory mapping

(VMM) addresses seen

by the VM

physical machine

addresses seen by the

ESX server

LPN table

PPN table

Machine pages

lpn2ppn

ppn2ma

(48)

Memory virtualization:

Translation look-aside buffer

LPN to PPN

translation

EPT: Hardware assist

PPN to MA

TLB hit

TLB

TLB hit

miss

(49)

Memory virtualization:

Shadow page table

LPN table

PPN table

Machine pages

LPN

PPN

MA

Shadow

page table

(50)

My Oracle Support Note 361468.1: HugePages on 64-bit Linux

My Oracle Support Note 317141.1: How to Configure REL/OEL 4 32-bit for Very Large Memory with ramfs and

More Pages:

Heavy weight kernel

Less Pages:

Light weight kernel

HugePages map:

5,000 2 MB pages

Default page map:

2.6 million 4 KB pages

Performance boost:

HugePages in Linux

(51)

2 MB addressable

memory

TLB cache

More TLB hits:

Better performance

Less TLB hits:

Worse performance

HugePages map:

5,000 2 MB pages

Default page map:

2.6 million 4 KB pages

16 MB addressable

memory

Performance boost:

HugePages in Linux

(52)

Un-virtualizable

Instructions

What is Binary Translation?

Light weight, efficient, near native performance

Translator Cache

Binary Translation

User Mode

Native Performance

Ring 3

Least privileged

Ring 1

Less privileged

Ring 0

Most privileged

(53)

Virtualization Management

Control Structure (VMCS)

Intel VT / AMD-V

Un-virtualizable

Instructions

What is Binary Translation?

Light weight, efficient, near native performance

User Mode

Native Performance

Ring 3

Least privileged

Ring 0

More privileged

Ring -1

Most privileged

Virtual Machine

Monitor

Hypervisor (ESXi)

(54)

VMware performance best practices

for Oracle RAC 11g

①

Disable dynamic coalescing on

private interconnect (.vmx file

hack)

②

Increase Interrupt rate on network

adapter to 30K

③

Keep full memory reservation

(DRS or resource sharing setting

at VM level)

④

Turn

off

Transparent

Page

Sharing (VM level setting)

⑤

Enable hyperthreading

⑥

Use PreferHT=TRUE (.vmx file

(55)

Enable CPU hyperthreading

• Hyperthreading adds

an additional 20% of

CPU capacity (YMMV)

• Both vSphere and

Oracle 11g RAC are

HT aware

• It costs you nothing

• Must be enabled from

(56)

4 socket 4 core ESX

server

Without Prefer HT

0

1

2

3

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

0

1

2

3

4

5

6

7

1

0 2 3

4

5 6 7

8

9 10 11

12

13 14 15

16

17 18 19

20

21 22 23

24

25 26 27

28

29 30 31

CPUs

Cores

Threads

vCPUs

(57)

4 socket 4 core ESX

server

With Prefer HT

CPUs

Cores

0

1

2

3

0

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15 vCPUs

0

1

2

3

4

5

6

7 Threads

0

1 2 3

4

5 6 7

8

9 10 11

12

13 14 15

16

17 18 19

20

21 22 23

24

25 26 27

28

29 30 31

(58)

(59)

vSphere 5:

(60)

Datastore cluster

vSphere 5:

Storage DRS

2

0

0 GB

S

D

data

s

to

re

10 T

B

S

A

S

data

s

to

re

100 T

B

NL

-S

A

S

data

s

to

re

DRS cluster

RAC VMs

.vmdk

(61)

VMFS-5 vs. VMFS-3 Comparison

Feature

VMFS-3

VMFS-5

2TB+ VMFS Volumes

Yes

(using extents)

Yes

Support for 2TB+ Physical

RDMs

No

Yes

Unified Block size (1MB)

No

Yes

Atomic Test & Set

Enhancements

(part of VAAI, locking mechanism)

No

Yes

Sub-blocks for space efficiency

64KB (max

~3k)

8KB (max

~30k)

Small file support

No

1KB

NOTE:

LUN count

limit for a

vSphere 5

cluster is

still 256,

but with

larger

datastores

the norm,

should be

less of an

issue…

(62)

(63)

Oracle 11i eBusin

ess

Suite: Replatform

• One of the largest single global

instances of Oracle 11i

• Core mission-critical applications

• 75+ Application tiers – VMware/RHEL

• Oracle Database 10g R2

• 8 TB Database; 8.8 billion rows of data

• 52 Million transactions per day

• 79K IOPS

• 40K Blocks Per Second Interconnect

Traffic

• 40,000+ Named users

(64)

• Cisco UCS B440

• Intel Nehalem EX processor

• CPU – 128 Cores

• CPU utilization 10%

• OS – Red Hat Linux / vSphere

• Storage – Symmetrix VMAX

• Sun Fire E25K

• UltraSPARC IV processor

• CPU – 224 Cores

• CPU utilization 80%

• OS – Solaris 10

• Storage – Symmetrix DMX-3

Sun Fire Server

Unified Computing System

EMC IT:

(65)

EMC IT:

Before and after architectures

Befo

re

A

ft

er

Server

Powerpath Powerpath CRS CRS JDBC SQL*Net JDBC SQL*Net

Database

Sun 25K OS - Solaris 10 CPUs – 224 Cores Veritas RAC 5.0 PowerPath 5.0.2 Cisco UCS B440 Intel Nehalem EX OS – RHEL 5.4 CPUs – 128 Cores PowerPath 5.3

Load balancer

ACE Load Balancer

Application server

E-Business Suite 11i Shared APPL_TOP Linux RHEL 4 VMware vSphere

Storage

Symmetrix DMX 3 720 x 146 GB drives 95 TB Total RAW RAID 10 Protection Symmetrix VMAX 450 x 235 GB drives 95 TB Total RAW RAID 10 Protection Oracle RAC 10g (version 10.2.0.3) 8 TB (8.8b rows SGA 60 GB each 2 nodes Oracle RAC 10g (version 10.2.0.4) 8 TB (8.8b rows) SGA 30 GB each 4 nodes

1

2

1 2 3 4

(66)

EMC IT:

(67)

EMC IT:

11i performance Improvements - Online

CXP Transaction Times (sec)

DXP Transaction Times (sec)

(68)

Replay duration will be about 4 times faster

Sales Job Timings (sec)

CS Job Timings (sec)

85% - 95% reduction in transaction times for the above jobs (i.e. up to 20 times faster)