CON9577 Performance Optimizations for Cloud Infrastructure as a Service

(1)

(2)

CON9577

Performance

Optimizations for

Cloud Infrastructure as

a Service

John Falkenthal, Software Development Sr. Director - Oracle VM SPARC and VirtualBox Jeff Savit, Senior Principal Technical Product Manager

October 26, 2015

(3)

Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a

commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

(4)

Program Agenda

Oracle VM performance in cloud

Performance concepts for the cloud and VM environments

Oracle VM Server for x86: optimizing CPU, Memory, I/O

Oracle VM Server for SPARC: optimizing CPU, Memory, I/O

Recommendations and Resources

1

2

3

4

5

(5)

Oracle VM performance in cloud

Performance concepts for the cloud and VM environments

Oracle VM Server for x86: optimizing CPU, Memory, I/O

Oracle VM Server for SPARC: optimizing CPU, Memory, I/O

Recommendations and Resources

1

2

3

4

(6)

Oracle VM performance in cloud

• Not just about performance of running VMs

– Also performance of management at scale

– Multi-tenancy, many clients, can’t give individual attention to all

• Oracle VM is designed to “engineer in” best practices

– Prevent problems through design features and default configuration – Reduce manual configuration for good “out of the box” performance – Best practices where configuration settings are necessary

• Oracle VM product family members address different markets

• Entire stack tested to find limits, and enhanced in each release

(7)

Oracle VM Portfolio

From Development, Fully Configurable Software to “Turn Key” Systems

Oracle VM VirtualBox

Desktop Private Cloud

On Premise Oracle VM x86 Oracle VM SPARC Public Cloud Oracle VM x86 Oracle VM SPARC

(8)

Oracle VM Family: Choice and Flexibility

• Oracle VM VirtualBox

– Development platform with multiple OS support

• Oracle VM Server for x86 & SPARC

– Zero cost license: pay only for support

– Full functionality including all management software – Your hardware infrastructure

• Private Cloud Appliance

– Enterprise class, turn key converged infrastructure system containing servers, storage, network, and software

– From power-on to production in about an hour – Automated installation and configuration

From Development, Fully Configurable Software to “Turn Key” Systems

(9)

Oracle VM performance in cloud

Performance concepts for the cloud and VM environments

Oracle VM Server for x86: optimizing CPU, Memory, I/O

Oracle VM Server for SPARC: optimizing CPU, Memory, I/O

Recommendations and Resources

1

2

3

4

(10)

Performance concepts for the cloud and VM environments

• Evaluating and performance: books-worth of material, but in short:

– Target service level objectives, not load factors

– Response time: in terms of X% of a specified transaction complete in Y seconds at a rate of N transactions per second

– Throughput: in terms of “units of work (jobs) accomplished per time interval”

• The simplified performance management loop:

– Proactively measure performance and resource utilization for baseline values – Measure: are we meeting service level objectives? If yes, take a break

– Identify bottlenecks getting in the way – Address those bottlenecks

– Re-measure. Lather, rinse, repeat…

(11)

Performance goals for the cloud and VM environments

• Run workloads as close to bare metal performance as possible

• Support large servers efficiently: >100 CPUs, multi-TB memory

– Typical: 2 (very common) to 8 socket, 256GB to 1TB RAM

• Support large VMs: >100 vCPUs, 512GB/1TB/2TB RAM

– Typical: 4 to 32 vCPUs, 16GB to 128GB RAM. Typical 4-8GB per vCPU

• Drive 10GbE networking near line rate

• Scale to dozens of servers, hundreds of VMs under single-pane control

• Scale to hundreds of LUNs, paths

(12)

Performance concepts for the cloud and VM environments

• Rules are sometimes different in the cloud. Sometimes “less is more”

• Virtual CPUs

– Excess CPUs can increase VM’s SMP overhead (true on bare metal too)

– Many applications don’t scale with more CPUs, so no point in adding more – Can over-subscribe on x86 – subject to contention and service requirements

• Jitter and performance can worsen if aggregate vCPUs > pCPUs, especially under load

• No individual VM should have its vCPUs > pCPUs. Ideally fits in single server socket

• Memory

– On bare metal: assign it all or it just goes to waste

– In cloud: excess memory assigned to a VM reduces number of VMs you can host – In a VM: excess memory increases NUMA latency since spreads over sockets

(13)

Oracle VM performance in cloud

Performance concepts for the cloud and VM environments

Oracle VM Server for x86: optimizing CPU, Memory, I/O

Oracle VM Server for SPARC: optimizing CPU, Memory, I/O

Recommendations and Resources

1

2

3

4

(14)

Oracle VM for x86 Architecture

Oracle VM Manager

• Centralized management server

• Web browser-based: No client required

• Manage hundreds or thousands of VMs centrally

• Support x86 and SPARC from same interface

• Advanced virtualization management included

Oracle VM Server

• Installs on “bare-metal” servers in about a minute

• Linux, Solaris and Windows guests: Supports paravirtualized- and hardware-virtualized kernels and drivers

• High performance at scale: Up to 320 physical CPUs and 6TB memory; Up to 128 vCPU and 2TB memory per VM

(15)

Oracle VM Server for x86: a little bit of architecture

• Xen thin hypervisor in software

• Dom0 for platform control and virtual I/O

• CPUs are time-sliced in software with Xen’s priority-based

credit scheduler

• Memory is not overcommitted or shared between VMs.

Hypervisor does not swap/page VM memory

(16)

Oracle VM Server for x86: optimizing CPU and memory

• Most important tuning options are automated

– Dom0 CPUs are pinned to first server socket, uses memory on that socket

– Guest vCPUs have node affinity: memory is assigned from a socket and CPUs are assigned to the same socket if possible. Reduces remote memory access

• Administrator guidance:

– Avoid guest #vCPU>#CPUs_per_socket to avoid NUMA effect

– Avoid excessive over-subscription of active virtual CPUs (with high %CPU)

• In guest, watch iostat %steal or vmstat %st to see if was dispatchable but preempted – If hyperthreading, count in units of threads (the effects are closely aligned) – Use Dynamic Resource Scheduling to load balance VMs across servers

– Can pin CPUs to ensure placement for NUMA and cache latency

(17)

Oracle VM Server for x86: optimizing CPU: hyperthreading

• Hyperthreading improves some cases, degrades others

• Workload dependent, so must be tested

• Linux tries to spread work over cores, but guest doesn’t know if virtual CPUs

are on the same core or socket. Real topology data isn’t available

• Can help scalable apps when #vCPUs > cores/socket but < threads/socket

when keeps allocation on same socket

• Best for multi-threaded applications that can drive multiple vCPUs

• Can reduce per-vCPU performance and affect single-thread dominated

applications

(18)

Oracle VM Server for x86: optimizing CPU, Memory, I/O

• Virtual machine ‘domain type’ and weights

– HVM + PV drivers gives best performance on current systems, 64-bit guests – Drivers available for Oracle Linux, Solaris, Windows

– PVM mode may still outperform with 32-bit guests, and on old systems

• Credit scheduler useful to reflect importance when CPU constrained

– but never make a guest (domU) weight higher than dom0

• Hugepages help large memory guests

– Especially for Oracle DB with large SGA to reduce TLB footprint and thrashing – Must be enabled for PVM guests

• There can be trade-offs for flexibility

(19)

Oracle VM Server for x86: optimizing I/O

• Storage

– Physical disk (FC or iSCSI LUN) outperforms virtual disks from repositories – Physical disk is needed for Oracle RAC

– Virtual disks easier to manage, more common in multitenant and cloud usage

• NFS vs. OCFS2 on iSCSI or FC LUN repositories

– No clear winner (IMO) between SAN and NAS – Ease of use and administration for NFS

– OCFS2-based repository provides thin clone, which is very handy – vdisk performance may not be much different. YMMV

(20)

Oracle VM Server for x86: optimizing network I/O

• NUMA sensitive on fast networks (10GbE)

• Main tuning actions (dom0 sizing and pinning) are now set by default

– Substantial benefit on network performance with this one setting – Should review if running old versions of Oracle VM

– Can pin key guests to same socket as dom0 (not a typical cloud use case)

• NUMA is top item, other tuning options listed in whitepaper (see

references at end of presentation)

– Balance IRQs if individual CPUs are saturated – Set TCP windows

(21)

Oracle VM performance in cloud

Performance concepts for the cloud and VM environments

Oracle VM Server for x86: optimizing CPU, Memory, I/O

Oracle VM Server for SPARC: optimizing CPU, Memory, I/O

Recommendations and Resources

1

2

3

4

(22)

Oracle VM Server for SPARC

Isolated OS and applications in each logical domain

Firmware-based hypervisor Each logical domain runs in dedicated CPU threads and RAM for bare-metal CPU performance

SPARC Server Oracle Solaris 11 Oracle Solaris 11 Database Domain Oracle Solaris 11 Oracle Solaris 11 Database Domain Oracle Solaris 10 Oracle Solaris 11 Database Domain Oracle Solaris 11 Oracle Solaris 11 Database Domain

Domain with zones Solaris 10 Domain GP Domain Domain with DB zones

SPARC Hypervisor

S10 zone S11 zone

(23)

Oracle VM Server for SPARC: a little bit of architecture

• Thin hypervisor in SPARC firmware

• Control domain used for platform control

– Acts as a service domain to provide virtual I/O

– Optional additional service domains for resiliency and non-disruptive upgrade

• CPUs multiplexed in hardware, and cores/threads dedicated to domains

• Memory is not over-committed. Hypervisor does not swap/page VM

memory. No memory side-channel security vulnerabilities like other

vendors, no performance anomalies due to sharing memory

• Physical I/O available via SR-IOV and PCIe bus assignment to domains

(24)

Oracle VM Server for SPARC Virtualized I/O

Architecture

` Device Driver LDC Hypervisor Virtual Device Service App LDC LDC App Virtual Device Driver Solaris I/O stack App LDC App Service Domain Guest Domain Guest Domain LDC LDC kernel kernel kernel Physical Devices Virtual Device Driver LDC Solaris I/O stack Solaris I/O stack

(25)

Oracle VM Server for SPARC: optimizing CPU, Memory

• Every domain has dedicated cores/threads

– Logical Domains manager assigns CPUs and memory to reduce NUMA latency

• CPU and memory tuning measures

– Size domains for their expected load, especially service domains

• Best practices whitepaper gives guidance for initial values

• Measure utilization (vmstat, mpstat, ldm list) and adjust suitably. Can add/remove CPUs on the fly – Allocate whole cores to domains rather than threads

– Chip multi-threading , hardware-assisted cryptography, large pages all available by default. Solaris knows how to dedicate cores to critical threads

– Really, that’s pretty much it

(26)

Oracle VM Server for SPARC: optimizing I/O

• Many I/O options. First choice is virtual I/O vs. physical

– Virtual I/O is proxied through service domains

– Physical I/O is directly performed by the application domain

• Virtual I/O can provide near-native network and disk performance

– Most flexible (permits live migration, easiest provisioning, most device sharing)

– Preferred in general case, and especially for cloud environment

– Very important: size service domains to drive I/O (based on load)

• Virtual disk: best performance with FC or iSCSI LUN backends

• Virtual network: Can use aggregations, jumbo frames, just as on bare metal

• Virtual HBA (NEW!)

Improves on vdisk: better multi-pathing, any SCSI device type (disk, tape, cd/dvd), improved LUN management, with similar performance

(27)

Oracle VM Server for SPARC: optimizing I/O

• Physical I/O for ‘extreme performance’ use case

• Native performance and scaled down service

domain requirements

• Additional considerations when using physical I/O:

RAS, live migration, scaling, device sharing

• Can assign entire PCIe bus – acts like bare metal

• SR-IOV with qualified Ethernet, IB or FC cards

• Useful for datacenter maximum performance, but

not generally a cloud deployment method

(28)

Oracle VM performance in cloud

Performance concepts for the cloud and VM environments

Oracle VM Server for x86: optimizing CPU, Memory, I/O

Oracle VM Server for SPARC: optimizing CPU, Memory, I/O

Recommendations and Resources

1

2

3

4

5

(29)

Recommendations / Summary

• Oracle VM is designed for out of the box performance wherever possible

– “low touch” performance important for cloud

• Apply best practices (which are often the defaults)

• Stay with current releases to benefit from enhancements

• Tuning may be needed to achieve service levels in some cases

• Measure and tune to target workloads

– Measure baseline performance before there is a problem

– Use baseline figures so you can see what’s different when there is a problem – Use metrics and tests that are meaningful to application and service level

requirements.

(30)

Oracle VM – Foundation for Cloud

• At the core of Oracle’s Cloud strategy

• Integrated VM lifecycle and cloud management solution with Oracle Enterprise Manager

• Supports both x86 and SPARC • Certified for Oracle products • Ready for OpenStack

• Cloud platform for Oracle & Non-Oracle applications

– Supports Oracle Linux, Oracle Solaris, Microsoft

Windows, and other Linux distributions

(31)

Stay Connected

• Oracle.com/virtualization

• @ORCL_Virtualize

• Facebook.com/OracleVirtualization

• Oracle VM Group

• Blogs.oracle.com/virtualization

Join our communities, visit our websites for more information

Youtube.com/OracleVirtualization

Download:

www.oracle.com/technetwork/server-storage/vm/downloads/index.html

(32)

Learn More

• Oracle VM Documentation landing page

http://www.oracle.com/technetwork/server-storage/vm/documentation/index.html

• Oracle VM whitepapers

http://www.oracle.com/technetwork/server-storage/vm/overview/index-160875.html

– Oracle VM Server for SPARC Best Practices: http://www.oracle.com/technetwork/server-storage/vm/ovmsparc-best-practices-2334546.pdf

• Oracle Virtualization blog: https://blogs.oracle.com/virtualization/

• MOS notes:

– “Oracle VM Server for SPARC (LDoms) Document Index (Doc ID 1367098.1)”

– “CPU Over Commitment and Scheduling on Oracle VM Server 3.x (Doc ID 1640862.1)”

– “How to detect an Hyper-Threading or/and a Multi-Core CPU on Oracle Linux” (Doc ID 462410.1

– “Oracle VM 3: 10GbE Network Performance Tuning” (Doc ID 1519875.1)

(33)

(34)