Understanding Host and Guest Memory Usage and Other Memory Management Concepts

(1)

Understanding “Host” and “Guest”

Memory Usage and Related

Memory Management Concepts

Kit Colbert Kit Colbert

Sr. Staff Engineer, VMware Sr. Staff Engineer, VMware

(2)

Disclaimer

This session may contain product features that are

currently under development.

This session/overview of the new technology represents

no commitment from VMware to deliver these

no commitment from VMware to deliver these features in

features in

any generally available product.

Features are subject to change, and must not be included in

contracts, purchase orders, or sales agreements of any

contracts, purchase orders, or sales agreements of any kind.

kind.

Technical feasibility and market demand will affect final delivery.

Pricing and packaging for

Pricing and packaging for any new technologies or features

any new technologies or features

discussed or presented have not been determined.

“These features are representative of feature areas under

“These features are representative of feature areas under development. Feature commitmendevelopment. Feature commitments arets are subject to change, and must not be

subject to change, and must not be included in contracts, purchase orders, or sales agreements ofincluded in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery.”

(3)

Agenda

Motivation

Define host and guest memory usage and

Define host and guest memory usage and ask some questions

ask some questions

Memory management concepts

Answer our questions

Best practices

Summary

Q & A

(4)

Motivation

(5)

Host and Guest Memory Usage

(6)

Using Host and Guest Memory Usage

Useful for quickly analyzing a VM’s

Useful for quickly analyzing a VM’s status

status

–

– Coarse-grained informationCoarse-grained information –

– Important for prompting further investigationImportant for prompting further investigation

Requires an understanding of memory management concepts

–

(7)

Define host and guest memory and

ask some questions

(8)

Host Memory Usage Defined

Host Memory Usage

–

– The amount of physical host memory in megabytes allocated to aThe amount of physical host memory in megabytes allocated to a guest

guest

•

(9)

Guest Memory Usage Defined

Guest Memory Usage

–

– The amount of memory in megabytes actively used by a guestThe amount of memory in megabytes actively used by a guest operating system and it’s applications

operating system and it’s applications –

(10)

Memory Terminology

memory size memory size

total amount of memory total amount of memory

presented to a guest presented to a guest allocated memory allocated memory memory assigned to memory assigned to applications applications unallocated memory unallocated memory memory not assigned memory not assigned

active memory active memory

allocated memory recently allocated memory recently

accessed or used by accessed or used by applications applications inactive memory inactive memory allocated memory not allocated memory not recently accessed or recently accessed or

used used

Guest memory usage measures this

Host memory usage

measures this, sorta…

(11)

Some commonly asked questions

Why is host memory usage so high?

Why is host memory usage greater than

Why is host memory usage greater than guest memory usage?

guest memory usage?

Why is host/guest memory usage different than what I see inside

the guest?

(12)

Memory Management Concepts

(13)

Discussion of Concepts

Focus on high-level concepts, not implementation details

–

– Simpler and easier to understandSimpler and easier to understand –

– Keep the fundamental ideas and drop the Keep the fundamental ideas and drop the unimportant onesunimportant ones

Virtual Memory

Creates uniform memory address space

–

– Operating system maps application virtualOperating system maps application virtual addresses to physical addresses

addresses to physical addresses –

– Gives the operating system memoryGives the operating system memory

management abilities that are transparent to management abilities that are transparent to the application the application “virtual” memory “virtual” memory “physical” memory “physical” memory “machine” memory “machine” memory guest guest hypervisor hypervisor

Hypervisor adds extra level of indirection

–

– Maps guest’s physical addresses toMaps guest’s physical addresses to machine addresses

machine addresses –

– Gives the hypervisor memory managementGives the hypervisor memory management abilities that are transparent to the guest abilities that are transparent to the guest

(15)

Virtual Memory

guest guest hypervisor hypervisor “machine” “machine” memory memory “physical” “physical” memory memory “virtual” “virtual” memory memory “virtual” memory “virtual” memory “physical” memory “physical” memory “machine” memory “machine” memory guest guest hypervisor hypervisor Application Application Operating Operating System System Hypervisor Hypervisor App App OS OS Hypervisor Hypervisor VM VM

(16)

Application Memory Management

Starts with no memory Starts with no memory

Allocates memory through syscall Allocates memory through syscall to operating system

to operating system

Often frees memory voluntarily Often frees memory voluntarily through syscall

through syscall

Explicit memory allocation Explicit memory allocation

interface with operating system interface with operating system

Hyper Hyper visor visor OS OS App App

(17)

Operating System Memory Management

Assumes it owns all physical memory

No memory allocation interface

with hardware

–

– Does not explicitly allocate or freeDoes not explicitly allocate or free physical memory

physical memory

Defines semantics of “allocated” and

“free” memory

–

– Maintains “free” list and “allocated”Maintains “free” list and “allocated” lists of physical memory

lists of physical memory –

– Memory is “free” or “allocated”Memory is “free” or “allocated” depending on which list it resides depending on which list it resides

(18)

How “Allocated” and “Free” Lists Work

OS OS App App A

Allllooccaatteed d LLiisstt FFrreee e LLiisstt

Memory becomes “allocated” or

“free” based on which list (piece

of memory) has a pointer to it

(19)

Hypervisor Memory Management

Very similar to operating system

memory management

–

– Assumes it owns allAssumes it owns all machine memory machine memory –

– No memory allocation interfaceNo memory allocation interface with hardware

with hardware –

– Maintains lists of “free” andMaintains lists of “free” and “allocated” memory “allocated” memory Hyper Hyper visor visor OS OS App App

(20)

VM Memory Allocation

VM starts with no physical

memory allocated to it

Physical memory allocated on

demand

–

– Guest OS will not explicitlyGuest OS will not explicitly allocate

allocate –

– Allocate on first VM accessAllocate on first VM access to memory (read or write) to memory (read or write)

(21)

VM Memory Reclamation

Guest physical memory not

Guest physical memory not “freed”

“freed”

in typical sense

–

– Guest OS moves memory to itsGuest OS moves memory to its “free” list

“free” list –

– Data in “freed” memory mayData in “freed” memory may not have been modified

not have been modified

Hypervisor isn’t aware when guest

frees memory

–

– Freed memory state unchangedFreed memory state unchanged –

– No access to guest’s “free” listNo access to guest’s “free” list –

– Unsure when to reclaim “freed”Unsure when to reclaim “freed” guest memory guest memory Guest Guest Free List Free List

(22)

Guest OS (inside the VM)

–

– Allocates and frees…Allocates and frees… –

– And allocates and frees…And allocates and frees… –

– And allocates and frees…And allocates and frees…

Hyper Hyper visor visor App App Guest Guest free list free list

VM

– – Allocates…Allocates… –

– And allocates…And allocates… –

– And allocates…And allocates…

Hypervisor can’t reclaim memory

through guest frees!

Inside the VM Inside the VM OS OS VM VM

VM Memory Reclamation Cont’d

(23)

What to do about VM memory

reclamation?

(24)

Why not just watch the free list?

Idea

–

– Hypervisor could identify memory used by guest for it’s free listHypervisor could identify memory used by guest for it’s free list –

– Watch that memory area for changes and free physical memory thatWatch that memory area for changes and free physical memory that backs guest memory added to free list

backs guest memory added to free list

Problem

–

– Free list may be hard to Free list may be hard to identifyidentify

•

• Depends on guest type and may change from release to releaseDepends on guest type and may change from release to release

–

– “Free” is not always free“Free” is not always free

•

• “Free memory” may be used in buffer cache or elsewhere by guest“Free memory” may be used in buffer cache or elsewhere by guest

–

(25)

What can the hypervisor do to reclaim

VM memory?

Nothing

–

– Route taken by pretty much everyone except VMwareRoute taken by pretty much everyone except VMware –

– No memory overcommitment! (Inefficient use of memory)No memory overcommitment! (Inefficient use of memory) –

– May try to convince you memory overcommitment isn’t importantMay try to convince you memory overcommitment isn’t important

Something

–

– VMware-innovated techniquesVMware-innovated techniques –

– Supports memory overcommitmentSupports memory overcommitment

•

• Efficient use of memoryEfficient use of memory •

(26)

Memory Overcommitment

Aggregate guest memory greater than host

Aggregate guest memory greater than host memory

memory

–

– Not enough physical memory to satisfy all Not enough physical memory to satisfy all VM memory needsVM memory needs –

– Eventually VM may want physical memory when none is leftEventually VM may want physical memory when none is left

V VMM 11 VVMM 22 VVMM 33 Hyper Hyper visor visor

(27)

Why is memory overcommitment important?

Ensures physical memory is actively used as much as possible

–

– Guest VMs may have lots of unused or Guest VMs may have lots of unused or inactive memoryinactive memory –

– No reason to back one VM’s unused/inactive memory with physicalNo reason to back one VM’s unused/inactive memory with physical memory if other VMs can use it

memory if other VMs can use it –

– Hypervisor can reclaim unused/inactive memory and redistribute toHypervisor can reclaim unused/inactive memory and redistribute to other VMs who will actively use it

other VMs who will actively use it

Increases VM-to-host consolidation ratio

–

– Each VM has a smaller physical memory footprint since Each VM has a smaller physical memory footprint since only itsonly its active memory is resident

active memory is resident –

(28)

VM Memory Reclamation

Techniques

(29)

Simple idea: why maintain many Simple idea: why maintain many copies of the same thing?

copies of the same thing?

–

– If 4 Windows VMs are running,If 4 Windows VMs are running, there are 4 copies of Windows there are 4 copies of Windows code

code –

– Only one copy is neededOnly one copy is needed

Share memory between VMs when Share memory between VMs when possible

possible

–

– Background hypervisor threadBackground hypervisor thread identifies identical sets of memory identifies identical sets of memory –

– Points all VMs at one set ofPoints all VMs at one set of memory, frees the others memory, frees the others –

– VMs are unaware of the changeVMs are unaware of the change

V VMM 11 VVMM 22 VVMM 33 Hyper Hyper visor visor V VMM 11 VVMM 22 VVMM 33 Hyper Hyper visor visor

Transparent Page Sharing (TPS)

(30)

Hypervisor wants to reclaim memory Hypervisor wants to reclaim memory Guest OS is not aware of this

Guest OS is not aware of this

–

– Thinks it owns all physical memoryThinks it owns all physical memory –

– Sits inside its own “box”, unaware it’sSits inside its own “box”, unaware it’s running in a VM or that other VMs running in a VM or that other VMs are running

are running

Goal: make the guest aware so it Goal: make the guest aware so it frees up some of its memory

frees up some of its memory

Hyper Hyper visor visor OS OS App App OS OS App App VM 1 VM 1

Solution: artificially create memory Solution: artificially create memory pressure inside the VM

pressure inside the VM

–

– “Push” memory pressure from the“Push” memory pressure from the hypervisor into the VM

hypervisor into the VM –

– Use “balloon” driver inside the VM toUse “balloon” driver inside the VM to create memory pressure

create memory pressure

VM 2 VM 2 memory pressure memory pressure

Ballooning

(31)

Hyper Hyper visor visor OS OS Guest Guest App App Guest Guest free list free list Balloon Balloon Driver Driver

1.

1. Balloon driver allocates

Balloon driver allocates

memory

2.

2. Balloon driver pins

Balloon driver pins

allocated memory

3.

3. Guest may reclaim other

Guest may reclaim other

memory

4.

4. Balloon driver tells

Balloon driver tells

hypervisor what

memory it allocated

5.

5. Hypervisor frees machine

Hypervisor frees machine

memory backing memory

allocated by balloon driver

6.

6. Hypervisor now has more

Hypervisor now has more

free physical memory

List of List of memory memory

Hypervisor: 8 pieces of

memory are allocated

Guest: 3 pieces of

memory are allocated

Hypervisor: 5 pieces of

memory are allocated

Guest: 6 pieces of

memory are allocated

Inflating Balloon

(32)

Hyper Hyper visor visor OS OS Balloon Balloon Driver Driver

Why can the hypervisor reclaim ballooned

Why can the hypervisor reclaim ballooned memory?

memory?

1. Balloon driver doesn’t

2. OS doesn’t

3. Nothing in the VM relies

on the memory’s value for

correctness

What in the VM assumes the

What in the VM assumes the memory has a specific value?

memory has a specific value?

Because of a contract with Because of a contract with

the hypervisor the hypervisor

Because the memory Because the memory

allocated to an app

allocated to an app _{List of}_{List of}

memory memory

Hypervisor can

safely

safely reclaim ballooned memory because the

reclaim ballooned memory because the

VM does not rely on a

VM does not rely on a particular value for that memory.

particular value for that memory.

Inflating Balloon Cont’d

(33)

Hyper Hyper visor visor OS OS Guest Guest App App Guest Guest free list free list Balloon Balloon Driver Driver

Guest OS swapping, a

possible

side effect of ballooning

Two possibilities for guest free memory:

2. VM doesn’t 2. VM doesn’t have much have much free memory free memory 1. VM has lots 1. VM has lots of free of free memory memory Guest Guest free list free list 1. No swapping 1. No swapping necessary! necessary! 2. Needs to 2. Needs to swap! swap!

Guest OS chooses whether

to swap or not!

Inflating Balloon Cont’d

(34)

Hypervisor can swap VM memory

–

– Swaps out to a per-VM swap fileSwaps out to a per-VM swap file –

– Transparent to the VMTransparent to the VM

Hypervisor swapping is a last resort

–

– Both TPS and ballooning preferredBoth TPS and ballooning preferred

•

• Low overhead for TPSLow overhead for TPS •

• Guest swapping due to ballooningGuest swapping due to ballooning

performs better than hypervisor swapping performs better than hypervisor swapping •

• But both take timeBut both take time

–

– Hypervisor swapping quickly reclaimsHypervisor swapping quickly reclaims memory, but is more expensive overall memory, but is more expensive overall

Swapping

(35)

When to reclaim memory

(36)

Quick Review

Physical memory is not reclaimed on

Physical memory is not reclaimed on a guest that is “free”

a guest that is “free”

–

– VM may accrue lots of physical VM may accrue lots of physical memorymemory

Transparent page sharing is always running

–

– May or may not help reduce a VM’s May or may not help reduce a VM’s physical memory consumptionphysical memory consumption

Ballooning or swapping is the only other way to reclaim memory

–

– Both have performance overheadBoth have performance overhead –

– Want to use only when necessaryWant to use only when necessary

Question: when to invoke ballooning or swapping?

(37)

When To Reclaim Memory

Memory overcommitment

–

– Aggregate guest memory is greater than Aggregate guest memory is greater than host memoryhost memory

Not memory

Not memory overcommitted

overcommitted

–

– Never reclaim memory from guest through ballooning or swappingNever reclaim memory from guest through ballooning or swapping

•

• No need to!No need to! •

• (Transparent page sharing is always running)(Transparent page sharing is always running)

Memory overcommitted

–

– Only reclaim once free physical Only reclaim once free physical memory drops below thresholdmemory drops below threshold

•

• Start ballooning when memory starts to fall toward thresholdStart ballooning when memory starts to fall toward threshold •

• Start swapping as memory nears/passes thresholdStart swapping as memory nears/passes threshold

* Assumes no VM or resource pool memory limits are set! * Assumes no VM or resource pool memory limits are set!

(38)

Answer our questions

(39)

High Host Memory Usage

Why is host memory usage so high?

–

– VM may have used lots of memory in VM may have used lots of memory in the pastthe past

•

• On boot, Windows zeros all guest memoryOn boot, Windows zeros all guest memory •

• Starting several guest applications together, e.g., at boot time or whenStarting several guest applications together, e.g., at boot time or when building a big project, can lead to high memory usage

building a big project, can lead to high memory usage •

• Guest workload spikeGuest workload spike

–

– Host memory usage represents “high water mark” of guest’s memoryHost memory usage represents “high water mark” of guest’s memory usage on non-memory overcommitted hosts

usage on non-memory overcommitted hosts

•

(40)

Host Memory Greater Than Guest Memory

Why can host memory be greater than guest memory?

–

– At some point in the past, the guest consumed a larger amount ofAt some point in the past, the guest consumed a larger amount of memory than it is actively using now

memory than it is actively using now –

– With little to no With little to no memory pressure on the physical host, there is memory pressure on the physical host, there is nono reason for hypervisor to reclaim memory from the guest

reason for hypervisor to reclaim memory from the guest –

– The guest maintains a high host memory usage even though itsThe guest maintains a high host memory usage even though its guest memory usage is small

guest memory usage is small

Expected behavior

–

(41)

Host/Guest Memory Usage Different

Inside Guest

Why is host/guest memory usage different than what I see inside

the guest OS?

–

– Guest memoryGuest memory

•

• Guest has better Guest has better visibility while estimating “active” memoryvisibility while estimating “active” memory •

• ESX active memory estimate technique can take time to convergeESX active memory estimate technique can take time to converge

–

– Host memoryHost memory

•

• Host memory usage doesn’t correspond to any memory metric within theHost memory usage doesn’t correspond to any memory metric within the guest

guest •

• Host memory usage size is based on a VM’s relative priority on theHost memory usage size is based on a VM’s relative priority on the physical host and memory usage by the guest

physical host and memory usage by the guest

Again, this is expected!

(42)

Best Practices

(43)

Host Memory Usage

Large host memory usage OK

–

– Expected behavior much of the time!Expected behavior much of the time!

Host memory usage shouldn’t limit guest

Host memory usage shouldn’t limit guest memory usage

memory usage

–

– Host memory usage should be Host memory usage should be large enough to accommodate guestlarge enough to accommodate guest workload

workload –

– Don’t over commit host memory forcing the guest to continuouslyDon’t over commit host memory forcing the guest to continuously swap

swap

Use shares to adjust relative priorities among VMs

Use shares to adjust relative priorities among VMs when memory is

when memory is

overcommitted

–

(44)

Guest Memory Usage

VM’s memory size should be slightly larger than the average guest

memory usage

–

– Accommodates workload spikes without guest swappingAccommodates workload spikes without guest swapping –

– Hypervisor will handle excess host Hypervisor will handle excess host memory usagememory usage –

(45)

Summary

(46)

Summary

Host/guest memory usage is not as simple as

Host/guest memory usage is not as simple as it looks!

it looks!

–

– Represents complex set of dataRepresents complex set of data –

– Requires understanding of memory Requires understanding of memory management conceptsmanagement concepts –

– Many aspects that are not obvious!Many aspects that are not obvious!

Memory Management Concepts

–

– Guest doesn’t “free” memory in a typical sGuest doesn’t “free” memory in a typical senseense –

– Hypervisor isn’t aware of memory a guest has Hypervisor isn’t aware of memory a guest has “freed”“freed” –

– Hypervisor uses transparent page sharing, Hypervisor uses transparent page sharing, ballooning, and swappingballooning, and swapping to reclaim memory

to reclaim memory –

– Ballooning and swapping used only when host is memoryBallooning and swapping used only when host is memory overcommitted!

(47)

Summary Cont’d

Host/guest memory can be very useful with proper knowledge

–

– Quick way to determine the status of VMQuick way to determine the status of VM –

– Important tool to prompt further investigationImportant tool to prompt further investigation –

– Requires good mental model of memory Requires good mental model of memory management conceptsmanagement concepts

•

(48)

Q&A

Kit Colbert Kit Colbert Sr. Staff Engineer Sr. Staff Engineer VMware VMware

(49)

(50)

Thank you for coming.

Rate your session and

watch for the highest scores!

(51)

(52)

Backup Slides

(53)

Measuring Active Guest Memory

Guest self-measurement won’t work

–

– Each guest uses different method to estimate active memoryEach guest uses different method to estimate active memory –

– Can’t compare activeness data from Windows guest with Linux guestCan’t compare activeness data from Windows guest with Linux guest –

– Comparable estimate important for making allocation decisiComparable estimate important for making allocation decisionsons

Statistical Sampling

–

– Select a subset of memory at randomSelect a subset of memory at random –

– Compute the percentage of pages accessed in a minuteCompute the percentage of pages accessed in a minute –

(54)

Ballooning and Swapping

Ballooning

–

– Provide adequate swap space in guest OSProvide adequate swap space in guest OS

•

• Ballooning can lead to high guest memory swappingBallooning can lead to high guest memory swapping •

• Ballooning not effective if a guest can’t swapBallooning not effective if a guest can’t swap

–

– Limit maximum balloon size if lots Limit maximum balloon size if lots of guest memory pinnedof guest memory pinned

•

• Pinned memory cannot be swappedPinned memory cannot be swapped •

• Use sched.mem.memctl.max to set maximum balloon sizeUse sched.mem.memctl.max to set maximum balloon size

Swapping

–

– Ensure adequate swap space on VMFS datastoreEnsure adequate swap space on VMFS datastore

•