Comparison of Memory Balloon Controllers
Presented by:
PNVS Ravali
Advisor:
Prof. Purushottam Kulkarni
Memory Overcommitment I
Server Consolidation
Give each VM only the memory it needs
Memory Overcommit: Total system memory is less than the memory configured to the VMs
Memory Overcommitment II
Techniques supporting Overcommitment
Demand Paging: The hypervisor swaps pages in and out of the memory
Virtual Machine Migration: Migration of VM to address the memory need
Content Based Page Sharing Ballooning
Ballooning
Ballooning - Memory reclamation mechanism
Balloon driver: module inside the Guest OS
VMM inflates and deflates to reclaim some pages from a VM
Balloon Controller : Logic to operate the balloon driver (monitors memory need for a VM)
Balloon Controller
When to balloon out?
Motivation and Problem Description
Motivation
Many Balloon controllers - use different parameters for decision making
Given a workload scenario, which controller to use?
Problem Statement
Evaluate the controllers with different workload settings
Goals
Controllers and set of workloads
The behaviour of a controller with different workload setups
Evaluate controller behaviour in a memory overcommit scenario
Outline
Algorithms of balloon controllers Experiment Setup and Workloads Results in non-overcommit scenario Experiments with overcommit setup Conclusion
Balloon Controller Algorithms
TWS based ballooning [Chiang]
Estimates the True Working Set (TWS) size of the VM using measures like swapin, refault and Committed AS When Guest’s physical memory allocation is greater than its WSS, the swap in and refault events are close to zero.
Modifications to tws controller - CAS based controllers
mod-tws controller
When in FAST or SLOW state, the reduction is by some % of current allocation
When there is a change in CAS, increase or decrease the allocation by the difference in the amount of CAS Gives importance to swapin and refault
new-mod-tws controller
Decrease in CAS indicates decrease in memory allocation When there is swapin or refault, consider decrease in CAS as well
bbox controller
Estimation of WSS [vmware-esx-server]
Tracks VM’s memory pages Uses sampling
bbox controller
Algorithm 1bbox controller logic
1: x←numb of pages tracked
2: wss←numb of pages accessed in sampled interval 3: rss←numb of resident pages
4: if wss >70%of xthen
5: increment curr alloc by 20 % of curr alloc 6: else
7: next alloc= rss∗xwss
RSS based controllers I
rss-swap and new-rss-swap are the controllers which are based on RSS.
Make use of swap and refault
rss-swap controller
Initial memory allocation is set to RSS
If swapin or refault occurs, increase allocation by swapin and refault
Else set allocation to RSS
new-rss-swap controller
Set memory allocation to sum of averages of swapin, refault and RSS observed in the decision interval period
Managing overcommit scenario I
Controllers take per VM decisions
Need for a global policy or controller which manages memory across the VMs
Global controller policy
Take reclaimable memory
Host free memory is subtracted from total demand Take deficit in the ratio of the memory allocations Distribute the obtained memory in the ratio of their demands
ratio= curr−minthresh
Example overcommit memory management
Total memory at host is 1100 MB, min threshold is 100 MB
Table : Example of an overcommit scenario
VM1 VM2 VM3 VM4 curr alloc 250 250 300 300
max alloc 500 500 500 500
demand (controller) 100 200 200 100 max mem (reclaimed) 150 150 200 200 ratio 0.375 0.375 0.5 0.5 normalized ratio 0.21 0.21 0.29 0.29 reclaim 128.57 128.57 171.43 171.43 ratio of demands 0.17 0.33 0.33 0.17 give 100 200 200 100 final alloc 221.43 321.43 328.57 228.57
Workloads used
Workloads
Kernel Compile: CPU intensive, disk io
-Dacapo: Set of workloads, memory need fluctuates over time
Memcached: in memory key-value store
Sysbench OLTP: Database benchmark, read-only, read-write
Metrics for Comparison
Performance Comparison Memory Saved Index (msi):
msi=Xmax allocation possible−curr alloc
Analysis in non overcommit
scenarios
Experiment Setup
IBM blade server with 8 CPU cores with 24GM RAM with 3.10.52 Linux kernel
Steps for the experiments
Cleanup inside the VM
Invoke resource utilization capture scripts inside the VM Start the server which sends the values of metrics as seen inside the VM to the client at the host
Start the workload
Stop the resource capture scripts inside the VM and save them once the workload finishes
Controllers performance vs memory savings
Kernel Compile Workload - Performance and Memory
Savings
Observations
tws saves slightly more memory than mod-tws and new-mod-tws. CAS is less (around 250MB) and keeps fluctuating so tws ignores refault and swapin
Dacapo Workload - Performance and Memory Savings
Observations
CAS value varies between 500 and 700 MB and thus not much msi when compared to kernel compile
tws saves more when compared to mod-tws, new-mod-tws as it follows CAS and ignores swap and refault
Sysbench read-only workload - Performance and
Memory Savings
Observations
tws saving memory at the cost of performance degradation Minimal memory savings for mod-tws, new-mod-tws and
Msi vs performance metrics (di=12 sec)
tws m-tws nm-tws rss-sw n-rss-sw bbox wl p m p m p m p m p m p m kc X X X X X X X X x X x X dac X X X X X X X x x X x X mc(1:9) X X X X X X X X X X X X mc(7:3) x X X X X X X X x X x X sb ro x X X x X x X x x X x X sb rw x X X x X x X x x X x XWorkloads and controllers - relative comparison
Xindicates a performance which is atleast 75% of baseline
Conclusions based on the results
bbox: Does not perform well for any of the workload settings except memcached with a set to get ratio of 1:9
new-rss-swap: Shows results similar to those of the bbox. Points out the importance of refault
tws : Performs well in all cases except sysbench readonly, sysbench read-write and mecached with a set to get ratio of 7:3.Might be an indication that tws controller is not
suitable for workloads which exhibit swap or refault. The mod-tws,new-mod-tws andrss-swap: Close in terms of performance given or the memory savings for all the workload settings
Analysis in overcommitted
scenario
Experiment Setup
Workloads : kernel compile, dacapo, memcached, sysbench, dacapo2
Each VM is set to have a maximum memory allocation of 1GB
Sysbench workload is set to have a read-only transaction load
For memcached, the set to get ratio is set to 1:1
The experiment involves cleanup inside the VMs, starting all the workloads, moving the logs once they finish
The overcommit ratio was varied from 1 to 3 to 6 with decision interval fixed at 12 seconds
Overcommit ratio is defined as :
sum of memory conf igured to V M s
Table : Performance Values of workloads with ocr=3 relative to ocr=1
kc time dac time dac2 memc tput sysb tws 0.96 1.05 0.90 0.99 1.20 mod-tws 1.27 1.40 1.50 0.35 0.70 new-mod-tws 1.25 1.26 1.20 0.37 0.76 rss-swap 1.16 1.33 1.68 0.45 0.61
Table : Performance Values of workloads with ocr=6 relative to ocr=1
kc time dac time dac2 memc tput sysb tws 1.56 3.16 0.77 0.41 0.10 mod-tws 2.55 4.00 3.43 0.18 0.00 new-mod-tws 2.22 3.08 2.31 0.15 0.00 rss-swap 2.25 4.50 3.78 0.21 0.01
As the overcommit ratio increased, the performance of workloads deteriorated.
Conclusions and Future Work
Conclusions
Evaluation of controllers in non-overcommit scenarios Overcommit scenario - more work to be done
Extensions
Analysis in overcommit scenario space - combinations of workloads, decision interval
Make the black box controller swap aware
Incorporate sharing and evaluate overcommit levels when ballooning and sharing co-exist
Thank You
I would like to thank my advisor, Prof. Purushottam Kulkarni for his constant guidance and support throughout the project. I also would like to thank Debadatta Mishra for his support in the form of useful discussions related to work.
Bibliography I
[1] Carl A.Waldspurger“Memory Resource Management in VMware ESX Server”,In proc. Fifth Symposium on Operating Systems Design and Implementation,OSDI ’02
[2] Shashank Rachamalla, Debadatta Mishra, Purushottam Kulkarni “Share-o-meter: An empirical analysis of KSM based memory sharing in virtualized systems”,20th Annual International Conference on High Performance Computing, HiPC 2013, Bengaluru (Bangalore), Karnataka, India, December 18-21, 2013
[3] Jui-Hao Chiang, Han-Lin Li, Tzi-cker Chiueh“Working
Set-based Physical Memory Ballooning”,10th International Conference on Autonomic Computing, ICAC’13
[4] Weiming Zhao, Zhenlin Wang, Yingwei LuoDynamic memory balancing for virtual machines,. In Proceedings of the 2009 ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE), ACM Press, pages 21- 30, March 11-13, 2009, Washington, DC, USA
Bibliography II
[5] L. Chin-Hung“Evaluating the effectiveness of memory overcommit techniques on kvm-based hosting”,2012
[6] Sailee JainAn empirical analysis of ballooning based memory management techniques,Technical report,
Department of Computer Science, Indian Institute of Technology, Bombay, 2014
