Individual hosts have finite hardware resources, and are susceptible to failure. To mitigate against failure and resource exhaustion, hosts are grouped into clusters, which are essentially a grouping of shared resources. A Red Hat Enterprise Virtualization environment responds to changes in demand for host resources using load balancing policy, scheduling, and migration. The Manager is able to ensure that no single host in a cluster is responsible for all of the virtual machines in that cluster. Conversely, the Manager is able to recognize an underutilized host, and migrate all virtual machines off of it, allowing an administrator to shut down that host to save power.
Available resources are checked as a result of three events:
Virtual machine start - Resources are checked to determine on which host a virtual machine will start. Virtual machine migration - Resources are checked in order to determine an appropriate target host. Time elapses - Resources are checked at a regular interval to determine whether individual host load is in compliance with cluster load balancing policy.
The Manager responds to changes in available resources by using the load balancing policy for a cluster to schedule the migration of virtual machines from one host in a cluster to another. The relationship between load balancing policy, scheduling, and virtual machine migration are discussed in the following sections.
6.1. Load Balancing Policy
Load balancing policy is set for a cluster, which includes one or more hosts that may each have different hardware parameters and available memory. The Red Hat Enterprise Virtualization Manager uses a load balancing policy to determine which host in a cluster to start a virtual machine on. Load balancing policy also allows the Manager determine when to move virtual machines from over-utilized hosts to under- utilized hosts.
The load balancing process runs once every minute for each cluster in a data center. It determines which hosts are over-utilized, which are hosts under-utilized, and which are valid targets for virtual machine migration. The determination is made based on the load balancing policy set by an
administrator for a given cluster. There are three load balancing policies: None, Even Distribution, and Power Saving.
6.1.1. Load Balancing Policy: None
If no load balancing policy is selected, virtual machines are started on the host within a cluster with the lowest CPU utilization and available memory. To determine CPU utilization a combined metric is used that takes into account the virtual CPU count and the CPU usage percent. This approach is the least dynamic, as the only host selection point is when a new virtual machine is started. Virtual machines are not automatically migrated to reflect increased demand on a host.
An administrator must decide which host is an appropriate migration target for a given virtual machine. Virtual machines can also be associated with a particular host using pinning. Pinning prevents a virtual machine from being automatically migrated to other hosts. For environments where resources are highly consumed, manual migration is the best approach.
6.1.2. Load Balancing Policy: Even Distribution
An even distribution load balancing policy selects the host for a new virtual machine according to lowest CPU utilization. The maximum service level is the maximum CPU utilization that is allowed for hosts in a cluster, beyond which environment performance will degrade. The even distribution policy allows an administrator to set a maximum service level for running virtual machines. The length of time a host is allowed to continue at this maximum service level before the Red Hat Enterprise Virtualization Manager intervenes is also set by an administrator. If a host has reached the maximum service level and stays there for more then the set time, virtual machines on that host are migrated one by one to the host in the cluster that has the lowest CPU utilization. Host resources are checked once per minute, and one virtual machine is migrated at a time until the host CPU utilization is below the maximum service threshold.
6.1.3. Load Balancing Policy: Power Saving
A power saving load balancing policy selects the host for a new virtual machine according to lowest CPU utilization. The maximum service level is the maximum CPU utilization that is allowed for hosts in a cluster, beyond which environment performance will degrade. The minimum service level is the minimum CPU utilization allowed before the continued operation of a host is considered an inefficient use of electricity. The even distribution policy allows an administrator to set a maximum and minimum service level for running virtual machines. The length of time a host is allowed to continue at this maximum or minimum service level before the Red Hat Enterprise Virtualization Manager intervenes is also set by an administrator. If a host has reaches the maximum service level and stays there for more the the set time, the virtual machines on that host are migrated one by one to the host that has the lowest CPU utilization. The process continues until the host CPU utilization is below maximum service level. If a host CPU utilization falls below the minimum service level the virtual machines are migrated to other hosts in the cluster if their maximum service level permits. When an under-utilized host is cleared of its remaining virtual machines, it can be shut down by an administrator to preserve power.
In Red Hat Enterprise Virtualization, scheduling refers to the way the Red Hat Enterprise Virtualization Manager selects a host in a cluster as the target for a new or migrated virtual machine.
For a host to be eligible to start a virtual machine or accept a migrated virtual machine from another host, it must have enough free memory and CPUs to support the requirements of the virtual machine being started on or migrated to it. If multiple hosts are eligible targets, one will be selected based on the load balancing policy for the cluster. For example, if an even distribution policy is in effect, the Manager chooses the host with the lowest CPU utilization. If the power saving policy is in effect, the host with the lowest CPU utilization between the maximum and minimum service levels will be selected. The storage pool manager status (SPM, see Section 3.2, “Role: The Storage Pool Manager” for more information) of a given host also affects eligibility as a target for starting virtual machines or virtual machine migration. A non-SPM host is a preferred target host, for instance, the first virtual machine started in a cluster will not run on the SPM host if the SPM role is held by a host in that cluster.
6.3. Migration
The Red Hat Enterprise Virtualization Manager uses migration to enforce load balancing policies for a cluster. Virtual machine migration takes place according to the load balancing policy for a cluster and current demands on hosts within a cluster. Migration can also be configured to automatically occur when a host is fenced or moved to maintenance mode. The Red Hat Enterprise Virtualization Manager first migrates virtual machines with the lowest CPU utilization. This is calculated as a percentage, and does not take into account RAM usage or I/O operations, except as I/O operations affect CPU utilization. If there are more than one virtual machines with the same CPU usage, the one that will be migrated first is the first virtual machine returned by the database query run by the Red Hat Enterprise Virtualization Manager to determine virtual machine CPU usage.
Migration Statistics
A bandwidth limit of 30 Mbps is imposed on each virtual machine migration. A migration will time out after a certain amount of time has passed. The time out happens out after either 300 seconds, or 300 seconds multiplied by a factor of the virtual machine memory divided by 2048, which ever is larger. By default, concurrent outgoing migrations are limited to one per CPU core per host, or 5, which ever is smaller.