Hyper9 Inc 9015 Mountain Ridge Drive
Suite 220 Austin, TX 78759 [email protected] http://www.hyper9.com
Capacity Planning 2.0
Executive Summary
Capacity Planning is the process of ensuring that the IT infrastructure can support agreed upon or target Service Levels in a cost effective and timely manner. Once thought of as nice to have, Capacity Planning has become a must have for virtual deployments where the sharing of underlying hardware resources (and the contention that inevitably arises) is built in by design.
At its simplest level, Capacity Planning can be thought of as the task of balancing supply (CPU, Memory, Storage, I/O) with demand
(applications/SLAs). The virtual environment brings a number of new twists on Capacity Planning however. Firstly, we have a different level of abstraction to deal with – we primarily care about the supply side of capacity planning as pools or clusters of resources. With a certain degree of automation essentially “built in” to the virtualization platform, deploying a new application will frequently boil down to “which cluster or pool should I deploy this application to?” Secondly, whilst Capacity Planners have always been concerned about “waste” – or how much oversupply or “headroom” to bake in – “waste” takes on a whole new meaning in Virtual Environments. The ease which Virtual Machines (VMs) and applications can be created, cloned and moved has led to a proliferation or “unplanned waste” problem, often referred to as “VM Sprawl.” Depending on the environment, waste due to Sprawl could be considerable – proper insight is
key when adequately planning for capacity. Without proper insight into genuine consumption vs waste, we risk vastly over estimating the Virtual Infrastructure resources required.
Finally, VMs don’t exist in isolation. They run complex (possibly multi-tiered) applications that support different lines of business. Being able to profile how applications and business departments utilize (or potentially waste) the underlying physical resources is key. Even without the formal mantle of Chargeback, most companies need to understand how resources are being consumed by the business, so proper contributions can be calculated for budgetary planning and sensible use of resources can be encouraged on the demand side of the equation.
The Hyper9 Approach
Hyper9 has built a powerful search based platform from which multiple Capacity Planning use cases can be satisfied. With search at its heart, multiple artifacts can be generated from a single unified solution such as reports, alerts and trends that give both a real time and historical perspective of the supply and demand side of Capacity Planning. Searching across Hyper9’s rich multi-faceted data set also provides the basis for different perspectives of the collected information - for example, a traditional infrastructure view of the underlying cluster capacity and consumption, but also additional views such as application aware usage “profiles” and mapping of Virtual Infrastructure usage (such as CPU, memory and
storage consumption) to departments or other important business constructs.
Virtual Infrastructure
As covered earlier, managing capacity of a Virtual Infrastructure requires tracking of supply and demand at a more abstract level. For CPU and memory resources, this relates to the underlying clusters or pools of resources. For storage space (and generally I/O), this relates to the datastores or shared storage resources where the Virtual Machines are stored. In either case, Capacity Planning at the Infrastructure level typically distills to two fundamental questions:
1. When will I run out of capacity?
2. How many VMs (or applications) can I add?
For both questions, it is desirable for a management solution to provide immediate answers, and refine those over time as more data becomes available. Hyper9 Performance Analyzer provides answers to the first question. A common question for example is to understand when clusters will run out of memory and CPU resources.
Performance Analyzer renders both a best fit trend line and a projection of 100% utilization over the timeframe of interest. A common follow up question that arises is “who are the top resource consumers?” Search provides an extremely powerful way for narrowing down the top consumers in an interactive fashion. For example, we can quickly find all of the ESX hosts that are part of the “Corp HA” cluster, understand their average CPU utilization, and plot them relative to the overall cluster utilization.
Searching for top host consumers and overlaying against the parent cluster
To answer the second over riding use case of “How many more VMs can I add?” we can leverage Hyper9 trends. Trends are simply a way of tracking the results of a search over time to give a historical context. Lets again take the cluster as the resource container we are working with – we want to be able to track how many more VMs can be added to a cluster from the perspective of the overall memory (or CPU) resources available. This obviously depends on what kind of VMs we are adding (or the typical “profile” – more on that later), but for now, lets extrapolate based on the average VM size across the cluster. The formula to do this is:
SUM Hosts In Cluster {((100 – Host Memory Utilization)/Host Memory Utilization) * #VMs per host}
We have set the target utilization at 100%, but we can also set it to say 80% to allow for some headroom – for example to account for the failure of a host in the cluster (assuming 5 equally powered hosts per cluster say) to ensure there is sufficient capacity even if we lose a host. It is also possible to set the target utilization greater than 100% to simulate adding capacity to the cluster as well.
Number of VMs that can fit in each cluster from a memory perspective
What if we want to be proactively notified if the number of VMs that will fit is getting low? Hyper9 alerts like trends are simply based on an underlying search. We can take this capacity planning use case one step further and construct a proactive alert that will fire when the number of VMs that will fit on a cluster is below some threshold. In this example, we are setting up an alert that will e-mail (and/or send an SNMP trap) when the number of VMs that will fit on our production cluster from a memory perspective falls below 20.
A note on VM Sprawl
As mentioned earlier, genuine consumption and waste are really two sides of the same coin from the demand side of capacity planning. Consider the previous capacity calculations, what part of the predicted growth is coming from genuine consumption vs waste? Hyper9 contains a wealth of pre-built searches and reports to identify common VM Sprawl conditions. Some examples are:
Sprawl Type Description Resources Wasted
Stale VMs VMs that have been powered off or suspended for extended periods
Disk Space Zombie VMs VMs that are powered on but not “actively” being
used – for example, not being logged into in a dev/test environment.
CPU, memory, disk space, I/O
Orphaned Files VMs (and other files) that have been removed from inventory but not deleted from disk
Disk Space Old and Large Snapshots VMs with excessively large or old snapshots Disk Space
Rogue VMs VMs that are powered on or off, that we do not “expect” to see in the environment
Remember, it should also be possible to tie both consumption and waste back to the applications and departments that are generating them.
Application Awareness and Profiling
VMs are not an island! They exist to serve applications and services back to the business. When planning for capacity, we are not just concerned about the “average VM size” within the virtual infrastructure. The pattern of resource consumption will vary greatly across workloads – is this VM a web server, application server or database server for example? Is it a machine being used in dev/test/QA or in production? Hyper9 was designed from the outset to understand the complete stack of the virtual infrastructure and the workloads that run inside it. Again leveraging search, we can build up arbitrary “profiles” of usage of the virtual infrastructure and trend these over
time. For example, we can search for all VMs running SQL server in our environment, or possibly only those being used in production, and leverage the Hyper9 Trend Preview feature to understand the average memory (or CPU, disk space etc…) consumed by a typical VM belonging to this “profile.”
A memory “profile” of SQL Server VMs
These application aware profiles can help to more accurately predict the resource needs of an application, whether we are moving an existing application from the physical world (and want to leverage the experience of prior
conversions), or creating it from scratch in the virtual environment. Furthermore, most critical applications consist of multiple servers, broken into tiers such as presentation, application and database tiers, offering an overall “application service” back to the business (billing, inventory, ERP etc...). An application service could be thought of as a “vApp” [ref] in a purely virtual context – the distinction is important because many companies are rolling out “hybrid” distributed application services that span both physical and virtual servers. Hyper9 provides a number of ways to define and track these abstract distributed applications and their corresponding resource requirements on the virtual infrastructure – this will be covered in depth in a future paper.
Tying it back to the business
We have talked about the need to tie consumption (and waste) back to the business departments / entities that are utilizing the virtual infrastructure. Companies are looking for transparency into the “black box” of virtualization, to justify existing investments, build the confidence to virtualize mission critical applications and ensure costs are equitably divided at budgeting time. Hyper9 provides open APIs and interfaces, both inbound and outbound, so that resource consumption by department / business unit can be reported on and shared with the business.
Hyper9 Trend - Resource Utilization by department
Once business mappings are imported, Hyper9 treats this as just another piece of data so that searches, reports, alerts and trends can be created specific to business departments, units, teams, projects and so on. These insights can be shared in a variety of ways such as embedding Hyper9 views into external portals so teams can get a handle on their consumption (and waste) patterns.
Summary
In this paper, we described a holistic approach to capacity planning that meets the needs of today’s modern virtual deployments. Whilst understanding Capacity Planning from a virtual infrastructure perspective is necessary – it is not sufficient – we need to understand both consumption and waste as it pertains to the applications and business departments that utilize the underlying infrastructure. Only with this complete view can businesses appropriately plan and build up the experience necessary to virtualize the most demanding and mission critical applications.
