Designing the formal system model is complicated and time consuming process. It has to be repeated each time when the workload pattern changed or and application is updated. The goal of our identification experiments is to understand how different control knobs affect the application performance. During application lifetime the quantitative relationships between re- source capacity available for the application and its performance can change due to workload dynamics or application updates. However, the fundamental impact of each resource to the application performance remains the same. We aim to design controller that controls CPU and RAM assignment to VM running components of interactive application. Therefore we run set of experiments to understand the impact of each of control knobs on application performance and its components (tiers).
For the experiments we created a Xen-based test-bed that consists of representative multi- tier web application benchmark - RUBiS [121] (PHP version). RUBiS is a free, open source auction site prototype which simulates real users’ behavior of a popular auction eBay.com. The front-end tier is a Apache http web-server (WS), the back-end is MySQL database (DB). Each tier runs on VM with centOS6. Client requests are issued by dedicated group of machines running RUBiS client emulator. With certain probability clients access different pages of the application. The clients can browse or bid. Browsing does not utilize database tier, while bidding causes significant load on DB. We run mixed workload that has mixed types client accesses. The experiments consist of two parts.
6.3.1 CPU USAGE AND PERFORMANCE
In first part we analyze impact of virtual CPU power on the application response time and re- source demand. Therefore we periodically changed the CPU entitlement and monitor applica- tion performance. We use Xen credit scheduler [37] to assign virtual CPU capacity to the VM. CPU allocation cap value varied from 8 to 100. To understand the effect of workload fluctuation we also modified number of clients sending requests. The number of clients changes from 400 to 1600 by step 400. CPU allocation experiment was conducted for each VM running the application tiers.
Figure1shows the mean response time (MRT) as function of front-end tier CPU entitlement. If we increase CPU cap value, then for each request rate there is a maximum CPU entitlement value that affects MRT. Above the value MRT does not change any more. The graph 2gives alternative view on CPU and response time correlation. It represents relationship between CPU utilization and MRT. Each data point is an average from 20 samples. The presented mea- surements show that changing CPU entitlement provides smooth control of MRT when CPU utilization above 80%. The same correlation we found between DB CPU entitlement and the application response time. Lower utilization levels do not allow to control CPU. It means that we need to keep CPU utilization above certain threshold to control the application response time.
The change of CPU entitlement also affects web server memory consumption. It is shown in figure3. Memory utilization increases with increase of CPU utilization. The reason is that higher CPU utilization leads to increase of MRT (see figure2). Therefore incoming requests instead
100
101
102
103
25 50 75 100
CPU entitlement (Xen Cap)
Response time (ms) Number of clients 400 800 1200 1600 2000
Figure 1:MRT vs CPU entitlement
100 101 102 103 0.7 0.8 0.9 1.0 CPU utilization Response time (ms) Number of clients 400 800 1200 1600 2000
Figure 2:MRT vs CPU utilization
0.2 0.3 0.4 0.5 0.6 25 50 75 100
CPU entitlement (Xen Cap)
Memor y utilization Number of clients 400 800 1200 1600 2000
Figure 3:WS memory utilization vs CPU entitlement
0.61 0.62 0.63 0.64
25 50 75 100
CPU entitlement (Xen Cap)
DB memor y utilization Number of clients 400 800 1200 1600 2000
Figure 4:DB memory usage vs WS CPU entitlement
of being served immediately go to the application queue. As a result the queue size increases and it needs additional memory space. The same effect has web server CPU entitlement on database memory usage (see figure4). Web server cannot process responses of DB tier and DB has to keep them in memory. Reduction the power WS VCPU leads to decrease of DB CPU usage. Figure5shows the effect. Basically low power web server CPU needs more time to process incoming requests.
6.3.2 MEMORY USAGE AND PERFORMANCE
In the second group of experiments we evaluated the effect of VM memory capacity on the application performance. The memory allocation varied from 240 MB to 896 MB. The experi- ment was repeated for different workload intensity of 400, 800, 1200, 1600 clients. We limit minimal VM memory capacity to 240 MB. This is a minimal RAM size required by underlying OS. We cannot go below the value, even if actual memory usage is lower.
Figure 6 presents the relationship between response time and memory utilization. MRT sharply increases when memory utilization reaches 90% threshold. Memory pressure (ratio
0.050 0.075 0.100 0.125
25 50 75 100
CPU entitlement (Xen Cap)
DB CPU utilization Number of clients 400 800 1200 1600 2000
Figure 5:DB CPU utilization vs WS CPU entitlement
100 101 102 103 0.6 0.7 0.8 0.9 Memory utilization Response time (ms) Number of clients 400 800 1200 1600 2000
Figure 6:MRT vs memory utilization
between VM capacity and actual memory consumption) does not affect response time below the threshold. The reason is swapping activity that is shown in figure 7. During swapping process OS tries to free up memory by saving memory pages to a disk. The speed of the disk orders of magnitude slower than memory, therefore the application performance is affected dramatically. We conclude that memory cannot be used for smooth MRT control. Memory is non-compressible resource that cannot be reclaimed without severely affecting application performance. In some case it can lead OOM (out of memory) events, when OS start to kill tasks to free up memory.
6.3.3 CLUSTER WIDE CORRELATION
Resource provisioning of multi-tier applications should provide fair resource allocation across all tiers to avoid shift of resource bottlenecks. Therefore it is important to understand how the change of resource allocation on one of tiers affects the resource consumption of another tier. Based on our previous experiments we analyze cluster wide correlation effects. In figure 8
presented the correlation between DB CPU entitlement and WS memory utilization. The graph shows that the memory usage increases if the CPU entitlement is reduced. Higher request rate leads to higher memory utilization levels. The reason is that request service rate µ provided by available CPU capacity of DB tier is less than incoming request rate λ. DB cannot accept connections from WS and WS stores incoming requests in the queue instead of sending them down to the DB tier. In figures 4 and5 we show that different web server CPU entitlement changes memory and CPU usage of database tier. The knowledge of cluster wide correlation is important when one of the tiers runs under resource pressure. In such situations the second tier is over-provisioned. If scaling policy does not consider this effect, then it would add more resources to the first tier and reclaim resources from the second tier. As a result it brings the second tier to saturated state and the first tier to over-provisioned state. Finally, the policy would not be able to solve instability problem. To avoid described scenario we can use simple approach: increase allocation of all tiers if one of them is close to saturation. After the application performance stabilizes go back to normal scaling policy.
The following can be concluded from the experiments results. CPU and RAM belong to different groups of resources. CPU is compressible resource (as well as network and disk I/O bandwidth) and it can be reclaimed from the application by decreasing it performance. In contrast memory is non-compressible resource (as well as disk space) and we cannot reclaim memory if it is already used by the application. Only free VM memory can be reclaimed. Hence,
0.0 0.1 0.2 0.4 0.5 0.6 0.7 0.8 0.9 Memory utilization Sw ap r ate MB/sec Number of clients 400 800 1200 1600 2000
Figure 7:Swap rate vs memory utilization
0.2 0.3 0.4 0.5 0.6 25 50 75 100
DB CPU entitlement (Xen Cap)
WS Memor y utilization Number of clients 400 800 1200 1600 2000
Figure 8:Effect DB CPU entitlement on WS memory usage
it is important to keep memory utilization below the value that triggers swapping process. To regulate the application response time we need to change CPU entitlement.