Performance Testing of a Cloud Service

(1)

Performance Testing of a Cloud Service

Trilesh Bhurtun, Junior Consultant,

Capacitas Ltd

UKCMG Annual Conference 2012 - Performance Testing of a Cloud Service

Agenda

• Introduction

• Objectives

• Environment

• Tests and Results

• Issues

• Summary

(2)

Customer

• Over 99% of the company's revenue is earned via the ecommerce service

• Highly Competitive Market

• Business Critical Service

• 20% year on year growth

• Frequently introducing new products to keep up with the competition and to increase revenue

• Large ICT infrastructure mirrored across 2 data centres

Introduction

• Look at benefits arising from migrating the web and logic infrastructure that currently supports the ecommerce service to a cloud service

• Proposed a private cloud service with virtual machines (VMs)

• Some initial sizing of the number of virtual machines (VMs) required to meet business demand had been delivered by the service provider

(3)

Introduction

• This sizing had been achieved using assumptions around the capacity of the VM environment relative to the current production environment

• Customer wanted to conduct a proof of concept to assess whether the cloud service is fit for purpose from a capacity and performance perspective

Introduction

• Capacitas offers performance testing as a managed service to this customer amongst other services

• This includes – Risk Assessment – Writing Test Plan – Run Performance Tests – Produce Test Report

(4)

System Architecture

• Three tier layer

• Interfacing to various B2B and other 3^rd parties

Software Development

• Agile Development Process

• Sprint Planning

• Large number of developers working towards a monthly release

• Pre release activities include:

– Functional Testing – Performance testing

(5)

Objectives

• The main objective for the customer was to understand the capacity and performance characteristics of the ecommerce application on the cloud offering compared to the existing physical infrastructure

• In order to do this, Capacitas planned to carry out a series of tests to measure the ‘resource cost per transaction’ on both virtual servers in the cloud and the customer’s physical infrastructure

Objectives

• In addition, the virtual machines were running on Windows 2008, Capacitas attempted to see if the current Windows 2003 limitation of maximum process size of 1.2 GB is removed in Windows 2008

• Customer had opted to use the existing Load Runner load

generation capabilities in the Test environment for this Cloud Proof of Concept project

(6)

Objectives

Objective Description

1 Prove that the ecommerce web application is stable when running within a VM instance

2 Quantify the capacity throughput per web server VM in business terms

3 Quantify the capacity throughput of the web server VM, relative to the current production web server

4 Quantify the capacity throughput per Logic server VM in business terms

5 Quantify the capacity throughput of the Logic server VM, relative to the current production Logic server

6 Demonstrate that the known web server memory constraint which exists in production is alleviated on a web VM

7 Demonstrate that the CPU cost per business transaction is constant under increasing load

8 Demonstrate that the web server application memory leaks are no worse than those on the current production service, as far as can be reproduced in test

9 Prove that the operational management tools are fit for purpose from a capacity management perspective

Environment

Cloud servers were connected to the existing physical performance testing environment with load generation capability.

(7)

Environment

• The Test environment needed to have the following configured:

– SSL turned on at the web server

– Compression turned off at the web server

• The virtual environment needed to have the following configured:

– SSL turned off at the load balancer

– Compression turned off at the load balancer – Caching turned off at the load balancer – SSL turned on at the web server

– Compression turned off at the web server

Environment

• Service Provider will be able to move the virtual servers over the two physical boxes according to the requirements of the test plan

• The underlying architecture was based on two clustered HP DL585- G7 – 4 CPU * 2.1 GHz AMD Opteron Twelve-Core, 128GB RAM servers

• The servers were running vSphere v4.1

(8)

Limitations

• This approach had the following limitations because of the limited load generation capabilities:

– The logical layer was not loaded realistically during web tests – There was no testing of the multiple machine logical memcached

configurations under significant load – however this issue exists with existing test environment

– There was no significant load on the underlying hardware that supported the virtual machines

Tests

• The tests consisted of the following transactions:

– Homepage – Search

– Product Selection – Additional Options – Login

– Purchase

– Order Confirmation – Order History

(9)

Tests

Setup Performance Counters

Test LoadRunner Scripts Against Cloud Servers

Prepare Test Scenarios

• Service times at the WEB server were much higher for some steps when measured on the cloud server compared to the physical server

Test 1 – 4 vCPU WEB

(10)

Test 1 – 4 vCPU WEB

• Page sizes from the cloud tests were half than those measured from the baseline

• This indicated that compression could be turned on

Test 1 – 4 vCPU WEB

(11)

Test 2 – 4 vCPU WEB (Compression Off)

• Turning Compression off did not improve the performance at the WEB server by much

Test 2 – 4 vCPU WEB (Compression Off)

• Page sizes were similar to the ones measured on the baseline

(12)

Test 3 – 8 vCPU WEB

• The 8 Cores WEB Server was performing worse than the 4 Cores WEB server

• Further investigation into these issues showed that the web site was running in debug mode

Test 3 – 8 vCPU WEB

(13)

Test 4 – 4 vCPU WEB (Debug Off)

• Service times decreased considerably but were still higher than those measured on the physical servers

• The 8 cores web server was still performing worse than the 4 cores

Test 5 – 8 vCPU WEB (Debug Off)

(14)

• Service times on the cloud Logic server was nearly similar to the ones measured on the physical Logic server except for the purchase step

Logic CPU

VMware Counters

• VMware Performance Counters Collected

(15)

VMware Counters

(16)

Response Times

• Response times were worse across the main steps on the cloud servers when compared to physical servers

Home Search Purchase

Physical 3.049 2.655 2.415

4 Cores 6.515 4.795 8.456

8 cores 6.775 4.514 8.924

0 2 4 6 8 10

Response Time (s)

Response Time

Home Search Purchase

Physical 10.24 5.55 2

4 Cores 5.38 3.54 1.3

8 cores 5.29 3.56 1.49

0 2 4 6 8 10 12

Transactions Per Second

Transactions Per Second

• The increase in response time had a negative impact on throughput as well

Issues

• Setup

• Network Connectivity

• Configuration inconsistency

• Firewall

• Incompatible Code

(17)

Conclusion

• Moving the ecommerce service to the private cloud was not a technically and from the discussions economical feasible proposition

• They would need 288 and 64 physical cores (3.0 GHz) at the web and logic server respectively, if they used 4 VCPU servers 2.1 GHz AMD Opteron) then they would need 480 and 80 cores at the web and logic servers respectively

Performance Testing of a Cloud Service