The Design and Execution of Performance Testing
Strategy for Cloud-based System
Muhammad Dhiauddin Mohamed Suffian
Faculty of ComputingUniversiti Teknologi Malaysia Johor, Malaysia [email protected]
Fairul Rizal Fahrurazi
Product Quality and Reliability Engineering MIMOS Berhad
Kuala Lumpur, Malaysia [email protected]
Suhaimi Ibrahim
Advanced Informatics School Universiti Teknologi Malaysia
Kuala Lumpur, Malaysia [email protected] Abstract— This study illustrates the design and implementation
of strategy employed for testing the performance of cloud-based system. The strategy involves the analysis of usage scenario of the system, formulation of performance test cases, test environment setup, performance test execution and reporting with the support of open source testing tool. It evaluates the system under test in handling different volume of users, both at server-end and user-end under different network environment from the perspective of system response time. Distributed testing approach was applied for server-side performance while manual approach was used for testing the performance at user-side. The result has demonstrated the practicality and reliability of this strategy in measuring the performance of the software hosted in cloud environment towards passing the performance criteria set by the stakeholders, which is maximum response time of five seconds. Keywords — performance testing, cloud, open source, response time.
I. INTRODUCTION
Performance testing is one of the types of testing, mostly conducted at system testing level. Performance is one of the non-functional requirements that describe the expected normal behavior of any software under specified condition. For a web-based application, it is usually measured in terms of response time under variety of load condition. Performance testing can be in a form of load testing or stress testing. Load testing measures the performance of a system under different load and usage pattern while stress testing determines at what usage pattern or user loads make the system fails to response [1].
As today’s systems are mostly moving to cloud environment, performance and reliability of the system become one of the main concern. Once the system is hosted on cloud platform, it is expected that the system’s administration,
services and transactions should be faster and easier as compared to hosting it on average dedicated servers [2].
A research organization is tasked to develop a cloud-based solution that consists of client-based application and web-based application that provides online services to the end-users via the client application. The idea behind this solution is to enable end-users to access common services offered by government as well as private organizations and perform related transactions. By having this initiative, end-users able to pay utility bills, perform banking transactions, reading news and learn new knowledge via one application. This client application is shipped together with netbook.
As the target users for the system is the public people, it is expected that the system particularly the web-based application that stores all the services can handle huge volume of users. For this reason, a right approach is required to evaluate and measure the performance of the overall system, both at client-side as well as server-client-side. Thus, a realistic performance testing strategy needs to be formulated so that the system meets the performance criteria specified by the stakeholder. This realistic strategy does not only comprises of employing good practices as suggested by the tool and other related works, but also realistic performance test cases, coverage of both server-side and end-user side as well as simulation of various network connection speed.
This paper is organized into several sections. Section II describes related works on performance testing for cloud computing system while Section III gives the overview of activities involved in formulating the performance test strategy. Overview of the system under test is explained in Section IV Section V describes the design and execution of formulated performance test strategy in detail while the performance test results are presented in Section VI. Findings and discussion
sare provided in Section VII while Section VIII summarises the key points of this study together with the recommendation for future works.
II. RELATED WORKS
Nowadays, cloud computing has attracted a big numbers of research communities. The big portion of the researches so far talked about how the availability of the cloud can be fully utilized for the next generation of test technology [3][4][5][6][7].
Parasoft.com [8] mentioned about security and performance reliability of the application layer is critical to the success of cloud-based applications. While security is still a concern, there is a very limited work related to performance test against cloud-based application. The assumption, as cloud computing supports scalability on the fly and cheap, the performance will be taken care by the cloud service provider. To some extent, this is not true since the software testers still need to test and tune the system under tests against a realistic and broad range of performance and behavior conditions. It is even more complex now as there is a new variable introduces which is the cloud that can be the performance bottleneck.
Clear requirements are required before any performance testing can be executed for cloud-based system. Among the important performance requirements that should be specified are throughput level, latency, capacity and ability to operate in lower mode level [9]. The capability of meeting or exceeding the performance requirements will determine the quality of the cloud-based solution.
Similar effort has been taken by using commercial tool in testing cloud application. Important tasks during and after the performance test execution of the cloud should involve environment deployment, monitoring of environment status, controller-agent model implementation, logs analysis, analysis of slow queries, errors visualization as well as statistics comparison [10]. In [11], a framework for monitoring and testing the scalability of web application on the cloud platform was introduced, which emphasizes on various software that play specific roles during the performance test. In the framework, there is software that serves as HTTP server, load balancer, database, cache and benchmark tool.
One important aspect that is rarely specified in detail when discussing performance testing for cloud-based system is regarding formulation of scenario or test cases for performance testing. Creation and generation of test cases or test scripts for performance were mentioned as one of procedures or strategies for testing the cloud system performance [12][13][14] but no further elaboration were done on how the test cases should look like. However, [15] managed to comprehensively discuss the detail elements of a complete test case for web services deployed in cloud environment. Besides that, there is also no clear evidence by existing works on how performance testing for cloud-based system incorporates network speed simulation as part of the strategy.
Since performance testing is a resource intensive activity especially for cloud application, test engineers need to realize various challenges and issues in conducting such test. The
issues may consist of utilization of computing resources, location of test agents, cost and concern on the security. Testing cloud application shall also deal with quality of workload generation, complexity in data analysis and service level agreement [16].
Our research effort is illustrating the strategy employed for conducting performance testing against cloud-based software using an open source testing tool that consists of following elements: distributed approach, clear performance testing process mapping to systematic test process, real world performance test case scenarios, network speed simulation and cover performance behavior for both server-end and user-end.
III. PERFORMANCE TESTING STRATEGY
There are various approaches in formulating the strategy for performance test. [17] outlined test definition, test design, test build, test execution and posttest execution as the integrated approach to web performance testing. According to [18], performance testing should comprise the following activities: identify test environment, identify performance acceptance criteria, plan and design tests, configure test environment, implement test design, execute tests as well as analyse, report and retest. In the context of this study, the activities that have been carried out inherit most activities mentioned in previous approaches but are based on the phases in fundamental test process. Table I below describes the performance testing activities conducted in relation with the phases in test process.
Table 1 Performance testing activities against test process Phase Performance Test Activities Planning and
Control
• Identify performance requirement of the system under test
• Identify requirement for test environment Analysis and
Design • Design the performance test environment • Design performance test cases Implementation
and Execution
• Setup performance test environment
• Execute the performance test for end-user side and server side scenarios
Reporting • Analyse and evaluate performance test results against performance criteria
Closure • Deliver performance test artefacts to stakeholders As this exercise measures the performance of the system, it is essential for the performance test engineers to be clear on which focus area they are addressing; either load test, stress test, scalability or profiling. Thus, this study emphasizes more on evaluating the behavior and response of the system under test under certain expected increasing loads [19], which is load test.
IV. OVERVIEW OF THE SYSTEM UNDER TEST
The software or system under test is mainly made up several sub-systems: client-side system that resides on the netbook or mobile devices, intelligent services portal, location aware services, content aggregator services as well as the cloud or virtualization platform that host all the online services system.
The client-side system is developed using Java language and executed as Java Network Launch Protocol (JNLP) in order to get services from remote web server. This system will request the related online services from the intelligent services portal. The services stored in the portal are also integrated with the content aggregator services and location aware services. This will allow suitable content to be pushed to the end-users of the client system, profiling the users and enable users themselves to mash-up the relevant contents that suit their needs. On the other hand, the cloud platform serves as the host for storing virtual machines, operating the virtual machine instances and providing scalability for the virtual machines’ load. The logical architecture could be depicted as below in Figure 1:
Fig. 1. Logical architecture of the system under test. Physical configuration of the overall system can be viewed in Figure 2 as below:
Fig. 2. Physical architecture of the system under test.
V. DESIGN AND EXECUTION OF PERFORMANCE TEST In designing the test cases for performance test, it should be governed by the performance criteria which have been set earlier during the planning stage. The system needs to respond fast to meet the performance criteria for passing the performance test. If the system can perform beyond those criteria, i.e. faster than the threshold value, the system is said to have better performance and is scalable to handle more users in the future.
The design and execution of this performance test mainly involve two aspects: performance scenario, performance test cases and performance test environment setup.
A. Performance Scenario
Scenario of the performance testing is summarized as below in Table 2:
Table 2 Performance scenario
Sub-System Scenario
Number of Virtual Users
(VU) Client System Response time to load services home page 1 Server System (Intelligent Services Portal + Content Aggregator + Location Aware)
Response time to load services
home page (via client test page) 1, 100, 200, 300, 400, 500,
1000, 1500, 3000
As the default network connection is on typical Local Area Network (LAN) speed, there were also scenarios to simulate real world experience when end-users use the client system in the netbook to access the online services via the Internet. It involved 1 Mbps wired broadband connection and 0.44 Mbps wireless broadband connection.
Stakeholders have specified the threshold level of response time or performance requirements for the overall system, whether client-side or server-side. The response time for the system is specified as below in software requirement specification:
“The response time of the system should be less than or equal to 5 seconds under 100 concurrent users or as specified
by stakeholders.”
This means when the above performance criteria is met, the performance test is pass. In the event when the response time is greater than that although the system does not crash, the performance test is considered as fail. Virtual users of 1500 are the anticipated highest number of users that could access the system at particular time. However, performance test on 3000 virtual users were added to observe how the system will behave if greater number of concurrent users access the system at the same time.
B. Performance Test Cases
Based on identified performance scenario in Table III, comprehensive test cases are derived to cover end-user side and server-side performance. The test cases for executing a performance test for this cloud-based system are as follows in Table 3:
Table 3 Performance Test Cases Test
Case ID Scenario Number of Virtual Users Network Connection Speed Server Load Client-side system TC 1 Response time to load services home page 1 LAN High TC 2 LAN Low
TC 3 Wireless LAN Low
TC 4 1 Mbps Wired Broadband High TC 5 1 Mbps Wired Broadband Low TC 6 0.44 Mbps Wireless Broadband Low Cloud system TC 7 Response time to load services home page (via client test page) 1
LAN Applicable Not
TC 8 100 TC 9 200 TC 10 300 TC 11 400 TC 12 500 TC 13 1000 TC 14 1500 TC 15 3000
C. Performance Test Environment Setup
JMeter, an open source performance test tool was used in conducting the performance test for the system. This tool was selected due to its scalability in increasing number of load or virtual users without user license restriction as imposed by
other proprietary performance test tools. Besides that, the features and capabilities of JMeter are also comparable with those commercial tools, hence it is also free.
The test setup has adopted the distributed testing approach as recommended in JMeter user manual [20]. The pre-conditions as described in the manual have been adhered to before further configuration is done to the JMeter itself. In this setup, there are three terms need to be understood: Master, Slave and Target. Table 4 describes these roles.
Table 4 Terms in JMeter Distributed Performance Testing
Term Scenario
Master • Machine that run JMeter GUI that governs the whole test Slave • Machines that run the JMeter clients generating the loads to the target system Target • System under test
In general, the steps taken for configuring and running the performance test in the JMeter distributed testing environment are as follows:
• In the Master’s machine, add the IP address for all machines used as Slave in the “jmeter.properties” file. • Start JMeter in Master’s machine and record the test
plan (recording the scenarios as described in Table I) • Verify the correctness of test plan.
• Select the test plan for particular scenario (one scenario is one test plan)
• Start all remote JMeter clients in Slave’s machines from JMeter GUI in Master’s machine
• Observe the performance test and record all results once the test completed for further analysis.
Figure 3 depicts the proposed environment setup for JMeter distributed performance testing.
Fig. 3. JMeter distributed testing environment setup [21]. Table 5 outlines the hardware specification of both Master and Slave’s machines used in the test as well as the netbook
Table 5 Hardware Specification for JMeter Machines JMeter
Machine Specification
Master
• Processor : Intel Pentium 2.8 GHz • Memory: 4GB RAM
• Operating System: Windows Vista • Quantity: 1
Slave
• Processor : Intel Pentium 2.8 GHz • Memory: 4GB RAM
• Operating System: Windows Vista • Quantity: 8
Client (Netbook)
• Processor: Intel Atom 1.66 GHz • Memory: 1GB RAM
• Operating System: Windows 7 Starter • Quantity: 3
Based on the proposed environment setup as depicted in Figure 3, the actual performance test environment for the cloud-based system is configured to accommodate the increasing number of user loads up to 3000 concurrent users. Figure 4 below illustrates the actual environment setup for the distributed performance test execution used in this exercise.
Fig. 4. Actual environment setup for distributed performance
test execution.
VI. PERFORMANCE TEST RESULTS
The results of performance test are divided into two groups: server-side performance test results and client-side performance test results.
A. Server-side Performance Test Results
Performance test for the server-side was conducted to evaluate and measure the performance of the online services system (intelligent services portal + content aggregator services + location aware services) hosted in the cloud platform. This was also to determine whether the virtual machines need to be added when the number of concurrent users increases. Table 6 lists the response time for every set of concurrent users while Figure 5 represents the results in graphical format.
Table 6 Response Time Results for Server Performance Number of
concurrent (virtual) users
Response Time (seconds)
1 4.70 100 4.59 200 4.81 300 4.77 400 4.61 500 4.45 1000 4.33 1500 4.85 3000 8.86
Fig. 5. Performance test result for server-side- online services
system.
The yellow-dotted line represents the threshold level for response time of five seconds specified by the stakeholders of the system as described in Section V (A). At 3000 concurrent users, 4 virtual web servers (machine) were used to successfully response to that huge client requests.
B. Client-side Performance Test Results
On the client-side system, only 1 user was tested for all scenarios as described before in Table I. This is because in real situation, only one client-system can run on one netbook at a particular time. The detail results are presented below in Table 7:
Table 7 Response Time Results for Client Performance Client Internet
Connection Server Load Response Time (seconds)
LAN speed High 7.01
LAN speed Low 4.81
Wireless LAN Low 5.33
1 Mbps wired
broadband High 8.57
1 Mbps wired
broadband Low 5.33
Client Internet
Connection Server Load Response Time (seconds) broadband
From the above results, it is obvious that end-users will face slower response time to access online services via client-system when the cloud platform hosting the online services system is under heavy load, especially if end-users have low speed Internet connection.
C. Summary of Performance Test Results
Based on the results from both server-side and client-side performance test, the overall results for this performance test can be summarized in Table 8 below:
Table 8 Performance Test Cases Test Case
ID
Number of Virtual Users
Response Time Result TC 1 1 7.01 PASS TC 2 4.81 PASS TC 3 5.33 PASS TC 4 8.57 PASS TC 5 5.33 PASS TC 6 16.52 – 24.88 PASS TC 7 1 4.70 PASS TC 8 100 4.59 PASS TC 9 200 4.81 PASS TC 10 300 4.77 PASS TC 11 400 4.61 PASS TC 12 500 4.45 PASS TC 13 1000 4.33 PASS TC 14 1500 4.85 PASS TC 15 3000 8.86 FAIL
VII. FINDINGS AND DISCUSSION
From all test results in measuring the overall system’s performance, several findings can be deduced:
• The online services system hosted in the cloud platform could response to specified performance requirement of five seconds from 1 user to 1500 concurrent users without any break.
• At 3000 concurrent users, the system still can response to huge requests without fail but with delay in response time.
• Performance test was rated as FAIL at 3000 concurrent users due to response time of more than 5 seconds performance criteria, not because of the system crashes.
• When the server is under heavy load or many concurrent users are accessing the online services system, a particular end-user may experience slower response time accessing the online services via the client system.
• End-user with fast Internet connection will access the online services faster than those with slow speed in any server load condition.
• Cloud platform allows the virtual web servers to be scaled up in order to meet the performance requirements for faster response time without having to incur additional cost in purchasing physical servers. Additionally, load balancing mechanism can also be incorporated to handle the load.
This effort has demonstrated several areas of concern with regard to the performance test strategy as well as performance test cases design and execution. Performance testing activities should be planned earlier in the life cycle and not during execution or after execution completed. Furthermore, performance test cases should mimic the real-world scenarios. . Real world test cases will upsurge the reliability of test results. When identifying the test cases to simulate, the focus is against “Most common Usage scenarios” and “Business Critical Usage scenarios”. At one point, test cases have to predict the most common scenarios as the system has not been in the market and the only information available is a severity of the requirements.
Apart from that, by understanding the performance requirements, suitable performance test tools can be selected to accommodate different usage profile, loads, strategy and environment setup. This is because different performance test tools have different capabilities and addressing different areas of performance.
One more thing that should not be left out is the performance testers need to be equipped with knowledge on understanding, analyzing and evaluating the performance test results so that the performance behavior of the system under test can be justified clearly with data. This will help testers together with developers to recommend next course of actions for improving and tuning the system’s performance. In more specific context, deeper analysis could be prepared in ensuring that the software performance evaluation for a cloud-based system could be done in the correct ways.
VIII. CONCLUSION AND RECOMMENDATION
The study has demonstrated the success of the test strategy in measuring the performance of cloud-base system, especially under heavy loads. The use of open-source tool for this exercise has proven the capabilities of free performance test tool to conduct large scale performance testing, in comparable with other commercial tools without incurring cost. Besides that, it is important to have proper planning and come out with realistic scenarios for performance testing. This is to ensure the measurement and evaluation on the system’s performance is done correctly. Having the right approach with correct measurement will determine that the system or solution provided to end-users meet the performance criteria, thus improving their experience in dealing the system. Performance testing should not be treated as “when required” or “when requested” basis. When performance test is done using the right strategy, right scenarios and appropriate testing tool, any system would be able to deal with increasing loads once
deployed in the production environment, more stable and more responsive to serve its users.
In the future, several improvements could be done to the strategy. Instead of using physical machines to install the tool for master and slave, it is preferable to host the tool in the cloud. This will allow the ‘virtual’ slave to be added incrementally without having to purchase physical machines for larger volumes of performance testing. Furthermore, the analysis and measurement of the performance test can be expanded further to not just on response time, but also other aspects such as CPU utilization, memory utilization, disk I/O and network latency. It is also better to add more scenarios for the test so that more aspects of the system’s performance can be learned and explored further.
One more element that should be taken into consideration for future improvement is to incorporate performance testing analysis and evaluation at the beginning of coding phase. This means, performance testing should be done by developers at code level during unit, module and integration testing. Having this in place performance defects or bottlenecks related with the codes can be detected, isolated and fixed early before final integration into system level, thus allows performance test at higher test levels to focus on performance defects associated with architecture and network design.
ACKNOWLEDGMENT
Thank you to Product Quality and Reliability Engineering of MIMOS Berhad for sharing the related data and evidences in preparing this experience-based paper.
REFERENCES
[1] B. Hambling, “Software Testing: An ISTQB-ISEB Foundation Guide Second Edition,” British Informatics Society Limited, Swindon, 2010.
[2] VServer Center, “Cloud server vs dedicated server”,
http://www.vservercenter.com/dedicated-server, accessed on 11 September 2013.
[3] Z. Li, C. Yinghui, and T. Fan, “Design and implementation of cloud-based performance testing system for web services,” 6th International ICST Conference on Communications and Networking in China, Harbin, China, 2011, pp. 875-880. [4] Q. Gao, W. Wang, G. Wu, X. Li, J. Wei and H. Zhong,
"Migrating load testing to the cloud: a case study", 2013 IEEE 7th International Symposium on Service Oriented System Engineering (SOSE), Redwood City, 2013, pp. 429 – 434. [5] W. Jenkins, S. Vilkomir, P. Sharma, and G. Pirocanac,
"Framework for testing cloud platforms and infrastructures", 2011 International Conference on Cloud and Service Computing (CSC), Hong Kong, 2011, pp. 134 – 140.
[6] K. Inçki, I. Ari, and H. Sozer, "A survey of software testing in the cloud", 2012 IEEE Sixth International Conference on Software Security and Reliability Companion (SERE-C), Gaithersburg, 2012, pp. 18 – 23.
[7] P. Robinson, and C. Ragusa, "Taxonomy and requirements rationalization for infrastructure in cloud-based software testing", 2011 IEEE Third International Conference on Cloud Computing Technology and Science (CloudCom), Athens, 2011, pp. 454 – 461.
[8] Cloud,SOA and API Testing , “Comprehensive enterprise-grade testing for cloud-based Application”,
http://www.parasoft.com/jsp/solutions/cloud-testing.jsp, accessed on 11 Sept 2013.
[9] M. Hussain and H.M. Abdulsalam, “Software quality in the clouds: a cloud-based solution”, Journal of Cluster Computing, Springer, 2012.
[10] M. Lynch, T. Cerqueus and C. Thorpe, “Testing a cloud application: IBM SmartCloud inotes: methodologies and tools”, Proceedings of the 2013 International Workshop on Testing the Cloud, Lugano, Switzerland, 2012, pp. 13-17.
[11] M. Vasar, S.N. Srirama and M. Dumas, “Framework for Monitoring and Testing Web Application Scalability on the Cloud”, Proceedings of the WICSA/ECSA 2012 Companion Volume (WICSA/ECSA '12), Helsinki, Finland, 2012, pp. 53-60.
[12] G. Denaro, A. Polini and W. Emmerich, “Performance Testing of Distributed Component Architectures”, Book Section of Testing Commercial-off-the-Shelf Components and Systems, Springer, 2005, pp. 293-314.
[13] K. Markande and S.J. Murthy, “Leveraging Potential of Cloud for Software Performance Testing”, Book Section of Cloud Computing: Methods and Practical Approaches, Springer, 2013, pp. 293-322.
[14] I. Chana and P. Chawla, “Testing Perspectives for Cloud-Based Applications”, Book Section of Software Engineering Frameworks for the Cloud Computing Paradigm, Springer, 2013, pp. 145-164.
[15] H.M. Sneed, “Testing Web Services in the Cloud”, Book Section of Software Quality: Increasing Value in Software and Systems Development, Springer, 2013, pp. 70-88.
[16] J. Zhou, S. Li, Z. Zhang and Z. Ye, “Position Paper: Cloud-based Performance Testing: Issues and Challenges”, Proceedings of the 2013 international workshop on Hot topics in cloud services (HotTopiCS '13), Prague, Czech Republic, 2013, pp. 55-62.
[17] B.M. Subraya, “Integrated Approach to Web Performance Testing: A Practitioner's Guide”, IRM Press, 2006.
[18] J.D. Meier, C. Farre, P. Bansode, S. Barber, and D. Rea, “Performance Testing Guidance for Web Applications”, Microsoft Press, United States of America, 2007.
[19] E.V. Veenendaal, “Standard glossary of terms used in Software Testing version 2.2”, International Software Testing Qualification Board, Brussels, 2012.
[20] JMeter User Manual, “JMeter distributed testing step-by-step”,
http://jmeter.apache.org/usermanual/jmeter_distributed_testing_ step_by_step.pdf accessed on 11 September 2013.
[21] Distribution Testing JMeter, “JMeter 2.9 - (remote test/ distributed testing)”, http://gerardnico.com/wiki/jmeter/remote, accessed on 11 September 2013.
