The Stress/Load Testing Simulator

King Saud University

College of Computer and Information Sciences

Computer Science Departement

MSC Project-1

_______________________________________________________________

The Stress/Load Testing

Simulator

Huda Bin Sadiq

426221465

Under the supervision of

Prof. Ghazy Assassa

Revision History

Date Document Changes

12/3/2007 Document creation

19/3/2007 Related Work

20/4/2007 Requirement specification, tools compression 29/5/2007 High Level architecture, literature review

31/5/2007 Literature reviwe, SLTS planning

2/6/2007 SLTS planning update

Table of Contents

Chapter 1... 6

1.1 Introduction... 6

1.2 Performance Testing Overview ... 7

1.3 Related Work (Literature Review)... 9

Chapter 2... 14

2.1 A Sample of Existing Tools... 14

1. Proxy Sniffer…………..………. ... 14 2. Apache Jmeter………….……… ... 14 3. Hammer-Head2…………..……. ... 14 4. Jcrawler………..……... 14 5. Mercury LoadRunner……..….... ... 15 6. Peassler………. ... 15

2.2 Comparing the Tools Features ... 15

2.3 SLTS Requirements Specification ... 19

2.4 SLTS High-Level Architecture Description ... 20

Chapter 3 ... 21

3.1 The Stress/Load Testing Process ... 21

3.2 Stress/Load-Test Planning ... 22

1.Analyzing the application:... 22

1.1 Identifying System Components... 22

1.2 Describing the System Configuration... 23

1.3 Analyzing the Usage Model ... 23

1.4 Task Distribution ... 24

2. Defining Stress/Load testing objectives………..………..………...24

3.1 Defining the Scope of Performance Measurements ... 25

3.2 Defining Virtual User (Vuser) Activities... 27

3.3 Selecting Virtual Users ... 28

3.4 Choosing Testing Hardware/Software... 28

Chapter 4 ... 29

4.1 Metrics and Measurements ... 29

Table of Figures

Figure1: Types of performance testing ... 7

Figure 2: Manual testing is problamatic ... 8

Figure3: A single console controlling several thousand Vusers replaces manual testers ... 9

Figure4. SLTS Architecture…………...……….20

Figure5: The stress/load testing process ... 21

Figure6: An online system accessed by many web users ... 23

Figure7: Measuring end-to-end response time... 26

Figure8: Measuring N/W & server response times... 26

Figure9: Measuring GUI response time... 26

Figure10: Measuring server response time ... 27

Figure11: Measuring MW-to-server response time ... 27

Figure12: The stress/load testing process ... 28

Chapter 1

1.1 Introduction

Software Testing is the process used to help identify the correctness, completeness, security, and quality of developed computer software. Testing is a process of technical investigation, performed on behalf of stakeholders, that is intended to reveal quality related information about the product with respect to the context in which it is intended to operate; it includes the process of executing a program or application with the intent of finding errors.

Testing a system is harder than building it in the face of new risks introduced by the ever-increasing complexity of software and hardware because without programs that properly work, the system will never process information and produce the output for which it was designed, therefore testing procedures should be established, testing roles should be applied and testing data that can test the limits of the program should be created. But performing tests using the manual traditional way is a tedious, time consuming and costly job which has led to the development of testing tools. These tools may be the only practical way to conduct tests easily and efficiently. Also they can help in avoiding trouble, providing vital information, and they can enable an organization to take new opportunities with greater performance and strength.

There are many types of software testing which include unit testing, integration testing, system testing, acceptance testing, functional testing and non-functional testing.

Performance testing is a type of non-functional testing that includes other types of tests which are:

1. Component Testing: Find the behavior and performance of each tier.

2. Load Testing: Testing an application against a requested number of users (real-world

load). The objective is to determine whether the site or system can sustain this requested number of users with acceptable response times.

3. Capacity Testing: Testing to determine the maximum number of concurrent users an

application can manage. The objective is to benchmark the maximum loads of concurrent users a site or a system can sustain before experiencing system failure.

4. Volume or Stress Testing: Load testing over an extended period of time to find stess

points. It includes testing beyond normal operational capacity often to a breaking point to determine the stability of a given system or application. The objective is to validate an application’s stability and reliability.

Figure1: Types of performance testing

1.2 Performance Testing Overview

Application load testing is the measure of an entire web application’s ability to sustain a number of simultaneous users and/or transactions, while maintaining adequate response times. Because it is comprehensive, load testing is the only way to accurately test the end-to-end performance of a website or system prior to going live.

Application stress/load testing enables developers to isolate bottlenecks and find errors in any component of the infrastructure.

Two common methods for implementing this process are manual and automated testing. Manual testing has several built-in challenges, such as determining how to:

ƒ Emulate hundreds of thousands of manual users that will interact with the application to generate load.

ƒ Coordinate the operations of users. ƒ Measure response times.

ƒ Repeat tests in a consistent way. ƒ Compare results.

Because load testing is iterative in nature, the tester must identify performance problems, tune the system, and retest to ensure that tuning has had a positive impact — countless times. For this reason, manual testing is not a very practical option.

Figure 2: Manual testing is problamatic

With automated stress/load testing tools, tests can be easily rerun and the results automatically measured. In this way, automated testing tools provide a more cost-effective and efficient solution than their manual counterparts. Plus, they minimize the risk of human error during testing.

Today, automated stress/load testing is the preferred choice for stress/load testing a web application or a system. Most testing tools typically use three major components to execute a test. These include:

ƒ A control console, which organizes, drives, and manages the stress/load test.

ƒ Virtual users (Vusers), which are processes used to imitate the real user performing a

ƒ Load servers, used to run the virtual users requests.

Figure3: A single console controlling several thousand Vusers replaces manual testers

Using

ƒ Replace manual testers with automated virtual users (Vusers).

ating machine.

These advanced functionalities in turn allow an organization to save time and costly resources.

r capability of a program or ystem and determining that it meets its required results [14]_{. Although crucial to software quality}

these components, automated stress/load testing tools can: ƒ Simultaneously run many virtual users on a single load-gener ƒ Automatically measure transaction response times.

ƒ Easily repeat load scenarios to validate design and performance changes. ƒ Easily generate reports.

1.3 Related Work (Literature Review)

Software testing is any activity aimed at evaluating an attribute o s

and widely deployed by programmers and testers, software testing still remains an art, due to limited understanding of the principles of software [12], [13], [17]_{. The difficulty in software testing}

stems from the complexity of software. The purpose of testing can be quality assurance, verification and validation, or reliability estimation [16], [18], [23]_{. Software performance testing is a}

major area of software testing. Software testing is a trade-off between budget, time and quality

A search of the published literature using the on-line version of Science Citation Index, returned nly two papers: "Performance Testing of Software Systems"[11] _{by Vokolos and Weyuker, and}

oftware (functionality) testing , software testing strategies , software testing

testing" uncovered lots of rising businesses rganizations that offered some sort of tool to aid people doing software performance testing (Ex:

lier performance testing papers are very rare in literature, although erformance is what most people care about because it is an extremely significant issue for man o

the other one is: "Deriving Workloads for Performance Testing" by Avritzer and Weyuker [2]. Similarly, using the ACM on-line search engine to search all available ACM papers, journals and conference proceedings returned only little research, Also the search facility for both the IEEE and IEE journals and conference papers uncovered only little research on the subject.

There is significant literature describing work on software performance modeling [6], [8], [10], [5], [7] [12], [17], [21]

s

techniques [13] _{and automated testing} [14], [20], [23]_{, although all these subjects are related, this work}

does not directly consider how to establish or generate test sets for testing the performance of software, which is the primary goal of our research.

A search of the internet for "software performance o

Proxy Sniffer[36]_{, Apache JMeter}[37]_{, Hammerhead2-Web Testing Tool}[38]_{, DBMonster}[39]_,

JCrawler[40]_{, WAPT}[41]_{, Mercury Load Runner}[42] _{and Peassler}[43]_{). Also their where workshops or}

seminars that included a section on the subject, but the search uncovered very little research papers. So, it appears from our search that there is neither an existing body of research literature nor an extensive collection of practical or experiential information available to help testers and researchers interested in software performance testing, and this is expected since performance testing is a relatively new field. We brief some of the scientific researches that we found in the following paragraphs.

As we mentioned ear p

large industrial projects. Often, the primary problems that projects report after field release are not system crashes or incorrect system responses, but rather system performance degradation or problems handling required system throughput, although the software system has gone through extensive functionality testing, it was never really tested to assess its expected performance. This paper "Performance Testing of Software Systems" by F.I. Vokolos and E.J. Weyuker [11]_{, gives a}

discussion on approaches to software performance testing. A case study describing the experience of using these approaches for testing the performance of a system used as a gateway in a large industrial client/server transaction processing application is presented.

In "Experience with Performance Testing of Software Systems: Issues

[1]

, an Approach, and Case tudy" by Vokolos and Weyuker , an examination of performance testing issues and description

has amined a substantial amount of data collected during architecture reviews. Their goal was to

Testing Real-Time Systems with Genetic lgorithms" suggest that during development, some performance analysis be performed to

eneration" talk about utomating the regression testing process which involves testing the modified program with test S

to an approach to address some of these issues for certain classes of software systems is addressed. This work is based on experience gathered while testing the performance of different industrial software systems. The paper classifies the objectives of performance testing and describes a technique for creating appropriate workloads needed for the testing. Also it looks at the role of requirements and specifications in successful performance testing. A case study is presented describing the results of some recent work that was done to test the performance of an existing system. The approach used drew on an earlier technique that was developed for performance testing, as described in "Deriving Workloads for Performance Testing"[2] _paper.

A. Avritzer and E.J. Weyuker paper: "Investigating Metrics for Architectural Assessment" ex

develop metrics to predict the likely success or failure of a project [4]_{. The goal of an architecture}

review is to assure that the architecture is complete and to help identify likely problems that can be easily addressed at this stage and may have negative impact on the software system that is ultimately produced and categorized according to the cause of the problem and the severity, but would be much more expensive to correct once implementation is in progress. Performance problems identified at this stage might include such things as the lack of performance estimates, the failure to have proposed plans for data collection, or the lack of a performance budget. Developing comprehensive performance testing strategies is essential. The issues that have to be addressed when doing performance testing differ in a number of ways from the issues that must be addressed when doing typical functionality testing.

Briand[1]_{, Labiche and Shousha in their paper "Stress}

A

determine whether designed tasks will likely meet their deadlines. At design time, such performance analysis requires that task execution times be estimated (e.g., based on the expected number of lines of code) since the whole software is not fully developed they propose a Real-Time Scheduling Theory that helps designers determine whether a group of tasks whose individual execution times have been estimated, will meet their deadlines.

Korel[2] _{and Al-Yami in their paper "Automated Regression Test G}

cases in order to establish the confidence that the program will perform according to the (possibly modified) specification. During software maintenance the existing software is modified. There are three types of software maintenance: corrective maintenance is performed to correct an error that has been discovered in some part of the software. Adaptive maintenance is usually performed when the software is modified to ensure its compatibility with the new environment in which it will operate. Perfective maintenance is performed to add new features to the software or to improve performance of the software. Also Korel and Al-Yami proposed a regression test generation method to automatically generate test data that causes a modified program under test to behave incorrectly.

Yang[3] _{and Pollock in there paper "Towards A Structural Load Testing Tool" introduce a testing}

ol for structural load testing which takes a program as input, and automatically determines

aper "A Model-Based Approach for Testing the Performance of Web Applications" by hams, Krishnamurthy and Far studies the performance of Web-based systems. Poor performance

troduced new failure odes and added complexity to protocol design and testing. In addition, the advent of multicast to

whether that program needs to be load tested, and if so, automatically generates test data for structural load testing of the program. There exist many structural testing methods with the main goal of generating test data for executing all statements, branches, definition-use pairs, or paths of a program at least once to discover load sensitive faults, which are programming errors with which the program executes successfully if executed for a short time or with a small workload, but causes the program to fail when it is executed under a heavy load or over a long period of time.

The p [4]

S

can adversely impact the profitability of enterprises that rely on the web applications. As a result, effective performance testing techniques are essential for understanding whether a web-based system will meet its performance objectives when deployed in the real world. The workload of a web-based system has to be characterized in terms of sessions; a session being a sequence of inter-dependent requests submitted by a single user. Dependencies arise because some requests depend on the responses of earlier requests in a session. To exercise application functions in a representative manner, these dependencies should be reflected in the synthetic workloads used to test web-based systems. This makes performance testing a challenge for these systems. A model-based approach to address this problem is proposed. The proposed model uses an application model that captures the dependencies for a Web-based system under study.

Also the recent growth of the Internet and its increased heterogeneity has in m

applications has introduced new challenges of qualitatively different nature than the traditional point-to-point protocols. A method for automatic synthesis of worst and best case scenarios for these new protocols was addressed by Helmy [5]_{, Gupta and Estrin in "The STRESS Method for}

Boundary-Point Performance Analysis of End-to-End Multicast Timer-Suppression Mechanisms". The algorithm uses timing semantics to handle end-to-end delays and address performance criteria. Also The paper "Traffic-aware Stress Testing of Distributed Systems Based on UML Models"[6] _{by Garousi, Briand and Labiche studies a stress test methodology aimed at}

increasing chances of discovering faults related to network traffic in distributed systems.

The paper "An Intentional Approach to the Specification of Test Cases for Database

[7]

Applications" by Willmor and Embury proposes a new approach in which the database states

ftware testing is an art. Most of the testing methods and ractices are not very different from 20 years ago [17], [20]_{. It is nowhere near maturity, although}

required for testing are specified intentionally, as constrained queries, that can be used to prepare the database for testing automatically.

From our search we conclude that so p

there are many tools and techniques available to use. Good testing requires a tester's creativity, experience and intuition, with proper techniques. Plus testing is more than just debugging. Testing is not only used to locate defects and correct them. It is also used in validation, verification process, and reliability issues that include performance measurements.

Chapter 2

2.1 A Sample of Existing Tools

1.

Proxy Sniffer

Proxy Sniffer is a web load/stress testing tool that measures the response time and the stability of web applications under real load conditions, by simulating hundreds or thousands of web users. This product is especially suited for testing the performance of:

ƒ E-Banking applications. ƒ Web Portals.

ƒ E-Commerce shops.

2.

Apache Jmeter

Apache JMeter is a 100% pure Java desktop application designed to load test functional behavior and measure performance. Apache JMeter may be used to test performance both on static and dynamic resources (files, Servlets, Perl scripts, Java Objects, Data Bases and Queries, FTP Servers and more). It can be used to simulate a heavy load on a server, network or object to test its strength or to analyze overall performance under different load types.

3.

Hammer-Head2

Hammerhead-2 is a stress testing tool designed to test out any web server and web site. It can initiate multiple connections from IP aliases and simulated numerous (256+) users at any given time. It also can give numerous options for trying to create problems with a web site (so the developer can fix them). Hammerhead can destroy a web site very quickly so it should only be used to generate heavy loads on local web sites for testing purposes.

4.

Jcrawler

JCrawler is an open-source Stress-Testing Tool for web-applications. It comes with the crawling/exploratory feature. I tester can give JCrawler a set of starting URLs and it will begin

crawling from that point onwards, going through any URLs it can find on its way and generating load on the web application. The load parameters (hits/sec) are configurable.

5.

Mercury LoadRunner

Mercury LoadRunner prevents costly performance problems in production by detecting bottlenecks before a new system or upgrade is deployed. A tester can verify that new or upgraded applications will deliver intended business outcomes before go-live, preventing over-spending on hardware and infrastructure. With LoadRunner, a tester can:

ƒ Obtain an accurate picture of end-to-end system performance.

ƒ Verify that new or upgraded applications meet specified performance requirements. ƒ Identify and eliminate performance bottlenecks during the development lifecycle.

6.

Peassler

Peassler is a web server performance, load, and stress-test tool, it is a powerful HTTP-client/server test application designed to pinpoint critical performance issues in a web site or web server that may prevent optimal experience for a site's visitors. By simulating the HTTP requests generated by hundreds or even thousands of simultaneous users, a tester can test a web server performance under normal and excessive loads to ensure that critical information and services are available at speeds the end-users expect.

2.2 Comparing the Tools Features

Tool1: Proxy Sniffer Tool2: Apache JMeter Tool3: Hammer-Head2 Too4: JCrawler Tool5: Mercury LoadRunner Tool6: Peassler

Feature Tool1 Tool2 Tool3 Tool4 Tool5 Tool6 Priority My

System Justification

1. GUI support √ √ √ √ √ √ High √

2. The generation of summary

reports √ √ √ √ √ √ High √ 3. Customizable PDF, Word reports √ X X X X √ Low X Due to limited time constraints 4. PDF Comparison Report (response-Time comparisons between several test-runs) X X X X √ √ Medium √ 5. The support of

performance tests √ √ √ √ √ √ High √

6. The support of load tests √ √ √ √ √ √ High √

7. The support of stress tests √ √ √ √ √ √ High √

8. The support of ramp tests X X X √ √ √ Medium √

9. The support of regression

testing X √ √ X √ √ High √

10.The generation of result

diagrams √ √ √ √ √ √ High √

11.Test Scenarios generation √ √ √ √ √ √ Medium √

12.Recording & playing back √ √ √ √ √ √ Medium √

13.Simulating large numbers

of simultaneous users √ √ √ √ √ √ High √

14.Specifying no. of current

users √ √ √ √ √ √ High √

15.Simulate user load with

minimum intrusion (shouldn't produce new problems).

√ √ √ √ √ √ High √

16.Specifying Max. loops per

user √ √ √ √ √ √ Medium √

17. The use of data feeders X X X X X X High √

18.Data export of load test

details (from files, excel sheets)

√ √ √ √ √ √ Medium √

20.The support of data driven testing – ability to read and execute the same script over a multiple set of data.

√ √ √ √ √ √ Medium √

21.The support of test

management activities like Test Preparation and Planning, Test Execution, Test Results (Provide templates for test script preparation).

√ √ √ √ √ √ Medium √

22.Maintain logs of all

different runs and executions.

√ √ √ √ √ √ Medium √

23.Is the tool capable of

integrating and interfacing with different systems?

√ √ √ √ √ √ High √

24.The ability to perform end

to end testing with all interfaces and different protocols. √ √ √ √ √ √ Medium X Due to limited time constraints we cannot include all protocols and different interfaces

25.Does the tool support

protocols such as TCP/IP, SNA, X.25 and IFX

√ √ √ √ √ √ Low X Due to limited

time constraints

26.Can send last transaction

status and details along with the new transaction requests.

X X X X √ √ Low X Due to limited

time constraints

27.Touch screen support? X X X X X X Low X

Inapplicable with current requirements

28.Specific command reject

message support? X X X X √ X Low X

Due to limited time constraints 29.Reversal transaction support? X X X X √ X Low X Due to limited time constraints

30.The ability to encrypt data. √ √ √ √ √ √ Low X Inapplicable

requirements

31.Does it allow printing the

transactions and requests? X X X X √ X Low X

Inapplicable with current requirements

32.Tool compliance with

different formats and messages.

√ √ √ √ √ √ Low X Due to limited

2.3 SLTS Requirements Specification

The focus of this work will be the development of a Stress/Load Testing Simulator (SLTS). SLTS will provide the user with all the necessary means in which to adequately test his/her environment.

1. SLTS will be a tool that generates realistic heavy loads that typically results from the activity of many users.

2. A virtual environment is going to be built.

3. The virtual environment will have the capability to include as many users as needed to stress the

system.

4. The virtual users' queries and resource utilization information from the different system servers, e.g. web server, application server and database server that are currently undergoing testing will be gathered and saved.

5. The testing results are going to be analyzed.

6. The analysis should provide precise performance measurements that will be collected during stress/load tests for further examination and analysis.

7. SLTS will provide a graphical real-time view graph of the system undergoing the stress/load test. 8. A summary report is going to be generated.

9. Different types of diagrams are going to be generated.

10.SLTS will operate within a three-tier client/server architecture.

11.The stress/load-testing concept and SLTS is going to be implemented using JAVA for multithreading and for making SLTS platform independent.

2.4 SLTS High-Level Architecture Description

Chapter 3

3.1 The Stress/Load Testing Process

The stress/load testing process consists of several phases:

1. Plan the stress/load test: Define performance testing reqirments, ex: number of concurent users,

transcations to ne performed and required response time.

2. Create the testing scripts:Capture the end-user activties into automated scripts. 3. Define a scenario.

4. Run the defined scenario. 5. Analyze the results.

Figure5: The stress/load testing process

Terminology

1. Vuser: Virtual users (Vusers) emulate the steps of real users. The steps that Vusers perform

should be recorded in a Vuser script.

2. Vuser Scripts: The actions the Vusers performs during the scenario are described in a Vuser

script.

3. Scenario: A scenario is a file that defines the scripts to execute, the number of Vusers to run,

the goals of the test, the computer that will host the Vusers, and the conditions under which the stress/load test runs.

4. Controller: The controller is the machine chosen to manage and maintain the scenarios and

allows controlling all the Vusers of the scenario from a single location. All the elements of a scenario should be defined in the controller.

5. Transactions: To measure the performance of the server, a transaction is defined. A transaction

is an end-to-end measurement of time elapsed in the execution of one or more steps in a business process that the tester is interested in measuring.

3.2 Stress/Load-Test Planning

Stress/Load-Test planning is a three step process that includes: 1. Analyzing the application.

2. Defining Stress/Load testing objectives. 3. Planning Stress/Load testing implementation.

1.

Analyzing the application:

The first step to stress/load test planning is analyzing the application under test. The tester should become thoroughly familiar with the hardware and software components, the system configuration, and the typical usage model. This analysis ensures that the testing environment the tester creates using SLTS will accurately reflect the environment and configuration of the application under test, so the steps to be done can be summarized to:

1. Identifying system components. 2. Describing the system configuration. 3. Analyzing the usage model.

4. Task distribution.

1.1 Identifying System Components

Draw a schematic diagram to illustrate the structure of the application. If possible, extract a schematic diagram from existing documentation. If the application under test is part of a larger network system, the tester should identify the component of the system to be tested. The diagram should include all system components, such as client machines, network, middleware, and servers.

The following diagram illustrates an online system that is accessed by many web users. The web users each connect to the same database to get special data. The customers connect to the database server through the web, using multiple browsers.

Figure6: An online system accessed by many web users

1.2 Describing the System Configuration

Enhance the schematic diagram with more specific details. Describe the configuration of each system component. The tester should be able to answer the following questions:

1. How many users are anticipated to connect to the system?

2. What is the application client's machine configuration (hardware, memory, operating system, software, development tool, and so on)?

3. What types of database and Web servers are used (hardware, database type, operating system, file server, and so on)?

4. How does the server communicate with the application client?

5. What is the middleware configuration and application server between the front-end client and back-end server?

6. What other network components may affect response time (modems and so on)?

7. What is the throughput of the communications devices? How many concurrent users can each device handle?

1.3 Analyzing the Usage Model

Define how the system is typically used, and decide which functions are important to test. Consider who uses the system, the number of each type of user, and each user's common tasks. In addition, consider any background load that might affect the system response time.

For example, suppose 200 employees log on to the accounting system every morning, and the same office network has a constant background load of 50 users performing various word processing and printing tasks. The tester could create a SLTS scenario with 200 virtual users signing in to the accounting database, and check the server response time.

To check how background load affects the response time, the tester could run the scenario on a network where he can simulate the load of employees performing word processing and printing activities.

1.4 Task Distribution

In addition to defining the common user tasks, the distribution of these tasks needs to be examined. For example, suppose that an organization uses a central DB to serve clients across many states and time zones. The 250 application clients are located in two different time zones, all connecting to the same web server. There are 150 in Chicago and 100 in Detroit. Each begins their business day at 9:00 AM, but since they are in different time zones, there should never be more than 150 users signing in at any given time. The tester can analyze task distribution to determine when there is peak DB activity, and which activities typically occur during peak load time.

2.

Defining Stress/Load testing objectives:

Before beginning testing, the tester should define exactly what he wants to accomplish. The following table shows common application testing objectives that our proposed SLTS helps test:

Objective Answers the Question

Measuring end-user response time How long does it take to complete a task? Defining optimal hardware

configuration Which hardware configuration provides the best performance?

Checking reliability How hard or long can the system work without errors or failures? _{Or How stable is the system under a heavy work load?} Checking hardware or software

upgrades How does the upgrade affect performance or reliability?

Evaluating new products Which server hardware or software should you choose? or _{What is the best server for 100 users?}

performance degradation?

Identifying bottlenecks Which element is slowing down response time? Or

What is the cause of degradation in performance?

Accept the system Is the system stable enough to go live? Or The system is not stable due to perfomance issues and an immediate

action needs to be done?

Also it is important to decide when to test. Stress/load testing is necessary throughout the product life cycle. The following table illustrates what types of tests are relevant for each phase of the product life cycle:

Planning and Design

Development Deployment Production Evolution

Evaluate new

products Measure response time Check reliability Measure response time Check HW or SW upgrades

Measure response

time Check optimal hardware configuration Measure response time Identify bottlenecks Measure system capacity

Check HW or SW upgrades

Measure system capacity

Check reliability

3.

Planning Stress/Load testing implementation:

The next step is to decide how to use SLTS to achieve the testing goals, a tester should: 1. Define the scope of performance measurements.

2. Define virtual user activities. 3. Select the virtual users.

4. Choose the testing hardware/software.

3.1 Defining the Scope of Performance Measurements

A tester can use our SLTS to measure response time at different points in the application. The tester should determine where to run the virtual users and which virtual users to run according to the test objectives:

ƒ Measuring end-to-end response time:

A tester can measure the response time that a typical user experiences by running a GUI Vuser. GUI Vusers emulate real users by submitting input to and receiving output from the client

application. A tester can run the GUI Vusers at the front end to measure the response time across the entire network, including a terminal emulator or GUI front end, network, and server.

Figure7: Measuring end-to-end response time

ƒ Measuring network and server response times:

A tester can measure network and server response time, excluding response time of the GUI front end, by running Vusers on the client machine. Virtual users emulate client calls to the server without the user interface. When a tester runs many virtual users from the client machine, the tester can measure how the load affects network and server response time.

Figure8: Measuring N/W & server response times

ƒ Measuring GUI response time:

A tester can determine how the client application interface affects response time by subtracting the previous two measurements:

GUI response time = (end-to-end) – (network and server)

ƒ Measuring server response time:

A tester can measure the time it takes for the server to respond to a request without going across the network. When a tester runs virtual users on a machine directly connected to the server, he can measure server performance.

Figure10: Measuring server response time

ƒ Measuring middleware-to-server response time:

A tester can measure response time from the server to middleware if he has access to the middleware and it's API (Application Programmable Interface). He can create virtual users with the middleware API and measure the middleware-server performance.

Figure11: Measuring MW-to-server response time

3.2 Defining Virtual User (Vuser) Activities

A tester should create Vuser scripts based on his analysis of Vuser types, their typical tasks and his test objectives. Since Vusers emulate the actions of a typical end-user, the Vuser scripts should include the typical end-user tasks. For example, to emulate an online accounting client, a tester should create a Vuser script that performs typical online accounting tasks. He would browse the pages that he normally visits to perform his tasks.

The tester decides which tasks to measure based on his testing objectives and define transactions for these tasks. Transactions measure the time that it takes for the server to respond to tasks submitted by Vusers (end-to-end time). For example, to check the response time of a hospital web server supplying a search for a patients file, define a transaction for this task in the Vuser script. And to load the web server a tester can emulate peak activity by defining for example 100 users simultaneously updating or retrieving the patients files information.

3.3 Selecting Virtual Users

Before deciding on the hardware configuration to use for testing, a tester should determine the number and type of Vusers required. To decide how many Vusers and which types to run, the tester should look at the typical usage model, combined with the testing objectives.

3.4 Choosing Testing Hardware/Software

The hardware and software should be powerful and fast enough to emulate the required number of virtual users.

3.3 Other Phases of the Stress/Load Testing Process

After completing the planning phase, which defines the performance testing reqirments, for example the number of concurent users, transcations to be performed and the required response time the tester can move to the second phase which is creating the testing scripts to capture the end-user activties into automated scripts. After that the tester can go to phase three where he defines the testing scenarios. The next phase can be done where he runs the defined scenario. Finally the last phase where he can analyze the results.

Chapter 4

4.1 Metrics and Measurements

During Performance Testing, a tester should track the following metrics for each server within the topology:

1.

SLTS Counter Metrics

For each server utilized (including application servers, database servers, etc.), a tester can obtain the following metrics from monitoring the test:

1. Server Level Metrics:

a. CPU Metrics

i. % User CPU

ii. % Privileged CPU (System CPU) iii. % Total CPU

b. System Metrics

i. System Calls/sec ii. Context Switches/sec c. Memory Metrics

i. Pages /sec

2. SLTS Response Times: Monitor average transaction response times for each state.

3. Transactions per Seconds (TPS): Monitor average transactions per sec (Hits/Sec) for each

state.

4. Throughput: Monitor average throughput (KB) for each state.

References

[1]

Vokolos and Weyuker, "Experience with Performance Testing of Software Systems: Issues, an Approach, and Case Study", IEEE transactions on software engineering, vol. 26, no. 12, December 2000.

[2] A. Avritzer and E.J. Weyuker, "Deriving Workloads for Performance Testing" Software-_{Practice and Experience, vol. 26, no. 6, pp. 613-633, June 1996.} [3] A. Avritzer and E.J. Weyuker, "Monitoring Smoothly Degrading Systems for Increased _{Dependability", Empirical Software Eng. J., vol. 2, no. 1, pp. 59-77, 1997.} [4] A. Avritzer and E.J. Weyuker, "Investigating Metrics for Architectural Assessment", Proc. IEEE _{Fifth Int'l Conf. Software Metrics (Metrics '98), pp. 4-10, Nov. 1998.} [5] Beizer, Boris, Black-box Testing: techniques for functional testing of software and systems, _{New York: John Wiley & Sons, 1995.} [6] D. Ferrari, G. Serazzi, and A. Zeigner, Measurement and Tuning of Computer Systems. Prentice _{Hall, 1981.} [7] E. Kit, Software Testing in the Real World. New York: Addison-Wesley, 1995.

[8] M. Loukides, System Performance Tuning. O'Reilly & Associates, 1990.

[9] B. Marick, The Craft of Software Testing. Englewood Cliffs, N.J.Prentice Hall, 1995.

[10] C.U. Smith, Performance Engineering of Software Systems. New York, Addison-Wesley, 1990. [11]

F.I. Vokolos and E.J. Weyuker, "Performance Testing of Software Systems", Proc. ACM, Proceedings of the first international workshop on Software and performance , Oct 1998, Pages 80 - 87.

[12]

Victor R. Basili, Richard W. Selby, Jr. "Comparing the Effectiveness of Software Testing Strategies", Technical Report, Department of Computer Science, University of Maryland, College Park, 1985.

[13] Boris Beizer, Software Testing Techniques. Second edition. 1990

[14] Hetzel, William C., The Complete Guide to Software Testing, 2nd edition, Wellesley, Mass. _1988. [15] William E. Howden. Functional program Testing and Analysis. McGraw-Hill, 1987.

[16] Michael R. Lyu, Handbook of Software Reliability Engineering. McGraw-Hill publishing, 1995. [17] Myers, Glenford J., The art of software testing, Wiley 1979.

[19] Norman Parrington and Marc Roper, Understanding Software Testing, Published by John Willey _{& Sons, 1989.} [20] Progress toward automated software testing; Richard A. DeMillo; Proceedings of the 13_{international conference on Software Engineering, 1991, Page 180.} th [21] Dick Hamlet; Foundations of software testing: dependability theory; Proceedings of the second _{ACM SIGSOFT symposium on Foundations of software engineering, 1994, Pages 128 – 139.} [22] Smith, C. U. Performance Engineering of Software Systems. Addison-Wesley, 1990.

[23] Yang, M.C.K.; Chao, A. Reliability-estimation and stopping-rules for software testing, based on _{repeated appearances of bugs; IEEE Transactions on Reliability, vol.44, no.2, p. 315-21, 1995.} [24] Joe W. Duran, Simeon C. Ntafos, "An Evaluation of Random Testing", IEEE Transactions on _{Software Engineering, Vol. SE-10, No. 4, July 1984, pp438-443.} [25] http://www.cs.cmu.edu/afs/cs/project/edrc-ballista/www/

[26] http://www.numega.com/devcenter/bc.shtml

[27] http://www.rstcorp.com/definitions/software_testing.html

[28] http://www.cnet.com/Content/Features/Dlife/Bugs/?dd

[29] Lionel C. Briand, Yvan Labiche, Marwa Shousha ,June 2005, Proceedings of the 2005 _{conference on Genetic and evolutionary computation GECCO '05 , ACM Press.}

[30] Cheer-Sun D. Yang, Lori L. Pollock ,May 1996, ACM SIGSOFT Software Engineering Notes , Proceedings of the 1996 ACM SIGSOFT international symposium on Software testing and analysis ISSTA '96, Volume 21 Issue 3 , ACM Press.

[31] Mahnaz Shams, Diwakar Krishnamurthy, Behrouz Far, November 2006, Proceedings of the 3rd _{international workshop on Software quality assurance SOQUA '06, ACM Press.}

[32] Bogdan Korel, Ali M. Al-Yami ,March 1998,ACM SIGSOFT Software Engineering Notes , Proceedings of the 1998 ACM SIGSOFT international symposium on Software testing and analysis ISSTA '98, Volume 23 Issue 2, ACM Press.

[33] Ahmed Helmy, Sandeep Gupta, Deborah Estrin, February 2004, IEEE/ACM Transactions on _{Networking (TON), Volume 12 Issue 1 , IEEE Press.} [34] Vahid Garousi, Lionel C. Briand, Yvan Labiche ,May 2006, Proceeding of the 28th international _{conference on Software engineering ICSE '06 , ACM Press.} [35] David Willmor, Suzanne M. Embury ,May 2006, Proceeding of the 28th international _{conference on Software engineering ICSE '06, ACM Press.} [36] http://www.proxy-sniffer.com

[37] http://jakarta.apache.org/jmeter/ [38] http://hammerhead.sourceforge.net/ [39] http://sourceforge.net/projects/dbmonster/ [40] http://jcrawler.sourceforge.net/ [41] http://www.loadtestingtool.com/ [42] http://www.mercury.com/us/products/performance-center/loadrunner/ [43] http://www.paessler.com/webstress [44] http://ltp.sourceforge.net/ [45] http://OpenSTA.org/ [46] http://www.hpl.hp.com/research/linux/httperf/ [47] http://www.acme.com/software/http_load/ [48] test-tools.technologyevaluation.com [49] http://www.extremetech.com/article2/0,1697,1178548,00.asp [50] http://proquest.umi.com/pqdweb [52] http://search.ebscohost.com/ [52] http://infolab.stanford.edu/~burback/watersluice/node24.html [53] http://agiletesting.blogspot.com/2005/02/performance-vs-load-vs-stress-testing.html [54] http://www.west-wind.com/presentations/webstress/webstress.htm [55] http://www.adventnet.com/products/qengine/performance-testing.html?gclid=CMX1wt-_{W1IoCFRJxMAodO2ijgg} [56] http://www-128.ibm.com/developerworks/webservices/library/ibm-stress/ [57] http://www.riskglossary.com/link/stress_testing.htm