Learning Objective
After completing this chapter, you will be able to:
Test performance of a software
What is Performance testing?
The performance testing is a measure of the performance characteristics of an application. The main objective of a performance testing is to demonstrate that the system functions to specification with acceptable response times while processing the required transaction volumes in real-time production database. The objective of a performance test is to demonstrate that the system meets requirements for transaction throughput and response times simultaneously. The main
deliverables from such a test, prior to execution, are automated test scripts and an infrastructure to be used to execute automated tests for extended periods.
Performance testing of an application is basically the process of understanding how the web application and its operating environment respond at various user load levels. In general, we want to measure the latency, throughput, and utilization of the web site while simulating attempts by virtual users to simultaneously access the site. One of the main objectives of performance testing is to maintain a web site with low latency, high throughput, and low utilization.
Why Performance testing?
Performance problems are usually the result of contention for, or exhaustion of, some system resource. When a system resource is exhausted, the system is unable to scale to higher levels of performance. Maintaining optimum Web application performance is a top priority for application developers and administrators.
Performance analysis is also carried for various purposes such as:
During a design or redesign of a module or a part of the system, more than one alternative presents itself. In such cases, the evaluation of a design alternative is the prime mover for an analysis.
Post-deployment realities create a need for the tuning the existing system. A
systematic approach like performance analysis is essential to extract maximum benefit from an existing system.
Identification of bottlenecks in a system is more of an effort at troubleshooting. This helps to replace and focus efforts at improving overall system response. As the user base grows, the cost of failure becomes increasingly unbearable.
To increase confidence and to provide an advance warning of potential problems in case of load conditions, analysis must be done to forecast performance under load.
Typically to debug applications, developers would execute their applications using different execution streams (i.e., completely exercise the application) in an attempt to find errors. When looking for errors in the application, performance is a secondary issue to features; however, it is still an issue.
Performance Testing Objectives
The objective of a performance test is to demonstrate that the system meets requirements for transaction throughput and response times simultaneously. This infrastructure is an asset and an expensive one too, so it pays to make as much use of this infrastructure as possible. Fortunately, this infrastructure is a test bed, which can be re-used for other tests with broader objectives. A comprehensive test strategy would define a test infrastructure to enable all these objectives be met.
The performance testing goals are:
End-to-end transaction response time measurements.
Measure Application Server components performance under various loads.
Measure database components performance under various loads.
Monitor system resources under various loads.
Measure the network delay between the server and clients
Pre-requisites for Performance Testing
We can identify five pre-requisites for a performance test. Not all of these need be in place prior to planning or preparing the test (although this might be helpful), but rather, the list defines what is required before a test can be executed.
First and foremost thing is, the design specification or a separate performance requirements document should:
Defines specific performance goals for each feature that is instrumented.
Bases performance goals on customer requirements.
Defines specific customer scenarios.
Quantitative, relevant, measurable, realistic, achievable requirements
As a foundation to all tests, performance requirements should be agreed prior to the test. This helps in determining whether or not the system meets the stated requirements. The following attributes will help to have a meaningful performance comparison.
Quantitative - expressed in quantifiable terms such that when response times are measured, a sensible comparison can be derived.
Relevant - a response time must be relevant to a business process.
Measurable - a response time should be defined such that it can be measured using a tool or stopwatch and at reasonable cost.
Realistic - response time requirements should be justifiable when compared with the durations of the activities within the business process the system supports.
Achievable - response times should take some account of the cost of achieving them.
Stable system
A test team attempting to construct a performance test of a system whose software is of poor quality is unlikely to be successful. If the software crashes regularly, it will probably not withstand the relatively minor stress of repeated use. Testers will not be able to record scripts in the first instance, or may not be able to execute a test for a reasonable length of time before the software, middleware or operating systems crash.
Realistic test environment
The test environment should ideally be the production environment or a close simulation and be dedicated to the performance test team for the duration of the test. Often this is not possible.
However, for the results of the test to be realistic, the test environment should be comparable to the actual production environment. Even with an environment which is somewhat different from the production environment, it should still be possible to interpret the results obtained using a model of the system to predict, with some confidence, the behavior of the target environment. A test
environment which bears no similarity to the actual production environment may be useful for finding obscure errors in the code, but is, however, useless for a performance test.
Performance Testing Requirements
Performance requirements normally comprise three components:
Response time requirements
Transaction volumes detailed in ‘Load Profiles’
Database volumes
Response time requirements
When asked to specify performance requirements, users normally focus attention on response times, and often wishes to define requirements in terms of generic response times. A single response time requirement for all transactions might be simple to define from the user’s point of view, but is unreasonable. Some functions are critical and require short response times, but others are less critical and response time requirements can be less stringent.
Load profiles
The second component of performance requirements is a schedule of load profiles. A load profile is the level of system loading expected to occur during a specific business scenario. Business scenarios might cover different situations when the users’ organization has different levels of activity or involve a varying mix of activities, which must be supported by the system.
Database volumes
Data volumes, defining the numbers of table rows which should be present in the database after a specified period of live running complete the load profile. Typically, data volumes estimated to exist after one year’s use of the systems are used, but two year volumes or greater might be used in some circumstances, depending on the business application.
Performance Testing Process
Phase 1 – Requirements Study
This activity is carried out during the business and technical requirements identification phase. The objective is to understand the performance test requirements, Hardware & Software components and Usage Model. It is important to understand as accurately and as objectively as possible the nature of load that must be generated. Following are the important performance test requirement that needs to be captured during this phase.
Response Time
Transactions Per Second
Hits Per Second
Workload
No of con current users
Volume of data
Data growth rate
Resource usage
Hardware and Software configurations
Phase 2 – Test Plan
The following configuration information will be identified as part of performance testing environment requirement identification.
Hardware Platform
Server Machines
Processors
Memory
Disk Storage
Load Machines configuration
Network configuration
Software Configuration
Operating System
Server Software
Client Machine Software
Applications
Phase 3 – Test Design
Based on the test strategy detailed test scenarios would be prepared. During the test design period the following activities will be carried out:
Scenario design
Detailed test execution plan
Dedicated test environment setup
Script Recording/ Programming
Script Customization (Delay, Checkpoints, Synchronizations points)
Data Generation
Parameterization/ Data pooling
Phase 4 –Scripting
Phase 5 – Test Execution
The test execution will follow the various types of test as identified in the test plan. All the
scenarios identified will be executed. Virtual user loads are simulated based on the usage pattern and load levels applied as stated in the performance test strategy.
The following artifacts will be produced during test execution period:
Test logs
Test Result
Phase 6 – Test Analysis
Phase 7 – Preparation of Reports
The test logs and results generated are analyzed based on Performance under various load, Transaction/second, database throughput, Network throughput, Think time, Network delay, Resource usage, Transaction Distribution and Data handling. Manual and automated results analysis methods can be used for performance results analysis.
The following performance test reports/ graphs can be generated as part of performance testing:-
Transaction Response time
Transactions per Second
Transaction Summary graph
Transaction performance Summary graph
Transaction Response graph – Under load graph
Virtual user Summary graph
Error Statistics graph
Hits per second graph
t analysis, suggestions on improvement or tuning will
ents to application software, middleware, database
Upgrades to client or server hardware, network capacity or routing.
Throughput graph
Down load per second graph
Based on the Performance repor be provided to the design team:
Performance improvem organization.
Changes to server system parameters.
Common Mistakes in Performance Testing
echniques, Metrics, Workload
R system is better than THEIRS”
the Problem
No general purpose model
Goals =>T
Wrong Evaluation Technique
Overlook Important Parame
Ignore Significant Factors
Inappropriate Experimental D
Inappropriate
No Analysis
Erroneous Analysis
No Sensitivity Analysis
Ignoring Errors in Input
Improper Treatment of Outliers
Assuming No Change in the Future
Results
Omitting Assumptions and Limitations
formance
mance issues must be identified as soon as ossible to prevent further degradation.
asure
hould agree that a performance issue is not just a bug; it is a software rchitectural problem.
r Application Programming terface and application server provider (ISAPI/ASP) applications.
ls ar purpose, at how to instrument tests to best easure various response times.
nce
d into the build that would revent the performance test suite from successfully completing.
irements change and ou must modify a test, perturb only one variable at a time for each build.
Ignoring Variability
Too Complex Analysis
Improper Presentation of
Ignoring Social Aspects
Benchmarking Lessons
Ever build needs to be measured. We should run the automated performance test suite against every build and compare the results against previous results. Also, we should run the per test suite under controlled conditions from build to build. This typically means measuring performance on "clean" test environments. Perfor
p
Performance goals need to be ensured. If we decide to make performance a goal and a me of the quality criteria for release, the management team must decide to enforce the goals.
Establish incremental performance goals throughout the product development cycle. All the members in the team s
a
Performance testing of Web services and applications is paramount to ensuring an excellent customer experience on the Internet. The Web Capacity Analysis (WebCAT) tool provides Web server performance analysis; the tool can also assess Internet Serve
In
Creating an automated test suite to measure performance is time-consuming and labor-intensive.
Therefore, it is important to define concrete performance goals. Without defined performance goa or requirements, testers must guess, without a cle
m
The performance tests should not be used to find functionality-type bugs. Design the performa test suite to measure response times and not to identify bugs in the product. Design the build verification test (BVT) suite to ensure that no new bugs are injecte
p
The performance tests should be modified consistently. Significant changes to the performance test suite skew or make obsolete all previous data. Therefore, keep the performance test suite fairly static throughout the product development cycle. If the design or requ
y
Strive to achieve the majority of the performance goals early in the product development cycle because:
Most performance issues require architectural change.
Performance is known to degrade slightly during the stabilization phase of the development cycle.
Achieving performance goals early also helps to ensure that the ship date is met because a product rarely ships if it does not meet performance goals. You should reuse automated performance tests Automated performance tests can often be reused in many other automated test suites. For example, incorporate the performance test suite into the stress test suite to validate stress scenarios and to identify potential performance issues under different stress conditions.
Tests are capturing secondary metrics when the instrumented tests have nothing to do with measuring clear and established performance goals. Although secondary metrics look good on wall charts and in reports, if the data is not going to be used in a meaningful way to make
improvements in the engineering cycle, it is probably wasted data. Ensure that you know what you are measuring and why.
Performance Testing Tools
Testing for most applications will be automated. Tools used for testing would be the tool pecified in the requirement specification. The tools used for performance testing are Loadrunner 6.5 and Webload 4.5x.
LoadRunner 6.5
LoadRunner is Mercury Interactive’s tool for testing the performance of client/server systems.
LoadRunner enables you to test your system under controlled and peak load conditions. To generate load, LoadRunner runs thousands of Virtual Users that are distributed over a network.
Using a minimum of hardware resources, these Virtual users provide consistent. Repeatable and measurable load to execute your client/server system just as real users would. Load Runner’s in depth reports and graphs provide the information that you need to evaluate the performance of your client/server system.
WebLoad 4.5
Webload is a testing tool for testing the scalability, functionality and performance of Web-based applications – both Internet and Intranet. It can measure the performance of your application under any load conditions. Use WebLoad to test how well your web site will perform under realworld conditions by combining performance, load and functional tests or by running them individually.
Webload supports HTTP1.0 and 1.1, including cookies, proxies, SSL, TSL, client certificates, authentifications, persistent connections and chunked transfer coding. Webload generates load by creating virtual clients that emulate network traffic. You create test scripts (called agendas) using Java Scripts that instruct those virtual clients about what to do. When Webload runs the test, it gathers results at a per-client, per-transaction and per-instance level from the computers that are generating the load. Webload can also gather information server’s performance monitor. You can watch the results as they occur- Webload displays them in graphs and tables in real-time and you can save and export the results when the test is finished.
Performance Testing Tools - summary and comparison
Architecture Benchmarking
Hardware Benchmarking - Hardware benchmarking is performed to size the application with the planned Hardware platform. It is significantly different from capacity planning exercise in that it is done after development and before deployment
Software Benchmarking - Defining the right placement and composition of software instances can help in vertical scalability of the system without addition of hardware resources. This is achieved through software benchmark test.
General Tests
What follows is a list of tests adaptable to assess the performance of most systems. The methodologies below are generic, allowing one to use a wide range of tools to conduct the assessments.
Methodology Definitions
Result: provide information about what the test will accomplish.
Purpose: explains the value and focus of the test, along with some simple background information that might be helpful during testing.
Constraints: details any constraints and values that should not be exceeded during testing.
Time estimate: a rough estimate of the amount of time that the test may take to complete.
Type of workload: in order to properly achieve the goals of the test, each test requires a certain type of workload. This methodology specification provides information on the appropriate script of pages or transactions for the user.
Methodology: a list of suggested steps to take in order to assess the system under test.
What to look for: contains information on behaviors, issues and errors to pay attention to during and after the test.
Performance Metrics
The Common Metrics selected /used during the performance testing is as below
Response time
Turnaround time = the time between the submission of a batch job and the completion of its output.
Stretch Factor: The ratio of the response time with single user to that of concurrent users.
Throughput: Rate (requests per unit of time) Examples:
Jobs per second
Requests per second
Millions of Instructions Per Second (MIPS)
Millions of Floating Point Operations Per Second (MFLOPS)
Packets Per Second (PPS)
Bits per second (bps)
Transactions Per Second (TPS)
Capacity:
Nominal Capacity: Maximum achievable throughput under ideal workload conditions.
E.g., bandwidth in bits per second. The response time at maximum throughput is too high.
Usable capacity: Maximum throughput achievable without exceeding a pre-specified response-time limit
Efficiency: Ratio usable capacity to nominal capacity. Or, the ratio of the performance of an nprocessor system to that of a one-processor system is its efficiency.
Utilization: The fraction of time the resource is busy servicing requests.
Average Fraction used for memory.
As tests are executed, metrics such as response times for transactions, HTTP requests per second, throughput etc., should be collected. It is also important to monitor and collect the statistics such as CPU utilization, memory, disk space and network usage on individual web, application and database servers and make sure those numbers recede as load decreases.
Cognizant has built custom monitoring tools to collect the statistics. Third party monitoring tools are also used based on the requirement.
Client Side Statistics
Running Vusers
Hits per Second
Throughput
HTTP Status Code
HTTP responses per Second
Pages downloaded per Second
Transaction response time
Page Component breakdown time
Page Download time
Component size Analysis
Error Statistics
Errors per Second
Total Successful/Failed Transactions
Server Side Statistics
System Resources - Processor Utilization, Memory and Disk Space
Web Server Resources–Threads, Cache Hit Ratio
Application Server Resources–Heap size, JDBC Connection Pool
Database Server Resources–Wait Events, SQL Queries
Transaction Profiling
Code Block Analysis
Network Statistics
Bandwidth Utilization
Network delay time
Network Segment delay time
Conclusion
Performance testing is an independent discipline and involves all the phases as the mainstream testing lifecycle i.e strategy, plan, design, execution, analysis and reporting. Without the rigor described in this paper, executing performance testing does not yield anything more than finding more defects in the system. However, if executed systematically with appropriate planning, performance testing can unearth issues that otherwise cannot be done through mainstream testing. It is very typical of the project manager to be overtaken by time and resource pressures