White Paper
Cloud Testing
Production Applications
Executive Summary ... 3
1. Intended Audience ... 4
2. The Past and Present of Performance Engineering ... 4
3. Product and Methodology - Working Together in Harmony ... 5
4. Select Topics From SOASTA’s CloudTest Methodology ... 6
4.1 Security ...7
4.2 Test Data in Production ...10
4.3 Potential Live Customer Impact ...12
5. Customer Case Studies - Testing in Production ... 12
5.1 Security Case Studies ...12
Conclusion ... 13
About SOASTA, Inc. ... 13
SOASTA, Inc.
Table of Contents
CLOUD TESTING PRODUCTION APPLICATIONS WHITE PAPER | 2Online application performance is critical - no one would challenge this statement. Yet the “dirty little secret” in the web world is that applications are rarely performance tested. When performance
for web applications has delayed the introduction of load testing methodologies for production
®
testing in production.
Web sites are the most accessible consumer-facing outlet for most companies.
The brand impact from poor performance or an outage is far reaching and usually
garners visibility at the highest levels. Anyone involved in web site performance
and operations of consumer-facing applications.
Applications are increasingly event driven, needing to serve a larger than
normal customer base for perhaps only a few weeks out of the year. We are
SOASTA, Inc. CLOUD TESTING PRODUCTION APPLICATIONS WHITE PAPER | 4
1. Intended Audience
The most common test tool in use today for web application testing was never designed for this purpose.
3. Product and Methodology - Working Together in Harmony
There are certain pieces of critical functionality that are required in a testing
tool to be successful at production testing. CloudTest is a purpose-built
production performance testing tool. It has the ability to scale up to generate
true concurrent user loads up into the millions of users while still delivering the
analytics in real time.
4. Select Topics From SOASTA’s CloudTest Methodology
The methodology SOASTA has developed is a full lifecycle system. It starts with strategy and planning, and then proceeds into implementation details. The third pillar is execution. Finally, a full iteration of the process ends with measurement of success and changes in the processes to improve the system, ultimately looping back to begin again.
Lower left: Tests are created and controlled via web browser from anywhere.
Center: Load is generated from one or more cloud environments around the world.
directed at the target applica-tion wherever it resides (private datacenter, cloud, etc) over standard HTTP and HTTPS ports just like real users.
4.1 Security
4.1.1 Sensitive Data Applications
CLOUD TESTING PRODUCTION APPLICATIONS WHITE PAPER | 8
SOASTA, Inc.
Sensitive Data Sites
Pre-Release and Early Access Sites
IP Address Restriction
Cookies and Headers
Transaction taggingCloudTest dashboards combine data from the test controller perspective with data from the
Another data stream that might be present during a running cloud test is the monitoring data (CPU, Memory, etc). Whether the data is coming from SOASTA’s monitoring agents themselves or from 3rd party monitoring systems, the data is always relayed through SOASTA’s agent called and then it uses ‘push’ technology to send data to the CloudTest main instance. While this data is not generally considered sensitive, as it is only related to the health of systems under test, it can be encrypted for further security and transferred over HTTPS.
the servers being monitored and the outside world. This creates a secure relay point for the monitoring data to travel through.
leaves the datacenter. Unfortunately, by choosing this option, customers lose the ability to see results.
Test data, in the context of this paper, refers to data that resides in an environment (production, in this case) to enable testing. It can also refer to data that was created in the production environment as a result of testing, such as test orders, test accounts, and things of that nature.
Test data is sometimes used loosely to refer to data collected during a test by the CloudTest load generators, such as result reports, dashboards, monitoring data, etc. In some cases, added security to companies whose performance and benchmarking are critical to their business, such as platform performance information about their software.
CLOUD TESTING PRODUCTION APPLICATIONS WHITE PAPER | 10
SOASTA, Inc.
4.2 Test Data in Production
on the coverage
4.2.1 Using Test Data in Production
User Accounts
query plan to optimize for
database fragmentation
Handling Other Types of Test Data
A common performance problem that databases encounter over time is
fragmentation. If you delete large chunks of data from a database, there will
be gaps in the data and the individual pages that store data will become
non-contiguous. This causes SQL statements to take much longer to process.
Test Data Cleanup
database fragmentation
CLOUD TESTING PRODUCTION APPLICATIONS WHITE PAPER | 12
SOASTA, Inc.
alternate code paths
4.3 Potential Live Customer Impact
Three Requirements for Successful Live Testing
kill switch
5. Customer Case Studies – Testing in Production
5.1 Banking and Financial Data Security
IP address restriction
Conclusion
About SOASTA, Inc.
©2012 SOASTA. All rights reserved. SOASTA,
