Performance Testing in Production
Performance Testing in Production
Using the Cloud to your advantage
P t d b 1 Presented by: Rob Holcomb VP Performance Engineering SOASTA Chris Cho
Director, BTO Division Hexaware Technologies
Agenda
Why Performance Matters
Hexaware Introduction
Best Practices for Performance Testing in Production
Cloud Testing with SOASTA CloudTest
Using the Cloud to Test Your Site
Using the Cloud to Test Your Site
More Today Than Ever…… More Today Than Ever……
Recent Headlines
More Today Than Ever……as It Is Measured in Lost Sales More Today Than Ever……as It Is Measured in Lost Sales
Recent Headlines
Recent Reports
Google loses 20% Traffic with .5 second of Latency
USERS
NETWORK PLATFORM
Code USERS NETWORK PLATFORM APPLICATION PERFORMANCE MANAGEMENT Quantity Type Location Browser Servers Settings Load Balancer Firewalls Databases Memory Content Architecture External Com API CDNs Location Traffic Memory Connections
Load Generation – Inside a Test Lab
Load Generation – Inside a Test Lab
Load Generation – Public Cloud
Load Generation – Public Cloud
Load Generation: Private & Public Clouds
Load Generation: Private & Public Clouds
Global Test Cloud
Global Test Cloud
Hong Kong Hong Kong UK UK New York New York San Francisco San Francisco Target Application: Behind a Firewall Managed Service Provider Cloud Platform
The Leading Cloud Service For Performance Testing The Leading Cloud Service For Performance Testing
Net
FIRST Cloud Testing Company (2007)
Net
FIRST Cloud Testing Company (2007)S C oud est g Co pa y ( 00 ) Provisioned Over 200,000 Cloud Servers AWARD Winning Technology:
Top Ten New Cloud Services (IW) Best Cloud Application (IT Mgmt) Visionary Leader MQ (Gartner)
S C oud est g Co pa y ( 00 ) Provisioned Over 200,000 Cloud Servers AWARD Winning Technology:
Top Ten New Cloud Services (IW) Best Cloud Application (IT Mgmt) Visionary Leader MQ (Gartner) Visionary Leader MQ (Gartner)
Only Integrated Cloud Based Test Solution
Visionary Leader MQ (Gartner)
Only Integrated Cloud Based Test Solution
Millions of eFilers each TAX year Several Performance issues in Past Millions of eFilers each TAX year Several Performance issues in Past Several Performance issues in Past Internal Test Lab (4K User Limit)
TEST TOOLS: Silk-Keynote-Wily
CLOUDTEST LIVE PRODUCTION SITE
Tested over (33) Days, (7) Hours Per Day Slowly Ramping up User Traffic
Several Performance issues in Past Internal Test Lab (4K User Limit)
TEST TOOLS: Silk-Keynote-Wily
CLOUDTEST LIVE PRODUCTION SITE
Tested over (33) Days, (7) Hours Per Day Slowly Ramping up User Traffic
Net
Cost: Using the Cloud as the NEW Test Lab is affordable Net
Cost: Using the Cloud as the NEW Test Lab is affordable
Cost: Using the Cloud as the NEW Test Lab is affordable
Agility: Scalable Test Lab at your Fingertips
Scale: Simulate hundreds up to millions of Users
Cost: Using the Cloud as the NEW Test Lab is affordable
Agility: Scalable Test Lab at your Fingertips
Scale: Simulate hundreds up to millions of Users
Hexaware’s Wide Range of Enterprise Services
Travel, Transportation, Hospitality & Logistics
Banking, Financial
Services & Insurance Emerging Verticals Vertical Competency
Human Resources
Hospitality & Logistics Services & Insurance g g
Horizontal Competency
Enterprise Solutions
Application Management Business Process Outsourcing BTO & Testing Solutions
Global Presence
17
Hexaware Quality Management
Strategic APT™Methodology
O A h P l T h l (APT) th d l
End-to-end strategic testing services with:
Our Approach, People, Technology (APT) methodology
honed over hundreds of engagements
Proven Center of Excellence approach for Quality Assurance
Best practice approach to lifecycle QA management
Hexaware Quality Management
Global Presence
End-to-end strategic testing services with:
Worldwide development centers
Flexible options for onsite, near-shore, off-shore,
and outsourcing capabilities
Deep-Testing Competencies
Spanning industries including financial services, insurance,
19
telecommunications, healthcare, travel, transportation, manufacturing, hi-tech, and hospitality
Formal partnerships with HP, IBM, Oracle, SAP, SOASTA
5 Key Stages of a Performance Test
APPROACH Reduce Risks • Reduce Failures • Reduce Costs Define Test Requirements
& Objectives, Build Test Plan, Design Test Steps
Build Script Modules, Begin Data-Creation Activities
Execute Performance Tests; Analyze Results & Perform Root-Cause Defect Analysis Optimize Application Performance, Prioritize Defects PEOPLE • Global Resources • BTO Professionals • Independent QA Team I d t C tifi ti TECHNOLOGY • CloudTest On-Demand • Real-Time Infrastructure Monitoring • CloudTest Conductor Plan & Discovery Design Develop Unit System Strategy, Discovery
& Design Construction Execution Analysis
Optimization & Defect Management
SDLC
Software • Industry Certifications • SDLC QA Methodology &Tools Experience • Cloud and Web-basedApplication Expertise
• CloudTest Conductor • Post-Production Monitoring &
and CloudTest Certification Develop Iteration UAT Train Deploy DevelopmentSo t a e Lifecycle
PPM Design & Implementation Requirements Design & Validation Center of Excellence (CoE) Strategy & Implementation Test Strategy Development & Implementation
Why Performance Testing is Critical
Agile Architecture such as SOA allows performance issues to mask themselves at many levelsy
Additional Hardware & Software without Root-Cause Analysis can compound the problem
The Old Mantra: Reduce Risk, Reduce Defects, Reduce Cost
21 Today’s Paradigm: Increase performance, produce a
positive business outcome
When to Performance Test
Technical Drivers Anytime Change or Release Management is InvokedAnytime Change or Release Management is Invoked Business Drivers
Seasonal Surge
Special Events
Staging and Production Testing
Staging Production
23 Staging
• Attack the “low-hanging fruit” • Optimize application parameter settings • Tune based on initial sizing estimates • Conduct preliminary interface volume testing • If available, use production data source
• 1-1 staging to production extrapolation not always possible • Take into consideration fact that staging may be inside the LAN
Production
• Optimize for throughput • Identify data volume errors • Tune infrastructure components
• Identify capacity limits of production environment
• Verify that boundary systems do not trigger unintended Transact. • Take proper care with setting customer expectations
• Test data usage in production must be tagged or handled appropriately
Production Testing Best Practices
Address the Taboo of Production Testing Bend but don’t breakBend but don t break
Focused, Real-Time Monitoring
Controlled Load Injection
Operational support is key
Understand data and its impact
Data Management
Data Security
Workflow Consequences
Typical Performance Test Types
Performance Test Endurance Test
Time Load Stress Test Time Load Elasticity Test 25 Time Load Time Load
Data and Its Role In Performance Testing
Data Misuse
Using data correctly can add
Data Management Data Security Using data correctly can add realism to your performance testing but can also be disastrous if not well planned out.
Data Management
Data use and consumption play a key role in performance test planning.
When sensitive or production data items are used, special care needs to be taken to minimize risks.
DO:
Create a data management spreadsheet to calculate your data needs that includes information on virtual users, rates/hour, reusable data, single-use data
DO:
Identify potential data risks from data management spreadsheet
Masquerade potentially sensitive information (SS#, credit card numbers, account numbers, names)
DO:
Obtain approvals for data management spreadsheet Obtain approval from Information
Security group for sensitive data use Take into consideration how
Operational Support & Monitoring
Enterprise Monitoring leads to Real-Time Optimization
Business and IT owners are both involved
Open line of communication via War Room and Webinar; all have real-time access to the results
Change Control and Release Management must keep
27 Change Control and Release Management must keep
pace with performance optimization
Deployment Options
On-Demand CloudTest Services
External, web-scale load testing, g
Hexaware develops Cloud Tests with your test plans and involvement
Hexaware consultants deploy/execute on CloudTest Global Test
Platform
Collaborative analysis and tuning using real-time analytics
CloudTest Virtual Appliance
Internal Testing Solution (Build Private Test Clouds)
CloudTest Summary
No hardware or software to buy
Hexaware will provision the hardware from leading cloud IaaS providers p g p
(Amazon EC2, GoGrid, Rackspace, Savvis, Terremark), design and execute tests, and deliver analysis in a fraction of the time and cost traditional performance vendors can deliver
Load is driven from the Web using the Web via Cloud Computing
Real-time analytics allows “on-the-spot” configuration changes to your system while the test is still running—
29
g y y g
true Agile testing
Removal of serious functional flaws related to data volume and synchronization issues not identified by the functional team
