27th March 2015
Istanbul, Turkey
Performance Testing
Best Practice
•
Leads the Intechnica performance team
•
More years in IT than I care to remember
•
Author of “ The Art of Application Performance Testing”
Your Host..
Ian Molyneaux
27th March 2015
Istanbul, Turkey
Why Performance
Matters
INTERNET CUSTOMERS Mobile Carriers Content Delivery Networks Major ISP Local ISP Third-‐party/ Cloud Services INTERNAL USERS DATA CENTER Storage DB
Servers Servers Web
App Servers Middleware Servers Mainframe Load Balancers Network
System complexity
So what represents good performance?
Unacceptable (10s) > Tolerable (7s) > Good (3s) > Average (6s)> Excellence (Uneconomic) (1s)> Compe33ve advantage27th March 2015
What is performance testing?
•
Testing to ensure that your application
works
under real-world
loads
before
you release it to production.
•
Works?
- staying
available
and
performant
under conditions of
peak load.
•
Performant?
- providing a
good
end user experience.
•
The problem is defining what
good
means. It really comes down to
Good performance definition refined..
•
A well-performing application is one that lets the end user carry out
a given task without undue perceived delay or irritation.
Important..
•
Performance testing is distinct from any other form of software
testing.
27th March 2015
Istanbul, Turkey
Performance Testing
Scope
13
Performance Testing helps identify…
NETWORK ISSUES EUE EUE CONFIG ISSUES CAPACITY CODE QUALITY CONTENTION (No) SQL PERFORMANCE EUE
•
Local users
•
Remote users
•
Mobile users
In reality this is server response time
• Plus the impact of latency
•
Last-mile
•
Service Providers
•
Cloud Providers
•
Content Delivery Networks
•
Affiliates
Understand the technology…
Finding the upper limits of
• Availability• Acceptable response time
The ability to remain available and performant
• As a complete integrated deployment
• In the presence of other applications and services
Performance and availability over an extended duratio
n
• No memory leaks
• No configuration hard-stops
• No disk space issues
The What if…?
Useful for optimizing deployment
• Load balancing choices• Web, App and DB configuration settings
Don t forget your DR environment
• Hopefully you have one!• Understand DR performance compared to
production
• Failover testing under load
Try before you buy!
• Performance evaluation of potential suppliers
• Particularly relevant in a SaaS world
Production Benchmarking
• Periodic non-destructive performance evaluation and
benchmarking of live deployments
• Detecting performance and capacity regressions
• Performance impact of configuration changes
• Canary approach
27th March 2015
Istanbul, Turkey
Performance Testing
Foundations
Built on 6 pillars…
•
Business knowledge
•
Technical knowledge
•
People
•
Tooling
•
Environments
•
Process
27th March 2015
Why are we performance testing?…
•
Pre-existing performance issues?
•
Concern about an upcoming peak event?
•
A desire to improve the quality of delivery?
Understand the application…
•
What does the application do?
• Get familiar with the functionality
• Get familiar with the terminology
•
Was it written in-house or is it a package?
•
Is this a new release or a pre-existing application?
•
Understand the size and distribution of end-user
Understand the performance targets…
•
Typically..
• Availability • Concurrency • Response time•
Consider..
• How have they been derived?
• Are they realistic?
Availability…
•
The application
has
to remain available under peak load.
Concurrency…
Two types.
.
•
Application
•
Automation
Impacted by.
.
•
Session persistence
•
Capacity constraints
Response time…
The end user experience.
.
• From the perspective of the end user
Service consumption
• Service SLA’s
Impacted by.
.
• Performance by design – Code quality
• Capacity constraints
Use cases…
Selecting functionality to render as scripts
• Also called user journeys or transactions
• The building blocks of (performance) test scenarios
• Ideally represent high volume, high load activity
Use cases…
Several different types..
• UI driven
• Service driven
• API driven
• Batch driven
Use cases…
Critical to get these correct.
.
• Sub-divide into logical timings
• e.g. Login, Search, Add to Basket, Logout
• Organise into iterative and non-iterative steps
Load model…
Derived from
..
• Business intelligence data
• Application usage profile
• Web server logs
• Performance targets
• Use cases
Together represent performance test scenarios
Load injection profile
27th March 2015
Understand the technology…
•
Application architectu
re
• Tech-stack
• Integrations
• Service consumption, internal and external
•
Hosting infrastructure
• Servers
Determine the technical requirement…
•
Tooling options
• Automation effort and cost
• Instrumentation effort and cost
•
Monitoring
• Tech-stack KPI’s
KPI’s , a many layered approach..
Business Level Response Time Availability Application Level Rendering Time TTFB Application Level Cache Performance HTTP Errors O/S Level CPU Loading Available Memory Application LevelConnection Pool Use Garbage Collection
Application Level
SQL Query Performance
Full Table Scans
I/O Performance Free Disk Space Page File Usage
O/S Level
CPU Loading
Available Memory I/O Performance Free Disk Space Page File Usage
O/S Level
CPU Loading
Available Memory I/O Performance Free Disk Space Page File Usage
APM Level Method Performance Business Transaction APM Level Database Calls Service Calls Service Level Response Time Availability Conversions
The first deliverable… Statement of Work
•
Performance KPI’s
•
Tooling options
•
Load Model
•
Test Plan
•
Time-scales
•
Aim to achieve
consensus
and
agreement
on
27th March 2015
•
The right people are essential to effective performance
testing.
•
Experienced performance testers are a rare commodity to be
nurtured.
•
They should ideally have..
• Background knowledge of application design and deployment.
• An understanding of application architectures.
• Understand the importance of accurate requirement capture.
• The ability to use tooling and implement process.
•
Promote a culture of performance
• Raise awareness of performance in IT
• Consider performance at project start.
• Test in Dev, QA and Production
• Be a performance warrior
27th March 2015
You
cannot
performance test effectively without
automation.
•
S
calability
•
C
onsistency
•
C
orrelation
•
R
epeatability
Tooling components
•
Most automated performance testing tools have the following
components..
• Workbench for script recording
• Load injector to generate application traffic
• Controller to manage and execute tests
• Monitors to capture non-application KPI s
Test Controller Test Scenarios
Web Server Tier
App Server Tier
Database Cluster Load Injectors Metrics Load Load Load Load
Tooling components
• Performance Test Delivery
• JMeter
• Facilita Forecast
• Neotys Neoload
• Borland Silk Performer
• HP LoadRunner
• dynaTrace APMaaS
• Windows O/S and Application
Monitoring
• Windows Performance Monitor
• Solar Winds
• Linux / Unix kernel tools
Tooling choice..
• APM • dynaTrace • Appdynamics • New Relic • Correlsense Sharepath • HP Diagnostics27th March 2015
Most modern performance testing requirements involve
web
•
Good support for Websockets, Async and similar current
tech
However you may also come across the following..
•
Thin-client Citrix, MSTC
•
SAPGUI
•
Oracle eBusiness
•
BMC Remedy IT Service Management
Tech-stack requirements other than web..
• Tooling choice – Non-web support is typically not OpenSource
• Licensing considerations other than performance tooling
• i.e Does the client have enough Citrix licenses?
• Specialist domain knowledge
27th March 2015
•
As close as practical to production in terms of..
• Horizontal and Vertical scale
• Service consumption
• Data volume and quality
• Software Build, Deploy and Configuration
•
Exclusive access during performance test execution
•
Roll-back strategy
Appropriate Instrumentatio
n
• Application KPI s
• Infrastructure KPI s
Appropriate monitorin
g
• Baked into performance test tooling
• Leverage existing tooling investments
27th March 2015
Make sure you follow a consistent approach..
• Scoping
• Design
• Execution and Analysis
• Closure
Scoping..
• NFR Capture
• Six Pillars approach
• Delivers a Statement of Work
Design..
• Based on Statement of Work and Load Model
• Validate test environment
• Create scripts from use cases
• Create performance test scenarios
• Configure environment monitoring
• Ensure you are ready to execute
Execution and Analysis..
• Generally an iterative process
• A lot of “watchful waiting”
• Feedback is critical
• Highly vulnerable to scope creep
Closure
• Creation of engagement report
• Must report against performance targets and goals
• Clarity is key
• Workshop results with the client
• Follow up with quality review
27th March 2015
Built on six pillars…
•
Business knowledge
•
Technical knowledge
•
People
•
Tooling
•
Environments
•
Process
•
Performance testing is a tactical deliverable but should be
part of a strategic process
•
Test early, test often, test effectively
•
Be a performance warrior
27th March 2015
Istanbul, Turkey