The Business Case for
Performance Testing
Introducing Joseph Ours
About Joseph
•
Owns development and delivery
of Cohesion's Quality Assurance
and Testing Services
•
More than 15 years of IT
experience spanning several
industries and roles
•
Notable success managing
multi-million dollar initiatives
while introducing innovative
solutions for Fortune 1000
companies, resulting in
improved quality, increased
client satisfaction, cost
avoidance, on-time delivery, and
establishing a true team culture
Contacting Me
@justjoehere
614.423.5253
Why poor performance is worse than a full
blown production outage
1
How better performance drives actual
business revenue
2
How performance affects operating expenses
3
Typical stakeholder expectations of
performance
4
2007, 2009, 2011
•
iPhone Activations
take 3 to 36 hours to
complete
•
Complaints trended
as top topic on
Twitter, made CNN
and other new outlet
top stories
•
AT&T reputation
cemented as one that
cannot handle iPhone
•
Servers overwhelmed
due to concurrent
activation requests
and length of
activation process
2011
•
5 days of network
outages globally
•
$50MM in lost sales
due to outage
•
Free Tech Support
and Free Apps offered
to customers
•
Network
overwhelmed with
processing backlog of
data after momentary
switch failure
Amazon Cloud Service - 2011
•
North America - East
Region unavailable
for over 11 hours
•
Sites like Reddit,
Foursquare, and
NYTimes affected
•
Customers offered
10 days of free credit
•
Traffic incorrectly
switched to backup
network. Data stores
overwhelmed
network with
read/write operations
JP Morgan - 2010
•
874k transactions
held up worth
$132MM
•
Customers charged
late fees that had
to be reversed
•
60 hours to resolve
•
Resulted from code
update
2010, 2011(twice)
90%+ of users
received page
time-outs on home
page
Users could not
access online
accounts
All attributed to
routine code
changes
*Bank of America Site Suffers, Washington Post, 2011Why are Performance Issues Worse
Than Production Outages?
•
Production Outages
▫ Immediately Known
▫ Outages tend to have singular
root cause
▫ Customers know when its
down
▫ Usually fixed within hours
▫ “Break-Fix” is appropriate, it
breaks, it gets fixed
•
Performance Outages
▫ Not always immediately
known
▫ Often gives a false sense of
being available
▫ Multiple causes due to
interdependent impacts
▫ Often takes days or weeks to
resolve
100
ms
Increased Revenue
by 1%
Improve Page Load
By
9%
• Traffic increased when
400ms
• Page time decreased
*Source: Yahoo!Come see
presentation
Come see
presentation
100ms-400ms
Increased Traffic
by
.2-.6%
Improve Page Load
By
*Source: GoogleCome see
presentation
Come see
presentation
Average Impact of Performance
Improvement
Of the 178+ companies
evaluated, those who
improved application
performance saw…
*Application Performance Management: Getting IT on the C-Level's Agenda, 2009, Aberdeen
Revenue Increase 9%
presentation
Come see
Revenue Impact of Poor Performance
1 second delay
11 % fewer views
16 % drop in
satisfaction
A site generating
$100k/day
That incurs a
1 second delay
Will lose
$2.5MM/year
*Application Performance Management: Getting IT on the C-Level's Agenda, 2009, Aberdeen
The bottom line
Come see
presentation
Slow Sites Generate Calls
•
56% of online bank
customers would
move to offline
channels to ask a
general account
question
•
54% of online bank
customers would
move to offline
channels to
complete
transactions
*Application Performance Management: Getting IT on the C-Level's Agenda, 2009, Aberdeen
Slow is a page
taking longer
than 4 seconds to
load
Come see
presentation
Internal and External Customers
58% Respondents said poor
application performance
lowered
employee
satisfaction
47% indicated that they had
decreased responsiveness to the
needs of external customers
60% external users are not
satisfied with the performance
of business critical applications
32% of companies experienced
damage to their brand
reputation as a result of poor
performing applications
Primary Source: Application Performance Management: Getting IT on the C-Level's Agenda, 2009, Aberdeen Alternate Sources: Multiple
39% customers rank speed as more
important than functionality
57% of online consumers will abandon a
site after waiting 3 seconds for a page to
load.
8 out of 10 people will not return to a
site after a disappointing experience
Executive Customers – The “C” Level
60% report inability to identify
performance issues before end
users are impacted
Organizations are planning to
increase the number of business
critical applications by 67% over
the next 12 months
41% of CIO’s have issues with rolling
out new applications without them
being performance tested
31% believe virtualization would
further challenge application
performance management
Primary Source: Application Performance Management: Getting IT on the C-Level's Agenda, 2009, Aberdeen Alternate Sources: Multiple
When you don’t know what your users want - find out what the
public wants
•
External Web Page Load Times: <2 seconds
•
Abandonment: 33% abandon between 1-5 seconds, 56% after 4 seconds
•
100 percent abandon sites near 1 minute
•
Internal performance response times: 5 seconds
•
Web site performance is second only to security in user expectations
•
39% users say speed is more important than functionality
•
Mobile users expect better performance than full sites
•
70% of mobile sites are slower than full sites
•
Baby Boomers are more impatient than Gen-Y
User EXPECTATIONS
Primary Source: Application Performance Management: Getting IT on the C-Level's Agenda, 2009, Aberdeen Alternate Sources: Multiple
User Perceptions of Performance
Recalled (1 week
later) Perceived Load
Time
Immediate Perceived
Load Time
Actual Load Time
• 35% slower
than actual
• 15% slower
than actual
User’s perception of performance degrades with
time
*Source: Psychology of Performance, Stoyan Stefanov
Come see