Driving Development for Performance Through Targeted Testing

(1)

1

Driving Development for

Performance Through Targeted

Testing

Thomas Barns

UKCMG Annual Conference 2010

Driving Development for Performance Through Targeted Testing

Agenda

• Goal of performance testing • System overview

• Performance test results • The performance testing cycle

• Release performance testing case study • Benefits

(2)

3

Background

• www.easyJet.com

• Business critical

• Continued growth in business • Continued growth in functionality • Frequent releases

• Cost-efficiency is critical

Goal of Performance Testing

• Delivery of well performing software which will continueto perform to business requirements

• Integrated with development • Integrated with capacity planning

(3)

5

System Architecture

• Three tier architechture

… … … Web Business Database

User Transactions

• Corresponding to web pages or groups of web transactions on easyJet.com

• Each is a well defined unit which can be linked directly to the code that provides it

(4)

7 © Capacitas 2002-2010

System

Database Business Web

• Each transaction traverses one or more of the logical

tiers of the system Hom

e Se arch Purchas e Se le ct Log in

Results of Performance Testing

• What should the results of

performance testing this system look like?

Database Business Web H ome Se arch Purchas e Se le ct Log in

(5)

9

Results of Performance Testing

• Tick in the ‘performance tested’ box

• High management level

Results of Performance Testing

• ‘Performance is good/bad’

(6)

11

Results of Performance Testing

• ‘Web search is slow’

• Current user experience level

Results of Performance Testing

• ‘CPU usage at business tier is high’

(7)

13

Results of Performance Testing

• ‘Searching uses excessive CPU at Business tier’ • Transaction breakdown level

Results of Performance Testing

• ‘GetCheapestFlights’ method uses excessive CPU’ • Targeted transaction level

(8)

15

Timing of Performance Testing

• When should testing of this system’s performance take place? • Stages of testing

• Unit Testing

• User Transaction Testing • System Testing

• Final Regression Testing

The Performance Testing Cycle

Unit Testing

User Transaction Testing System Testing Final Regression Testing

(9)

17

The Performance Testing Cycle

• Run most complete test

possible at current development stage

• Then drill down the stack to find where the issue

occured

• Agile sprint-based development

• Working on a set number of items from a product backlog • Monthly releases

Release Timeline

Development Sprint Regression

Sprint Planning

Risk Assessment/Planning Performance Sprint Report

…

(10)

19

Risk Assessment

• Performance risk of proposed Product Backlog Items assessed based on experience

• This affects

• Development sprint content • Development scheduling • Design decisions

Sprint Planning

…

Test Plan

• Tests based on risk assessment

• Timeline based on development schedule

(11)

21

• Pick out individual services and procedures • Targeted Transaction Level

• Start as soon as possible in development as soon as any component transactions are built or changed

Unit Testing

Sprint Planning

…

Example – Flight Availability Caching

• In December a fix was to be made to change some caching

(12)

23

Example – Flight Availability Caching

0 5 10 15 20 25 GetAvailableFlights() GetCheapestAvailableFlights() M illi se co n d s C P U Se rv ic e T im e

Targeted Tests - Database CPU Impact

November December

Sprint Planning …

User Transaction Testing

• Tests to break down the impact of different user transactions • Transaction Breakdown Level

• Can start to be executed as soon as any user transactions are functional

(13)

25

Example – New Dynamic Picklist

• During the course of a release some new dynamically generated javascript was added to the home page and elsewhere, to populate a dropdown box

Example: New Dynamic Picklist

Home Availability Select Flight

C P U Se rv ic e T im e

Transaction Web CPU Cost

Package 1 Package 2 Package 3 Package 4 Package 5 Package 6 Package 7 Package 8 Package 9 Package 10 Package 11 Package 12 Package 13 Package 14 Package 15 Package 16 Package 17 Package 18 Package 19

(14)

27

Sprint Planning

…

System Testing

• Tests bringing together different user transactions to calculate the overall impact

• Current user experience Level • Platform Level

• Can start to be executed as soon as any user transactions are functional

Example – List All Bookings Page

• A new version of the All Bookings List page was developed for the April release

• This involved moving over from an old 2 tier platform to the new 3 tier platform

• A test was run over the Booking retrieval pages • This involved

• Logging in

• Viewing the latest booking • Viewing a list of all the bookings • Filtering and sorting this list

(15)

29

Example – List All Bookings Page

• Severe .NET Locking on the Business Tier

• High CPU 0 500 1000 1500 2000 2500 3000 3500 4000 4500 10 20 30 40 50 C P U m illis e co n d s Virtual Users Business Tier CPU

Sprint Planning

…

Final Regression Testing

• Tests bringing together the whole system • System Level

(16)

31

Example – New Dynamic Picklist

• After the dynamic javascript was cached, it was only

regenerated every 24 hours

• This meant this was not picked up on standard testing • The full regression test however used

• Multiple languages so the javascript was generated once for each

• IIS recycles to force regeneration of the javascript

Example – New Dynamic Picklist

• A synchronisation issue occasionally caused the javascript

generation to fail

• This failure would then be cached and presented to every user! • And user transaction testing found that also the web CPU usage

increased, as the system was attempting to regenerate the cache on every visit

(17)

33

• Report on the results of the performance testing • The report is not the result!

• Report focussed on what needs to be known • What remains unfixed

• The less there is to report on the better

Sprint Planning

…

Reporting

What does a final result look like?

• Delivery of well performing software which will continueto

perform to business requirements • Inefficient code costs real money

• In a growing business performance over-engineering helps ensure future performance while minimising hardware spend • Also in a public cloud environment the changing performance of

a system can have a direct impact on the budget required to support it

• Performance testing used together with capacity modelling for future predictions

(18)

35

Benefits

• Providing development with the information they need, when they need it

• Putting the extra effort in to drive development for performance can save a lot of time and money later

Questions?

Thomas Barns

Consultant

[email protected]