THE LAST PRESENTATION ON TEST ESTIMATION YOU WILL EVER NEED TO ATTEND

BIO

PRESENTATION

International Conference On Software Testing Analysis and Review

May 15-19, 2006 Orlando, Florida USA

F1

Friday, May 19, 2006 10:00AM

T

L

P

T

E

Y

W

E

N

A

Geoff Horne Geoff Horne Testing

Geoff Horne

Geoff Horne is based in New Zealand and has founded and run two testing

consultancy companies which grew to enjoy an international clientele. He has over 28 years experience in IT including software development, sales and marketing and IT and project management. In 1994, almost by accident, he found himself involved in testing a complex fault management system, which led to further testing and QA assignments covering a wide range of applications and tools. Geoff’s companies were subsequently founded to bring a full range of testing consultancy services to the IT industry. Latterly, Geoff has focused on a few select clients running complex test projects in a programme test management capacity.

Geoff has written a variety of white papers on the subject of software testing and has been a regular speaker at the Star testing conferences. He is married with four children and in his spare time (which there is not a lot of) enjoys writing and recording original contemporary Christian music.

The Last Presentation on Test Estimation

You Will Ever Need to Attend

(I think!)

Geoff Horne - Principal

Why estimate testing?

Why is estimating so hard?

So many variables…

• Product - complexity, availability, stability • Test environment

-

• Issues - volume, severities, time to fix, regression • Personnel - suitability, experience

• Test stages - analysis, design, development, execution • Evolving project – re/descoping, priority shifts

And what’s the usual response?

Something like, and I quote...

• “We don’t have that luxury”

• “We don’t have time for best practice”

• “We’re building a Holden not a Rolls Royce” • “I think you’re over-complicating the exercise” • “We brought you in to save time”

• “Ha ha ha” (sic: maniacal laughter)

Test estimation

Two types:

• “Guesstimation” • “Testimation”

Examples of “Guesstimation”

Based on previous exercises and prior knowledge: • Percentages - of development efforts

• Comparative size

-

• Application metrics - no. of screens, functions etc. • And the one we all love…..

you’ve got

n

n

“Guesstimation:”

Good starting point however…

“Guesstimation:”

Only a starting point!

• Educated “finger in the wind”

• Beware of getting locked into a “guesstimation” • You don’t know what you don’t know

• Watch out for the monkey on your back!

“Guesstimation” to “Testimation”

Applying some science

• Start with assumptions • Add what you do know • Allow for what you don’t

• Refine as “unknowns” become “knowns” • Develop “testimation” models

“Guesstimation” to “Testimation”

Key: Gaining buy-in to the process, from: • Project managers

• Test team

• Other project team leaders • Sponsors

• Developers • Business

“Testimation” models:

Simple model

• Based on averages

• Assumes some sort of previous metrics to work from • Provides “lump sum” result

• Good for structured testing initiatives

“Testimation” models:

Simple model

“Testimation” models:

Simple model OK however…

• Usually not 8 productive work hours in a day • Test effort cannot always be divided evenly • Test team productivity not always linear

• No allowance for complexity or priority • Tester productivity differences

• Time estimates are flat

“Testimation” models:

Simple model OK however…

• Time models need to allow for: • Development

• Requirements analysis • Test case design

• Test script development

• Reviews and other overheads

“Testimation” models:

Simple model OK however…

• Time models need to allow for: • Execution

• Pretest – test data setup, prerequisites etc. • Execution of script

• Retesting

• Wait time for issue repair

• Documentation and other overheads

“Testimation” models:

More comprehensive model… • Still based on averages

• Still requires some sort of metrics to start, however… • Allows for: different levels of tester

multiple complexities multiple time models

productivity diminishing returns variables work hours

“Testimation” models:

More comprehensive model…

Test Lifecycle:

T est P lan R eq u irem en ts

T est R eq u irem en ts (re vie w ) T est C a se s (re vie w ) T est S c rip ts (re vie w ) In c id en t T est Lo g s (re vie w ) R ep o rts A N A LY S E D E S IG N D E V E LO P E X E C U T E S IG N O F F R E P A IR A N A L Y S E R E V I E W M E A U S U R E R E P O R T D E V E L O P E X E C U T E R E V I E W R E V I S E www.ghtest.com

Test estimation; “Testimation”

Stage View Method Activity

40,000’ 20,000’ 10,000’ 5,000’

sea level

Rough estimate based on % of development effort Better estimate based on expected no. of test cases and modelling Estimate Estimate Monitor and manage Monitor and manage Viscosity

Better estimate still, based on

test case design and schedules Estimate

“Guesstimation” vs “Testimation”

Spot the difference:

• “Guesstimation” is a good starting point however… • Not what you want to get locked in to

• “Testimation” provides quantifiable estimates • Can be used to create buy-in

• Refining the “testimation” provides basis for test plan

“Testimation”

Feedback loop:

• Capture test metrics as plan is executed • Regularly measure against estimates

• Full analysis at test completion • Use to measure test efficiency • Document for future projects • Learn!

Summary

• Estimating test effort is an evolutionary process • Modelling provides tangible estimates

• Gaining buy-in allows estimate to be refined

• Reviewing estimates vs actuals measures accuracy • Feeding back lessons learned improves process