Translating Business Risks into a Risk-Based Test Plan

(1)

www.polteq.com

slide nr. 1

Translating Business Risks

into a

Risk-Based Test Plan

Ruud Teunissen

Polteq IT Services BV – The Netherlands

Vught – April 21

st

2006

(2)

www.polteq.com

slide nr. 3

Operational

Financial

Customer Satisfaction

Regulatory

Security

……

Business Risks

•

Consequences of a failing product / system

-

Loose business (opportunities)

-

Too little, too late!

-

Loose customers

-

Break the law

-

Cost of repair

-

Image loss, bad publicity!

-

……

•

Ask the business

(3)

www.polteq.com

slide nr. 5

Prioritize Business Risk

…

10. ……

Low

3.

Unable to print invoice overview

Low

7.

Interfacing with other systems is impossible

Medium

9.

New user interface is difficult to understand

Medium

8.

Unable to update client information

High

6.

Unable to add new orders

Low

5.

“Idle time” due to low performance

High

4.

Invoices are incorrect

High

2.

Incorrect inventory information leads to “no sale”

Medium

1.

Users are able to update all information

Business

Priority

Business Risk

Divide the system into testable parts

and define priorities

1

2

Sub-Systems,

(User) Functions,

Transactions,

Processes,

……

Complete system

CRM

Invoicing

Inventory

Product / System Part

Low

Medium

High

Product

Priority

(4)

www.polteq.com

slide nr. 7

Quality Characteristics

Business Priority versus Quality Characteristics

Usability

Efficiency

Security

Interoperability

Accuracy

Quality

Characteristic

…

Medium

Low

Medium

Low

Medium

High

Business

Priority

3.

Unable to print invoice overview

9.

New user interface is difficult to

understand

5.

“Idle time” due to low performance

1.

Users are able to update all information

7.

Interfacing with other systems is

impossible

…

8.

Unable to update client information

6.

Unable to add new orders

4.

Invoices are incorrect

2.

Incorrect inventory information leads to

“no sale”

(5)

www.polteq.com

slide nr. 9

Business Priority versus Product Priority

Low

Medium

High

Product

Priority

System

CRM

Invoicing

Inventory

Product /

System Part

Low

3.

……

Accuracy

Low

5.

……

Efficiency

Medium

9.

……

Usability

High

6.

……

Accuracy

Security

Interoperability

Accuracy

Quality

Characteristic

…

Medium

Low

Medium

High

Business

Priority

1.

……

7.

……

… …

8.

……

4.

……

2.

……

Business

Risk

Low

Medium

High

Product

Priority

System

CRM

Invoicing

Inventory

Product /

System Part

Low

……

Efficiency

Medium

……

Usability

Security

Interoperability

Accuracy

Quality

Characteristic

…

Medium

Low

Medium

High

Business

Priority

……

(6)

www.polteq.com

slide nr. 11

Low

Medium

Low

Medium

High

Medium

Business

Priority

Medium

High

Critical

High

Low

Medium

High

Product Priority

Priority

…

Medium

Low

Medium

High

Low

Medium

High

Business

Priority

Low

Medium

High

Product

Priority

System

CRM

Invoicing

Inventory

Product /

System Part

Medium

…

Efficiency

High

…

Usability

Security

Interoperability

Accuracy

Quality

Characteristic

…

Medium

Critical

Priority

…

(7)

www.polteq.com

slide nr. 13

Risk Analysis

Product

Risks

Business

Priority

Probability

Intensity

Chance of

error

(In)direct

loss

Probability

•

Chance of Error

-

Did they perform a Unit and Integration Test?

-

Time pressure! Deadlines…

-

“Fresh” and ambitious, yet inexperienced developers…

-

New tools

-

No or little Requirements

-

Performance was not a design issue!

-

…

•

Intensity

-

The more usage / exposure,

… the higher the probability that it will go wrong!

… the higher the impact

(8)

www.polteq.com

slide nr. 15

High

Critical

Low

Medium

Low

Medium

High

Medium

Priority

High

Medium

Low

Medium

High

Probability

Test Priority

Priority versus Probability

Test Priority

Low

Medium

Low

…

High

Medium

High

Probability

Medium

High

…

Medium

Critical

Priority

System

CRM

Invoicing

Inventory

Product /

System Part

Medium

…

Efficiency

Medium

…

Usability

Security

Interoperability

Accuracy

Quality

Characteristic

…

High

Medium

High

Critical

Test

Priority

…

(9)

www.polteq.com

slide nr. 17

Test Design Techniques

•

coverage

-

formal or informal

-

the principle by which the test cases are derived

•

type of functionality to be tested

•

test level (white-box or black-box)

•

quality characteristic(s) to be tested

•

required test basis

•

required skills

T Ma p - t e c h n i e k Wh i t e - b o x ( WB ) o f B l a c k - b o x ( B B ) , F o r me e l o f I n f o r m e e l T e s t b a s i s P r i n c i p e v a n a f l e i d e n ( m e t d i e p g a n g ) : V e r w e r k i n g s l o g i c a , E q u i v a l e n t i e k l a s s e n , O p e r a t i o n e e l g e b r u i k , C R U D , O v e r i g K w a l . a t t r i b u t e n T o e p a s s i n g s g e b i e d e n A l g o r i t m e t e s t WB , f o r m e e l I n t e r n e s t r u c t u u r , b i j v o o r b e e l d p r o g r a m m a - c o d e o f t e c h n i s c h o n t w e r p V e r w e r k i n g s l o g i c a : d e c i s i o n c o v e r a g e i n c o m b i n a t i e m e t p a t h c o v e r a g e , n a a r k e u z e t e v e r z w a r e n m e t t e s t m a a t 2 o f h o g e r F u n c t i o n a l i t e i t V e r w e r k i n g B e s l i s s i n g s -t a b e l l e n -t e s -t WB é n B B , f o r m e e l B e s l i s s i n g s -t a b e l l e n , z o w e l i n t e r n e s t r u c t u u r a l s ( f u n c t i o n e l e ) s p e c i f i c a t i e s V e r w e r k i n g s l o g i c a : d e c i s i o n c o v e r a g e i n c o m b i n a t i e m e t p a t h c o v e r a g e , n a a r k e u z e t e v e r z w a r e n m e t d e c i s i o n / c o n d i t i o n c o v e r a g e e n / o f m e t e e n h o g e r e t e s t m a a t F u n c t i o n a l i t e i t C o m p l e x e v e r w e r k i n g D a t a f l o w t e s t B B , i n f o r m e e l F u n c t i o n e l e s p e c i f i c a t i e s E q u i v a l e n t i e k l a s s e n F u n c t i o n a l i t e i t C o n t r o l e e r b a a r h e i d V e r w e r k i n g , i n t e g r a t i e t u s s e n f u n c t i e s e n g e g e v e n s E l e m e n t a i r e v e r g e l i j k i n g e n t e s t WB é n B B , f o r m e e l I n t e r n e s t r u c t u u r ( WB ) o f f o r m e l e f u n c t i o n e l e s p e c i f i c a t i e s ( B B ) , b i j v o o r b e e l d p s e u d o - c o d e o f g e s t r u c t u r e e r d N e d e r l a n d s V e r w e r k i n g s l o g i c a : m o d i f i e d d e c i s i o n / c o n d i t i o n c o v e r a g e F u n c t i o n a l i t e i t C o n t r o l e e r b a a r h e i d C o m p l e x e v e r w e r k i n g E r r o r g u e s s i n g B B , i n f o r m e e l A l l e s o o r t e n t e s t b a s i s O v e r i g : o p b a s i s v a n v e r m o e d e n s w a a r d e f o u t e n z i t t e n B e v e i l i g i n g C o n t r o l e e r b a a r h e i d F u n c t i o n a l i t e i t G e b r u i k e r s v r . h e i d , I n p a s b a a r h e i d , P e r f o r m a n c e , Z u i n i g h e i d A l l e G e g e v e n s c y c l u s -t e s -t B B , i n f o r m e e l F u n c t i o n e l e s p e c i f i c a t i e s C R U D : o p b a s i s v a n d e l e v e n s c y c l u s v a n g e g e v e n s F u n c t i o n a l i t e i t I n t e g r a t i e t u s s e n f u n c t i e s e n g e g e v e n s P r o c e s c y c l u s t e s t B B , f o r m e e l A d m i n i s t r a t i e v e O r g a n i s a t i e ( A O ) p r o c e d u r e s V e r w e r k i n g s l o g i c a : d e c i s i o n c o v e r a g e , s t a n d a a r d m e t t e s t m a a t 2 , m a a r n a a r k e u z e i s d e z e t e s t m a a t t e v e r l i c h t e n o f t e v e r z w a r e n B e v e i l i g i n g B r u i k b a a r h e i d G e b r u i k e r s v r . h e i d I n p a s b a a r h e i d I n t e g r a t i e t u s s e n d e a d m i n i s t r a t i e v e o r g a n i s a t i e ( A O ) e n h e t s y s t e e m

DCoT

Data Combination Test

SYN

Syntax Test

ECT

Elementary Comparison Test

RLT

Real Life Test

SEM

Semantic Test

PCT

Process Cycle Test

SUMI

Software Usability Measurement Inventory

PCT and

Usability

Laboratory Test

Use Cases or

PCT

and SUMI

Use Cases or

PCT

and SUMI

Error Guessing

and SUMI

Usability

SEM

user profiles and

overall system

SEM

user profiles

SEM sample

user profiles

Error Guessing

Security

The

thoroughness

of the RLT is variable and will thus be determined by

the rating available as a consequence. and the amount of hours that

comes

Efficiency

SEM and

sample SYN

Sample both

SEM and SYN

Sample SEM

Error guessing

• Screen checks

ECT

DCoT

Error Guessing

Functionality

• Processing

Critical

High

Medium

Low

(10)

www.polteq.com

slide nr. 19

Test Strategy

Medium

…

High

Medium

High

Critical

Test

Priority

System

CRM

Invoicing

Inventory

Product /

System Part

RLT

…

Efficiency

PCT and SUMI

…

Usability

Security

Interoperability

Accuracy

Quality

Characteristic

…

SEM, user profiles

PCT

DCoT

Sample SEM

Error guessing

ECT, DCoT

Sample SEM

Error guessing

ECT

SEM and sample SYN

Test Design

Technique(s)

…

Test Effort Estimation

Size

Strategy

Productivity

Estimation Method

(11)

www.polteq.com

slide nr. 21

Test Effort Estimation

400 hrs

Estimated Effort

…

High

Medium

High

Medium

Critical

Test

Priority

…

∑

S

S3

S2

S1

Size

…

High

Medium

High

Productivity

System

CRM

Invoicing

Inventory

Product /

System Part

…

Efficiency

…

Usability

Security

Interoperability

Accuracy

Quality

Characteristic

…

Effort

Estimate

…

An Iterative Process thus far…

Operational

Financial

Customer Satisfaction

Regulatory

Security

……

(12)

www.polteq.com

slide nr. 23

Example : Test Progress – Test Design

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Inventory

Invoicing

CRM

System

Accuracy

Usability

Efficiency

Interoperability

Security

Prioritize test activities in such

a way that whenever you

have to stop, you have done

the best testing in the time

available

(13)

www.polteq.com

slide nr. 25

Example: Defects - Status

0

20

40

60

80

100

120

New

Open

Rejected Retest

Solved

Total

Low

Medium

High

Acceptance Criteria

25

17

10

7

5

Total

Pending Defects

8

5

3

2

1

Medium

50%

75%

100%

Perc.

Exec.

3

2

1

0

High

14

Total

Low

Medium

High

Critical

Priority

P3

P2

P1

Product /

System Part

5

QC 2

QC 4

QC 3

QC 1

Quality

Characteristic

10

6

4

Low

(14)

www.polteq.com

slide nr. 27

Pending Defects versus Priority

78

0

21

8

2

0

26

Total

Pending Defects

31

…

0

14

4

1

0

12

Medium

32%

…

0%

100%

80%

20%

0%

60%

Perc.

Exec.

11

…

0

3

2

0

4

High

36

Total

…

Medium

Critical

Medium

High

Critical

Priority

System

CRM

Invoicing

Inventory

Product /

System Part

1

Efficiency

0

Usability

Security

Interoperability

Accuracy

Quality

Characteristic

…

0

12

2

10

Low

Example : Test Progress – Test Execution

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

P1

P2

P3

P4

P5

Critical

High

Medium

Low

(15)

www.polteq.com

slide nr. 29

Ruud Teunissen

Polteq IT Services B.V. – The Netherlands

Orlando - May 19

th

2005

Ruud Teunissen