ISTQB Expert level Improving the testing process

First practical experience

ISTQB Expert level

Improving the testing process

www.ip-consult.be Version 1.1 Isabelle Robrechts ●Training content

Agenda

● Practical experience ● Conclusion

ISTQB Structure

IP-Consult bvba 3

Bron : www.istqb.org

Goal ISTQB Expert ITP

• Coaching/Leading test process improvement tracks in own organisation

tracks in own organisation

• Goal part 1 : Assessing the test processes • Improvement models

• Linking and situate

• Goal part 2 : Improving the test processes • Change Management

IDEAL

Part 1: Assessing

© 2014 Improve Quality Services B.V. 5

Part 2: Implementing

Approaches

Overview Improvement Approaches

Model based IT/SW Testing Analytical GQM MeasuresMetrics Indicators Hybrid / Mixed Analytical & Models Analytical & Content Other People Skills Standards & Regulations Infrastructure & Tools CMMI IEEE15504 Process models TMMi TPI Next Content models CTP STEP (TMap) Causal Analysis

Infrastructure & Tools Evaluations

● Training content

Agenda

● Training content ●Practical experience ● Conclusion 7 ● Overview Environment

Practical experience:

− Environment − Business goals ● Retrospective − Starting point

− Transfer to improvement: Causal analysis ?

¾exercise

● Improvement proposition

− Short term − Long term

Environment

•Financial institution

•SDLC : Combi V-model / ‘Agile’ (inhouse invention) •SDLC : Combi V-model / Agile (inhouse invention) •IS : tester is not a profession but a role (<-> Bsn)

9

● To go quickly to the market with new products

− Risk-based testing !

Business goals

Risk-based testing ! − Regression testing !

● Good quality of the requested products

− Fit for purpose − Passive testing !g

Retrospective

• Assignment 2009 : IS

•Set-up :Set-up : • test organization • test methodology • test culture • testing

•Representing the department in an organizational

11

g g

testing competence center

•Representing the IS testing at business meetings •Report to the department management

• just CTAL

● What is present ?

− Test work products ?

Retrospective : where do I start ?

Test work products ?

¾Test Policy ¾Test Strategy ¾(Master)Test Plan ¾Test Cases − Test culture − Test process ? − Test Tooling ¾Test Management ¾Defect Logging

● Knowledge through measuring is power...

Retrospective : where do I start ?

● But what are you going to measure ?

− Inspired by the circumstances ?

− Inspired by others (lecture, management,...) − Inspired by own interestsInspired by own interests

13

● Defects :

− # incidents in production

Retrospective : Data Collect

p

− # scope changes per phase − # defects per phase (incl.priority)

● Test cases :

− # written test cases per phase (incl.priority) − # executed, (in-)correct test cases

● Budget :

− Planned & Actual test budget − Planned & Actual total budget

− Planned & Actual Realization (Analysis & Development) − Planned & Actual repair budget

● Can we obtain information from this data ?

Test Effectiveness

Retrospective : first metrics...

− Test Effectiveness

¾How many defects are found per test case ? ¾How many defects are found per test phase ? HOW EFFECTIVE ARE OUR TESTS?

− Test EfficiencyTest Efficiency

¾What’s the price of a defect found during the ST/SIT phase vs a defect found during the UAT phase ?

15

● How effective are our tests ??

Retrospective : Test Effectiveness

Retrospective : Test Efficiency

17

● Pre-condition : an effective defect and/or incident management process

Transfer to improvement : Causal Analysis ?

management process ● 4 steps :

1. Select the items for causal analysis 2. Collect and organize the information 3. Identify any root causes

3. Identify any root causes 4. Draw conclusions

● Defects/incidents

● Irregularities reported by stakeholders

1. Select the items for Causal Analysis

● Irregularities reported by stakeholders ● Problems found during the analysis ● Aids:

− The pareto principle : 80/20 − Categorization of defects − Analysis of statisticsAnalysis of statistics − Results of retrospectives

19

Analysis of statistics

Reality:

• incidents doubled the last two years

• normal requests (no projects) decreased

Peaks are indicating the close-out of a project.

Analysis of statistics

21

● Let someone take the lead ● Aid while structuring the data :

2. Collect and organize the information

● Aid while structuring the data :

2. Collect and organize the information

Management Infrastructure Planning Minimal budgets for resources Tests are  delayed by strict  authorizations Unavailability of the test environment  is 1 on 5 Unrealistic Project planning # Incidents  doubles in the last  2 years No money for trainings authorizations Monthly refresh of test data Testing is a role, no profession No budget for resources Lack of van PM  capacities No commitment to any test work Missing milestines is easily  Accepted. Bsn is often seen as the enemy by IS 23

Communication Culture Human

No professional IS testers Motivation/work spirit often very low IS : why loose time testing now if it is  thoroughly tested later ? Defects are found much too  late No transparancy in the IS  tests

● Identify the ‘most-likely’ causes ● Work top down through the causes

3. Identificy the root causes

● Work top-down through the causes

● Add (if necessary) extra information to the analysis ● Identify any Tipping Points or Broken windows ● Maintain brainstorm rules

● Are there any conclusions ? ● Be Aware for single events ?

4. Draw conclusions

● Be Aware for single events ?

● Do we find any problems in the test process which can be avoided in the future ?

● Are there any other processes involved ? ● Mutual causes ?

25

− Short term : some quick-wins are defined:

¾More transparancy between IS and Bsn

Improvement proposal

p y

● Visualisation of the businessplan with all milestones ● Weekly reporting on planning at the Division Management ● Definition of quality gates and its controles

● Better communication (1N)

− LongTerm : elaborate an improvement track based on TPI Next, with possiblity to expand to CMMi

Improvement Proposal

, p y p

¾Why TPI Next ? And not TMMi ?

● Test process exists (TMAP)

● Test process has already a certain maturity ● Emotional sensitive to improvement

● Clear visibility of the impact on the business goals ● Link with CMMi exists

¾Fit in the model ...

● How does it look dd 2014Q1 ? ● TPI model in oktober ● Where do we want to go ?

27

Improvement Proposal : Long Term

TPI Next : Test Maturity Matrix (dd 2014Q1)

● TPI Next : Test Maturity Matrix (2016)

Improvement Proposal : Long Term

Maturity Level :

- Controlled on all key areas

- Evolution from Efficient on the business critical key areas

29

● Training content

Agenda

● Practical experience

●Conclusion

31

● Models and causale analyse are complement

Conclusion

− Models for Long Term visions

− Causale analyse for selection and short term activities

● In the training links between techniques are shown ● Assignments connect the theory with own practical

experiences