Encombrement maximum du logotype depuis le bord inférieur droit de la page
Testing @ Axa
UA Leerstoel - Testing - May 2011
Agenda
AXA in the world, and in Belgium
IBT² @ AXA Belgium
Testing @ AXA Belgium
Testing Approaches
Test Roles
Encombrement maximum du logotype depuis le bord inférieur droit de la page
(logo placé à 2/3X du bord; X = 2
A fresh look at AXA
The AXA Group in key figures
80 million customers
present in 56 countries
91 billion euros in revenues
4 billion euros in underlying earnings
981 billion euros of assets under management
AXA In Belgium
AXA in Belgium
6,256 employees (exclusive distributors included)
more than 3.1 million customers
375 million euros in underlying earnings
4.7 billion euros in insurance turnover
23.4 billion euros of total consolidated balance sheet
for AXA Bank Europe
an activity model integrating bank and insurance
central services supporting all local activities of AXA in
Encombrement maximum du logotype depuis le bord inférieur droit de la page
(logo placé à 2/3X du bord; X = 4
AXA in top 5 In Belgium
Ranking 2008 Group % market in turnover
% market in
non life % market in life
1 Fortis 22.3 17.2 25.0
2 AXA 16.1 21.8 13.2
3 Ethias 12.6 13.3 12.2
4 KBC 12.2 9.4 13.6
5 Dexia 9.5 4.3 12.2
Source: Assuralia 2009: 2008 market shares
AXA is one of the 5 leading insurance
Agenda
AXA In The World, and in Belgium
IBT
²
@ AXA Belgium
Testing @ AXA Belgium
Testing Approaches
Test Roles
Encombrement maximum du logotype depuis le bord inférieur droit de la page
IBT
²
Structure
•
Global Business Roadmap
•
Business Architecture
(incl. Process)•
Change Management
•
Method & Coaching
•
Portfolio Management
•
Project Management (prepare & deliver)
•
Application management
•
Application Architecture
•
Specialized Architecture
•
Solution Architecture
•
Risk & Security
•
Technology oriented
delivery teams
•
Testing
•
Production monitoring
•
QoS improvement
•
IT Infrastructure management
•
IT Services transition
•
IT Security
•
Budget & Costing
•
Sourcing
•
Reporting
•
Capacity management
•
People processes
Key features of AXA Software Engineering
Axa IBT
²
has chosen to work with
Delivery Centers
Industrialized approach to deliver components of high quality.
Members of Delivery Centers are technological specialists (Silverlight, Java, ETL) in charge of implementing the business requirements.
Specialization means more efficiency – Reduced time to market, reduced cost
Delivery Centers Services, Delivery Center Front End, Delivery Center Testing,…
Releases
Most applications and projects depend on each other, changing one application can have impacts on the entire AXA system environment.
Working in releases means to put all the changes together in production, after careful synchronized testing. This increases quality and availability of production.
Off shore partners
Encombrement maximum du logotype depuis le bord inférieur droit de la page
Agenda
AXA In The World, and in Belgium
IBT² @ AXA Belgium
Testing @ AXA Belgium
Testing Approaches
Test Roles
What is Testing?
Delivery Center Testing is an independent Test Team
14 Test Coordinators
35 Test Analysts
40 Off shore testers
4 Test managers
Defines, monitors and controls its own processes
Supported by HP Quality Center as Test Management Tool
Supported by HP Quick Test Pro as Test Automation Tool
Supported by HP Load Runner As Performance Testing Tool
N° of defects found last year – 8.781
Encombrement maximum du logotype depuis le bord inférieur droit de la page
Benefits of IBT
²
organization for testing
Delivery center
-
Test professionals are continuously gaining on-the-job experience in testing-
They are now involved in an education program aimed at improving theirtesting skills.
Off shoring
-
Off shore provides capacity required to execute tests and report defects-
Test execution is made off shore at off shore costs.-
Good test professionals are scare people, therefore a dedicatedinvestment in knowledge acquisition is needed in order to keep their skills and motivation high. This investment would not be possible in an organization
where testing is not made in an independent team
-
Good test professionals are rare and testing is a matter of absorbing peaks of activity in last weeks before Put in Production. Off shoring the testexecution and detailed design, offers the possibility to handle multiple parallel testing peak periods while keeping the costs under control.
Benefits of IBT
²
organization for testing
Release mode
-
Imposes code freeze period 2 weeks before Put in Production in order to execute regression and performance tests. All developments must be finished 2 weeks before Put in Production-
Imposes a scope freeze 12 weeks before Put in Production. Scope of testing activities is known and can be planned in advance.-
Regression tests and performance tests could be forgotten in the rush to production, therefore a freeze period of 2 weeks is welcome. These areimportant tests to execute as a
regression or slow performances in an application could have much more impact on the users than a defect on a new functionality
-
The test team has a few weeks of relative calm to organize itself in order to have a planned and controlled test session. Most often it will eventually turn to an uncontrolled rush to testing, but we want this to happen as late as possible in the project in order to minimize risks of putting defects into production.Encombrement maximum du logotype depuis le bord inférieur droit de la page
Added-value of testing
Why spending money on testing before Put in Production?
Any defect found AFTER Put in Production is found by many customers
-
Bad Imago !
-
Loss of new business !
-
Financial errors !
-
Loss of time by the broker
-
Customer not returning
Multiplied by the number of customers finding the defects at the same time.
Multiplied by the internet-speed !
This cost is most often hidden, because difficult to compute,
but it exists
Benefit of IBT
²
organization for testing
Let’s assume the
(very)
conservative hypothesis that a
defect in production costs to the company 400
€
more than
the test activities would have cost to find it.
8000 defects found per year * 400
€
= 3 200 000
€
saved by testing activities.
Encombrement maximum du logotype depuis le bord inférieur droit de la page
Added-value of a dedicated test team
Decreasing Cost of Testing Improving Quality of Production
Deeper and Broader Test Coverage
Systematic Regression Testing
Improved Reviews (V-model)
Off-Shore component takes detailed test design and test execution in charge
Test automation is made possible
Re-use of test scenarios
Some organization are not working with an independent
tests team
Tests are created and executed by the business analyst, system analyst or
representative of ‘the business’.
There is no opportunity to capitalize on experience and trigger a continuous
improvement loop.
A dedicated test team can dedicate time on continuous
testing improvements
KPI:
N° defect prod/N° defect in testAXA: 4%
KPI:
Euros spent on test activities/ N° ofAgenda
AXA In The World, and in Belgium
IBT² @ AXA Belgium
Testing @ AXA Belgium
Testing Approaches
Test Roles
Encombrement maximum du logotype depuis le bord inférieur droit de la page
2 Different Testing Approaches
Large Projects are tested in a systematic, rigorous way
dictated by the V-model.
-
This increases the probability to put in production on time a high quality
software because such a rigorous approach allows to better control the
schedule and scope uncertainties, the dependencies with other projects/
applications and the test efforts spent.
-
But it has a cost !
Small demands and small projects are tested in a less
rigorous, more agile way.
-
Limited risk level, limited uncertainties
-
No or limited dependencies
-
Minor test effort invested.
Project Testing
Validate that the application delivered meets the
business need
Verify that the design
documents are implemented as specified in the
application
Verify that the critical features of the application are still working fine (Functional + Performance)
Requirement Acceptance Tests
System Tests
Technical Design Integration Tests
Implementation And Unit Tests
Regression Tests
Functional Design
Out of Test Process Scope
Acquire project knowledge and identify incoherencies thanks to
Encombrement maximum du logotype depuis le bord inférieur droit de la page
IDEA CONCEPT INCEPTION ELABORAT* CONSTRUCT* TRANSIT* Warranty
Go “concept ” 0 0 Go “concept ” 0 0 Go “mobilize ” 1 1 Go “mobilize ” 1 1 Go “delivery” 2 2 Go “delivery” 2 2 Go “execute” 3 3 Go “execute” 3 3 Go “production” 4 4 Go “production” 4 4 End “project” 5 5 End “project” 5 5
RUN
OFFER
PROJECT
AM
• Support Solution Design
• Define Test Strategy
• Finalize collaboration agreement with
project. Commit on:
• Scope • Cost • Schedule • Quality Software factory 1101 0011 0001 0110 1000 11
BUILD : Coordinate and deliver
• Testability Reviews
• Technical Design
• Coding
• Integration testing
• Maintenance
TEST : Coordinate and deliver
• System Testing
• Regression and Performance Testing
• Support to User Acceptance Testing
Small Demands Testing
1 Week Cycle Approach
Once per week, new demands are included in the test session.
The demands are tested as soon as possible, with a light formalism
Focus On Regression Tests
During the 2 weeks preceding the Put in Production, the applications are
completely tested for regression, on a pre-production environment containing the
entire release to be Put in Production.
Encombrement maximum du logotype depuis le bord inférieur droit de la page
Agenda
AXA In The World, and in Belgium
IBT² @ AXA Belgium
Testing @ AXA Belgium
Testing Approaches
Test Roles
Test Roles
Test Coordinators
In charge of the daily management of the test sessions : day to day schedule, facilitation, fire-fighting, reporting, defect tracking, administration in Quality Center
This is an organization role, not a « content » role. The goal of the Test coordinator is to « make-it-happen » that requires pro activity and drive.
Test Analysts
In charge of the test coverage The test analyst identifies the tests to be executed and the expected results that will guarantee that the major risks are covered by test scenarios.
This is a « content » tasks, the test analyst has a deep knowledge of the application/project and guarantees the quality of the test coverage.
Off Shore Testers
In charge of the detailed design of the test scenarios identified by the Test Analyst, test data identification and test execution.
This an execution job, that requires the ability to absorbs peaks of activity and good handling of stressful situations
Encombrement maximum du logotype depuis le bord inférieur droit de la page
Agenda
AXA In The World, and in Belgium
IBT² @ AXA Belgium
Testing @ AXA Belgium
Testing Approaches
Test Roles
Challenges Facing a Test Delivery Center
Testing is between Development and Put in Production
Testing is on the Critical path of the project and is the step just before the Put in Production (very) high pressure to shorten/skip the testing
Testing is not perceived as a « productive » task. Testing does not add lines of codes or provides analysis documents that describe a solution to a problem.
(very) high pressure.
Axa Answer: release mode and mandatory freeze periods, off shore testing to add capacity, rigorous reporting based on functional risks taken.
Testing finds defects in the code BUT also in all other
processes
Requirements Engineering, Requirement Management, Configuration Management, … are also generating defects. The earlier in the life cycle a defect is generated and not discovered, the most it costs to correct.
Testing could be perceived as a rude guest that underlines dysfunctioning. The tester
needs good relational and communication skills in order to be perceived as a ally in the quest for good quality in production.
Encombrement maximum du logotype depuis le bord inférieur droit de la page
Challenges Facing a Test Delivery Center
Skills shortage, competence erosion
Test specialist are scare. The developments techniques are speeding up in order
to shorten the time to market and applications are evolving fast in order to meet
market needs
Axa answer: My.Future program (upskills and career moves development
program) for test analysts, investments in knowledge acquisition and off shoring
of test execution
If testing gets a cold, it could be because other
processes have the flu…
Requirements unclear
generates false defects, rework in test design
No design
No testing ! (V-Model )
Uncontrolled changes
uncontrolled rework (budget!), false defects
Late deliveries, uncompleted code baselines,…
Testing is at the end of the project, it can be crippled by all the other preceding
processes if they dysfunction.
Axa Answer: QA team accompanies all the processes in order to make sure they
Encombrement maximum du logotype depuis le bord inférieur droit de la page