The Definition of Metrics for Continuous
Integration in SCRUM
––––––
How Continuous Is Our Continuous
Integration?
Christian Facchi
University of Applied Sciences
I l t dt
Ingolstadt
Jochen Wessel Jochen Wessel
Nokia Siemens Networks SMEF 2010
Outline
SCRUM and Continuous Integration
1
Definition of Metrics for Continuous Integration
2
Implementation
3
4 Case Study
Conclusion & Future Work
A il M if t Agile Manifesto:
Individuals and interactions over processes and tools
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaborationCustomer collaboration over contract negotiationover contract negotiation
Respond to change over following a plan
Î small feedback cycles
Î focus on product /deliveries
H t i b d?
S b i i i l f SCRUM Some basic principles of SCRUM:
continuous feedback loop
continuous feedback loop
every backlog item should produce deliverable code Supported by continuous integration:
complete build and test after every modification (checkin)
if build or test fails problems are solved
Î early detection of integration problems (interfaces)
Î bi b t th d f th j t/i t
R i t f C ti I t ti
Requirements for Continuous Integration:
automated build and test
automated build and test
automated configuration management Available toolsupport (visualisation):
Cruise control
Hudson
Software Development
Project Landscape
Software Development
UMTS/LTE Telecommunication System
WCDMA e cifi c a ti o n U P S W LTE GSM WCDMA e cifi c a ti o n U P S W LTE GSM UMTS / LTE o n W WCDMA SW RA T s p e Applic a CP and U LTE SW GSM SW o n W WCDMA SW RA T s p e Applic a CP and U LTE SW GSM SW n BTS Co mm o A pplicat io n S BTS Site M RFSW Common Application SW Co mm on SW TRSW n BTS Co mm o A pplicat io n S BTS Site M RFSW Common Application SW Co mm on SW TRSW B T S C o mmo n Pl at fo rm SW Mgr FR/AAS BTS
Common Platform Services SW
B T S C o mmo n Pl at fo rm SW Mgr FR/AAS BTS
Common Platform Services SW
Flexi BTS System Module HW
FR/AAS Module HW
PC/
Workstation Flexi BTS System Module HW
FR/AAS Module HW
PC/ Workstation
Wh t i f C ti I t ti Why metrics for Continuous Integration
no objective input from SCRUM teams only feelings
no objective input from SCRUM teams only feelings
prioritisation of impediments among different SCRUMprioritisation of impediments among different SCRUM teams
Î these metrics can be used not only in SCRUM
Î i t t t d b ild
SCRUM and Continuous Integration 1
Definition of Metrics for Continuous Integration 2
Implementation
3
4 Case Study
Conclusion & Future Work
P l
f C
ti
I t
ti
Pulse of Continuous Integration
CI frequency: = number of successful integrations per day CI frequency: = number of successful integrations per day heartbeat of the SCRUM team
heartbeat of the SCRUM team
target is CI frequency ≥ 1* |developers|g q y | p |
only successful integrations are measured (correct delivery of a product); not erroneous checkin (error correction phase)
Q
lit
f C
ti
I t
ti
Quality of Continuous Integration
#CI{OK ERR}{OK,ERR} are defined as number of correct or erroneous checkins
ErrorRatio has been determined for each phase (build,
itt t t t t )
unittest, systemtest, …)
indicator for:
indicator for:
process problems (before checkin a successful test is required)
D
ti
f C
ti
I t
ti
Duration of Continuous Integration
defined as maximal value of an automated tool completion
defined as maximal value of an automated tool completion time per day for a special development phase of a
component
if e.g. the build time is to long the number of integration problems will grow, due to parallel checkins
as target limit 10 minutes
A il billit
f C
ti
I t
ti
Availabillity of Continuos Integration
uptime := time where system in <phase> is up and running
uptime<phase> := time where system in <phase> is up and running
downtime := time where system in <phase> is not running
downtime<phase> := time where system in <phase> is not running
availabillity should be very high (basic requirement for
d l t t k
development tasks
SCRUM and Continuous Integration 1
Definition of Metrics for Continuous Integration
2
Implementation 3
4 Case Study
Conclusion & Future Work
I
l
t ti
Implementation:
fully automated test and build environment (UNIX scripts)
fully automated test and build environment (UNIX scripts)
extension of scripts to write metrics information
extension of scripts to write metrics information
(component name, phase, state, timestamp, …) in logfiles
METRICS < t>< h >< t t ><ti > METRICS:<component><phase><state><time>
state = {started| ok| failed}
state = {started| ok| failed}
PERL scripts to analyze logfilesPERL scripts to analyze logfiles
SCRUM and Continuous Integration 1
Definition of Metrics for Continuous Integration
2
Implementation
3
4 Case Study
Conclusion & Future Work
Case study:
2 SCRUM teams
d l t it
one development site
one team in bugfix phase and one team in development phase Observations:
low pulse of integration: less than 1 per team member and day; reason: already known task split (developer, tester)
constant pulse of integration; no rush hour at the end of a sprint;
Î ok
slow development cycles:
slow development cycles:
long maximal build time (up to 1.5 hours)
long maximal test time (up to 2 hours)long maximal test time (up to 2 hours)
Actions:
buildprocess has been optimized
Î immediately reduced to 50% further improvement
Î immediately reduced to 50%, further improvement scheduled
remove tasksplitt between team members (tester, developer); areas of competence destabilized
Î long term issue (education) Advantages for SCRUM teams:
improved buildtime leads to reduce cycle time and to less integration problems (not reported)
Results of the case study:
proposed metrics are very helpful for SCRUM teams. Despite the fact that metrics are suspected in general Despite the fact that metrics are suspected in general
analyzing metrics needs time and deep insight (wrong y g p g ( g conclusions can be drawn very fast)
objective results to achieve improvements (measureable) less is more (concentrate on some al able metrics)
less is more (concentrate on some valuable metrics)
better visualisation should help SCRUM teams (see burn
better visualisation should help SCRUM teams (see burn down chart)
SCRUM and Continuous Integration 1
Definition of Metrics for Continuous Integration
2
Implementation
3
4 Case Study
Conclusion & Future Work 5
Conclusion
Summary
Summary
metrics in an agile environment are useful
careful selection of metrics
careful interpretation of results
to much metrics leads to control by management (contradiction to SCRUM)
( )
Î metrics should help SCRUM teams, so the teams should define the metrics
best practice: Continuous Integration can be used in every SW development process: Try itp p y
Future Work
Wh t
i
t b d
?
What remains to be done?
li ti t th ( l d i j t th il
• application to other areas (plan driven projects; other agile
methods)
• better visualization (Integration in Continuous Integration
tooling); feedback to SCRUM teams
Thank you for your attention
Questions?
Q
University of Applied Sciences Ingolstadt Hochschule Ingolst adt
Prof. Dr. Christian Facchi
SW Engineering and Distributed Systems Esplanade 10 85049 Ingolstadt Phone: +49-841-9348-365 F +49 841 9348220 Fax: +49-841-9348220 E-mail: christian.facchi@haw-ingolstadt.de www: www.haw-ingolstadt.de/
SCRUM t LTE C Pl SW i N ki Si N t k SCRUM at LTE C-Plane SW in Nokia Siemens Networks
(environment):
development of a part of eNodeB telecommunication SW (LTE C-Plane)
~90 software developers
2 development sites 8 SCRUM t
8 SCRUM teams
embedded in bigger frame program (using waterfall model)
S i l t f SCRUM t LTE C Pl SW i N ki Special setup of SCRUM at LTE C-Plane SW in Nokia
Siemens Networks:
Feature Expert Team:
interface to traditional (waterfall) environment (frame process)
continuity of development
introduction of features to SCRUM teams
introduction of features to SCRUM teams
support of Product Owner to build up backlog
Feature Expert Team has been divided furthermore:
Subject Matter Expertsj p
one central Product Owner and one proxy Product owner per site
