A Taming of the
Tornado
U i
Using
Outline
• Integration Hell
• Integration Hell
• Software Development Nirvana
• What is Continuous Integration?
• Implementing Continuous Integration
p
g
g
• CI Toolbox
Success Criteria
• Success Criteria
• Good Resources
• Questions, comments, stories,…
• Contact info
Integration Hell
Evolution of Continuous Integration
What is Continuous Integration?
“The macro process of object-oriented
d
l
t i
f "
ti
i t
ti
"
development is one of "continuous integration."
... At regular intervals, the process of "continuous
integration" yields executable releases that grow
in functionality at every release. ... It is through
these milestones that management can measure
progress and quality, and hence anticipate,
p
g
q
y,
p
,
identify, and then actively attach risks on an
ongoing basis.”
-- Grady Booch, Object-Oriented Analysis and
Grady Booch, Object Oriented Analysis and
Design with Applications, 2nd ed, 1993
What is Continuous Integration?
“A common practice at Microsoft and some other
shrink-wrap software companies is the ‘daily build
and smoke test’ process. Every file is compiled,
p
y
p
,
linked, and combined into an executable program
every day, and the program is then put through a
‘smoke test’ a relatively simple check to see
smoke test , a relatively simple check to see
whether the product ‘smokes’ when it runs.”
-- Steve McConnell, Daily Build and Smoke Test,
IEEE Software July 1996
IEEE Software, July 1996
Microsoft called daily builds the
“Sync Pulse” of a project
-- Microsoft Secrets, 1995
7
Microsoft Secrets, 1995
What is Continuous Integration?
“Continuous Integration is a software
Continuous Integration is a software
development practice where members of a team
integrate their work frequently, usually each
i t
t
t l
t d il
l
di
t
person integrates at least daily - leading to
multiple integrations per day. Each integration is
verified by an automated build (including test) to
detect integration errors as quickly as possible.
Many teams find that this approach leads to
significantly reduced integration problems and
g
y
g
p
allows a team to develop cohesive software more
rapidly.”
What is Continuous Integration?
Continuous Integration and the Möbius Strip
Continuous Integration and the Möbius Strip
Mobius Strip
¨
never-ending, one-sided shape
¨
continuous
A surface with only one side and only one boundary
p
g,
p
su ace t o y o e s de a d o y o e bou da y component.
¨ mathematical property of being non-orientable
¨ a ruled surface
¨ discovered independently by the German
mathematicians August Ferdinand Möbius and Johann Benedict Listing in 1858
What is Continuous Integration?
P
id
f l
th d f fi di
d f
t
Continuous integration
• Provides a powerful method of finding defects
early enough to fix them easily
• Continuous integration
automatically:
Integrating code frequently
–
Integrating code frequently
(2 hours recommended)
–
Monitors source code
–
Compiles after every
change
–
Unit tests compiled code
–
Notifies developers of
problems as encountered
problems as encountered
Return on Investment
Cost
=
Complexity
Each connection represents potential for defect
11
What is Continuous Integration?
• Many statistics show
projects fail more
often than they succeed
• All studies agree:
– Projects very often
Projects very often
fail
fail
.
– They are
late
.
They are
over budget
– They are
over-budget
.
– They
fail to deliver
the product they were
designed to produce
designed to produce.
– Some projects
don't produce
any product at
all
Why is it Important? -- History Lessons
Toyota - Recalled 160,000 Prius (new hybrid) for software bug
Mars Climate Orbiter - Destroyed due to navigation error
Th 25 D li d l th l di ti d 5 ti t di d
Therac-25 - Delivered lethal radiation doses – 5 patients died
National Cancer Institute - In Panama City, therapy planning software miscalculates radiation dosage - 28 overdoses, 8 died
UK Air Traffic Control Upgrade problems caused thousands
UK Air Traffic Control - Upgrade problems caused thousands of passenger delays and plane groundings across the country
Denver International Airport - Baggage handling system problems Cost estimate $1 1 million lost per day
problems. Cost estimate - $1.1 million lost per day
FAA - Advanced Automation System overran planned budget by ~ $3 billion
IRS - $8 billion software modernization program cost U S
IRS $8 billion software modernization program cost U.S. taxpayers $50 billion per year in lost revenue
California motor vehicle - Registration system cancelled after topping $44 million in a long series of overruns
topping $44 million in a long series of overruns
B-2 bomber - Wouldn’t fly on its maiden flight because of a software problem
Ariane 5 rocket - Software error caused blow up on maiden
13
p launch
Seattle, Computer-controlled ferries problems caused dozens of dozen dock crashes with damages > $7M
Why is it Important? -- The Future
True Global Civilization - complete with a unified language, culture and planet-wide technological prowess. A global
it bl f t i i d t lli it l t community capable of sustaining and controlling its planet.
Control of Global Warming - Human-caused global warming is visible in the air, water and melting ice, and is destined to get much worse in the future
much worse in the future.
Alternate Energies and Fuel - Managing the planet's
resources effectively using advances in computing, engineering and other fields Geothermal power showing promise
and other fields. Geothermal power showing promise.
Cloak of Partial Invisibility - a cloaking device that reroutes certain wavelengths of light, forcing them around objects. the cloaked object would appear nearly invisible in microwave light cloaked object would appear nearly invisible in microwave light.
Molecular Memory - ultradense integrated memory circuit from nanowires and molecules developed. Problems are still being worked out of device; expect these as common by 2020.
worked out of device; expect these as common by 2020.
Microrobot - group of international scientists developing
microrobot able to swim through arteries and digestive system.
Exploding Robots - fleet of exploding probes could prepare
14
p g p g p p p
Reality: Projects Fail
-- especially IT projects
especially IT projects
Battle cry is loud and clear
Battle cry is loud and clear
especially IT projects
especially IT projects
Battle cry is loud and clear
Battle cry is loud and clear
“Projects are failing more often than they
“Projects are failing more often than they
are successful Something must be done!”
are successful Something must be done!”
are successful. Something must be done!
are successful. Something must be done!
But what ???
But what ???
15
One Good Answer: Continuous Integration
One Good Answer: Continuous Integration
Not Just For Agile
• Popularized in the XP World
Popularized in the XP World
• Any Software Development Lifecycle
benefits from Continuous Integration
g
– Find problems early
– Correct problems while fresh and less
Scrum
complex
– Promote communications/collaboration
Spiral Project Life Cycle Rational Unified Process (RUP)
Definition Review Plan C it t Risk Analysis Review
1. Define Approach 2. Analyze & Avert Risk
• Identify Risks • Analyze Risks • Evaluate Risks • Develop/Update Estimations • Define Cycle Requirements Analysis Overlapping Waterfall Project Life Cycle
Requirements Analysis Preliminary Design Detailed Design Evolutionary Project Life Cycle Build 1
..
.
Rational Unified Process (RUP)
Requirements Serial Incremental or Iterative Build
Development Plan Commitment Risk Walkthrough Management Review Commitment • Monitor & Review • Develop & Verify Product • Plan Risk Mitigation /Update Cycle Definition • Update Plan • Update Procedures • Review Progress • Baseline Product Preliminary Design Detailed Design Code & g Code & Unit Test System Integration & Test Formal Test Requirements Analysis Preliminary Design Detailed Design Build n Requirements Analysis Preliminary Design
Serial, Incremental, or Iterative Build Project Life Cycle
Build n Build 1 Requirements Analysis Preliminary Design Detailed Design
Waterfall Project Life Cycle
16 Product Review Code & Unit Test System Integration Design Code & Unit Test
..
.
Final System Integration & Test Build nCode & Unit Test
Getting from Here to There
• Are you in the tornado?
• Are you in the tornado?
• How can you start the path to Nirvana?
• It is a matter of continuous incremental
i
!
improvements!
• A CI platform becomes a place to “bolt on”
repeatability
17
The CI Process (
continued
)
Getting There
• Backlog:
• Backlog:
– Establish source repository and standards
Identify and configure the Continuous Integration
– Identify and configure the Continuous Integration
Server (the “build machine”)
– Write a continuous integration script
– Write a continuous integration script
– Automate the build process and install into CI tool
Automate the deployment process and install into CI
– Automate the deployment process and install into CI
tool
– Automate the Unit Test Suite execution and install into
Automate the Unit Test Suite execution and install into
CI tool
Source Repository and Standards
• We must be able to revert to previous versions
• We must be able to revert to previous versions
of any artifact. This is critical in being able to
quickly recover from a CI build failure
quickly recover from a CI build failure.
Rule Whyy
Version everything needed for the build. “Everything” goes into the repository. Everything includes: test scripts,
properties files, database schema, install scripts, SQL scripts, third party libraries, documentation …
Don’t version things that get built. Nothing that gets built through a tool should
be stored in the repository, only those things that are needed to perform a build
21
The CI Server
• This is the “brains” of CI
This is the brains of CI
• Install and configure a CI tool
Candidates:
– Candidates:
• CruiseControl from ThoughtWorks (www.thoughtworks.com)
• Anthill Professional from Urbancode (
Anthill Professional from Urbancode (
www urbancode com
www.urbancode.com
)
)
Repository
“Clean Machine” Deployment
• “Clean Machine” Automation
• Clean Machine Automation
• This should include only those tools that are too
difficult to script the installation This would include
difficult to script the installation. This would include
the operating system, compilers, JVM, .Net run-time,
database servers …
• Deb’s picture here
Deb s picture here
RepositoryCI S D l t S
23
Communications Portal (email, web interface,…)
Script the CI Process
•
See Process Diagram from previous slide
•
See Process Diagram from previous slide
• Timely Feedback
T
i
ll
l
th
15
i
t
If
id
“ t
d
• Typically, less than 15 minutes. If more, consider a “staged
build” approach
Rule
Why
Maintain “clean machine” build
Scripts are written and maintained in
scripts
such a way that someone should be
able to walk up to a “clean
machine”, execute the scripts, and
,
p ,
fully build the system.
Automated Builds
•
Automate the build from source to executable
•
Common tools such as
make
and
ant
can be used
•
Can identify other tools/best practices that should be included in
automated build
– Candidates:
• Build documentation (e.g. JavaDocs)
• Automated code standards (e.g. FxCop, CheckStyle, and Jalopy)
• Automated test coverage checkers (Did the developer embed automated • Automated test coverage checkers (Did the developer embed automated
testing?)
Rule
Why
The build process must be reusable
across Dev, QA, and Production
This creates repeatability as moving
from one software development stage to
the next
the next.
The output of the build should be
placed in a location where anyone
Builds need to be accessible to anyone
on the team
25
Automated Deployments
• Automate the Deployment to the “Clean Machine”
• Typically done with the OS and DB scripting languages
R l
Wh
Rule
Why
The deployment script should allow
different deployment targets
This creates repeatability as moving
from one software development stage to
different deployment targets.
from one software development stage to
the next.
After deployment, system should
This will allow the production support
execute a “Smoke Test” to ensure
proper configuration
team, and others, to use the script for
deployment and have some validation of
proper configuration without running
Automated Unit Tests
• Automate the execution of a Test Suite of Unit Test Cases
Automate the execution of a Test Suite of Unit Test Cases
• Choose a testing tool for each language
– Candidates: Java - Junit, .Net - Nunit, …
,
,
• The entire suite of tests should be able to be executed
from a single command
• Establish automated testing standard
Rule
Why
Rule
Why
Always add tests to code when a code
module is created or modified
This will maintain and/or increase test
coverage of code base.
f
g
f
Synchronize with repository before
checking in.
Always synchronize your working copy
with changes in the repository (mainline
b
h) b f
h ki
d i
27
Other Standards
Rule Why
Only check it in if it works. The mainline and branch must be kept in working
condition so that others can safely check out working copies
working copies.
Run the scripts before you check in. Always run the automated build/deploy and test
script in your local working copy and p y g py
development environment before checking code in. This approximates the CI process as closely as possible and will avoid those pesky “failure” emails.
Everyone commits every day. By doing this frequently, developers quickly find
out if there's a conflict between two f f developers. The key to fixing problems quickly is finding them quickly.
If h CI b ild b k fi i i h Th h l i f ki i h CI i h '
Success Criteria
• Be prepared to invest some time and effort in the
Be prepared to invest some time and effort in the
beginning
– Saves time later
Builds quality into product
– Builds quality into product
• Ensure you are working with the “latest” source code
• Never commit “broken” code
• Always do a “clean build”
• Never use CI as a “shaming event”
Di
ib
f il
f
db
k
i kl
ibl
• Distribute failure feedback as quickly as possible to team
• Build/Compile/Test cycle timing should be reasonable
– Break build up into logical chunks
Break build up into logical chunks
– Add more machines
– Minimize testing if
Make the CI process a team decision
29
• Make the CI process a team decision
Continuous Integration Toolbox
• Tool Considerations
F
ti
lit
– Functionality
– Compatibility
– Reliability
– Longevity
Longevity
– Usability
Continuous Integration Toolbox
T
i
l C
ti
I t
ti
T
lb
• Typical Continuous Integration Toolbox
– Continuous Integration System – provides
t
t ti ti
tifi
ti
t
t f
i h
reports, statistics, notifications, etc. to furnish
the framework of the CI build process
B ild T
l
i ti
t
ti
d
– Build Tools – scripting or automating code
compilation to convert software to run on a
computer
computer
– Repository – where software components are
stored
stored
– Automated Testing Tools – automating the unit
or developer level testing
31
Continuous Integration Toolbox
Continuous Integration
System Build Tools Repository
Automated Testing Tools
CruiseControl Beetlejuice Shell/command SubVersion NUnit ( NET) CruiseControl Hudson LuntBuild Beetlejuice Gump Sin Parabuild Shell/command script Ant SubVersion CVS Visual NUnit (.NET) JUnit (Java) SUnit (Smalltalk) Continuum Draco.NET CI Factory Parabuild Pulse TeamCity NAnt Maven Groovy Sourcecode PVCS MKS (Smalltalk) PYUnit (Python) y Drumbeat CI Tinderbox BuildBot Bamboo y MSBuild Make Visual Studio Dimensions Vault Surround CPPUnit (C++) UnitTest Dogtail (GUI) BuildBot Anthill
Visual Studio Surround Mercurial CMSynergy
g ( ) HttpUnit (Web) Parasoft
