Continuous Integration: Put it at the heart of your development

(1)

Continuous Integration:

Put it at the heart of your

development

Susan Duncan

(2)

Program Agenda

§

_{What is CI?}

§

_{What Does It Mean To You?}

§

_{Make it Hudson}

§

_{Evolving Best Practice}

–

_{For Developers}

(3)

What is CI?

§

_{Continuous Integration is a Software Development Process}

– _{Frequent Integration}

● Code Repository ● Automatic Build ● Automated Tests

§

_{Discover errors more quickly thereby reducing fix time}

§

_{Makes the project available to everyone to install}

§

_{Everyone is involved and takes a stake in maintenance}

Code Test Build

(4)

Insert Picture Here

What Does CI Mean to

You?

(5)

Diff’rent Strokes

Who uses CI?

CI Infrastructure The “Management” QA Build Managers Developers Release Managers

(6)

What do They all Want …?

§

_{Yes it needs to be agile}

– _{Iterative, incremental, collaborative, rapid, flexible}

§

_{We seek improved development process through}

– _{Better builds, faster} – _{Better testing}

– _{Better visibility} – _{Better feedback} – _{Better metrics}

§

_{Hudson provides the opportunity to grow beyond builds}

(7)

Introduction to Hudson CI

§

_{Hybrid Eclipse / Java.net project}

– _{Eclipse / MIT licensed}

§

_{Hudson 2.2.1 (production)}

– _{Java.net project accessed through hudson-ci.org}

§

_{Hudson 3.0 (currently at release candidate)}

– _{Eclipse Foundation - top level technology project} – _{www.eclipse.org/hudson}

– _{Plug-ins still through java.net}

(8)

Hudson Has All the Right Bits

§

_{Runs automatically}

– _{SCM: SVN, Git, CVS, Perforce …} – _{Builds: Ant, Maven, scripts …}

– _{Integrate tests: JUnit, Selenium, Abbott…} – _{Deploy: files, App server…}

§

_{Feedback via}

– _{Email, IM, RSS feeds …}

§

_Analysis

– _{reporting, graphs: Clover, Sonar, PMD …}

(9)

Introduction to Hudson

• Easy installation and configuration

• Runs from the WAR out of the box • You only need a JDK installed

• Web-based management interface

• Extendable through plug-ins

• Maintained outside Eclipse • Tiered

(10)

Hudson 3.0 at Eclipse

§

_{IP cleanup}

– _{Core 63MB/109 jars in 2.2.1  26MB / 84 jars in 3.0} – _{Required extensive core re-write in places}

– _{Wholesale technology replacement in some cases e.g.}

§ _{Winstone  Jetty} § _{JFreeChart  Birt}

§

_{Some limited new features}

– _{Switchable update centers}

– _{Improved plug-in management} – _{Bootstrap configuration}

(11)

Gerard Davidson

Long suffering CI user

“It is really easy to set up a sever that is very useful but can go

away at a moments notice – oh, and present a very large hole in

your networks security…”

(12)

Insert Picture Here

Getting There – Through

Best Practice

(13)

Focus First on the Developer

§

_{Time & convenience}

§

_{Make their life better with smaller, faster builds}

– _{Don’t try and build / test the whole world on every check-in. Make it}

modular

– _{Stick to sanity tests on check-in, full re-build and regression nightly or}

manually §

_{Notifications!}

– _{Tell ‘em when it’s failed – email, jabber, RSS, lava lamps} – _{Have the right identity from VCS}

(14)

Cascading Jobs

§

_{Provides job template with}

standardized properties

§

_{Inherited by dependent jobs}

(and the downstream chain)

§

_{Over writable on a per-job}

basis

§

_{Inherited unless overridden}

(15)

Modular Jobs

§

_{Job definitions in Hudson are flexible so take advantage}

– _{Parameterized builds for boilerplate tasks D.R.Y.} – _{Upstream and downstream job dependencies} – _{Build pipeline plug-in}

– _{Join plug-in – great for dependency}

“diamonds”

§ _{E.g. ensure DB is seeded and}

App server configured (in parallel) before triggering deployment

§

_{Simpler jobs are better!}

(16)

Job Management

§

_{Put job intelligence into parameterized scripts}

– _{Put them in source control}

– _{Can be executed externally from Hudson}

– _{Hudson jobs become much simpler (which, as we’ve said, is better!)}

§

_{Use smart job naming that can be}

used with regular expressions to simply / predictably create views

§

_{Just show me the jobs}

I care about!

What’s Important

(17)

Focus on Build Engineering

Don’t wake up one day and suddenly discover how much you

depended on your CI – plan it!

§

_{You need to manage}

– _Metadata – _Resources – _Security

– _{Infrastructure} – _Upgrades

(18)

Managing Your Metadata

§

_{Backup, backup, backup}

– _{Several plug-ins for this: Backup, thinBackup} – _{Move / offsite the backup images}

– _{Like any backup regimen – test that it works (before you need to)}

§

_{Record job history}

– _{JobConfigHistory plugin – useful when several folks are configuring}

§

_{Treat your CI like a production system}

– _{Test first, then promote into “production”}

(19)

Resource Management

§

_History

– _{Restrict job history depth – improves startup, reduces memory}

§

_{Disk space}

– _{Only fingerprint when necessary}

– _{Start with one Maven repository per executor to give the best balance}

between concurrency and space usage. Refine as needed

§

_{Run with sufficient memory allocated to the application server VM}

(20)

Insert Picture Here §

_{Keep no. builds restricted}

§

_{Promote builds you want}

to keep

§

_{Restrict debugging info in}

tests and scripts

§

_{Manage artifacts externally}

– Copy Artifacts plug-in

Lessons Learned

(21)

Security

§

_{Out of the box Hudson is open}

– _{This may be OK but probably isn’t, particularly in larger enterprises and}

servers accessible via the internet

– _{Future direction will be “secure by default”}

§

_{So yes, secure your Hudson, please!}

§

_{Consider the danger areas}

– _{Operating system script execution}

– _{REST API – disable it if you don’t need it} – _{Credentials passed to scripts}

– _{Open HTML fields in Job descriptions etc.} – _{DOS attacks}

(22)

Security

§

_{Out of the box Hudson is open}

– _{This may be OK but probably isn’t, particularly in larger enterprises and}

servers accessible via the internet

– _{Future direction will be “secure by default”}

§

_{So yes, secure your Hudson, please!}

§

_{Consider the danger areas}

– _{Operating system script execution}

– _{REST API – disable it if you don’t need it} – _{Credentials passed to scripts}

– _{Open HTML fields in Job descriptions etc.} – _{DOS attacks}

(23)

Typical Practice

§

_{Enable security!}

– _{+ switch on “Prevent Cross Site Request Forgery exploits”}

§

_{Hand off responsibility for}

authentication

– _{You’re a build engineer not a}

security officer

– _{Use LDAP or container security}

§

_{Upgrade from Winstone/Jetty}

& front with Apache

§

_{Think about resource caching}

(24)

Infrastructure

§

_{Start simple – no need to create a build grid on day-one}

§

_{Take advantage of plug-ins, but only load the ones you need and}

standardize on them.

– _{Consider control through a private plug-in site (Hudson 3.0)}

§

_{Consolidate on a single primary O/S}

– _{Only use multiple O/Ss for operating system specific tests and packaging} – _{Again, think how long it will take to recover from a disaster}

(25)

Infrastructure Cont.

§

_{Keep an eye on free space}

– _{A filled disk is one of the most common causes of problems}

§

_{Maintain identical environments across the farm}

– _{Ant, JDK etc.}

§

_{Consider an infrastructure management methodology / tool}

– _{VM snapshots, Puppet, Chef etc.}

– _{You’ve automated your build, so now automate your infrastructure!}

(26)

Upgrades

§

_{Upgrades should not be approached lightly, given that:}

– _{a) Hudson is critical infrastructure in your org, and} – _{b) Every Hudson install is different}

§

_{Test it first}

– _{Set up a parallel instance}

– _{Test (and possibly upgrade) your plug-in mix}

– _{Use of cascading jobs and parameterized scripts can really make this}

simpler

(27)

Summary

§

_{Treat Hudson like any other critical infrastructure.}

– _Manage – _Plan – _Backup

§

_{Simpler is better for your individual build jobs}

– _{Hudson is not a automation DSL}

– _{Externalize complex tasks into scripts}

§

_{Use the available plug-ins}

(28)

Insert Picture Here

§

_{Winston prakash}

§

_{Duncan mills}

§

_{Steve Christou}

§

_{Henrik Lynggaard Hansen}

§

_{Geoff Waymark}

§

_{Manfred Moser}

(29)