Agile Development c/w Continuous
Integration and Testing
……….………...
JOHN
PAYNE
………...
SENIOR DATA AND SERVICES DEVELOPER
UK DATA ARCHIVE
UNIVERSITY OF ESSEX
……….………..
Overview of today’s talk
• Brief background: the UK Data Archive and the UK Data Service
• Historical and Current Structure of ADM
• Why use Agile?
• Why use CIT?
• How we use Agile and CIT together
• Where we are going with Agile and CIT?
……….………..
The UK Data Archive and the UK Data Service
• based at the University of Essex since 1967• curator of the largest collection of digital data in the social sciences and humanities in the UK
• see data-archive.ac.uk for more details
• makes these available via the new UK Data Service
• UK Data Service also provides value-added services for UK Census data, government surveys and beyond
• UK Data Service includes Universities of Essex, Manchester (Mimas, CCSR), Leeds, Southampton, Edinburgh (Edina) and
……….………..
Historical Structure of ADM
• Until two years ago, Application Development and Maintainance (ADM) used the waterfall method of application development.
• One developer assigned to work on a product from inception to implementation across all layers of development.
• This had lead to some of the following all too common issues • Repetition and duplication of code across projects
• Isolated effort – “Stove Pipes”
• Repetition of data across systems – but not quite!
• Poor quality code and lack of consistency
• Unmaintainable code
• Poor scheduling and long lead times
• Unfocused meandering development cycles
……….………..
Current Structure of ADM
Application Development and Maintenance (ADM) now consists of two teams which opens up Agile possibilities:
• Applications
• Front end development
• Umbraco/MVC/Webforms Java/Solr /Sharepoint…..
• Services
• Back end development
• C# Web services/Tools/CIT/Databases/Data feeds……
……….………..
Why use Agile
• The introduction of Agile techniques has had many benefits:
• Focused project management and decision making during a sprint
• Reduced development time
• Better utilisation of resources and skills
• Technology specialists rather than application specialists
• Improved communication within ADM itself • Stand up Meetings
• Whiteboards
• Improved communication outside of ADM
• All interested parties encouraged to attend stand ups – not just the team and the project manager.
……….………..
Why use CIT
• To avoid the mistakes and pitfalls of the past and improve turnaround, quality and reuse.
• With reuse comes dependency – not always visible to a developer • Small changes can lead to big problems – library reuse
• Errors are spotted early
• Resource changes should not lead to unmaintainable code • Enforces stylistic rules for readability
• Force developers to produce documentation
• Improve quality and reliability
• Resharper encourages refactoring – BUT handle with care!
• Builds (DLLs) are automatically versioned. • Essential for managing deployments
……….………..
How we use Agile
• Our main ‘tools’ for Agile are
• Requirements
• User Stories “Given that … when … then ….”
• JIRA – Issue Tracking software
• Time logging and sprint management
• Greenhopper – Plugin to JIRA • Scrum
• Kanban
• Scrumban!
• Prioritising
• Burndown graphs
• Sprints – One or Two Weeks
• Daily standups
……….………..
Sprint Planning in JIRA using Greenhopper
Sprint
Commitment
Sprint Backlog
……….………..
KanBan Board within JIRA
Swimlanes are
……….………..
Jira Project reporting showing sprint activity
Sprint 1 Planning Sprint 1 activity Sprint 1 review Sprint 2 Planning Sprint 2 activity Go Live!
……….………..
How we use CIT
• Our main ‘tools’ for CIT are
• User Stories – Requirements
• JIRA – Issue Tracking software
• SVN – Source Control Repository
• Jenkins – CIT Server
• Resharper – Visual Studio Tool
• Stylecop – Code style ruleset
• Nunit (Rhino Mocks and MOQ)
• Selenium – UI Testing
• Dogygen – API documentation
• Robocopy – Depoloyment
……….………..
How does Jenkins fit in?
Developer PC • Source Code • Test Code • Shared Libraries • Resharper • Stylecop Jenkins • Build Script • Stylecop • Tools Workspace • Builds, • Test results • Documentation SVN • Source Code • Test Code • Shared Libraries Check in/out Monitor builds Build occurs
……….………..
Jenkins Screenshots
Job Status
……….………..
Complete CIT flow
Developer PC Jenkins SVN Test Server Live Server Staging Server
Monitor builds and failures Email notifications sent
Code development Test development
SVN Trigger • Pre-Build – DLL Version• Build • Unit Tests • Selenium Tests • Code Coverage • API Documentation Jenkins Success! Triggers further ‘lift and shift’ job from Jenkins’s Workspace. Updates JIRA issues
Successful build triggers further jenkins job and updates JIRA
Deployment script using Robocopy
Jenkins (Manual)
Manually triggered
‘lift and shift’ job Jenkins (Manual)
Manually triggered ‘lift and shift’ job
fires remote build
……….………..
Agile and CIT Issues
• Because of the team structure within ADM, any application sprint requires a pre-sprint by the services team.
• How far to go with testing? • Code Coverage and Mocking
• ‘Functional’ Tests and Unit Tests
• Interfaces!
• Estimating
• Time logging is good
• R&D during a sprint is bad.
• Creep (requirements churn)
• UI developer can end up building to an outdated requirement.
• Communication becomes paramount. ‘Speak up at a stand up’!
……….………..
Where we are going with Agile and CIT
• Database and deployment tools – Redgate Developer Tools
• All new developments involving 3 or more people to be developed as sprints.
• Thorough testing and continual upskilling • Test-Driven Development (TDD)
• Behaviour-Driven Development (BDD)
• Improved planning and estimation – Jira time tracking
……….………..
Useful Links
• Jira http://www.atlassian.com/software/jira/overview • Greenhopper http://www.atlassian.com/software/greenhopper/overview • SVN http://subversion.apache.org/ • Jenkins http://jenkins-ci.org/ • Selenium http://docs.seleniumhq.org/ • Stylecop http://stylecop.codeplex.com/ • Resharper http://www.jetbrains.com/resharper/ • Redgate http://www.red-gate.com/products/sql-development/sql-developer-bundle/ • Doxygen http://www.stack.nl/~dimitri/doxygen/index.html • Sonar http://www.sonarsource.org/……….………..
Thanks for listening
……….………..
