Development Process A case study

(1)

Development Process

A case study

(2)

Legal disclaimer

While we have made every attempt to ensure that the information in this publication has been obtained from reliable sources, bbv Software Services AG (bbv) is not responsible for any errors or omissions, or for the results obtained from the use of this information. All information is provided with no guarantee of completeness or accuracy, and without warranty of any kind. In no event will bbv or its employees thereof be liable to you or anyone else for any decision made or action taken in reliance on the information in this publication. The information in this publication should not be used as a substitute for consultation with professional bbv advisers. Before making any decision or taking any action, you should consult a bbv professional.

The names of actual companies and products (e. g. Microsoft, bbv Software Services AG, .Net) mentioned in the publication may be the trademarks of their respective owners.

(3)

1 Overview

1.1 Introduction

Some software development processes

• Top-down and Bottom-up • Waterfall model

• Evolutionary prototyping (Code and fix) • Unified Process

• RUP (Rational Unified Process) • Extreme Programming

There are a lot of documents describing conceptual software development processes. They usually divide the process itself in phases and describe these phases and their particular organization in depth. What is missing or sometimes just hidden beneath a lot of specification details are the answers to questions like these:

• How is software development within a team organized, so that the developers do not get in each others way?

• How to ensure the quality of the software during its development?

• How to organize continuous development of software, which is already installed at the customer’s site?

• How to avoid tedious, repetitive work?

• How to prevent the release management from going to hell?

There are a lot of other desperate calls for help of frustrated software developers where the questions above are coming from. Of course, like with all the big questions in life simple answers are rare. Unfortunately there is no religion we know of, which fills the gap and provides salvation for the nagging thoughts of so many that anywhere out there might be the ideal way to develop software. Now and then there is one who tells us he has found the ultimate answer and promptly gets a group of faithful believers. But time shows us that the right way to do it in one situation will be the wrong in another. What is left is to take the experience with real life projects, find the approaches that worked out well and use them again the next time.

And this is exactly where this booklet jumps in. At the end of a three year development process of an invoicing system in the medical business our team looks back at a lot of ups and downs. Now, instead of just using what we learned for the next project, we decided to share our ideas and insights. But please keep in mind, that this is just the way we finally did it. Take what you think are the good parts and for what is left, it’s up to you to find a better way.

1.2 The Project

A detailed project description is not necessary to understand the concepts we applied, but some background information might be helpful:

The goal of the project was an invoicing system for medical services with a capacity of up to four million invoices per year. It replaced a sixteen years old application with a character based user interface running on a Novell server. It had to be replaced because of new country specific standards for invoicing medical services and because the database of the old application had reached its limits some years ago and data was constantly reorganized into backup databases. The new system is a Server-Client application based on .Net Technologies and SQL Server as Database. Invoices are entered manually, are digitally sent directly from the provider of the medical service to the application or are imported from different file formats (the most important one is XML). The application sends the invoice electronically to the insurance or prints invoices ready for mailing. It also provides a web interface where medical service providers can enter their invoices themselves or update patient data. Additionally it takes care of the business process specific accounting before the condensed accounts are transferred to the actual accounting software.

(5)

The development team consisted of one project manager, three to six developers and one to three testers. The first deadline was given by the day, when the standards for invoicing medical services were to change and we had to meet this deadline even if hell froze. Unfortunately, we had less than half a year and it was impossible to replace the whole application in this time. Therefore we had to deliver a first release, which consisted just of the utmost important parts, so that the business of the customer could continue. The old system ran in parallel and we had to implement several interfaces between the two worlds. Because the first release was, as mentioned, everything else than complete, we had to provide the missing functionality as fast as possible, which lead to a two week release cycle. This booklet tells you how we coped with this tricky situation.

1.3 The Map

Developer Workstations

Daily Build Server Release Server Test system

Customer

VPN

Test database

Customer database

Every night:

- Get the latest source code and diff scripts from the source repository. - Build a test database - Update the customer database - Build the application - Run the unit tests

- Run the basic sanity tests _Hotfix: Add description here

Release Build:

- Branches the Release in the source repository

- Updates the assembly versions of the release

- Get the latest source code and diff scripts from the source repository to a directory with the name of the release.

- Build the application

Resembles the customers system as exact as possible . Release stays on test system at least one week before installed at customers site.

Testing department executes release tests on test system .

Source Repository Check In- Get latest source code and diff scripts - Run diff scripts on test database

- Build the application - Run diff scripts

Check out for daily build

Build Release

Test database Run unit tests

Hotfix

Install Release on test system

Tourist guides sometimes start the chapter about a particular city with a map and then tell you about all the interesting places where you can go. Just looking on the map does not help you very much. But just reading the text without having seen the map before or being able to look places up on it, will give you a hard time to find, where you actually are.

For this reason we start this booklet with a map too. It is a map of the final situation in our project. The following sections will show you parts of it to describe a specific problem or solution but you can always go back to The Map to get the big picture.

(6)

2 Source and version control

Source Repository

Check Out Check In

2.1 Do we really need it?

Examples for source and version control systems

• CVS

• SubVersion • Visual Source Safe • Rational Clear Case • Suround SCM • BitKeeper • SuperVersion • Perforce • SourceGear Vault • SourceOffsite • ...

There are still development teams out there, which manage their source and release versions by hand. Some of them even have it organized well and with enough discipline they get the upper hand over the threatening chaos. Still, it is the hard way to do it. Nowadays you can choose from a wide range of source and version controlling applications. There are free ones like CVS, which have entered the scene quite some time ago and can easily swim with the big fish in the pound. So if you ever had a mess if multiple developers worked on the same code or if more than one version of your Software had to be maintained at the same time, its worth to consider using a source and version control system.

2.2 Experiences with SourceSafe

In our project we used Microsoft Source Safe 6.0 but we do not recommend it. SourceSafe is the typical software, which comes with a bigger package, so that you have at least something. But at least something is normally not enough if you have to get serious work done. Here some disadvantages of SourceSafe we encountered during the project:

Workaround for branching with SourceSafe

Create a share of the project in the repository. Pin the version of the shared files. If changes have to be applied both in the branch and the current version, you can just move the pin. If the change has only to be applied in the branch, use the Branch function of Source Safe to create an independent copy.

• SourceSafe does not support branching very well (for the importance of branching for your version management see below). It only supports branching at file level, which simply creates a copy of the file in the repository. If you have to branch the project for a release version, you are actually creating a copy of your project in the repository tree (bloating your repository in size every time).

• Merging of branches is not supported by SourceSafe.

(7)

• Using SourceSafe only files that are explicitly checked out from the repository can be changed. This is an acceptable behavior also implemented by other source control applications. Still, it is problematic if more than one developer is working on the same code (even if you have configured Source Save for multiple Checkout).

SourceSafe terminology

Check Out

Users check out files to make changes to them. In the default configuration, VSS allows only one user at a time to check out a file but multiple Check Out can also be configured. Checking out a file copies its latest version into the user's working folder.

Check In

Files are stored in the VSS database and are unavailable for modification

Branching

Process of sharing a file with another project and then separating it into two or more branches. Once a branch has been created, two files (the file in the project and its counterpart in other projects) will have a shared history up to a certain point and divergent histories after that time.

Sharing

In VSS, one file can be shared among multiple projects. Changes to the file from one project are automatically seen by other projects sharing the file.

Pin a file

If you pin a file, you cannot thereafter make changes to it. Pinning is not the same as branching. You are not making a copy of the file. If the file is shared first, and then pinned in one project, other projects can still change and update the file.

Example:

Developer A has to implement a new functionality. He gets all the source files from the repository to work on the most current version. Developer A does not yet know which files exactly have to be changed. So, he checks them out along the way. Meanwhile, developer B works in the same area on another problem. He finishes his changes and puts them back into the repository (check in). Now, developer A has to check out a file, which has been changed by developer B. SourceSafe provides him with the newest version. Here we have the Problem: The newest version of this file belongs to all the code, developer B has checked in recently. So, there is a major chance that after a check out developer A has code, which is not compiling anymore. True, this is solved simply by getting all the sources form the repository, but the developer is unnecessarily interrupted in his workflow.

• SourceSafe knows no atomic check in. If you check in a bunch of files back into the source save repository and in the middle of the process you have a problem with merging a file, which cannot be immediately solved, you end up with half of your files checked. The rest of the team cannot get the latest version of the code until you have solved the problem.

• SourceSafe has an Analyze and Fix function. Now, how trustworthy is an application, which already assumes, that it will finally corrupt its data? There is even a chance that the analyzer tool will tell you that the SourceSafe repository is corrupted beyond repair and you have to start over. It happened once during our project, so it is something you really have to expect.

(8)

2.3 Branches

By creating a branch of a project in your source control repository you create a copy of the project, so that parallel development can proceed simultaneously on two independent versions of the project. You may create a branch of your application to add changes exclusively for one customer or to use it as the base for a new application. Two teams can work on two branches of the project in parallel and merge them together if a certain milestone is reached. Finally, you can (and should) use branching for your release management. Every release, which is installed at the customer site, has to be branched. Like this, you can easily fix bugs in the customer version while rest of the team is already developing new features.

In our project we used branching mainly for the release management (described in the sections Release and Hotfix). But there was also a time, when we had two branches for parallel development. As mentioned in the The Project, the existing application and its replacement, which we were developing, had to run hand in hand for some time. Now, a milestone was about to come, where a major part of the old application was going to be disabled. The corresponding changes in our application made it impossible to deliver another release before this milestone. At the same time, there were still a lot of urgent feature requests for the version currently installed. So, we created a branch for the current version and for the milestone version. Like this, we still could deliver releases on the current version, while already developing the milestone version. Unfortunately, SourceSafe does not support the merging of branches and so when we reached the milestone and had to merge the two branches mostly manually, we had quite a hard time. Another reason to use another tool the next time.

2.4 Token

As mentioned in 2.2 Experiences with SourceSafe SourceSafe does not have an atomic check in. If one developer is getting the latest Version of the code, while another one is checking in some changes, the first developer may receive an incompletely updated repository, normally resulting in code, which is not running.

Because in our project a check in consisted of more steps than just updating the repository (see 5.3 Check in), even if the check in of the source control system had been atomic, we would have had the same problem.

So, we had to serialize the access to the repository somehow. We finally did it with what we called The Token. It was actually a cuddly toy of the children of our project manager, which was not in use anymore. A developer, who was about to check in, had to get the token first. Like this, only one developer was checking in at the same time. Getting the latest version from the repository while the token was with a developer was not exactly forbidden, but you were doing it on your one risk.

(9)

3 Database

If your application accesses a database, you may also have to develop the database schema. We know how to manage code changes but what about database changes? If you add a new table to the database and write code, which reads from this table, other developers, which are to run this code, have to have an updated database too. Additionally, the customer database has also to be updated, when a release is installed but only up to the state the developer database had, when the release was created.

You could create a lot of SQL scripts with the changes and then run them manually if you need an updated database. But which script has to be run and in which order? Manually running them is tedious and mistakes are easily made. On the other hand, one single SQL script, where the developers add their changes individually is not a better solution at all. The script has to be run on a specific state of the database. So, if one developer changes the single script, all developers have to restore this state and run the complete script again. Additionally, for both solutions you do not know when the database has to be updated, if you are not informed about every change.

Confronted with this problem, we have developed the following concept:

• The developer creates a SQL script for all database changes necessary for his next update into the source control repository. We call it a diff script.

• Diff scripts are numerated (e.g. 00100Diff.sql) and stored in the same directory. A new diff script gets the next available number (e.g. 00100Diff.sql, next one 00101Diff.sql).

• The database contains a table VersionInfo, where the last executed diff script is stored. • The ExecuteDiffScripts tool (written for this project) runs all diff scripts in the directory. It

begins with the next diff script after the LastExecutedDiffScript value in the database (e.g. LastExecutedDiffScript 00235Diff.sql -> ExecuteDiffScripts starts with script 00236Diff.sql). If the diff scripts are finished successfully, the LastExecutedDiffScript value is updated.

This gives us an incremental and automated update of the database. When a release version is branched, the diff script directory is branched too. So, we can create the exact state of the database that corresponds to the release version.

(10)

Now let’s improve the concept:

• The database table VersionInfo contains also a database version number.

• The application checks the expected database version on startup. If the table VersionInfo does not contain the expected version, the startup is aborted with a corresponding error message.

• If the developer writes a diff script, which has depending code, he increases the database version in the VersionInfo table and the application.

If developer gets the new code without updating the database or updates the database with out getting the new code, a start of the application will tell the developer that the expected and the actual database version are different.

You could do without the LastExecutedDiffscript value and use only the database version number, but then the developer would have to recompile after every new Diffscript. With two separate values you can apply changes to the database, which do not directly affect the code (like creating an index) without increasing the database version. Other developers do not have to execute the (perhaps time consuming) Diffscript to run Application.

But there is still another improvement:

• You can set a file filter when running ExecuteDiffScripts (e.g. ExecuteDiffScripts – db:DemoDB –p:*Diff.sql)

• Diff scripts that do not match the filter criteria (e.g. 00103Test.sql) are not executed.

Like this, you can create diff scripts that should not be executed on the productive database of the customer (e.g. inserting test data). We used this feature to populate our test database (see below).

(11)

4 Testing

4.1 Unit tests

Unit tests are a great asset to the quality assurance of your application. By writing unit tests the developer checks the quality while developing. Unit tests serve as examples for other developers how to use the code and they are good protection against the dreaded side effect bug.

The side effect bug occurs, if a developer adds an innocent little change in code, which is called from different locations. The developer checks the change from the user interface or writes a unit test but normally this check is just to verify if the developer’s immediate goal is fulfilled. Then, some time later, perhaps after installing the next release at the

customers site, a nasty bug appears, which after a long search is pin-pointed to a small side effect the deliberate change of the developer has caused. Even with average unit test coverage of this code, the bug would most probably have been discovered, the next time all unit tests were run.

Unit testing

In computer programming, a unit test is a method of testing the correctness of a particular module of source code.

The idea is to write test cases for every non-trivial function or method in the module so that each test case is separate from the others if possible (Wikipedia).

In our project we used NUnit to write and execute unit tests. It works well with NAnt(see Automation) and was therefore easily integrated into our build process.

The most important problem with unit tests: They cost time and money. Of course, every book about unit testing tells us that more money is saved with better quality than spent by writing unit tests. But project managers usually do not trust anything that costs you now and may or may not save you money in the future. So you will end up with the choice to write code simply without bugs or to try fitting the unit tests into the original time estimation. In your project we had a compromise:

• As much unit tests as the budget allows. • Always a unit test for a fixed bug.

Another problem was the database. According to the rules it should not be part of any unit tests. It resembles a separate unit and is therefore to be replaced by a stub. We tried database stubs but finally abandoned the idea because it was just too expensive. Because the database can be accessed in succession or by writing data, which changes its state, it is extremely difficult to create the correct stub reaction for every situation. Additionally, we’d liked to test the SQL statements, which are passed from the application to the database. This was impossible with a stub. Therefore, we allowed unit tests with access to the database with one condition: After the unit test has finished (successfully or failed) the database has to be in the same state as before the unit test. So, we placed every unit test, which changed the contents of the database, inside of a transaction and caused a

Some unit test tools for .NET

• NUnit • NUnitForms • csUnit • MbUnit • dotUnit • Zanebug •

(12)

4.2 Basic Sanity Tests

With unit tests it’s easy to test the basic elements of your application. But what about a system test, which checks if the application itself shows the expected behavior. Unit tests are not supposed to be applied in this situation.

Daily Build Server

Every night:

- Get the latest source code and diff scripts from the source repository. - Build a test database

- Update the customer database - Build the application

- Run the unit tests - Run the basic sanity tests Because of the short release cycle, we needed as much

automatic tests as possible, to ensure that the application would still be completely functional after every installation on the customers system. Because unit test could not test the overall workflow of the application, we came up with what we called Basic Sanity Tests.

Other than with unit tests, the Basic Sanity Tests are executed on a running System. They feed the application with test input (medical invoices from different sources), trigger the different workflow steps and check the file output and the database contents. To get this working, you need to fulfill the following requirements:

• The tests run on a database with a defined state.

• The database has to be restored before running the Basic Sanity Tests because they will change the database and you cannot enclose them in transactions like the unit tests. • Because the database has to be restored before the Basic Sanity tests, it should only

contain the most necessary data. Although a copy of the customer’s database would produce more realistic tests, it is normally just too big.

• The application has to provide a non-graphical interface to execute basic operations, which can be operated by an automatic build tool.

• The build tool can execute queries on the database, for example with the help of a tool like osql.

Based on this we created a build target, which run a basic system test. Because the Basic Sanity Tests where automatically started once every day on the most current source, we had an almost immediate feedback if changes unintentionally affected the major functions of the application.

4.3 The test database

Both our unit tests and our Basic Sanity tests needed a small database with a defined state to be executed on. Therefore, we created a test database, which was filled manually with the data specifically needed by basic sanity and unit tests. The concept of the test diff scripts came in handy. Whereas the normal diff scripts run on both the test database and the customer database, we could use the test diff scripts to update the test database only.

Test database

But when our database model grew more complex, it got more and more difficult to fill the test database with consistent data. Although a copy of the customer’s database with productive data was available, it was a painstaking work to move new data to the test database and mistakes were almost unavoidable.

(13)

Although there are tools, which help you to copy data between databases, most of them were not selective enough for our need. The suffering of our project team together with an article of bbv in the Dot.Net Magazine finally lead to the internal development of a tool called CloneDB.

CloneDB checks the dependencies on the database to find all tables to copy along with your selected data. Then the user can decide up to which level these dependencies should be applied. Finally, the tool creates insert scripts based on the configured values and the selected database, which then can be run on the test database (see Database, diff scripts).

(14)

5 Automation

5.1 Build tool

We used the command line build tool NAnt to create application builds outside of the Visual Studio Environment. With a build tool you are not just limited to compiling your code. We used build targets to update the source repository, to run basic sanity tests, to update the database and so on. If you still have repetitive tasks, which you are doing by hand, consider using a build tool or something similar to automate the process.

5.2 Update the database

We used two kinds of databases during the development: The test database and a copy of the customer’s database with productive data. Changes were applied with diff scripts, so you had to run them on your local database regularly to update the database you were working with. Instead of having every developer running the ExecuteDiffScripts tool manually for every new diff script, we had the two databases on a server, where the diff scripts in the source repository were executed automatically every night. The test database was set up from scratch where as on the customer database only the new diff scripts since the pervious night were executed. The developer could update his local databases with one build target, which copied the server version and ran any remaining diff scripts from the current day. Of course, for efficiency, there was also a build target, which just ran the diff scripts from the repository if you still hat a working database, which was just not up-to-date.

5.3 Check in

Source Repository

Check In

- Get latest source code and diff scripts

- Run diff scripts on test database

In our project, you were only allowed to check in if all unit tests succeeded. For this step we used a build target called Integrate, which consisted of the following steps:

• Clean • Build • Unit tests

So, a developer, which was about to check in, would first get the latest source from the repository, then update the test database and finally run Integrate. If the build succeeded, the developer could check in his changes.

(15)

5.4 Daily Build

Another way to check the quality of your code is the Daily Build. We had a server, which did the following steps every night:

• Delete any project data at the build destination. • Get the latest source code and diff scripts from the

source repository. • Build a test database

• Update the customer database • Build the application

• Run the unit tests

• Run the basic sanity tests

If such a Daily Build finishes successfully, you know that… • No files necessary for a build are missing in the

source repository.

• All diff scripts in the repository can be executed without error on the test as well as on the customer database.

• The application can be built.

• All unit and basic sanity tests succeed in the current version.

In the beginning of the project, we did not have a daily build. Instead, during the week every developer checked

out the sources he was working on at will. Finally, on Friday (also called the integration day) all developers checked in and one poor soul had to get the application running again. If you are still doing it like this, replace the process with a daily build. It’s like Stone Age compared with the 21st century.

Daily Build Server

Test database

Customer database

Source Repository Check In Check out for daily build

5.5 Release Build

Creating a release to be delivered to the customer does not just mean compiling a release build instead of a debug build of the application. Our process of getting a release consisted of the following steps:

• Create a branch in the source repository.

• Update the release numbers of the assemblies in the branch. • Build the release version of the application.

• Install the release on the test system. • Create a delivery CD for the customer.

First we did all these steps manually. But tasks, which have to be done repeatedly, are always subject to automation. We moved everything apart form creating the delivery CD to a build target. So, if we had to create a new release we just executed this target and after patiently waiting

(16)

6 Test

system

The customers system was far more complex than the typical setup on a developer’s workstation. Two instances of the server application ran at the same time on different servers cooperatively. The user did start the client application through a terminal server. Printing and communication with the partners outside the company were placed on separate servers too.

Because we had a very short release cycle and there was no way to check the application on the customers system without running it in productive mode, we set up a test system, which resembled the customers system as much as possible.

After creating a release, it was installed on the test system and tested by the testing department for at least a week. Additionally, it could by used after installing the release on the customers system to reproduce bugs reported by the customer on the actual release instead of the working version on the developers platform.

`

Test system

Release Server

Resembles the customers system as exact as possible .

Release stays on test system at least one week before installed at customers site.

Testing department executes release tests on test system.

Install Release on test system

(17)

7 Release

Cycle

Because almost from the beginning we were developing on a productive system, which as far from complete, the release cycle was very short and every release absolutely urgent. To cope with this rigid timetable, we had to plan the release process in detail.

Developers Testing In s tal la ti on of t h e r e leas e at t he c u st omer s s ite 1 2 :0 0 – 1 3: 00 F in a l b u ild o f c u rr en t r e le as e . C re a te del iv er y C D . 16: 0 0 B ran ch r e le a se. I n st a ll on t e st sys te m . 1 6 .0 0 D e ve loper t es t day . A ll re leas e bu gs c los e d . T e st in g gi ve s O K fo r I n s ta lla tion . 12 :0 0

Development of next Release

R e le a se t e st in g s tar ts .

Developers are fixing bugs in the release branch.

Regression tests. Tests of completed features on developer version. Hotfixtests.

MO TU WE TH FR SA SU MO TU WE TH FR SA SU

Because the application was complex and the customer did not provide exact requirements, it was difficult for the testers to come up with in depth tests in the short time given. Therefore, we decided to support them with what we called the developer test day. The day after the internal release was created, every developer had to check part of the changes listed in the release notes (of course not his own changes). Together with the work of the official testers, this resulted in quite good test coverage. The down side: The development team is occupied with testing for at least half a day every two weeks, which is quite expensive.

(18)

8 Hotfix

Even with the two weeks release cycle, some bugs could not wait until the next release to be fixed. Any bug which prevented the customer’s employees to do their job meant constantly loosing money and set the clock ticking for us developers.

Test system

Customer

This is what we call a hotfix. First the bug is reproduced on the working branch. If it cannot be reproduced, it may already have been fixed for the next release. In this case, you just had to merge the fix from the working branch into the release branch. It is also possible that the bug just cannot be reproduced on the developer’s workstation. So, the next step is to

check the bug on the test system. If the bug can be reproduced, it has to be fixed on both the working and the

release branch. Then a new release with an updated release number for the affected assemblies is created on the release server and hotfixed to the test system, so that it can be verified by the testing department. Finally, if everything is OK, the hotfix is applied to the customer system. We were fortunate because we had a VPN connection to the customer. So, installing a hotfix was quite easy as long as no shutdown of the server was necessary.

VPN

2. Create a new release build

3. Install Hotfix on test system

1. Fix the bug in the working branch and the release branch

4. Verification of the bug fix by the testing department

5. Install Hotfix on the customer’s system

(19)

9 Putting

all

together

Now have another look on The Map we started with. The decisions we made and the roads we took might not all have been the one-and-only ones. But for our team most of them work quite well and will be also applied for future projects. We hope this booklet gave you some insights or ideas for your next project or was at least an interesting look at the way other developers cope with some of the everyday-problems.

Developer Workstations

Daily Build Server Release Server Test system

Customer

VPN

Test database

Customer database

Every night:

- Get the latest source code and diff scripts from the source repository. - Build a test database - Update the customer database - Build the application - Run the unit tests

- Run the basic sanity tests _Hotfix: Add description here

Release Build:

- Branches the Release in the source repository

- Updates the assembly versions of the release

- Get the latest source code and diff scripts from the source repository do a directory with the name of the release.

- Build the application

Resembles the customers system as exact as possible . Release stays on test system at least one week before installed at customers site.

Testing department executes release tests on test system .

Source Repository Check In- Get latest source code and diff scripts - Run diff scripts on test database

Check out for daily build

Build Release

Hotfix

Development Process A case study