Reaching Software Development Maturity with Continuous Delivery. Fredrik Stål Viktor Ekholm Faculty of Engineering, Lund University

(1)

Reaching Software Development

Maturity with Continuous

Delivery

Fredrik St˚

al

Viktor Ekholm

Faculty of Engineering, Lund University

July 23, 2013

(2)

(3)

Abstract

In the area of software configuration management a configuration manager works with almost every part in the software development process. A part of the responsibility is to ensure efficiency at the highest possible level for a development team, which is simplified a great deal with a mature and well defined development process.

The purpose of this thesis was to analyze and locate key areas in a software development process for small teams and define methods of improvement that will serve as goals for to reach process maturity. We chose to base our work upon the well-known concept of continuous delivery, which aims at maintaining a releasable product after each software code integration. The project was carried out on a consulting firm called Softhouse that also has in-house development projects with mobile applications in teams ranging from one to seven developers.

The work started with interviewing the developers with the goal of finding the root-cause of the current process, as well as identifying both the possibilities of improvement and determine the requirements they had on their own work. To narrow the scope we focused on working together with a project that started and had its deadline during this thesis. That project was split into three smaller sub-projects, which gave us the chance to explore how more than one development process was normally carried out and how collaboration was done.

Together with the developers in one of the sub-projects we designed and implemented a new environment with the aim of solving both the perceived and real problems that were discovered during analysis. The purpose of the environment was to perform manual tasks by automation and introduce the starting points of continuous delivery. Measurements on the process was done before and after the improvement to find what benefits gained and any eventual costs. The results of the measurements were positive and although the change was a very small one the benefits was still detectable.

Based on our analysis and results from measurements we developed a maturity model. The primary objective of the model is define a set of goals to improve a software development process. The model is split up into different levels of maturity and to reach a higher level the process first has to achieve the goals for the next maturity level. This maturity model is the result of our master’s thesis and is still in its experimental stages. It would require further testing and research to be fully developed.

Keywords

Software development maturity, software process improvement, software configuration manage-ment, continuous delivery, continuous integration, automation, agile development

(4)

(5)

Preface

Configuration management has had different definitions over the years. One of the older definitions tells us that the goal of configuration management is to “maximize productivity by minimizing mistakes”. Another definition is that configuration management “represents general techniques for controlling the creation and evolution of objects”.

As its name applies configuration management manages configurations and a configuration can been seen as a photograph of an object at a certain point in time. As the amount of variations increases for the components of the object, so does the number of available configurations. Without sufficient ways of maintaining all these configurations we lose the ability to tell what components and dependencies that was put into a specific configuration and we lose history on versions of all components. Variations will become blurred and we would soon enough, simple put, end up with a mess.

This thesis is targeted for individuals with some expertise within the subject of software devel-opment, be it developers, project managers, configuration managers or undergraduate students in the field of computer science. We will attempt to keep our discussions at a level so that any person belonging to the aforementioned groups can make use of the information presented and discussed in this report.

(8)

(9)

Acknowledgments

First of all we would like to thank our supervisor, Christian Pendleton, and our examiner, Lars Bendix. Their invaluable assistance and great engagement in our work became fundamental in keeping us on our toes and spur our own engagement. We would also like to thank all employees of Softhouse in Malm¨o, the company where we carried out the work you are about to read, for their hospitality and the willingness to participate in our interviews and demonstrations.

(10)

(11)

Chapter 1

Introduction

As fine spirits mature and develop a more complex and interesting taste over the years, the same can, almost, be said about software development. Having a mature software development process may not produce any extraordinary taste sensations, but it does give you a comfortable and effective work process. The prerequisites are to put in effort to actually mature the software process.

In short, improving your software process aims to rid you of any manual, repetitive processes and possible downtimes due to events you have to wait for to complete. Unnecessary rework is considered an anti-pattern and a very bad practice. Automating this hard labor intends to give developers a break from tedious, manual tasks, but in theory quality and consistency would improve, since there is less chance of careless mistakes due to automation.

A fundamental building block in software process improvement is the practice of continuous de-livery. Through this practice we expect less integration problems as software code is integrated in small portions several times per day. Through automated test executions upon integration, we receive iterative feedback which tells the status of our software.

1.1

Problem Statement

Softhouse is first and foremost a consulting firm but also conduct some in-house development. These development projects became our focus and it was initially perceived that the in-house developers were in need of a common build server.

Through interviews with developers and stakeholders, our take on the in-house environment was that the nature of the company required developers to be given assignments on other companies between projects. Their knowledge of processes is then temporarily lost. As there is no collective knowledge of processes, there is a lot of work made ad-hoc at project start. In a way this has a relation to open-source projects, where it is assumed that contributing developers will not contribute during the entire lifetime of the project.

Another problem is that no value is seen in taking the required time to mature and baseline processes. Work hours are considered to be billable and the customer is most certainly not interested in paying good money for advancing the processes of Softhouse after they already accepted them as developers. The interest for Softhouse in developing more efficient processes is to place more time and focus on developing the software and thus increase productivity.

(12)

1.2

Purpose

The purpose of this thesis was to, through interviews and analysis, locate key areas in the develop-ment process that could benefit the most from improvedevelop-ment. Although being a smaller company, Softhouse had multiple projects running simultaneously with different number of developers in each one, ranging from one to six. Our first target was only the projects that developed applica-tions for the Android platform and try to extend the process by adding additional features and tools. The intention was to improve, not only to increase efficiency of the development, but also the communication with customers.

After a certain amount of weeks we changed our target to a newly started project that had three smaller parts. These parts consisted of an ASP.NET website, an Android and an iPhone application, respectively. The motivation of this change of target was to have the opportunity of accessing and monitoring a live project taking shape in real-time, and not just basing our work on theoretical or old projects. Also, the diversity aspect of this three-parted project interested us. The primary goal then became to analyze the process used for each part and together with the developers work with improving beneficial areas during the course of the project. It was discovered that one of the smaller project contained a deployment procedure that was performed several times a day to a testing environment, which we saw as a perfect opportunity to study how processes are carried out on Softhouse.

There was no unified or general approach for projects, which had led to that developers on each part of the studied project had performed implementations on an ad-hoc basis. Besides analyzing the reasons and root-causes of the chosen methods we also wanted to implement an environment with the purpose of introducing automated procedures for as many of the manual activities as possible. This would further benefit our own work, but our initial hypothesis was that it would also help to increase knowledge of well defined practices and make the company see the benefits of such process improvements from a practical perspective.

1.3

Chapter 7 evaluates and discusses the “validity” of the work. Any restrictions and limitations that are posed on the implementation and its general application are brought up and finally relations to other work in the same area and how to extend what was accomplished. Chapter 8 is a summary of all the results of the thesis and potential thoughts for the future.

(13)

Chapter 2

Background

In this chapter we will begin with outlining the methodology that was used during the course of the project. The sections of this chapter will also discuss how to compare projects and analyze the current process. In the final section we discuss how these kinds of analysis will lead to a discovery of initial problems that are perceived while exploring a specific project case with relations to the problem domain.

2.1

Methodology

During the stages of our work we reviewed several methodologies that could be of use for defining the domain, discover the problems that the developers were faced with, analyze their cause and designing our solutions. Since our task was to produce a template for a development process applicable for the company, our approaches had to reflect that. We wanted to derive requirements on a process improvement from a problem analysis and use appropriate methods to implements these into the current process.

We will begin by describing all of the methodologies that we reviewed during the course of our masters thesis.

Interview. An ordinary interview, individually or in group, where we ask questions and, hope-fully, get answers.

Questionnaire. An approach where questionnaires (or forms) are sent to the interview subjects, be it in physical or digital form, with fixed and/or free-text answers.

Workshop. Doing a workshop with the developers to get a view of how they really work in their actual environment.

Demonstration. Present a solution for a group of developers to discuss the implementation and receive feedback.

Test environment. To see if a solution is practical it can first be implemented and tested on an environment which is not used by development teams.

Documentation. Every step is documented as well as discussions and results from tried solutions, which later put into the final report.

Literature study. The method of acquiring relevant literature for the problem domain and read-ing it.

(14)

We decided on conducting individual interviews with persons of interest. We felt comfortable with this approach and believed the interviewees would feel the same as they would not have to do anything but sit back and speak their mind. The basis for this choice, other than previous stated beliefs, was that we could talk and discuss with developers directly and freely. Based on the answers from interviewees we could also manufacture new questions on the fly if we stumbled upon an interesting subject.

The questionnaire approach would not, in our perspective, have produced these spontaneous dis-cussions as the questions would have been static. A workshop-like interview is also believed to be unsatisfactory as developers could rely on their tools to speak for them. In a regular interview the developers are extracted from their work environment and would have to explain the problems and their processes in their own words, making it easier for us to determine e.g. the level of competence based on their phrasing.

The downsides to the interview approach that we were aware of, or quickly became aware of, was that that it was difficult to steer the interview in the desired direction. When we gave the interview subjects too much freedom during the interview they quickly diverged from our original question and began discussing technical solutions, which seemed to be a more interesting topic. By doing the interviews as a pair we could have one of us conducting and steering the interview, while the remaining one jotted down notes and answers from the interview.

Using a demonstration to present a plausible solution turned out to be very effective. We were able to get more feedback from a demo than any another method. It was also effective with a testing environment, since we could be free to try out many kind of solutions without hindering someones work. It became very necessary for us to document every step we took during our work, since at the later stages the test environment, for instance, contained a lot of information and experiments that would otherwise not had made much sense and their connection to our analysis would been hard to detect.

We also did some literature studies within the problem area, but relevant literature was scarce. As the initial problem statement involved the need of a continuous integration (build) server, we initially made some studies within the practice of continuous delivery—specifically Humble and Farley’s book on the subject [HF10] and other sources [FF06, HRN06]. Amongst our literature we also studied previous work on similar conditions.

2.2

Domain

As was briefly mentioned in the introduction and in the previous section, we were presented with some initial problems from the organization. There had been some issues concerning the development environment and a they had a wish for a more simplified, but still controlled, manner to handle integration and deployment. The problems had originated, as was explained to us, from a build server that was no longer maintained and no one on site had the knowledge to maintain it. Since the company is a consulting firm, the personnel may receive assignments and take their knowledge and competence with them.

Before making an attempt at analyzing a project, we would need to define our domain, to know our boundaries as the analysis is dependent on the domain. The domain specifies what we need to look for when doing the analysis. With a definition of the context and a comparison between the three studied project types (case descriptions) we can start explaining the initial problems in chapter 1 in more detail.

(15)

2.2 Domain 5

2.2.1

Defining the Contexts

As was described in previous section, we chose to conduct interviews to gain a better understanding of the problem domain we were dealing with. The goal of our interviews was to root out what and what was not part of our problem domain. Whatever approach that we choose to conduct it is the initial problems the company has laid out that are in the focus, because they are the most visible symptoms, so to speak. Most of the answers to our questions during the interviews turned out to be focused on technical issues at the organization.

There were a lot of suggestions for solutions which involved implementing different types of tool support into the development process. The tools suggested ranged from more control of the build environments to cloud support. Given a suggested solution we tried to work out which problem this solution or tool would actually solve and if the underlying reason was that “it is nice or cool to have” we became skeptical. During our interviews we tried to have the following question in the back of our minds: if this is the solution, what would be the problem?

We learned that asking which problems that would be solved and what area in the development process that would benefit from these solutions, we started to receive information about the context and the way of working. We also detected the existence of three contexts. We defined them as company context, process context and project context. With this division we can narrow the scope on an interview depending on the position an employee holds. Our assumption was that a developer would be able to give the most accurate answers related to the project context and a finance manager to the company context, as it is part of their daily work.

Company context contains items with information like the status of the organization, its avail-able resources in terms of economy and the average competence on its employees of config-uration management or knowledge of technologies.

Process context is the context of what type of work model (agile, waterfall, Lean, etc.) that are used in the development and planning processes. It also explains if any well-known practices are followed, for example test-driven development or planning poker.

Project context describes a unique project in an organization in terms of the team size and its setup, as well as the different personalities in the team.

In our case, we were mostly able to collect information related to the project context. The rea-son for this was because the majority of the interviewed perrea-sonnel were developers. An optimal approach would have been to put most focus on the process context, since it was the process at Softhouse we wanted to work with. But given that the organization did not have any employees with the sole assignment to maintain the development process, or something similar to a config-uration manager, on-site the interviews gave us accurate descriptions of how project and teams were set-up and some loosely information about the development process.

The properties we looked at when defining our contexts are as follows:

Company size. Softhouse has, at the writing of this report, just about 100 employees and have offices in four cities in Sweden. Most of the employees are on consulting assignments at other organizations.

Competence levels. There is no configuration manager in the in-house projects, but there still exist knowledge of configuration management and its processes. The employees with the highest experience and knowledge of configuration management are often those that are assigned to other companies.

Knowledge of tools. Through our interviews we identified that tool support is very com-mon in projects. Version control tools exists in every project and there has been some

(16)

experiments with tool support for integration, code review, task management and issue tracking purposes.

Process work model. In almost all of the projects and the one that we studied Scrum was used as the work model. It is not followed “by the book”, but the planning process inherits much from the model.

Practices followed. What we were able to extract was that the only practice followed was continuous integration. The practice itself contains several concepts, that will be ex-plained in chapter 2, but we only saw evidence that integration and merging was done few times per day.

Project type. The project was a client-server application for cell phones. It was split up into three smaller project, two client projects and one server project. See the case descriptions in the following section 2.2.2 for details.

Project size. The project started with eight members in total. Later it grew as more em-ployers were called in to assist.

Team setup (activity distribution). The team consisted of a requirement engineer, a project coordinator and the rest of the members were developers. Testing was done at the developers own discretion and there was planned test phase in the final weeks before deadline.

With the contexts defined it is now relevant to locate a connection back to the initial problem to lay the foundation of further analysis. Since the three contexts will be the basis for further analysis, making a list of perceived problems on the whole domain will help us achieve traceability when analyzing a specific project in the same way that the initial problems defined by the organization helped us when defining the contexts.

The issues the company been having with their experiments with build environments can be related to the fact that there was no employer designated as configuration manager and there had been no formal education on the subject. Another piece of evidence is the knowledge of both tools and practices are different from employer to employer. This is evident in the initial problem that when a certain individual with knowledge of a particular tool or practice takes charge of setting up an environment, there is no guarantee that the environment can be maintained if that employee receives an assignment. We did not find any evidence that there had been any prior attempts at a tool evaluation. All of these three factors has lead to that projects are set up ad-hoc.

Summarizing the issues we found during our context definitions we have: • No designated configuration manager

• Scarce knowledge of well-defined practices • No evaluations on tools have been performed

• Integration and deployment problems are solved ad-hoc

2.2.2

Case Descriptions

As we saw in section 2.2.1 the project we were tasked to target contained three different smaller projects. As we wanted to fully understand the domain we also was required to understand these project cases. With understand we mean that we wished to know of any standard conventions, if any particular tools are required for development and how objects and artifacts are managed.

(17)

2.2 Domain 7

The reason for this is because it puts a limit on the amount of variant solutions we can have for each case.

Following is a list of our cases, which were the focus for all of the studies in the rest of the thesis. ASP.NET website. An ASP.NET Web Site Project (WSP) using Microsoft’s .NET framework. This case is commonly locked to Microsoft tools during development and thus complicate matters if developed on another platform than Windows. Compilation of source code is performed at runtime, which means that no binaries are produced after a build. In other words, The source code files are the primary artifacts and to verify the system we have to verify that they can be built on a specific platform.

Android client. Excluding the functionality for implementing graphical interfaces it is quite similar to regular Java applications. The development can be done on any platform, with the development kit installed, but running the application requires either a simulated version of an Android phone or actual hardware. A build produces a single binary file which is deployed and executes the application.

iPhone client. Produces binary artifacts similar to a regular application written in C. Developing these applications, however, requires an OS X platform. Running these kind of applications during development requires that the cell phone is explicitly granted access in the source code.

The next step is to perform a comparison against a general project type and other project types within the organization, in a software configuration management perspective. This is required to extract the differences among the cases to later establish a solution for each case. The project were compared to each other, as well as a general case.

General Project. So what does the general software project entail? Well, for one thing we have the production of binaries. A binary in this context is the output file or files produced via the process of compilation. While this binary may not qualify as a true binary file, it is no longer the source code.

Moving on, our general real-world software project employs some kind of version-control system (VCS for short). With a VCS we have the shared codebase in our VCS repository. This repository can be accessed by the project team members, having been granted access to it. Developers can update themselves against the central repository, write their code locally, and then merge and commit (integrate) their work to the repository so that the rest of the team can update their local copy with the new code.

As for unit tests, they are most commonly a part of the project, often contained in a separate test package inside the source code folder. Unit test source files are version-controlled in the VCS repository together with the actual application code. It is, on the other hand, common to have acceptance tests in a separate, external project, as these kinds of tests are to perform black-box tests targeting the application project.

Summarizing the above we have the characteristics of the general software project: • Source code is compiled into binaries which become build artifacts.

• A version-control system (VCS) is used for storing and version-controlling the source code in a shared repository.

• Unit tests are a part of the project.

• Acceptance tests are housed in a separate project and performs black-box testing against the application.

(18)

Comparing this project type to the previously defined general project, we can notice one major difference: the lack of binaries. A WSP utilizes runtime compilation, meaning that the source code is compiled at runtime on the server, instead of compiling and deploying binaries beforehand. If we refrain from diving into the technical aspects of how this runtime compilation is carried out and look at it from an SCM perspective, we have a set of artifacts contained in a specific file structure that we are to deploy. This file structure includes directories and many different file types as the web format allows for many different types of scripts (both server- and client-side) to interact with each other upon access via a web browser.

The found characteristics of the ASP.NET project, differing against the general project: • A WSP project uses runtime compilation, the source code and file structure of the project

is the artifact to be deployed and are compiled and executed when accessed [Mit09]. • The production environment (operating system, runtime compiler) decides the output

from the deployed source code.

• WSP’s can alternatively be (pre)compiled [Mit09] for deployment or testing purposes. • Necessarily not only one programming language, a website may use client-side scripts,

markup files, style sheets, etc.

• Officially supported tools for ASP.NET, such as compilers or test frameworks, are only available for Windows.

Android Project. We also studied an Android application, which would serve as a client to the ASP.NET server application. Android applications are written in Java, and in many ways an Android project is similar to a regular Java project. However, there are some distinct differences which we will highlight here, together with the differences that Java development introduces compared to the generalized project.

Compared to the above described ASP.NET WSP project where the source code files forms a set of artifacts, Android applications are compiled and packed into a single binary. This binary or executable can only run in an Android environment, meaning an emulator or an Android phone. The unit testing strategy differs a bit from the general project in that it is suggested to have a separate unit test project [and]. This is because an Android application must be executed in an Android environment, i.e. an emulator or on a physical Android phone. Therefor the test project is also compiled and packed into a binary and deployed to an Android environment alongside the actual Android application, making it possible to test the Android application in its natural habitat. Worth noting is that tests not utilizing the Android API need not be tested in an Android environment.

The unique characteristics for the Android project compared to the general project are as follows:

• The source code is compiled and packed into a single binary file.

• An Android executable can only run within an Android environment, i.e. an emulator or Android phone.

• Android development is cross-platform.

• Unit tests dependent on the Android SDK must be contained in a separate project from the application.

• Unit testing is carried out by deploying the compiled test binary alongside the applica-tion binary to an Android environment.

(19)

2.3 Preliminary Analysis 9

• No restrictions on development environment or build tools.

2.3

Preliminary Analysis

Now that the domain and the three cases have been defined, analyzed and compared we can move forward with analyzing the process followed on a project at the organization. In our case we had three smaller projects and we chose to treat them individually by analyzing the process for each case separately. As we will see in a while there was distinct differences in the process followed in every case, which is precisely the reason for why we decided on this approach. To gain the information required to establish the current process we can again make use of interviews by the same reason that was discussed in section 2.1.

The information we are interested in here is mostly the kind that related to configuration man-agement, e.g. the procedure to identify configuration items and how build artifacts are managed. It cannot hurt to find out what tools or development environments that are used and, more im-portantly, what problems they solve or how they support the process. This is because it might become evident that some tools are required in a particular case, which will become a required when discussing solutions.

• Project setup procedure • Configuration identification • Artifact management • Tools used and to what end • Test management

• Integration and deployment procedures

The first three bullets will give us information on how a project is managed in terms of general organization and configurations. Tools used and to what end was explain in the previous paragraph. By covering test management we will know how the process handles the organization of tests related to a case description of the project type. The last bullet covers challenges with how different platforms are integrated, how environments work together and when integrations are performed.

For each case analyzed we will list any perceived problem that we identified as a potential issue. Any listed problem is linked to the preceding paragraph and is labeled unique for the purpose of traceability in later chapters. Unfortunately, due to limitation in time we were not able to perform any further analysis on the client application project for iPhone.

ASP.NET Project. We started with the ASP.NET project because it was the only case with a deployment to another node through network channels. In that project the developers had chosen to work with Visual Studio which, together with the project type, required that unit test code be placed in a separate Visual Studio project. As there was different opinions on how and what to test they had decided to only write tests for their own code.

A.PP1. Tests are not part of the project, they are a project of their own. A.PP2. Tests are written by the same person that wrote the code.

They had made use of what should be a production-like environment. However, during our interviews we found out that the actual production environment was actually running on a different operating system than the production-like. Looking back to the case description of this project type, a part of verifying the system is to ensure that it can be built on a chosen environment.

(20)

A.PP3. Test environment much different than the production environment.

A deployment to the test environment is performed by copying the source code via a manually executed script. The files are not version controlled on the test environment and are thus replaced at each deployment. When the system receives a connection after a change has been made it is rebuilt.

A.PP4. Builds are performed on development environment and then again on test/production environment.

A.PP5. Copies of source code exists on both development and test environment. A.PP6. Tests on the client platforms that calls the server can fail if a rebuild fails.

We were also able to discover that on certain occasions, changes to the system located on the test environment and database it contained, were executed manually by directly accessing the node. The issue we saw with this procedure was that these changes never got recorded in any way and it had the risk of leading to a divergence between the two copies of the system.

A.PP7. Changes to the database and the system are not always recorded.

Android Project. For the Android application project the process was a very much common approach according to the documentation and case description of how an Android project is carried out. This project started out with only one developer and that developer had chosen to do all of his work in the Eclipse IDE. Eclipse creates certain configuration files for both a project and a workspace and we found that these were present in the repository but not libraries that the application required to compile and build.

B.PP1. Workspace-specific files are checked into the repository.

B.PP2. Libraries, upon which the application is dependent, are not checked in to the repository. According to the case description, unit tests are kept in a separate project. This was also the case with the project we analyzed, although the “test project” was also kept in a separate repository, which we did not find any evidence that it was a requirement. Actually it is not entirely correct to call the tests unit tests, since most of them called the API and making them more of acceptance tests or black-box tests. This also led to that execution of the tests required that the entire project be packaged and deployed to either a simulation of a cell phone or a physical one.

B.PP3. Unit tests are kept in a separate repository.

B.PP4. Production-like environments are required for all test executions.

During our interviews we become aware that it was an important that the graphical interfaces on both client platforms being identical. To ensure this developers from each platform manually compared their implemented interfaces during development. These sessions were not documented and performed at their own discretion, i.e. not planned.

B.PP5. Verification of graphical interfaces are performed manually and not documented.

2.3.1

Perceived Problems

Looking back to the discussion of the problem domain as well as the initial problems we can see that an analysis of a specific case have revealed a link between the problems listed in the previous section with the problems for the domain. For the ASP.NET case there is evidently a lot of manual and ad-hoc procedures, e.g. testing, deployment and changes to the database. The developers had

(21)

2.3 Preliminary Analysis 11

different opinions of how to manage the project and the lack of a configuration manager might lead to the results that are repeated in the following list.

A.PP1. Tests are not part of the project, they are a project of their own. A.PP2. Tests are written by the same person that wrote the code.

A.PP3. Test environment much different than the production environment.

A.PP4. Builds are performed on development environment and then again on test/production environment.

A.PP5. Copies of source code exists on both development and test environment. A.PP6. Tests on the client platforms that calls the server can fail if a rebuild fails. A.PP7. Changes to the database and the system are not always recorded.

If we instead look at the Android project we had the case of a single person project. This lead to a majority of issues related to the structure and organization of the repositories and their artifacts. We chose not to delve into the standard convention laid out by the documentation for Android development. We were more interested in how the developer had chosen to manage his project based on the unique case.

B.PP1. Workspace-specific files are checked into the repository.

B.PP2. Libraries, upon which the application is dependent, are not checked in to the repos-itory.

B.PP3. Unit tests are kept in a separate repository.

B.PP4. Production-like environments are required for all test executions.

(22)

(23)

Chapter 3

Locating the Gremlin [6]

Originating from the air forces of the United Kingdom, a gremlin is a mythical creature who spends its days performing mischief. The airmen at that time used to blame faults with their aircrafts, for which they could find no cause to, on a gremlin responsible for sabotaging the craft. The main focus of this chapter is to discover the cause—the gremlin—to the perceived problems that we saw in the final two sections of chapter 2.

Figure 3.1: A gremlin sabotaging a computer, just for the fun of it! Figure by Larry Wentzel, August 26, 2012. Licensed under CC BY 2.0. http://www.flickr.com/photos/wentzelepsy/ During the analysis of root causes we also want to discover problems that are on a more abstract level than the ones we found in the previous chapter. We can later use these abstract problems to derive requirements for suitable solutions, which will cover more than only the cases that we analyzed compared to if we instead used the perceived problems. We will call these new problems “real problems”.

3.1

Discover the Real Problems [5]

Before we start with detailing how we performed our analysis of the ASP.NET case and the Android case we want to explain the purpose of doing a root-cause analysis. After gaining knowledge about the domain in chapter 2 and interviewing employees to discover the problems from sections 2.3 and 2.3.1 we could have begun to derive requirements and later solutions. However, what we do have to understand is that the perceived problems are only the consequence of a cause.

(24)

If we could compare it to a disease; having a patient showing symptoms, the doctor is unable to treat the symptoms effectively until he or she actually knows the disease. By performing a root-cause analysis this is exactly what we want to do: we want to treat the disease—the root cause—not its symptoms. If we only solve the perceived problems then we will add the risk of them manifesting themselves again or perhaps another problem is introduced because we did not take into account the root cause.

What is also important to remember is that a technical solution might not always be your best answer to a problem we have in a development environment. If we have a build process that seems to take forever or the lack of visual representations, a technical solution can assist us by other approaches to managing dependencies to speed up a build and construct graphical views based upon some calculation on e.g. the source code. But these are technical issues that are in turn solved by technical solutions.

We did not find a technical solution as an answer to our cases satisfactory, but instead as a means to support and implement solutions that are based on root causes. We wanted to break up the development process into smaller pieces and then focus on deriving requirements for solutions from these pieces: divide and conquer. The pieces are going to be the real problems and how they were analyzed for our two cases is the topic of the following two subsections.

3.1.1

ASP.NET Case [2]

In section 2.3.1 we discussed what we called perceived problems and the first perceived prob-lem (A.PP1) was developers had chosen to have their tests in a separate project, which we viewed as an uncommon approach when we discussed the general properties of a software project in sec-tion 2.2.2. More commonly a test package containing all unit tests is employed, but we found that because of the project type characteristics this was not a desired design. As all the source code files of the project were to be deployed to the test environment and compiled at runtime there when accessed, unit tests were suggested to be contained in a separate project which had a dependency to compile the actual application and performing its tests via this.

As for the perceived problem of having copies of the same source code on both development and test environments (A.PP5) we gathered through interviews with the developers that this was a wanted feature. Developers wanted to be able to make changes to the code directly in the test environment as a way to by-pass the deployment process when only a few quick code changes were needed, even though these changes were not be version-controlled in the shared repository. A.RP1 By having two copies of the source the double maintenance problem is introduced So here we arrive at Wayne Babich’s famous problem of double maintenance, since changes made in the test environment will diverge from the original code base and will be overwritten at the next deployment if all changes are not remembered to be copied back to the development environment. We derived this behavior to be related to the automation process of the deployment, since it was deemed as inefficient (it would taker longer time) to perform a new deploy for just a small change in code.

Moving on, we also had the perceived problem that tests are written by the same developer that wrote the code to be tested (A.PP2). Through our interviews with the developers of the ASP.NET project we found that one of the two developers only wrote unit tests for his own code, testing that e.g. his functions worked the way they were intended to.

The second developer had another approach where he only wrote API tests similar to the practice of black-box testing. With many years of experience this developer was convinced that these types of test had a tendency to capture more failures than unit testing. So there were some differences in testing techniques among the two team members and they did not adopt the understanding of collective code ownership as they only cared about their own test project not failing.

(25)

3.1 Discover the Real Problems [5] 15

For example, test runs were made at the developers own discretion, meaning that there was no process of pre-commit testing or testing after each commit to the repository. Taking into account that the developers mainly only ran their own test suite one developer could break the other developer’s tests quite easily.

A.RP2 No certainty that tests are run after or before each commit/push, which means feedback for a longer history of changes

Another possible problem we noticed when doing our preliminary analysis in section 2.3 was that the test environment differed from the production environment quite a bit (A.PP3). To clarify the test environment is the server environment used as the deployment target during development, while the production environment is the actual server at the customer site.

E.g. the test environment was hosted at a Windows XP machine, while the server at the customer used a server operating system, Windows Server 2008. We tried to work out the root cause for this difference in setup, because if we had an identical configurations for both test and production environment we would have the same “parameters” for the runtime compilation.

However, from our interviews we gathered that the choice of using an old Windows XP machine boiled down to economical reasons. They already had this XP machine in place, and buying a Windows Server license just to assert that the application would without doubt work on the customer site was not considered valuable enough or economically feasible.

A.RP3 Proof in test environment that the application works does not prove that it works in production environment

From this we arrive at yet another perceived problem, namely that builds are performed first in the development environment (i.e. via the Visual Studio IDE) and then again on the test/production environments at runtime (A.PP4). This is of course a common approach within script-like pro-gramming languages, but it raises some concern as the development environment may differ in configurations compared to test/production environments. Perhaps the code is dependent on some special version of the ASP.NET runtime compiler or framework, and when the application code is deployed the lack of this prerequisite renders in failure.

A.RP4 Since a build is performed at the production environment after a deployment, there is no guarantee that it will work and failures are caught later rather than sooner

We also observed the perceived problem that tests on the client platforms which call the server can fail if a “rebuild” of the server application fails (A.PP6). What we mean by a rebuild is that whenever source code files change—e.g. due to a deploy or manual changes on the server—the changed files are built again upon access. Should there be “faulty” code on the server or if the configuration (e.g. IIS version) does not comply with the code, the rebuild will fail and client applications are unable to run their tests.

A.RP5 Future versions of IIS might result in failed builds in production environment

At last we noticed some issues for concern relating to database management. We did not really find the time to pursue this concern properly, but the perceived problem was that changes to the database are not always recorded (A.PP7). Since an arbitrarily chosen version of the application code is dependent on a specific version of the database, the application can not run if it can not find the desired database entries or tables, etc. So we have no rollback strategy since the database is not version-controlled in any specific manner, but only modified directly on the test environment. A.RP6 The application cannot be rollbacked to an earlier version if no database version for that

application version exists

To conclude this root cause analysis of the studied ASP.NET project, we mainly found problems relating to the design with having source code files as the deployment artifacts. We discovered for example double maintenance and uncertainty of build status at another platform. Issues like this are likely to be visible in e.g. other web projects utilizing runtime compilation too.

(26)

Real problems for ASP.NET project:

A.RP1 By having two copies of the source the double maintenance problem is introduced A.RP2 No certainty that tests are run after or before each commit/push, which means

feedback for a longer history of changes

A.RP3 Proof in test environment that the application works does not prove that it works in production environment

A.RP4 Since a build is performed at the production environment after a deployment, there is no guarantee that it will work and failures are caught later rather than sooner A.RP5 Future versions of IIS might result in failed builds in production environment A.RP6 The application cannot be rollbacked to an earlier version if no database version for

that application version exists

3.1.2

Android Case [2]

The same type of analysis as we saw in previous section will also be conducted for the Android case. Starting with the first perceived problem derived in section 2.3 we had discovered that the repository of the Android project contained configuration files that were specific to the workspace of the developer. These configuration files are automatically generated by Eclipse, which was the choice of development environment in the project.

By further interviewing the developer and performing a quick study of the Android documentation for managing projects [and] we learned that Eclipse had been chosen because of the fact that it contains all that the developer required in one packaged deal. It is also recommended in the Android documentation and the development kit can be directly integrated. Going back to the perceived problem, when trying to check-out the project ourselves failures were generated because of non existing, dependent libraries on a location that corresponded to the file system of the developer (B.PP2).

If the project had consisted of more than just one developer we believe that this problem would never had manifested in this way. It is fair to assume that the configuration files for a workspace would not contain exactly the same information on two workspaces and thus they would be varia-tion to each other (B.PP1). As a configuravaria-tion item, having them tracked in version control would result in an overwrite on a pull or update. In our particular context the cause of the perceived problem is because only one developer is assigned to the project.

B.RP1 Single person project

We also had the condition with the dependent libraries (B.PP2). Having the project set up this way such as every additional developer are required to download the libraries from some location, make sure it is the same version and manually configure Eclipse so it knows where the libraries are installed do we not consider efficient. We would therefore like to add to the previous analysis that the project is not reproducible, i.e. it would not possible to clone the project to a new workspace without any manual configuration.

B.RP2 The project is not reproducible

The third problem that was perceived for the Android case in section 2.3 was that all of the unit tests was kept in a separate Eclipse project and a separate repository (B.PP3). According to the Android documentation it is a standard convention that all test code is separated from the production code. It is, however, not explicitly stated that test code should be in its own

(27)

3.1 Discover the Real Problems [5] 17

repository, in fact it is recommended to place all tests in a sub directory under the main directory of the application.

The only explanation we were able to discover was that the developer was used to working this way, but it still has its consequences. Again, should additional developers be assigned to the project we risk that the “test repository” are not synchronized as often as it should which leads to the incorrect number of executed tests. By analyzing the log entries of the repository we could also detect that the test directory received very little attention, i.e. few tests. Our conclusion was that the perceived problem originates from an ad-hoc project setup and it gives us a problem that failures are not captured during implementation of a feature.

B.RP3 Failures are captured later rather than sooner

The fourth and fifth perceived problems both have a connection to a production environment. The former stated that such an environment is required for all test executions (B.PP4) and the latter showed us that verification of graphical interfaces are performed manually on unscheduled occasions (B.PP5). We performed some experiments with both simulated and actual hardware to gain a deeper understanding of the variables included in software testing for Android applications. We performed the experiments with the same software project we analyzed so to avoid having worrying about scale.

What was almost immediately discovered was that during test executing, the majority of total time was spent on deploying the application and the test code to the hardware. On a simulated device it took about twice the time for a deployment. It was also very demanding for the device we performed the experiments on, granted it was not the latest super computer, but it still showed a significant increase in hardware usage.

It is possible to prevent a deployment for certain test executions as long as they do not utilize any functionality restricted to Android devices and its API. Since this was not the case in the project we analyzed the cause for the fourth perceived problem is a choice made by the developer whom only has interest in testing functionality directly on the device. With further interviews we were able to discover that the reason for manual verification of the graphical interface was because of no knowledge of a tool that could perform a similar visual verification as the human eye.

The fact that production environment is always required and that new tests are scarce has led to a lack of management for older tests as well. It is not considered efficient to get a test to work correctly after a change or overhaul has been implemented when we “know” that the application works. The long wait time for packaging and deploying the application becomes unacceptable if it is to be performed several times a day or during fault correction. Without any regression we have no guarantee that changes introduce undetectable defects.

B.RP4 A change can introduce undetected failures in production environment

To sum up the analysis of the Android case we are dealing with a project that has most of its problems related to how it is set up, i.e. its structure of files and how tests are managed. All that we discovered might have a strong connection to the fact that only one developer was assigned to the project, together with no company standards. It is a typical example of an ad-hoc project, which do not always have to be regarded in a negative perspective.

Real problems for Android project: B.RP1 Single person project

B.RP2 The project is not reproducible

B.RP3 Failures are captured later rather than sooner

(28)

3.1.3

Commonalities

With the two cases analyzed we can try to abstract the root causes by first identifying any common-alities. If we focus on other activities that are part of development except only implementation, our analysis has revealed that testing the software is considered an inefficient usage of time and is instead pushed back to the final stages of a project. Granted some testing is done during de-velopment as well, but in such small quantities it cannot be deemed as part of the dede-velopment process.

This gives us the common factor in both cases that a failure may be captured weeks, or even months, after a functionality has been implemented or changed. Preferable is to capture and correct errors in functionality when they are still fresh in memory.

On the surface the two projects were set up quite differently (mostly because variations in the target platform) but the cause to their issues were not. With the lack of company standards the developers had organized the applications based on early projects with similar resources and customized for these specific cases. Evidence to support our claim was discovered in the repository of the first case. It contained old scripts for deployments which almost looked identical to those that were used in this project.

All of the discovered commonalities for root causes and real problems from the analyzed cases: • No company standard or templates for a development process.

• No seen value in testing during agile development.

• Feedback on implementation is only given at the final stages of a project. • Repetitive procedures are performed manually.

By constructing an abstract view of the real problems in both cases we can derive requirements for solutions that are not solely targeting a specific case or project context, but instead tries to target the whole domain. We realize that with further analysis of more cases on the company would have been preferable and perhaps required. Unfortunately we did not have the required time.

3.2

Define Requirements [2]

Having discussed how we performed our in-depth analysis to discover the causes to the perceived problems we can now conclude its result into a set of requirements that we place on a solution. The requirements has to cover the real problems we are facing and take the problem domain into consideration.

As a first requirement on our solutions we will have to accept the fact that having a too complex solution will lead to it not being used or maintained. This was the case with the previous build environments that had been as an experiment in company. Only the ones responsible for that environment had the knowledge to configure it, which eventually resulted in no more maintenance, once they got assignments to work on other companies.

To define what is and what is not considered a complex solution is no small matter. It does, however, put a limit on the amount of available solutions we can have and that are seen as acceptable. We would settle with the definition that if an activity can be inserted into the new system without the knowledge of some other approach that is instead performed in the new system, then the solution is not considered complex.

(29)

3.2 Define Requirements [2] 19

The deploy procedure in the ASP.NET case involved a transfer of a certain directory to the test environment and in the Android case the whole application was transfered as a single packaged file. If we cannot configure the new system by explicitly telling it to transfer a certain directory or file, then it would be considered as complex. This requirement will always exist in the background and can never be considered validated, because of the fact that every future addition to the solution has to consider it.

R1. It must be possible to perform activities with the same approach as is done without the new system.

If we now focus on what we call the commonalities in section 3.1.3, namely the identified root causes shared by both of the studied projects, and attempt to argue for requirements that will solve these behaviors, they will most likely make a greater positive impact on the development as both project types will be affected.

For example, we saw the lack of feedback during agile development. In the ASP.NET project there was no certainty that the tests were run before or after each integration (A.RP2) as well as no verification in a production-like envrionment (A.RP3) and no guarantee that the application will build (A.PR4).

In the Android project we had that the product was not reproducible (B.RP2) and feedback on implementation were given late in the development process (B.RP3). If we are given feedback a long time after from where the functionality is actually written, we argue that finding errors will take more time as the developer has to go through his code again to actually work out what went wrong. To achieve feedback on the code committed to the repository, we need to enforce that builds, as well as tests or code metrics are actually executed.

R2. Support has to exist for the implementation, execution and publication of results for tests and code metric analysis.

Still on the topic of feedback and again relating back to A.RP2 and B.RP3, we would like to extend the previous requirement by adding a second one detailing an acceptable time span between integrated changes and the generation and publication of feedback. Amongst our primary literature we were able to find that it is reasonable to at least accept 60 seconds [HF10] before we can start pulling out our hair in despair. Like our first requirement, this third is also considered as a background requirement.

R3. Feedback on checked in changes should not take more than 60 seconds

Having a test or production-like environment (see for example A.RP3 and B.RP4) is not only limited to the two cases we studied, since almost every project within the company is targeting cell phone platforms. A suitable and abstract requirement would then be one that covers support for multiple environments in a development project. A solution has be able to work with more than environment, thus be aware they exist.

R4. The solution needs to support working with multiple environments and be aware of their existence.

If we are to aid the need for a company standard for setting up and maintaining projects, as was discussed in section 3.1.3, we require a centralized environment on which applications can be built, tested, etc. In the cases we had the existence of two copies of the source code (A.RP1) and a root cause of a single developer spawning issues related to file structure (B.RP1). Without a standard we would have to decide on every project how and where to manage files such as build and deploy scripts, which goes against what we are trying to solve.

R5. We require a centralized node where we can check out or clone from a repository, perform builds, execute tests, i.e. perform the same activities as are done during development. The final requirement we are going to place on our solutions is related to automation. As we discussed in section 3.1, a common behavior is to perform repetitive procedures manually. Example

(30)

of this was discovered as a test related issue (see A.RP2 and B.RP3). In the long run it will be considered inefficient and even to the point where it is tried to be avoided because of the opinion that implementing is much more fun then transferring a couple of files from one location to another. R6. It has to exist support for automating procedures and activities.

Now we have defined a set of six somewhat abstract and general requirements, which from our derivations would help us find practices and design a solution which would be an attempt to im-prove the software development process at Softhouse. We want a non-complex, multi-environment oriented system for which the user (developer) does not need to adapt to very much. Quick feed-back should be considered essential and therefor we need a centralized system which can clone a repository, perform a build, run tests, publish results from tests and code metric analysis—all within 60 seconds. Last, but not least, again automation is key and we need support in our solution for automating repetitive and manual processes.

Figure 3.2: The connection between the real problems and the requirements.

In the next chapter we will use the discussed requirements to limit our scope when attempting to choose practices and techniques to use in our solution. The requirements should be matched are to be matched against, maybe not explicitly, but at least the general concept, against the promises of the reviewed practice. Should we find practices that suit our domain and takes care of our root causes to our problems we should be golden, and can start designing a solution in chapter 5 which we can then validate and evaluate against the stated requirements for assurance.

(31)

Chapter 4

The Cornerstone [4]

Having gone through the process of identifying the problem domain followed by a root-cause analysis on the perceived problems that were discovered during an initial analysis and interviews we can now begin to start thinking about solutions. We have our real problems and requirements for solutions from the previous chapter, but now we require methods and approaches that are theoretical in nature which will then be the basis for the actual solutions, which are more technical. This chapter will entail the approaches we found during our research and how they relate to the problem domain we were facing. We will define any requirements and constraints placed on a development team when improving the development process as well as argue the advantages and drawbacks for the approaches we have chosen. The chapter can be seen as an introduction to designing solutions.

4.1

Software Development Maturity [1.5]

Being able to perform something effectively means that we have the capability to successfully accomplish goals that has been set out for us. It might not be a efficient solution, which refers to the notion of making sure to minimize required time and effort during our process, but it is at least a complete solution. Questions raised are now related to how we can be certain that goals are accomplished, i.e. how can we validate our solution? The goals might also not always be laid out for us, so we would then be first required to define them and again how do we accomplish that?

During our research we came across a model called the Capability Maturity Model (CMM) [Bur03, PCCW93], which uses a concept called software development maturity. Now the whole idea be-hind software development maturity and the CMM is to assist in software process improvement by defining both goals but also procedures to validate how mature a process is, i.e. the level of maturity. For the interested reader there is a whole book on the CMM with much further detail [PWCC95].

In our case we will relate to our own context and problem domain when defining software de-velopment maturity. What we require is first and foremost a means to effectively measure the processes we have analyzed to determine any anomalies from standards and if a solution actually was an improvement. The purpose is to create awareness of how much or how little a process has developed. With the analysis on the domain that was made in chapter 2 half our work is already done for us, now we just need to value our findings.

The cost of an implementation of a software process improvement can vary depending on what area of the development process that is targeted and so can the benefits. We can, for instance,

(32)

have a solution that requires a week of training before it can be efficiently used by the developers. Since these variations exist it would make the lives of the software process improvement personnel easier, as well as the developers if areas in the development process which require the least effort to improve that also give high benefit could be identified. The same goes for the vice versa, which we will call bottlenecks.

The final piece to the puzzle of our definition of software development maturity is the ability to produce goals which will serve as the main purpose of reaching higher levels of maturity. With information gathered for the factors in the previous paragraphs, we now have the possibility to set the goals for process improvement. The main purpose of having maturity goals is for, again, validation.

We have discussed a total of four different purposes of our definition of software development maturity that are outlined in the following list.

• Measure how far a process has deviated from standards

• Determine key areas that requires the least effort for improvement • Identify bottlenecks

• Set goals for process improvement

The previously mentioned topics had one major thing in common and that was validation. The next topic we would like to discuss is therefore how we can effectively perform a validation on a software process improvement. But before we do that, a question had certainly been raised as to why all this matters and what a company can gain by improvement through software development maturity.

First of all we have the, potentially obvious, factor of increasing efficiency. In one of our cases we had a situation were a project was setup ad-hoc with dependent libraries only on a local location. Now this was fine until the point came when another developer joined and required a clone of the repository on his own machine. Since he was not present from the start the first developer had to assist in setting it up which halted his work. This might be seen only as a common mistake, but the fact is that without any standards in project setup these kinds of problems will repeat themselves.

Standards and template procedures are therefore wanted in our context and we would argue that it is another benefit given by a maturity model. As we will see in the next chapter, designing solutions for software processes requires some level of abstraction and that is where templates come in. These standards are defined based upon the business goals of the company. In fact, having a maturity model specifically designed for the company domain will thus make sure that the company goals are, drum roll please, validated.

So far we have discussed how maturity models work to increase efficiency in development, validating company goals and turning ad-hoc processes into standards or templates. There is one more point we think is worth mentioning and that is awareness. Now it is not about general awareness and to know when to look over your shoulder, but awareness of the development process and the tools it uses. Producing and maintaining templates and maturity goals forces us to study and ponder on what before was taken for granted.

What is left is of course a practice or approach that will detail how to reach higher levels of maturity. As we see it, there are no general cases for this. Even though the CMM has been around for a while it cannot cover all types of company, process and project contexts that exists. Before starting design for solutions we first have to outline a methodology they will be based upon. Considering our domain, we have a requirement of continuously integrating and deploying.

(33)

4.2 Continuous Delivery [2] 23

4.2

Continuous Delivery [2]

Before moving onwards we should again reflect on the results from the previous chapter. What is shared between the two analyzed cases is that they both require a production-like environment during development to run specific tests. We have seen that both cases struggled with issues related to capturing failures as fast as possible and no regressions testing can lead to undetectable faults. The Android project also suffered some issues related to reproducibility and the ASP.NET project lacks guarantees that the system will successfully build.

Since our project context involves smaller development teams we do not need to worry about any constraints on the network load if more information should be passed after each code check-in. We do however need to take into consideration that we do not have an elite special force of testers and thus that burden is places on the developers. We are dealing with an agile development team, which means that it is feasible to assume that change requests from the customer is somewhat common.

What we would like to have on our recipe for designs is an approach that gives us the benefits outlined in the following list.

• Fast feedback on changes

• Always make sure that the code has passed tests and quality checks

• Larger scope of history, i.e. more traceability, then just changes to the source code • Repetitive procedures are automated

A concept that fulfills the preceding list, that we were able to find, is called continuous deliv-ery [HF10]. This section will discuss how the different parts of continuous delivdeliv-ery will assist in designing solutions, which is performed in chapter 5.

4.2.1

Continuous Integration [0.5]

Having already established that continuous delivery is a practice which may suite our domain, following it places a requirement on developers. If we want to have the possibility to capture failures early and increase our traceability we need the increments to be small or else it will take too much time and we lose our fast feedback. Relating back to the definition of our contexts (see section 2.2.1) we mentioned that integration to the shared repository was performed a few times per day. Thus in our case, we already had evidence of continuously integrating small changes to the repository with an up-to-date code base.

If we look to the real problems discovered in section 3.1 we found that for both the ASP.NET and Android projects there was no way of knowing if the project compiled and pas

Reaching Software Development Maturity with Continuous Delivery. Fredrik Stål Viktor Ekholm Faculty of Engineering, Lund University