Towards continuous delivery in system integration projects : introducing a strategy to achieve continuous delivery and test automation with FitNesse

(1)

Towards Continuous Delivery

in System Integration Projects

Introducing a Strategy to Achieve Continuous

Delivery and Test Automation with FitNesse

Sandra Drenthen

(2)

2

Master's Thesis

Towards Continuous Delivery in

System Integration Projects

Introducing a Strategy to Achieve Continuous

Delivery and Test Automation with FitNesse

Author

Name S. Drenthen BSc

Study Master Student Computer Science, Software Engineering Department of Computer Science

Software Engineering group

University of Twente, Enschede, The Netherlands Student Number s0146110

E-mail s.drenthen@alumnus.utwente.nl

Graduation Committee

First Supervisor Dr. M.I.A. Stoelinga Associate Professor

Department of Computer Science Formal Methods and Tools group

University of Twente, Enschede, The Netherlands Second Supervisor C.M. Bockisch

Assistant Professor

Department of Computer Science Formal Methods and Tools group

University of Twente, Enschede, The Netherlands Everett Supervisor Drs. René Mulder

(3)

3

Preface

This master thesis is the result of the final research project for the Computer Science master at the University of Twente. This project was performed externally at Everett. During the research project, I have learned a lot about continuous delivery, test automation, research methodologies and the process of conducting research. Furthermore, since the project was performed externally, I have experienced Everett's projects which enabled me to experience the practical field of computer science as well as the theoretical field.

First of all, I would also like to thank my supervisors Mariëlle, Pascal (during the first months), Christoph (during the last months), and René. I had the privilege to meet them all together every month, and my Everett supervisor René even every week. During these meetings, they provided feedback on my work, asked critical questions, guided me in the right direction and René also helped me with the practice. I believe their commitment in the guidance really did improve my work. Furthermore, I would like to thank Everett for making this research project possible and the (anonymous) client of Everett for enabling me to apply the designed strategy at their project. I would like to thank the colleagues of both companies for making me feel welcome and part of the team, for all the fun we had and for their interesting insights on the project. Special thanks to the colleagues who invested their time to apply and evaluate my strategy in the case study and to my carpool-buddy for the fun trips, driving me almost every day to the case study and back.

Finally, I would like to thank my friends and family who supported me during the whole process, especially when I wasn't able to see them very often. They were always there for me: they helped me moving, they coped with my busy agenda, they cheered me up when needed, they gave me give good advice and they provided a lot of fun in the evenings and weekends. Special thanks for Pim for reviewing my thesis during his busy schedule, giving useful insights and improvements.

I hope that you like reading this thesis and that Everett and others can benefit from the results.

(4)

4

Abstract

This thesis presents a strategy to introduce continuous delivery and test automation with FitNesse in system integration projects. It was designed for Everett's identity solutions projects, but is expected to be mostly applicable for other system integration projects as well.

The strategy consists of tools, tutorials, approaches and guidelines:

 NetBeans, Git, Ant, JUnit, FitNesse and Jenkins were chosen as tools, used to achieve continuous delivery and test automation.

 Tutorials were written for FitNesse, Jenkins and a solution to automatically push and pull to Git.

 Several approaches and guidelines were chosen and tailored to Everett's projects and showed how to incorporate test driven development, how to decide which tests should be automated, how FitNesse and Jenkins needs to be configured and used, how to cope with test data changes and how to communicate with the system under test.

The strategy was constructed by first creating a high level strategy, which was then applied, evaluated, supplemented and improved during a case study at a client of Everett. During the case study, ten automated tests were made by the team-members of the project and the continuous delivery cycle was set up. Evaluations were held with team-members of the case study project, which showed that the strategy did need some time investments in creating the strategy and tests, but can deliver an added value as it is expected to reduce faults, increase the quality, enhance trust from the client and possibly in the long run, also save time. However, in order to show this with hard facts and significance, the strategy should be applied once again where the strategy is complete in the beginning and more tests are written during the whole project.

(5)

5

1 Introduction

1.1 Problem Statement

Everett's test strategy for their system integration projects, called identity solutions was ad-hoc and based on much manual work, resulting in high costs for testing and sometimes finding bugs in a late stadium (e.g., at the last sprints of the project). Everett wanted to reduce both the costs of testing and the amount of delivered faults by introducing test automation in their projects with the use of FitNesse. Furthermore, Everett wanted to automate and improve the process of software delivery further by introducing the continuous delivery pattern in their projects as well.

In order to set up the continuous delivery framework, several foundations are needed: good configuration management, automated build and deploy scripts, automated tests and a continuous delivery framework to manage the steps in the deployment pattern. In Everett's case, the introduction of test automation and a continuous delivery framework were the last two steps that needed to be taken in order to introduce continuous delivery in their projects.

Everett was founded in 1999 and has nearly 80 employees throughout the Netherlands, Italy and the United Kingdom [1]. Everett's mission is to help organizations around the world to be successful with identity solutions through consulting, system integration, and support services.

System integration projects are projects where different computing systems and software applications are linked together, physically or functionally, to act as a coordinated whole [2].

Identity solutions [3] is a name that Everett gives to their defined set of several solution areas[3]. Identity solutions projects are system integration projects which revolve around the scalable and timely management of users and their access to information and applications. These projects characterize themselves as consisting of highly configured and customized third party software and being highly integrated, data-driven and short term (10-15 weeks). Identity solutions projects are performed at different clients at several sectors and may be

implemented with various techniques and third party products. Due to these characterizations, the projects are hard to test. The aspect that the projects are short term and different third party software is used for different projects gives less time and less reuse in the target of achieving a return on investment from test automation. See section 2.2 for more information on identity solutions. 1.2 Goal

(8)

8 The designed strategy for test automation and continuous delivery defines the test and development process and consists of tool-selections, tutorials,

approaches and guidelines. The strategy was documented in a wiki.

The requirements and evaluation criteria for the designed strategy were defined as follows:

 Efficacy - It must tackle the problem in the problem scope (i.e., enable parts of continuous delivery)

 Flexibility - It must be useable in different projects with different circumstances.

 Implementation time - It must be easy and fast to install, learn and use.

 Cost-effectiveness - It must have an early return on investment.

 Transferability - It must enable the client to keep using and maintaining the tool.

1.3 Research Questions

The main research question of this research project has been:

 How can continuous delivery and test automation with FitNesse be introduced in system integration projects?

This question was divided into the following underlining questions, which needed to be answered in order to provide an answer to the main research question:

1. What is a good1_{strategy to introduce continuous delivery and test}

automation with FitNesse?

a. Which guidelines and tools are used in this strategy?

b. How will these guidelines and tools be tailored to system integration projects and to each other?

2. What is are the costs and benefits of applying the strategy?

a. Which effects has the application of the strategy on a project?

b. Is the strategy an improvement compared to the test- and development strategy used in earlier projects?

c. Where is the break-even point to recoup the effort of applying this strategy?

i. How does this differ in several factors of the projects (e.g. different test types, different features, different projects and different software)?

d. To which extent is the strategy applicable for other system integration projects?

1.4 Research Method

The first research question (i.e., what is a good strategy) has been answered by creating a high level strategy for test automation and continuous delivery based on literature by selecting promising combinations of methods, guidelines and tools and creating tutorials when needed. This strategy is described in chapter 5.

(9)

9 When the high level strategy was created, it was applied, evaluated,

supplemented and improved during a case study which is discussed below. The second research question (i.e., what are the costs and benefits) has been answered by evaluating the strategy in a case study. The case study approach allows to investigate the validity of the strategy in the real-life context of a identity solutions project in a qualitative manner. The case study followed an iterative research pattern, which has intermediate evaluations and strategy changes in order to design the strategy in steps and improve it along the way. The case study has been performed in five iterations, each during two weeks. Data was gathered from documents, observations, surveys and personal interviews. The validity of the strategy is determined by the outcome of the evaluations and personal interviews.

A more detailed research method description is given in chapter 3. 1.5 Scope

Three scopes have been defined in order to have some control on the size and the time needed to carry out the research project. First of all, his research used FitNesse as test automation tool, which was selected in earlier research as a promising tool to be used with Everett's projects. Furthermore, the strategy has been created for and applied on Everett's Identity & Access Governance projects, and even more specific, Identity & Access Governance projects using SailPoint IdentityIQ software. However this was a specific project and software choice, the goal was to be able to use (most of) the strategy for other projects and software as well.

1.5.1 Test Tool: FitNesse

In earlier research[4], several test automation tools have been compared and judged on the degree of how well they meet the requirements mentioned in section 1.2. The investigated tools were web browser automation tools like Selenium [5] and Watir [6] and test automation tools FitNesse [7], GreenPepper [8], Cucumber [9] and Root Framework [10]. After evaluating these tools (see section 2.5), FitNesse was selected as the most suitable test tool for Everett's purposes. FitNesse is an acceptance testing framework that is lightweight, open source and easy to use. See section 4.6 for more information on FitNesse.

1.5.2 Solution area: Identity & Access Governance

The strategy has been created and applied on the solution area of Identity & Access Governance projects (see section 2.2.2). This solution area has been chosen because Everett experienced a high demand for these projects, making it more likely that such a project would be available for the case study than with other solution areas. Furthermore, Identity & Access Governance projects are short term in particular (10 to 15 weeks).

1.5.3 Identity & Access Governance Software: SailPoint IdentityIQ

(10)

10 other products as well. IdentityIQ from SailPoint [11] was chosen as this representative product, as it is a well-known vendor software for Identity & Access Governance, and it is widely used at the time of this research project. SailPoint IdentityIQ is a governance-based identity and access management suite [12]. It offers an identity warehouse where the identities are stored in a central repository, a role model, a policy model, an advanced risk model [13] and a workflow engine. Together, it enables the client to, for instance, create and manage roles, automate access certifications, automate changes based on lifecycle events (i.e., hiring, transferring, leaving), calculate access risks and define, detect and enforce policies. During the project, SailPoint IdentityIQ) becomes the center of all the main applications at the client's business for identity and access related information (see figure 1 on page 12).

1.6 Outline

This thesis is divided in 7 chapters, starting with this chapter which provides an introduction to the thesis. Chapter 2 gives background information of an earlier research project, providing preliminary knowledge for this thesis. Then, in chapter 3, the research method is described in depth. Next, a literature review is given in chapter 4, discussing important concepts and giving the proper

(11)

11

2 Background

This chapter is based on the author's earlier study, performed during the course 'Research Topics' [4], and offers background information for this thesis. The earlier study has been performed by a literature study, an observing case study at a project of Everett (including observations and personal interviews) and an analysis between theory and Everett's practice.

2.1 Chapter Summary

Everett's identity solutions projects are system integration projects that revolve around the scalable and timely management of users and their access to

information and applications [3]. These projects are characterized as consisting of highly configured and customized third party software and being highly integrated, data-driven and short term (10-15 weeks) [4]. Identity solutions projects are performed at different clients at several sectors and may be implemented with various techniques and third party products [14].

The current system development process of Everett is based on the Scrum and Prince2 methodologies [15]. Everett already has Continuous Integration

(revision control using GIT) and automated builds (using Maven or Ant) in place. Furthermore, Everett uses the Atlassian Software Stack (Bitbucket, JIRA,

GreenHopper and Confluence) for revision control management, issue management, scrum project management and documentation [4].

The current test strategy of Everett consists of mostly manually and ad-hoc testing; tests are risk-driven; there are no formal test scenario's (except for user acceptance tests), negative test cases are seldom tested, testing is often

performed with production data and there are no special test tools used in the project [4].

An analysis between the current strategy and the literature resulted in a set of improvement points [4]:

 Everett should be more test-oriented.

 A general test-strategy should be created.

 The handling of the system's high level of integration should be improved.

 Test automation with FitNesse should be introduced.

 Fictional data should be used instead of production data.

Based on the requirements given in section 1.2, FitNesse has been selected as a promising test automation tool for Everett's projects [4].

2.2 Identity Solutions Projects

Identity solutions revolves around the scalable and timely management of users and their access to information and applications. Everett delivers solutions with which organizations have, even across organizational boundaries, means to[3]:

 Reduce the operational and development costs of IT.

 Increase security.

(12)

12

 Simplify business processes.

[image:12.595.152.440.112.286.2]

 Personalize services.

Figure 1: An architectural overview of an identity solutions project

Figure 1 shows an example of the architectural overview of an identity solutions project: An identity solutions system is introduced, configured and connected with several applications of the client and an authoritative source for identities (for instance, a human resource application).

The identity solutions system can perform several actions with identities, their access to information and source applications. The result is integrated management and use of identity information regarding employees, partners, suppliers and other stakeholders to support a complete service chain. Examples of identity solutions projects can be found from section 2.2.1 to 2.2.6.

Identity Solutions projects (ISP) are often realized in a short period of time (10 to 15 weeks). Everett consults, designs, implements and supports [16] identity solutions for their clients and uses various products and technologies in order to create the best fit for their clients requirements. The various products that act as an identity solutions system include products and suites from vendors like ForgeRock, iWelcome, Microsoft, NetIQ, Oracle, RM5, SailPoint and various open source products [14].

Identity solutions is a name that Everett gives to their defined set of several solution areas [3]. These solution areas are defined as groups of functionalities which are present in identity solutions projects. Multiple solution areas can be combined in one identity solutions project and there is some overlap between them. The following solution areas are defined:

 Identity management

 Identity & access governance

 Access management

 Identity federation

 Identity cloud solutions

 Authentication

(13)

13 2.2.1 Identity Management

Identity management [17] revolves around managing the lifecycle of identities and the subsequent relevant impact on their access to applications and services. It enables the company to give a new user quickly and automatically access to systems that match his relation to the company, using the same account for each system. Identity management can include self-service (manager can assign rights to its employees through an access request portal) and provisioning (automatic process that executes the changes in access rights the ICT landscape).

Example When the role of the employee changes (e.g., via promotion), he automatically gets the correct access for his new role. This includes withdrawing the access he had for his previous role but does not need for his new role. Furthermore, when the employee leaves the company, all access rights are automatically removed. Secondly, the manager can inspect some company data (name, personnel number and function) and access data of the employees that he supervises.

2.2.2 Identity & Access Governance

Identity & Access Governance [18] revolves around being in control of access rights to the information systems and the ability to demonstrate and prove it. It enables companies to comply with the regulations on access control set by regulatory bodies and gives these companies the ability to prove it as well.

Example Identity & access governance can enable companies to comply with the report from the Basil Committee on Banking Supervision [19], which states that the e-banking security process should include, among others, sufficient logical controls and monitoring processes to prevent unauthorized internal and external access to e-banking applications and databases.

In order to be in control of access rights and to be able to prove it, certification-cycles and reports are performed and generated periodically (for instance every quarter). In the certification-cycle, access rights of employees and accounts are verified by, for instance, the employee's manager or the system owner. During this verification, the manager can accept, revoke or redirect de decision to another employee. If the right is revoked, the identity solution system will take the appropriate actions: either by directly revoking the access via provisioning, by entering a ticket to the associated ticketing system or by sending an email to a specified address with the message to change the access right. Reports can be generated to provide an overview of information, for instance, describing which identities had access to which systems and information in a specified period of time.

2.2.3 Access Management

(14)

14 sign on. Access Management relies on authentication solutions to validate the identity of the users and relies on Identity & Access Governance solutions to determine authorizations.

Example When a user wants access to information (e.g. log in the system with customer data), access management techniques identifies the user (for instance by asking a username and password), and checks whether the user has the right authorizations to access the information (by verifying the credentials). Finally, the user gets access to the information (e.g. gets access to the system with customer data) when both steps are performed successfully. When the user does not have the correct rights, an error-message will be shown.

2.2.4 Identity Federation

Identity Federation [21] revolves around all processes and underlying technology which makes it possible to exchange identity data across organizational boundaries in a secure and controlled manner. It implements a comprehensive and robust architecture for Identity Federation, establishing a solution for integrated services in a supply chain.

Example Identity Federation enables secure collaboration across organizational boundaries, giving parties access to each other services. This can be between, for instance, suppliers and buyers: the buyers can access the stock information of suppliers and place orders directly.

2.2.5 Identity Cloud Solutions

Identity Cloud Solutions [22] revolves around integrating cloud applications, enabling one-click access to all applications, anywhere, anytime and from any device, while keeping identities safe. It delivers a single point of access to the company's public and private applications.

Example Identity Cloud Solutions enables companies to integrate and manage identities for cloud applications like Google, Salesfore and Office365 with their internal applications, letting their users access these cloud applications via the same portal as the company's internal applications, possibly with single sign on as well.

2.2.6 Authentication

(15)

15 Example Authentication can verify a user by something that someone knows (e.g., a password or PIN code), something that someone possesses (e.g., a key, token or phone) or something that someone is (e.g., a fingerprint-scan or iris-scan) or a combination of these. With these authentication techniques, a user can ensure his identity in order to get the access rights that are assigned to this identity.

2.3 Current System Development Strategy

Everett develops according to the Scrum methodology [24]. Scrum works with iterations called sprints, typically lasting between two and four weeks. A Sprint is a fixed period of time for developing functionality as part of a product release (final product). Each Sprint will need to deliver some form of business value, adding on to the previous Sprints [15]. Each sprint starts with a planning and ends with a review. In between, user stories are implemented and tested. A user story is a software system feature specified by the customer in everyday business language. An example of a user story is: "As a customer, I want to be able to login on the site in order to see my purchase history". Working with the Scrum methodology helps the project in continuously delivering working software to the client and keep on moving forward to success.

[image:15.595.92.504.504.722.2]

Everett combines the Scrum methodology with the project management methodology Prince2. Prince2 provides a control and governance framework that aligns with the stakeholders, business and budget owners and their project organization [15]. Figure 2 shows how Everett combined the stages of Prince2 with the Scrum development methodology. Prince2 identifies several stages of the project (shown in yellow), such as directing, startup, initiation, planning and so on. Everett has added scrum (shown in orange) to this diagram, which shows that the product delivery is done with sprints and a daily standup (i.e., the 24h-cycle), that a sprint starts with planning and ends with a demo where after a new sprint starts.

(16)

16 As part of the development environment, Everett uses the Atlassian software stack [25] in their projects; Bitbucket combined with Git [26] for revision control (integrating code and manage the different versions of this code), JIRA and GreenHopper for issue and scrum project management (managing user stories and issues to work on), and Confluence for documentation (wiki).

Furthermore, Everett uses mostly Maven [27] and Ant [28] as automated build tools (automatic compiling and building the projects code into an application that can be run). With these tools, Everett has build scripts that can be run via, for instance, a command line in order to build the project.

2.4 Previous Test Strategy

When using scrum, a project has a definition of done, which specifies the criteria of when a task in the project is considered done. An example/standard definition of done of Everett has three specified test activities, performed by three different parties/individuals:

 Developer: The developer tests his implementation of the user story.

 Review: The implementation of the user story needs to be reviewed by another team member.

 Acceptance testing: When the sprint is finished, the client tests the implementation of the user story, as part of acceptance testing.

Earlier research [4] notices that, except from the statements in the definition of done, there was no clear or standard strategy defined at Everett, so an observing case study was held in order to investigate the test process in practice. In this case study, team members of a project were interviewed on the test process. This project was also an Identity & Access Governance project, also using SailPoint IdentityIQ. The case study showed that:

 Most tests are performed manually without special test tools or test scripts; the developer checks his solution with a couple of debug statements and visually checking the output based on the requirements and the user story where the developer is working on.

 Tests are risk-based, meaning that for standard solutions, only the configurations are checked. For instance, for a standard connection, the hostname, logs and whether or not the data is received are checked visually, but there is no deep test for every detail. Negative (i.e., test cases that should cause an error or failure) test cases probably seldom or not performed by the developers.

(17)

17

 Unit tests are seldom used, since IdentityIQ uses a script language called BeanShell which doesn't support unit tests. However, it is possible to let BeanShell call functions and perform unit tests on those Java-functions.

 Instead of unit tests, integration tests are performed. Ideally end-to-end, although it is almost never possible to obtain full test coverage of the whole software chain. The missing parts of an end-to-end test are then simulated by mocking.

During interviews in an observing case study in earlier research [4] several improvement points have been mentioned as well: According to the Junior Engineer, testing was started too late, which gives a risk that problems are discovered in a late stage where a lot of work may be redone. The project manager stated that it probably would help to add someone to the project that monitors the test process and helps with defining correct test cases. Secondly, the project manager stated that it might help to formulize test cases so they can be reviewed and correlated to requirements and risks, introducing transparency to the client. These test cases are then probably reusable for other projects as well.

2.4.1 Improvement Points

The earlier research project [4] had performed an analysis over the literature combined with the observed strategy, identifying gaps between the applied practice and theory. These gaps were presented as improvements over the current strategy:

 More focus on testing should be created throughout the company in order to improve the test process. This focus is needed in order to investigate and apply the other improvements mentioned below.

 A new test strategy should be created and standardized to improve the test process, making testing more structured, documented and well thought out.

 Decompose highly-integrated systems into manageable and testable units to deal with the system's high level of integration. Mocking should be investigated in detail in order to help with this decomposition.

 Test automation should be introduced. In order to do this, a strategy should be designed that uses FitNesse.

 Techniques like data anonymization and generation should be investigated, including proper tools which can assist in applying these techniques for identity solution projects.

2.5 Test Automation Tool Selection

As said in the introduction, several automation steps are needed in order to set up continuous delivery: configuration management, automated build and deploy scripts, automated tests and a continuous delivery framework. As statded in section 2.3, Everett has the configuration management and automated builds already in place.

(18)

18 Jenkins was chosen during the case study of the thesis as continuous delivery framework (see section 6.2.1). The rest of this section will elaborate on the choice for FitNesse.

There are several tools available for the purpose of test automation, such as web browser automation tools like Selenium [5] and Watir [6] and the test automation tools FitNesse [7], GreenPepper [8], Cucumber [9] and Root Framework [10]. These tools are compared with each other based on the requirements in section 1.2.

A short list of the functionalities and usage of the possible tools is given:

 Web browser automation tools like Selenium and Watir perform tests on the web GUI of the systems under test. These tools want several interactions with the web browser and validations (i.e. the current pane must contain the word "x") as input and give true or false for each validations. Selenium is based on multiple programming languages: Java, C#, Groovy, Perl, Php, Python and Ruby whereas Watir is a Ruby library. Both tools support multiple browsers and Selenium also has the feature to record and playback tests [5, 6].

 FitNesse is an open source wiki and acceptance testing framework, enabling teams to collaboratively define acceptance tests. This tool expects test tables in a wiki format, which interact with Java programs (called fixtures) that communicate with the system under test. When the acceptance tests are run, the results are given in the wiki. It can be used with one of the two frameworks as basis: Framework for integrated tests (FIT) [33] or Simple List Invocation Method (Slim) [34]. FitNesse has plug-ins for Maven and Git and can be connected with the continuous integration tool: Jenkins [7].

 GreenPepper is very similar to FitNesse, although it is not open source and does only supports the FIT framework. It has build-in support to connect it with the Atlassian software stack, Maven and some IDE's [8].

 Cucumber is an open source framework where users can create behavior and scenarios in plain text and write a definition in Ruby that interprets this plain text and calls the system under test [9].

 Robot Framework is an open source framework where users create tests in a tabular test data syntax. It uses keywords provided by the test libraries (written in Python or Java) to interact with the system under test. The results are given in HTML format and XML output [10].

(19)

19

req. →

tools ↓

Efficacy Flexibility Implementatio

n time Cost-effectiveness Transferability

Web browser automatio n tools - tests higher level than needed -- although multiple browsers, only tests web browser apps ++

can record tests, Java

?-

depends on implementation time but has maintenance when GUI changes

+-

open source but maintenance when GUI changes

FitNesse +

can test multiple test levels (e.g., unit, integration and acceptance tests)

wiki to create and show test cases and its results

+

can test all apps that can be called via Java (optionally via other languages with language plug-ins) ++ separation between test definition (wiki tables) and test coding

easy to deploy ? depends on implementation time + open source GreenPep

per + same as FitNesse

+ same as FitNesse

++

mostly same as FitNesse, however: it connects to the Atlassian software stack that Everett uses and does not support the slim framework ? depends on implementation time - closed source

Cucumber +

multiple test levels

+-

can test all apps that can be called via ruby

-

separation between test definition (plain text) and test coding, but need to learn ruby ? depends on implementation time + open source Robot Framewor k +- multiple test levels

reports in html and xml format

+

can test all apps that can be called via python and Java

+- separation between test definition (tabular data syntax) and test coding ? depends on implementation time + open source

[image:19.595.86.512.85.611.2]

(20)

20

3 Research Method

This chapter explains by which method the research questions from section 1.3 have been answered by this research project. The main research question and its underlining research questions are listed here as reminder:

 How can continuous delivery and test automation with FitNesse be introduced in system integration projects?

1. What is a good strategy to introduce continuous delivery and test automation with FitNesse?

2. What is are the costs and benefits of applying the strategy?

The first underlining research question (i.e., what is a good strategy) has been answered by creating a high level strategy for test automation and continuous delivery based on literature by selecting promising combinations of methods, guidelines and tools and creating tutorials when needed. When the high level strategy is created, it will be applied, evaluated, supplemented and improved during a case study which is discussed below.

The second underlining research question (i.e., what are the costs and benefits) has been answered by evaluating the strategy in a case study. The case study approach is chosen to investigate the validity of the strategy in the real-life context of an identity solutions project in a qualitative manner. The case study followed an iterative research pattern, which has intermediate evaluations and changes in order to design the strategy in steps and improve it along the way. The case study is performed in five iterations, each during two weeks. Data is gathered from documents, observations, surveys and personal interviews. 3.2 High Level Strategy Design

The high level strategy design has been created prior to the case study in order to have a well-thought-out solution basis at the beginning. Much details were then yet to be determined. This high level strategy consist of tool-selections, tutorials approaches and guidelines which are documented in informative documents. The high level strategy design is listed in chapter 5. The strategy has been given more details and is adjusted and improved during the case study. 3.3 Case Study

The case study's project needed to be a project at a client of Everett and consists of implementing, testing, evaluating and improving the strategy design.

The case study's main goal is to determine the validity of the designed strategy with as secondary goal to add details, improve and adjust the strategy. With the case study approach, the validity of the strategy has been investigated in the real-life context of an identity solutions project in a qualitative manner. Data is gathered from observations, surveys, interviews and documents.

(21)

21 phenomenon in its real-life context, and for answering how and why questions when the investigator has little control over the events.

3.3.1 The Case Study's Project

The case study's project has been performed at a client of Everett where an Identity & Access Governance project takes place. This project uses the SailPoint IdentityIQ software and was the second phase of a project at this client.

The first phase introduced SailPoint IdentityIQ at the client, migrated the

management of role models from Excel to IdentityIQ, integrated six applications with IdentityIQ, and finally introduced periodic certifications, periodic

verifications of access rights and periodic reports on risks and policy violations. The second phase, which is project of the case study, will extend the first phase by using SailPoint IdentityIQ's LifeCycleManager [37] as an application portal with lifecycle events, introducing provisioning and connecting more applications for certifications. Phase 2 will be conducted with the same team members as in phase 1 with four people of Everett and three people of the client. The project is conducted with sprints of two weeks whereas Everett consults three days in a week.

3.3.2 Case Study Approaches

Iterative Research Pattern

[image:21.595.326.502.396.590.2]

The case study had been used an iterative research pattern as described by Pratt [38]. This model allows the design of the strategy to be constructed in steps, beginning with a basic strategy design which will be applied, evaluated and adjusted and in several iterations. This differs from the non-iterative method, which requires a final and complete strategy before validation. To create such a final and complete strategy at once is difficult to achieve without practical experience in both performing identity solutions projects and in creating and applying test strategies. Because of this, the iterative research pattern suits the case study better.

Figure 3: Iterative research pattern [38]

As shown in figure 3, the iterative research pattern consists of four primary steps with a cyclic relationship: observe the application, identify problems, develop the solution and test the solution. The pattern starts and ends with the observe step. The pattern should be used with several iterations. If not, the pattern degenerates to a waterfall development model [38].

(22)

22 chapter 5). Because of this, the case study has been started halfway of the first iteration at the test-phase.

Interpretive perspective

There are several philosophical perspectives on how to conduct valid qualitative research, including the commonly known positive and interpretive perspective. The positive perspective assumes that that the reality is objectively given and can be gathered objectively, whereas the interpretive perspective assumes that the reality is subjective and can only be gathered through social constructions [39, 40, 41].

The effects of the strategy is not well or at least not completely measurable through only objective data (e.g., the amount of errors found). Therefore, the subjective opinions (e.g., how does a participant experience the strategy) have been considered of great value. resulting in the choice of the interpretive perspective.

Using the interpretive perspective, the strategy is evaluated through the

opinions of the team members (what did they find difficult, what did they want to have improved, what worked well, etc). This gives the opportunity to adjust the strategy during the case study, based on these opinions.

In order to perform the interpretive approach properly, the case study has been following the set of principles for conducting and evaluating interpretive field studies in information systems given by Klein and Myers [42] (see appendix A for their summary of these principles).

Inquiry from the inside

Next to perspectives, there are multiple inquiry modes; inquiry from the inside and inquiry from the outside. With inquiry from the outside, the researcher is totally detached from the organizational setting of the investigation, where with inquiry from the inside, the researcher is personally involved in the investigation [43, 44].

The case study has been performed as an inquiry from the inside, as the researcher has been a part of the team and helped the team implementing the new strategy. It has been chosen above the inquiry from the outside for several reasons. First of all, more experience and knowledge on practical issues and opportunities of the strategy can be gathered with inquiry from the inside, which helps with making more founded choices when adjusting the strategy.

(23)

23 3.3.3 Case Study Detailed Design

Iterations

[image:23.595.110.487.251.589.2]

With the chosen project and approaches, a detailed case study design has been made. First of all, a time scope of the iterations has been defined. Since the project already works with sprint as development cycles, one iteration matches one sprint. In total, five sprints have been used for the case study. It is decided to held five sprints in order to gather a proper amount of feedback, enough room for improvements and a duration that is long enough to be able to perform a final validation, but not longer in order to stay in schedule of the thesis. In total, with an occupation of three days a week, the case study had been performed in thirty workdays, spread over ten weeks.

Figure 4: Case study detailed design

Figure 4 gives an overview on which activities (blue) took place at which moment of the iteration/sprint (yellow), how this is done (green) and by who (orange):

 Explain strategy: The sprint planning has been the ideal time to explain strategy (adjustments), since the sprint planning marks a new sprint and is used as a new start.

 Apply strategy: Naturally, the application of the strategy has been done during the sprint, since then the user stories will be implemented and tested.

 Evaluate strategy: The strategy has been evaluated during the

(24)

24 mood. These evaluations will be hold by presenting everyone a survey, which was filled in on the spot.

In the week after the final iteration, in-depth interviews with the team members have been hold for each team member separately.

 Adjust strategy: The strategy was adjusted before the new sprint, so that changes can be communicated before the new sprint. Because sprints end at Thursdays and start again at Mondays, it leaves the Fridays open to design the adjustments based on the evaluations on Thursdays. Adjustments that took more time were designed during the following sprint and introduced in the sprint after that.

The activities shown in figure 4 matches the iterative pattern of figure 3 on page 21, in a way that is shown in table 2: The application has been observed by observing how the strategy is applied, the problem has been identified by evaluating the strategy, the solution has been developed by adjusting the strategy, then the strategy adjustments has been explained, where after the solution has been tested by applying the adjusted strategy.

Step of the iterative research pattern (figure 3)

Steps of the case study detailed design (figure 4)

[image:24.595.121.463.305.448.2]

Observe the application <-> Apply Strategy Identify the problem <-> Evaluate Strategy Develop the solution <-> Adjust Strategy <inbetween> <-> Explain Strategy Test the solution <-> Apply Strategy

Table 2: matching steps of the iterative research pattern with steps of the case study detailed design

During the sprint, I (i.e., the researcher) was present on location: observing the project, supporting team members with the strategy, designing adjustments and implementing some example test cases.

Surveys

The survey at the end of each iteration has been used to gather information on how the team members experience the new strategy: what is clear, what could be improved, which problems did occur, etc. The main goal of the survey has been to determine which parts of the strategy worked well, which parts were unclear, which parts were needed to be adjusted and what was missing. Secondly, it has been used to identify the costs and benefits of the strategy, including outliers and trends over the iterations.

The survey has been chosen instead of a verbal group setting to let the

participants not influence each other. Furthermore, it has been performed on paper instead of on a digital medium, so it could be filled in directly at the retrospective and it would not have been be postponed or forgotten.

(25)

25 were open questions for those where I couldn't provide pre-defined answers, but the answers could be very valuable when improving the strategy.

The complete survey is enclosed in appendix B. Below states what each question of the survey contributes to the thesis.

 The first question asked about the amount of experience from the team members on Sailpoint IdentityIQ, FitNesse, test-driven development and test automating. The answers on this question can be used in order to determine with how much experience the other questions are answered and possibly show a growth in expertise during the case study.

 The second question asked how well the strategy parts lend their selves to do their task and to what extent the tutorial and explanations were good to follow and apply. These questions are used to evaluate and adjust the strategy during time.

 The third question asked the amount of hours spend on test-activities in order to give rough estimations for the time it costs to apply the strategy

 The fourth question asks whether problems are encountered, how restrictive these problems are and if they are solved (and how). This question is used for strategy improvements in order to determine limitations for the strategy and possible improve the strategy by fixing the problem in a next iteration.

 The fifth question asked how the strategy effects several aspects of the project by comparing these effects between the first phase with the previous strategy and the second phase with the new designed strategy and is mainly used for evaluation purposes.

 The sixth and seventh questions asked which part of the strategy could be reused in which manner and/or to which extent in order to determine the reusability of this strategy in other projects.

 Finally, there was room for comments, tips, ideas, complaints or pitfalls of the strategy.

Interviews

The final interviews after the last sprint have the goal to gather an overall evaluation of the strategy. This evaluation is done by comparing my strategy with the strategy used in the first phase of this project and comparing my strategy with the goals and requirements given in section 1.2. All interviews are recorded, transcribed and send back to the participant in order to be able to verify the statements done in the interviews.

The interview mainly contains open questions, asking the participants to

evaluate the strategy as a whole. The interview gives the opportunity to respond on their answers, asking for more clarification, details and examples. The

interviews helped to achieve a qualitative and complete evaluation of the strategy.

The complete list of interview questions is enclosed in appendix C. Below states what each question of the survey contributes to the thesis.

(26)

26 team member the change to give his opinions without further influences or steering towards an answer. The answers given on this question were a guidance for other questions as well, so the interviewer knows where it could ask for more information or clarification.

 The second part of the interview consists of questions that asked about the costs, benefits, effects and specific evaluations of the strategy. The answers given on this question determines mainly how the strategy performed and is evaluated.

 The third part of the interview consists of questions that asked to which extent the goals of this thesis has been achieved in order to use that for evaluations and future work.

3.4 Evaluation

The strategy was evaluated through analyzing the results of surveys and interviews.

Surveys were summarized by first grouping answers on questions. For the questions that have a scale as answer, the answers are grouped by survey iteration, calculating the averages per iteration (with min and max values) and the overall average (with min and max values), and creating a plot from this, both with individual values and average values. For the open questions, all individual answers are listed, sometimes grouped per iteration.

Interviews were summarized by grouping answers on questions or subjects, listing the opinions of all individuals on that question or subject.

(27)

27

4 Literature Review

This chapter provides theoretical background and the state of the techniques on important concepts of the research project. The following concepts are

discussed:

 Continuous delivery

 Test models

 Test-driven development

 Test automation

 FitNesse

Continuous delivery is used in software development to automate and improve the process of software delivery by using a deployment pipeline that consists of continuously integrating, building, testing and releasing software [45, 46]. Test models describe test activities with correlation to development activities. The v-model shows that testing can (and should) start at the beginning of the project. An improved version of the v-model [47], called the advanced v-model [48], is designed to reflect the relationship between the development activities, test activities and maintenance activities. Another improved version of the v-model, called the w-model [47], is designed to define more clearly when which test activity starts, what the connections are between various test stages, and it shows the link between the tasks of testing, debugging and changing during test phases.

Test-driven development (TDD) is a development and testing practice from the agile software movement where tests are written before coding in small iterations existing of: write a failing test, make the test pass, refactor [49]. Test automation is the use of a mechanism for running test cases without a tester, which can improve the quality of testing but also requires an investment that should repay itself [50, 51].

FitNesse is a test automation tool that is used in the strategy design of this thesis. It is a lightweight, open source framework that enables teams to collaboratively define acceptance tests, run those tests and see the results. FitNesse tests can be used very early in the project and tests can be written by both technical and non-technical stakeholders [7].

4.2 Continuous Delivery

(28)

28 4.2.1 The Deployment Pipeline

The Continuous delivery deployment pipeline [45, 46] is shown in figure 5:

 First, the delivery team delivers commits to the version control, integrating code in one central place.

 The update in the version control software triggers the build and unit tests, which verifies that the system compiles and passes a suite of automated unit tests.

 When the build and unit tests pass, a series of automated acceptance tests are triggered, which tests whether the system works at the functional and nonfunctional level.

 When the automated acceptance tests also succeed, the manual user acceptance tests can start, testing whether or not the system is usable and fulfills its requirements.

 When the manual tests succeed, the delivery team will get this success as feedback and the software can be released.

[image:28.595.96.499.321.572.2]

 When one of the tests fails, the delivery team will get this fail as feedback.

Figure 5: Changes moving through the deployment pipeline [46]

4.2.2 Prerequisites of Introducing Continuous Delivery

The deployment pipeline depends on having some foundations in place [46]:

 Good configuration management.

 Automated scripts for building and deploying your application.

 Automated tests to prove that your application will deliver value to its users.

With these foundations, a continuous delivery framework can be set-up in order to perform the triggers and feedback steps shown in figure 5.

(29)

29 4.2.3 Supporting Tools

Several tools exists which support steps of the continuous delivery pattern. Below, some tools are listed as starting point for the automated steps of figure 5:

 Version control software like Git [26] and Subversion [52] for configuration management.

 Build tools like Maven [27] and Ant [28] for automated builds and tools like the xUnit frameworks [53] for unit tests.

 Test automation tools like FitNesse [7], GreenPepper [8], Cucumber [9] and Root Framework [10] and - for automating web browsers - tools like Selenium [5] and Watir [6] for automated acceptance tests.

 Continuous Integration tools like Jenkins [54] and Bamboo [55], for managing the overall process, triggers and feedback steps of continuous delivery. These tools offer to perform and monitor external jobs that build and test software projects continuously.

4.3 Test Models

There are several test models that describe test activities in relation to development activities. The V-Model is the most commonly known. Later on, the Advanced V-Model and W-Model were introduced as an improved version of the V-Model.

4.3.1 V-Model

The v-model describes the graphical arrangement of individual software development phases, connecting development and test activities in different abstraction levels. The V points to both the form of the graphical representation shown in figure 6 and to the terms verification and validation [47].

The v-model shows that testing can (and should) start at the beginning of the project [48]. In figure 6, all grey arrows have the meaning of based on. For example, acceptance testing is based on the requirements, coding is based on the detailed design, and so on. The purpose of the v-model is to improve efficiency and effectiveness of software development- and maintenance activities by connecting

development and test activities. Figure 6: The v-model[47] 4.3.2 Advanced V-Model

[image:29.595.330.501.449.587.2]

(30)

30 The advanced v-model shown in

[image:30.595.293.516.70.209.2]

figure 7 has an extra line on the right compared with the v-model in figure 6. The figure illustrates that, when the test activity in the middle line is done, the test activity on the right of that activity needs to be performed afterwards (perform test cases after unit testing, perform regression testing after integration testing, and so on.) Again, all grey arrows have the meaning of based on.

Figure 7: The advanced v-model [48]

4.3.3 W-Model

Another improved version of the v-model, called the w-model, is designed to define more clearly which test activity initiates which other test activities, what the connections are between various test stages, and what the link between the tasks of testing, debugging and changing during test phases are [47]. Figure 8 illustrates this w-model. If you compare the w-model in figure 8 to the v-model in figure 6, two lines are added:

 The second line from the left. This line states test should be planned and test activities should start early on.

 The line on the right. This line states that test activities lead to change activities by the discovery of faults and errors.

[image:30.595.132.464.461.626.2]

Again, all grey straight arrows have the meaning of based on, and as addition, the grey round arrows have the meaning of a cycle of debugging, changing and re-testing.

Figure 8: The w-model [47]

4.4 Test-Driven Development

(31)

[image:31.595.340.496.68.226.2]

31 Figure 9 shows the most common

development cycle of TDD [49]:

 Write an automated test case that defines a desired improvement or new function.

 Produce the minimum amount of code to pass the test.

 Refactor the new code to acceptable standards.

 (Repeat)

Figure 9: Test-driven development[56]

Using TDD as development style makes your code self-testing, in general more suitable for testing and it sets the focus on testing requirements instead of testing if the implementation is satisfied [57].

4.5 Test Automation

In order to achieve continuous delivery, tests should be automated. Manual testing can be described as a person initiating tests, interact with them, interpret, analyze and report the results. Automated testing is the use of a mechanism for running test cases without a tester [50].

4.5.1 Benefits

Test automation provides an improvement on the quality of testing; automated tests are formalized and can be run repeatedly for many times with minimal effort. The repeating property enables tests to run often, therefore finding errors more quickly than when manual testing is used, especially when modification in one component breaks another component [51]. Finding errors quickly saves time overall in the development processand can reduce overall development time by 8–15% [58, 59, 60].

4.5.2 What/When to Automate

The benefits mentioned above sound very promising, but test automation is not always profitable, since test automation comes with an investment: the costs of automating a GUI-test may be several times as expensive as a manual GUI test (although, relatively cheaper when a capture or replay tool is used). However, when automating compiler testing, automation is only a little more expensive than running manual tests (because both manual and automatic tests of compiles use the same syntax and it is fairly easy to put these syntax in a script) [51]. Automated tests have a lifecycle; they are run every time after the code changes until they need to be repaired or discarded. The investment of automating a test should repay itself before that test breaks.

To estimate the costs and benefits of automating, three questions should be asked [51]:

(32)

32

 How likely is it that the test dies(e.g., it needs maintenance or is thrown away)? Which events are likely to end it?

 During its lifetime, how likely is this test to find additional bugs (after the first run) and how does this balance with the costs of automation?

Note that there will always be a role for manual testing [51, 61]. First of all, it is the only way to sanity test the automation itself and secondly, some tests are not worth repeating due to high automation costs.

4.5.3 Tools

One of the key elements when automating tests for a system is knowing which tools can support test automation in the system's environment [50]. As stated in section 4.2.3, test automation tools like FitNesse [7], GreenPepper [8], Cucumber [9] and Root Framework [10], Selenium [5] and Watir [6] are a selection of automated solutions for acceptance tests.

Next to tools that automate acceptance tests, there are many tools for supporting other test activities as well, such as tools that automatically generate test cases or test data. Finding, selecting and adjusting these tools so that they are capable to work and deliver value in identity solutions projects will be a challenge on its own and out of the scope of this project due to time constraints and the choice of focusing on introducing continuous delivery.

4.5.4 Common Test Automation Problems

Pettichord illustrated several common problems that plague test automation projects [61], including that test automation does not get the focus it needs, that the goals of automation are not clear and that projects suffer from a lack of test experience from employees. Pettichord contends that test automation projects need to be run like other software development processes, needing test designs, source code management, user documentation etc. It is good to be aware of these problems to be able to early identify if these problems occur at your project. 4.6 FitNesse

As stated in section 1.5.1 and section 2.5, FitNesse will be used as test automation tool in the strategy. FitNesse [7] is a lightweight, open source framework that enables teams to collaboratively define acceptance tests, run those tests and see the results. FitNesse tests can be used very early in the project: it works well with test-driven development, where the tests are written first. FitNesse also offers an internal version control system that stores old versions of wiki pages automatically in ZIP-files as a backup [62].

(33)

33 4.6.1 Components

[image:33.595.97.496.171.297.2]

A simplified version of the FitNesse architecture, is given in figure 10. This figure shows only the parts that need to be created when using FitNesse: test cases, fixtures and the system under test [63]. Tests are specified in the wiki, from this wiki, the corresponding fixtures are called, which in their turn perform calls to the system under test and report the results back to the fixture, back to the wiki.

Figure 10: FitNesse components and the system under test (based on [64] and [63])

Wiki

The wiki has three types of pages: static, test and suite pages. Static pages are simple text pages, test pages are executable pages which contain test tables and suite pages are collections of test pages or other suites in order to group en order test pages. A screenshot of a test page with a script table is shown in figure 11. FitNesse stores the wiki as folders and plain text [62]. The wiki-pages can be run manually (via the Wiki) or automatically (via Ant, Maven, JUnit, REST-commands) by anyone with web access to the server, as frequently as required.

[image:33.595.130.468.446.751.2]

(34)

34 Fixtures

[image:34.595.129.466.240.555.2]

The test tables in the wiki are associated with tests programs, called fixtures. These fixtures take the input data from the wiki, test the system under test and delegate the output back to the wiki again. FitNesse then compares the actual outcome with the expected outcome and reports the results by color-coding the table rows in the wiki: green for success and red for failure and, when a test fails, giving the errors and differences between what was expected and what was received from the system under test (SUT). Figure 12 shows the results after running the test page of figure 11. As you can see in figure 12, the test found an error ( color coded with red) which is in this case due to a bad value in the script table.

Figure 12: Running the test

FitNesse supports two programming languages for these fixtures: Java and DotNet [65]. Support for other languages can be build in manually; some extensions for other languages exists already (e.g., plug-ins for Ruby, C, and PHP) [66]. In general, fixture code should be as thin as possible, only piping and wiring between the test table and the application code under test.

4.6.2 Detailed FitNesse Info

Test Systems

(35)

35 Test Tables

The SLIM and FIT test system supports several table styles to define your tests in. The most common used table styles are [34, 68]:

 ColumnFixture (FIT) and Decision Table (SLIM), which allows series of inputs and outputs to be defined.

 RowFixture (FIT) and Query Table (SLIM), which allows testing queries that should return an exact set of values.

 ActionFixture (FIT) and Script Table (SLIM), which allows series of events to be performed.

 Comment Table (FIT) and Comment Table (SLIM), which allows to write a tabular comment that is not executed as test.

 Import Table (FIT) and Import Table (SLIM), which allows to add a path to the fixture search path.

Architecture

[image:35.595.90.507.386.667.2]

The FitNesse architecture with both test systems is shown in figure 13. As shown on the left of the picture, test cases are defined in a wiki in FitNesse. Dependent on the specified test system, either the Fit Client or the Slim runners will be used which eventually execute fixtures that perform test API calls to the system under test. The blue blocks in the picture are given and generally not changed and the orange blocks are the places where application specific development needs to be done.