MA S A R Y K O VA U N I V E R Z I T A FA K U L T A I N F O R M A T I K Y
}w !"#$%&'()+,-./012345<yA|
Effectively Integrating Unit
Testing and Code Coverage into
Continuous Integration
DI P L O M A T H E S I S
Tom´aˇs Svrˇcek
Hereby I declare, that this paper is my original authorial work, which I have worked out by my own. All sources, references and literature used or excerpted during elaboration of this work are properly cited and listed in complete reference to the due source.
I would like to thank my supervisor, Barbora B ¨uhnov´a, for all valuable ad-vice during the writing of this master thesis.
I would like to thank Ronny Kolb as well for the patience, guidance and feedback during the whole process of implementation and writing part.
I cannot forget to thank all my colleagues in ACT team for the great cooperation and willingness to help. Without these people the work would not be completed.
These days, automated testing is a fundamental part of the software devel-opment process. It helps the developers to avoid the basic mistakes imme-diately and therefore it saves the development time. Thanks to the rapid errors discovering, the quality of the developed product is increasing.
This master thesis investigates how to effectively integrate Unit Testing and Code Coverage to the Continuous Integration process. The idea is ana-lyzed in theState of the Artchapter and the current situation is inspected in
theState of the Practice in Honeywellchapter.
Based on the results of the analysis, the plugin for the Atlassian Bamboo server is designed and implemented. Evaluation of the developed product is done and its improvements are proposed for the future work.
Atlassian, Bamboo, Code Coverage, Continuous Integration, Unit Testing, xUnit
1 Introduction . . . 3
1.1 Context . . . 3
1.2 Goals . . . 4
1.3 Thesis Structure . . . 4
2 State of the Art . . . 6
2.1 Software Development Process . . . 6
2.1.1 Rational Unified Process . . . 7
2.1.2 Other Approaches . . . 8
2.2 Capability Maturity Model Integration . . . 9
2.3 Continuous Integration. . . 10
2.3.1 Continuous Integration Servers . . . 11
2.3.2 Summary . . . 12
2.4 Automated Software Testing. . . 12
2.4.1 Reasons and Strategies for Test Automation . . . 14
2.4.2 Unit Testing . . . 15
2.4.3 xUnit Framework Testing . . . 16
2.5 Code Coverage . . . 17
2.6 Unit Testing in Continuous Integration. . . 19
2.7 Summary . . . 19
3 State of the Practice in Honeywell . . . 20
3.1 ACS Software Development Process . . . 20
3.2 Software Development Process in ACT. . . 22
3.3 Continuous Integration in Honeywell . . . 22
3.4 Bamboo. . . 23 3.5 Unit Testing . . . 24 3.6 Summary . . . 25 4 Feasibility Study . . . 26 4.1 Overview. . . 26 4.2 Bamboo Architecture . . . 26 4.2.1 Project Structure . . . 27
4.2.2 Agents and Server . . . 27
4.4 xUnit Frameworks . . . 31
4.5 Code Coverage Tools . . . 33
4.6 Tools Integration . . . 34
4.7 Summary . . . 35
5 Implementation. . . 36
5.1 Architecture Design . . . 36
5.2 Bamboo Plugin Composition . . . 37
5.3 Used Technologies . . . 37
5.3.1 Java . . . 37
5.3.2 JavaScript . . . 37
5.3.3 XML, XSD and XSLT . . . 38
5.3.4 Active Objects . . . 38
5.3.5 FreeMarker Template Language . . . 38
5.4 Data Flow . . . 39
5.5 Analysis of Potential Problems revealed during the Imple-mentation . . . 41 5.6 Presentation Layer . . . 41 5.7 Summary . . . 43 6 Evaluation . . . 45 6.1 First Round . . . 46 6.1.1 Evaluation Method . . . 46 6.1.2 Scenarios . . . 46 6.1.3 Questionnaire Results . . . 47 6.1.4 Proposed Improvements . . . 48 6.2 Second Round . . . 49 6.2.1 Evaluation Method . . . 49 6.3 Summary . . . 50 7 Conclusion. . . 51 7.1 Future Work . . . 51 7.2 Obtained Benefits . . . 52 7.3 Summary . . . 53 Appendices . . . 57
A Terms and Abbreviations . . . 58
B Package Diagram . . . 59
Introduction
1.1 Context
Honeywell is an international company with headquarters in the USA. It employs over 132,000 people all over the world. The company develops and manufactures a wide range of products from systems for residential homes and commercial buildings such as thermostats and security devices, through chemicals and materials up to turbochargers and equipment for airplanes. The company is divided into four strategic business groups:
• Aerospace
• Automation and Control Solutions (ACS)
• Performance Materials and Technologies (PMT)
• Transportation Systems (TS)
This master thesis has been designed as improvement of the current state for the ACS Center of Technologies (ACT) team. ACT was founded to drive soft-ware excellence such as development of Best Practices, training, mentoring and improvement of the software development tools used in Automation and Control Solutions (ACS). For the purposes of software improvement, Honeywell ACS follows ACS Software Development Process (ASDP) which is compliant with Capability Maturity Model Integration (CMMI) as de-scribed in Section 3.1.
1.2 Goals
Bamboo from Atlassian is a continuous integration, deployment, and release management solution that is used by Honeywell ACS. The latest version of the Bamboo Continuous Integration (CI) server offers only a few plugins for xUnit Testing and does not provide any Code Coverage functionality by default. Honeywell needs better support of mentioned tools and therefore this master thesis has been designed. The objective is to adjust Bamboo functionality for Honeywell users by implementing the plugin for this tool.
The integration xUnit framework family and Code Coverage analysis into Continuous Integration (CI) process is chosen to fulfill the assignment. The goal of this master thesis is to improve part of the development process and save the development cycle time. The result should be saving develop-ers‘ time during their activities. Based on the information from theState of
the Artand also from theState of the Practice in Honeywellstudies the
exten-sion for this Continuous Integration server is developed.
1.3 Thesis Structure
This master thesis is further divided into the following six chapters.
Chapter 2 State of the Artprovides an overview of Unit Testing automation and Code Coverage as important parts of software development process. It also describes Continuous Integration process, tools and their usability. Chapter 3 State of the Practice in Honeywelldescribes how Unit Testing, Code Coverage and Continuous Integration are used in Honeywell ACS (Appendix A). Moreover, it discusses possibilities for improvement.
Chapter 4 Feasibility Study discusses the possibility of the Unit Testing tools integration into the Bamboo CI (Appendix A) server. It describes Bam-boo architecture, its extensibility and possible integration with testing and Code Coverage tools.
Chapter 5 Implementation describes the implementation process. It con-tains a section about the architecture, code sharing and used technologies. The end of this chapter outlines the potential problems revealed during the implementation.
Chapter 6 Evaluationdescribes the methodology of thesis evaluation and summarizes gathered results. Measurements and findings are also provided. Chapter 7 Conclusionsummarizes the results of this master thesis and its fulfillment. It also contains suggestions for the future improvements of the extension.
State of the Art
This chapter describes Continuous Integration and its importance in the software development process. It describes Unit Testing, Code Coverage and other parts of the Continuous Integration process. The chapter also mentions Automated Software Testing.
2.1 Software Development Process
Software Development Process (SDP) also known as a Software Develop-ment Life-Cycle (SDLC) [1] is division of software developDevelop-ment work into distinct phases. In other words it is a set of steps that a developed software program goes through. First mentioned in 1960, SDLC was the first frame-work for building information systems. This frameframe-work is meant to develop software in a structured and methodical way from the first birth of an idea to the deployment and support. There are more approaches to define the SDP (Appendix A). Figure 2.1 describes one of the possible approaches [2].
Figure 2.1: Phases of Software Development Process
• Plan represents planning of the whole project and obtaining users‘ requirements.
• Analysisof the problem and design suggestions are the second step of the SDP. The output is usually expressed in the graphical notation such as the Unified Modeling Language (UML) [3].
• Developmentis programming of the application.
• Testsfor verification and validation should be executed to reveal the defects and therefore to increase the product quality.
• Deploymentof the application to the customer‘s environment is done after the test results.
• Maintenanceis the last phase of the SDP. The customer can find errors during the use of the system. The customer can also suggest improve-ments for the system.
These activities are in all software development processes. Depending on the methodology of the software development, small differences in terms may arise.
2.1.1 Rational Unified Process
The SDP tries to introduce the basic steps of the development to ensure the quality of the produced software. The company can use one of the certified processes to ensure the quality. Good example is Rational Unified Process (RUP) from Rational Software Corporation, a division of IBM. It is the soft-ware engineering process which gives a disciplined approach to distribute tasks, roles and responsibilities within a company. Its goal is to produce high-quality software. It is made to standardize and divide software devel-opment into four distinct phases [4]. Every phase is focusing mainly on one part of the process.
Project overview and basic analysis are established in the Inception phase. The business context, expected revenue and also financial situation of the company are introduced. Potential risks during the construction phase, potential threats and time management are considered.
TheElaborationphase provides the basic use-case model which should be fulfilled from at least 80% at the end of the project. The architecture is defined and prototypes are the outcomes of this part. This information helps to identify technical risks. The principle is to go forward to the vision. The vision has to be set wisely and must not change the main idea of the process. The current plan and architecture have to be set to fulfill the vision.
Figure 2.2: RUP diagram [5]
The main components and other features are developed during the con-struction phase. This part may consist of several iterations for better out-come. The end of this part is the first release of the developed product.
TheTransitionphase does the switch (‘transit‘) from the development phase to the production. Training the users and beta testing of the applica-tion is provided. The comparison of the requirements from the incepapplica-tion phase is set during this phase.
After the evaluation there is the final decision if the product has suc-ceeded or not. The cycle can begin next iteration regardless of the result.
2.1.2 Other Approaches
There are several models for such processes, each describing approaches to a variety of tasks.
In thewaterfallprocess, every phase is done only once. The next phase is started only after the successful end of the previous one. The customer is supposed to define the requirements before implementation of the product. In case of new ideas the customer is usually not able to change or improve functionality due to the fixed requirements. As the customer is mostly not a part of the implementation phase, she is not able to see the product until after the implementation phase.
Commonly used models areiterationandspiral. Both approaches con-sist of the four phases as mentioned before. These four parts are iterating more times during the development process until customer‘s needs are fulfilled. Spiral development is a combination of waterfall approach and rapid prototyping methodology. It begins with concept of customer‘s re-quirements. The first prototype is designed based on the rere-quirements. The requirements are improved by customer and the new prototype is designed again. The process is repeated until the customer is satisfied. After several rounds of prototypes the development begins.
Nowadays a lot of companies use theagile[6] model of software devel-opment. It is based on a new different way of managing IT projects. Basic principles are adaptive planning and flexible response to change. Due to this way of thinking, projects are usually split into smaller pieces. The manage-ment has quick status of work done in shorter iterations. More information about agile in Honeywell is provided in Section 3.1.
There are several possible approaches to do the software development. Every methodology has its pros and cons and depending on project capabil-ities the project manager has to choose the one that fits the project.
2.2 Capability Maturity Model Integration
Capability Maturity Model Integration (CMMI) is a process improvement framework which helps to define and improve the development of the prod-ucts. It involves the best practices covering the whole product life-cycle from conception through delivery to maintenance. CMMI (Appendix A) for de-velopment provides the concept of core components which can be extended by additional material to provide application-specific models which have common content.
The maturity level of an organization (Figure 2.3) consists of related specific and generic practices that improve the organization‘s overall perfor-mance. Maturity levels are used to characterize organizational improvement relative to a set of process areas [7].
AtInitialmaturity level, the processes are usually ad hoc and chaotic. The organization environment is not stable to support processes. Success depends on the individual people in the projects instead of the following rules of the processes.
Figure 2.3: CMMI Maturity Levels
AtManagedmaturity level, the projects must execute the processes in ac-cordance to policy. Projects contain plans and are also monitored, controlled, reviewed and evaluated for the future use. In other words management is able to control the milestones of the projects. This level helps to retain the plan during the times of stress.
AtDefinedmaturity level, the processes are well characterized and un-derstood. Established organization‘s standard processes are improved over the time. The performance of the defined processes is managed. These stan-dards are used to establish the consistency across the company.
AtQuantitatively Managedmaturity level, quantitative objectives are based on the customer‘s needs, end users and organization. Processes and projects are based on measures and statistic terms. The performance of projects and selected sub-processes is controlled using statistical and other quantitative techniques.
AtOptimizingmaturity level, the main goal is focused on continuous improvement process. The effects of deployed process improvements are measured using statistical and other quantitative techniques. This level is focused on improvement across all the projects in the organization.
2.3 Continuous Integration
Continuous Integration (CI) is a software development practice to simplify the developers‘ commit of source code into the repository followed by other steps such as static code analysis, Unit Testing, Code Coverage analysis, functional tests and others (Figure 2.4) to the one automatized unit. The
basic idea is to add the new piece of code to the build process as soon as possible. After every commit to the repository the CI (Appendix A) process should be executed. The developers subsequently know possible errors and can fix them. CI process can take more time depending on its content. The vision is to automate as many procedures as possible. It should be as easy as clicking a button.
Figure 2.4: Continuous Integration Cycle [8]
2.3.1 Continuous Integration Servers
CI process provides automation of work which is achieved by CI servers. There are several solutions on the market and the following part discusses and compares three of them [9].
JenkinsCI server was forked from Hudson in 2011 and is developed as an open source project. Hudson focuses on improving stability and perfor-mance for the enterprise organizations. The main advantage of the Jenkins is the number of plugins and big community support. Development of new plugins is easy way how to extend this server. The negative side is its sta-bility. Plugins have big differences in the implementation which leads to unexpected behavior of the system.
BambooCI server is developed by Atlassian company. The biggest dif-ference compared to Jenkins is its price. Bamboo is not an open-source soft-ware and provides reliable support especially on the portal named Atlassian Answers [10]. Extensibility of this tool is more complicated than Jenkins. The biggest advantage is the simplicity of integration with other Atlassian products. Bamboo provides also great integration with GIT revision control system. Moreover, this solution has remarkably better User Interface than Jenkins.
TeamCityis the product of JetBrains and it is a commercial tool. Team-City offers integration with version control systems (VCS) though does not contain sufficient support. On the other hand Bamboo provides great branch and merge management for the version systems. TeamCity also offers suf-ficient integration with defect tracking tools. The use with Amazon Elastic Compute Cloud (EC2) is included as it is in Bamboo. TeamCity provides great integration with other JetBrains products.
2.3.2 Summary
Each mentioned CI solution is capable to be used for the development. Bam-boo is good for bigger teams because of integration and more complex jobs. Jenkins wins in the price aspect and ease of use. TeamCity provides the easiest workflow and project definitions.
2.4 Automated Software Testing
Software testing is an essential and expensive activity in the Software De-velopment Life Cycle (SDLC). Automated Software Testing (AST) is the ap-plication and implementation of software technology throughout the entire software testing life-cycle (STL) with the goal of improving STL efficiency and effectiveness [11].
• Team members do not have to spend time on inspections and reviews and can spend time on higher-priority testing tasks.
• Reduction in missed findings related to human errors.
• Allowance for measurement of testing coverage, i.e. identification of areas that are not covered by a test (almost impossible to find manu-ally).
Nowadays the developers are able to convert most of the types of manual tests into the automated tests. Automation of many recurring actions offers another potential source of saving company‘s money. Of course, it is neces-sary to consider whether all tests need to be automated. The used metric is called Return of Investment (ROI). If automated software testing is imple-mented correctly, in many cases it helps to save the time, lower investments (costs), and also helps to raise the quality of the development software prod-ucts.
AST is the integral part of the CI process and contains the following four levels.
Unit Testingis the first part of testing and is described in more detail in Section 2.4.2.
Since the Unit Testing checks the smallest parts of code such as classes, Integration Testing tests cooperation of bigger parts of the system called modules. The user can add individual applications to the modules integra-tion and also is able to add client-server applicaintegra-tions on a network. This type controls interactions between different parts of the system such as an operating system, file system and also hardware interfaces. Many modules are developed in the same time and they need to be integrated together. It falls into the black-box and also white-box testing.
System Testingtests the scope and verifies all requirements from the customer. It also includes tests based on risks and/or requirement speci-fications, business processes, use-cases or other high-level descriptions of system behavior. It tests the interactions with the different parts of the sys-tem such as file syssys-tem and hardware or interfaces between syssys-tems. The basic practice is to test interactions with the different operating systems and system resources. This type falls into the black-box testing.
Acceptance Testingis the last part of testing process. The system is deliv-ered to the customer for the Acceptance Testing. The customer verifies if the system meets all the requirements and if he/she accepts the final product. It falls into the black-box testing category.
AST (Appendix A) encompasses many types of testing phases, includ-ing cross-platform compatibility, and is process-independent. Advantage of AST against the manual code testing is the possibility to avoid repetitions of scenarios many times by a tester. AST is a part of software development and does not replace the need for manual testers‘ analytical skills, test strat-egy know-how, and understanding of testing techniques. Manual software testing and AST cannot be separated. Basically, they complement each other. Automation is the natural solution where a lot of personnel resources are saved and more errors are revealed.
2.4.1 Reasons and Strategies for Test Automation
The main advantages of Test Automation are the following [11]
• Reduces the time and cost of software testing.
• Improves software quality.
• Enhances manual testing efforts via increased testing coverage.
• Does what manual testing can not accomplish, such as concurrent test-ing, performance testtest-ing, etc.
There are many strategies how to approach test automation. Many compa-nies use the white-box, black-box or optionally gray-box testing as a power-ful method.
White-box testingis one of the three well-known methods of testing software. Developers can see the source code of an application during this approach therefore they are able to predict outputs of their tests [12]. They can see if the program diverges from its intended goal. It is a well-known process for all manual testers. The big problem is that the testers do not test it sufficiently, as they are influenced by knowledge of the source code. Also the developer is only a human and can forget to test some situation. The most important knowledge is which line of the code is being executed and if line‘s result is correct or not. The advantage of this technique is the simplicity of the automation.
White-box testing can be applied at the unit, integration and system test-ing levels of the program. This technique can include Code Coverage which includes branch coverage, statement coverage, decision coverage and modi-fied condition/decision coverage criteria (more explained in Section 2.5).
Black-box testing[13] is unlike the white-box testing based on the spec-ification and requirements of the applications supposed functionality. Test cases are made in the same manner. The tester usually knows what result the program returns. On the other hand during black-box testing the tester does not have to know how the program is executed.
Black-box testing can be applied to all levels of the software testing such as unit, integration, system and acceptance testing.
Figure 2.5: Black-box
Gray-box testing[13] is a combination of white-box testing and black-box testing. The testers do not have the access to the source code. They have access to the documentation and they are familiar with the used algorithms. The testers usually have the specification and requirements of the applica-tion. Their task is to create feasible test cases from this data. Gray-box testing maintains boundary for testing between the tester and the developer. Find-ings from the statistics show that combination of black-box and white-box testing is the best solution.
The problem of the software testing in general is money. In fact, 30-40% of the total costs are spent for the application testing.
The other strategy how to test the product is horizontal vs. vertical test-ing [14].
Horizontal automation testing represents point to point flow where each test case is independent of the others and is following strict scenario.
Vertical automation testingis executing test cases sensibly to save the execution time. The tester uses the state of the process and can execute more test cases at once.
2.4.2 Unit Testing
Unit Testing is the first step of software testing. This part tests individual hardware or software units or groups of related units. In objected-oriented programming, a unit is a software component that cannot be subdivided into other components [15]. The developers write Unit tests concurrently with
the source code to avoid the first wave of QA returns. Approximately 65% of all the bugs can be caught by Unit Testing. The tests should be done by the developers and falls into the white-box testing category (Section 2.4.1). It also helps to reduce the cost of bug fixes when the bugs are found in the earlier stage during the testing. As the nameUnit Testingsuggests, this part ensures that the blocks of the software work correctly independently from each other. Unit Testing might include Static Code Analysis which provides early defect detection. In general, it is used to ensure that code meets the architecture and behaves as intended.
Benefits of Unit Testing are isolation of each part of the program and insurance of the individual parts as correct. Automation of the work is prob-ably the most used benefit and so there is no need for group of testers who have to test the application manually.
2.4.3 xUnit Framework Testing
xUnit is a common name for a set of Unit Testing frameworks such as JU-nit, CppUJU-nit, PHPUJU-nit, etc. The unified name was chosen for the structure similarities of all frameworks.
The generic name xUnit [16] is originally created from SUnit from 1998 from Kent Beck and was known mainly in Smalltalk developers‘ community. JUnit had arose on the flight from Z ¨urich to Atlanta when Kent Beck traveled with Erich Gamma and introduced him his idea about xUnit frameworks. The red-green progress bar with passed and failed results was the first time used in JUnit. This idea subsequently came back to the SUnit framework as an improvement. Then the idea had appeared in many others xUnit frame-works. First from new wave was CppUnit where Michael Feathers found way how to use a core from JUnit. He developed a CppUnit which is the C++ port of the famous JUnit framework for Unit Testing. The developers worked with other languages made their ports of JUnit in a short time.
In the meantime the JUnit became the ‘standard‘ of xUnit family frame-works and was the essential idea in expansion of Extreme Programming (XP) and Test Driven Development (TDD). Unfortunately, JUnit is still not a part of JDK and therefore it is a third-party project.
All tools of the xUnit family frameworks include following four phases to clearly identify the intent of the test [17]:
• Set up test fixture/pre-test state (more in Section 4.4).
• Interaction with the system.
• Tear down the test fixture.
While the developers are able to run the same test many times in a row with xUnit, the xUnit framework is not particularly well-suited for automating performance tests [17].
2.5 Code Coverage
Code Coverage analysis is a technique that measures the percentage of sys-tem‘s code that is being tested by particular test suite while the automated tests are running. Code Coverage analysis is a powerful tool for finding more of the software bugs enabling a release of a software product of bet-ter quality. More technically, Code Coverage analysis reveals the areas in a program that are not covered by any test cases.
Code Coverage tools are divided into two main categories. The first type adds the statements into the code and calls for recompilation of the code. The second type is the group of tools which instrument the code before run-ning or during the execution. In this process a specialized tool is used to instrument the binaries to add the tracing calls and run a full set of auto-mated tests against the instrumented product. During the process the data is stored and subsequently the report is generated. Good Code Coverage tools give to user the percentage of the code that is tested and also allow seeing exactly which lines of code were executed during the process.
This part of the Continuous Integration process is expanding because software developers would like to know how many lines of the source code are covered by Unit tests. Usually, the result of this analysis is the ratio between the lines of code which are covered by Unit tests and all lines of the code. Code Coverage tools also provide other measures. There may also be information about function call coverage and branch coverage in the report. Code Coverage tests the code directly, therefore it is a white-box testing.
The Code Coverage analysis compares the results of percentages against the given set of criteria. The main types of criteria are following [18]:
• Statement Coverage(also known as Line Coverage, Segment Cover-age or Block CoverCover-age)
• Function Coverage
• Function Call Coverage
• Condition Coverage(also known as Decision Coverage or Predicate Coverage)
• Branch Coverage(also known as Path Coverage)
• Modified Condition Coverage(also known as Modified Decision Cov-erage)
Statement Coverage reports the number of lines in the source code that have been executed. A variant of Statement Coverage is Block Coverage. This measurement is based on code blocks instead of statements. Basically, block is a set of statements. From a test quality point of view, there is a one-to-one relationship between Statement Coverage and Block Coverage. 100% success indicates that every line of code in an application has been tested.
Function Coveragereports whether a function has been called or not. It does not say anything about execution inside the function or method. Nei-ther it does not count how many function calls are executed. 100% covered indicates that every function/method in an application has been called by a test suite.
Comparing to the Function CoverageFunction Call Coveragemeasures the count of the available function calls. Every function and all its parts (i.e. if-statement) have to execute all branches in the code section at least once to achieve 100% Function Call Coverage.
Condition Coverageinspects if each Boolean expression has been eval-uated to both values: True and False. 100% success indicates that for each conditional statement, both outcomes have been tested.
Branch Coverageis an advanced type of code coverage. It requires that all code blocks and all execution paths have been tested. This type adds more advanced requirements on top of statement or Block Coverage. For the if-statement, Branch Coverage is fulfilled when the code is executed at least two times. First time it is evaluated to the false and second time to the true. Only then all the code blocks and all the execution paths have been tested.
Modified Condition Coverage is considered as advanced type of the Code Coverage analysis. In general, it is a combination of all types of Code Coverage. It requires that all code blocks and all execution paths have been tested. It adds the requirement that all the sub-expressions in the complex branch expressions (with more statements in if-condition) are considered as well. Basically, this means that the code section must be executed many times, until all the sub-expressions have been tested independently of the others. For example the block of code
if( (A||B) &&C){
result=0;
a=evaluateA(B); }
result++;
should be evaluated 8 times because in if-condition there are three vari-ables A, B and C.
In many cases Branch Coverage and Statement Coverage have the same effect. For example, if the code only contained if-then-else statements, the metrics are equivalent. Testing all paths through code implicitly tests State-ment Coverage and Function Coverage. StateState-ment Coverage implicitly tests Function Coverage.
2.6 Unit Testing in Continuous Integration
Automated Unit Testing and Code Coverage analysis are in most cases pro-vided by the Continuous Integration solutions. They exist in the form of plugins for the CI tools.
Jenkins (Section 2.3.1) provides many useful plugins for Unit Testing util-ities for a various programming languages. It also offers code coverage tools for better control of Unit tests. The examples of Code Coverage tools are Cobertura[19] and Clover[20] which exist in Jenkins by default. The work and settings with these plugins are simple and the workflows are direct. Overall results look transparent and therefore they bring the inspiration for the new Bamboo extension. Jenkins also provides xUnit Plugin by default which accepts formats as XML, TXT, CSV, etc. as input file and generates the results.
Bamboo offers poor support for xUnit frameworks in comparison to Jenkins. This may be due to smaller user base of this tool. Moreover, Bamboo does not provide any Code Coverage tools by default.
2.7 Summary
This chapter provides introduction to the various tools, methods and re-lated terms. It compares available solutions on the market and discusses their advantages and disadvantages. The Unit Testing and Code Coverage implemented as parts of Continuous Integration have a high importance in the development of high-quality software. They offer metrics and also reduce time of the development process.
State of the Practice in Honeywell
Honeywell ACS developed ACS Software Development Process (ASDP) to gain CMMI Level 5. The best practices for each of the software development disciplines have to be implemented to achieve this standard. This chapter describes ASDP and outlines current situation in Honeywell ACS teams regarding the use of Unit Testing, Code Coverage in Continuous Integration.
3.1 ACS Software Development Process
The first version of ACS Software Development Process (ASDP) was re-leased in January 2010 and current version is 3.6 published in September 2013. ASDP (Appendix A) aims at being CMMI Level 5 compliant software development (Section 2.2). ASDP defines an iterative and incremental life-cycle. ASDP is based on best practices from existing Honeywell ACS Proce-dures and Industry Standards as Rational Unified Process (RUP) and others. The content of the ASDP is organized into the two sets of the Disciplines [21]. Each discipline has its responsible person. The first type are the life-cycle disciplines calledCore Disciplines.
• Requirements • Architecture • Design • Implementation • Test • Deployment
The second type is calledSupport Disciplineswhich are performed through-out the whole project life-cycle, from the inception phase to release. They help to manage and organize the whole process and to resolve unexpected accidents.
• Project Management
• Quality Assurance
• Organizational Management
ASDP defines various roles, one for each activity. The process provides 43 roles by disciplines and 27 project roles (i.e. DB Designer, Project Manager, Security Architect, Tester, UI Designer, etc.). They perform a task and are responsible for a work product. Each discipline has to produce a compre-hensible outcome.
There are several categories of work products within the ASDP process. The two most important are Deliverables and Mandatory Work Products.
• Deliverablesare the expected outcomes from the concrete activities. Typical outputs are the product (including source code), its parts and project and product documentation. Deliverables are those critical work products generated in a project that need to be verified. Pre-defined templates are used to unify the outputs of the reports.
• Mandatory Work Products(outputs) are all types of the reports. ASDP supports the reports by templates as it was mentioned before. Reports have to exist for evidence during process compliance audits.
Finally there are work products that are optional in ASDP. These work prod-ucts are related to the best practices. For example, a Static Code Analysis Report or Code Coverage Analysis Report would only be generated if a team chose to perform that practice.
Measures and Indicators are needed to measure the success of the pro-cess. Typical examples of measurement arenumber of defects, effort, project
duration, defect leakage or defect density. Honeywell‘s ASDP provides different
points of view on the measurement. Measures aligned with the business goals aredelivery on timeandreducing cycle time.
As ASDP aims at being CMMI level 5 compliant, its basis is a continuous improvement of itself, the projects and the company. It evolves the method-ology from the past and recommends the best estimation in general. For this purpose ASDP contains the knowledge how to estimateexpected
deliv-ery date,required resourcesandproject complexity. Honeywell supports project
management to improve processes in the organization.
Measurement is important for comparing more projects in the company and therefore for the future improvement. There are four main reasons for measuring software and system processes, products, and resources.
After the measurement it is able to bettercharacterizethe process and therefore establish the baseline for future comparisons.
Theevaluationhelps to recognize when the project is going out of the track and therefore brings it back under the control.
Theprediction is powerful ability for better planing. Gathered values from observation of modeled relationships can be used to predict other rela-tionships between the processes. The prediction and the estimates based on historical data are used to analyze the risks. This helps to establish achiev-able goals for cost, schedule and quality.
Futureimprovementis the global goal of all measures. ASDP uses mea-surement to identify inefficiencies, root causes and other places for improv-ing product quality and process performance.
As it was mentioned, the ASDP also contains Templates, Guidance and Tools suggested from disciplines and supported by Honeywell.
UNITY Project
Integration of used tools and Atlassian plugins sticks these two together under the one project called UNITY. The main idea is to remove data dupli-cation and incompatibility across the Honeywell ACS and therefore there will be less human errors during the development process as well.
3.2 Software Development Process in ACT
One of the most important decisions is the strategy to approach the problem. There are several known proven solutions such as agile software develop-ment or waterfall model (described in Section 2.1.2). ACT tries to use agile software development process with the SCRUM methodology. ACT usually practices strategy of 2 to 4-week sprints to achieve partial deliverables (ex-plained in Section 3.1). All development approaches fall into the general iterative PDCA (Appendix A) process shown in Figure 3.1. There is four-phases similarity with the RUP process from Section 2.1.1. Recommended approach is agile software development with ASDP support. The strategy is set by Project Manager depending on the individual project. There is still used iterative waterfall development process a lot across the ACS teams.
3.3 Continuous Integration in Honeywell
Continuous Integration is a current trend of software development and Hon-eywell aims towards this process as well. The main goal is to simplify the whole development process as much as possible.
Figure 3.1: Continual Improvement
Honeywell ACS uses parts of CI and there is still a space for improve-ment. ACT (Appendix A) brings the integration of the particular tools to the CI process. Current situation of ACS shows that not all teams in Honey-well ACS are on the same level of using Continuous Integration. Nowadays, ACT team uses only Static Code Analysis and Unit Tests to speed-up bug findings and increase the quality.
Current state of ACS Honeywell CI runs only the Static Code Analysis tool Klocwork during the nightly builds. Not all teams are able to build, test and deploy applications. A few of them are only in the previous stages of CI and obviously the ACS effort is to unify the work process and using CI effectively as much as possible.
Future plans include more types of Unit Tests, Code Coverage analy-sis, Dynamic Code analyanaly-sis, Automated Functional Tests and application Deployment to the Continuous Integration process.
3.4 Bamboo
ASDP recommends various tools for most of the disciplines. Atlassian Bam-boo is proposed as useful tool for the implementation within Continuous Integration. This tool is not used only for running builds and test. It connects issues, commits and test results and is able to deploy applications. It serves the results across the whole team from developers and testers to the project management.
Bamboo is one of the few tools for Continuous Integration. In general, most of the CI tools offer the same functionality. Honeywell uses Bamboo mainly for its smooth integration with the Atlassian Software Suite. Other benefits are comfort User Interface and easy extensibility via plugins with Atlassian SDK. Another useful feature is automatic branch management. When this function is enabled Bamboo creates the copy of current plan for every new branch in the project and everything is done automatically.
3.5 Unit Testing
The big expansion of software testing during the past decades improves the quality of the developed software. ASDP leads to improve quality by several different approaches. Contrary to Integration Testing, which focuses on the possible interactions between two or more parts, Unit Testing only considers the separate parts in isolation. Unit Testing is a practice to examine the individual testing of source code units to determine if they behave as intended.
At this time most of the Unit Testing frameworks used in ACS belong to the xUnit test frameworks. The following list shows the most popular tools across the ACS.
• JUnitandTestNGfor Java language.
• NUnitand Visual Studio plugins for .NET languages (C#, VB).
• CppUnitfor C++ language.
Using these xUnit frameworks gives way to unification the results from var-ious Unit Testing tools. Integration of tools under UNITY project is partially conditioned by the use of xUnit frameworks.
Code Coverage analysisis closely related to the Unit Testing. There are only few attempts of the Code Coverage analysis tools such a dotCover for .NET tested. Nowadays there is no Code Coverage analysis performed across the Honeywell ACS. Teams working with .NET platform would like to use Visual Studio plugins. Real addition of the analysis to the CI process is plan for the future.
3.6 Summary
This chapter describes current state of processes and tools used within Hon-eywell ACS. ASDP, mentioned in Section 2.2, is a CMMI Level 5 compliant software development process. For this purpose Honeywell ACS uses At-lassian products, Bamboo especially for Continuous Integration process. As it is analyzed in this chapter, ACT team uses Unit Testing to increase quality of developed software but the Code Coverage is still missing.
Feasibility Study
The goal of this master thesis is to develop a plugin for Bamboo Continuous Integration server. For this purpose the initial analysis before the imple-mentation is performed. The extension gathers the data from various Unit Testing frameworks and Code Coverage Analysis tools. Atlassian uses its own SDK to develop the plugins.
This chapter provides an overview of the Bamboo server and its extensi-bility. Next parts contain basic information about xUnit Frameworks, Code Coverage tools and their integration into Bamboo.
4.1 Overview
ACT team has a lot of experience with the Atlassian products extensions. The essential requirement declares to simply run one or more tools in the build. This is done by design uniform input XML file for all user inputs (example in Listing 4.4). It is the only place for information from the user. All tools are usable via command line, so the user can easily write commands into input XML file. The basic assumption for the tools integration is similar structure of the XML output files produced by xUnit testing frameworks. Code Coverage tools produce the similar output files as well.
4.2 Bamboo Architecture
Analysis and understanding of Bamboo architecture is necessary for better integration. This section describes project structure, build possibilities and capabilities and requirements.
Figure 4.1: Bamboo Plan Architecture [22] 4.2.1 Project Structure
Bamboo CI server contains projects which contain plans which are struc-tured as depicted in Figure 4.1. Tasks are executed sequentially within a Job. Jobs are executed in parallel within a Stage. Stages are executed sequentially within a Plan. The project may contain more plans. Other CI servers can use different terms for the project structure parts.
4.2.2 Agents and Server
Agent is a resource for Bamboo server to perform builds. The server is in-stalled on the central machine. Bamboo is able to run jobs on different agents (machines) for parallelism. There are two types of agents. Local agents run on the same machine as Bamboo server and remote agents obviously on separated machines. Remote agents communicate with the server via the network. Bamboo uses more agents to increase the computing power and save the time.
The user is not able to manage administration without logging in. The user should check if at least one agent is capable to run the job in case there are any job requirements.
Administration section Agents provides view of all local and remote agents and their statuses. Agent can be either:
• idle(available to execute builds);
• cancelling(currently cancelling a Job build);
• disabled(not available to execute builds);
• disabled-building(currently executing a build, but disabled so cannot execute further builds);
• disabled-cancelling(currently cancelling a build, and disabled so can-not execute further builds).
Bamboo provides interesting function called Elastic Bamboo (Figure 4.2) which grants user access to computing resources from the Amazon Elastic Compute Cloud (EC2) to run builds. Amazon EC2 (Appendix A) is preferred for the companies and it changes the economics of computing by allowing a customer to pay only for capacity that he/she actually needs. The customer can ‘rent‘ the virtual machines from Amazon for building his/her builds for monthly fee. Using this feature is a big advantage for the companies as for every job requirement there is at least one agent capable to build it.
4.2.3 Capabilities and Requirements
Every job on Bamboo has a possibility to set the requirements. Requirements basically define the set of the agents where the job can be performed. The job inherits the requirements of its tasks by default. Administration section of Bamboo contains a Server capabilities tab where there are displayed ‘Match-ing Agents‘ and ‘Match‘Match-ing Images‘ numbers that is constantly updated.
Each agent provides executable capabilities which are external programs that Bamboo uses during the build process. Current version of Bamboo in ACS supports following executable capabilities:
• Ant(to build projects by Ant)
• Custom command(e.g. ‘make‘)
• devenv.com(to build projects from Visual Studio)
• Grails(to build Grails projects)
• Maven(to build Maven projects)
• msbuild.exe (to build .NET/C# projects)
• NAnt(to build .NET projects)
• PHPUnit(to build PHP projects)
• Script
Agent can be either chosen by the administrator or Bamboo assigns the job to the random capable agent by itself. Capability is a feature of an agent and defines the path to an executable program. Capability can be shared by all local agents or all remote agents. Bamboo has its own predefined capabilities such as Maven, Mercurial or JDK. Job requirements offer only agents‘ defined capabilities. Bamboo offers an option to add specific custom capability if there is missing one. Agent specific custom capability is defined by key and value. Key represents capability name in Bamboo and value contains the path to the specific capability.
The user may setup Bamboo job requirements via input XML file (Fig-ure 4.4).
4.3 Bamboo Extensibility
Bamboo CI server is extensible by plugins. ACT team has a lot of experience with extending Atlassian products. Atlassian SDK is designed to create ex-tensions for Atlassian applications such as Bamboo, Confluence, JIRA and Stash. Bamboo plugin is compound from plugin modules. The essential part is a plugin descriptoratlassian-plugin.xml which contains the definition of all modules of the extension. There are four categories of module types.
• User Interface plugin modules
• Build Life-cycle plugin modules
• Notification plugin modules
• System plugin modules
Special separate category represents Bamboo Event Listeners. Nowadays, there are 43 plugin modules available but for the purpose of this master thesis only the following few plugin modules are used [24].
Task plugin module(from Build Life-cycle category) represents a basic unit of work during the Build (Task in Figure 4.1). Every Job can have multiple Tasks which run in predefined order on the Agent. However, if only one Task fails, the following tasks will not be executed except for the Final Tasks.
Tasks are designed to be easily extensible. The only need is to implement single Java interface to activate the Task. Moreover, there are optional exten-sion points allowing user to add a configuration screen and other powerful features to the Task.
WithWeb Item plugin module(from User Interface category) developer is able to add a link to some defined context on certain place on the web page. The link should have an ID and can be modified by CSS classes.
XWork plugin module (from User Interface category) allows the user to react to the action. The module allows users to create their own XWork package and accessible actions. Developer usually needs to add a Web Item module to link XWork action with the system. Example of this plugin mod-ule is in Listing 4.1.
<xworkkey=”xworkResult”name=”CoveragexWork”> <package name=”resultXwork”extends=”buildView”>
<action name=”xworkAction”class=”plugin.Result”> <result name=”buildKey”type=”ftl”>results.ftl</result> <result name=”error”type=”ftl”>err.ftl</result> <result name=”fourOhFour”type=”ftl”>404.ftl</result> <result name=”none”type=”ftl”>none.ftl</result> </action>
</package> </xwork>
Listing 4.1: XWork plugin definition example
The developer often needs to use external downloadable web resources during the plugin development.Web Resource plugin module(from User Interface category) allows developers to define them. CSS and JavaScript are usually also included web resources for the web design into the specific contexts of the application.
WithComponent plugin module(from System category) the developer is able to share Java components between modules and optionally with other plugins running at the same time. The typical use is for singleton class used internally in the plugin. For the purpose of this thesis the pooling is used to save the time of the user. The biggest savings are during the view of builds history because the data is cached.
4.4 xUnit Frameworks
The frameworks from the xUnit family (Section 2.4.3) generate XML file with a similar structure. The file always contains test suites, test cases and their results. ACT comes in with a requirement to develop a Bamboo plugin supporting these frameworks which simply and logically displays the data. The results from all used xUnit tools are displayed in one tab under the build results.
These frameworks are used especially in bigger companies where de-velopers work on more projects written in the different programming lan-guages. Using the frameworks provides the unification of the output XML file.
All frameworks from the xUnit family have the same following parts in common.
Figure 4.3: Core Parts of xUnit framework
Test Caseis usually one test. The conventions prescribe to write one test method with only one assertion. This practice produces a lot of redundant tests or unwanted lines of code. On the other hand, many lines of code are better than hidden failures. It is hard to find which assertion had failed in multi-assertion methods without this practice.
Test Suiteis a set of Test Cases. It is possible to create multilevel Test Suites (means Test Suite of Test Suites) during the test development.
Test Runnerruns Test Suites, Test Cases, or multilevel Test Suites as all of these objects implement the same interface. Test Runner is reliable for the reports and results especially for the information how many tests had run or how many tests had failed the assertions. It also reports the number of tests that raised errors or exceptions. Another capability of Test Runner is simultaneous run of all the tests, automated by different people.
Test Fixture (also known as a Test Context) is a set of preconditions for the running Test Cases. The developer is able to set the state of test environment for running tests. After the test, developer should return the program to the previous state without dependency on the other tests. Test Fixture is used for better identification of the errors (independent of the environment).
<report>...
<method name=”getAllUserRoles (): List”>
<coverage type=”method, %”value=”0% (0/1)”/> <coverage type=”block, %”value=”22% (2/9)”/> <coverage type=”line, %”value=”50% (1/2)”/> </method>...
</report>
Listing 4.3: Emma output example
4.5 Code Coverage Tools
Nowadays there are many tools for Code Coverage Analysis on the market. The most wide-spread solutions are free open-source tools Cobertura[19] and Emma[25] however they are not maintained anymore. Each Code Cov-erage tool generates XML file with the similar structure like it is in xUnit frameworks case. XML file contains file coverage, package coverage, class coverage, method coverage and line coverage. Most of the results are ex-pressed in percentage.
The outputs of the tools in the Listings 4.2 and 4.3 contain basically the same data. The generic XML file is designed to gather information from various tools. The XSLT file is used for the transformation of the XML output files. The examples of the given tools outputs are following.
<coverage>...
<method name=”getCar”line−rate=”1.0”branch−rate=”1.0”> <lines>
<linenumber=”35”hits=”2”branch=”false”/> <linenumber=”36”hits=”2”branch=”false”/> </lines>
</method>...
</coverage>
Listing 4.2: Cobertura output example
According to the task specification of the thesis it is preferable to use Emma which is ancestor of JaCoCo[26]. All of mentioned tools support com-mand line execution, ready to use and even for free (more details in Table 4.1).
Table 4.2 also includes non-free Atlassian‘s Clover tool, which is simply integrated with other Atlassian products. The advantage is various report types and also statement coverage metric. The assignment of this thesis is to find free and useful tools. Other paid tools are the part of suggestions for the future work.
Tool Commercial use Licence Cobertura free GNU General Public Licence Emma free Common Public Licence v1.0 JaCoCo free Eclipse Public Licence (EPL)
NCover non-free End-User Licence Agreement (EULA) PHPUnit non-free Creative Commons Attribution 3.0 Atlassian Clover non-free proprietary
CodeCover free Eclipse Public Licence (EPL) Pitest non-free Creative Commons Licence
Table 4.1: Code Coverage tools licences Cobertura Emma/JaCoCo Atlassian Clover Coverage metrics
Method no yes yes
Statement no no yes
Line yes yes yes
MC/DC no no no
Report types
HTML yes yes yes
PDF no yes yes
XML yes yes yes
JSON no no yes
CSV no yes no
Table 4.2: Code Coverage tools comparison
4.6 Tools Integration
ACT team decided to design a solution with the XML input file shown in Listing 4.4. The user defines used tools, all necessary directories and require-ments of the task if there are any. There is an option to define directories of test files on the local machine or on the remote machine. The input XML file is parsed and commands are executed. Used tools generate intermediate data placed into one result file which is parsed again and final test results are displayed in Bamboo.
<input xsi:noNamespaceSchemaLocation=’inputValidator.xsd’> <name>CoberturaTest</name>
<execution>
<type>COMMAND</type> <commands>
<command>
<basedirrelative=”true”>JAVA\</basedir> <query>mvn cobertura:cobertura</query> </command>
</commands> </execution>
<results> <result>
<name>Cobertura</name> <location>LOCAL</location> <format>XML</format> <filePath>result.xml</filePath>
<parsing type=”XSL”>convert.xsl</parsing> </result>
<result>
<name>JUnit</name> <location>LOCAL</location> <format>XML</format> <filePath>result.xml</filePath>
<parsing type=”XSL”>junit.xsl</parsing> </result>
</results> <requirements>
<requirement>Executable.Ant.Ant1.4</requirement> <requirement>Executable.MSBuild.32</requirement> </requirements>
</input>
Listing 4.4: XML input file example
4.7 Summary
This chapter provides the basic information about Bamboo architecture, its extensibility, and tools integration. Essential idea is similar XML output file of the used tools. The goal of this master thesis is to develop simple exten-sible Bamboo plugin for the Honeywell ACS. It is easy to extend Bamboo CI server thanks to Atlassian SDK and its implemented modules. Analy-sis of current state of ACS environment implies that it is possible to fulfill customer needs. The development of Bamboo plugin for xUnit testing frame-works and Code Coverage ensures time saving and better results summary.
Implementation
This chapter describes implementation and technologies used during the implementation process. The data flow in the plugin, potential problems and user interface issues are also discussed in following sections of this chapter.
5.1 Architecture Design
Before the actual implementation of the integration, a number of architec-tural design decisions had to be made. Honeywell ACT team aims at in-tegrating Continuous Integration process into the development work. The plugins for Bamboo are developed for this purpose. Implementation of this master thesis is closely interconnected with the thesis of Samuel Peˇtovsk ´y. Samuel writes about effective integration of static code analysis into the Con-tinuous Integration process. Further analysis identifies related connections between these two theses. Due to this fact, ACT decided to implement the logic of the application into the common library. We named itCore.
Another issue was the development of the Core as a Bamboo plugin or as a standard Maven JAR library. The Bamboo plugin is accessible via Bamboo component system and therefore there is bigger overhead. The resolution was implementation as the Maven library for simplicity of integration.
The Core contains functions for plugin configuration, commands execu-tion and result conversion and parsing. The Core contains parsers for XML files and classes for conversions to other formats such as JSON and plain text. Package diagram of the plugin and Core relations are in Appendix B. The database is accessed by the ORM framework called Active Objects [27]. Ev-ery access to the database must be in the transactions for the data persistence. Plugin also contains available REST API for the external communication of Core with other systems.
Unit tests and Code Coverage tools are either launched from command line or they are included in the build systems as Maven or Ant for Java. .NET platform uses MSBuild for this purpose. Command line option is chosen because Bamboo provides interface supporting script execution (precisely execution method) and therefore Unit Test tools and Code Coverage tools may be executed.
5.2 Bamboo Plugin Composition
Atlassian extension is the JAR file which contains XML plugin descriptor, source code and front-end files. The plugin descriptoratlassian-plugin.xml
encloses labels and linkage to all necessary files. It includes plugin modules (mentioned in Section 4.3), external front-end files (.ftl, .js, .css) and core component injected from the Bamboo server. Attlasian API offers simple plugin establishment for the Bamboo plugin by only one command
atlas-create-bamboo-plugin. During the plugin creation the artifact ID, group ID,
version and package for the source code must be set.
5.3 Used Technologies
The following section describes used technologies during the implementa-tion process.
5.3.1 Java
Java is a programming language and computing platform originally devel-oped by James Gosling at Sun Microsystems and released in 1995. Java is fast, secure, reliable and especially cross-platform language which is almost in every device. Nowadays, there are more 9 million Java developers world-wide and the community is still growing. Some applications and websites do not work properly until Java is installed. This language is mostly used in client-server web applications. Atlassian also uses Java programming language in their products and provides a good support.
5.3.2 JavaScript
JavaScript (JS) is a dynamic programming language used for special actions on the client side of the application. JavaScript[28] is widely used as it allows web pages to display dynamic content and transfers part of computational power to the client side. Another big advantage is using the JS as cheap
tech-nology because there is no license needed. JavaScript together with HTML and CSS are among the most important parts in the web development. An-other reason to use JS is using external graphic libraries for visualization of results into charts. Developers in ACT Honeywell use HighCharts [29], JS library to model uniform charts.
5.3.3 XML, XSD and XSLT
XML is well-known abbreviation for Extensible Markup Language. This metalanguage is used to define documents with a standard format. It is supported by various programming languages and can also serve as a small database. All the tools used for this project support this format without any problem.
XSD(XML Schema Definition) is a recommendation of the World Wide Web Consortium (W3C) which specifies how to formally describe the ele-ments in an XML document. This thesis uses the technology as the valida-tion of the users‘ input. It is easy and fast validavalida-tion opvalida-tion which helps to avoid most of the incorrect inputs already in the beginning.
XSLTmeans Extensible Stylesheet Language Transformations and is able to transform XML documents to other forms or formats. Due this fact, this transformation feature is used to create a generic XML file from various tools outputs.
5.3.4 Active Objects
Active Objects is the Object Relational Mapping (ORM) layer in Atlassian‘s products which was designed to simplify the work with the database. Active Objects use dynamic interface proxies as entities to avoid writing beans and to work with database objects more comfortably.
5.3.5 FreeMarker Template Language
FreeMarker Template Language (FTL)[30] is a free template engine, licensed under the Apache License. FreeMarker is a tool which generates the text output based on the templates. The developer prepares the data to display and FTL engine just fills the variables from Java program into the FTL tem-plates. FTL is designed primarily for servlet-based applications following the Model View Controller (MVC) pattern. Due to this fact the designers and the programmers are often split to the separate groups of developers and
they will not touch the logic nor the code of the other group. If the designer and programmer is the same person, this pattern helps him to maintain the product. FTL is not an application for end-users but something that programmers can embed and use in their programs.
5.4 Data Flow
Figure 5.1 shows the data flow of extension and user‘s communication with the used tools through the Bamboo. Following text describes each of the four steps in detail.
Figure 5.1: Data Flow
Inputfrom the user is the first part of the data flow diagram in Figure 5.1. The user‘s input file has gone through several design changes. The result solution is available in Figure 5.2. Final proposal allows user to configure the plugin with three options:
• Use of predefined configuration.
• Upload of the XML file.
• Editing of current configuration.
Predefined configuration reads the XML files from the particular directory and offers them to the user in drop-down menu. Example of the input XML file was shown in Figure 4.4. The validation of the XML input file is per-formed by the XML Schema Definition (XSD) file to avoid basic failures and therefore failed builds. There is a more user-friendly possibility to configure the plugin via Graphical User Interface (GUI).
The object design and entities are created after the parsing of the in-put file. Every assignment to the Active Objects must be in the transaction for the data persistence. Active objects (Section 5.3.4) represent the Object-Relational Mapping (ORM) database while the entities are created.
Triggerstep (Unit Tests or Code Coverage analysis) is running when the build is started. Command execution usually produces the data from the used tool. Bamboo provides command execution by Windows PowerShell or by the command line. Designed solution uses the Windows PowerShell because it allows pipelining of the commands. The plugin supports two types of execution. Local machine is able to run commands from command line and remote machines via REST web-services.
Resultshave to be transformed in third phase. Multiple tools generate multiple output files. Various formats of the output files are generalized by XSL Transformation. All JSON outputs are transformed to the XML files at first and after that all files are transformed to the generic form.
Visualizationtakes the last step of the process. This part collects the data from the back-end and displays them on the user‘s screen with the help of FreeMarker Template Language (FTL) [30]. More details about visualization are provided in Section 5.6.
5.5 Analysis of Potential Problems revealed during the Implementation
Problems might occur when the used tool does not produce output of the analysis in the form of the XML file. The proposed solution does not support such tools.
Another problem might be using a tool not supporting command line operations. In general, most of the tools from xUnit frameworks supports command line by default therefore the threat is limited.
In case the Unit Testing tools produce XML file output with structure different from the one from the xUnit frameworks family, the transformation into generic the XML file may not be possible at the moment.
The performance problem had been discovered during the testing part of the development. Parsing of a long XML document takes too much time. The investigation shows that the document with 6500 unit tests takes 7 minutes to get parsed. Solution was found quickly and the removal of the child after the reading in every loop solved the problem. Moreover it takes only two lines of the code. Such simple improvement of the code reduces parsing time from 7 minutes to 20 seconds.
Node node = testSuites.item(i);
node.getParentNode().removeChild(node);
Listing 5.1: Cutting the list
5.6 Presentation Layer
The idea is to keep the same look-and-feel as the original Bamboo applica-tion for the consistency.
Mock-ups for the purpose of this master thesis are created in Balsamiq Mockups tool. Mock-up prototype of an application is in Figure 5.3 and a few comments occurred from the users. None of them were critical. Mock-up prototype displays all the data on one tab.
Input data for xUnit frameworks is from thetestdirectory of the project and Code Coverage analysis takes the data from thesource codedirectory. Due to this fact ACT decided to split the results view into two separate tabs called Unit Tests and Code Coverage and therefore it does not fit with the proposed mock-up in Figure 5.3. Unfortunately this occasionally happens in the business because unexpected findings are detected during further analysis or implementation.
Figure 5.3: Build Results Mock-up
The screenshots of the real application are in Figures 5.4 and 5.5. The upper part contains the most important information, numeric results and charts and lower part contains the table with the package structure and its results distribution. The results can be filtered by test status and alternatively by used tools.
The ACT department suggests the idea to put the results history to the plan information and therefore more space on the results screen will be available.
Visual Flow
Human Computer Interaction (HCI) teaches to use visual flow as a user-friendly property of the applications. Grid is a basic principle for screen composition and visual consistency (repetition of the signs used). Computer literacy is used in many applications because the users are trained to use the learned operations such asclose the windowby red cross in Windows oper-ating systems. This behavior is caused by repeoper-ating the actions and objects used. These patterns are also used in extensions of Atlassian products in Honeywell ACS for the filtering as a clickable hand cursor.
