}w!"#$%&'()+,-./012345<ya

(1)

MASARYKUNIVERSITY

FACULTY OFINFORMATICS

}w !"#$%&'()+,-./012345<yA|

Metrics and Indicators on Software

Testing Process Effectiveness based

on Requirements Coverage and

Traceability Information

MASTERTHESIS

Jan Dolecek

(2)

(3)

Assignment

(This page will be replaced by official assignment description from the university)

(4)

(5)

Declaration

Hereby I declare, that this thesis is my original authorial work, which I have worked out by my own. All sources, references and literature used or excerpted during elaboration of this work are properly cited and listed in complete reference to the due source.

Advisor: Ing. RNDr. Barbora Bühnová, Ph.D.

(6)

(7)

Acknowledgement

I would like to thank my supervisors in Honeywell, in particular Ronny Kolb and Ste-fan Topp, and my advisor at the university Bára Bühnová, who all guided my work. Without them this thesis could not have been completed.

I would also like to thank all my colleagues, lead by Tomáš Hrdliˇcka, who I worked with at Honeywell and who kindly supported me all the time.

(8)

(9)

Abstract

Testing discipline is a very important part of software development process. It helps to improve and then retain quality of a software product, and also makes it possible to measure required quality. When done properly, it also speeds up development and brings more confidence to developers. Properly written and rigorously executed test suites help ensuring that all the required functionality is implemented and is working according to specification. The best way would be automated tests with automated test execution, which can be repeated without any manual work. Or, even a manual test is sufficient when it is executed by a tester before product release.

Software testing can not prove that a product has no defects at all, but with each test case a potential defect or class of defects can be detected and avoided. It is never possible to reduce the number of defects in the final product to zero, but we should be always reaching for it and getting as close to this target as possible.

This thesis was written within Honeywell Automation and Control Solutions (ACS), namely in the software excellence team called SoftCo.

(10)

(11)

Keywords

requirements management, software testing, effectiveness, metrics, measures, indica-tors, CMMI, traceability, requirements coverage

(12)

(13)

3.1 Overview . . . 27 3.2 Life-Cycle Models . . . 28 3.3 Requirements Discipline . . . 29 3.4 Test Discipline . . . 30 3.4.1 Activities . . . 31 3.5 Traceability Information . . . 36 3.6 Tools Suite . . . 38 3.6.1 Contour . . . 38 3.6.2 Jira . . . 40 3.6.3 Tools Integration . . . 40 4 Data evaluation . . . 43 4.1 Data Sources . . . 43 4.2 Available Tools . . . 45 4.3 Data Overview . . . 45 1

(14)

4.4 Traceability Information . . . 47

4.4.1 Traceability in Practice . . . 47

4.4.2 Requirement–Test Case Links . . . 48

4.4.3 Link Chains . . . 49

4.4.4 Requirements Coverage . . . 50

4.4.5 Product Features Coverage . . . 51

4.4.6 Unassociated Test Cases . . . 52

4.4.7 Links to Jira . . . 52

4.4.8 Suspect Flags . . . 53

4.4.9 Failing Tests for Requirements . . . 55

4.4.10 Defect Solving Progress . . . 55

4.5 Summary . . . 56

4.6 Areas for Improvements . . . 56

4.6.1 Stakeholders . . . 57 IV PRACTICAL CONTRIBUTION 59 5 Dashboard Definition . . . 61 5.1 Metrics Definition . . . 61 5.1.1 Requirements Coverage . . . 61 5.1.2 Test Status . . . 62 5.1.3 Defect Status . . . 62 5.1.4 Unassociated Requirements . . . 63

5.1.5 Unassociated Test Cases . . . 63

5.1.6 Tests to Re-run . . . 63

5.1.7 Links to Defects . . . 63

5.1.8 Product Feature Outcome . . . 63

5.2 Traceability Dashboard . . . 64

5.2.1 Dashboard Principles . . . 64

5.2.2 Alternative View . . . 66

5.2.3 Unassociated Requirements and Test Cases . . . 66

5.3 Summary . . . 67 6 Implementation . . . 69 6.1 Prototype Overview . . . 69 6.2 Employed Technologies . . . 70 6.3 Architecture . . . 70 6.3.1 Workflow . . . 70 6.3.2 Data Sources . . . 71

6.4 Traceability Dashboard Views . . . 72

6.4.1 Product Features View . . . 72

6.4.2 Requirements Folders View . . . 73

6.4.3 Tags View . . . 73

6.5 Unassociated Requirements & Test Cases Browser . . . 73

(15)

6.6 Summary . . . 74 7 Evaluation . . . 77 7.1 Form of Evaluation . . . 77 7.2 Questionnaire . . . 77 7.2.1 Chosen Projects . . . 78 7.2.2 Stakeholders . . . 79 7.2.3 Questions . . . 79 7.3 Results . . . 81 7.4 Summary . . . 81 V SUMMARY 83 8 Conclusion. . . 85 8.1 Provided Metrics . . . 85 8.2 Future Work . . . 85 3

(16)

(17)

PART I

(18)

(19)

Chapter 1

Introduction

This chapter describes the motivation of this thesis, introduces broader context and scope of the project and outlines overall structure of the thesis.

Software testing is essential to ensure quality of software, i.e. that it conforms to customer’s requirements and needs. Testing is time-consuming – it normally consumes 30–50% of project development time and cost (for safety critical applications, it is even higher). Therefore, improvements of both efficiency and effectiveness of testing process is critical for successful product.

1.1 Context

This thesis was written within Honeywell Automation and Control Solutions (ACS), namely in the software excellence team called SoftCo.

Honeywell is a Fortune 100 company focused on many aspects of technology. It has more than 132,000 employees worldwide, including 20,000 engineers and scien-tists. The company is divided into four Strategic Business Groups (SBGs) based on their targeted market segments [8]:

• Automation and Control Solutions (ACS) • Aerospace

• Transportation Systems

• Performance Materials and Technologies

These groups are sub-divided into Strategic Business Units (SBUs).

ACS provides its customers with environmental controls, life safety, security, sens-ing, scannsens-ing, and mobility products, as well as building and process solutions. Their products currently perform in 120 million homes, 10 million buildings, 5 thousand in-dustrial facilities, and support hundreds of public and private utilities worldwide.

ACS teams and individual team members are spread all over the world working on many different projects. The projects differ in size as well as in specialization. There are pure software projects just like projects focused on development of embedded so-lutions.

(20)

1. INTRODUCTION

SoftCo is software excellence team with headquarters in Rolle, Switzerland. It was founded within ACS business group as a support for other teams, with the mission to coordinate efforts in software development across all ACS SBUs. SoftCo’s respon-sibilities are software process improvements, best practices, coaching, and mentoring. SoftCo also configures, administers, and supports the ACS Tools Suite.

The ACS Tools Suite is a set of recommended software engineering tools which help teams during development activities. Among the activities of SoftCo is also the integration and development of extensions for these tools. This topic is described in details in Chapter3.

To improve software projects development effectiveness and efficiency and product quality across ACS, SoftCo analyzed CMMI guidelines [3] and adapted them for the specific needs in ACS. The result of this adaptation is custom development process called ACS Software Development Process (ASDP) [18].

Other ACS teams are internal customers of SoftCo team, using its products and services.

Work on the thesis was carried out in SoftCo team, in its division in Brno, Czech Republic.

1.2 Motivation

Test management differs across ACS teams, usually dependent on their experience, do-main and technology. Very important factor is also knowledge of the tool, since there may be many features not known by the team, thus decreasing efficiency and effec-tiveness of testing. On the other hand, the tools themselves do not solve every possible problem. Although problems in Honeywell are usually very specific and the tools are customized, they usually do not solve all needed questions of the test manager, or may not solve them efficiently.

Motivation for this thesis is to find the problems stakeholders have regarding to soft-ware testing and develop metrics and indicators which would help solving the prob-lems more efficiently (faster and with less resources) and more effectively (better, more precisely). Eventually, existing tools may be adapted, or a new tool proposed.

Common problem is lack of overview on the whole project. This is an inevitable result of big projects being created by a group of people. These projects are evolving rapidly as teams try being agile. At such a fast pace it is easy for a single person to loose track of what is really happening in the project. The tool should maximally help and provide up-to-date overview of the whole project, with very high-level and brief information listed first. Should the user want more detailed info, he shall be able to drill down into more details via several levels of project decomposition.

(21)

1. INTRODUCTION

1.3 Scope

The thesis focuses mainly on defining and implementing indicators related to Trace-ability Information, i.e. links between Product Features, (Non-)Functional Requirements, Test Cases and Defects. This is not the only useful indicator to accomplish tasks of this thesis, but having more indicators discussed is beyond the scope of this thesis. There-fore, after initial analysis and discussion with SoftCo team, only some indicators and metrics were chosen.

Because of the complexity of the original topic, it has been divided into two parts and thus two distinct theses. The other thesis is being written on complementary topic, Indicators and Measures on Test Effort and Test Progress, by my colleague David Schimmel. Some work on these theses has been shared, mainly in the beginning of the research and data analysis. These two theses are related and defined prototypes have links to each other. Ideally, they both should be incorporated into ACS tools.

Schimmel [17] claims that effort on testing is often estimated poorly, which may lead to harmful effects – testing may not be finished by deadline, or some tests may be skipped because of exceeded budget. He aims to provide managers with time related metrics, which shall help to establish better estimates on test planning and thus prevent such problems.

1.4 Goals

Main goals of this thesis were identified before the actual work the thesis started: • Perform state-of-the-art surveyon test management and related indicators • Analyze state-of-the-practiceregarding test management and related

measure-ment in ACS

• Identify information needs of project and test managers regarding test planning, execution, tracking, and reporting

• Extract, transform and load (ETL) data from source application(s) • Definerelevant indicators, measures, reports, and charts

• Defineand implement data collection mechanisms and procedures

• Develop prototype for collecting and visualizing defined indicators, reports, and charts

• Documentdesign of implemented solution for data analysis and visualization • Test and evaluate defined indicators and implemented extensions with selected

data from ACS projects and representative users

(22)

1. INTRODUCTION

• If necessary refine indicators and prototype These main goals of the prototype were identified: • provide the managers with quick overview of project • provide ability to drill down to details

• recommendationswhat tasks to perform (effectiveness) • information how well tasks are being performed (efficiency)

1.5 Unity Project

There are many tools used among ACS teams. Fields of responsibility of these tools naturally overlap, that implies some data is redundantly stored and managed by more tools. This brings overhead on one side, as teams need to enter the same data twice (or even more times) into separate systems, and on the other side this approach is also very error-prone. Someone may forgot to update the data in one tool after he changed it in another one; or he may actually not even know about the change needed in another system, as that is not part of his responsibility.

For these reasons, shared data should be synchronised or linked between projects using tools integration. These are developed by SoftCo as the needs are very specific to Honeywell and ACS. Project Unity is an umbrella project for all these connectors. There are many connectors needed to be developed for full integration, and they are being developed incrementally.

Final implementation of the metrics and indicators proposed by this thesis shall be implemented under this Unity project.

The tools are shown in Figure 1.1. Regarding to testing, most important are two tools: Contour and Jira. Contour is a requirement management tool which also helps to manage and execute tests. When a defect is found in product during test execution, it is recorded into Jira issue tracker. Connection between these two systems is most critical and thus was started first. It was highly requested feature by ACS teams and it went into production in August 2012. Since then, it is possible to establish fully automatic bi-directional links between test runs in Contour and defects in Jira.

1.6 Methodology

Figure1.2shows the followed methodology for this project. The main activities map to the previously stated goals.

1.7 Thesis Outline

This document is divided into several parts, sub-divided into chapters: 10

(23)

1. INTRODUCTION

Figure 1.1: ASDP project tools and portals

Figure 1.2: Methodology overview • Part I contains introduction to the master thesis.

• Part II shows the issues in Test Management as described in literature and exam-ines metrics and indicators proposed by or used in companies across the world. • Part III describes state-of-the-practice in Honeywell ACS. It starts with descrip-11

(24)

1. INTRODUCTION

tion of internal processes, mainly with ASDP and its Testing Discipline in Chap-ter3. After that, real data recorded by ACS teams are analyzed in Chapter4, which discusses data quality together with several issues identified and possi-ble solutions. The data analysis is performed on a management tool called Jama Contour.

• Part IV is divided into three chapters, as follows:

– Chapter 5 defines the useful metrics properly and places them into a dash-board.

– Chapter 6 discusses implementation choices for a working prototype of the proposed dashboard.

– Chapter 7 talks about the performed informal interview and validation among several ACS teams.

• Part V provides a general conclusion on the work and suggest future improve-ments.

(25)

PART II

(26)

(27)

Chapter 2

Research & Literature Review

This chapter gives overview of current state-of-the-art in software testing. It explains what Test Management is and why it is important. Then it follows with reasoning about importance of measurement and metrics in software development, eventually present-ing several interestpresent-ing metrics found in the literature.

2.1 Test Management

The director asked the tester, “So you tested it? Is it ready to go to pro-duction?” The tester responded, “Yes, I tested it. It is ready to go.” The director asked, “Well, what did you test?” The tester responded, “I tested it.”

— Marnie L. Hutcheson [7]

The ultimate purpose of the test management is to ensure that the software, system or product under development behaves as expected and meets the business and techni-cal requirements that guided its design and development. Testing activities evolved in many companies over time as their maturity grew, especially when dealing with new big projects or big changes in existing projects (e.g. when preparing for euro) [16].

Most companies start with intuitive testing. Development teams work on a product and when they think it is ready for release, they present it to a team of testers. Testers usually do not need to be professionals but rather co-workers from another department, or appointed from a client. These people “play” with the product and try to perform their usual tasks and if no major issues occur, the product gets released to production environment.

There is however no or very little opportunity for any adjustments if major problems or inconsistencies with the documentation are found, because the project is already in its final stage. Expectations of end users may vary, so it is not easy to tell which issues are critical. Also, there is no test documentation and therefore it is very difficult to reproduce a problem if found, because testers may not remember which tests were actually performed. However, the most critical issue is that some parts of the system were not tested at all, just because nobody knew they needed to be tested. There is 15

(28)

2. RESEARCH& LITERATUREREVIEW

no record what has been tested and what has not. Such testing provides insufficient insight into quality of product being delivered. Therefore, such approach cannot be recommended and luckily is not performed in most companies nowadays (though all companies I worked for took exactly this approach and, as far as I know, they continue doing so) [16].

Next step in testing maturity evolution is based on documentation. Testers begin col-lecting documentation for the product under development and for each piece of func-tionality they write a test case, which contains clear steps how to proceed. Testers should start writing test cases as early as some functionality is finished, and continue over time, until everything is covered. However, they usually end when their time runs out, not depending whether all needed test cases have been already written [16]. This ap-proach has many improvements when compared to intuitive testing. First, testing is documented by test cases and thus it is clear, which parts of the product have been tested, how thoroughly and what exactly has been tested. If an issue is identified later, relevant test case can be added. Also, as test steps are recorded, issues found during testing can be easily reproduced and thus it is easier to fix them. The test manager has much more information about performed testing tasks, than in case of intuitive testing, and thus can be more confident with his decisions and recommendations. However, he may not be sure that tests covered every functionality, especially the most critical parts. A subsequent improvement and next step in test management evolution is testing based on requirements. Requirements specification is critical for project development and there are many papers studying and improving maturity of requirements management [2]. Requirement specification shall cover all important parts of the product, also with pri-orities, and thus testers can base the test cases on these requirements and make links between them. When such links are created, manager can easily confirm that all require-ments are covered by test cases and thus the requirerequire-ments specification as a whole can be tested properly [16].

Possible problem can be caused when requirements are not stable and change over time [2]. If a test case had been written before relevant requirement was modified, it is possible that it is not properly tested anymore. These changes must be tracked and all associated test cases must be reviewed after requirement changes. Also, no tool can automatically confirm that one particular requirement is completely tested by a test case. There must be a human confirming, that nothing had been forgotten in the test case description.

It is also important to look at testing from another point of view, which is repro-ducibility. Intuitive testing mentioned earlier is usually not reproducible, i.e. it is not possible to repeat a test exactly as it was done before. Reproducibility is a measurement of the precision of a test method. Reproducible test case must be recorded in some form of documentation or code, which can be followed. When a tester runs a test twice and he precisely follows the documentation, he shall always obtain the exactly same results.

Good reproducible test cases are [6]:

• Isolated– the test should be runnable in isolation. Tests, that expect the environ-16

(29)

ment to be in a particular state when run, violate this premise.

• Standalone– the test should be as stand-alone as possible. Ideally the test should be a single class (or package) that can be imported into a developer’s workspace. If necessary the test might also be parceled up in a new project.

• Lean– the test should be as minimal as possible. This means that it should be as simple as possible and it should introduce a minimal set of dependencies (ideally none).

Special class of reproducible tests are automated tests. When a test case is described by executable code instead of human language, it can be followed by an interpreter and thus reproduced easily. The code does not need to be written in an inherently program-ming language like C++ or Java. There are special Domain Specific Languages (DSLs) intended for recording test cases.

These days, most commonly used DSLs are RSpec and Cucumber. RSpec is a library for Ruby programming language which defines Ruby based DSL for defining test cases. On the other hand, Cucumber defines a plain-text based language anyone can under-stand easily. Figure2.1shows an example test case for a hypothetical calculator1. Feature: Addition

In order to avoid silly mistakes As a math idiot

I want to be told the sum of two numbers Scenario Outline: Add two numbers

Given I have entered <input_1> into the calculator And I have entered <input_2> into the calculator When I press <button>

Then the result should be <output> on the screen Figure 2.1: Example of Test Case written in Cucumber DSL

Once automated test cases are written, they can be executed entirely by a com-puter and do not need any human interaction at all. They can be also executed with marginally almost no cost and in a very short time. Naturally, idea of executing the whole test suite as often as possible comes in mind. This idea created a concept called Continuous Integration [4]. Usually, the company or team has a build server which runs the test suite after each commit to Version Control System, or regularly (e.g. every night). With this technique, defects can be spotted very quickly.

1. Downloaded from https://github.com/cucumber/cucumber/blob/master/examples/ i18n/en/features/addition.feature

(30)

2.2 Measurement

Would you hire a carpenter who did not have and would not use a tape measure? Probably not, because the carpenter who does not use a tape measure probably will not deliver a satisfactory job. Most people recog-nize readily that measuring tools are necessary to make sure that a struc-ture is laid out according to the plan, that the floors are level and the walls plumb. Yet we buy software to do important work that has been devel-oped and validated by people who do not use any type of measurement. — Marnie L. Hutcheson [7]

When the whole requirements specification is properly recorded along with test cases, and when these are linked, processes and algorithms can be defined to measure quality and validate consistency of these specifications. The results are several mea-sures, metrics or indicators.

The goal of using metrics is to measure your project’s success, to know whether everything is running according to plan, and if it is not, what might be done to correct it [15].

These terms are described in the following chapter. 2.2.1 Measures, Metrics, Indicators

The terms measures, metrics and indicators are often used interchangeably and incor-rectly. In scope of this thesis, these terms are properly defined.

When you obtain/observe/measure a value of directly observable property of an entity, you have a measure. Each measure has a standard unit of measure (UOM) like sec-onds, meter, kilograms etc. In software development and software testing, most com-monly used measures are:

• Number of Defects found in a system or component • Lines of Code (LOC, kLOC)

• Number of Test Cases

A metric, in contrast, is a derived value which cannot be observed/measured di-rectly. It is a number derived from one or more measures by a formula (or estimation). Best known metrics in software development and software testing are:

• Number of defects found per kLOC, which serves as an estimation of quality of code

• Productivity, i.e. Size / Effort 18

(31)

• Defect Density, i.e. number of defects related to size

There are two types of metrics, objective and subjective. Objective metrics can be quantized and are readily available, subjective metrics rely on opinions, gut feelings, personal attitudes etc. An example is CSAT (customer satisfaction), though the former is more reliable, than the later, the reliability of subjective metrics can be improved by having checklists and guidelines. For e.g.: survey question need to have, probe areas, facets, scale definitions before an option can be chosen.

An indicator is “a thing that indicates the state or level of something” 2_{, thus it}

can be simply just a number showing value of a particular measure or metric. A better indicator could be a chart comparing two measures/metrics or projecting how a mea-sure/metric developed during a time period. Also, a semaphore where red means bad and green means good is also a very simple indicator, which can be helpful in particular class of situations. Thus, indicator is most general of these terms.

2.2.2 Requirements on Metrics

When it is possible to measure something, it can be comprehended in a better way and more can be known about it. With better comprehension, there comes better chance in attempts of improving it.

Effective metrics must be simple, objective, measurable, meaningful and have easily accessible underlying data [9]. The most important attributes of a metric are summa-rized in this section.

• Simple– so that errors in computation or interpretation are avoided, and also that the metric can be comprehended.

• Objective– based on goals and objectives of the company.

• Measurable– based on things which can be reliably measured, not estimated or guessed.

• Meaningful– so that they can help the managers to understand important as-pects of their projects.

• Easy to collect– metric shall be automated and non-intrusive, i.e. not interfere with other activities of the developers.

• Easy to interpret – so that it is easy to comprehend it, understand the causes which affect it.

• Hard to misinterpret– even when the metric may be easy to interpret, there may be cases leading to wrong conclusions. Metrics shall be designed with caution so that such cases are avoided.

2. Oxford Dictionary - "indicator" -http://oxforddictionaries.com/definition/english/ indicator

(32)

• Valid – to ensure, whether the metric really measures what it is supposed to; prevent systematic errors.

• Reliable(Consistent) – they must perform their required functions under stated conditions for a specified period of time, providing consistent results; prevent random deviations in measurement.

2.2.3 Efficiency and Effectiveness

These two terms are often used interchangeably and incorrectly, which causes confu-sion. In some languages (e.g. Czech) these are even translated to just single word which makes it hard to differentiate between the two terms. There are several articles about the differences of these two terms, some of them confusing as well. For the sake of reading and understanding this thesis, the reader shall properly understand these terms and be able to recognize the difference [14].

Efficiency is the ratio of provided input and useful output. It is a measure of the resources (money, time, effort, etc.) that are needed to produce a desired result. Being efficient is about doing the things right.

Effectivenessdescribes impact of your decisions. It represents the degree to which something is successful in producing a desired result. Being effective is about doing the right things.

Efficiency and effectiveness depend on each other. Usually, when improving one, the other worsens. Therefore it is essential to focus on both.

2.2.4 Reliance on Metrics

Metrics and indicators usually play an important role in software quality and matu-rity. They provide a quick overview on the project and the direction it is moving. With metrics it is simpler to target improvements and asses them.

However, metrics can have also negative impact on developers and the code itself. The problem occurs when people forget their original goal and fall into false proxy trap (quoted below). Usually it is not possible to directly measure quality of a product, so we choose to measure something much easier – e.g. number of defects found. This measure becomes an approximation of the original measure we targeted for.

Once you find the simple proxy and decide to make it go up, there are lots of available tactics that have nothing at all to do with improving the very thing you set out to achieve in the first place. When we fall in love with a proxy, we spend our time improving the proxy instead of focusing on our original (more important) goal instead.

— Seth Godin [5] 20

(33)

Gaming the system to improve metrics is not the goal people shall focus on. Though, they usually do, because it is easier for them and much easier for their managers. Such metrics are not useful, or the worse, they can lower the quality of final product. They can also falsely increase confidence the manager has about the quality of his product. The metrics themselves do not solve problems, they help to avoid them when used properly.

Description of several metrics follows, which, when blindly followed and/or im-proved, may and probably will cause issues to the project:

• Code Coverage by TestsTest cases are written to cover use cases. When there is a piece of code not covered by any test case, it means either it does not need to be there or that there is a use case not recorded in specification and not tested. It makes no sense to add a new test case only to cover a line of code, if it does not cover the whole use case. On the other hand, some pieces of code are so simple that they do not need to be tested (simple constructors, getters, setters) – errors in these should be found by other tools, like heuristics in static code analysis. • Number of Test CasesThe more critical part of the system or the more critical

use case, the higher likelihood there should be that it is properly covered by test cases. When a team’s goal is to write e.g “ten test cases a month”, they will choose the simplest parts to test – parts so simple they do not even need to be tested at all, or parts which are not critical for the software. In such a case, it might have been better if only two test cases were written instead while covering critical use cases and covering them properly.

These two are just simple examples. Similar situation can happen with any metric or indicator if a team starts blindly following them. Keep your goal in mind and think if you are aiming for it. A metric can simplify this for you, but human judgement is still important. Educate team about real goals and how to follow them. Make sure they understand these goals and will not follow the metrics blindly.

2.3 Selected Metrics

This section describes several metrics found in literature, which seem useful for use in ACS. These are described and their relevance to SoftCo is discussed. Possible problems are also outlined, which may occur if the metric would be followed in a narrow focus. These problems are usually not described in the literature found.

Requirements Coverage by Test Cases

Requirements Coverage by Test Cases [9] solves problem with ever increasing banks of requirements and test cases. Bidirectional links need to be established between require-ments and test cases - for each requirement, there is at least one test case which checks that the requirement is implemented properly and thus the product works as expected. 21

(34)

Also, each test case must be linked to at lest one requirement, so that it is clear, why it exists. If it does not test a requirement, it does not need to be in the test repository in the first place. If a test case exists because of a requirement but they are not linked, a prob-lem can occur when the requirement is amended or removed. In such a case, related test cases need to be amended to reflect new state of the requirements, or it needs to be deleted. If this does not happen, there is inconsistency in project specification.

This metric displays the amount (percentage) of requirements which have associ-ated test cases. It also provides the project manager with confidence on the product quality and acts as an important indicator of effectiveness of testing team.

Possible problem can occur when a requirement is not fully covered by linked test cases. E.g a requirement can be complicated and test cases cover only a part of cases which can possibly happen. Such a situation cannot be covered by this metric. Solution exists on a different layer - someone responsible for that particular requirement shall validate completeness of test cases and update status of that requirement, making it clear it is tested completely.

Test Coverage

Test Coverage [9] provides the project manager with progress and state of testing, dis-playing number of test cases executed. It can be displayed for the project as a whole (i.e. overall test coverage) or per component/project feature. This metric shows number of executed tests in relation to all test cases recorded in the system (resp. in a compo-nent/feature). Example is shown in Figure2.2.

Figure 2.2: Test Coverage metric (Overall)

Possible problem can occur when test cases are still being written and thus the final number of them is not known. In such case, this metric can provide high numbers only because the test bank is not complete. Solution is in combination with previous metric, requirements coverage by test cases, in combination with which it provides better overview of tests already executed.

(35)

There may be more distinct states in which a test case can be (than just passed/failed), which would result into more detailed Test Status Breakdown. It may be interesting to show also this detailed breakdown, with following test cases states:

• Not Scheduled– test case had been written, but not yet scheduled for execution nor assigned to a tester.

• Scheduled– manager scheduled the test case for execution and assigned it to a tester.

• In Progress – a test is being executed at the moment. This state may be useful for long-running tests.

• Blocked– test execution had to be paused for some reason (e.g. part of the testing infrastructure is not working at the moment). Tester will continue on this test when the problems have been resolved.

• Passed • Failed

Component-wise Defect Distribution

Component-wise Defect Distribution [9] tries to answer which components/features are most problematic, i.e. have most defects found. Example is shown in Figure2.3.

Figure 2.3: Component-wise Defect Distribution

Defect Status

In addition to Component-wise Defect Distribution, defects may be broken down by status. It provides additional information about which parts of the software need to be 23

(36)

fixed by developers (when the defects are open) or which needs to be focused on by testers for re-testing.

Typically, a defect passes through several states:

• New – Defect has just been recorded. It needs to be validated, whether it con-tains all relevant information or whether it is not a duplicate. When not properly specified, the issuer shall provide any additional information required.

• Open– Defect has been recorded and validated and can be assigned to a devel-oper for resolution.

• In Progress

• Resolved – Developer resolved the problem. It needs to be tested again by a tester.

• Closed– A tester has confirmed, that the defect is no longer present. Summary

This section described several metrics found in the literature, which seem potentially useful for being implemented in ACS projects.

(37)

PART III

(38)

(39)

Chapter 3

ACS Software Development Process

This chapter describes ACS Software Development Process (ASDP), a customized software development process used in Honeywell ACS. It is mandatory for individual teams, and thus any proposed metrics and indicators shall be based on its work products. This chapter also describes the tools recommended by SoftCo, namely Contour for require-ments and test management and Jira for issue tracking.

3.1 Overview

ASDP [18] is a set of rules, schemes and templates which are required to be followed by Honeywell ACS teams when dealing with the software part of new projects develop-ment. It tells the teams what to do, not how to do it. The approach is up to them – if they provide desired work product. However, SoftCo provides some recommendations and best practices, together with recommended tools.

ASDP is highly inspired in CMMI1 and UP2, but tries to map closely to Honey-well’s specific position and its needs. ASDP incorporates best practices and guidance for successful software development.

Full ASDP documentation is available on Honeywell’s internal network and is not available to public, therefore this thesis describes its fundamentals to the readers out-side of Honeywell.

High-level overview on the project development is depicted in Figure3.1. It is di-vided info several disciplines (on Y axis in the chart) and passes through several phases (in X axis), which are mapped to New Product Introduction (NPI) process3. The chart shows rough work intensity for each individual discipline in a particular phases. The work intensity is based on the data and observations from previous ACS projects.

When a project is about to proceed to the next phase, an exit-gate check must be passed. This check validates if certain criteria are met and thus required level of quality of the product is ensured.

For the scope of this thesis, most important are two disciplines: naturally Test Disci-pline, but also Requirements Discipline simply because testing is based on requirements, as stated in previous chapter.

1. Capability Maturity Model Integration 2. Unified Process

3. Complete process of bringing a new product to market

(40)

3. ACS SOFTWAREDEVELOPMENTPROCESS

Figure 3.1: ASDP lifecycle overview - activity intensity

Requirements Discipline is very important for testing, because work products of this discipline are essential for creation of test specification. It is described in more details in the following sections.

Test Discipline covers the main focus of this thesis. The requirements ASDP puts on individual teams regarding this discipline is described in more details in following section, together with the reasons and its relation to Requirements Discipline. Activities of Testing Discipline slowly start in planning phase, where plans for testing are created. Most of testing activities however take place during development and validation phases, during which test assets (Test Cycles) are created and executed respectively.

Because ACS projects differ a lot from each other and some are much larger than others, it would be unreasonable to require the same procedures and level of details for all of them. For small projects, some deliverables can be delivered in reduced form or omitted completely. ASDP clearly specifies it in ASDP Tailoring, where it lists all deliverables as specified by ASDP together with level of their need. Each part of the process and deliverable can be mandatory, which is a default value for full ASDP, reduced or optional. Thanks to tailoring small projects can focus more on their main goals and not on the paperwork the processes bring with them. More on ASDP Tailoring can be found in full ASDP documentation.

3.2 Life-Cycle Models

Looking at the chart activity intensities in Figure3.1, ASDP seems very much like wa-terfall life-cycle model. It however promotes itself not to support only wawa-terfall, but 28

(41)

wants to be life-cycle model independent. ASDP puts more focus on what needs to be done, i.e. inputs and outputs, and not how it needs to be done. Though, most projects have been developed using waterfall method so far.

New documentation about Agile ASDP is being created while this thesis is being written. It is intended for teams using the ASDP and interested in adopting an Agile framework. It is a guideline which provides instructions on how to execute a software development project using ASDP with an Agile life-cycle model, namely the Scrum process framework.

It may be interesting to compare ASDP to different models used in other large com-panies. For example, when Google starts developing a new project, testing is one of the most important disciplines ever since early stages [19]. While Google designs the architecture of its new product, experienced testers are responsible for preparing a test-ing infrastructure so that integration tests can be executed as soon as small pieces of the product are being completed. This approach is heavily based on Test Driven Develop-ment (TDD).

In ASDP, on the other hand, testing is planned in early stages of development, but most activities are performed later, usually when the development is reaching to its finish.

3.3 Requirements Discipline

Because this thesis is focused on links and traceability information of test cases, we need to start with requirements engineering. Shall we have test management based on re-quirements, we first need to have requirements recorded properly, as stated in Chapter2. ASDP defines that requirements discipline is concerned with the elicitation, analy-sis, specification, validation, and management of requirements [18].

Requirement Discipline is highly related to Test Discipline, as testing evolves from recorded requirements.

The Requirements Discipline consists of several distinct goals. Figure3.2 presents a high-level overview how individual tasks are divided into several phases of project development. For this thesis, the most important part are deliverables (outputs) from these activities, namely Requirements Specification together with Traceability Information (Note that Traceability Information is described in detail later in this chapter).

The requirements of a product being developed are recorded in several item types, each with a specific goal, scope and targeted audience. The basic and most commonly used requirement types and relations among them are shown in Figure3.3.

• Product Features are useful and convenient way to describe the functionality of a product without getting bogged down in too much detail [13,18].

• Operational Scenarios describe proposed use of given system or product. Opera-tional Scenarios are best expressed as “casual” use cases [18].

(42)

Figure 3.2: ASDP Requirement Discipline and its Work Products

• Use Cases are a derived from Operational Scenarios describing the interactions with the product or system in more detail. They describe how the product will be used [18].

• Declarative Requirements are low-level refinements of product features, which de-scribe in detail how product shall behave. They can also dede-scribe internal states of the product which are not visible to the user but are needed for implementa-tion.

Figure 3.3 shows work products of requirements management according to ASDP, with relationship between individual document types. ASDP however defines a variety of subtypes of declarative requirements.

Requirements Discipline is very important for further development activities. It is then followed by Architecture, Design and Implementation disciplines. After these, Test Discipline comes. Therefore, requirements must be carried out properly.

3.4 Test Discipline

Test Discipline activities could be divided in time into two intervals. First occurs during development, where individual pieces are tested as soon as they are implemented. Sec-ond interval of testing occurs in Validation phase, i.e. after development has finished and just before product release or before project deployment to production environment.

ASDP defines that test discipline covers testing performed after the implementation is complete and prior to any kind of user acceptance testing with customers in the field. Because the developers may overlook their mistakes or may not see all the ways their product can be used, it is better when someone else than them performs the tests. 30

(43)

Figure 3.3: Hierarchy of ASDP Requirement Types

ASDP states that “this type of testing is typically performed by someone independent of the team that has done the development – this may include engineering test teams, as well as independent QA/Test teams”.

Test discipline is related to other ASDP disciplines. It closely follows Requirement discipline as test assets are generated to cover functional and non-functional require-ments, so that they can be validated as described in SoA chapter. All requirements should be covered as much completely as possible. Requirements discipline thus pro-vides input to testing.

3.4.1 Activities

Figure3.4shows individual activities of Test Discipline together with work products, which represent their output.

Planning

Similarly to other disciplines, also testing starts with planning. That includes high level overview on what will be tested, how and when the test are going to be executed. Also plan for test cases development is prepared. Individual tasks of Test Planning are described in detail in ASDP.

Test Assets development

After testing approach has been chosen and plan prepared, the most important part of Testing Discipline takes part. During this phase, test cases need to be generated. These 31

(44)

Figure 3.4: Test Discipline with Work Products

test cases serve as documentation for proper testing based on documentation as previously described in Chapter2.

Test Cases make sure that testing is deterministic and thus can be repeated, espe-cially that when an issue is encountered, it can be easily reproduced. Also, test cases provide documentation so that the manager can easily tell, and with confidence, what is being tested and what is not. This is very important so that situations like the one in preface Chapter2will not occur.

Figure 3.5 shows main tasks which need to be done during development of test assets as described by ASDP.

Figure 3.5: Activity Diagram of Test Assets Development

(45)

These Test Cases can be manual or automated. Of course automated tests are usually better because they can be repeated without marginally low cost and in a very short time. In spite of that, manual tests are very common, mainly because they are easier to write. Automated test cases would be generated through a test automation tool.

Thanks to all project documentation being recorded in one tool, it is easier not to forget anything. ASDP states that “test cases need to be generated for all types of testing identified in the Test Plan”.

Also stated in Chapter2 is that tests shall be based on requirements. When a tester is writing new test case based on one particular requirement (as recorded by Require-ments Discipline), he shall create a link between those two. With these links, traceabil-ity information is recorded and it extends traceabiltraceabil-ity information captured in require-ments discipline. Furthermore, ASDP states that “as part of developing test cases, trace-ability of the test cases to the requirements “must” be established”.

Test Cases need to be structured, so that they can be executed easily. Also, it makes it easier for test manager to verify correctness and completeness of such test case. A test case consists of small steps to be executed to check whether the software, system or product under test performs as specified or expected. Figure3.6shows how a Test Case is created in Contour.

While Test Cases are being written, they are also grouped into Test Suites so that they can be managed easily. Test Cases and Test Suites are being developed at the same time. Test Execution

Once Test Cases are written and organized into Test Suites, they can be executed to verify that the product satisfies elicited requirements. Test execution usually happens in several phases during product development, as features are added and can be tested and confirmed/verified.

Test Execution Plan is prepared in advance. It has list of test cases which need to be executed, assigns them to individual testers (resources), and outlines an expected schedule. Schimmel’s thesis [17] is more focused on this planning and how accurate the schedules are.

Figure 3.7 shows main tasks performed when tests are being executed, which is detailed view on the third step, Execute Test Cycles, of overall Test Discipline tasks presented earlier in this chapter.

Each time a Test Case is executed, the results and also time spent needs to be logged. This is based on concept of Test Runs (in Contour) – each test run means one execution, or scheduled execution, of a test case.

When a test is executed, it is expected either to pass when everything works prop-erly, or to fail when there is a defect. ASDP states that. . .

During the execution of tests, if it is found that there are defects, or changes are needed to be made in the test cases or suites themselves, these issues should be reported in the issue tracking system and tracked for

(46)

Figure 3.6: Contour’s definition of a Test Case artifact

ture resolution [18].

Jira is the recommended tool for logging these defects. Thanks to its integration with Contour, defects can be logged directly from Contour’s Test Run screen (as shown in Figure3.8). When this integration is used, a link is also created between newly cre-34

(47)

Figure 3.7: Activity Diagram of Test Execution Activity

ated issue in Jira and existing Test Run in Contour. This again extends the amount of Traceability Information recorded.

Figure 3.8: Log a Defect button in Contour

Test Closure Approval

Once test execution is finished, a report is created determining status of testing, i.e. which tests passed and which failed. It should also contain list of defects, which were created as results of failed tests.

The report confirms that the software, system or product under test behaves as de-scribed in the Requirements Specification [18]. This sentence from ASDP also implies the importance of tests based on requirements and the necessity of traceability infor-mation.

The report is usually generated by a software tool used by the team. But given the fact that a report view in the tool itself can act as the report work product, it does not 35

(48)

need to be generated at all.

3.5 Traceability Information

Traceability is essential to lager projects because it provides context for each item recorded in project, especially:

• why does it exist,

• what are its consequences, • what does it impact.

ASDP defines that Traceability Information work product represents bidirectional trace link information between requirements and other work products such as test cases or architecture design decisions [18]. Additionally, links can be created between test cases and defects, which is not however stated in ASDP, but it is important for proper traceability.

These links are directed, i.e. connecting an parent item with its subitems. ASDP requires certain types of links, which are shown in Figure3.9.

Figure 3.9: Traceability relationships among different work products

Links are transitive, i.e. when multiple links are chained, they provide longer im-plicit links which do not need to be exim-plicitly recorded in the system at all. Thus, they 36

(49)

can be used e.g. to trace each defect to a product feature, via test run, test case and declar-ative requirements, as depicted in the Figure3.10.

Figure 3.10: Chain of links

Implications

When testing is based on requirements (as mentioned in Chapter2), traceability informa-tion also helps to ensure that all requirements are tested. ASDP provides some rules to ensure this coverage:

• requirements trace backwards to higher level requirements and thus indeed sup-port the product’s vision and purpose,

• all stakeholder needs have been addressed with some derived requirements, and further trace forward to architecture design decisions,

• requirements are sufficiently covered by test cases.

With traceability information, every recorded item has its meaning in the specifica-tion.

Traceability information also helps with Change Impact Analysis where the evalu-ator investigates which elements are affected by a proposed change.

Also, thanks to traceability it is possible to see results for particular parts of the project. For example, if all defects are linked properly, they can be traced to high level product features and thus it can be determined, whether a given feature is ready for release or whether is still too buggy.

Exceptions

ASDP is somehow benevolent in the amount of traceability recorded in requirements discipline. It states that

Although tracing product features backwards to CTQs, CTQs to Stake-holder Needs, and StakeStake-holder Needs to StakeStake-holder Profiles is good practice, it is not mandatory to do so. It is assumed that these relation-ships are shown in the respective QFD documents.

(50)

Also, there are exceptions related to links between requirements and tests: Traceability needs to be maintained between Test Cases and Require-ments. However, in certain cases test teams may generate and document test cases that do not relate directly to any documented requirement. Traceability information does not need to be provided under such cir-cumstances.

Usage

Traceability information is very useful for validation of work on a product. Because it is required by ASDP and thus mandatory for all teams, it can be greatly used by proposed metrics and provide relevant information to managers.

3.6 Tools Suite

Continuing on Unity project description from Chapter1, more detailed overview of the used and recommended tools follows.

The ASDP Tools Suite is a set of third party software tools recommended by SoftCo to individual ACS teams. These tools cover most parts of software development process and it is SoftCo’s task to make them as compliant to and compatible with ASDP as possible. Note that usage of any of these tools is not mandatory to ACS teams, as long as they follow ASDP rules and produce all required work products.

These tools are configured and supported by SoftCo. The ones which are web- or server-based are also hosted and maintained by SoftCo.

However, because the tools are meant for different tasks, their field of responsibility is naturally connected and also overlaps. SoftCo is therefore continuously working on their mutual integration. Thanks to that, data would not need to be duplicated in mul-tiple tools, which would reduce the amount of work and also decrease chance of errors when data are changed.

Parts of the integration are already fully operational, while other are still being de-veloped.

Overview of the disciplines and provided tools is depicted in Figure 1.1 in Chap-ter1.

Regarding to requirements management, testing and defect tracking, and thus scope of this thesis, two of these tools are at utmost importance: Contour for requirements and test management and Jira for issue tracking. They are described in more details in this section.

3.6.1 Contour

Contour by JAMA Software is a recommended tool by SoftCo to be used by ACS teams for

(51)

• requirements management,

• test management and test execution.

It was chosen by SoftCo after evaluating several requirement management tools. One of the reasons for choosing Contour was also its simplicity when compared to other professional tools. Such a tool must be as simple as possible, so that users understand it properly. Otherwise, they would not be able to use it properly which would reduce both efficiency and effectiveness of testing.

Contour is a web based application. Data are organized into projects. It provides possibility to specify a template when a new project is about to be created.

Customization functionality is used by SoftCo team by means of ASDP Contour De-fault Configuration4, which provides several item types, like ASDP Product Feature, ASDP Declarative Requirement or ASDP Test Case. These items are designed to follow ASDP documentation as closely as possible, though it is not always possible to be followed completely. Default configuration also provides a folder structure which provides a de-fault way how to organize items. It unifies structure among projects and thus makes it easier and quicker for the user to get understanding of a new project. It also makes possible for definition of several metrics and reports.

Folder structure of ASDP Contour Default Configuration is presented in Figure3.11.

Figure 3.11: Folder layout of Contour Default Configuration

4. can be downloaded from Contour’s internal Wiki

(52)

Requirements management in Contour

Requirements management in Contour is very similar as described in ASDP. For the scope of this thesis, it does not need to be discussed any further.

Test management in Contour

Test management in Contour follows ASDP very closely, mainly because of the cus-tomization and default configuration prepared and maintained by SoftCo. Although, there are slight differences.

According to ASDP, testing shall start with planning and creation of ASDP Test Plan, which is however not possible in Contour. Therefore, this artifact is usually created in MS Word. Prior to the actual execution of the test cases, a ASDP Test Execution Plan is created5. It specifies which test cases shall be executed and assigns them to testers, and also defines a time schedule.

In the complementary thesis [17], Schimmel focuses on this planning and accuracy of its estimates.

3.6.2 Jira

Jira by Atlassian is an issue tracker recommended by SoftCo, which is used by most teams in ACS and Aerospace as well. Jira is used to record any type of issues, which can be general tasks, logs or most importantly for this thesis, defects.

Jira has several item types, each of which is defined by a set of fields and its behavior. For example defects pass through several states: reported, open, resolved, closed, . . . .

Like in Contour’s case, also Jira has been customized by SoftCo by means of ACS Jira Default Configuration6. It defines best practices for Jira usage, as well as custom workflows which closely follow ASDP.

Jira itself provides several metrics and indicators, like number of defects in each state.

3.6.3 Tools Integration

Jira is greatly extensible and customizable. Widgets can be simply added into it which makes it the ideal base for any integration with other tools. This extensibility has been recognized by SoftCo and provided start for ACS Jira Extensions project.

The integration is important because it reduced the amount of information user need to enter into the tools. It also speeds up navigation, because it is now possible to navigate directly to an item in another tool, just by following a hyperlink.

Thanks to all of these features, quality of the product specification becomes more clear, resulting in fewer problems and thus higher quality of the final product.

5. Note, that the naming is inconsistent and it is called Test Plan in Contour 6. It can be downloaded from internal Wiki

(53)

At this time, it is possible to create inter-tool links between Contour and Jira. Most obvious are links between a failed test case (in Contour) and a recorded defect (in Jira).

This first integration plugin was deployed full ACS in August 2012.

SoftCo is working on further integration plugins, which will bridge the gaps be-tween other tools. Thanks to them, more tools will get connected, resulting in higher amounts of traceability information available. In the future, it may add possibilities for several other metrics and indicators.

(54)

(55)

Chapter 4

Data evaluation

Chapter 2reviewed best practices of test management found in literature. Chapter 3

reviewed Honeywell’s point of view, namely description of requirements and test man-agement as specified in ASDP. This chapter continues by looking at the quality of real data, recorded in real projects in ACS. First, it describes how data can be gathered and where from. Then it evaluates to what extent the data fit practices from Chapter2and documentation in ASDP, what problems it may cause and the ways to prevent them.

Finally, it identifies stakeholders, who may get most benefit by improving the qual-ity of data by means of metrics.

4.1 Data Sources

This chapter describes data analysis we performed mainly on Contour database, while adding some notes about Jira database as well. Contour was chosen as the primary target of this analysis, because it contains requirement- and test- related data. On the other hand, data about defects are stored in Jira, therefore we also touched Jira database. However, defects and their statuses form only a small fraction of all data available in Jira, and thus only this small fragment of the database has been analyzed.

Access to Contour data

Data in Contour can be accessed in several ways, from which most prominent are inter-nal channel, web services and direct database access. These approaches are discussed in this section.

The first way, which can be used by any Contour user without any configuration or special privileges, is the communication channel Contour uses internally. Though, it is not officially supported nor documented by Jama Software. As said in Chapter3, because Contour is a web based application, its frontend is running in the end user’s browser and heavily uses JavaScript and processing on the client side. This client side JavaScript communicates via DWR1 library interface with the web server and fetches data from it. It would be possible to leverage this internal JavaScript API and emulate

1. Direct Web Remoting,http://directwebremoting.org/

(56)

4. DATA EVALUATION

the communication browser normally makes. But, as this form of access is not docu-mented, it would require reverse engineering Contour’s code. Also, it is not scalable and it may change in the future easily. For these reasons, we decided not to use this communication channel for obtaining the data for our analysis.

The second and most preferred way is by means of web services, namely via a SOAP API provided by Contour [12]. To be able to use it, an endpoint URL for HTTP connections is needed. Access privileges need to be configured as well so that a third party application can use this API. Via this interface, Contour exposes access to its ContourSoapService [11] implementation via HTTP protocol. Con-tour provides both read and write capabilities via functions like getItem, addItem, getDocumentTypeetc. This API reflects many use cases Jama software expected their customers may have, and thus, for these use cases it is very convenient and simple. But on the other side, it is not very flexible and thus it is difficult or even impossible to obtain specific data, or to perform specialized data analysis. Therefore, we decided not to use this SOAP API for our data analysis either.

Perhaps, it would be needed to extend Contour’s SOAP API in the future, so that the production version of dashboard proposed in Chapter6could access the data via a preferred interface.

Third and also the easiest way is direct access to Contour’s database [10]. Contour uses an SQL database, in Honeywell’s case it is Microsoft SQL Server, as a storage for all its data. Such a database can be accessed by many tools and SQL queries can be executed to obtain data. This direct form of access is very low-level and provides almost no abstraction on the raw data, but it comes with high flexibility on the other side. We were able to execute complex SQL queries to perform various calculation directly on the database server. Great advantage of this approach were also database indices which made this kind of processing computationally efficient and very fast (in comparison to Contour’s SOAP API which is said to be pretty slow).

We have chosen to use the direct connection approach for accessing Contour data not only because of its flexibility and speed, but also because SQL is very common language which we know very well. Thus, it was easy for us to write even very complex queries and perform super cool analysis.

Access to Jira data

Jira, similarly to Contour, also provides access via web services, namely it has a REST API [1]. REST itself is much simpler to use than SOAP provided by Contour.

Nevertheless, like Contour, Jira uses SQL database for storing its data as well. For consistence, and also for reasons mentioned in previous paragraph, we have chosen to access Jira database directly as well. In case when both databases are stored on the same machine (RDBMS), it is even possible to execute JOIN queries across both these databases and thus get combined and aggregated results directly from the database. 44

(57)

4. DATA EVALUATION

Database access

To access the data for our analysis, we used direct access to Contour and Jira database. We did not have access to production database, but we were provided with a snapshot copied to a development server. Therefore, we were allowed to perform experiments on real data.

4.2 Available Tools

For the data analysis, we used several ETL techniques and experimented with several ways of data visualisation, from simple MS Excel charts to a customized and interactive HTML5 web applications.

MS SQL Studio is a starting point for the data analysis, because it provided us with native access to the database and allowed us to write complex custom queries. The results help with basic understanding about the data under analysis.

In second stage of the analysis, simple SQL access was not sufficient enough any-more, as more complex statistics as well as charts were needed. We decided to use MS Excel which has great integration with SQL databases. Because output from a SELECT query in SQL database is in tabular format, Excel can run a query against a database and populate its sheets with the results. Then, Pivot tables and several built-in charts can be used for a decent analysis.

When fresh data are needed after some time, it is possible to Refresh the data sources in Excel and it will re-execute the queries and update all data sheets. However, for more sophisticated statistical data analysis, Excel does not seem to be very sufficient tool – it lacks more advanced statistical methods commonly used by statisticians. It is also very limited with the flexibility of its charts and data structures.

Tableau2_{is a professional tool for fast analysis and business intelligence. Like Excel,}

it can fetch the data directly from a database, and it provides much more flexible options for data analysis. Though, because of its complexity, is learning curve is steep making it very difficult for inexperienced users.

To achieve best flexibility and also interaction with the user, we tried visualising the data using HTML5 & SVG, with help of d3.js library.

Several tools were used for the data analysis, each with particular advantages and disadvantages. The more flexible tools used, the more difficult was the analysis and the longer it took. Therefore the tools must be chosen carefully, based on the aims of the expected result.

4.3 Data Overview

Before mentioning any results form the data analysis, it is essential to understand the scope of Contour and Jira usage. These tools were adopted by ACS only recently and

2. http://www.tableausoftware.com/

}w!"#$%&'()+,-./012345<ya

}w  !"#$%&'()+,-./012345<yA|

Metrics and Indicators on Software

Testing Process Effectiveness based

on Requirements Coverage and

Traceability Information

Jan Dolecek

Assignment

Declaration

Acknowledgement

Abstract

Keywords

Contents

PART I

Introduction

PART II

Research & Literature Review

PART III

ACS Software Development Process

Data evaluation

}w !"#$%&'()+,-./012345<yA|