Before mentioning any results form the data analysis, it is essential to understand the scope of Contour and Jira usage. These tools were adopted by ACS only recently and
2. http://www.tableausoftware.com/
45
4. DATA EVALUATION
thus they can be viewed as very young in terms of such a large company. In particular, Contour started in ACS in 2009, but teams are adopting it slowly. The main reason is scope of the projects in ACS, which usually spread across several years. Many projects, which are already in active development at the moment, actually started either before Contour was deployed in ACS, or when it was still a new and unproven. Teams and managers already had their requirements and tests recorded on paper or in other tools, and had no reason to migrate these to a new tool. Some managers only experimented with Contour and they will probably use it with their future projects.
More teams are using Contour over the time. Some switched to Contour only par-tially, for example only for their test management, while they still keep their require-ments in a separate tool or in static docurequire-ments.
Basic statistics
At the time of writing this thesis, there are 573 projects in total recorded in Contour.
Though, some of them are only examples created for training purposes, some are play-grounds used for experimenting with the tool, and some stand as templates from which new projects are being created. Naturally, we needed to exclude these from our analy-sis.
After the initial cleansing there are 543 real projects to work with. Though, as men-tioned previously, most of these project are not completely recorded in and managed by Contour. Some were created only for their managers and team to try Contour and then the tool was abandoned. Or, these projects do not have both requirements and tests recorded, and thus are not relevant for this thesis - there can be no traceability information recorded. We found that there are 197 (∼ 35% ) projects which have at least some requirements recorded, and 147 (∼ 27%) projects which have at least some test cases.
For the sake of simplicity, we decided to choose only TOP 20 projects from Con-tour for the full data analysis. We did not want to have a representative sample of all Contour projects, because most of them are not recorded properly. Rather we tried to choose a small set of project with as much meaningful data as possible. We also tried to find projects using most functionality offered by Contour. Such projects would help us to understand what kind of results we can obtain from Contour and thus what im-provements may be achieved. Moreover, our imim-provements may motivate teams to use Contour more in general and more properly, when they see what results are possible to obtain from it. We have chosen the TOP 20 projects based on number of recorded items of several types (requirements, test cases, test runs) and also based on discussion with other members of SoftCo team, who provided us with examples of well recorded projects.
46
4. DATA EVALUATION
4.4 Traceability Information
ASDP defines rules specifying which types of documents shall exist in projects and how traceability information shall be recorded between them. However, despite ASDP defines guidelines which should work in theory, they are not always practiced in reality.
Traceability information in real projects can differ largely from ASDP.
This thesis shall respect mainly ASDP rules, since they are supposed to be even-tually adapted by all ACS teams. Their adaption is also validated by several audits performed by SoftCo audit team.
When a development team does not follow these rules, it may be caused just by lack of information, knowledge or training for that team. Eventually, because the metrics and indicators developed by this thesis will be adhering to ASDP, teams will get even more motivated to follow these rules. The only way for them to obtain proper results from proposed indicators will be compliance with ASDP.
Anyway, this thesis shall also consider “badly” created projects (i.e. projects not completely adhering to ASDP) and proposed indicators shall be useful for them as well.
Reasoning is as follows, that if existing projects which had been recoded badly would not be able to benefit from those indicators, they would not be motivated to comply with ASDP (or at least not motivated enough). Also, it is not possible to change a large existing project to follow ASDP precisely, as it would be too expensive and would cause huge delay and confusion to existing processes in the project team. For these reasons, we will try to provide the teams at least with some metrics and thus motivation.
4.4.1 Traceability in Practice
ASDP specifies that each product should have a (decently) small number of ASDP Product Features (or shortly product features or just features), which form high-level overview of the product under development. These features should be then linked to more detailed ASDP Declarative Requirements (shortly requirements), which in turn de-scribe individual parts of the product in details.
Such a project structure is not always the practice. For example there are no product features in Project A 3. Instead, this team has two “kinds” of requirements – high-level ones, which are to some extent an equivalent to product features, and low-level ones, which describe individual requirements in detail.
When we asked test lead of this team why they chose this particular separation for requirements over the one described in ASDP, he had no answer. Probably, when they started the project, they were not familiar enough with the tools and so they used a solution which worked well for them. Such situation only complicates our data analysis and the final implementation of metrics as well. Data quality and compliance to ASDP must improve.
3. Project names and other information is only available to Honeywell and has been omitted from public version of the thesis.
47
4. DATA EVALUATION
4.4.2 Requirement–Test Case Links
Requirements and Test Cases are internally stored as something called a document in Con-tour. One can create directed links between them to describe relations. Contour does not impose any restriction on these links, any document can be linked to any other one.
In practice, there may sometimes appear strange artifacts like loops – a chain of links (directed), where an item is its own ancestor (and thus also its own descendant). These artifacts make data analysis more complicated, as they need to be filtered out first.
Ideally, as by ASDP description, links shall be made between ASDP Declarative Re-quirements and ASDP Test Cases.
Figure 4.1 presents a table summarizing link counts between certain document types, having source documents on the left (in rows) and target documents on the top (in columns).
Figure 4.1: Summary of direct document links
It can be seen that links to ASDP Test Cases originate only from one document type of requirement, namely ASDP Declarative Requirement. Other types of requirements do not have direct links to test cases. Such state is compliant to ASDP.
However, if we follow more steps of links, also the other requirement types are indirectly connected to test cases. As described in ASDP, all links should origin from ASDP Product Features.
Figure 4.2 shows summary of transitive links between items (i.e. more links con-nected together). Overall, all types of requirements (ASDP Product Features, ASDP Declarative Requirements, . . . ) eventually link to test cases. Which is good.
48
4. DATA EVALUATION
Figure 4.2: Summary of transitive document links
Some problems can be spotted though. For example, there are some direct links to test cases not originating from requirements – but these form only a small number.
Bigger confusion make links between test cases themselves, which form a large por-tion of all links. It is not clear to us, why such situapor-tion happens.
In summary, the links appear to be created between correct types of documents.
There are some small data errors, which may be pointed out to the teams and fixed by them.
4.4.3 Link Chains
As stated in Chapter3, when more links are transitively combined, they form chained, implicit links. Such a chained link should reach from product features, via declarative requirements, test cases, test runs (on Contour’s side) to defects (on Jira’s side).
Though this kind of traceability information is required by ASDP, data shows that it is not a common practice and it is one thing for future improvement. Table4.3presents projects under analysis together with number of documents of each type. It also shows how many links there are created between each pair of document types. As can be seen, most project do not have product features recorded at all, which confirms the hypothesis stated in beginning of this chapter. Some project do not even have declarative require-ments recorded. Unfortunately, the quality of traceability information in these project cannot be recorded at all. When looking at the rest of the projects, the situation does not get any better.
49
4. DATA EVALUATION
Figure 4.3: Link chains
A problem has been also noted in Project B , where links between requirements and test cases are formed in the other direction than expected. In the table it is represented by negative number.
Number of links is often much lower than it would ideally be (in case where testing and test cases are based on requirements). It can be seen that most project under the analysis do not have enough data properly recorded.
4.4.4 Requirements Coverage
It is essential to have all requirement sufficiently covered with tests in order to ensure that the requirements are implemented correctly and the product under test behaves as it is supposed to.
Figure 4.4 shows number of declarative requirements in each project, compared with the part of those being linked to at least one test case. Ideally, all requirements should be linked to the test cases and thus this metric should be getting to ~100%.
Though, data suggest that requirements coverage by test cases is only 31% in total.
However, the percentage of requirements covered with some tests does not guaran-tee, that every requirement is covered completely. More complicated requirements can have more test cases associated with them.
On the other hand, one requirement should not be linked to a large number of test cases. In such case, the requirement itself is not probably properly written and should be broken down into several “smaller” requirements.
We analyzed number of test cases associated to each particular requirement and created a histogram of these requirements within one project. Figure4.5shows this his-togram, where height of the bar shows number of requirements with given number of associated test cases. Ideally, each requirement should have a (decently) small number of associated test cases.
It can be seen that the highest number of requirements are those with no tests asso-ciated. Interestingly, there is also non-trivial number of requirements which are linked to more than 10 test cases4.
There are many requirements without linked test cases and thus it is not possible to validate, whether these requirements are correctly implemented. When looking from
4. Some requirements have even more than 100 test cases linked, the record goes to a requirement with 629 test cases
50
4. DATA EVALUATION
Figure 4.4: Requirements Coverage per project
Figure 4.5: Histogram of Requirements Coverage
the other side, there are also requirement with too many associated test cases, which clearly indicate misuse of the traceability information.
4.4.5 Product Features Coverage
Similar coverage analysis like in previous section can be made for product features, checking to what extent they are refined by requirements.
Each high-level product feature should have linked a (decent) number or low-level requirements, so that each of these requirements can be assigned to a feature for better overview on the project.
51
4. DATA EVALUATION
4.4.6 Unassociated Test Cases
Previous section looked at the requirements and their down-ward relations/links to test cases, analyzing how many of these links are missing. We can obviously have a similar look from the other side, i.e. on the up-ward relations from test cases while looking for missing links as well. This approach will provide us with test cases which exist in the test repository in Contour, but are not linked to any particular requirement. Therefore, it may not be easy to understand reasons for their existence. Usually, they are correct tests which somebody only forgot to link. But they can be also become invalid when requirements get changed or even slightly modified. Also, these test cases need to be reviewed or updated when a related requirement is modified – which will probably not happen because of the missing links. Because testing shall be based on requirements (as stated in Chapter2), such not connected test shall not exist – rather, all of them shall be linked to a requirement (which is also required in ASDP).
For example, in Project C , 155 of total 243 (i.e. ∼ 64%) of test cases are not linked to any declarative requirement nor to a product feature.
These not connected test cases shall be linked to the correct requirements in the better option, or deleted as irrelevant in the worse option. Unluckily, Contour does not provide any built-in way how to find these test cases, thus they stay behind in the test repository and may cause troubles in the future.
4.4.7 Links to Jira
Each execution of a particular Test Case is called a Test Run in Contour and is recorded, so that it is clear whether the test passed or not. In the case when Test Run fails, the tester performing it should record a defect into the Jira issue tracker. It is possible by a dedicated button Log defect displayed on the top of test run screen. Such a defect is bi-directionally linked with the failed Test Run, and thus also implicitly linked to a Test Case and possibly to requirement and/or to a product feature.
The situation with these links is not that clear however, because of historical changes and the availability of this integration.
Link reconstruction
It is important to note that Contour-Jira integration was deployed to production in Au-gust 2012, i.e. in a late phase of writing of this thesis. Before this integration, it was possible to create links thanks to the nature of HTML. When tester wanted to link a de-fect, he would simply add a hyperlink pointing to its URL in Jira to the test run results field. This link would then become clickable and point to Jira, which in turn would provide the user with more details.
However, because such functionality was not built in the tools and because these links were not first class links recognized by the tools, links between test runs and defects were not used very often.
52
4. DATA EVALUATION
Because there were no “native” links at the time of the data analysis, the links had to be reconstructed by parsing the URLs by regular expressions and added to the database.5
Overview on links to Jira
Naturally, because of the lack of built-in support, the quality of links is very poor. Only 12% of all failed test runs have links to a defect.
Consequently, it is not easily possible to tell, whether a failed test should be executed again. If these links would be present, an automated tool might be able to select all failed test runs which have defects closed, and offer the test manager to re-execute them.
4.4.8 Suspect Flags
When a requirement gets changed, it is possible that all linked test cases may need to be changed as well, so that they properly reflect the new content of given requirement.
It is not always necessary though, because somebody perhaps just fixed a typo in re-quirement description or performed a small change with small or no impact. Bulla [2]
describes types of possible changes and defines a Requirement Volatility Index metric, which he proposed in his thesis last year as an interesting metric for ACS teams.
Contour does not try to estimate significance of the performed change, but rather it always marks linked test cases as suspects to a needed change (or shortly just sus-pect)6. These suspects shall be reviewed to ensure they still test the requirement correctly and completely. If not so, these test cases must be updated as well to reflect previous changes made in requirements. Either way, the suspect flag is cleared afterwards.
When there is a high number of test case (or other documents) marked as suspects to a needed change, it may indicate a possible problem with data consistency. Therefore we analysed how many items in a project have suspect flag, and what part of the whole project does it make.
Overall status was satisfactory. Only ~16% of Test Cases in total had this suspect flag assigned. Figure4.6presents portion of suspect in top 20 projects.
This number seemed to optimistic to us so we started looking for the reason, which we probably found. Contour offers a table view on all suspects in the project, with a
“Clear all” button. This button clears the suspect flag for all items in the project and thus reduces usefulness of this feature. On the other hand, without being able to clear the suspect flag for multiple items at the same time, this feature would be useless as well. In such case nobody would be interested in clearing the suspect flags after fixing many typos in requirements.
5. Script used for this reconstruction is available in ACS Subversion repository in scripts/links.php 6. actually, it marks the link itself and not the target item
53
4. DATA EVALUATION
Figure 4.6: Suspect Test Cases
Suspects in Contour
Some results we obtained during the analysis can be already obtained from Contour.
But these results can not be obtained easily or do not provide sufficient details.
Contour provides a feature called Coverage Explorer, which is supposed to present recorded traceability information to the user. However, Contour displays only the links themselves, without any further information and without the possibility to group and collapse sets of items. The amount of information the user can get from this view is very limited. Example is displayed in Figure4.7. Great feature, though, is the possibil-ity to define custom filters for items being displayed, which provides somewhat good flexibility.
In a large project, view provided by coverage explorer is not really usable anymore, and it also takes non trivial time to load. When the test manager needs to ensure all requirements are covered by test cases, he can use a custom report called Coverage Re-port, which exports selected set of requirements into a Word document and highlights those without any test cases.
Another important view provided by Contour is Suspect Links, which displays links with suspect flag mentioned in previous chapter. It provides very limited view and is shown in figure.
Contour completely lacks a view giving the manager an overview on the traceability information for the whole project. This provides a great chance for improvements.
Contour completely lacks a view giving the manager an overview on the traceability information for the whole project. This provides a great chance for improvements.