Identifying and Tracking Quality

In this section, we present our approach for finding quality trends in evolving soft- ware. We start by discussing issues pertaining to the analysis of temporal data. Then, we present an approach to analyse the evolution of quality.

5.3.1 Dealing with Temporal Data

When trying to study evolution, a decision must be made on how to represent the passing of time. Generally, we use either calendar time or the version numbers of soft- ware releases. An analysis using calendar time implies that a project is audited at regular intervals (e.g., every month, or even nightly in a continuous integration process). Ex- tracting information regularly can produce some misleading results because these dates might not correspond to an actual release. Many projects tend to be developed actively until they become relatively stable. At that point, the development slows down. This is the case with open-source projects like Xerces-J, which went from having releases every month to every year. If weeks go by without any changes to the software, a quality audit might be a waste of time.

Analysing the evolution of software according to release dates ignores specific times- tamps and has the added advantage of consistently capturing a project in the same de- velopment phase. Furthermore, this approach can indicate what type of maintenance was performed (e.g., bug-fix versions indicate that developers did not add functionality, and mostly removed bugs). This choice can have an adverse effect on models when releases become rare. Xerces-J is now maintained by a small number of developers who have limited knowledge of the inner workings of most of the project. Typically, the quality of software tends to degrade when key knowledge of a system is lost due to turnover [HBVW08]. As releases become infrequent, there should be even more atten- tion paid to a system when it is changed. If the time between changes and releases is long, then the feedback from an IV&V team might arrive too late to be useful.

A final problem concerns the tracking of new classes. We might assume that “new” classes go through a period of debugging before becoming stable. But, not all new

73

Figure 5.1: Quality signal: the quality of the module degrades over six versions

classes exist in the first version of the software; some are introduced much later. We would need a technique that is able to adapt to the notion of class-centered time as opposed to project-centered time.

We consider that the quality of every class is represented by a time series. These series can be of different lengths and have different rates of change. The technique used to identify trends should be able to support the fact that classes evolves somewhat independently from each other, and can appear and disappear.

5.3.2 Quality Trend Analysis

The evolution of a class with respect to its level of quality can be represented as a signal Q = (q1, q2, q3, ..., qn) where qi is the quality score of a class at the version i.

Version 1 represents a version in which a class appears in the system and n is the total number of versions in which the class exists and is analysed. Such a signal is illustrated in Figure 5.1.

Our approach consists of two steps. First, we seek to find different trends given a set of signals representing how all classes have evolved. Then, we want to classify a specific signal Q according to these trends to guide a subsequent, in depth-study of evolution. Both these steps require a way to compare these signals given the aforementioned issues with temporal data. We chose to use dynamic time warping.

5.3.2.1 Dynamic Time Warping

Dynamic time-warping (DTW) is a well known signal analysis algorithm. DTW was first presented by Kruskal and Lierberman [KL83] to compute a distance measure between pairs of signals independent of timescale. This technique finds a “topological” distance between two signals by modifying the time axis of each one. For example, it can align two signals by matching peaks that occur at different times. It is commonly used in speech recognition software to handle different speaking speeds. This ability to modify time is important in the context of change pattern detection because changes tend to happen irregularly, often independently of the system versions.

5.3.3 Finding Quality Trends

Identifying quality trends can be done completely manually or semi-automatically using data mining techniques from a set of quality signals. We opted for a semi-automatic process. By using mining techniques, it is possible to process all known quality signals to find regular patterns, and present this information to an analyst for review.

To find candidate trends, we used a technique called clustering. Clustering is an unsupervised classification technique that takes a set of items and tries to group items together using a notion of similarity. To find possible trends, we applied agglomerative hierarchical clustering using the DTW distance metric as a measure of dissimilarity. This clustering algorithm starts by considering every item as a cluster and iteratively merges similar clusters together. The specific algorithm is described in [BGA06]. The result is a dendrogram representing all possible groupings as shown in Figure 5.2. A manual analysis of these groupings provides a set of possible trends. A quality engineer can select a set of clusters that makes senses.