5.4 Experimental Evaluation
5.4.3 Comparison to Related Work
So far, the existing call-graph-based techniques [CLZ+09, DFLS06, LYY+05] have not been compared to the well-known techniques from software engineering dis- cussed in Chapter 3.3 We now compare our best-performing approach, Esubtree
comb , to
the Tarantula technique [JHS02], to two of its improvements [AZGvG09] and to the SOBER method [LFY+06] (see Section 3.1.2 for details). These techniques can be seen as established defect-localisation techniques as they have outperformed a num- ber of competitive approaches (see Section 3.1.2).
For the experiments in this section, we have implemented Tarantula, its improve- ments and SOBER for our programme used in the evaluations in this chapter. We have done so as no complete implementations are publically available. For SOBER, there is MATLAB source code available from the authors that performs the statistical calculations. However, there is no tool for the instrumentation of predicates avail- able. For SOBER we have therefore implemented an automatic instrumentation, and we have reimplemented the statistical calculations in Java. For Tarantula and its improvements, we have implemented both steps, automated instrumentation and the calculations.
2In this comparison, FREQT was restricted as in [DFLS06] to find subtrees of a maximum size of
four nodes. Such a restriction was not set in gSpan. Furthermore, we expect a further significant speedup when CloseGraph is used instead of gSpan.
3Only [CLZ+09] has been compared to the sequence-mining-based approach RAPID [HJO08] which
Both techniques, Tarantula and SOBER, work on granularities that are finer than the method level used by our approach (see Section 3.1.2). However, the Taran- tula authors describe a means of mapping the results to the method level [JH05]. Concretely, the authors assign the score from its highest ranked basic block to the method. For our comparisons we rely on this mapping, and we do the same for SOBER, which originally works on the predicate level.
In our experiments with Tarantula, we have noticed that it happens frequently that methods have the same likelihood score, which worsens the results from the ap- proach. Although the authors have not applied this technique in the original evalua- tions [JH05, JHS02], we use the brightness score from Tarantula (see Section 3.1.2) as a secondary ranking criterion. This is, we let this criterion decide the ranking po- sition in case the original score is the same for some methods. This approach seems to be natural, as the brightness would be a secondary source of information for a developer who uses the original visualisation from Tarantula.
When looking at Tarantula and our approach from a theoretical perspective, our approach considers more data than code coverage as utilised by Tarantula, but at the coarser method level. The information analysed by our approach additionally to the information analysed by Tarantula includes (1) call frequencies, (2) subgraph con- texts and (3) the information which method has called another one. This data is poten- tially relevant for defect localisation, e.g., to localise frequency-affecting bugs (1) and structure-affecting bugs (2, 3). We therefore expect well results from our approach in comparison to Tarantula.
As discussed before (see Section 3.1.2), SOBER overcomes some of the short- comings of previous approaches mentioned. It analyses the frequencies of predi- cate evaluations and is therefore better suited than Tarantula to localise frequency- affecting bugs. However, it does not analyse subgraph contexts as our approach does, but its predicate analysis takes information into account that we do not consider (e.g., return-value predicates). It is therefore hard to formulate theoretical expectations whether SOBER or our approach will perform better.
In the following, we compare our Esubtree
comb approach (values taken from Table 5.4)
to Tarantula in experiments ETarantula and ETarantulab (with and without the brightness
score), to the Jaccard coefficient variations in experiments EJaccard and EJaccardb , to
the Ochiai coefficient variations in experiments EOchiai and EOchiaib and to SOBER in
experiment ESOBER.
Table 5.4 contains the results from the comparison. The table clearly shows that our approach (Esubtree
comb ) performs best in 12 out of the 14 defects and as well best on
average. Only for defect 1 some of the other approaches perform a little better, and for defect 9 all other approaches perform better than Esubtree
comb . The explanation for the
latter is as before: Defect 9 does not affect the call frequencies at all which are the most important evidence for our approach. The other comparisons are as expected: Eb
Tarantula leads to better results than ETarantula, and EJaccard and EOchiai perform better
exp./ defect 1 2 3 4 5 6 7 8 9 10 11 12 13 14 ∅ Esubtree comb 3 2 1 1 1 2 2 1 18 2 2 3 3 3 3.1 ETarantula 6 8 5 6 7 9 8 6 5 6 9 9 11 9 7.6 Eb Tarantula 1 6 1 3 7 7 6 3 5 6 7 7 11 7 5.5 EJaccard 1 6 1 3 6 7 6 4 4 5 7 7 11 7 5.4 Eb Jaccard 1 6 1 3 6 7 6 4 4 5 7 7 11 7 5.4 EOchiai 1 6 1 3 6 7 6 4 4 5 7 7 11 7 5.4 Eb Ochiai 1 6 1 3 6 7 6 4 4 5 7 7 11 7 5.4 ESOBER 3 3 6 3 3 4 3 3 4 5 4 4 9 4 4.1
Table 5.4: Comparison to related work (bold face indicates the best experiments).
improve the results of EJaccard and EOchiai. This can be explained by the fact that
EJaccardand EOchiaiare initially a lot better than ETarantula, which has more potential for
improvements. In our experiments, the results of EJaccardand EOchiaido not differ. This
is as well not unexpected, as in [AZGvG09] the Ochiai coefficient is only in very few cases better than the Jaccard coefficient. ESOBER in turn performs on average better
than all Tarantula variations, although there are few defects where ESOBERperforms
worse. This is consistent to the results in [LFY+06].
Looking at the average values, a developer has to consider 3.1 methods when using our approach (Esubtree
comb ). In contrast, when using the best performing approach from
the related work considered in this comparison, SOBER (ESOBER), the developer
would have to consider 4.1 methods. Based on the benchmark defects used in these experiments, our approach therefore reduces the effort for defect localisation by 24% compared to SOBER.
Besides Tarantula and SOBER, we have also experimented with the static Find- Bugs [AHM+08] defect-localisation tool (see Section 3.1.1). However, FindBugs
was not able lo localise any defect in our 14 defective programme versions. This is not surprising, as the defects of our benchmark (listed in Table 5.2) mostly represent defects affecting the programme logic rather than defect-prone programming patterns that can be identified by FindBugs.
5.5 Subsumption
The experiments in this chapter have shown that our approach performs well (regard- ing localisation precision) compared to both related approaches based on call-graph mining and established approaches from software engineering. However, as in the related work [CLZ+09, DFLS06, LYY+05], our evaluation is based on a relatively small number of defects and it is hard to draw conclusions for arbitrary defects in arbitrary programmes. Nonetheless, the defects in our evaluation serve as a bench-
mark. According to Zeller [Zel09], it is likely that a technique that performs better than another one on a benchmark will perform better on other programmes, too.
More concretely, our experiments presented in this chapter as well as the ones in the closely related work [CLZ+09, DFLS06, LYY+05] suffer from two issues related to the question if the results can be generalised:
• The experiments are based on artificially seeded defects. Although these de- fects mimic typical defects as they occur in reality, a study with real defects from an existing software project would emphasise the validity of the tech- niques. (This also applies to most of the techniques described in Section 3.1.2, as the evaluations rely on the Siemens Programmes [HFGO94] featuring artifi- cial defects.)
• All experiments feature rather small programmes containing the defects (i.e., roughly ranging from 200 to 700 LOC). The programmes rarely consist of more than one class and represent situations where defects could be found rel- atively easy by a manual investigation as well. (This also applies to most of the techniques described in Section 3.1.2.) The approaches considered here will probably not scale without any further effort for programmes that are much larger than the programmes considered currently.
In the remainder of this dissertation we tackle these two issues. In Chapter 6, we investigate a scalable solution that builds on the techniques proposed in this chapter. We evaluate the approach with real defects from an open-source software project that is two orders of magnitude larger than the programmes in the evaluations considered so far. Furthermore, we present a constraint-based approach in Chapter 8, which leads to better scalability of the underlying graph-mining algorithms.
So far, we have not considered multithreaded programmes in our evaluations. In Appendix A, we present and evaluate a variation of the technique presented in this chapter for the localisation of defects in such programmes.
In the previous chapter, we have presented our approach for defect localisation (Sec- tion 5.3). Despite good results, we have identified two issues of this approach, namely poor scalability with the size of the software project and a desired evaluation with real defects (see Section 5.5). Both issues as well apply to the closely related work [CLZ+09, DFLS06, LYY+05]. In this chapter, we aim at generalising our approach for defect localisation to scale for larger software projects. To this end, we propose a hierarchical procedure that works with call graphs at different levels of granular- ity. We furthermore evaluate our new approach with real defects from a real-world software project.
We first present an introductory overview in Section 6.1. Sections 6.2 and 6.3 explain the call-graph representations we use in this chapter and defect localisation based on them, respectively. Section 6.4 contains the evaluation, and Section 6.5 is a subsumption of this chapter.
6.1 Overview
In this chapter, we aim at a scalable method for call-graph-mining-based defect local- isation and at an evaluation with real defects. Solving the scalability issues is chal- lenging, as seemingly possible solutions have issues: (1) Using increased computing capabilities or distributed algorithms is not feasible due to exploding computational costs. We have experienced this effect in preliminary experiments as well. Further, spending a lot of computing time for graph mining might be inappropriate for defect localisation. (2) Solving the scalability issue with approximate graph-mining algo- rithms might be a solution, but might miss patterns which are important for defect localisation. For instance, [CLZ+09] (see Section 5.2.3) does not report any prob- lems caused by the better scaling LEAP algorithm [YCHY08] (see Section 3.2.2), but does not analyse large programmes either.
A different starting point to deal with the scalability problem in call-graph-based defect localisation is the graph representation. In this chapter, we investigate graph representations at coarser abstractions than the method level (see Section 4.1), i.e., the package level and the class level, and we start at such a coarse abstraction before zooming-in into a suspicious region of the call graphs. These graphs are a lot smaller than conventional method-level call graphs, and they cause scalability problems in much fewer cases. However, this idea leads to new challenges:
1. Call-graph representations have not yet been studied for levels of abstraction higher than the method level. How do representations well-suited for defect localisation look like?
2. When zooming-in into defect-free regions by accident, the following question arises: How to design hierarchical defect localisation in a way that minimises the amount of source code to be inspected by humans?
3. It is unclear which defects can indeed be localised in coarse graph representa- tions.
Our approach for hierarchical defect localisation builds on the zoom-in idea and solves these challenges. It relies on weighted call graphs, making the localisation of certain defects a lot easier. In more detail, this chapter makes the following contribu- tions:
Granularities of Call Graphs. We define call graphs at different levels of granu- larity, featuring edge-weight tuples that provide further information besides the graph structure (challenge 1). We do so by taking the specifics of defect localisation into account: We explicitly consider API calls as well as inter-/intra-package and inter- /intra-class method calls.
Hierarchical Defect Localisation. We describe the zoom-in operation for call graphs, present a methodology for defect localisation for the graphs at each level and describe hierarchical procedures for defect localisation (challenge 2). In concrete terms, we present different variants of a depth-first search strategy to hierarchically mine a software project.
Evaluation with a Large Software Project. An essential part of this chapter is the evaluation featuring real programming defects in Mozilla Rhino (challenge 3). To this end we use the iBUGS repository [DZ09] and the original test suite. Rhino consists of≈ 49k LOC, and the defects in the repository were obtained by joining information from a bug-tracking system with data and source code from a revision- control system.
Ideas related to zooming-in into call graphs, namely Graph OLAP, have been de- scribed in [CYZ+09]. The authors propose data-warehousing operations to analyse graphs, e.g., drill-down and roll-up operations, similar to our zoom-in proposal. How- ever, [CYZ+09] does not help in defect localisation, as it aims at interactive analyses, and it does not consider specific requirements (e.g., API calls).