Statistical analysis

As well as presenting a selection of metrics, this chapter has also attempted to perform statistical analysis of the results of applying them to two Haskell pro- grams. This work has shown that some of the metrics are strongly correlated, indicating that the attributes they measure are related, for instance the number of patterns in a function is closely related to the number of scopes in the function. These correlations between metrics are examined in greater detail in Chapter 5.

Looking at the correlation between the metric values and the number of bug- fixing or refactoring changes each function has undergone has shown that in gen- eral there is no single metric which gives good predictions, although some of the metrics such as “Outdegree” (c3) can give reasonable predictions. Combining the metrics which exhibit the highest correlations can give better predictions. From this we can see that there is no single attribute that makes a Haskell program hard to understand, but rather that the subjective complexity lies in the combination of features.

Unfortunately, due to the time consuming nature of analysing program change histories, the analysis of correlation between metrics and bug-fixing changes is lim- ited to only two programs. Because of this one must be careful not to over gen- eralise the results of this study. To acquire greater confidence in these statistical results it would be necessary to carry out further case studies.

However, there are other interesting studies that can be performed with the metrics that were presented in this chapter. Therefore the work in this chapter is extended in Chapter 5 to examine a wider selection of Haskell programs.

Trends and Characteristics of

Haskell Metrics

Chapter 4 presented a selection of metrics that measure various individual at- tributes of Haskell programs and analysed the correlation between the individual metrics and the change history of two case study programs, as well as the cross- correlation between the various metrics.

This chapter extends that work by examining a wider selection of Haskell programs. Due to the time consuming nature of examining change histories this chapter does not compare metrics with change histories, but instead concentrates on the relationships between attributes measured by the metrics.

The methodology used for the work presented in this chapter was described in detail in Chapter 3. It is worth noting that the following case study programs all perform tasks which involve language processing.

• Haddock - A tool for generating API documentation from Haskell source code.

• Happy - A parser generator for Haskell, similar to yacc.

• Hat - A collection of tools for debugging Haskell programs by using tracing. • Haxml - A library of tools for processing XML, including parsing, pretty

printing and transformations.

• Hunit - A unit testing framework for Haskell.

• Thih - A Haskell program that implements a type checker for the Haskell language.

Typically, when writing programs that involve processing of complex tree-like data structures such as parse trees, programs are constructed in a manner that closely follows the data structure. It is therefore possible that because these programs perform similar tasks they may cause the results to be biased. However for the work presented in this chapter, the presence of these programs does not appear to cause bias because these six programs appear to show the same common metric characteristics as the other eight programs. Therefore this issue is not addressed further.

This chapter also studies the values that the metrics produce in order to de- termine the typical values for the metrics and to discover what values may be classed as unusual, and which therefore indicate programs and functions which deviate from the typical style of Haskell programs.

The commonly held view of the style of typical Haskell programs can be sum- marised as the following.

Functional programs typically contain many pattern expressions, such as in the argument lists of functions, for manipulating data types and for control flow such as performing if-then-else style selective execution. Looping behaviour is typically achieved using groups of recursive functions. Functional programmers tend to break problems down into small parts, which results in programs which consist of many small functions “plumbed” together with a few large functions. Abstraction is usually achieved through the use of polymorphism and higher-order functions.

However, there is little hard evidence that this “folklore” is actually represen- tative of real Haskell programs. Thus by examining the metric values it may be possible to provide a more concrete idea of the features that typify the “average” Haskell program.

The remainder of this chapter is divided into the following sections.

• Section 5.1 examines the interaction between the various metrics on a larger body of programs.

• Section 5.2 discusses the values that the various metrics typically show on a selection of programs and suggests ways in which these values may be incorporated into a tool.

• Section 5.3 summarises the conclusions from the work in this chapter and suggests some ways to utilise this information.

5.1

Analysis of Cross-correlation of Metrics on

a Larger Body of Programs

The work in Chapter 4 concentrated on examining the correlation between the various metrics and the number of changes a program has undergone, but that work is based on only two case study programs.

To obtain a much clearer picture of the behaviour of the metrics it would be useful to analyse more programs. However, analysing the program change histories is a prohibitively time consuming process, as described in Chapter 3, so this has not been done in this study, and is instead left as future work.

An alternative method of analysing the metrics on a larger body of programs is to study the relationships between the metrics by analysing a single snapshot of each program. Because analysing a single snapshot of a program can be done quickly and mostly automatically it is possible to examine a larger sample of programs. This allows for greater confidence in any conclusions drawn from the study.

In the following sections the cross-correlation of the metrics will be analysed on a further fourteen Haskell programs. The methodology used for this work and the programs studied were described in Section 3.3.4 of Chapter 3. The cross- correlation matrices for each of the programs are shown in Appendix E, and will be examined in detail in the following sections.

• Section 5.1.1 examines the cross-correlation of the recursion metrics. • Section 5.1.2 studies the cross-correlation of the callgraph metrics.

• Section 5.1.3 looks at the cross-correlation of the distance measurements. • Section 5.1.4 investigates the cross-correlation of the pattern metrics. • Section 5.1.5 studies the cross-correlation of the miscellaneous metrics. • Section 5.1.6 examines the cross-correlation between metrics of different

• Section 5.1.7 presents the conclusions that can be drawn from the study of the cross correlation.

5.1.1

Cross-correlation of recursion metrics

The tables of cross-correlation values for the recursion metrics, shown in Section E.1, indicate that only five of the fourteen programs feature any non-trivial re- cursion, as indicated by Tables 32, 33, 35, 36 and 40. However only one program, Getopt, does not include any recursion. Thus one might make the following observation.

Observation 5.1.1 The occurrence of non-trivial recursion in Haskell programs is quite unusual, and is associated with complex program behaviour. However the occurrence of trivial recursion is common.

Because of this observation, and those discussed in Section 4.3 of Chapter 4, it is not surprising that for most of the programs in the study the recursion metrics were exactly correlated.

It is worth noting that four of the programs which did contain non-trivially recursive functions were language processing programs. As was described at the start of this chapter, the program structure of language processing programs often quite closely follows the structure of the data types used in the program.

However, three of the programs which did contain non-trivially recursive func- tions still exhibited strong correlation between the recursion metrics. As discussed in Section 4.3 of Chapter 4, this suggests that only a small proportion of the re- cursive functions in those programs are non-trivially recursive, adding another indication that such functions are unusual.

For the two remaining programs with non-trivial recursion, Haddock and Hat, it appears that “Number of non-trivial recursive paths” (r2), “Number of recursive paths” (r4) and “Sum of lengths of recursive paths” (r5) are strongly correlated. This seems to suggest that for these programs there is a significant pro- portion of the recursive paths that are non-trivially recursive, because the number

of non-trivial paths is strongly correlated with the total number of paths. Both of these programs contain large parsers for the Haskell language which have complex recursive behaviour, and are therefore the probable cause of this correlation.