Scalability in Code Recommenders

Weimer et. al. [159] applied Maximum Margin Matrix Factorization to code recom- menders. Their technique builds a single model for a complete framework, rather than a model for each API type. The authors point out the scalability issues they face due to the complexity of the optimization problem of the underlying factorization algorithm. This forced them to limit the size of the input data they provide to the algorithm.

Recommender techniques that treat code as plain text [51] or as some form of struc- tured sentences with underlying statistical language models [129], naturally scale to large repositories. On the other hand, GraLan [105] needs a large number of trees/- graphs to capture the context information of the code under editing. Therefore, it uses two thresholds to limit the number and size of the generated trees/graphs. However, such techniques don’t consider some of the structural information of the code, which is important to make more accurate code predictions.

3.6 Discussion

Intelligent code completion systems learn models by analyzing a large number of code repositories to increase the probability of detecting relevant patterns for developers.

With the vast increase of available data in such repositories, scalability becomes an issue especially with respect to the learned model sizes. Another factor that influences model sizes is the use of additional contextual information to improve prediction quality. In this part of the thesis, we investigate Boolean Matrix Factorization (BMF) as a means to create smaller models by adapting a previously developed Pattern-based Bayesian Network (PBN) framework [123], and replacing the originally used canopy clustering with BMF. Matrix factorizations are more robust than clustering approaches as they do not require partitioning. Furthermore, using the MDL principle together with the BMF allows automatic selection of the correct number of clusters.

We compare both approaches on the SWT framework, and show that we obtain model sizes that are up to 80% smaller, which in return reduced the inference speed by up to

78%, all while not compromising prediction quality (F1-measure). In the experimental

evaluation, BMF seems to be the best all-round performer, but this quality comes with a cost in running times. As each type is dealt independently, however, the work can be trivially distributed in the cloud, alleviating the problem. It is of interest to study whether we can use the specific knowledge of the domain to improve the BMF’s running time (e.g. by the sparsity of the matrices, or using the Buckshot-type approach used by SONEX [84]).

However, our experimental results suggest that BMF is promising in the context of intelligent method call completion, and we speculate that other software engineering applications should benefit from it.

Usage Patterns

Application Programming Interfaces (APIs) provide means for effective code reuse. How- ever, for the reuse to succeed, developers need to know the API and apply it correctly, i.e., according to the API usage pattern intended by the creators. An API usage pattern encodes a set of API elements that are frequently used together, optionally complemented by constraints like the order in which elements must be used. Researchers have proposed several learning approaches that are able to find API usage patterns by analyzing code repositories. These approaches commonly analyze API usages, i.e., code snippets that use a given API. They mine usage patterns, i.e., equivalent API usages that occur fre- quently. The proposed approaches are then used as the basis for various applications such as API documentation generation [4, 93], automated code completions [102, 123], bug or anomaly detection [91, 158], and code search [45, 74]. However, a major chal- lenge when learning from programs is how to represent programs in a way that facilitate effective learning. All the above mentioned approaches learn indeed different pattern representations.

Our survey presented in Section 2.4.2, shows that previous learning techniques learn three different types of pattern representation:

(1) No-order patterns are unordered sets of frequently used code elements (e.g., [100, 102]). Such patterns encode that calls of methods, say a, b, and c, frequently co-occur in code, but do not feature any information about the order of calls. (2) Sequential-order patterns (e.g., [117, 129]) additionally encode facts such as that a

has to be called before b, and b before c.

(3) Partial-order patterns (e.g., [106]) are usually represented as graphs or FSMs. these patterns encode that, for example, a must be called first, but how b or c are called afterwards is irrelevant.

Approaches from each of the above categories justify their choice of the respective pattern representation, apply their patterns to solve a specific recommendation task, and eventually compare their results with approaches within the same category in terms of precision and recall. However, so far, we lack systematic studies of the tradeoffs between the different types of patterns in representing source code in practice. For instance, an empirical study could explore whether the increased computation complexity required to mine partial-order patterns is justified when compared to mining unordered sets.

Furthermore, previous approaches predefine the code structure (pattern type) they want to learn a-priori, by designing specific program analysis and input formats that guide the learning process. This makes it hard to judge the impact of research that ad- dresses API usage pattern learning. A comparison of different pattern types with regards

to some predefined metrics is challenging, because each approach in the literature uses a different learning technique with configurations specific to its data set (e.g., frequency threshold), a different representation for usage examples and patterns, and might even be specifically tied to a particular programming language or input form (e.g., source code vs. byte-code vs. execution traces). Moreover, the different approaches are evaluated on different sets of target projects, such that the respective results are hardly comparable. In many cases the exact versions of the projects are not reported or became unavailable, which makes it impossible to reproduce results and to evaluate other approaches on the same project versions. Ideally, we would need a unified learning technique that can provide the three different patterns representations discussed above. Applying such a learning technique to the same data set and with the same settings would provide a fair comparison between API pattern types.

In this part of the thesis, we address this challenge and present, to the best of our knowledge, the first empirical comparison of API pattern types and investigate their effectiveness in representing API usages in the wild. The different pattern types we compare, consider constraints of different nature between method calls, and thus un- derstanding what exactly they are able to mine in a concrete setting constitutes an interesting and relevant subject in many software engineering applications (e.g. code recommendation or misuse detection). To provide a fair setting, we use a common data set of 360 open-source Github C# repositories with over 68M lines of code [122], and adopt an established mining algorithm that can be customized to mine all three types of patterns, episode mining [1]. Episode mining is a well known machine learning tech- nique used to discover partially ordered sets of events from a stream, called episodes (patterns in our terminology). In our setting, events are method declarations or invo- cations (cf. 4.3.2). Episode mining has already been used in several domains such as neuroscience [1], text mining [2], and positional data [48]. We can mine all three pattern types discussed above by adjusting certain parameters of the episode mining algorithm. With this experimental setup in place, we can produce sequential, partial and no-order patterns using the same mining algorithm and same data set. Our experimental setup

is publicly available as a benchmark (PtBench1_{), and can be used by other researchers}

to perform similar empirical studies.

In this first study, we compare pattern types in terms of three metrics (defined in Sec- tion 4.4.3):

Expressiveness quantifies the richness of the language corresponding to a pattern type

whose grammar rules are the mined patterns. We measure expressiveness as the number of words (i.e., derived sequences of method calls) in the language. This measure indicates how well the mined patterns abstract over the variety of concrete API usages observed in source code. Conceptually, one would expect that less structure patterns encode a richer language. The question is, though, to what extent do the differences in expressiveness between pattern types materialize in the wild.

Consistency quantifies the extent to which the words in the language defined by the

mined patterns are actually found in the code. This is to judge how truthful the mined API usage patterns represent actual API usage constraints implicitly encoded in source code. From a practical perspective, this metric gives us insights about the relevance of the order information encoded in sequential and partial-order patterns.

Generalizability measures whether the usages a pattern encodes are specific to a single

code context or if they generalize to multiple contexts. In language terminology, this metric indicates whether the learned model is applicable across domains/projects or whether we learn domain-specific languages (models). This is important to understand the applicability of the information encoded in the learned patterns.

We also present some statistics in terms of the pattern size that refer to the number of events within the learned patterns, and in terms of API types that investigates whether the learned patterns encode interactions between method calls of the same or multiple API type. Results obtained on the empirical comparison of the three pattern types, on an existing dataset of 360 C# code repositories highlight interesting evidences in terms of expressiveness, consistency, generalizability, patterns size and number of API types.

The remainder of this chapter is organized as follows: In Section 4.1, we provide an overview over related work on different API usage representations and empirical studies based on API usages. Section 4.2 introduces some conceptual differences between dif- ferent pattern type representations. The adaptation of the general episode mining algo- rithm within the domain context of pattern mining for software engineering is presented in Section 4.3. In Section 4.4 we continue with the evaluation setup, and Section 4.5 presents the complete PtBench pipeline. In Section 4.6, we use PtBench to empiri- cally evaluate and compare the three different pattern types that we analyze. From our empirical evaluation, we derive a set of implications that might be considered useful to other researchers working with code patterns. The derived implications are presented in Section 4.7, followed by the discussion of threats to validity in Section 4.8. At the end, we conclude our discussion in Section 4.9.

4.1 Related Work

To summarize the state of the art and motivate the work presented in this chapter, we look into two different directions of related work. The first is with regards to how order information is treated in existing API usage mining techniques and representations, and the second are empirical studies that have investigates API usages in practice.