Future Work

Research Question Q3.1:

How to learn heuristics using genetic algorithms in order to improve prediction quality of configuration systems?

In Section 3.3 (Erdeniz et al., 2018a), we have proposed a novel heuristics learning method with which historical transactions are clustered and a genetic algorithm is employed on each cluster to learn cluster-specific heuristics. When a new configuration/diagnosis task need to be solved, the closest cluster is iden-tified and its pre-learned heuristic is used for solving the new problem. This heuristic (CLCO) is provided in the CSPHeuristix Library as well (see Chapter 5). Using this approach,

• constraint ordering heuristics for consistency based diagnosis (Erdeniz et al., 2018a) (see Section 3.3)

are learned. As presented in experimental evaluations of Section 3.3 (Erdeniz et al., 2018a), CO (constraint ordering) heuristics improve quality of configuration in terms of prediction quality and minimality.

Research Question Q3.2:

How to learn heuristics using matrix factorization techniques in order to improve prediction quality of configuration systems?

In Section 3.5 (Erdeniz et al., 2019a), we have proposed novel heuristics learning method with which historical transactions are gathered into a matrix (a sparse matrix) and matrix factorization is employed to estimate a full matrix from which we then obtain learned heuristics. When a new configuration/diagnosis task need to be solved, the most similar historical transaction is identified and its pre-learned heuristic is used for solving the new problem. This heuristic (MFCO) is provided in the CSPHeuristix Library as well (see Chapter 5). Using this approach,

• constraint ordering heuristics for consistency based diagnosis (Erdeniz et al., 2019a) (see Section 3.5)

are learned. As presented in experimental evaluations of Section 3.5 (Erdeniz et al., 2019a), proposed CO (constraint ordering) heuristics improve quality of configuration in terms of diagnosis minimality.

6.2. Future Work

In configuration and recommendation technologies, little research exists to simultaneously improve both the runtime and prediction quality. In this context, we have provided novel solutions based on learn-ing heuristics uslearn-ing learnlearn-ing algorithms and historical transactions. Our future work will include further heuristics learning approaches. In this chapter, relevant topics for future research are discussed.

Further Learning Approaches

Machine learning algorithms are made up of three main components: representation, evaluation, and op-timization. A classifier must be represented in a formal language that the computer can understand and interpret. Creating a set of classifiers that the learner can learn is crucial. Evaluation function is needed

to distinguish good classifiers from bad ones. Finally we need a method to search in a specialized lan-guage for the highest scoring classifier. The choice of optimization technique is key to the efficiency of the algorithm (Domingos, 2012). The goal, of course, is to generate a rule that makes the most accurate pre-dictions possible on new test examples. On one hand, building a highly accurate prediction rule is certainly a difficult task. On the other hand, it is not hard at all to come up with very rough rules of thumb that are only moderately accurate. An example of such a rule is something like the following: "If the phrase ’buy now’ occurs in the email, then classify it as spam.". Such a rule will not even come close to covering all spam messages; for instance, it does not say anything about what to predict if ’buy now’ does not occur in the message. However, this rule will make predictions that are significantly better than random guessing (Schapire, 2003).

Based on the training dataset, four groups of learning methods can be used. They are as follows: su-pervised learning (Linear Regression, Support Vector Machines, Neural Networks, Decision Trees, Naive Bayes, Nearest Neighbor, etc.), unsupervised learning (k-means clustering, Association rule, etc.), semi-supervised learning, and reinforced learning (Q-Learning, Deep Adversarial Networks, etc.).

In this thesis, we have utilized historical transaction as the training dataset and unsupervised learning methods based on this dataset. We have employed k-means clustering algorithm, genetic algorithms, and matrix factorization techniques. However, even based on the same historical transactions, various combi-nations of various machine learning techniques can be also investigated.

Datasets for Evaluation Purposes

In the world of recommender systems, it is a common practice to use public available datasets from dif-ferent application environments (e.g. MovieLens, Book-Crossing, or Each-Movie) in order to evaluate recommendation algorithms. These datasets are used as benchmarks to develop new recommendation al-gorithms and to compare them to other alal-gorithms in given settings (Verbert et al., 2011). The relevance of these datasets however has faded over the years as they become outdated (e.g., most recent movie in MovieLens 100K dataset is from 1998). While still useful for offline evaluation, online experiments with actual users may fail because of the lack of recent and relevant movies in the dataset. Moreover, datasets themselves are also often filtered so to only contain users with e.g. a minimum number of ratings (e.g., 20 ratings for MovieLens). Because of this filtering, a systematic bias is introduced which may prevent experimental results to be generalizable to real-life scenarios (Harper and Konstan, 2016).

Configuration systems need a larger and richer open-source evaluation datasets as is the case with rec-ommendation systems (e.g. Movielens Dataset). In these datasets, configuration and diagnosis problems (with many variables and constraints) should be defined with their corresponding real-world user prefer-ences (e.g., user requirements and their purchase transactions on a online shopping with personalization features). On the other hand, the development of recommendation technologies for IoT scenarios is a rather young discipline and research in the field would strongly profit from the availability of more IoT datasets that enable corresponding tests of, for example, the prediction quality of recommendation algorithms. In the context of end-user development support in IoT scenarios, datasets that include logs are helpful for development of IoT workflows on remote gateway installations. This information can be exploited to opti-mize user support, for example, by predicting relevant code-fragments and sensors that should be included.

We are interested in gathering such big datasets for our quality based evaluations of configuration and recommendation systems in IoT.

CSPHeuristix Library

New heuristics learning approaches can be easily tested and compared with our approaches by using our open-source library, CSPHeuristix. CSPHeuristix is a modular and extendable library, so these new

ap-6.2. Future Work

proaches can be implemented inside the library using its already-implemented constraint solving objects.

CSPHeuristix can also be extended with new constraint-solvers, diagnosis algorithms, complex variable/-constraint structures, and even various input file formats.

CSPHeuristix-v1.0 supports:

• Heuristics: four novel heuristics learning methods which are described in this thesis in Section 3.2, Section 3.3, Section 3.4, and Section 3.5,

• Variables: integer,

• Constraints: unary,

• Constraint solvers: choco-solver (as a constraint solver)

• Consistency-based diagnosis algorithms Solvers: FASTDIAGand FLEXDIAG

Appendix A

Appendix