This chapter presented our instance-based learner on data streams, IBLStreams, for tackling the tasks of classification and regression. IBLStreams is, to some extent, a continuation of IBL-DS; IBLStreams does not only exhibit the desirable properties of an adaptive system proposed by [57], but it also respects all the relevance factors
20000 40000 60000 80000 100000 120000 0.00 0.05 0.10 0.15 Hyperplane, distance 20000 40000 60000 80000 100000 120000 0.0 0.1 0.2 0.3 0.4
Hyperplane, square distance
20000 40000 60000 80000 100000 120000 0.0 0.2 0.4 0.6 0.8
Hyperplane, cubic distance
AMRules FIMTDD
FLEXFIS
IBLStreams Adapt σ
IBLStreams Adapt k IBLStreams Fixed k & σ
(R1) (R3)
(R2)
Figure 3.16: RMSE for the pure distance to hyperplane data (distance, squared and cubed distance).
20000 40000 60000 80000 100000 120000 0.00
0.05 0.10 0.15
Hyperplane, distance, concept drift
20000 40000 60000 80000 100000 120000 0.0 0.1 0.2 0.3 0.4
Hyperplane, square distance, concept drift
20000 40000 60000 80000 100000 120000 0.0 0.2 0.4 0.6 0.8
Hyperplane, cubic distance, concept drift
AMRules FIMTDD
FLEXFIS
IBLStreams Adapt σ
IBLStreams Adapt k IBLStreams Fixed k & σ
(R1) (R3)
(R2)
Figure 3.17: RMSE for the distance to hyperplane data (distance, squared and cubed distance), with a concept drift.
1000 2000 3000 4000 5000 0 2 4 6 8
Parkinson’s Motor UPDRS
1000 2000 3000 4000 5000 0 2 4 6 8 10
Parkinson’s Total UPDRS
10000 20000 30000 40000 50000 0 10 20 30 40 50 60 Slice Localization AMRules FIMTDD FLEXFIS IBLStreams Adapt σ IBLStreams Adapt k IBLStreams Fixed k & σ
(R1) (R3)
(R2)
Figure 3.18: RMSE for the real data sets: Parkinson’s motor UPDRS, Parkinson’s total UPDRS and slice localization.
introduced by [17] for an IBL approach while maintaining a case base. In addition, parameter adaptation strategies are suggested for a dynamic fit to the current concept. The experiments presented here suggest that IBLStreams competes with the state-of-the-art instance-based and model-based learners on data streams. Indeed, IBLStreams seems to be less “inert” when a concept drift occurs and, moreover, re- covers its original performance more quickly when the drift comes to an end. This is arguably due to the advantage of not having to adapt a possibly complex model. Additionally, IBLStreams seems to quickly reach a high performance compared to the other learners, this is seen as a learning curve that rapidly reaches the saturation level. For these reasons, IBLStreams is comparable, if not superior, to the state-of-the-art instance-based and model-based learners on data streams
Chapter 4
Evolving Fuzzy Pattern Trees
This thesis starts by introducing the aspects of learning and the need to develop statistical solutions for transforming data to knowledge; it also shows how learn- ing becomes challenging when the data becomes immense and continuous as in the streaming settings.
Chapter 3 shows an example of how one of the widely used machine learning techniques, namely the simple nearest neighbor approach, can be adapted to make the learning from non-stationary environments possible.
In this chapter, we present a different learning technique that draws its elements from the theory of fuzzy sets [184]. Fuzzy logic is a multivalued logic in which truth values go beyond the binary set {true, false} or even the many-valued sets. In this type of logic, truth values are taken from the unit interval, with the ability to employ linguistic terms characterizing the space of underlying variables.
Models that utilize the theory of fuzzy sets are capable of expressing more realistic representation of world’s problems than two-valued logic. Fuzzy logic allows prepo- sitions to be satisfied, unsatisfied or even partially satisfied; even more, satisfaction is quantified through the notion of membership degree for an element in a set, or the satisfaction degree of a proposition.
The advantage of fuzzy modeling becomes more obvious when considering fuzzy rule-based systems, which allow a fuzzy representation of the data; a fuzzy rules-based model allows rules to become partially satisfied. Because fuzzy logic allows sets to be identified with linguistic terms, the set of rules representing a concept becomes a generalized representation of the concept that is easier to interpret and to understand due to its expressibility in the natural language.
H¨ullermeier [83] refers to the advantage of extending machine learning and data mining methods with fuzzy concepts. This extension leads to models that are more comprehensible and less complex; however, it is unlikely that the fuzzy extension
would lead to major improvements in the generalization performance, especially be- cause these fields have reached a mature state.
Motivated by these developments, we propose an extended version of the fuzzy pattern trees suitable for learning from data streams. More specifically, by building on the (batch learning) algorithm for pattern tree induction as proposed in [146], we develop an evolving variant for the problem of binary classification.
This chapter is organized as follows: By way of background, Section 4.1 recalls some basic information about the theory of fuzzy sets. Section 4.2 presents a few data- driven approaches that utilize the aspects of fuzzy logic. Section 4.3 introduces the fuzzy pattern tree and its main induction methods, which we extend to the streaming setting in Section 4.4. Experimental results are presented in Section 4.5, prior to concluding the chapter in Section 4.6.
4.1
Introduction to Fuzzy Sets
Proposed as an extension to the set theory, fuzzy sets theory relaxes the crisp defini- tion of the set membership “∈”. This extension is motivated by the natural way we represent the continuity of our knowledge and belief, which suffers from information loss when discretized. Thus, an element now belongs to a set to some degree and is characterized by the notion of membership. The characteristic function of a subset
A of a reference set Ω is defined as follows: A(x) =
{
1 if x∈ A
0 if x /∈ A , (4.1)
whereas, a fuzzy set [184] is defined by a membership function A that assumes values in the unit interval:
A : Ω→ [0, 1] .
A large number of fuzzy membership functions have been proposed in the literature [127], such as triangular function, β-function, S-function, trapezoidal, and Gaussian, among others. The triangular functions take the form of a triangle with the mode at
b and the support at [a, c]:
A(x) = x−b c−a if x∈ [a, b] c−x c−a if x∈ [b, c] 0 if x /∈ [b, c] . (4.2)
4.1.1
Operations on Fuzzy Sets
Fuzzy sets require new definitions of three main set operations, intersection, union and complement in order to fit their multivalued nature. These definitions can be achieved based on the generalization of the logical operators. Triangular norms were formally defined as generalization of the triangular inequality in probability metric spaces [118]. Subsequently, triangular norms [96] were used as a substitute for the conventional conjunction and disjunction operations as shown in the following two definitions.
A t-norm is the generalization of the logical conjunction and it is a function
⊤ : [0, 1] × [0, 1] → [0, 1] that needs to satisfy the following conditions: • Commutativity: ⊤(a, b) = ⊤(b, a)
• Associativity: ⊤(a, ⊤(b, c)) = ⊤(⊤(a, b), c)
• Monotonicity: if a ≤ c and b ≤ d, then ⊤(a, b) ≤ ⊤(c, d) • Identity element: ⊤(a, 1) = a
A t-conorm is the generalization of the logical disjunction and it is a function
⊥ : [0, 1] × [0, 1] → [0, 1] that needs to satisfy the following conditions: • Commutativity: ⊥(a, b) = ⊥(b, a)
• Associativity: ⊥(a, ⊥(b, c)) = ⊥(⊥(a, b), c)
• Monotonicity: if a ≤ c and b ≤ d, then ⊥(a, b) ≤ ⊥(c, d) • Identity element: ⊥(a, 0) = a
Each t-norm has a dual t-conorm for which
⊥(a, b) = 1 − ⊤(1 − a, 1 − b) ,
or equivalently
⊤(a, b) = 1 − ⊥(1 − a, 1 − b) .
4.1.2
Aggregation Operations on Fuzzy Sets
The rich representation of fuzzy sets allows for a class of operators that aggregate multiple fuzzy sets into a single set. A fuzzy aggregation operator [127] is an n-ary
ψ : [0, 1]n× [0, 1] → [0, 1] operator for which the following holds:
• Monotonicity: ψ(a1, . . . , an)≥ ψ(b1, . . . , bn) if ai ≥ bi, i = 1, . . . , n
• Boundary conditions ψ(0, . . . , 0) = 0 and ψ(1, . . . , 1) = 1
Obviously, the set of fuzzy aggregation operators contains the set of triangular norms and conorms. The aggregation operators include: the compensatory opera- tors [186], symmetric sums [58], averaging operators [59] and the ordered weighted averaging [181]; many data-driven approaches focus and utilize the last two.
The weighted average (WA) operator is an n-ary function WA : [0, 1]n → [0, 1]
identified by the vector w = (w1, . . . , wn)∈ [0, 1]n with
∑n
i=1wi = 1 such that
WA(a1, . . . , an) = n
∑
i=1
wiai .
Similarly, the ordered weighted average (OWA) [181] is an n-ary function OWA : [0, 1]n→ [0, 1] that takes the weighted average of the n arguments after sorting them; it is identified by the vector w = (w1, . . . , wn)∈ [0, 1]n with
∑n
i=1wi = 1 such that
OWA(a1, . . . , an) = n
∑
i=1
wif (i, a1, . . . , an) ,
where the value f (i, a1, . . . , an) is the ith smallest value in the vector (a1, . . . , an).
The OWA operator exhibits the property of generalizing other operators:
• The arithmetic mean: if w = (1/n, . . . , 1/n) then OWA(a1, . . . , an) = 1n
∑n
i=1ai
• The minimum operator : if w = (1, . . . , 0) then OWA(a1, . . . , an) = min(a1, . . . , an)
• The maximum operator : if w = (0, . . . , 1) then OWA(a1, . . . , an) = max(a1, . . . , an)
Notably, the t− DRA is the smallest t-norm and the MIN is the largest, whereas t-conorms are bounded by the MAX and the co− DRA. The averaging operators WA and OWA take values in the wide spectrum of operators between the least strict t-norm and the most strict t-conorm:
t− DRA ≤ t − EIN ≤ t − LUK ≤ t − ALG ≤ MIN
≤ WA, OWA ≤
Operator t-norm t-conorm
G¨odel MIN(a, b) = min{a, b} MAX(a, b) = max{a, b} algebraic t− ALG(a, b) = ab co− ALG(a, b) = a + b − ab Lukasiewicz t− LUK(a, b) = max{a + b − 1, 0} co − LUK(a, b) = min{a + b, 1} Einstein t− EIN(a, b) = 2−(a+b−ab)ab co− EIN(a, b) = 1+aba+b
drastic t− DRA(a, b) = b if a = 1 a if b = 1 0 otherwise co− DRA(a, b) = b if a = 0 a if b = 0 1 otherwise Table 4.1: Fuzzy triangular operators.