7.4 A case study
7.4.5 Incremental recomputations in the general case
In the case study, all incremental MapReduce programs were initially derived manually. This was an insightful exercise that led to two important observa- tions. First, we found ourselves following a common pattern for each sample MapReduce program (view). Second, we realized that each of these views be- longs to a common class – the class of self-maintainable aggregate views. In this section, we will formally define this class of MapReduce views and provide a general solution to the problem of deriving incremental MapReduce programs from members of this class of views.
Aggregate MapReduce views are evaluated in three steps. First, the source key-value pairs are translated into intermediate key-value pairs. Second, the intermediate key-value pairs are grouped together based on the intermediate key. Third, an “aggregate function” is applied to each group of intermediate values resulting in a single (possibly set-valued) result value. The result value along with its grouping key is stored at the materialized view.
Abstractly, a MapReduce aggregate view can thus be defined by a translation function and an aggregate function. The translation function translates source key-value pairs into intermediate key-value pairs. Recall, that a single source key-value pair may be translated into multiple intermediate key-value pairs. We denote the domain of source keys, source values, intermediate keys, and intermediate values as Dsk, Dsv, Dk, and Dv, respectively. We denote the
power set of a set S as P(S). A translation function is defined as translate :
Dsk× Dsv → P(Dk× Dv).
An aggregate function is defined by a binary function◦ : Dv×Dv→ Dv. Let
v1, . . . , vn∈ Dvbe a group of intermediate values, then the aggregated value
a∈ Dvin the materialized view is computed as a := v1◦ v2◦ . . . ◦ vn. Because
require the function◦ to be commutative and associative.
A view is called self-maintainable if it can be maintained using the base deltas and the content of the view itself, without accessing the base data [GJM96]. Let ∆V ={an+1, . . . , am} be inserted values (deltas). The latest aggregate value
a′ in the materialized view is computed as a′ := v1◦ . . . ◦ vn◦ vn+1◦ . . . ◦ vm.
Alternatively, a′ can be computed using the content of the materialized view itself as a′:= a◦vn+1◦. . .◦vm. Thus, an aggregate view based on a commutative
and associative function◦ is always self-maintainable w.r.t. insertions. Let∇V = {vi, . . . , vk}, 1 ≤ i < k ≤ n be deleted values (deltas). The latest
aggregate value a′ in the materialized view is computed as a′ := v1◦ . . . ◦
vi−1◦ vk+1◦ . . . ◦ vn. To achieve self-maintainability a′needs to be computable
using only the content of the materialized view a and the deltas ∇V . This can be done if each element v∈ Dvhas an inverse element v∗∈ Dvsuch that
v◦ v∗= e, with e being the neutral element w.r.t. function◦. If this condition holds, the latest aggregate value can be computed as a′ := a◦ v∗i ◦ . . . ◦ vk∗.
Note that a self-maintainable aggregate function is thus defined by an Abelian group (Dv,◦).
In summary, a MapReduce aggregate view definition can be specified by four user-defined functions.
A function translate : Dsk× Dsv → P(Dk× Dv)
A function ◦ : Dv× Dv→ Dvthat is commutative and associative.
A function ∗ : Dv → Dv that maps each element of the Abelian group
(Dv,◦) to its inverse element.
A nullary function e : ∅ → Dv that returns the neutral element of the
Abelian group (Dv,◦).
Given these functions, MapReduce programs that fully compute or incremen- tally recompute aggregate views can be assembled. Consider Algorithm 8 that performs a full (re)computation. In the Map phase the translation function is applied to obtain intermediate key-value pairs. Intermediate values having the same intermediate key are grouped together. In the Combine and Reduce phase, the elements of each group are composed using the function◦ to obtain an aggregated result value.
Algorithm 9 and Algorithm 10 maintain aggregate views using overwrite or increment installation, respectively. The Map function of Algorithm 9 reads in the deltas and the content of the materialized view. The deltas are fed into the translate function, while the content of the materialized view is simply passed through. The Map function of Algorithm 10 reads and translates the deltas only. The Combine and Reduce functions of Algorithm 9 (overwrite installation) are similar to Algorithm 8. Increment installation differs in the
Algorithm 8 Full (re)computation
1: function Map(key, value)
2: for all v in translate(value) do
3: emit(v.key, v.value)
4: end for
5: end function
6: function Combine(key, values) ◃ optional
7: temp := e ◃ neutral element
8: for all v in values do
9: temp = temp◦ v
10: end for
11: emit(key, temp)
12: end function
13: function Reduce(key, values)
14: temp := e
15: for all v in values do
16: temp = temp◦ v
17: end for
18: write(key, temp)
19: end function
Algorithm 9 Incremental recomputation - overwrite installation
1: function Map(key, value) ◃ Reads deltas and the view
2: if key is inserted then
3: for all v in translate(value) do
4: emit(v.key, v.value)
5: end for
6: else if key is deleted then
7: for all word in translate(value) do
8: emit(v.key, v.value∗) ◃ inverse element
9: end for
10: else
11: emit(key, value)
12: end if
13: end function
14: function Combine(key, values) ◃ see Algorithm 8
15: ...
16: end function
17: function Reduce(key, values) ◃ see Algorithm 8
18: ...
Algorithm 10 Incremental recomputation - increment installation
1: function Map(key, value) ◃ Reads deltas only
2: if key is inserted then
3: for all v in translate(value) do
4: emit(v.key, v.value)
5: end for
6: else if key is deleted then
7: for all word in translate(value) do
8: emit(v.key, v.value∗) ◃ inverse element
9: end for
10: end if
11: end function
12: function Combine(key, values) ◃ optional
13: temp := e ◃ neutral element
14: for all v in values do
15: temp = temp◦ v
16: end for
17: emit(key, temp) ◃ If Reducer is used. Alternatively, write to the materialized view.
18: end function
19: function Reduce(key, values)
20: temp := e
21: for all v in values do
22: temp = temp◦ v
23: end for
24: begin ◃ single-row transaction
25: old := read(key) ◃ read from the materialized view
26: new := old◦ temp
27: write(key, new) ◃ write to the materialized view
28: commit
29: end function
way deltas are applied. Algorithm 10 uses the notion of a single-row transaction to atomically read a value from the materialized view, update it, and write it back (lines 25 - 29). Note that HBase provides an atomic increment operation that can be used if◦ is the addition operation.
We review the MapReduce programs to compute word histograms and reverse web-link graphs discussed in the previous section to show how these fit into our theoretical framework. In both cases, Dsk is the set of all URLs and Dsv is
the set of all html documents. The translate function of the word histogram view produces intermediate key-value pairs where Dkis the set of all words and
Dv is the natural numbersN. The aggregate function is based on the Abelian
group (+,N), i.e. the operation ◦ is addition, the inverse element of n ∈ N is
n∗:=−n, and the neutral element e is zero.
mediate key-value pairs where Dk is the set of all URLs and Dvis the power
multiset of all URLs. The aggregate function is based on the Abelian group (⊎, Dv), where⊎ denotes multiset union. The inverse element is computed by
negating the multiplicity of the elements in a multiset and the neutral element is the empty set.
The lessons learned from the case study allow for designing a programming model on top of MapReduce for defining self-maintainable aggregate compu- tations. Such a programming model will provide hooks to plug-in user-defined functions including a translation function, a commutative and associative ag- gregate function, an inverse element function, and a neutral element function, as discussed earlier. For any computational problem that can be described in this form, MapReduce jobs for the incremental recomputation of previous re- sults can be assembled automatically. A framework supporting the proposed programming model for self-maintainable aggregates is currently being devel- oped in the course of a Master thesis2.