We already discussed the parallelization approach for the matrix-vector multiplication kernels in Algorithms 9 to 11, which is mainly based on parallelizing the for loop (as indicated in the code comments), and depending on the algorithm, by introducing a thread-local intermediate result vec- tor y.
The parallelization of the matrix-matrix multiplication is more complex. Hence, the pseu- docodes for Algorithms 9 to 11 only contain the sequential version for the sake of simplicity. There- fore, we now discuss the general parallelization scheme, which we use in all of our kernel imple- mentations. Note that in the context of a main-memory machine, we only consider shared-memory parallelism. Algorithms based on distributed systems are not discussed in the scope of this thesis.
Similar to the parallelization of the matrix-vector multiplication, the parallelization of each of our kernels is primarily realized by splitting the outer for-loop over the rows of matrix A (e.g., line 4 in Algorithm 10). From a more conceptual perspective, we effectively parallelize the algorithm by separating A into multiple, disjunct row chunks A(i), which form independent work units that are
processed in parallel. Figure 3.8a sketches the access pattern of a partitioned matrix multiplication, and Figure 3.8b shows the separation into the work units. The advantage of the row-centric parti- tioning is that each matrix row i of A, multiplied with (potentially the complete) matrix B, creates
×
=
A B C A(1) A(2) A(3) A(4) C(1) C(2) C(3) C(4)(a) Access pattern of a row-partitioned matrix multiplication. The pattern indicates which parts are involved in the calculation of C(2): A(2)and the full matrix B
A(1) A(2) A(3) A(4) C(1) C(2) C(3) C(4)
(b) Separation in independent work units. The surrounding paths denote the input and output matrices of the two exemplary work units 1 and 2.
Figure 3.8: Parallelization approaches for row-based matrix multiplication algorithms. the corresponding row i in the result C. Hence, each work unit will produce a disjunct chunk C(i)
of the result matrix.
In case of a dense result matrix, the threads just write into the disjunct memory locations of the previously created dense matrix representation of C. In contrast, if the target matrix is in a sparse layout, each of the disjunct part results C(i)have a CSR layout, which can not be trivially appended.
Instead, an additional step is required to assemble the row chunks together into a final CSR result representation. This step is done in a separate parallelized procedure.
It should be mentioned that each thread, which is processing any of the working units (i), po- tentially reads the complete matrix B. At least, it has to read all rows that are required in the multiplication with A(i). In a naive shared-memory parallel model, where each thread is assumed
to have equal access times to any memory location, this should not have any impact on the perfor- mance. In practice, however, blocking of the dense or sparse matrix B yields a far better scaling behavior, due to an increased cache locality. In particular we observed substantial performance improvements by using our tiled data structure in the evaluation of Chapter 5. This observation was also made by Patwary et al. (2015), who used a columnar blocking of B (and simultaneously C) to improve the cache locality of both reads and writes. Furthermore, on modern multi-core machines, memory read latencies for different physical cores diverge due to effects of non-uniform memory access (NUMA). A main-memory system with many memory sockets behaves somewhat
similar to a distributed multi-host landscape. In a distributed environment with communication overhead, different aspects come into place: Buluç and Gilbert (2010) found that parallel sparse matrix multiplication using an 1D partitioning of matrix A (as of Figure 3.8b) fails to scale, whereas a 2D partitioning exhibits a superior, scalable asymptotic behavior.
We revisit the topic of matrix multiplication in connection with our adaptive tile matrix data structure in Chapter 5. Therein, we present a tile-based multiplication algorithm, which will be discussed with respect to parallel scheduling, cache-efficiency, and NUMA-partitioning.
3.8
SUMMARY
In this chapter, we have set the foundations for the following chapters of this thesis. In particular, we introduced the column-oriented storage engine of the main memory system, on which the Lapeg is based on. Moreover, we presented ways to represent dense and sparse in a columnar DBMS that are significantly more efficient than the alternative approaches of related work (Zhang et al., 2009). Furthermore, we outlined the analogy between efficient sparse matrix formats and clustering as well as indexing relational tables in column-stores.
We surveyed different strategies of multiplying dense and sparse matrices with vectors and other matrices based on the column-oriented storage layer matrix data structures, and introduced basic parallelization constructs. In particular, we emphasized the importance of having a full set of mixed sparse- and dense matrix multiplication kernels. In fact, these are not provided by most numerical libraries, e.g. the NIST sparse Blas(NISTb, n.d.), limiting their applicability for flexible, matrix- based ad-hoc data analysis.
The different multiplication kernels and especially their runtime behavior are again a topic of Chapter 4, where we discuss the optimization of arbitrary multiplication expressions. Further- more, based on these kernels we present a sophisticated matrix multiplication operator in Chapter 5, which exploits heterogeneous tile multiplications, dynamic task scheduling, and cost-based runtime optimizations. We also discuss related work about advanced matrix storage formats in greater detail in Section 5.6.
4
EXPRESSION OPTIMIZATION
The optimizer forms the fundamental building block of the logical layer in the Lapeg (Figs. 1.1 and 4.2). For each linear algebra expression there might be multiple execution plans, and only few exe- cute the requested operations in an efficient, optimal way. Similar to that of a relational execution engine, the Lapeg generates a plan that is composed of physical plan operators, including multipli- cations and storage type transformations of matrices. This chapter is devoted to the optimizer of the Lapeg, with focus on the optimization of matrix multiplication expressions by considering physical properties of matrices and matrix algorithms.4.1
MOTIVATION
Many workflows contain one or more linear algebra expressions that are computationally expensive. In this chapter, we focus on optimizing those expressions that amount for the major part of the anal- ysis execution time. Therefore, we recall the randomized SVD method described in Section 2.3.1: the core computation in Algorithm 3 is a matrix chain multiplication of multiple instances of the sparse term-document matrix with a dense, Gaussian random matrix.
Matrix chain multiplications occur in a variety of other applications. Examples are transitive closure computations, Markov chains (Yegnanarayanan, 2013), linear transformations (Edelman et al., 1994), linear discrete dynamical systems (Feng, 2002), or multi-source, multi-level breadth first search (Kepner and Gilbert, 2011). In fact, an efficient execution of sparse matrix chain multi- plications is nontrivial, especially if intermediate result matrices become dense. In many situations the runtime performance can be significantly improved by changing the execution order of the ex- pression, or by switching from a sparse to a dense multiplication algorithm during execution. The Density of Intermediates. To illustrate the problem of optimizing the execution of sparse matrix expressions, we present a brief experimental study of the graph analysis scenario introduced in Section 2.3.2. The breadth-first search algorithm can be rewritten by unrolling the main loop (line 5 in Algorithm 4), and assembling the matrix-vector multiplications into a single multiplication expression. The resulting expression yields:
yT= (...(((xTG)G)G)...G
search depth
). (4.1)
1 2 3 4 5 10−4 10−2 100 102 Search Depth Quer y Runtime [s] 100 101 102 103 104 105 # found nodes
x: sparse intermediate (SpI) x: dense intermediate (DI) # of found vertices Figure 4.1: Comparison of the execution duration of the breadth-first search (Equation 4.1) on a social network graph by using a sparse (SpI) and a dense (DI) intermediate structure. The x-axis denotes the depth parameter of Algorithm 4. The adjacency matrix is stored in a CSR format.
Indeed, Equation 4.1 is a matrix chain multiplication of the sparse 1 × n matrix xT, which is
effectively a row vector, with d (search depth) instances of the sparse adjacency n × n matrix G. We now illustrate the impact of densities and data structures on the execution time when ex- pression (4.1) is executed naively from left to right by using the outer product sparse matrix-vector multiplication algorithm (Algorithm 7 in Section 3.5.2.) In this small experiment depicted in Fig- ure 4.1, we used two different versions of the intermediate result vector; a dense vector representa- tion (DI) as shown in Figure 3.2a, and a sparse one (SpI) as of Figure 3.2b. Furthermore, we took the adjacency matrix (G) of a social network graph1and varied the search depth parameter d, which
corresponds to the multiplication chain length and is denoted along the x-axis of the plot.
In the initial phase of Algorithm 4 the vector x is only filled with a single element, which represents the start vertex ID. With every multiplication, new vertices are found, i.e., more non- zero elements are added to the intermediate result vector. Hence, the density of the intermediate vector increases, which has a considerable impact on the multiplication runtime. Furthermore, it reveals the different complexity between the DI and the SpI version. The DI implementation of the multiplication iterates over every matrix entry (zero- and non-zero-valued), whereas the SpI implementation only iterates over non-zero elements. As a consequence, the DI approach performs worse for small depths than the SpI one. However, with increasing search depth the x vector density reaches a turning point at which the DI-based implementation becomes more efficient. The exact value of this point depends on the details of the respective implementation. In our experiment, it is reached between depth two and three for the social graph Gra1. In the analogous measurement on Gra2, the turning point is at a considerably larger depth, which exceeds the x-range of the plot.
From this experimental study it can be concluded that switching internally from a sparse to a dense representation of vector x at a certain density clearly offers a tuning opportunity, which should be exploited by an optimizer. In fact, this example only shows one aspect of optimization, namely the internal representation of intermediate results. Other aspects that can affect the run-
time include parenthesization and execution order, which we discuss in more detail below and which are both addressed by our optimizer.
The problem is closely related to join enumeration in relational algebra, where the optimal join order and the selection of join algorithms is largely determined by the cardinality of intermediate results. Since it is usually impossible for database users to manually overview the cardinalities of intermediate relations, the RDBMS takes over this task by generating an optimized execution plan based on cardinality estimates and physical properties of the system. Likewise, data scientists are usually not familiar with algorithmic details of multiplication kernels and system parameters, and do not have a profound knowledge of the characteristics of their matrices. Hence, it makes sense to leave these decisions as well to the system.
Despite the number of systems that offer a matrix-based language interface (some of which have been presented in Section 2.5ff,) only little has been done in the direction of optimizing matrix expressions. Our idea is the following: similar to the RDBMS optimizer that creates optimized exe- cution plans from SQL expressions, we propose an optimizer component that generates an optimal execution plan for linear algebra. As illustrated in Figure 4.2, this component forms the founda- tion of the logical layer of the Lapeg, and works on top of the physical storage engine. The latter accommodates matrices in the native column-oriented storage layer, using the data structures and algorithms that were presented in Chapter 2.
The optimizing component comprises the following integral parts:
• SPMACHO, a general matrix chain multiplication optimizer based on a dynamic programming approach, which leverages density estimations of intermediate results and different multipli- cation kernels to minimize the total execution runtime. The optimization problem and the SpMachO algorithm are presented in Section 4.2.
• A comprehensive cost model, which is used by SpMachO to determine the costs of subchain multiplications. It is derived from the number of memory accesses and floating point opera- tions of the different matrix multiplication kernels that were introduced in Section 3.6. The cost model is described in Section 4.3.
• SPPRODEST, a sparse matrix density estimator, which predicts the density structure of inter- mediate and final result matrices, by using a novel skew-aware stochastic density propagation method. It is described in detail in Section 4.4.
Moreover, we present an extensive evaluation and comparison of the execution runtime of Sp- MachO-generated plans against alternative execution strategies and other numerical algebra sys- tems in Section 4.5. Finally, we will discuss related work in Section 4.6, followed by the summary in Section 4.7.