Vectors and Matrices

Vector and matrix notation is a convenient and succinct way to represent dens data structures and complex operations. They are commonly used in the mathematical branch of linear algebra, though, in thesis they are mainly used for their expressive powers.

Data Structure

A vector v ∈ Rⁿ represents an ordered list of individual variables v_i ∈ R. The index i ∈ {1, · · · , n} denotes where a particular variable fits within a vector. By default the variables are considered to be listed vertically as in (7.1). A vector with this configuration is called a column vector. The orientation of the a vector matters when it comes to performing operations on them. Equation (7.2) illustrates the transpose operation T, where v is flattened. A neat way to represent a column vector is thus v = [v₁. . . v_n]^T.

v =







 v₁

... v_n









(7.1)

152 Mathematical Notation and Concepts

v^T =h

v₁ · · · v_n i

(7.2) An extension of vector notation is the more expressive matrix notation. A matrix A ∈ R^m×n is a rectangular array, or table, of single variables a_ij ∈ R. The first index i represents a row in the matrix, whereas the second index j denotes a column. An interpretation is that a matrix is composed of multiple vectors representing either rows (7.3) or columns (7.4). Conversely you could interpret a vector as a matrix with only one column or row, hence the terms column and row vectors.

There are multiple ways to perform multiplication of vectors and matrices. As with simple algebraic notation, two adjacent vectors have an implicit multiplication operation called the dot product (7.5). The vectors must be of equal dimensions, where each ordered pair of single variables are multiplied then summed.

v · u = v^Tu =h

General matrix multiplication is an extension, or generalization, of the dot product.

Using the dot product, each row vector of the first matrix A ∈ R^m×n is multiplied with every column vectors of the second matrix B ∈ R^n×k. This operation is performed in (7.6). Notice how the inner dimensions n of the matrices must agree.

That is, each row vector of matrix A must have the same dimension as the column

vectors of matrix B. The resulting matrix AB is thus of m-by-k dimensions where each entry is defined by (7.7).

AB =

A special case of the matrix multiplication in (7.6) is when k = 1, or rather the multiplication of a matrix A ∈ R^m×n to a vector b ∈ Rⁿ. This yields the case (7.8), where the dot product of vector b and each row vector of A produces the vector Ab ∈ R^m.

In the field of computer science a second more intuitive matrix and vector multi-plication is often used. The entrywise product, also known as the Hadamard product, applies normal algebraic multiplication of every corresponding entry pairs

154 Mathematical Notation and Concepts

of two matrices or vectors of identical dimensions.

A ◦ D =

An exception to the dimensionality requirements discussed above is the multipli-cation of a constant, or weight, to a vector or matrix. In this case the weight is simply multiplied to all entries, as (7.10) illustrates for the vector case. Unsur-prisingly, addition and subtraction also behave entrywise as in the vector case in 7.11. Although, for addition and subtraction the dimensions must agree.

cv = c

An important class of vector and matrix operations is the norm, which is denoted by some variation of || · ||. Both vectors and matrices have corresponding norm concepts however only norms of vectors are needed. There exists many types of norms, where they all have the common trait of being a measure of length or size of a vector. The most notable norm is the euclidean norm (7.12), often used to calculate the euclidean distance between two points (7.13). A generalization of the euclidean norm is the p-norm defined by (7.14). Besides the euclidean norm, the most important p-norms are the manhattan norm with p = 1 (7.15) and the infinity norm with p = ∞ (7.16).

d(v, u) = ||v − u||₂ =p

Matrix notation allows for succinct expression of linear and quadratic systems of equations. The set of linear equations y_i = a_i1x_i1+ · · · + a_inx_in + b_i = 0, i ∈

Sometimes it is useful to have a way to represent a vector of identical entries.

The null vectors 0 and one vectors 1 are vectors with entries of only 0 and 1 respectively. Usually the dimension of such a vector is implied by the context. An application of the one vector is that it can be used to sum the entries of a vector by 1^Tx = x₁+ · · · + x_m.

156 Technical SVM Details

The diag(·) operator transforms a vector v ∈ Rⁿinto a diagonal matrix V ∈ R^n×n by putting each entry of the vector into the diagonal of the matrix while keeping other entries zero.

An often used matrix concept is the identity matrix defined as I_n = diag(1). It has the property that multiplication with it does not have any effect, e.g., AIn = A, and can be viewed as a cousin of 1 in normal arithmetic.

Finally augmentation of vectors and matrices can be needed for application of machine learning and optimization software. Augmentation simply means that we concatenate two object of equal dimensions. For instance, a vector v = [v₁, · · · , v_n]^Tmay be augmented by a new variable v_n+1becoming v⁰ = [v₁, · · · , v_n+1]^T. Similarly a matrix A ∈ R^m×n may be augmented another with, say, matrix E ∈ R^m×k as in (7.21) or vector as in (7.22).

The basic SVM classifier assumes datapoint separability. It has a one-augmented dataset X ∈ R^m×n of n datapoints x_i and m features (including the augmented feature). This follows the dataset definition of (2.35), where X is a matrix with rows of datapoints x_i. Since this is not a multi-label problem there is only one label per datapoint. One can therefore compound all the labels related to dataset X into one single binary vector [y₁ · · · y_m]^T = y ∈ {−1, 1}^m which we want to use