Big Data Analytics: Optimization and Randomization

Tianbao Yang^†, Qihang Lin^\, Rong Jin^∗‡ Tutorial@SIGKDD 2015

Sydney, Australia

†Department of Computer Science, The University of Iowa, IA, USA

\Department of Management Sciences, The University of Iowa, IA, USA

∗Department of Computer Science and Engineering, Michigan State University, MI, USA

‡Institute of Data Science and Technologies at Alibaba Group, Seattle, USA

August 10, 2015

URL

http://www.cs.uiowa.edu/˜tyng/kdd15-tutorial.pdf

No

This tutorial is not an exhaustive literature survey

It is not a survey on different machine learning/data mining algorithms

Yes

It is about how to efficientlysolve machine learning/data mining (formulated as optimization) problems forbig data

Outline

Part I: Basics

Part II: Optimization Part III: Randomization

Outline

1 Basics

Introduction

Notations and Definitions

Model Optimization

20 40 60 80 100

0 0.05 0.1 0.15 0.2 0.25 0.3

iterations

distance to optimal objective

0.5^T 1/T² 1/T

Data

Big Data Challenge

Big Data

Big Model

60 million parameters

Learning as Optimization

Ridge Regression Problem:

min

w∈R^d

1 n

n

X

i =1

(y_i− w^>x_i)²+λ 2kwk²₂

xi ∈ R^d: d -dimensional feature vector yi ∈ R: target variable

w ∈ R^d: model parameters n: number of data points

Ridge Regression Problem:

min

w∈R^d

1 n

n

X

i =1

(yi − w^>xi)²

| {z }

Empirical Loss

+λ 2kwk²₂

xi ∈ R^d: d -dimensional feature vector yi ∈ R: target variable

w ∈ R^d: model parameters n: number of data points

Learning as Optimization

Ridge Regression Problem:

min

w∈R^d

1 n

n

X

i =1

(yi− w^>xi)²+ λ 2kwk²₂

| {z }

Regularization

xi ∈ R^d: d -dimensional feature vector yi ∈ R: target variable

w ∈ R^d: model parameters n: number of data points

Classification Problems:

min

w∈R^d

1 n

n

X

i =1

`(yiw^>xi) +λ 2kwk²₂

y_i ∈ {+1, −1}: label

Loss function `(z): z = y w^>x

1. SVMs: (squared) hinge loss `(z) = max(0, 1 − z)^p, where p = 1, 2

2. Logistic Regression: `(z) = log(1 + exp(−z))

Learning as Optimization

Feature Selection:

min

w∈R^d

1 n

n

X

i =1

`(w^>x_i, y_i) + λkwk₁

`₁ regularization kwk₁ =^Pd_{i =1}|w_i| λ controls sparsity level

Feature Selection using Elastic Net:

min

w∈R^d

1 n

n

X

i =1

`(w^>xi, yi)+λ^kwk₁+ γkwk²₂^

Elastic net regularizer, more robust than `1 regularizer

Learning as Optimization

Multi-class/Multi-task Learning:

minW

1 n

n

X

i =1

`(Wxi, yi) + λr (W)

W ∈ R^{K ×d}

r (W) = kWk²_F =^PK_k=1^Pd_j=1W_kj²: Frobenius Norm

r (W) = kWk∗ =^P_iσi: Nuclear Norm (sum of singular values) r (W) = kWk1,∞=^Pd_j=1kW_:jk∞: `1,∞mixed norm

Regularized Empirical Loss Minimization min

w∈R^d

1 n

n

X

i =1

`(w^>xi, yi) + R(w)

Both ` and R are convex functions

Extensions to Matrix Cases are possible (sometimes straightforward) Extensions to Kernel methods can be combined with randomized approaches

Extensions to Non-convex (e.g., deep learning) are in progress

Data Matrices and Machine Learning

The Instance-feature Matrix: X ∈ R^n×d

X =

















 x^>₁ x^>₂

·

·

· x^>_n



















The output vector: y =

















 y1

y₂

·

·

· y_n



















∈ R^n×1

continuous yi ∈ R: regression (e.g., house price)

discrete, e.g., yi ∈ {1, 2, 3}: classification (e.g., species of iris)

Data Matrices and Machine Learning

The Instance-Instance Matrix: K ∈ R^n×n Similarity Matrix

Kernel Matrix

Some machine learning tasks are formulated on the kernel matrix Clustering

Kernel Methods

Data Matrices and Machine Learning

The Feature-Feature Matrix: C ∈ R^{d ×d} Covariance Matrix

Distance Metric Matrix

Some machine learning tasks requires the covariance matrix Principal Component Analysis

Top-k Singular Value (Eigen-Value) Decomposition of the Covariance Matrix

Why Learning from Big Data is Challenging?

High per-iteration cost High memory cost

High communication cost

Large iteration complexity

1 Basics

Introduction

Notations and Definitions

Norms

Vector x ∈ R^d

Euclidean vector norm: kxk2=

√ x^>x =

q Pd

i =1x_i²

`_p-norm of a vector: kxk_p =^Pd_{i =1}|x_i|^p1/p where p ≥ 1

1 `2norm kxk2= q

Pd i =1x_i²

2 `1norm kxk1=Pd i =1|xi|

3 `_∞ norm kxk_∞= maxi|xi|

Vector x ∈ R^d

Euclidean vector norm: kxk2=

√ x^>x =

q Pd

i =1x_i²

`_p-norm of a vector: kxk_p =^Pd_{i =1}|x_i|^p1/p where p ≥ 1

1 `2norm kxk2= q

Pd i =1x_i²

2 `1norm kxk1=Pd i =1|xi|

3 `_∞ norm kxk_∞= maxi|xi|

Norms

Vector x ∈ R^d

Euclidean vector norm: kxk2=

√ x^>x =

q Pd

i =1x_i²

`_p-norm of a vector: kxk_p =^Pd_{i =1}|x_i|^p1/p where p ≥ 1

1 `2norm kxk2= q

Pd i =1x_i²

2 `1norm kxk1=Pd i =1|xi|

3 `_∞ norm kxk_∞= maxi|xi|

Matrix X ∈ R^n×d

Singular Value Decomposition X = UΣV^>

1 U ∈ R^n×r: orthonormal columns (U^>U = I): span column space

2 Σ ∈ R^{r ×r}: diagonal matrix Σii = σi> 0, σ1≥ σ2. . . ≥ σr 3 V ∈ R^{d ×r}: orthonormal columns (V^>V = I): span row space

4 r ≤ min(n, d ): max value such that σr > 0: rank of X

5 U_kΣ_kV_k^>: top-k approximation Pseudo inverse: X^†= V Σ⁻¹U^>

QR factorization: X = QR (n ≥ d ) Q ∈ R^n×d: orthonormal columns R ∈ R^{d ×d}: upper triangular matrix

Matrix Factorization

Matrix X ∈ R^n×d

Singular Value Decomposition X = UΣV^>

1 U ∈ R^n×r: orthonormal columns (U^>U = I): span column space

2 Σ ∈ R^{r ×r}: diagonal matrix Σii = σi> 0, σ1≥ σ2. . . ≥ σr 3 V ∈ R^{d ×r}: orthonormal columns (V^>V = I): span row space

4 r ≤ min(n, d ): max value such that σr > 0: rank of X

5 U_kΣ_kV_k^>: top-k approximation Pseudo inverse: X^†= V Σ⁻¹U^>

QR factorization: X = QR (n ≥ d ) Q ∈ R^n×d: orthonormal columns R ∈ R^{d ×d}: upper triangular matrix

Matrix X ∈ R^n×d

Singular Value Decomposition X = UΣV^>

1 U ∈ R^n×r: orthonormal columns (U^>U = I): span column space

2 Σ ∈ R^{r ×r}: diagonal matrix Σii = σi> 0, σ1≥ σ2. . . ≥ σr 3 V ∈ R^{d ×r}: orthonormal columns (V^>V = I): span row space

4 r ≤ min(n, d ): max value such that σr > 0: rank of X

5 U_kΣ_kV_k^>: top-k approximation Pseudo inverse: X^†= V Σ⁻¹U^>

QR factorization: X = QR (n ≥ d ) Q ∈ R^n×d: orthonormal columns R ∈ R^{d ×d}: upper triangular matrix

Norms

Matrix X ∈ R^n×d

Frobenius norm: kX k_F =^qtr (X^>X ) =^qPn_{i =1}^Pd_j=1X_ij² Spectral (induced norm) of a matrix: kX k₂ = max_kuk2=1kX uk₂

kAk2= σ1 (maximum singular value)

Matrix X ∈ R^n×d

Frobenius norm: kX k_F =^qtr (X^>X ) =^qPn_{i =1}^Pd_j=1X_ij² Spectral (induced norm) of a matrix: kX k₂ = max_kuk2=1kX uk₂

kAk2= σ1 (maximum singular value)

Convex Optimization

min_{x ∈X} f (x )

X is a convex domain

for any x , y ∈ X , their convex combination αx + (1 − α)y ∈ X

f (x ) is a convex function

Characterization of Convex Function

f (αx + (1 − α)y ) ≤ αf (x ) + (1 − α)f (y ),

∀x , y ∈ X , α ∈ [0, 1]

f (x ) ≥ f (y ) + ∇f (y )^>(x − y ) ∀x , y ∈ X local optimum is global optimum

Convex Function

Characterization of Convex Function

f (αx + (1 − α)y ) ≤ αf (x ) + (1 − α)f (y ),

∀x , y ∈ X , α ∈ [0, 1]

f (x ) ≥ f (y ) + ∇f (y )^>(x − y ) ∀x , y ∈ X local optimum is global optimum

Convex function:

f (x ) ≥ f (y ) + ∇f (y )^>(x − y ) ∀x , y ∈ X Strongly Convex function:

f (x ) ≥ f (y ) + ∇f (y )^>(x − y ) +λ

2kx − y k²₂∀x , y ∈ X Global optimum is unique

strong convexity constant

Convex vs Strongly Convex

Convex function:

f (x ) ≥ f (y ) + ∇f (y )^>(x − y ) ∀x , y ∈ X Strongly Convex function:

f (x ) ≥ f (y ) + ∇f (y )^>(x − y ) +λ

2kx − y k²₂∀x , y ∈ X Global optimum is unique

strong convexity constant

Non-smooth function

Lipschitz continuous: e.g. absolute loss f (x ) = |x |

|f (x ) − f (y )| ≤ Gkx − y k₂ Lipschitz constant

Subgradient: f (x ) ≥ f (y ) +∂f (y )^>(x − y )

−1 −0.5 0 0.5 1

−0.2 0 0.2 0.4 0.6 0.8

|x| non−smooth

sub−gradient

Smooth function

e.g. logistic loss f (x ) = log(1 + exp(−x )) k∇f (x ) − ∇f (y )k₂ ≤ Lkx − y k₂

smoothness constant

−5 −4 −3 −2 −1 0 1 2 3 4 5

−1 0 1 2 3 4 5

6 log(1+exp(−x))

f(y)+f’(y)(x−y) y f(x)

Quadratic Function

Non-smooth function vs Smooth function

Non-smooth function

Lipschitz continuous: e.g. absolute loss f (x ) = |x |

|f (x ) − f (y )| ≤ Gkx − y k₂ Lipschitz constant

Subgradient: f (x ) ≥ f (y ) +∂f (y )^>(x − y )

−1 −0.5 0 0.5 1

−0.2 0 0.2 0.4 0.6 0.8

|x| non−smooth

sub−gradient

Smooth function

e.g. logistic loss f (x ) = log(1 + exp(−x )) k∇f (x ) − ∇f (y )k₂ ≤ Lkx − y k₂

smoothness constant

−5 −4 −3 −2 −1 0 1 2 3 4 5

−1 0 1 2 3 4 5

6 log(1+exp(−x))

f(y)+f’(y)(x−y) y f(x)

Quadratic Function

Non-smooth function

Lipschitz continuous: e.g. absolute loss f (x ) = |x |

|f (x ) − f (y )| ≤ Gkx − y k₂ Lipschitz constant

Subgradient: f (x ) ≥ f (y ) +∂f (y )^>(x − y )

−1 −0.5 0 0.5 1

−0.2 0 0.2 0.4 0.6 0.8

|x| non−smooth

sub−gradient

Smooth function

e.g. logistic loss f (x ) = log(1 + exp(−x )) k∇f (x ) − ∇f (y )k₂ ≤ Lkx − y k₂

smoothness constant

−5 −4 −3 −2 −1 0 1 2 3 4 5

−1 0 1 2 3 4 5

6 log(1+exp(−x))

f(y)+f’(y)(x−y) y f(x)

Quadratic Function

Next ...

min

w∈R^d

1 n

n

X

i =1

`(w^>xi, yi) + R(w) Part II: Optimization

stochastic optimization distributed optimization

Reduce Iteration Complexity: utilizing properties of functions

Part III: Randomization Classification, Regression SVD, K-means, Kernel methods

Reduce Data Size: utilizing properties of data

Please stay tuned!

Big Data Analytics: Optimization and Randomization

Part II: Optimization

2 Optimization

(Sub)Gradient Methods

Stochastic Optimization Algorithms for Big Data

Stochastic Optimization Distributed Optimization

Learning as Optimization

Regularized Empirical Loss Minimization min

w∈R^d

1 n

n X i =1

`(w^>x_i, y_i) + R(w)

| {z }

F (w)

Optimization (Sub)Gradient Methods

Convergence Measure

Most optimization algorithms are iterative wt+1= wt+ ∆wt

bT w

Convergence Rate: after T iterations, how good is the solution

F (wbT) − min

w F (w) ≤ (T )

0 20 40 60 80 100

0 0.1 0.2 0.3 0.4 0.5

iterations

objective

T ε

Total Runtime =Per-iteration Cost×Iteration Complexity

Optimization (Sub)Gradient Methods

Convergence Measure

Most optimization algorithms are iterative wt+1= wt+ ∆wt

Iteration Complexity: the number of iterations T () needed to have

F (w_bT) − min

w F (w) ≤  (  1)

good is the solution F (wbT) − min

w F (w) ≤ (T )

0 20 40 60 80 100

0 0.1 0.2 0.3 0.4 0.5

iterations

objective

T ε

Total Runtime =Per-iteration Cost×Iteration Complexity

Optimization (Sub)Gradient Methods

Convergence Measure

Most optimization algorithms are iterative wt+1= wt+ ∆wt

Iteration Complexity: the number of iterations T () needed to have

F (w_bT) − min

w F (w) ≤  (  1)

Convergence Rate: after T iterations, how good is the solution

F (wbT) − min

w F (w) ≤ (T )

0 20 40 60 80 100

0 0.1 0.2 0.3 0.4 0.5

iterations

objective

T ε

Convergence Measure

Most optimization algorithms are iterative wt+1= wt+ ∆wt

Iteration Complexity: the number of iterations T () needed to have

F (w_bT) − min

w F (w) ≤  (  1)

Convergence Rate: after T iterations, how good is the solution

F (wbT) − min

w F (w) ≤ (T )

0 20 40 60 80 100

0 0.1 0.2 0.3 0.4 0.5

iterations

objective

T ε

Total Runtime =Per-iteration Cost×Iteration Complexity

Big O(·) notation: explicit dependence on T or 

Convergence Rate Iteration Complexity linear O^µ^T (µ < 1) O

 log

1





sub-linear O^_T¹α

 α > 0 O

 1

^1/α



Why are we interested in Bounds?

More on Convergence Measure

Big O(·) notation: explicit dependence on T or 

Convergence Rate Iteration Complexity linear O^µ^T (µ < 1) O

 log

1





sub-linear O^_T¹α

 α > 0 O

 1

^1/α



Why are we interested in Bounds?

Convergence Rate Iteration Complexity linear O(µ^T) (µ < 1) O^log(¹_)^ sub-linear O(_T¹α) α > 0 O^_1/α^{1 }

0 20 40 60 80 100

0 0.1 0.2 0.3 0.4 0.5

iterations (T)

distance to optimum

0.5^T

seconds

More on Convergence Measure

Convergence Rate Iteration Complexity linear O(µ^T) (µ < 1) O^log(¹_)^ sub-linear O(_T¹α) α > 0 O^_1/α^{1 }

0 20 40 60 80 100

0 0.1 0.2 0.3 0.4 0.5

iterations (T)

distance to optimum

0.5^T 1/T

seconds

minutes

Convergence Rate Iteration Complexity linear O(µ^T) (µ < 1) O^log(¹_)^ sub-linear O(_T¹α) α > 0 O^_1/α^{1 }

0 20 40 60 80 100

0 0.1 0.2 0.3 0.4 0.5

iterations (T)

distance to optimum

0.5^T 1/T 1/T^0.5

seconds

minutes hours

More on Convergence Measure

Convergence Rate Iteration Complexity linear O(µ^T) (µ < 1) O^log(¹_)^ sub-linear O(_T¹α) α > 0 O^_1/α¹



0 20 40 60 80 100

0 0.1 0.2 0.3 0.4 0.5

iterations (T)

distance to optimum

0.5^T 1/T 1/T^0.5

seconds

minutes hours

Theoretically, we consider

O(µ^T) ≺ O

 1 T²



≺ O

1 T



≺ O

 1

√ T



log

1





≺ 1

√ ≺ 1

 ≺ 1

²

Non-smooth `(z)

hinge loss: `(w<sup>></sup>x, y ) = max(0, 1 − y w<sup>></sup>x) absolute loss: `(w^>x, y ) = |w^>x − y | Smooth `(z)

squared hinge loss: `(w<sup>></sup>x, y ) = max(0, 1 − y w<sup>></sup>x)<sup>2</sup> logistic loss: `(w^>x, y ) = log(1 + exp(−y w^>x)) square loss: `(w^>x, y ) = (w^>x − y )²

Strong convex V.S. Non-strongly convex

λ-strongly convex R(w)

`2regularizer: ^λ₂kwk²₂

Elastic net regularizer: τ kwk1+^λ₂kwk²₂ Non-strongly convex R(w)

unregularized problem: R(w) ≡ 0

`1regularizer: τ kwk1

F (w) = 1 n

n

X

i =1

`(w<sup>></sup>xi, yi) +λ 2kwk<sup>2</sup><sub>2</sub> Suppose `(z) is smooth

Full gradient: ∇F (w) = ¹_n^Pn_{i =1}∇`(w^>x_i, y_i) + λw Per-iteration cost: O(nd )

Gradient Descent

w_t = w_t−1− γ_t∇F (w_t−1) step size

Gradient Method in Machine Learning

F (w) = 1 n

n

X

i =1

`(w<sup>></sup>xi, yi) +λ 2kwk<sup>2</sup><sub>2</sub> Suppose `(z) is smooth

Full gradient: ∇F (w) = ¹_n^Pn_{i =1}∇`(w^>x_i, y_i) + λw Per-iteration cost: O(nd )

Gradient Descent

w_t = w_t−1− γ_t∇F (w_t−1) step size

F (w) = 1 n

n

X

i =1

`(w^>x_i, y_i) +λ 2kwk²₂ If λ = 0: R(w) is non-strongly convex

Iteration complexityO(¹_)

If λ > 0: R(w) is λ-strongly convex Iteration complexityO(_λ¹log(¹_))

Accelerated Gradient Method

Accelerated Gradient Descent

w_t = v_t−1− γ_t∇F (v_t−1)

v_t = w_t + η_t(w_t − w_t−1)

Momentum Step

wt is the output and vt is an auxiliary sequence.

Accelerated Gradient Descent

w_t = v_t−1− γ_t∇F (v_t−1)

v_t = w_t + η_t(w_t − w_t−1)

Momentum Step

wt is the output and vt is an auxiliary sequence.

Accelerated Gradient Method

F (w) = 1 n

n

X

i =1

`(w^>xi, yi) +λ 2kwk²₂ If λ = 0: R(w) is non-strongly convex

Iteration complexityO(^√1_), better than O(¹_)

If λ > 0: R(w) is λ-strongly convex Iteration complexityO(^√1

λlog(¹_)), better than O(¹_λlog(¹_))for small λ

Consider a more general case min

w∈R^d

F (w) = 1 n

n

X

i =1

`(w^>x_i, y_i) +R⁰(w) + τ kwk₁

| {z }

R(w)

R(w) = R⁰(w) + τ kwk₁

R⁰(w): λ-strongly convex and smooth

Deal with `1 regularizer

Consider a more general case min

w∈R^dF (w) = 1 n

n

X

i =1

`(w^>x_i, y_i) + R⁰(w)

| {z }

F⁰(w)

+τ kwk₁

R(w) = R⁰(w) + τ kwk1

R⁰(w): λ-strongly convex and smooth

Accelerated Gradient Descent

w_t = arg min

w∈R^d

∇F⁰(v_t−1)^>w + 1

2γ_tkw − v_t−1k²₂ + τ kwk₁ v_t = w_t + η_t(w_t − w_t−1)

Proximal mapping

Proximal mapping has close-form solution: Soft-thresholding Iteration complexity and runtime remain unchanged.

Deal with `1 regularizer

Accelerated Gradient Descent

w_t = arg min

w∈R^d

∇F⁰(v_t−1)^>w + 1

2γ_tkw − v_t−1k²₂ + τ kwk₁ v_t = w_t + η_t(w_t − w_t−1)

Proximal mapping

Proximal mapping has close-form solution: Soft-thresholding Iteration complexity and runtime remain unchanged.

F (w) = 1 n

n

X

i =1

`(w<sup>></sup>x<sub>i</sub>, y<sub>i</sub>) +λ 2kwk<sup>2</sup><sub>2</sub> Suppose `(z) is non-smooth

Full sub-gradient: ∂F (w) = _n^1Pn_{i =1}∂`(w^>xi, yi) + λw Sub-Gradient Descent

w_t = w_t−1− γ_t∂F (w_t−1)

Sub-Gradient Method in Machine Learning

F (w) = 1 n

n

X

i =1

`(w<sup>></sup>x<sub>i</sub>, y<sub>i</sub>) +λ 2kwk<sup>2</sup><sub>2</sub> Suppose `(z) is non-smooth

Full sub-gradient: ∂F (w) = _n^1Pn_{i =1}∂`(w^>xi, yi) + λw Sub-Gradient Descent

w_t = w_t−1− γ_t∂F (w_t−1)

F (w) = 1 n

n

X

i =1

`(w^>x_i, y_i) +λ 2kwk²₂ If λ = 0: R(w) is non-strongly convex

Iteration complexityO(_¹2)

If λ > 0: R(w) is λ-strongly convex Iteration complexityO(_λ¹ )

No efficient acceleration scheme in general

Problem Classes and Iteration Complexity

min

w∈R^d

1 n

n

X

i =1

`(w^>x_i, y_i) + R(w)

Iteration complexity

`(z) ≡ `(z, y ) Non-smooth Smooth R(w) Non-strongly convex O^_¹2

 O^√1_^

λ-strongly convex O^_λ^{1 } O^√1

λlog^1_^

Per-iteration cost: O(nd ), too high if n or d are large.

2 Optimization

(Sub)Gradient Methods

Stochastic Optimization Algorithms for Big Data

Stochastic Optimization Distributed Optimization

Stochastic First-Order Method by Data Sampling

Stochastic Gradient Descent (SGD)

Stochastic Variance Reduced Gradient (SVRG)

Stochastic Average Gradient Algorithm (SAGA) Stochastic Dual Coordinate Ascent (SDCA)

Accelerated Proximal Coordinate Gradient (APCG) Assumption: kx_ik ≤ 1 for any i

F (w) = 1 n

n

X

i =1

`(w<sup>></sup>x<sub>i</sub>, y<sub>i</sub>) +λ 2kwk<sup>2</sup><sub>2</sub> Full sub-gradient: ∂F (w) = <sub>n</sub><sup>1Pn</sup><sub>i =1</sub>∂`(w^>x_i, y_i) + λw

Randomly sample i ∈ {1, . . . , n}

Stochasticsub-gradient: ∂`(w^Txi, yi) + λw

Ei[∂`(w^Tx_i, y_i) + λw] = ∂F (w)

Basic SGD (Nemirovski & Yudin (1978))

Applicable in all settings!

min

w∈R^dF (w) = 1 n

n

X

i =1

`(w^>xi, yi) +λ 2kwk²₂

sample: it ∈ {1, . . . , n}

update: w_t = w_t−1− γ_t^∂`(w^T_t−1x_it, y_it) + λw_t−1^

output: w_T = 1 T

T

X

t=1

w_t

Applicable in all settings!

min

w∈R^dF (w) = 1 n

n

X

i =1

`(w^>xi, yi) +λ 2kwk²₂

sample: it ∈ {1, . . . , n}

update: w_t = w_t−1− γ_t^∂`(w^T_t−1x_it, y_it) + λw_t−1^

output: w_T = 1 T

T

X

t=1

w_t