Machine learning and optimization for massive data

Machine learning and optimization for massive data

Francis Bach

INRIA - Ecole Normale Sup´ erieure, Paris, France

É C O L E N O R M A L E S U P É R I E U R E

Joint work with Eric Moulines - IHES, May 2015

“Big data” revolution?

A new scientific context

• Technical progress

– Computation, storage and sensor costs

Moore’s law - Processors

Moore’s law - Storage

DNA sequencing

“Big data” revolution?

A new scientific context

• Data everywhere: size does not (always) matter

• Science and industry

• Size and variety

• Learning from examples

– n observations in dimension p

Search engines - Advertising

Marketing - Personalized recommendation

Visual object recognition

Bioinformatics

• Protein: Crucial elements of cell life

• Massive data: 2 millions for humans

• Complex data

Astronomy

Square Kilometer Array (2024): 10

Go par jour

Context

Machine learning for “big data”

• Large-scale machine learning: large p, large n – p : dimension of each observation (input)

– n : number of observations

• Examples: computer vision, bioinformatics, advertising – Ideal running-time complexity: O(pn)

– Going back to simple methods

– Stochastic gradient methods (Robbins and Monro, 1951) – Mixing statistics and optimization

– Using smoothness to go beyond stochastic gradient descent

Context

Machine learning for “big data”

• Large-scale machine learning: large p, large n – p : dimension of each observation (input)

– n : number of observations

• Examples: computer vision, bioinformatics, advertising

• Ideal running-time complexity: O(pn) – Going back to simple methods

– Stochastic gradient methods (Robbins and Monro, 1951) – Mixing statistics and optimization

– Using smoothness to go beyond stochastic gradient descent

Context

Machine learning for “big data”

• Large-scale machine learning: large p, large n – p : dimension of each observation (input)

– n : number of observations

• Examples: computer vision, bioinformatics, advertising

• Ideal running-time complexity: O(pn)

• Going back to simple methods

– Stochastic gradient methods (Robbins and Monro, 1951) – Mixing statistics and optimization

– Using smoothness to go beyond stochastic gradient descent

Scaling to large problems with convex optimization

“Retour aux sources”

• 1950’s: computers not powerful enough

IBM “1620”, 1959

CPU frequency: 50 KHz Price > 100 000 dollars

• 2010’s: Data too massive

• One pass through the data (Robbins et Monro, 1956)

− Algorithm: θⁿ = θ_n−1 − γⁿℓ^′(y_n, hθⁿ⁻¹, Φ(x_n)i)Φ(xⁿ)

Scaling to large problems with convex optimization

“Retour aux sources”

• 1950’s: computers not powerful enough

IBM “1620”, 1959

CPU frequency: 50 KHz Price > 100 000 dollars

• 2010’s: Data too massive

• One pass through the data (Robbins et Monro, 1951) – Algorithm: θ_n = θ_n−1 − γⁿℓ^′(y_n, hθⁿ⁻¹, Φ(x_n)i)Φ(xⁿ)

Outline

• Introduction: stochastic approximation algorithms – Supervised machine learning and convex optimization – Stochastic gradient and averaging

– Strongly convex vs. non-strongly convex – O(1/(nµ)) vs. O(1/√

n) for all non-smooth functions

• Least-squares regressions – Constant step-sizes

– O(1/n) convergence rate for all quadratic functions

• Smooth losses

– Online Newton steps

– O(1/n) convergence rate for all convex functions

Supervised machine learning

• Data: n observations (xⁱ, y_i) ∈ X × Y, i = 1, . . . , n, i.i.d.

• Prediction as a linear function hθ, Φ(x)i of features Φ(x) ∈ R^p

• (regularized) empirical risk minimization: find ˆθ solution of

θ∈Rmin^p

1 n

n

X

i=1

ℓ y_i, hθ, Φ(xⁱ)i

+ µΩ(θ) convex data fitting term + regularizer

Usual losses

• Regression: y ∈ R, prediction ˆy = hθ, Φ(x)i – quadratic loss ¹₂(y − ˆy)² = ¹₂(y − hθ, Φ(x)i)²

Usual losses

• Regression: y ∈ R, prediction ˆy = hθ, Φ(x)i – quadratic loss ¹₂(y − ˆy)² = ¹₂(y − hθ, Φ(x)i)²

• Classification : y ∈ {−1, 1}, prediction ˆy = sign(hθ, Φ(x)i) – loss of the form ℓ(y hθ, Φ(x)i)

– “True” 0-1 loss: ℓ(y hθ, Φ(x)i) = 1y hθ,Φ(x)i<0

– Usual convex losses:

−30 −2 −1 0 1 2 3 4

1 2 3 4 5

0−1 hinge square logistic

Supervised machine learning

• Data: n observations (xⁱ, y_i) ∈ X × Y, i = 1, . . . , n, i.i.d.

• Prediction as a linear function hθ, Φ(x)i of features Φ(x) ∈ R^p

• (regularized) empirical risk minimization: find ˆθ solution of

θ∈Rmin^p

1 n

n

X

i=1

ℓ y_i, hθ, Φ(xⁱ)i

+ µΩ(θ) convex data fitting term + regularizer

Supervised machine learning

• Data: n observations (xⁱ, y_i) ∈ X × Y, i = 1, . . . , n, i.i.d.

• Prediction as a linear function hθ, Φ(x)i of features Φ(x) ∈ R^p

• (regularized) empirical risk minimization: find ˆθ solution of

θ∈Rmin^p

1 n

n

X

i=1

ℓ y_i, hθ, Φ(xⁱ)i

+ µΩ(θ)

convex data fitting term + regularizer

• Empirical risk: ˆf (θ) = _n¹ Pn

i=1 ℓ(y_i, hθ, Φ(xⁱ)i) training cost

• Expected risk: f(θ) = E^(x,y)ℓ(y, hθ, Φ(x)i) testing cost

• Two fundamental questions: (1) computing ˆθ and (2) analyzing ˆθ – May be tackled simultaneously

Supervised machine learning

• Data: n observations (xⁱ, y_i) ∈ X × Y, i = 1, . . . , n, i.i.d.

• Prediction as a linear function hθ, Φ(x)i of features Φ(x) ∈ R^p

• (regularized) empirical risk minimization: find ˆθ solution of

θ∈Rmin^p

1 n

n

X

i=1

ℓ y_i, hθ, Φ(xⁱ)i

+ µΩ(θ)

convex data fitting term + regularizer

• Empirical risk: ˆf (θ) = _n¹ Pn

i=1 ℓ(y_i, hθ, Φ(xⁱ)i) training cost

• Expected risk: f(θ) = E^(x,y)ℓ(y, hθ, Φ(x)i) testing cost

• Two fundamental questions: (1) computing ˆθ and (2) analyzing ˆθ – May be tackled simultaneously

Smoothness and strong convexity

• A function g : R^p → R is L-smooth if and only if it is twice differentiable and

∀θ ∈ R^p, eigenvaluesg^′′(θ) 6 L

smooth non−smooth

Smoothness and strong convexity

• A function g : R^p → R is L-smooth if and only if it is twice differentiable and

∀θ ∈ R^p, eigenvaluesg^′′(θ) 6 L

• Machine learning – with g(θ) = _n¹ Pn

i=1 ℓ(y_i, hθ, Φ(xⁱ)i) – Hessian ≈ covariance matrix _n¹ Pn

i=1 Φ(x_i) ⊗ Φ(xⁱ) – Bounded data

Smoothness and strong convexity

• A twice differentiable function g : R^p → R is µ-strongly convex if and only if

∀θ ∈ R^p, eigenvaluesg^′′(θ) > µ

convex strongly

convex

Smoothness and strong convexity

• A twice differentiable function g : R^p → R is µ-strongly convex if and only if

∀θ ∈ R^p, eigenvaluesg^′′(θ) > µ

(large µ) (small µ)

Smoothness and strong convexity

• A twice differentiable function g : R^p → R is µ-strongly convex if and only if

∀θ ∈ R^p, eigenvaluesg^′′(θ) > µ

• Machine learning – with g(θ) = _n¹ Pn

i=1 ℓ(y_i, hθ, Φ(xⁱ)i) – Hessian ≈ covariance matrix _n¹ Pn

i=1 Φ(x_i) ⊗ Φ(xⁱ)

– Data with invertible covariance matrix (low correlation/dimension)

Smoothness and strong convexity

• A twice differentiable function g : R^p → R is µ-strongly convex if and only if

∀θ ∈ R^p, eigenvaluesg^′′(θ) > µ

• Machine learning – with g(θ) = _n¹ Pn

i=1 ℓ(y_i, hθ, Φ(xⁱ)i) – Hessian ≈ covariance matrix _n¹ Pn

i=1 Φ(x_i) ⊗ Φ(xⁱ)

– Data with invertible covariance matrix (low correlation/dimension)

• Adding regularization by ^µ₂kθk²

– creates additional bias unless µ is small

Iterative methods for minimizing smooth functions

• Assumption: g convex and smooth on R^p

• Gradient descent: θ^t = θ_t−1 − γ^t g^′(θ_t−1)

– O(1/t) convergence rate for convex functions

– O(e^−ρt) convergence rate for strongly convex functions

Iterative methods for minimizing smooth functions

• Assumption: g convex and smooth on R^p

• Gradient descent: θ^t = θ_t−1 − γ^t g^′(θ_t−1)

– O(1/t) convergence rate for convex functions

– O(e^−ρt) convergence rate for strongly convex functions

• Newton method: θ^t = θ_t−1 − g^′′(θ_t−1)⁻¹g^′(θ_t−1) – O e^−ρ2t
convergence rate

Iterative methods for minimizing smooth functions

• Assumption: g convex and smooth on R^p

• Gradient descent: θ^t = θ_t−1 − γ^t g^′(θ_t−1)

– O(1/t) convergence rate for convex functions

– O(e^−ρt) convergence rate for strongly convex functions

• Newton method: θ^t = θ_t−1 − g^′′(θ_t−1)⁻¹g^′(θ_t−1) – O e^−ρ2t
convergence rate

• Key insights from Bottou and Bousquet (2008)

1. In machine learning, no need to optimize below statistical error 2. In machine learning, cost functions are averages

⇒ Stochastic approximation

Stochastic approximation

• Goal: Minimizing a function f defined on R^p

– given only unbiased estimates f_n^′ (θ_n) of its gradients f^′(θ_n) at certain points θ_n ∈ R^p

Stochastic approximation

• Goal: Minimizing a function f defined on R^p

– given only unbiased estimates f_n^′ (θ_n) of its gradients f^′(θ_n) at certain points θ_n ∈ R^p

• Machine learning - statistics

– f (θ) = Efn(θ) = E ℓ(yn, hθ, Φ(xⁿ)i) = generalization error

– Loss for a single pair of observations: f_n(θ) = ℓ(y_n, hθ, Φ(xⁿ)i) – Expected gradient:

f^′(θ) = Ef_n^′ (θ) = E ℓ^′(y_n, hθ, Φ(xⁿ)i) Φ(xⁿ)

• Beyond convex optimization: see, e.g., Benveniste et al. (2012)

Convex stochastic approximation

• Key assumption: smoothness and/or strong convexity

• Key algorithm: stochastic gradient descent (a.k.a. Robbins-Monro) θ_n = θ_n−1 − γⁿ f_n^′ (θ_n−1)

– Polyak-Ruppert averaging: ¯θ_n = _n+1¹ P_n

k=0 θ_k

– Which learning rate sequence γ_n? Classical setting: γ_n = Cn^−α

Convex stochastic approximation

• Key assumption: smoothness and/or strong convexity

• Key algorithm: stochastic gradient descent (a.k.a. Robbins-Monro) θ_n = θ_n−1 − γⁿ f_n^′ (θ_n−1)

– Polyak-Ruppert averaging: ¯θ_n = _n+1¹ P_n

k=0 θ_k

– Which learning rate sequence γ_n? Classical setting: γ_n = Cn^−α

• Running-time = O(np)

– Single pass through the data – One line of code among many

Convex stochastic approximation Existing work

• Known global minimax rates of convergence for non-smooth problems (Nemirovsky and Yudin, 1983; Agarwal et al., 2012)

– Strongly convex: O((µn)⁻¹)

Attained by averaged stochastic gradient descent with γ_n ∝ (µn)⁻¹ – Non-strongly convex: O(n^−1/2)

Attained by averaged stochastic gradient descent with γ_n ∝ n^−1/2

Convex stochastic approximation Existing work

• Known global minimax rates of convergence for non-smooth problems (Nemirovsky and Yudin, 1983; Agarwal et al., 2012)

– Strongly convex: O((µn)⁻¹)

Attained by averaged stochastic gradient descent with γ_n ∝ (µn)⁻¹ – Non-strongly convex: O(n^−1/2)

Attained by averaged stochastic gradient descent with γ_n ∝ n^−1/2

• Asymptotic analysis of averaging (Polyak and Juditsky, 1992;

Ruppert, 1988)

– All step sizes γ_n = Cn^−α with α ∈ (1/2, 1) lead to O(n⁻¹) for smooth strongly convex problems

A single algorithm with global adaptive convergence rate for smooth problems?

Convex stochastic approximation Existing work

• Known global minimax rates of convergence for non-smooth problems (Nemirovsky and Yudin, 1983; Agarwal et al., 2012)

– Strongly convex: O((µn)⁻¹)

Attained by averaged stochastic gradient descent with γ_n ∝ (µn)⁻¹ – Non-strongly convex: O(n^−1/2)

Attained by averaged stochastic gradient descent with γ_n ∝ n^−1/2

• Asymptotic analysis of averaging (Polyak and Juditsky, 1992;

Ruppert, 1988)

– All step sizes γ_n = Cn^−α with α ∈ (1/2, 1) lead to O(n⁻¹) for smooth strongly convex problems

• A single algorithm for smooth problems with convergence rate O(1/n) in all situations?

Least-mean-square algorithm

• Least-squares: f(θ) = ¹₂E(y_n − hΦ(xⁿ), θi)² with θ ∈ R^p

– SGD = least-mean-square algorithm (see, e.g., Macchi, 1995) – usually studied without averaging and decreasing step-sizes

– with strong convexity assumption EΦ(x_n) ⊗ Φ(xⁿ) = H < µ · Id

Least-mean-square algorithm

• Least-squares: f(θ) = ¹₂E(y_n − hΦ(xⁿ), θi)² with θ ∈ R^p

– SGD = least-mean-square algorithm (see, e.g., Macchi, 1995) – usually studied without averaging and decreasing step-sizes

– with strong convexity assumption EΦ(x_n) ⊗ Φ(xⁿ) = H < µ · Id

• New analysis for averaging and constant step-size γ = 1/(4R²) – Assume kΦ(xⁿ)k 6 R and |yⁿ − hΦ(xⁿ), θ_∗i| 6 σ almost surely – No assumption regarding lowest eigenvalues of H

– Main result: Ef(¯θ_n) − f(θ^∗) 6 4σ²p

n + 4R²kθ⁰ − θ^∗k² n

• Matches statistical lower bound (Tsybakov, 2003)

– Non-asymptotic robust version of Gy¨orfi and Walk (1996)

Markov chain interpretation of constant step sizes

• LMS recursion for fⁿ(θ) = ¹₂ y_n − hΦ(xⁿ), θi
2

θ_n = θ_n−1 − γ hΦ(xⁿ), θ_n−1i − yⁿ
Φ(x_n)

• The sequence (θⁿ)_n is a homogeneous Markov chain – convergence to a stationary distribution π_γ

– with expectation ¯θ_γ ^def= R θπ_γ(dθ) - For least-squares, ¯θ_γ = θ_∗

¯ θγ

θ₀

θn

Markov chain interpretation of constant step sizes

• LMS recursion for fⁿ(θ) = ¹₂ y_n − hΦ(xⁿ), θi
2

θ_n = θ_n−1 − γ hΦ(xⁿ), θ_n−1i − yⁿ
Φ(x_n)

• The sequence (θⁿ)_n is a homogeneous Markov chain – convergence to a stationary distribution π_γ

– with expectation ¯θ_γ ^def= R θπ_γ(dθ)

• For least-squares, ¯θ_γ = θ_∗

θ∗

θ₀

θ_n

Markov chain interpretation of constant step sizes

• LMS recursion for fⁿ(θ) = ¹₂ y_n − hΦ(xⁿ), θi
2

θ_n = θ_n−1 − γ hΦ(xⁿ), θ_n−1i − yⁿ
Φ(x_n)

• The sequence (θⁿ)_n is a homogeneous Markov chain – convergence to a stationary distribution π_γ

– with expectation ¯θ_γ ^def= R θπ_γ(dθ)

• For least-squares, ¯θ_γ = θ_∗

θ∗

θ₀

θ_n

¯ θ_n

Markov chain interpretation of constant step sizes

• LMS recursion for fⁿ(θ) = ¹₂ y_n − hΦ(xⁿ), θi
2

θ_n = θ_n−1 − γ hΦ(xⁿ), θ_n−1i − yⁿ
Φ(x_n)

• The sequence (θⁿ)_n is a homogeneous Markov chain – convergence to a stationary distribution π_γ

– with expectation ¯θ_γ ^def= R θπ_γ(dθ)

• For least-squares, ¯θ_γ = θ_∗

– θ_n does not converge to θ_∗ but oscillates around it – oscillations of order √γ

• Ergodic theorem:

– Averaged iterates converge to ¯θ_γ = θ_∗ at rate O(1/n)

Simulations - synthetic examples

• Gaussian distributions - p = 20

0 2 4 6

−5

−4

−3

−2

−1 0

log10(n) log 10[f(θ)−f(θ *)]

synthetic square

1/2R² 1/8R² 1/32R² 1/2R²n^1/2

Simulations - benchmarks

• alpha (p = 500, n = 500 000), news (p = 1 300 000, n = 20 000)

0 2 4 6

−2

−1.5

−1

−0.5 0 0.5 1

log10(n) log 10[f(θ)−f(θ *)]

alpha square C=1 test

1/R² 1/R²n^1/2 SAG

0 2 4 6

−2

−1.5

−1

−0.5 0 0.5 1

log10(n)

alpha square C=opt test

C/R² C/R²n^1/2 SAG

0 2 4

−0.8

−0.6

−0.4

−0.2 0 0.2

log10(n) log 10[f(θ)−f(θ *)]

news square C=1 test

1/R² 1/R²n^1/2 SAG

0 2 4

−0.8

−0.6

−0.4

−0.2 0 0.2

log10(n)

news square C=opt test

C/R² C/R²n^1/2 SAG

Beyond least-squares - Markov chain interpretation

• Recursion θⁿ = θ_n−1 − γfn^′ (θ_n−1) also defines a Markov chain – Stationary distribution π_γ such that R f^′(θ)π_γ(dθ) = 0

– When f^′ is not linear, f^′(R θπ_γ(dθ)) 6= R f^′(θ)π_γ(dθ) = 0

Beyond least-squares - Markov chain interpretation

• Recursion θⁿ = θ_n−1 − γfn^′ (θ_n−1) also defines a Markov chain – Stationary distribution π_γ such that R f^′(θ)π_γ(dθ) = 0

– When f^′ is not linear, f^′(R θπ_γ(dθ)) 6= R f^′(θ)π_γ(dθ) = 0

• θ_n oscillates around the wrong value ¯θ_γ 6= θ^∗

¯ θγ

θ₀

θn

¯ θn

θ∗

Beyond least-squares - Markov chain interpretation

• Recursion θⁿ = θ_n−1 − γfn^′ (θ_n−1) also defines a Markov chain – Stationary distribution π_γ such that R f^′(θ)π_γ(dθ) = 0

– When f^′ is not linear, f^′(R θπ_γ(dθ)) 6= R f^′(θ)π_γ(dθ) = 0

• θ_n oscillates around the wrong value ¯θ_γ 6= θ^∗ – moreover, kθ^∗ − θⁿk = O^p(√γ)

• Ergodic theorem

– averaged iterates converge to ¯θ_γ 6= θ^∗ at rate O(1/n) – moreover, kθ^∗ − ¯θ_γk = O(γ) (Bach, 2013)

Simulations - synthetic examples

• Gaussian distributions - p = 20

0 2 4 6

−5

−4

−3

−2

−1 0

log10(n) log 10[f(θ)−f(θ *)]

synthetic logistic − 1

1/2R² 1/8R² 1/32R² 1/2R²n^1/2

Restoring convergence through online Newton steps

• Known facts

1. Averaged SGD with γ_n ∝ n^−1/2 leads to robust rate O(n^−1/2) for all convex functions

2. Averaged SGD with γ_n constant leads to robust rate O(n⁻¹) for all convex quadratic functions

3. Newton’s method squares the error at each iteration for smooth functions

4. A single step of Newton’s method is equivalent to minimizing the quadratic Taylor expansion

– Online Newton step

– Rate: O((n^−1/2)² + n⁻¹) = O(n⁻¹) – Complexity: O(p) per iteration

Restoring convergence through online Newton steps

• Known facts

1. Averaged SGD with γ_n ∝ n^−1/2 leads to robust rate O(n^−1/2) for all convex functions

2. Averaged SGD with γ_n constant leads to robust rate O(n⁻¹) for all convex quadratic functions ⇒ O(n⁻¹)

3. Newton’s method squares the error at each iteration for smooth functions ⇒ O((n^−1/2)²)

4. A single step of Newton’s method is equivalent to minimizing the quadratic Taylor expansion

• Online Newton step

– Rate: O((n^−1/2)² + n⁻¹) = O(n⁻¹) – Complexity: O(p) per iteration

Restoring convergence through online Newton steps

• The Newton step for f = Efⁿ(θ) ^def= Eℓ(y_n, hθ, Φ(xⁿ)i)

at ˜θ is equivalent to minimizing the quadratic approximation

g(θ) = f (˜θ) + hf^′(˜θ), θ − ˜θi + ¹₂hθ − ˜θ, f^′′(˜θ)(θ − ˜θ)i

= f (˜θ) + hEf_n^′ (˜θ), θ − ˜θi + ¹₂hθ − ˜θ,Ef_n^′′(˜θ)(θ − ˜θ)i

= Eh

f (˜θ) + hf_n^′ (˜θ), θ − ˜θi + ¹₂hθ − ˜θ, f_n^′′(˜θ)(θ − ˜θ)ii

Restoring convergence through online Newton steps

• The Newton step for f = Efⁿ(θ) ^def= Eℓ(y_n, hθ, Φ(xⁿ)i)

at ˜θ is equivalent to minimizing the quadratic approximation

g(θ) = f (˜θ) + hf^′(˜θ), θ − ˜θi + ¹₂hθ − ˜θ, f^′′(˜θ)(θ − ˜θ)i

= f (˜θ) + hEf_n^′ (˜θ), θ − ˜θi + ¹₂hθ − ˜θ,Ef_n^′′(˜θ)(θ − ˜θ)i

= Eh

f (˜θ) + hf_n^′ (˜θ), θ − ˜θi + ¹₂hθ − ˜θ, f_n^′′(˜θ)(θ − ˜θ)ii

• Complexity of least-mean-square recursion for g is O(p) θ_n = θ_n−1 − γf_n^′ (˜θ) + f_n^′′(˜θ)(θ_n−1 − ˜θ)

– f_n^′′(˜θ) = ℓ^′′(y_n, h˜θ, Φ(x_n)i)Φ(xⁿ) ⊗ Φ(xⁿ) has rank one

– New online Newton step without computing/inverting Hessians

Choice of support point for online Newton step

• Two-stage procedure

(1) Run n/2 iterations of averaged SGD to obtain ˜θ

(2) Run n/2 iterations of averaged constant step-size LMS

– Reminiscent of one-step estimators (see, e.g., Van der Vaart, 2000) – Provable convergence rate of O(p/n) for logistic regression

– Additional assumptions but no strong convexity

Choice of support point for online Newton step

• Two-stage procedure

(1) Run n/2 iterations of averaged SGD to obtain ˜θ

(2) Run n/2 iterations of averaged constant step-size LMS

– Reminiscent of one-step estimators (see, e.g., Van der Vaart, 2000) – Provable convergence rate of O(p/n) for logistic regression

– Additional assumptions but no strong convexity

• Update at each iteration using the current averaged iterate – Recursion: θ_n = θ_n−1 − γf_n^′ (¯θ_n−1) + f_n^′′(¯θ_n−1)(θ_n−1 − ¯θ_n−1) – No provable convergence rate (yet) but best practical behavior – Note (dis)similarity with regular SGD: θ_n = θ_n−1 − γfn^′ (θ_n−1)

Simulations - synthetic examples

• Gaussian distributions - p = 20

0 2 4 6

−5

−4

−3

−2

−1 0

log10(n) log 10[f(θ)−f(θ *)]

synthetic logistic − 1

1/2R² 1/8R² 1/32R² 1/2R²n^1/2

0 2 4 6

−5

−4

−3

−2

−1 0

log10(n) log 10[f(θ)−f(θ *)]

synthetic logistic − 2

every 2^p every iter.

2−step

2−step−dbl.

Simulations - benchmarks

• alpha (p = 500, n = 500 000), news (p = 1 300 000, n = 20 000)

0 2 4 6

−2.5

−2

−1.5

−1

−0.5 0 0.5

log10(n) log 10[f(θ)−f(θ *)]

alpha logistic C=1 test

1/R² 1/R²n^1/2 SAG Adagrad Newton

0 2 4 6

−2.5

−2

−1.5

−1

−0.5 0 0.5

log10(n)

alpha logistic C=opt test

C/R² C/R²n^1/2 SAG Adagrad Newton

0 2 4

−1

−0.8

−0.6

−0.4

−0.2 0 0.2

log10(n) log 10[f(θ)−f(θ *)]

news logistic C=1 test

1/R² 1/R²n^1/2 SAG Adagrad Newton

0 2 4

−1

−0.8

−0.6

−0.4

−0.2 0 0.2

log10(n)

news logistic C=opt test

C/R² C/R²n^1/2 SAG Adagrad Newton

Conclusions

• Constant-step-size averaged stochastic gradient descent – Reaches convergence rate O(1/n) in all regimes

– Improves on the O(1/√

n) lower-bound of non-smooth problems – Efficient online Newton step for non-quadratic problems

– Robustness to step-size selection

• Extensions and future work – Going beyond a single pass – Pre-conditioning

– Proximal extensions fo non-differentiable terms

– kernels and non-parametric estimation (Dieuleveut and Bach, 2014) – line-search

– parallelization

– Non-convex problems

References

A. Agarwal, P. L. Bartlett, P. Ravikumar, and M. J. Wainwright. Information-theoretic lower bounds on the oracle complexity of stochastic convex optimization. Information Theory, IEEE Transactions on, 58(5):3235–3249, 2012.

F. Bach. Adaptivity of averaged stochastic gradient descent to local strong convexity for logistic regression. Technical Report 00804431, HAL, 2013.

Albert Benveniste, Michel M´etivier, and Pierre Priouret. Adaptive algorithms and stochastic approximations. Springer Publishing Company, Incorporated, 2012.

L. Bottou and O. Bousquet. The tradeoffs of large scale learning. In Adv. NIPS, 2008.

A. Dieuleveut and F. Bach. Non-parametric Stochastic Approximation with Large Step sizes. Technical report, ArXiv, 2014.

L. Gy¨orfi and H. Walk. On the averaged stochastic approximation for linear regression. SIAM Journal on Control and Optimization, 34(1):31–61, 1996.

O. Macchi. Adaptive processing: The least mean squares approach with applications in transmission.

Wiley West Sussex, 1995.

A. S. Nemirovsky and D. B. Yudin. Problem complexity and method efficiency in optimization. Wiley

& Sons, 1983.

B. T. Polyak and A. B. Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30(4):838–855, 1992.

H. Robbins and S. Monro. A stochastic approximation method. Ann. Math. Statistics, 22:400–407, 1951. ISSN 0003-4851.

D. Ruppert. Efficient estimations from a slowly convergent Robbins-Monro process. Technical Report 781, Cornell University Operations Research and Industrial Engineering, 1988.

A. B. Tsybakov. Optimal rates of aggregation. 2003.

A. W. Van der Vaart. Asymptotic statistics, volume 3. Cambridge Univ. press, 2000.