Optimized first-order minimization methods

Optimized first-order minimization methods

with applications to image reconstruction and ML

Donghwan Kim & Jeffrey A. Fessler EECS Dept., BME Dept., Dept. of Radiology

University of Michigan

http://web.eecs.umich.edu/˜fessler

TUM Seminar

Disclosure

• Research support from GE Healthcare

• Supported in part by NIH grant U01 EB018753 • Equipment support from Intel Corporation

Lower-dose X-ray CT image reconstruction

Thin-slice FBP ASIR Statistical

Seconds A bit longer Much longer

Image reconstruction as an optimization problem:

ˆ x = arg min x0 1 2ky − Axk 2 W + R(x)

y data, A system model, W statistics, R(x) regularizer (Same sinogram, so all at samedose)

Outline

Motivation Problem setting Existing algorithms

Gradient descent

Nesterov’s “optimal” first-order method

Optimizing first-order minimization methods Numerical examples

Logistic regression for machine learning CT image reconstruction

Further acceleration using OS

Generalizing OGM Summary / future work

Outline

Motivation

Problem setting

Existing algorithms Gradient descent

Nesterov’s “optimal” first-order method Optimizing first-order minimization methods Numerical examples

Logistic regression for machine learning CT image reconstruction

Further acceleration using OS Generalizing OGM

Optimization problem setting

ˆ

x ∈ arg min x f (x)

I Unconstrained

I Large-scale (Hessian ∇2f too big to store and/or undefined) I image reconstruction / inverse problems

I big-data / machine learning I ...

I Cost function assumptions (throughout) I _{f : R}M_{7→ R}

I convex (need not be strictly convex) I non-empty set of global minimizers:

ˆ

x ∈ X∗=x?∈ RM _{: f (x}

?) ≤ f (x), ∀x ∈ RM I smooth (differentiable with L-Lipschitz gradient)

Example: Fair potential function

Fair’s potential function [1] (similar to Huber function and hyperbola): ψ(z) = δ2[|z/δ| − log(1 + |z/δ|)] ˙ ψ(z) = z 1 + |z/δ| ¨ ψ(z) = 1 (1 + |z/δ|)2 ≤ 1. Thus L = 1. -1 0 1 3 0 1 z f(z) -1 0 1 3 -1 0 1 z f’(z)

Example: Machine learning

To learn weights x of binary classifier given feature vectors {vi}

and labels {yi = ±1}: ˆ x = arg min x f (x), f (x) = X i ψ(yihx, vii) . loss functions ψ(z) I _{0-1: I{z≤0}} I exponential: exp(−z)

I logistic: log(1 + exp(−z))

I hinge: max {0, 1 − z} -2 0 2 z 0 1 8 ψ ( z )

Loss functions (surrogates) exponential hinge logistic 0-1

Outline

Motivation Problem setting

Existing algorithms

Gradient descent

Nesterov’s “optimal” first-order method Optimizing first-order minimization methods Numerical examples

Logistic regression for machine learning CT image reconstruction

Further acceleration using OS Generalizing OGM

Gradient descent

Iteration with step size 1/L ensures monotonic descent of f :

xn+1 = xn− 1 L∇f (xn) . Telescoping: xn+1 = x0− 1 L n X k=0 ∇f (x_k) .

Gradient descent convergence rate

I Classic O(1/n) convergence rate of cost function descent:

f (xn) − f (x?) | {z } inaccuracy ≤ L kx0− x?k 2 2 2n .

I Drori & Teboulle (2014) derive tightest inaccuracy bound:

f (xn) − f (x?) ≤

L kx0− x?k22

4n + 2 .

I They construct a Huber-like function f for which GD achieves that bound =⇒ case closed for GD with step size 1/L.

Generalizing GD slightly

I GD with general step size h:

xn+1= xn− h

L∇f (xn) .

I Classical monotone descent result:

h ∈ (0, 2) =⇒ f (xn+1) < f (xn) when xn is not a minimizer. I If f is quadratic, then asymptotic best is

h∗ =

2L

Generalizing GD slightly

I GD with general step size h:

xn+1= xn− h

L∇f (xn) .

I More generally, Taylor et al. [3] conjecture:

f (xn) − f (x?) ≤ L kx0− x?k22 2 max  ₁ 2Nh+ 1, (1 −h) 2N_.

I Proof for 0 <h≤ 1 by Drori and Teboulle [2]

I Upper bounds achieved by Huber-like function and quadratic function f (x ) = (L/2)x2_{respectively.}

I Besthdepends on N !

Heavy ball method

Heavy ball iteration (Polyak, 1987):

xn+1= xn− α L∇f (xn) + β (x| n− x{z n−1)} momentum! (recursive form for implementing) xn+1= xn− 1 L n X k=0 αβn−k | {z } coefficients ∇f (x_k) (summation form for analysis)

• How to choose α and β?

General first-order method classes

I General “first-order” (GFO) method:

xn+1= function(x0, f (x0), ∇f (x0), . . . , f (xn), ∇f (xn))

I First-order (FO) methods with fixed step-size coefficients:

xn+1 = xn− 1 L n X k=0 hn+1,k∇f (xk) Primary goals:

I Analyze convergence rate of FO for any given {hn,k} I Optimize step-size coefficients {hn,k}

I fast convergence

I efficient recursive implementation

Example: Barzilai-Borwein gradient method

Barzilai & Borwein, 1988 g(n) , ∇f (xn) αn= kxn− xn−1k22 hxn− xn−1, g(n)− g(n−1)i xn+1= xn− αn∇f (xn) .

I In “general” first-order (GFO) class, but

I Not in class FO with fixed step-size coefficients. Nor are methods like

I steepest descent (with line search), I conjugate gradient,

Nesterov’s fast gradient method (FGM1)

zn+1= xn− 1 L∇f (xn) (usual GD update) tn+1= 1 2  1 + q 1 + 4t2 n 

(magic momentum factors)

xn+1= zn+1+ tn− 1

tn+1

(zn+1− zn) (update with momentum) .

Reverts to GD if tn= 1, ∀n. FGM1 is in class FO: xn+1= xn− 1 L n X k=0 hn+1,k∇f (xk) hn+1,k=          tn− 1 tn+1 hn,k, k = 0, . . . , n − 2 tn− 1 tn+1 (hn,n−1− 1) , k = n − 1 1 +tn− 1 tn+1 , k = n.    1 0 0 0 0 0 0 1.25 0 0 0 0 0 0.10 1.40 0 0 0 0 0.05 0.20 1.50 0 0 0 0.03 0.11 0.29 1.57 0 0 0.02 0.07 0.18 0.36 1.62   

Nesterov FGM1 optimal convergence rate

Shown by Nesterov to be O(1/n2_{) for “auxiliary” sequence:}

f (zn) − f (x?) ≤

2L kx0− x?k22

(n + 1)2 .

Nesterov constructed a function f such that any first-order method achieves 3 32L kx0− x?k 2 2 (n + 1)2 ≤ f (xn) − f (x?) .

Thus O(1/n2_{) rate of FGM1 is optimal.}

New results(Donghwan Kim & JF, 2016):

• Bound on convergence rate of primary sequence {xn}:

f (xn) − f (x?) ≤

2L kx0− x?k2₂

(n + 2)2 .

Overview

First-order (FO) method with fixed step-size coefficients:

xn+1= xn− 1 L n X k=0 hn+1,k∇f (xk)

I Analyze (i.e., bound) convergence rate as a function of I number of iterations N

I Lipschitz constant L

I step-size coefficients H = {hn+1,k} I Distance to a solution: R = kx0− x?k

I Optimize H by minimizing the bound

Ideal “universal” bound for first-order methods

try to bound the worst-case convergence rate of a FO method: B1(H, R, L, N, M) , max f ∈FL max x0,x1,...,xN∈RM max x?∈X∗(f ) kx0−x?k≤R f (xN) − f (x?) such that xn+1 = xn− 1 L n X k=0 hn+1,k∇f (xk), n = 0, . . . , N − 1.

Clearly for any FO method, this cost-function bound would hold: f (x ) − f (x?) ≤ B1(H, R, L, N, M).

Towards practical bounds for first-order methods

1

2Lk∇f (x) − ∇f (z)k

2_{≤ f (x) − f (z) − h∇f (z), x − zi,}

∀x, z ∈ RM. Drori & Teboulle (2014) use this inequality to propose a “more

tractable” (finite-dimensional) bound: B2(H, R, L, N, M) , max g₀,...,g_N∈RMδ0,...,δmaxN∈R max x0,x1,...,xN∈RM max x?: kx0−x?k≤R LRδ_N2 such that xn+1= xn− 1 L n X k=0 hn+1,kR gk, n = 0, . . . , N − 1, 1 2 gi− gj 2 ≤ δi − δj− 1 R hgj, xi − xji, i , j = 0, . . . , N, ∗, where g_n= _LR1 ∇f (x_n) and δn= _LR1 (f (xn) − f (x?)) .

For any FO method:

f (xN) − f (x?) ≤ B1(H, R, L, N, M) ≤ B2(H, R, L, N, M)

Numerical bounds for first-order methods

I Drori & Teboulle (2014) further relax the bound leading to a still simpler optimization problem:

f (xN) − f (x?) ≤ B1(H, . . .) ≤ B2(H, . . .) ≤ B3(H, R, L, N).

I For given step-size coefficients H, and given number of iterations N, they use a semi-definite program (SDP) to compute B3 numerically.

I They find numerically that for the FGM1 choice of H, the convergence bound B3 is slightly below

2L kx0− x?k2₂

(N + 1)2 .

Optimizing step-size coefficients numerically

Drori & Teboulle (2014) also compute numerically the minimizer over H of their relaxed bound for given N using a SDP:

H∗ = arg min

H

B3(H, R, L, N).

Numerical solution for H∗ for N = 5 iterations: [2, Ex. 3]

Drawbacks

• Must choose N in advance

• Requires O(N) memory for all gradient vectors {∇f (xn)}N_n=1

• O(N2) computation for N iterations

Benefit: convergence bound (for specific N) ≈ 2× lower than for Nesterov’s FGM1.

New analytical solution

I Analytical solution for optimized step-size coefficients [7], [8]:

H∗ : hn+1,k =      θn−1 θn+1hn,k, k = 0, . . . , n − 2 θn−1 θn+1 (hn,n−1− 1) , k = n − 1 1 +2θn−1 θn+1 , k = n. θn=        1, n = 0 1 2  1 +q1 + 4θ2_n−1, n = 1, . . . , N − 1 1 2  1 +q1 + 8θ2_n−1, n = N.

I Analytical convergence bound for this optimized H∗:

f (xN) − f (x?) ≤ B3(H∗, R, L, N) =

1L kx0− x?k22

(N + 1)(N + 1 +√2).

I Of course bound is O(1/N2_{), but constant is twice better} I No numerical SDP needed =⇒ feasible for large N.

Outline

Motivation Problem setting Existing algorithms

Gradient descent

Nesterov’s “optimal” first-order method

Optimizing first-order minimization methods

Numerical examples

Logistic regression for machine learning CT image reconstruction

Further acceleration using OS Generalizing OGM

Optimized gradient method (OGM1)

Donghwan Kim & JF (2016) also foundefficient recursiveiteration: Initialize: θ0= 1, z0 = x0 zn+1= xn− 1 L∇f (xn) θn=    1 2  1 +q1 + 4θ_n−12 , n = 1, . . . , N − 1 1 2  1 +q1 + 8θ_n−12 , n = N xn+1= zn+1+ θn− 1 θn+1 (zn+1− zn) + θn θn+1 (zn+1− xn) | {z } new momentum .

Reverts to Nesterov’s FGM1 if thenew terms are removed. • Very simple modification of existing Nesterov code • No need to solve SDP

Recent refinement of OGM1

zn+1= xn− 1 L∇f (xn) (usual GD update) tn+1= 1 2  1 + q 1 + 4t2 n  (momentum factors) xn+1= zn+1+ tn− 1 tn+1 (zn+1− zn) + tn tn+1 (zn+1− xn) | {z } OGM1 momentum New convergence bound for every iteration:

f (zn) − f (x?) ≤

1L kx0− x?k22

(n + 1)2 . Simpler and more practical implementation. Need not pick N in advance.

Optimized gradient method (OGM) is optimal!

xn+1= xn− 1 L n X k=0 hn+1,k∇f (xk),

we optimized OGM and proved the convergence rate upper bound:

f (xN) − f (x?) ≤

L kx0− x?k22

N2 .

Recently Y. Drori [11] considered the class of general FO methods: xn+1 =F(x0, f (x0), ∇f (x0), . . . , f (xn), ∇f (xn)) ,

and showed any algorithm in this case has a function f such that L kx0− x?k22

Worst-case functions for OGM

From [9], [10]:

OGM has two worst-case functions (like GM), a Huber-like function and a quadratic function.

Outline

Motivation Problem setting Existing algorithms

Gradient descent

Nesterov’s “optimal” first-order method Optimizing first-order minimization methods

Numerical examples

Logistic regression for machine learning CT image reconstruction

Further acceleration using OS Generalizing OGM

Machine learning (logistic regression)

To learn weights x of binary classifier given feature vectors {vi}

and labels {yi = ±1}: ˆ x = arg min x f (x), f (x) = X i ψ(yihx, vii) +β 1 2kxk 2 2. logistic: ψ(z) = log(1 + e−z), ψ(z) =˙ −1 ez_{+ 1}, ψ(z) =¨ ez (ez_{+ 1)}2 ∈  0,1 4  . Gradient ∇f (x) =P iyiviψ(y˙ ihx, vii) +βx

Hessian is positive definite so strictly convex:

∇2f (x) =X i viψ(y¨ ihx, vii) v0i+βI  1 4 X i viv0i+βI =⇒ L , 1 4ρ X i viv0i ! + β ≥ max x ρ  ∇2f (x)

Numerical Results: logistic regression

Training data (points); initial decision boundary (red); final decision boundary (magenta).

Numerical Results: convergence rates

Iteration

kx

−

x

k

Numerical Results: adaptive restart

O’Donoghue & Cand`es, 2014

Low-dose 2D X-ray CT image reconstruction simulation

Outline

Motivation Problem setting Existing algorithms

Gradient descent

Nesterov’s “optimal” first-order method Optimizing first-order minimization methods

Numerical examples

Logistic regression for machine learning CT image reconstruction

Further acceleration using OS

Generalizing OGM Summary / future work

Combining ordered subsets (OS) with momentum

I Optimization problems in image reconstruction (and machine learning) involve sums of many similar terms:

f (x) =

M

X

m=1

fm(x) .

I Approximate gradients using just one term at a time: ∇ f (x) ≈ M∇ fm(x)

I Ordered subsets (OS) in tomography

I Incremental gradients in optimization / machine learning

OS + OGM1 method

Initialize: θ0= 1, z0 = x0

For each iteration n

For each subset m = 1, . . . , M

k= nM + m − 1 zk+1 = xk− M L ∇fm(xk) (usual OSupdate) θk = 1 2  1 +q1 + 4θ2 k−1  (momentum factors) xk+1 = zk+1+ θk− 1 θk+1 (zk+1− zk) + θk θk+1 (zk+1− xk) | {z } new momentum .

• Simple modification of existing OS code • ≈ O 1/(Mn)2

Results: 3D X-ray CT patient scan

• 3D cone-beam helical CT scan with pitch 0.5

Initial FBP image x(0) Converged image x(∞) 800 850 900 950 1000 1050 1100 1150 1200

• Convergence rate in RMSD [HU], within ROI, versus iteration:

RMSD_ROI(xn) ,

||x_ROI(n)_√− ˆxROI||2

NROI

.

Results: RMSD [HU] vs. iteration: without OS

• Computation time: OGM <FGM  GD • OGM requires about √1

2-times fewer iterations than FGM

Results: RMSD [HU] vs. iteration: with OS

• Proposed OS-OGMconverges faster than OS-FGM.

Outline

Motivation Problem setting Existing algorithms

Gradient descent

Nesterov’s “optimal” first-order method Optimizing first-order minimization methods Numerical examples

Logistic regression for machine learning CT image reconstruction

Further acceleration using OS

Generalizing OGM

Generalizing OGM - gradient decrease

I Cost function decrease: f (xn) − f (x?) ∼ O(1/n2) I Gradient norm decrease? k∇f (xn)k → 0 at what rate?

Important especially for problems involving duality. Known (recent) result [15]:

GM: min 0≤n≤Nk∇f (xn)k = k∇f (xN)k ≤ √ 2 N LR FGM: k∇f (xN)k ≤ 2 NLR New results by DK & JF [16], [17]:

FGM: min 0≤n≤Nk∇f (xn)k ≤ 2√3 N3/2LR OGM: min 0≤n≤Nk∇f (xn)k ≤ k∇f (xN)k ≤ √ 2 N LR

Generalized OGM (GOGM)

Choose momentum factors: tn+1 > 0 s.t. tn+12 ≤ Tn+1,Pn+1k=0tk

xn+1= zn+1+ (Tn− tn)tn+1 Tn+1tn (zn+1− zn)+ (2t_n2− T_n)tn+1 Tn+1tn (zn+1− xn) .

Optimized choice of momentum factors (for decreasing gradient norm) [16], [17]: tn,      1, n = 0, 1 2  1 +q1 + 4t_n−12 , n = 0, . . . , bN/2c − 1, (N − n + 1)/2, n = bN/2c , . . . N.

OGM-OG convergence rate bounds

I Convergence bound for cost function for OGM-OG:

f (zN) − f (x?) ≤

2L kx0− x?k22

N2 .

I Same as FGM

I Convergence bound for gradient norm is best known:

min

0≤n≤Nk∇f (zn)k ≤ min0≤n≤Nk∇f (xn)k ≤

√ 6 N3/2LR

I √2 better than FGM’s smallest gradient norm bound

I Variations that do not require choosing N in advance, but that have slightly larger constants in bounds.

I Derivation uses relaxations that are not tight.

Summary of (fast?) gradient decreasing FO methods

From [16], [17]:

Summary

I New optimized first-order minimization algorithm (optimal!)

I Simple implementation akin to Nesterov’s FGM

I Analytical converge rate bound

I Bound on cost function decrease is 2× better than Nesterov Future work

• Constraints

• Non-smooth cost functions, e.g., `1

• Tighter bounds • Strongly convex case

• Asymptotic / local convergence rates • Incremental gradients

• Stochastic gradient descent • Adaptive restart

Bibliography I

[1] R. C. Fair,“On the robust estimation of econometric models,” Ann. Econ. Social Measurement, vol. 2, 667–77, Oct. 1974.

[2] Y. Drori and M. Teboulle,“Performance of first-order methods for smooth convex minimization: A novel

approach,” Mathematical Programming, vol. 145, no. 1-2, 451–82, Jun. 2014.

[3] A. B. Taylor, J. M. Hendrickx, and Franc¸ois. Glineur,“Smooth strongly convex interpolation and exact

worst-case performance of first- order methods,” Mathematical Programming, 2016.

[4] J. Barzilai and J. Borwein,“Two-point step size gradient methods,” IMA J. Numerical Analysis, vol. 8, no. 1, 141–8, 1988.

[5] Y. Nesterov,“A method for unconstrained convex minimization problem with the rate of convergence O(1/k2),” Dokl. Akad. Nauk. USSR, vol. 269, no. 3, 543–7, 1983.

[6] ——,“Smooth minimization of non-smooth functions,” Mathematical Programming, vol. 103, no. 1,

127–52, May 2005.

[7] D. Kim and J. A. Fessler,Optimized first-order methods for smooth convex minimization, arxiv 1406.5468, 2014.

[8] ——,“Optimized first-order methods for smooth convex minimization,” Mathematical Programming,

2016, To appear.

[9] ——,On the convergence analysis of the optimized gradient methods, arxiv 1510.08573, 2015.

[10] ——,“On the convergence analysis of the optimized gradient methods,” J. Optim. Theory Appl., 2016,

Submitted.

[11] Y. Drori,The exact information-based complexity of smooth convex minimization, arxiv 1606.01424, 2016.

Bibliography II

[13] B. O’Donoghue and E. Cand`es,“Adaptive restart for accelerated gradient schemes,” Found. Comp. Math., vol. 15, no. 3, 715–32, Jun. 2015.

[14] D. Kim, S. Ramani, and J. A. Fessler,“Combining ordered subsets and momentum for accelerated X-ray

CT image reconstruction,” IEEE Trans. Med. Imag., vol. 34, no. 1, 167–78, Jan. 2015.

[15] I. Necoara and A. Patrascu,“Iteration complexity analysis of dual first order methods for conic convex

programming,” Optimization Methods and Software, vol. 31, no. 3, 645–78, 2016.

[16] D. Kim and J. A. Fessler,Generalizing the optimized gradient method for smooth convex minimization, arxiv 1607.06764, 2016.

[17] ——,“Generalizing the optimized gradient method for smooth convex minimization,” Mathematical