Dimensionality reduction techniques for large-scale optimization

Dimensionality reduction techniques for large-scale optimization

Coralia Cartis (University of Oxford) Joint with

Jari Fowkes, Estelle Massart, Adilet Otemissov, Zhen Shao (Oxford) Lindon Roberts (ANU Canberra), Jan Fiala (NAG Ltd)

Research supported by the Alan Turing Institute for Data Science, NAG Ltd and NPL

Workshop on Mathematical Foundations of Optimization in Data Science November 24, 2020 (online)

Cantab Capital Institute for the Mathematics of Information

Johnson-Lindenstrauss Lemma and Random Embeddings

A ∈ ℝ^n×d

S ∈ ℝ^m×n SA ∈ ℝ^m×d

Let a(ny) real matrix, , .

Then is an -subspace embedding for if

for all .

A n × d n ≫ d ϵ

∈ (0,1]

S m × n ϵ

A

(1 − ϵ

)∥Ax∥

≤ ∥SAx∥

≤ (1 + ϵ

)∥Ax∥

x ∈ ℝ

Johnson-Lindenstrauss Lemma: [Woodruff,’14]

If is a scaled Gaussian matrix with , then

is an (oblivious) -subspace embedding for with probability at least .

S m = 𝒪 (d|log δ

|ϵ

)

S ϵ

A 1 − δ

But note the high cost of forming

SA ⟹ 𝒪(nd

)

Sparse Random Embeddings

Moving away from Gaussian sketching: uniformly sampling rows of

A ⟶

A ∈ ℝ^n×d SA ∈ ℝ^m×d But it does not work sometimes….

𝒪(1)

𝒪(10⁻⁶)

Chance of missing first row: (n-m)/n

A

Sampling provides an embedding when A has low coherence

If μ(A) ≪ 1 ∥U, _i∥₂ are similar in magnitude; intuitively, the rows of are similarly important in determining the solution.A

Definition [Leverage score, coherence]: Given , the leverage score of row is (row norm).

The coherence of , is the maximum of the leverage scores.

A = UΣV i ∥U

∥

A μ(A),

[Drineas et al’10,’11, Tropp’11]: If is a random sampling matrix with , then is an subspace embedding for with probability at least .

S m = 𝒪 (μ(A)

d log d|log δ

|ϵ

)

S ϵ

− A 1 − δ

Sparse Random Embeddings

Hashing: sparse sketching for dense and sparse matrices

Sampling: one non-zero per row

⎛

⎜⎜

⎝

0 · · · 0 1 0 0 0 0 · · · 1 0 0 1 0 · · · 0 0 · · · · 1 0 0

⎞

⎟⎟

⎠

Hashing: one non-zero per column

0 BB

@

0 1 0 1 0 · · · 0 1 0 0 0 0 · · · 1 0 0 1 0 0 · · · 0 0 0 0 0 1 · · · 0

1 CC A

<latexit sha1_base64="WPyQptR0n9joYXrp3aceajFQpSg=">AAACrnicbVHBThsxEPUutAXT0gDHXiwCVU/RblqpvRQhceGYSiQgZVeR1ztJLLz2yp5tG63yP/wSX9FfqBMWtARGGvn5zRt79CYrlXQYRfdBuLX95u27nV269/7D/sfOweHImcoKGAqjjL3JuAMlNQxRooKb0gIvMgXX2e3Fqn79G6yTRl/hooS04DMtp1Jw9NSkc3dCkwxmUtdlwdHKv0tGacQ+s9hn+0xEbtCtLklCH8l2Pgli5hXRRvvzJ9qKtrKloAno/GkoejLpdKNetA72EsQN6JImBpPOvyQ3oipAo1DcuXEclZjW3KIUCpY0qRyUXNzyGYw91LwAl9ZrR5fs1DM5mxrrUyNbs+2OmhfOLYrMK/2Ec7dZW5Gv1cYVTn+ktdRlhaDFw0fTSjE0bLUelksLAtXCAy6s9LMyMeeWC/RLpImGPzgHY6Gom3NZXzXAOxRv+vESjPq9+Guv/6vfPR80Xu2QT+SYfCEx+U7OySUZkCERwX7wLfgZnIVROArTcPIgDYOm54g8i3D+H4gVvnw=</latexit>

A ∈ ℝ^n×d SA ∈ ℝ^m×d

Action of Hashing S:

Compared to sampling,

Hashing uses every row of A.

•Expect better robustness

SA = ∑ⁿ

i=1

sⁱa_i

: column of S : row of A

sⁱ i^th a_i i^th

Can also consider nonzero per columns: s-hashing.

More robustness is achieved.s

Sparse Random Embeddings

Sketching with hashing matrices: theoretical results. [Shao, C, Fiala’20]

Sampling has better embedding properties when coherence of is low.

Is this true for hashing?

A

When is sufficiently small, hashing provides an -subspace embedding with an optimal dimensionality reduction bound, , better than the bound for sampling.

μ(A) ϵ

𝒪(d) 𝒪(d log d)

Result Coherence of A Size of sketching S

[Meng & Mahoney’13] —

[Bourgain et al’15]

[C, Fiala & Shao,’20]

Θ(d

|log δ

|ϵ

)

Throughout, can be replaced by rank of .d A

𝒪(log

d) 𝒪(d log

d|log δ

|ϵ

)

𝒪(d

) 𝒪(d|log δ

|ϵ

)

Using sketching for optimization?

Sketching in the observational domain (subsampling, batch)

reduces number of observations/measurements/data points

•

•

•

Sketching in the variable domain (block-coordinate, subspace methods) reduces the number of parameters/variables

•

•

How can we use sketching for improving efficiency and scalability of optimization algorithms ?

Today

Nonlinear least squares:

derivative-based methods

x∈ℝ

min

f(x) = 1

2 ∥r(x)∥

= ∑

i=1

(r

(x))

Gauss-Newton method for Non-linear Least Squares (NNLS)

where

r : ℝ

→ ℝ

J n × d

r

Gauss-Newton method(s): state-of-the-art for NNLS

At iterate , calculate direction by approximately minimizing a regularized/constrained/

unconstrained variant of the convex quadratic local model, over ,

x

s

∈ ℝ

s ∈ ℝ

q

(s) = 1

2 ∥J(x

)s + r(x

)∥

= f(x

) + ⟨J(x

)

r(x

), s⟩ + 1

2 ⟨s, J(x

)

J(x

)s⟩ .

Regularization, trust-region and linesearch variants have been successfully developed.

We will look at:

Sketching in the variable domain (subspace methods)

Randomised Subspace Gauss-Newton (R-SGN) methods: variable sketching

randomly draw sketching matrix ; calculate the subspace-Jacobian and the reduced local quadratic model, , ,

solve the reduced subproblem (inexactly) to find

compute the ratio .

set and if ; else, and

p × d S

J(x

)S

̂s ∈ ℝ

p ≪ d

̂s

∈ ℝ

,

ρ

= f(x

) − f(x

+ S

̂s ) f(x

) − q

( ̂s

)

x

= x

+ S

̂s σ

< σ

(Δ

> Δ

) ρ

≥ η

x

= x

σ

> σ

(Δ

< Δ

) .

min

q

( ̂s)(+ σ

2 ∥S

̂s∥

)  or  (such  that ∥S

̂s∥ ≤ Δ

) .

R-SGN with quadratic regularisation /trust region: at iteration , k

q

(s) = 1

2 ( J(x

)S

) ̂s+ r(x

)

= f(x

) + ⟨S

J(x

)

r(x

), ̂s⟩ + 1

2 ⟨ ̂s, S

J(x

)

J(x

)S

̂s⟩ .

Global rates of convergence for R-SGN methods

Assumptions:

are Lipschitz continuous; [smoothness]

Let . At each iterate , with probability at least ,

, and . [sketching accuracy]

Typical inexact model minimization conditions for quadratic regularisation/trust-region

r, J ϵ

, δ

∈ (0,1) x

1 − δ

∥S

∇f(x

)∥

≥ (1 − ϵ

)∥∇f(x

)∥

∥S

∥

≤ S

Theorem[R-SGN]: Let , and such that , where is a user- chosen parameter . Then the R-SGN algorithm takes at most

iterations and evaluations of the residual and sketched Jacobian such that

, with probability at least .

[ The constant connects the updates of the regularisation/trust-region parameter]

ϵ > 0 δ ∈ (0,1) (1 − δ

)δ > c c

*

N ≤ [(1 − δ

)δ

− c]

𝒪 (f(x

)(1 − ϵ

)

ϵ

)

min

∥∇f(x

)∥

≤ ϵ 1 − e

* c

This bound matches deterministic complexity bounds for first-order and Gauss-Newton methods despite having only partial Jacobian information available at each iteration. [C,Fowkes,Shao’20]

Global rates of convergence for R-SGN methods

Proof Idea: uses techniques from probabilistic models complexity analyses [Gratton et al’18;

C, Scheinberg’18]

True/false iterations, successful/unsuccessful iterations

There can be at most true and successful iterations (from sufficient decrease condition) and .

Sketching accuracy assumption gives: for any ,

where and are the total number of true and total iterations, respectively.

C[f(x

) − f

]ϵ

f

= 0

ℙ(T < δN) ≤ ϵ

δ ∈ (0,1)

T N

Global rates of convergence for R-SGN methods

Satisfying the sketching accuracy assumption

matrix with iid scaled Gaussian entries with -hashing matrix with

S

p × d p = 𝒪(|log δ

|ϵ

)

S

p × d s p = 𝒪(|log δ

|ϵ

)

Sufficient for each to be a (one-sided) -subspace embedding for one-dimensional vectors, so that the gradient can be embedded correctly.

S

ϵ

sampling matrix : need non-uniformity dependent subspace embeddings for vectors

with . Then . This implies that sampling embeds

correctly the gradient whenever , the gradient components are similar in magnitude.

S

p × d

∥y∥

⋅ ∥y∥

≤ ν

p = 𝒪(dν

|log δ

|ϵ

) S

∥∇f(x)∥

⋅ ∥∇f(x)∥

≤ ν

Comparison with probabilistic models

Our sketching assumption is weaker than probabilistic model conditions [Bandeira, Scheinberg, Vicente’13]:

one-sided length preservation of gradient; not required to embed subspace; numerical example

[C,Fowkes,Shao’20]

Block-Coordinate Gauss-Newton (BC-GN) methods

BC-GN= R-SGN with sampling matrix

S

Theorem[R-SGN] global rate of convergence of BC-GN (with quadratic

regularisation or trust region) with high probability, provided the gradient has similar components in magnitude

⟹

When is a sampling matrix,

S

J(x

)S

∂r

∂x

J(x

)

Under more general assumptions, we can obtain a weaker global rate analysis for BC-GN with fixed and arbitrary block size.

Assume that each coordinate block of size p is drawn with probability P_k (with replacement or from a partition). Then , where each coordinate appears R times in the

set of all possible block.

B

𝔼_Bk∥∇_Bk f(x_k)∥² ≥ P_minR∥∇f(x_k)∥²

[∇^Bk f(x_k) = J_Bk(x_k)^Tr(x_k)]

Theorem[BC-GN]: Assume Lipschitz continuous. Then the number of BC-GN iterations/evaluations until is at most . In particular, when blocks are drawn uniformly

at random, such as from a partition, then .

𝔼(∥∇f(x

)∥

) ≤ ϵ

r, J 𝒪 (f(x

)(P

R)

ϵ

)

P

R = pd

R-SGN/BC-GN methods: numerical experiments

[C,Fowkes,Shao’20; WPaper 20]

BC-GN with TR on logistic regression for chemotherapy

dataset (Python code)

R-SGN/BC-GN methods: numerical experiments

[C,Fowkes,Shao’20]

BC-GN with TR on logistic regression on gisette dataset

Nonlinear least squares:

derivative-free methods

Subspace derivative-free Gauss-Newton methods for NNLS

Sketching DFO-GN/DFO-LS in (number of variables/size of interpolation set)

d

Less evaluations and lower linear algebra cost per iteration. Global efficiency?

[Roberts, PhD Thesis’19; C, Roberts’20]

Use interpolation set , then solve

Underdetermined system take minimal norm solution.

Computational Cost= factorization + solve =

Evaluation cost: only need evaluations of on first iteration and a small number/multiple of subsequently

Choose based on computational resources/evaluation cost

{x

, y

, …, y

}

p < d

(y₁ − x_k)^T

(y_p − x⋮ _k)^T _k^T̂J = (r(y₁) − r(x_k))^T (r(y_p) − r(x⋮ _k))^T

⟹

𝒪(dp

) + (np

) ≈ 𝒪(np

)

p r p

p

Subspace derivative-free Gauss-Newton methods for NNLS

DFBGN (Derivative-Free Block Gauss-Newton) Algorithm

Build low-dimensional model and calculate trust-region step,

Evaluate , accept/reject step, and update (usual DFO choices)

(where is basis of interpolation set )

Add to interpolation set and remove points from the interpolation set

Add random orthogonal directions for until we have interpolation points

̂s

∈ ℝ

, min

1

2 ∥r(x

) + ̂J

̂s ∥

∥ ̂s∥ ≤ Δ

f(x

+ Q

̂s ) Δ

Q

𝒴

= {y

− x

, …, y

− x

}

x

+ Q

̂s p

≥ 2

x

+ Δ

d d ⊥ 𝒴

p + 1

Comments:

Linear algebra cost vs full space method

Choosing points to remove uses Lagrange polynomials (geometry-aware)

Choice of on successful iterations, on unsuccessful iterations

𝒪(np

+ dp

+ p

) 𝒪(nd

+ d

)

p

: p

= 2 p/10

Subspace derivative-free Gauss-Newton methods for NNLS

Numerical results for DFBGN algorithm

Choose test set CUTEst with , max 12hrs per problem

Relative accuracy=0.1 vs budget; Solver and timeout

DFBGN outperforms DFO-LS for low accuracy solutions …because it does not time out!

d ≈ 1000

1 2 4 8 16 32

Budget / min budget of any solver 0.0

0.2 0.4 0.6 0.8 1.0

Proportionproblemssolved

DFO-LS

DFO-LS (init n/100) DFBGN (p = n)

DFBGN (p = n/2) DFBGN (p = n/10) DFBGN (p = n/100)

DFOLS 93%

DFBGN (d/100) 35%

DFBGN (d/10) 74%

DFBGN (d/2) 82%

in figures

[n → d]

Subspace derivative-free Gauss-Newton methods for NNLS

Numerical results for DFBGN algorithm

Other advantage: DFBGN make progress after evaluations (especially important when large)

p ≪ d d

normalized objective reduction vs.~\# evaluations, 12hr timeout);

0.0 0.2 0.4 0.6 0.8 1.0

Budget (in gradients) 10⁰

2 × 10⁻¹ 3 × 10⁻¹ 4 × 10⁻¹ 6 × 10⁻¹

NormalizedObjectiveValue

DFO-LS

DFO-LS (init n/100) p = n

p = n/2 p = n/10 p = n/100

0.0 0.2 0.4 0.6 0.8 1.0

Budget (in gradients) 10⁰

5 × 10⁻¹ 6 × 10⁻¹ 7 × 10⁻¹ 8 × 10⁻¹ 9 × 10⁻¹

NormalizedObjectiveValue

DFO-LS

DFO-LS (init n/100) p = n

p = n/2 p = n/10 p = n/100

ARWHDNE, d=2000 CHANDHEQ, d=2000

in figures

[n → d]

Random embeddings for global optimization

Global optimization of functions with low effective dimensionality

Global optimization is generally NP-hard. Can global optimization algorithms be made

efficient for `simpler' problems? What is problem/data ‘simplicity'? Can algorithms adapt to data (without knowing it a priori)?

min

f(x) subject to x ∈ 𝒳 = [−1,1]

Problem simplicity: Functions which do not vary along certain linear subspaces.

Alternative names: low effective dimensionality, (multi-)ridge, planar waves, active subspaces

Applications: hyper-parameter optimization;

complex engineering simulations; parametric, stochastic PDEs; over-parametrized DNNs?

Global optimization of functions with low effective dimensionality

Challenging set-up: The objective function is black box. The orientation of the important subspace is not known.

Solution: Random embeddings [Ziyu Wang et al. Bayesian optimization in a billion dimensions via random embeddings. \textit{J. Artif. Int. Res.}, 55(1), 2016.]

Random embeddings lower dimensional problems

Replace by , where is an Gaussian matrix, and .

f(x) f(S

y) ⟶ S p × d p ≪ d

Functions with low effective dimensionality [Wang et al.’13]: has

effective dimensionality if there exists a linear subspace of dimension such that for all vectors in and in . [ is the smallest integer satisfying these properties]. Dimensions of interest: .

f : ℝ

→ ℝ

d

≤ d 𝒯 d

f(x

+ x

) = f(x

) x

𝒯 x

𝒯

d

d

≤ p ≤ d

Global optimization of functions with low effective dimensionality

(R) (AR)

y∈ℝ

min

f(S

y + u)

s.t.

y ∈ Y = [−a, a]

y∈ℝ

min

f(S

y + u)

s.t.

S

y + u ∈ 𝒳

Reduced optimization problems [C, Otemissov’20; C, Massart, Otemissov’20]

REGO algorithm: (single random embedding)

u=0; solves (R) once (using any global solver) , unconstrained solution

f(S

y

) ≈ f

S

y

AREGO algorithm: (multiple random embedding)

solves (AR) multiple times (with any global solver) updates to best point found so far

,

f(S

y

) ≈ f u

S

y

∈ 𝒳

Assumption: p ≥ d_e.

Theoretical analysis

REGO: best-known probability of success of (R) and suitable choices of , depends only on

, not on ambient dimension a

p, d_e d

Numerical experiments: confirm the theoretical findings; include replacing global solvers with local ones

AREGO: probability of success of (AR)

and convergence of AREGO, depends on algebraically, not exponentially

d

Global optimization of functions with low effective dimensionality

(R) reduced subproblem and REGO algorithm min

y∈ℝ^p f(S^Ty)

s.t. y ∈ Y = [−a, a]^p y*₂ = arg min

y: f(S^Ty)=f_* ∥y∥₂.

We can show that

∥x*

∥

∥y*

∥

∼ χ

⟹ ℙ((R) is successful) ≥ 1 − C(q)(1 + q

2 e

) ( c

2 )

q2

where

q = p − d

+ 1 c = ∥x*

∥/a

BARON on GO problems with low effective dimensionality

in figures

[D → d ^and d → p]

(R)

Global optimization of functions with low effective dimensionality

(AR) reduced subproblem and AREGO algorithm min

y∈ℝ^p f(S^Ty + u)

s.t. S^Ty + u ∈ 𝒳

ℙ((AR)

) ≥ ℙ(−1 ≤ S

y*

+ u ≤ 1) > τ(d) > 0

⟶

Convergence of AREGO, with prob one, proved to a neighbourhood of global minimum of original problem; multiple embeddings used.

S

BARON Local KNITRO

Same tests as for REGO, functions with low effective dimensionality

(AR)

in figures

[D → d ^and d → p]