**R E S E A R C H**

**Open Access**

## The modiﬁed accelerated Bregman method

## for regularized basis pursuit problem

### Yajun Xie

1,2_{, Yifen Ke}

1_{and Changfeng Ma}

1*
*_{Correspondence:}

1_{School of Mathematics and}

Computer Science, Fujian Normal University, Fuzhou, 350007, P.R. China

Full list of author information is available at the end of the article

**Abstract**

In this paper, a modiﬁed accelerated Bregman method (MABM) for solving the
regularized basis pursuit problem is considered and analyzed in detail. This idea is
based on the fact that the linearized Bregman method (LBM) proposed by Osher*et al.*

(Multiscale Model. Simul. 4(2):460-489, 2005) is equivalent to a gradient descent method applied to a certain dual formulation which converges to the solution of the regularized basis pursuit problem. The proposed method is based on an

extrapolation technique which is used in accelerated proximal gradient methods presented by Nesterov (Dokl. Akad. Nauk SSSR 269:543-547, 1983). It is veriﬁed that the modiﬁed accelerated Bregman method (MABM) is equivalent to the

corresponding accelerated augmented Lagrangian method (AALM). The theoretical
results conﬁrm that the method has a rapid convergence rate of*O*(1/(*k*+ 1)2_{).}
**Keywords:** compressed sensing; linear constraint; Bregman method; modiﬁed
accelerated method; convergence rate

**1 Introduction**

Compressed sensing, an interesting research ﬁeld involving how to obtain information,
acts as a crucial role in signal processing, image restoration,*etc.*The origin of its name is
based on the idea of encoding a large sparse signal exploiting a relatively small number of
linear measurements, then decoding the signal either through minimizing the -norm or
utilizing a combinational algorithm, a greedy algorithm,*etc.*It resulted from elementary
analysis and approximation theory by Kashin [], but was brought into the forefront by the
work of Candés*et al*. [–] and Donoho [] who constructed speciﬁc methods and showed
their application prospects. The concrete model considered in compressed sensing is the
so-called basis pursuit problem [, ]

min

*x*∈*nx* subject to*Ax*=*b*, (.)

where*A*∈ *m*×*n*_{,}_{b}_{∈ }*m*_{. It can be deemed a classical transform in the compressed sensing}
ﬁeld into the following NP-hard discrete optimization problem:

min

*x*∈*nx* subject to*Ax*=*b*, (.)

where*A*∈ *m*×*n*_{,}_{b}_{∈ }*m*_{,}_{ · }

denotes the number of nonzero elements.

In many practical circumstances, the data or signals are frequently represented by a few matrices, which, in fact, is convenient for data processing and analysis. However, the data

are often given with damage, loss or noise pollution,*etc.*In this case, how to restore the
original data is among the practical diﬃculties faced when solving the matrix
reconstruc-tion problem. Being similar to handling the original signal (vector form) in the compressed
sensing ﬁeld, matrix remodeling refers to an original compressible or sparse
representa-tion matrix which can be exactly or approximately reconstructed with a proper model.
The matrix reconstruction problem can be classiﬁed with matrix completion (MC) [, ]
and matrix recovery (MR) [, ], which is met in various areas,*e.g.*, a famous application
is the Netﬂix system [] in matrix completion; while the matrix recovery problem
orig-inates from face image processing, background modeling and so forth, it can be applied
to image alignment as by Peng*et al.*[]. Generally speaking, the matrix reconstruction
problem is written as

min rank(*X*) subject to*X*∈*C*, (.)

where*X*∈ *m*×*n*_{and}_{C}_{is a convex set. Normally, the following aﬃne constrained }
mini-mization problem can be considered:

min rank(*X*) subject to*A*(*X*) =*b*, (.)

where the linear map*A*:*m*×*n*_{→ }*d*_{,}_{b}_{∈ }*d*_{is a given vector.}

The model emerges in many ﬁelds, such as determining a low-order controller for a plant [] and a minimum order linear system realization [], and solving low-dimensional Euclidean embedding problems []. A natural generalization in the sense of compressed sensing, (.) can be formulated as the nuclear norm minimization problem

min*X*∗ subject to*A*(*X*) =*b*, (.)

where · ∗denotes the sum of singular values of the corresponding matrix. The matrix

completion problem

min rank(*X*) subject to*Xi*,*j*=*Mi*,*j*, (*i*,*j*)∈, (.)
where*X*∈ *m*×*n*_{,}_{M}_{∈ }*m*×*n*_{,}_{}_{is a certain index set, is considered as a special case of}
(.). The idea is inspired by the method presented in [, ]*etc.*Now, we consider the
so-called regularized basis pursuit problem

min

*x*∈*nx*+
*μ*

*x*

subject to*Ax*=*b*. (.)

Adding the penalized term of the constraint condition to the objection function of the regularized basis pursuit problem, we obtain

min

*x*∈*nx*+
*μ*

*x*

+

*λ*

*Ax*–*b*

. (.)

is also the solution of (.). This exact regularization property of (.) was studied in [, ]. Problem (.) has a -norm in the regularizer term, thus it is considered less sensitive to noise than the basis pursuit problem (.).

To solve (.), a linearized Bregman iteration method was presented in [], which was
motivated by []. The idea of the linearized Bregman method is to combine a ﬁxed point
iteration with the Bregman method in [, ], but the linearized Bregman method is a
bit slow in the convergence result. Hence, many accelerated schemes have been brought
up in various theses. For example, in [] the kicking technique was introduced. Yin []
veriﬁed that the linearized Bregman method is equivalent to a gradient descent method
applied to the Lagrangian dual problem of (.). They improved the linearized Bregman
method, utilizing the Barzilai-Borwein line of searching [], nonlinear conjugate
gradi-ent methods, and the method of limited memory BFGS []. Huang*et al.*[] proposed
an accelerated linearized Bregman method which is based on the fact that the linearized
Bregman method is equivalent to the gradient descent method applied to a Lagrangian
dual of problem (.) and the extrapolation technique, which is adopted in the
acceler-ated proximal gradient methods [] proposed by Nesterov*et al.*To solve problem (.),
Goldstein*et al.*[] used an alternating split technique and its Lagrange dual problem.

Based on these studies, we extend the accelerated Bregman method to solve (.) in which the object function might be not diﬀerentiable but have the ‘good’ performance (convex and continuous). We put forward a new improvement formula based on the accel-erated Bregman method. It can be proved to have the property that the modiﬁed Bregman method is equivalent to the corresponding accelerated augmented Lagrangian method, and the latter has a rapid convergence rate which can be deemed to be an improvement of [].

The rest of this article is organized as follows. In Section , we sketch the original Bregman method and the linearized Bregman method which are useful for the subse-quent analysis. In Section , we introduce the accelerated augmented Lagrangian method (AALM), and we present our modiﬁed accelerated Bregman method (MABM). Section is devoted to the convergence of the regularized basis pursuit problem and here we ana-lyze the error bound of the MABM in detail. In Section , we give some conclusions and discuss the research plans for our future work.

**2 The original Bregman method and its linearized form**

The Bregman method was introduced into image processing by Osher*et al.*in [] for
solving the total-variation (TV)-based image restoration problems. Let*ϕ*(·) be a Bregman
function, the Bregman distance [] of the point*u*to the point*v*is deﬁned as

*Dpϕ*(*u*,*v*) :=*ϕ*(*u*) –*ϕ*(*v*) –*p*,*u*–*v*, (.)

where*p*∈*∂ϕ*(*v*), the subdiﬀerential of*ϕ*at*v*. Note that updating the formula for (.), this
Bregman iterative regularization procedure recursively solves

*xk*+:=arg min

*x* *D*
*pk*

*ϕ*

*x*,*xk*+*λ*

*Ax*–*b*

(.)

for*k*= , , . . . , starting with*x*_{= ,}* _{p}*

_{= . Since (.) is a convex programming problem,}

the optimality conditions can be given by

from which we obtain the updated formula

*pk*+:=*pk*+*λATb*–*Axk*+. (.)

Hence, the Bregman iterative scheme is given by

*xk*+_{:=}_{arg min}

*xDp*

*k*

*ϕ* (*x*,*xk*) +_{}*λAx*–*b*,

*pk*+:=*pk*+*λAT*(*b*–*Axk*+). (.)

By the following lemma, we get the equivalent form of (.)

*xk*+:=arg min*xϕ*(*x*) +*λ*_{}*Ax*–*bk*,

*bk*+_{:=}_{b}k_{+ (}_{b}_{–}* _{Ax}k*+

_{),}(.)

starting with*x*_{= ,}* _{b}*

_{=}

_{b}_{.}

For the sake of the requirement for the whole theoretical analysis, in the following, we propose some signiﬁcant equivalence results, and we give the detailed proofs.

**Lemma .** *The Bregman iterative scheme*,*which is given by*(.),*will obtain the same*

*optimum point in the ﬁrst term as*(.)*if*

*pk*=*λATbk*–*b* (.)

*holds*,*whereλis a certain positive constant*.

*Proof* From the ﬁrst formula of (.) and the deﬁnition of Bregman distance (.), we get

*xk*+=arg min

*x* *ϕ*(*x*) –*ϕ*

*xk*–*pk*,*x*–*xk*+*λ*

*Ax*–*b*

=arg min

*x* *ϕ*(*x*) –

*λATbk*–*b*,*x*+*λ*

*Ax*–*b*

=arg min

*x* *ϕ*(*x*) –*λ*

*bk*–*b*,*Ax*–*b*+*λ*

*Ax*–*b*

+

*λ*

*b*
*k*_{–}* _{b}*

=arg min

*x* *ϕ*(*x*) –

*λ*

*Ax*–*b*
*k*

,

therefore, we complete the proof.

From the discussion above, we can give a crucial conclusion.

**Theorem .** *The original Bregman iterative scheme*(.)*is equivalent to its variant*(.).

*Proof* By induction, in fact, we only need to verify that (.) holds.

If*k*= ,*p*_{=}_{λ}_{A}T_{(}* _{b}*

_{–}

_{b}_{) = , (.) holds by the initial conditions}

**

_{p}_{= and}

**

_{b}_{=}

_{b}_{.}

Now, we suppose that (.) holds for*k*– . Then

*pk*=*pk*–+*λATb*–*Axk*

=*λATbk*––*b*+*b*–*Axk*

=*λATbk*–*b*,

where the ﬁrst equality is from the second term of (.), and the third equality is from the second term of (.). Therefore, the original Bregman iterative scheme (.) is equivalent

to its variant (.).

Noting that [, , ] and the references therein, it was argued that the Bregman iter-ative method is equivalent to the augmented Lagrangian method. The signiﬁcance of this statement will be demonstrated in our later analysis in Section .

For the following analysis, we deﬁne the Lagrangian function of (.),

*L*(*x*,*σ*) :=*ϕ*(*x*) +*σ*,*b*–*Ax*,

where *σ*∈ *m* is the Lagrangian multiplier. The corresponding augmented Lagrangian
function can be expressed as

*L*(*x*,*σ*,*λ*) :=*ϕ*(*x*) +*σ*,*b*–*Ax*+*λ*

*Ax*–*b*

, (.)

where*ϕ*(*x*) :=*x*+*μ*_{}*x*and*λ**Ax*–*b*

is the penalty term.

The augmented Lagrangian iterative scheme is

*xk*+_{:=}_{arg min}

*xϕ*(*x*) +*σk*,*b*–*Ax*+*λ*_{}*Ax*–*b*,

*σk*+_{:=}_{σ}k_{+}_{λ}_{(}_{b}_{–}* _{Ax}k*+

_{),}(.)

starting from*σ*_{= .}

**Lemma .** *The ﬁrst item of the iterative sequence*{*xk*}*of*(.)*and that of*(.)*are equal*
*if*

*bk*=*b*+

*λσ* (.)

*holds*.

*Proof* From the ﬁrst formula of (.) and (.), we get

*xk*+=arg min

*x* *ϕ*(*x*) +

*λ*

*Ax*–*b*–*σ*
*k*

*λ*

=arg min

*x* *ϕ*(*x*) +

*λ*

*Ax*–*b*

+

*σk*,*b*–*Ax*.

The constant of the objective function does not aﬀect the optimum point, so by com-paring with the above equations, we get the conclusion.

**Theorem .** *The augmented Lagrangian iterative scheme* (.) *is equivalent to the*

*Proof* It is not diﬃcult to see that the proof is the same as that of Theorem .. Similarly,
by the mathematical induction, we simply show that (.) holds.

If*k*= ,*b*_{=}_{b}_{+}*σ*

*λ*, (.) holds by the initial conditions*σ*

_{= and}* _{b}*

_{=}

_{b}_{. Now, we}

suppose that (.) holds for*n*≤*k*– . Then, when*n*=*k*,

*bk*=*bk*–+*b*–*Axk*=*b*+

*λσ*

*k*–_{+}_{b}_{–}_{Ax}k

=*b*+

*λσ*

*k*–_{+}
*λ*

*σk*–*σk*–=*b*+

*λσ*

*k*_{,}

where the ﬁrst equality is from the second term of (.), the second equality has its roots in induction, and the third equality is derived from the second term of (.). Therefore, the augmented Lagrangian iterative scheme (.) is equivalent to the Bregman iterative method variant (.). Moreover, we can get the equivalence of (.) and (.) from the

two theorems above.

For solving the subproblem of Bregman iterative method (.), diﬀerent algorithms were
conceived in [, , ], however, these methods have to perform signiﬁcant iterations
for solving the subproblem. In order to overcome the diﬃculties, the linearized Bregman
method is obtained by linearizing the last term in (.) into*AT*_{(}_{Ax}k_{–}_{b}_{),}_{x}_{}_{(where}_{λ}_{= )}
and adding the*l*-proximity term _{}* _{γ}x*–

*xk*. The concrete scheme is

*xk*+_{:=}_{arg min}

*xDp*

*k*

*ϕ* (*x*,*xk*) +*AT*(*Axk*–*b*),*x*+_{}* _{γ}x*–

*xk*

_{},

*pk*+_{:=}_{p}k_{+}_{A}T_{(}* _{Ax}k*+

_{–}

_{b}_{) –}

*γ*(*x*–*x*

*k*_{).} (.)

By the optimality conditions, we obtain the following formula:

∈*∂ϕxk*+–*pk*–*ATb*–*Axk*+

*γ*

*xk*+–*xk*.

That is, for the second term of (.)*pk*+_{∈}_{∂ϕ}_{(}* _{x}k*+

_{). In [, , ], the linearized Bregman}

method was analyzed; as*γ* <_{}* _{A}*

_{}(where·denotes the spectral norm of the

correspond-ing matrix) the iterates of the linearized Bregman method converge to the solution of the regularized basis pursuit problem

min

*x*∈*nx*+

*γx*

subject to*Ax*=*b*.

It has appeared in the accelerated Bregman algorithm in recent years, such as in []*etc.*
But we shall argue that the accelerated Bregman method has large room to advance. We
give some valid improvements on these accelerated Bregman methods. It can be veriﬁed
that the theoretical result as regards the proposed method has a rapid convergence rate of
*O*(/(*k*+ )_{).}

**3 The modiﬁed accelerated Bregman method**

In [], the linearized Bregman algorithm could be written as follows.

Step . Input:*J*(·),*H*(·),*γ* > ; initial point:*x*_{,}* _{p}*

_{.}

Step . Initialize:*k*= , let*x*_{= }_{and}* _{p}*

_{= .}

Step . Compute*xk*+_{:=}_{arg min}

*xDp*

*k*

*ϕ* (*x*,*xk*) +∇*H*(*xk*),*x*+_{}* _{γ}x*–

*xk*

_{}.

Step . Set*pk*+:=*pk*–∇*H*(*xk*) –* _{γ}*(

*xk*+–

*xk*). Step . Set

*k*:=

*k*+ , go to step .

For the iterative scheme above,*H*(*x*) =_{}*Ax*–*b*_{}. Next, we give its equivalence form
from the following lemma.

**Lemma .** *The linearized Bregman method*(*LBM*)*in Algorithm*.*is equivalent to the*

*iterative scheme*

*xk*+:=arg min*xϕ*(*x*) +_{}*γx*–*γu*

*k*_{}
,

*uk*+_{:=}_{u}k_{+}_{A}T_{(}_{b}_{–}* _{Ax}k*+

_{),}(.)

*starting from u*_{=}_{A}T_{b}_{,}_{where u}k_{=}_{p}k_{+}_{A}T_{(}_{b}_{–}_{Ax}k_{) +}*xk*

*γ*.

*Proof* From step of Algorithm ., we have

*pk*+=*pk*+*ATb*–*Axk*–

*γ*

*xk*+–*xk*=· · ·=
*k*

*i*=

*ATb*–*Axi*–

*γx*

*k*+_{.} _{(.)}

Thus, we denote

*uk*:=*pk*++

*γx*

*k*+_{=}_{p}k_{+}_{A}T_{b}_{–}_{Ax}k_{+}

*γx*

*k*_{=}
*k*

*i*=

*ATb*–*Axi*;

then we simplify step of Algorithm .,

*xk*+=arg min

*x* *ϕ*(*x*) +

*pk*,*x*–*ATb*–*Axk*,*x*+
*γx*–*x*

*k*

=arg min

*x* *ϕ*(*x*) +
*γ*

*x*–*γ* *pk*+*ATb*–*Axk*+*x*
*k*

*γ*

=arg min

*x* *ϕ*(*x*) +

*γx*–*γu*

*k*

.

On the other hand,

*uk*+=*pk*++*ATb*–*Axk*++

*γx*

*k*+

=*pk*+*ATb*–*Axk*–

*γ*

*xk*+–*xk*+*ATb*–*Axk*++*x*
*k*+

*γ*

=*pk*+*ATb*–*Axk*+

*γx*

*k*_{+}_{A}T_{b}_{–}* _{Ax}k*+

=*uk*+*ATb*–*Axk*+,

where the second equality is from (.). So, step and step in Algorithm . can be

Yin*et al.*presented all kinds of techniques, for example, line search, and L-BFGS and
BB steps, to accelerate the linearized Bregman method. It is interesting that for the latter
in [] the accelerated linearized Bregman algorithm is argued for as follows.

**Algorithm .**(Accelerated linearized Bregman method (ALBM))

Step . Initialize:*x*_{=}_{}* _{x}*

_{=}

**

_{p}_{=}

_{}

**

_{p}_{= ,}

_{γ}_{> ,}

_{λ}_{> ,}

_{k}_{= .}

Step . Compute*xk*+_{:=}_{arg min}

*xDp*

*k*

*ϕ*(*x*,*xk*) +*λAT*(*Axk*–*b*),*x*+_{}* _{γ}x*–

*xk*.

Step . Set*pk*+:=*pk*–*λAT*(*Axk*–*b*) – * _{γ}*(

*xk*+–

*xk*). Step . Set

*xk*+

_{:=}

_{α}*kxk*++ ( –*αk)xk*.
Step . Set*pk*+_{:=}_{α}

*kpk*++ ( –*αk)xk*.
Step . Set*k*:=*k*+ , go to step .

These methods motivated us to consider the following algorithm, and its acceleration idea is based on the extrapolation technique proposed by Nesterov in [, ]; see also the references therein.

**Algorithm .**(Modiﬁed accelerated Bregman method (MABM))

Step . Initialize:*x*_{=}* _{p}*

_{=}

_{}

**

_{p}_{= ,}

_{t}= ,*λ*> ,*k*= .

Step . Compute*xk*+_{:=}_{arg min}

*xDp*

*k*

*ϕ*(*x*,*xk*) +*λ*_{}*Ax*–*b*_{}.

Step . Set*pk*+_{:=}_{p}k_{+}_{λ}_{A}T_{(}_{b}_{–}_{Ax}k_{).}
Step . Set*pk*+_{:= ( –}–*tk*

*tk*+ )*p*

*k*+_{+}–*tk*

*tk*+ *p*

*k*_{.}

Step . Set*tk*+=_{}(

+ *t*

*k*+ ).
Step . Set*k*:=*k*+ , go to step .

The basic idea of the equivalence between MABA and the corresponding AALM can
be traced back to []. Especially, we can see that our updated iterative for*pk*_{is obviously}
better than the pre-iterative*pk*_{[], since we consider a suﬃcient amount of information}
about the former iterative. In this way, most of the better iterative eﬃciency could be
ex-pected, which is just our purpose in improving the method. Then we will be dependent
on a series of transformations in preparation for the convergence proof in Section .

**Lemma .** *The MABM in Algorithm*.*is equal to the following iterative scheme*:

⎧ ⎪ ⎨ ⎪ ⎩

*xk*+_{:=}_{arg min}

*xϕ*(*x*) +*λ*_{}*Ax*–*bk*,
*bk*+_{:=}_{b}k_{+ (}_{b}_{–}* _{Ax}k*+

_{),}

*bk*+_{:= ( –}–*tk*

*tk*+ )*b*

*k*+_{+}–*tk*

*tk*+ *b*

*k*_{,}

(.)

*starting with b*=*b*.

*Proof* Recalling (.) and noting Theorem ., we can prove that (.) holds for all*k*by
induction.

If*k*= ,*p*=*λAT*(*b*–*b*) = , (.) holds by the initial conditions*p*= and*b*=*b*. Now,
we assume that (.) holds for*n*≤*k*– , then

*pn*=*pn*–+*λATb*–*Axn*=*λATbn*––*b*+*λATb*–*Axn*

where the ﬁrst equality is directly derived from step of Algorithm ., the ﬁrst equality is derived from the induction hypothesis, the fourth equality utilizes the second step of (.).

Moreover, when*n*=*k*, we have

*pk*=

– – *tk*
*tk*+

*pk*+ – *tk*
*tk*+

*pk*–

=

– – *tk*
*tk*+

*λATbk*–*b*+ – *tk*
*tk*+

*λATbk*––*b*

=*λAT* – – *tk*
*tk*+

*bk*–

– – *tk*
*tk*+

*b*+ – *tk*
*tk*+

*bk*–– – *tk*
*tk*+

*b*

=*λATbk*–*b*,

where the ﬁrst equality is from step of Algorithm ., the second term is from (.) and
the induction hypothesis, the fourth equality is from the third term of (.), so (.) holds
for all*k*. Namely, the MABM in Algorithm . is equal to (.).

**Lemma .** *Iterative scheme*(.)*is equal to the following AALM iterative scheme*:

⎧ ⎪ ⎨ ⎪ ⎩

*xk*+_{:=}_{arg min}

*xϕ*(*x*) +*σk*,*b*–*Ax*+*λ*_{}*Ax*–*bk*,
*σk*+_{:=}_{σ}k_{+ (}_{b}_{–}* _{Ax}k*+

_{),}

*σk*+:= ( ––*tk*

*tk*+ )*σ*

*k*+_{+}–*tk*

*tk*+ *σ*

*k*_{,}

(.)

*starting fromσ*_{= ,}_{where}_{σ}k_{∈ }*m _{is the Lagrangian multiplier}*

_{.}

*Proof* It is not diﬃcult to see that the idea has likeness to Theorem ., we are just required
to verify that (.) holds. To this end, we proceed by mathematical induction.

If*k*= ,*b*_{=}_{b}_{+}*σ*

*λ*, (.) holds by the initial conditions*σ*

_{= and}* _{b}*

_{=}

_{b}_{. Suppose that}

(.) holds for*n*≤*k*– , then, when*n*=*k*, we have

*bk*=*bk*–+*b*–*Axk*=*b*+

*λσ*

*k*–_{+}_{b}_{–}_{Ax}k

=*b*+

*λσ*

*k*–_{+}
*λ*

*σk*–*σk*–=*b*+

*λσ*

*k*_{,}

where the ﬁrst equality is from the second term of (.), the second equality stems from induction, and the third equality is derived from the second term of (.).

Thus, we get

*bk*=

– – *tk*
*tk*+

*bk*+ – *tk*
*tk*+

*bk*–

=

– – *tk*
*tk*+

*b*+

*λσ*

*n*

+ – *tk*
*tk*+

*b*+

*λσ*

*n*–

=*b*+

*λ*

– *tk*
*tk*+

*σn*+ – *tk*
*tk*+

*σn*–

=*b*+

*λσ*

Therefore, AALM (.) is equivalent to iterative scheme (.). Moreover, we can get the equivalence of the MABM in Algorithm . to AALM from the above two lemmas.

**Theorem .** *The MABM in Algorithm*.*is equivalent to the corresponding AALM*(.).

**4 The convergence analysis**

A practical challenge for the regularized basis pursuit problem is to oﬀer an eﬃcient
method to solve the non-smooth optimization problems. Many algorithms have been
pro-posed in recent years []. In these methods, some schemes of approximation that have
to do with the non-smooth norm term are usually employed. However, a fast global
con-vergence is diﬃcult to guarantee. Due to the non-smooth nature of the -norm, a simple
method to solve these problems is the subgradient approach [], which converges only
as*O*(√

*k*), where*k*is the iteration counter.

In this paper, we present an eﬃcient method with fast global convergence rate to solve
the regularized basis pursuit problem. Particularly, we verify that this result is an extended
gradient algorithm with the convergence rate of *O*(√

*k*), like that for smooth problems.
Following the Nesterov method for accelerating the gradient method [, ], we show
that the MABM can be further accelerated to converge as*O*(/(*k*+ )).

A series of lemmas in the following are to ensure the convergence rate of the MABM.

**Lemma .** *Let*(*xn*_{,}_{σ}n_{)}_{be generated by the augmented Lagrangian iterative scheme}_{(.),}
*and let*(*x*∗,*σ*∗)*be a globally optimal solution of the problem*

min

*x*∈*nϕ*(*x*) *s.t. Ax*=*b*, (.)

*then the inequality*

*ϕxk*–*ϕ*(*x*)≥*σ*,*Axk*–*x* (.)

*for any*(*x*,*σ*) = (*xn*_{,}_{σ}n_{)}_{and}_{(}* _{x}*∗

_{,}

*∗*

_{σ}_{)}

_{holds}_{.}

*Proof* By the optimality conditions, we get

∈*∂ϕxn*_{–}* _{A}T_{σ}n*–

_{–}

_{λ}_{A}T_{b}_{–}

_{Ax}n_{.}From the second term of (.), we obtain

*ATσn*∈*∂ϕxn*. (.)

By the deﬁnition of subdiﬀerential, we have

*ϕxk*≥*ϕxn*+*σn*,*Axk*–*xn*.

So (.) holds for (*x*,*σ*) = (*xn*_{,}_{σ}n_{). Owing to the fact that (}* _{x}*∗

_{,}

*∗*

_{σ}_{) of (.) satisﬁes the KKT}condition, it is easy to obtain

Compared with (.), inequality (.) holds for (*x*,*σ*) = (*x*∗,*σ*∗), which completes the

proof.

**Lemma .** *Let*(*xk*,*σk*)*be generated by the augmented Lagrangian iterative scheme*(.).
*For any*(*x*,*σ*),*the inequality*

*Lxk*,*σk*–*L*(*x*,*σ*)≥

*λσ*

*k*_{–}* _{σ}k*–
–

*λ*

*σk*––*σ*,*σk*––*σk* (.)

*holds*.

*Proof* By the deﬁnition of the Lagrangian function and Lemma ., we get the following
result:

*Lxk*,*σk*–*L*(*x*,*σ*) =*ϕxk*–*ϕ*(*x*) –*σ*,*b*–*Ax*+*σk*,*b*–*Axk*

≥*σ*,*Axk*–*x*–*σ*,*b*–*Ax*+*σk*,*b*–*Axk*
=*σk*–*σ*,*b*–*Axk*

=

*λ*

*σ*–*σk*,*σk*–*σk*–

=

*λσ*

*k*_{–}* _{σ}k*–
–

*λ*

*σk*––*σ*,*σk*––*σk*.

The second inequality is derived from (.), and the third equality is from (.).

From the fact that*AT _{σ}k*

_{∈}

_{∂ϕ}_{(}

_{x}k_{),}

*∗*

_{A}T_{σ}_{∈}

_{∂ϕ}_{(}

*∗*

_{x}_{), and the deﬁnition of subdiﬀerential,}

we have

*ϕx*∗≥*ϕxk*+*σn*,*Ax*∗–*xn*,
then

*ϕx*∗–*σ*∗,*Ax*∗–*b*≥*ϕxk*–*σn*,*Axn*–*b*,
thus

*Lx*∗,*σ*∗≥*Lxk*,*σk*, (.)

where we exploit the fact that*Ax*∗=*b*.

**Lemma .** *Let*(*xk*_{,}_{σ}k_{)}_{be generated by the augmented Lagrangian iterative scheme}_{(.),}
*then*

*σk*–*σ*∗_{}≥*σk*––*σ*∗_{}–*σk*–*σk*–_{}– *λLx*∗,*σ*∗–*Lxk*,*σk*.

*Proof* Considering (.) and replacing (*x*,*σ*) with (*x*∗,*σ*∗), we get

*σk*–*σ*∗_{} =*σk*–*σk*–+*σk*––*σ*∗_{}

≤*σk*–*σk*–
+*σ*

*k*–_{–}* _{σ}*∗

– *σk*–*σk*–_{}– *λLx*∗,*σ*∗–*Lxk*,*σk*

=*σk*––*σ*∗
–*σ*

*k*_{–}* _{σ}k*–
–

*λ*

*Lx*∗,*σ*∗–*Lxk*,*σk*,

which completes the proof.

Moreover,*σk*–*σ*∗_{}≤ *σk*––*σ*∗_{}–*σk*–*σk*–_{}holds due to (.), this implies the
global convergence of (.). By summing the above inequality over*k*= , , . . . ,*n*, we get

*n*

*k*=

*σk*–*σk*–
≤*σ*

_{–}* _{σ}*∗

. (.)

Hence,

lim

*k*→∞

*σk*–*σk*–_{}= . (.)

**Theorem .** *Let* (*xk*,*σk*)*be generated by the augmented Lagrangian iterative scheme*
(.),*then L*(*x*∗,*σ*∗) –*L*(*xk*_{,}_{σ}k_{) =}_{O}_{(}

*k*).

*Proof* Noting (.) of Lemma . and substituting (*xn*–_{,}* _{σ}n*–

_{) for (}

_{x}_{,}

_{σ}_{), we get}

*Lxn*,*σn*–*Lxn*–,*σn*–≥
*λσ*

*n*_{–}* _{σ}n*–

. (.)

By multiplying with*n*– and summing it over*n*= , . . . ,*k*, we obtain

*k*

*n*=

(*n*– )*Lxn*,*σn*– (*n*– )*Lxn*–,*σn*–

=
*k*

*n*=

*nLxn*,*σn*– (*n*– )*Lxn*–,*σn*––*Lxn*,*σn*

=*kLxk*,*σk*–
*n*

*k*=

*Lxn*,*σn*

≥

*k*

*n*=

*n*–

*λ* *σ*

*n*_{–}* _{σ}n*–
≥.

From the above inequality, we have

*k*

*n*=

*Lxn*,*σn*≤*kLxk*,*σk*. (.)

On the other hand, it follows from Lemma . that

*Lx*∗,*σ*∗–*Lxn*,*σn*≤

*λσ*

*n*–_{–}* _{σ}*∗
–

*σ*

*n*_{–}* _{σ}n*–
–

*σ*

*n*_{–}* _{σ}*∗

Summing the inequality over*n*= , . . . ,*k*, we obtain

*k*

*n*=

*Lx*∗,*σ*∗–*Lxn*,*σn*

= –
*k*

*n*=

*Lxn*,*σn*+*nLx*∗,*σ*∗

≤

*λ*

*k*

*n*=

*σn*––*σ*∗_{}–*σn*–*σ*∗_{}–*σn*–*σn*–_{}

=
*λ*

*σ*–*σ*∗
–*σ*

*k*_{–}* _{σ}*∗
–

*k*

*n*=

*σn*–*σn*–

. (.)

Combining (.) with (.), we get

–*kLxk*,*σk*+*kLx*∗,*σ*∗≤–
*k*

*n*=

*Lxn*,*σn*+*kLx*∗,*σ*∗

≤

*λ*

*σ*–*σ*∗_{}–*σk*–*σ*∗_{}–
*k*

*n*=

*σn*–*σn*–_{}

≤

*λσ*

_{–}* _{σ}*∗
.

Consequently, we obtain

*Lx*∗,*σ*∗–*Lxk*,*σk*≤

*kλσ*

∗_{–}* _{σ}*
=

*O*

*k*

.

This completes the proof.

Comparing (.) with (.), we have the following two lemmas by replacing (*xk*_{,}_{σ}k_{) with}
(*xk*_{,}_{}_{σ}k_{).}

**Lemma .** *Let*(*xk*_{,}_{}_{σ}k_{)}_{be generated by the augmented Lagrangian iterative scheme}_{(.).}
*For any*(*x*,*σ*),*we have the inequality*

*Lxk*,*σk*–*L*(*x*,*σ*)≥

*λσ*

*k*_{–}* _{σ}k*–
–

*λ*

*σk*––*σ*,*σk*––*σk*. (.)

**Lemma .** *Let*(*xk*_{,}_{}_{σ}k_{)}_{be generated by the augmented Lagrangian iterative scheme}_{(.),}
*then*

*σk*–*σ*∗_{}≥*σk*––*σ*∗_{}–*σk*–*σk*–_{}– *λLx*∗,*σ*∗–*Lxk*,*σk*. (.)

**Theorem .** *Let*(*xk*_{,}_{}_{σ}k_{)}* _{be generated by the augmented Lagrangian iterative scheme}*
(.),

*then*

*Proof* We work from step of Algorithm .,*i.e.*,*tk*+=_{}(

+ *t*

*k*+ ), where*t*= . By a

simple calculation, we have

*tk*+(*tk*+– ) = *t**k* (.)

and

*tk*≥
*k*+

. (.)

Based on Lemma ., by replacing (*x*,*σ*) with (*xn*–_{,}_{}* _{σ}n*–

_{) and (}

*∗*

_{x}_{,}

*∗*

_{σ}_{), setting}

_{k}_{=}

_{n}_{, we}

get the two inequalities

*wn*––*wn*≥

*λσ*

*n*_{–}* _{σ}n*–
–

*λ*

*σn*––*σn*–,*σn*––*σn*, (.)

–*wn*≥

*λσ*

*n*_{–}* _{σ}n*–
–

*λ*

*σn*––*σ*∗,*σn*––*σn*, (.)

where

*wn*:=*L*

*x*∗,*σ*∗–*Lxn*,*σn*.

By multiplying both sides of (.), (.) with *tn*– and , respectively, and adding the
two sides, we obtain

(*tn*– )*wn*– *tnwn*+

≥*tn*

*λ* *σ*

*n*_{–}* _{σ}n*–
+

*λ*

*σ*∗+ (*tn*– )*σn*–– *tnσn*–,*σn*––*σn*

. (.)

For the sake of convenience, we denote

*Pn*:= *σ*∗+ (*tn*– )*σn*–– *tnσn*+*tnσn*–

and

*Qn*:= *σ*∗+ (*tn*– )*σn*–– *tnσn*–*tnσn*–.

By multiplying both sides of (.) with*tn*, we have

*tn(tn*– )*wn*– *t**nwn*+

≥

*λtn*

*σn*–*σn*–_{}

+

*λ*

*σ*∗+ (*tn*– )*σn*–– *tnσn*–,*tn*

*σn*––*σn*

=

*λ*

*σ*∗+ (*tn*– )*σn*–– *tnσn*,*tn*

*σn*––*σn*

=
*λ**σ*

∗_{+ (}_{t}

*n*– )*σn*–– *tnσn*+*tnσn*–

–
*λ**σ*

∗_{+ (}_{t}

*n*– )*σn*–– *tnσn*–*tnσn*–

=
*λPn*

–

*λQn*

,

where the second equality is derived from the fact that*α*+*β*_{}–*α*–*β*_{}= *α*,*β*.
Then we have

*Pn*–*Qn* =

*σ*∗+ (*tn*– )*σn*–– *tnσn*+*tnσn*–

–*σ*∗+ (*tn*– )*σn*–– *tnσn*–*tnσn*–

= *tnσn*–– *tnσn*

= *tnσn*–– *tn*

*tn*+*σn*– ( – *tn)σn*–

*tn*++ *tn*–

=:*K*

*σn*––*σn*,

where

*K*:=

*tntn*+

*tn*++ *tn*–
,

and the third equality is from the third iterative scheme of (.) and we make a transposi-tion to the third term. Thus,

*tn*–*wn*– *tn**wn*+≥

*λPn*

–

*λPn*– (*Pn*–*Qn)*

≥

*λPn*
–

*λ*

*Pn*+*Pn*–*Qn*

≥–

*λPn*–*Qn*

:= –*K*
*λσ*

*n*–_{–}* _{σ}n*

.

Summing the inequality over*n*= , . . . ,*k*, we get

*t*_{}*w*– *t**kwk*+≥–

*k*

*n*=

*K*

*λσ*

*n*–_{–}* _{σ}n*

. (.)

Then, combining (.), (.) with (.), we have

*wk*+≤

*t*_{}*w*+

*k*

*n*=

*K*

*λσn*––*σn*

*t*_{k}

≤ *t**w*+ *ε*

(*k*+ ) ,

as*k*tends to inﬁnity, where*ε*is an arbitrarily small positive number. Hence,

*Lx*∗,*σ*∗–*Lxk*,*σk*=*O*/(*k*+ ).

**Remark** The right formula of (.) exploits the fact that the choice of the penalty factor

*λ*can be seen as a monotonically increasing sequence (such as*λk) that depends on the*
selection of*tk*in Algorithm .. In this way, we are not only able to guarantee the
conver-gence in the formula of*K*divided by*λk, but also to play a critical role of punishment to*

the constraint condition.

**5 Conclusion**

In this paper, we put forward the modiﬁed accelerated Bregman method (MABM) for
solving the regularized basis pursuit problem. We give some beneﬁcial improvement tasks
on the basis of some recent literature on the accelerated Bregman method, and we perform
the theoretical feasibility analysis in detail. It can be showed that the proposed MABM has
a rapid convergence rate of*O*(/(*k*+ )_{). We will devote our future study to combining}

the advantages of LBM with our MABM as regards theory and numerical results.

**Competing interests**

The authors declare that they have no competing interests.

**Authors’ contributions**

All authors contributed equally to the writing of this paper. All authors read and approved the ﬁnal manuscript.

**Author details**

1_{School of Mathematics and Computer Science, Fujian Normal University, Fuzhou, 350007, P.R. China.}2_{Fujian Jiangxia}

University, Fuzhou, 350108, P.R. China.

**Acknowledgements**

The project is supported by the Scientiﬁc Research Special Fund Project of Fujian University (Grant No. JK2013060), Fujian Natural Science Foundation (Grant No. 2013J01006) and the National Natural Science Foundation of China (Grant No. 11071041).

Received: 9 November 2013 Accepted: 18 March 2014 Published:31 Mar 2014
**References**

1. Kashin, B: The widths of certain ﬁnite dimensional sets and classes of smooth functions. Izv. Akad. Nauk SSSR, Ser. Mat.41, 334-351 (1977)

2. Candés, E, Romberg, J, Tao, T: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory52, 489-509 (2006)

3. Candés, E, Romberg, J, Tao, T: Stable signal recovery from incomplete and inaccurate measurements. Commun. Pure Appl. Math.59, 1207-1223 (2006)

4. Candés, E, Tao, T: Decoding by linear programming. IEEE Trans. Inf. Theory51, 4203-4215 (2005) 5. Donoho, D: Compressed sensing. IEEE Trans. Inf. Theory52, 1289-1306 (2006)

6. Candés, E, Wakin, M: An introduction to compressive sampling. IEEE Signal Process. Mag.21, 21-30 (2008) 7. Cai, J, Osher, S, Shen, Z: Linearized Bregman iterations for compressed sensing. Math. Comput.78, 1515-1536 (2009) 8. Cai, J, Candés, E, Shen, Z: A singular value thresholding algorithm for matrix completion. SIAM J. Optim.20,

1956-1982 (2010)

9. Candés, E, Recht, B: Exact matrix completion via convex optimization. Found. Comput. Math.9, 717-772 (2009) 10. Candés, E, Li, X, Ma, Y, Wright, J: Robust principal component analysis. Preprint (2009)

11. Figueiredo, MAT, Nowak, RD, Wright, SJ: Gradient projection for sparse reconstruction: application to compressed sensing and other inverse problems. IEEE J. Sel. Top. Signal Process.1(4), 586-597 (2007)

12. Beck, A, Teboulle, M: Fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2, 183-202 (2009)

13. Peng, Y, Ganesh, A, Wright, J, Xu, W, Ma, Y: RASL: robust alignment by sparse and low-rank decomposition for linearly correlated images. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2010) 14. Ghaoui, LE, Gahinet, P: Rank minimization under LMI constraints: a framework for output feedback problems. In:

Proceedings of the European Control Conference (1993)

15. Fazel, M, Hindi, H, Boyd, SP: A rank minimization heuristic with application to minimum order system approximation. In: Proceedings of the American Control Conference, vol. 6, pp. 4734-4739 (2001)

16. Linial, N, London, E, Rabinovich, Y: The geometry of graphs and some of its algorithmic applications. Combinatorica 15, 215-245 (1995)

17. Kang, M, Yun, S, Woo, H, Kang, M: Accelerated Bregman method for linearly constrained*l1*–*l2*minimization. J. Sci.
Comput.56, 515-534 (2012)

21. Osher, S, Mao, Y, Dong, B, Yin, W: Fast linearized Bregman iteration for compressive sensing and sparse denoising. Commun. Math. Sci.8, 93-111 (2010)

22. Darbon, J, Osher, S: Fast discrete optimization for sparse approximations and deconvolutions. Preprint (2007) 23. Osher, S, Burger, M, Goldfarb, D, Xu, J, Yin, W: An iterative regularization method for total variation-based image

restoration. Multiscale Model. Simul.4(2), 460-489 (2005) (electronic). MR2162864 (2006c:49051)

24. Yin, W, Osher, S, Goldfarb, D, Darbon, J: Bregman iterative algorithms for1-minimization with applications to

compressed sensing. SIAM J. Imaging Sci.1(1), 143-168 (2008)

25. Barzilai, J, Borwein, J: Two point step size gradient methods. IMA J. Numer. Anal.8, 141-148 (1988)

26. Liu, D, Nocedal, J: On the limited memory method for large scale optimization. Math. Program., Ser. B45, 503-528 (1989)

27. Bennett, J, Lanning, S: The Netﬂix prize. In: Proceedings of KDD Cup and Workshop (2007)

28. Goldstein, T, Osher, S: The split Bregman method for L1-regularized problems. SIAM J. Imaging Sci.2(2), 323-343 (2009)

29. Bregman, L: The relaxation method of ﬁnding the common points of convex sets and its application to the solution of problems in convex programming. USSR Comput. Math. Math. Phys.7, 200-217 (1967)

30. Powell, MJD: A method for nonlinear constraints in minimization problems. In: Fletcher, R (ed.) Optimization, pp. 283-298 (1972)

31. Rockafellar, RT: Augmented Lagrangians and applications of the proximal point algorithm in convex programming. Math. Oper. Res.1, 97-116 (1976)

32. Hale, ET, Yin, W, Zhang, Y: Fixed-point continuation for*l1*minimization: methodology and convergence. SIAM J.
Optim.19, 1107-1130 (2008)

33. Berg, EVD, Friedlander, MP: Probing the Pareto frontier for basis pursuit solutions. SIAM J. Sci. Comput.31, 890-912 (2008)

34. Nesterov, YE: A method for unconstrained convex minimization problem with the rate of convergence*O*(1/*k*2_{). Dokl.}

Akad. Nauk SSSR269, 543-547 (1983)

35. Nesterov, YE: Introductory Lectures on Convex Optimization, vol. 87, pp. 220-236 (2004)

36. Rennie, JDM, Srebro, N: Fast maximum margin matrix factorization for collaborative prediction. In: Proceedings of the International Conference on Machine Learning, pp. 713-719 (2005)

37. Bertsekas, D: Nonlinear Programming. Athena Scientiﬁc, Nashua (1999)

10.1186/1029-242X-2014-130