5.3 The FAST-QPA Active Set Algorithm
5.3.3 Convergence Analysis
The convergence analysis of FAST-QPA is based on two key results, namely Propo- sition 5.2 and Proposition 5.3 stated below. These results show that the algorithm obtains a significant reduction of the objective function both when fixing to zero the variables in the active set estimate and when we perform the projected Armijo line search.
Proposition 5.2 completes the properties of the active set identification strategy defined in Section 5.3.1. More specifically, it shows that, for a suitably chosen value of the parameter ε appearing in Definition 5.2, a decrease of the objective function is achieved by simply fixing to zero one or more variables whose indices belong to the estimated active set.
Proposition 5.2. Assume that the parameter ε appearing in Definition 5.2 sat- isfies
0 < ε < 1 2λmax( ˜Q)
. (5.10)
Given the point xk and the set Ak, let Ay be a set of indices and let y be a point
such that
Ay ⊆ {i ∈ Ak: xk i 6= 0},
5.3. THE FAST-QPA ACTIVE SET ALGORITHM 81 with I = {1, . . . , m}. Then,
q(y) − q(xk) ≤ − 1
2εky − x
kk2.
Proof. By taking into account the definition of the set Ay and the point y, we have q(y) = q(xk) + ∇qAy(xk)>(y − xk)Ay+ 1 2(y − x k)> Ay(∇2q)AyAy(y − xk)Ay .
Since ∇2q = 2Q, the inequality
q(y) ≤ q(xk) + ∇qAy(xk)>(y − xk)Ay+ λmax( ˜Q) k(y − xk)Ayk2
holds. Using (5.10) we obtain
q(y) ≤ q(xk) + ∇qAy(xk)>(y − xk)Ay+ 1 2εk(y − x k )Ayk2 and hence q(y) ≤ q(xk) +∇qAy(xk) + 1 ε(y − x k) Ay > (y − xk)Ay− 1 2εk(y − x k) Ayk2.
It thus remains to show ∇qAy(xk) + 1 ε(y − x k) Ay > (y − xk)Ay ≤ 0 ,
which follows from the fact that, for all i ∈ Ay, ∇qi(xk) + 1 ε(yi− x k i) > (yi− xki) ≤ 0.
Indeed, for all i ∈ Ay we have xki ≥ 0 and yi = 0, hence xki − yi = xki ≤ ε ∇qi(xk),
so that ∇qi(xk) + 1 ε (yi− x k i) ≥ 0 .
The following result shows that the projected Armijo line search performed in Algorithm 14 terminates in a finite number of steps, and that the new point obtained guarantees a decrease of the objective function of QP.
Proposition 5.3. Let γ ∈ (0,12). Then, for every ¯x ∈ Rn
+ with ∇q(¯x)N (¯˜ x) 6= 0,
there exist ρ > 0 and ¯α > 0 such that
q([x + αd]]) − q(x) ≤ γ α d>N (¯˜ x)∇q(x)N (¯˜ x) (5.11)
for all x ∈ Rn+ with x ∈ Bρ(¯x) and for all α ∈ (0, ¯α], where d ∈ Rnis the direction
82 CHAPTER 5. ACTIVE SET BASED BRANCH-AND-BOUND Its proof is similar to the proof of Proposition 2 in [24].
Proof. From the continuity of the objective function of problem QP, we have that the gradient ∇q is Lipschitz continuous on Rn+, so that there exists L < ∞ such
that, for s ∈ [0, 1]
k∇q(x) − ∇q(x − s[x − x(α)])k ≤ sLkx − x(α)k ∀ x ∈ Bρ(¯x) ∩ Rn+,
where x(α) := (x + αd)]. Furthermore, ∇q is bounded on Bρ(¯x) ∩ Rn+, hence,
there exists σ > 0 such that
k∇q(x)k ≤ σ σ2
, ∀ x ∈ Bρ(¯x) ∩ Rn+
and by (5.9) d is also bounded on Bρ(¯x) ∩ Rn+. For x(α) ∈ Bρ(¯x) ∩ Rn+ we have
q(x(α)) − q(x) = ∇q(x)>(x(α) − x) + Z 1 0 ∇q(x − s[x − x(α)]) − ∇q(x)> x(α) − xds ≤ ∇q(x)>(x(α) − x) + kx(α) − xk Z 1 0 sLkx(α) − xkds = ∇q(x)>(x(α) − x) + L 2kx(α) − xk 2.
Then, from the definition of dA(¯˜x) in Algorithm 14, we have
x(α)A(¯˜x) = xA(¯˜x).
Therefore, it is possible to write
q(x(α)) − q(x) ≤ ∇q(x)>N (¯˜ x)(x(α) − x)N (¯˜ x)+
L
2k(x(α) − x)N (¯˜ x)k
2. (5.12)
We now majorize the terms in the right hand side. of (5.12). We first analyze the term k(x(α) − x)N (¯˜ x)k2.
Let x ∈ Bρ(¯x) ∩ Rn+. We distinguish between di ≥ 0 and di < 0.
• If di ≥ 0, we have
xi(α) = xi+ αdi
for all α ≥ 0. Thus,
5.3. THE FAST-QPA ACTIVE SET ALGORITHM 83 • If di < 0, we have xi(α) ≥ xi+ αdi for all α ≥ 0. Hence, 0 ≤ xi− xi(α) ≤ −αdi, so that 0 ≤ (xi− xi(α))2 ≤ α2d2i. (5.14)
Then by (5.13) and (5.14) we have k(x(α) − x)N (¯˜ x)k 2 ≤ α2kd ˜ N (¯x)k 2 . Using (5.8) and (5.9) k(x(α) − x)N (¯˜ x)k2 ≤ α2kdN (¯˜ x)k2 ≤ α2σ22k∇q(x)N (¯˜ x)k2 ≤ −α2 σ2 2 σ1 d>N (¯˜ x)∇q(x)N (¯˜ x). (5.15) Now we consider the first term in the right hand side of (5.12). By condition (5.9) on the search direction we have that for all x ∈ Bρ(¯x) ∩ Rn+ and for all i ∈ ˜N (¯x):
|di| ≤ kdN (¯˜ x)k ≤ σ2k∇q(x)N (¯˜ x)k ≤ σ.
Then we introduce
ω = min
i:∇qi(x)>0
ε∇qi(x).
Consider first the case that i ∈ ˜N (¯x) and ∇qi(x) = 0. For such indices we have
of course that
∇qi(x)(x(α)i− xi) = α ∇qi(x)di.
When i ∈ ˜N (¯x) and ∇qi(x) 6= 0 according to our estimation of the non-active
set, we have two different subcases:
• If ∇qi(x) > 0, then by choosing α ∈ (0, ω/σ] we have that:
xi(α) − xi = αdi. (5.16)
• If ∇qi(x) < 0, we have for all α > 0
xi(α) − xi ≥ αdi,
so that multiplying by ∇qi(x) < 0 we have
84 CHAPTER 5. ACTIVE SET BASED BRANCH-AND-BOUND Therefore, choosing α ∈ (0, ω/σ] we have, from (5.16) and (5.17), that
∇q(x)> ˜ N (¯x)(x(α) − x)N (¯˜ x) ≤ α ∇q(x) > ˜ N (¯x)dN (¯˜ x). (5.18)
Substituting (5.15) and (5.18) into (5.12) we obtain that for all α ∈ (0, ω/σ] and x(α) ∈ Bρ(¯x) ∩ Rn+
q(x(α)) − q(x) ≤ α (1 − ασ22L/2σ1) ∇q(x)>N (¯˜ x)dN (¯˜ x). (5.19)
Hence from (5.19) we see that (5.11) is satisfied by choosing α in such a way that the following inequality holds:
1 − αL σ
2 2
2σ1
> γ.
We can conclude that the statement holds for all x ∈ Bρ(¯x) ∩ Rn+, α ∈ (0, ¯α],
where ¯ α = min ( ω σ, 2σ1(1 − γ) σ2 2L ) . (5.20)
Proposition 5.4. Suppose that (5.10) holds and that FAST-QPA produces an infinite sequence {xk}. Then the sequence {q(xk)} converges.
Proof. Let ˜xk be the point produced in Algorithm 14. By setting y = ˜xk in
Proposition 5.2, we have
q(˜xk) ≤ q(xk) − 1 2εk˜x
k− xkk2 .
Furthermore, by the fact that we use an Armijo line search in Algorithm 14 and by Proposition 5.3, we have that the chosen point xk+1 satisfies inequality (5.11),
that is
q(xk+1) − q(˜xk) ≤ γ α d>Nk∇q(˜xk)Nk.
By taking into account (5.8), we thus have q(xk+1) − q(˜xk) ≤ γ α d>Nk∇q(˜x k )Nk ≤ −σ1k∇q(˜xk)Nkk2 . In summary, we obtain q(xk+1) ≤ q(˜xk) − σ1k∇q(˜xk)Nkk2 ≤ ≤ q(xk) − 1 2εk˜x k− xkk2− σ 1k∇q(˜xk)Nkk2 . (5.21)
In particular, the sequence {q(xk)} is monotonously decreasing and bounded
from below by the minimum of QP, which by our assumption is finite. Hence it converges.
5.3. THE FAST-QPA ACTIVE SET ALGORITHM 85 Finally, we are able to prove the main result concerning the global convergence of FAST-QPA.
Theorem 5.5. Assume that the parameter ε appearing in Definition 5.2 satis- fies (5.10). Let {xk} be the sequence produced by Algorithm FAST-QPA. Then either
an integer ¯k ≥ 0 exists such that x¯kis an optimal solution for Problem QP, or the
sequence {xk} is infinite and every limit point x? of the sequence is an optimal
solution for Problem QP.
Proof. Let x? be any limit point of the sequence {xk} and let {xk}
K be the
subsequence with
lim
k→∞, k∈Kx
k= x?.
By considering an appropriate subsequence, we may assume that there exist sub- sets ¯A ⊆ {1, . . . , m} and ¯N = {1, . . . , m} \ ¯A such that Ak= ¯A and Nk = ¯N for
all k ∈ K, since the number of possible choices of Ak and Nk is finite. In order
to prove that x? is optimal for Problem QP, it then suffices to show
(i) min {∇qi(x?), x?i} = 0 for all i ∈ ¯A, and
(ii) ∇qi(x?) = 0 for all i ∈ ¯N .
In order to show (i), let ˆı ∈ ¯A and define a function Φˆı : Rm → R by
Φˆı(x) = min {∇qˆı(x), xˆı} ,
we thus have to show Φˆı(x?) = 0. For k ∈ K, define ˜yk ∈ Rm as follows:
˜
yki = 0 if i = ˆı xk
i otherwise.
(5.22) Recalling that ˜xkA¯ = 0, as set in Algorithm 14, and using ˆı ∈ ¯A, we have
k˜yk− xkk2 = (˜yk− xk)ˆ2ı = (˜xk− xk)2ˆı ≤ k˜xk− xkk2 . From (5.21) and Proposition 5.4 we obtain
lim k→∞, k∈Kk˜x k− xkk2 = 0 , hence lim k→∞, k∈Ky˜ k = x? . (5.23) By Definition 5.2, we have 0 ≤ xk
ˆı ≤ ε ∇qˆı(xk) for all k ∈ K. Using Assump-
tion (5.10), there exists ξ ≥ 0 such that
ε ≤ 1
2 ˜Qˆıˆı+ ξ
86 CHAPTER 5. ACTIVE SET BASED BRANCH-AND-BOUND As ˜yk ˆı = 0, we obtain xˆkı − ˜yˆık= xˆkı ≤ ε ∇qˆı(xk) ≤ 1 2 ˜Qˆıˆı+ ξ ∇qˆı(xk) and hence (2 ˜Qˆıˆı+ ξ)(xˆkı − ˜y k ˆı) ≤ ∇qˆı(xk),
which can be rewritten as
∇qˆı(xk) + 2 ˜Qˆıˆı(˜yˆık− xˆkı) ≥ ξ(xˆkı − ˜yˆık) ≥ 0,
yielding ∇qˆı(˜yk) ≥ 0. Together with ˜yˆık = 0, we obtain Φˆı(˜yk) = 0. By (5.23) and
the continuity of Φˆı, we derive Φˆı(x?) = 0, which proves (i).
To show (ii), assume on contrary that ∇q(x?) ¯
N 6= 0. By Proposition 5.3, there
exists ¯α > 0 such that for k ∈ K sufficiently large q([˜xk+ αdk]]) − q(˜xk) ≤ γα X i∈Nk ∇qi(˜xk)dki for all α ∈ (0, ¯α], as ˜xk ¯ N converges to x ? ¯
N. As we use an Armijo type rule in
Algorithm 14, we thus have δj−1 ≥ ¯α and hence
αk= δj ≥ δ ¯α . (5.24)
Again by the Armijo rule, using (5.8) and (5.24), we obtain q(˜xk) − q([˜xk+ αkdk]]) ≥ −γαk X
i∈Nk
∇qi(˜xk)dki
≥ γσ1αkk∇q(˜xk)Nkk2
≥ γσ1δ ¯αk∇q(˜xk)Nkk2 .
As the left hand side expression converges to zero by Proposition 5.4, while the right hand side expression converges to
γσ1δ ¯αk∇q(x?)N¯k2 > 0 ,
we have the desired contradiction.
Corollary 3. If (5.10) holds, then the sequence q(xk) converges to the minimum
of QP.
As a final result, we prove that, if strict complementarity holds, FAST-QPA finds an optimal solution in a finite number of iterations.
Theorem 5.6. Assume that there exists an accumulation point x? of the sequence {xk} generated by FAST-QPA such that
5.3. THE FAST-QPA ACTIVE SET ALGORITHM 87 (i) strict complementarity holds in x? and
(ii) the problem
min y>Q˜N Ny + c>Ny
s.t. y ∈ R|N |, (5.25)
with N = N (x?), admits a finite solution.
Then FAST-QPA produces a minimizer of Problem QP in a finite number of steps. Proof. Let {xk} be the sequence generated by FAST-QPA and let {xk}
K be a con-
verging subsequence. By Theorem 5.5, the limit point x? of {xk}
K is a minimizer
of Problem QP. Now define
ξ = min
i∈N (x?){x
?
i} > 0 .
From Corollary 2, we have that for k ∈ K sufficiently large
Ak = A(x?) (5.26) and min i∈Nk{x k i} = min i∈Nk{˜x k i} ≥ ξ 2 . The search direction dk
Nk computed in Algorithm 14 is gradient related and, since
lim k→∞, k∈Kk∇qN k(˜xk)k = 0, (5.27) we have lim k→∞, k∈Kkd k Nkk = 0.
Therefore, for k sufficiently large,
(xk+ dk)Nk = (˜xk+ dk)Nk and (˜xk+ dk)i ≥ ξ 2 > 0, ∀i ∈ N k.
This means that
[(˜xk+ dk)Nk]]= (˜xk+ dk)Nk ,
and ˜xk+dkis feasible for Problem QP. For k sufficiently large, k ∈ K, Corollary 2,
(5.26) and (5.27) ensure that
A(x?) = Ak= Ak+1 (5.28)
and
88 CHAPTER 5. ACTIVE SET BASED BRANCH-AND-BOUND Since, taking into account (5.29) and point ii) in our assumptions, (˜xk+ dk)
Nk
is the optimal solution for problem QPk, we have
∇qNk(˜xk+ dk) = 0, (5.30)
q(˜xk+ dk) = q(˜xk) + 1 2∇q(˜x
k
)>dk. (5.31) By (5.31), the test in the projected Armijo line search of Algorithm 14 is satisfied with α = 1. Hence, at the end of Algorithm 14, we set xk+1 = ˜xk+ dk. By (5.30),
we obtain ∇qNk(xk+1) = 0, so that xk+1 is stationary with respect to the non-
active variables.
To conclude the proof, we need to show that xk+1 is stationary with respect to the active variables, too. By (5.28), we have
xk+1Ak = 0, ∇qAk(xk+1) ≥ 0,
and the stationarity of xk+1with respect to the active variables is also proved.