2.5 Methods for Nonlinear Programming
2.5.3 Constrained Quadratic Optimization
Essentially two classes of methods have been developed for solving linearly con- strained quadratic programs of the form QP, namely active set methods (ASM) (see, e.g., [94, 168]) and interior point methods (IPM) (see, e.g., [95, 152]). For a detailed comparison, we refer to [162]. In this thesis, we will focus on inequal- ity constrained convex problems, i.e., we assume Q to be positive semidefinite. Equality constrained quadratic programs of the form EQP do not require spe- cial algorithms to be solved, since solving its KKT-system, which is a system of linear equations, is sufficient (see Section 2.4). We therefore concentrate on quadratic programs with inequalities and describe a solution technique based on the reduction to equality constrained problems.
Active Set Methods
The main idea of active set methods is to solve a sequence of equality constrained quadratic programs by considering only the active constraints at the current iterate. For sake of simplicity, assume we have an optimization problem of the form QP without equations, i.e., m = 0. Recall that by Definition 2.3 the active and non-active sets at some point xk are denoted by
A(xk) := {i | a> i x
k = b
i} and N (xk) := {1, . . . , m} \ A(xk).
It is well known that active set methods are preferable to interior point methods in practice, if the number of constraints is small or medium.
The basic idea of general active set methods for convex quadratic programming problems relies on the following observation. If x? is the minimizer of QP, we
have
x? = argmin f (x) s.t. Ax ≤ b
= argmin f (x) s.t. a>i x = bi for all i ∈ A(x?).
Knowing the optimal active set of x? would solve the problem immediately by
finding the solution to (2.8). Since this is not the case, the approach starts by choosing a working set Wk ⊆ {1, . . . , m} and computing
xk+1 = argmin f (x) s.t. a>i x = bi for all i ∈ Wk.
If the new iterate xk+1 is not the minimizer of QP, adjust Wk to Wk+1 in a clever
way and solve the updated equality constrained quadratic programming problem. It is common to distinguish between primal and dual active set methods. While a primal active set method ensures primal feasibility of all iterates, a dual active set methods ensures dual feasibility.
2.5. METHODS FOR NONLINEAR PROGRAMMING 27 Let Wk denote the current working set and Ak the matrix that consists of all
rows of A belonging to Wk. We assume that the row vectors ai, i = 1, . . . , m,
are linearly independent. Algorithm 4 gives a basic version of a primal active set method for quadratic optimization problems with linear inequalities.
Theorem 2.26 ([91]). Consider Algorithm 4 for Problem QP.
(i) If Q is positive definite and ai (i ∈ Wk) are linear independent, (2.20) has
a unique solution.
(ii) If ai (i ∈ Wk) are linear independent at any iteration k and the algorithm
does not terminate after solving (2.20), this also holds for ai (i ∈ Wk+1).
(c) If Q is positive definite and ∆xk 6= 0, we have ∇f (xk)>∆xk < 0, i.e., ∆xk
is a descent direction.
In each iteration k, the algorithm considers the feasible iterate xkand checks if it
minimizes the quadratic objective function in the subspace defined by the current working set Wk. If not, it computes a descent direction ∆xk by solving (2.20).
This corresponds to solving the equality constrained quadratic program with constraints a>i x = bi for all i ∈ Wk while all other constraints are relaxed. From
Theorem 2.17 we know that ∆xk 6= 0, since otherwise the iterate would already be optimal in that subspace. On the other hand (xk+ ∆xk, λk+1
Wk) is a KKT-point
of that equality constrained problem. Since Ak∆xk = 0, every constraint in the
working set will still be satisfied with equality by the new iterate xk+ ∆xk. If
(c) holds, i.e., xk+ ∆xk is feasible for all constraints, we update our iterate and
the working set is not changed. If there are constraints i /∈ Wk that are violated
by the new iterate, we choose the step size αk as large as possible until the first
constraint that is violated becomes active and any other constraints stays strictly feasible. Therefore we choose the step size as αk := mini /∈Wk, a>i∆xk>0
bi−a>i xk
a>i∆xk ,
update our iterate by xk+ α
k∆xk and add this blocking constraint to the working
set. This iterative process is repeated until a point is reached that is optimal for the current working set, i.e., when ∆xk = 0. Then, either all of the multipliers in
the working set Wkare non-negative and we can stop since (xk+1, λk+1) is already
a KKT-point of our original problem, or if there is a negative multiplier in the working set Wk, we remove this constraint. It can be shown that the solution of
the resulting subproblem (2.20) gives a feasible direction for the constraint that has just been dropped.
Assuming that it is always possible to take a nonzero step along a nonzero descent direction guarantees that the algorithm does not cycle. It can also be shown that no working set is considered twice. Since the number of different working sets is finite, the algorithm stops after a finite number of iterations with a KKT-point of the problem. Nevertheless active set methods can have exponential running time
28 CHAPTER 2. NONLINEAR PROGRAMMING Algorithm 4: Active Set Method
input : symmetric, positive definite matrix Q ∈ Rn×n, matrix A ∈ Rm×n, vectors c ∈ Rn, b ∈ Rm
output: minimizer of q(x) := x>Qx + c>x such that Ax ≤ b Choose feasible x0 ∈ Rn for QP and λ0 ∈ Rm. Set W
0 := {i | a>i x0 = bi}.
while (xk, λk) is not a KKT-point do Set λk+1i := 0 for all i /∈ Wk. Solve
2Q A> k A>k 0 ∆x λA =−∇f (x k) 0 . (2.20) Distinguish between: (a) If ∆xk= 0 and λk+1
i ≥ 0 for all i ∈ Wk: STOP
(b) If ∆xk= 0 and min{λk+1 i | i ∈ Wk} < 0: choose q := arg min{λk+1i | i ∈ Wk} and set xk+1 := xk, Wk+1 := Wk\ {q}.
(c) If ∆xk6= 0 and xk+ ∆xk is feasible for QP:
set
xk+1 := xk+ ∆xk, Wk+1 := Wk.
(d) If ∆xk6= 0 and xk+ ∆xk is infeasible for QP:
choose r := arg min bi− a > i xk a>i ∆xk | i /∈ Wk s.t. a > i ∆x k > 0 and set αk := br− a>rxk a> r∆xk , xk+1 := xk+ αk∆xk, Wk+1 := Wk∪ {r}. Set k := k + 1.
2.5. METHODS FOR NONLINEAR PROGRAMMING 29 in the dimension of the problem [125], due to the existence of up to 2n possible
working sets, but their practical performance is usually much better. A significant advantage of active set methods is that they can be easily warmstarted. We will profit from this fact in the design of our branch-and-bound algorithms.