Active Set Method

Given a feasible solution , we say that the -th constraint is active at if holds and

inactive otherwise. We define a set of indices for active constraints: . is named the active set at . The active set method starts with an initial estimation of an active set and iteratively refines the estimation.

If we've known the optimum solution for the convex quadratic program in (3.4), we can define a problem with only equality constraints specified by the active set :

(3.7)

subject to

We denote the original problem by and the equality constrained problem by . Obviously, an optimum solution of must be optimum for . Since is unknown in advance, has to be estimated. We can start with a rough estimation of the active set, and refine our estimation step by step. We call the estimated active set by the working active set, and denote it by . is not necessarily the current active set . Instead, is often a subset of .

To solve the equality constrained problem , we can remove the equality constraints by introducing some auxiliary variables. For example, we have equality constraints, and we

36

represent them by . Since we assume the rows are linear independent, the null space has a basis of vectors. We put the vectors into the columns of matrix

. Any solution of can be represented by

(3.8)

where is a solution of and is a vector of auxiliary variables. Substituting for in the objective function , we get an unconstrained optimization problem . There are many iterative ways to solve the unconstrained problem , for example, gradient descent, Newton method, conjugate gradient, etc. The gradient descent looks for the steepest direction from the current position. The Newton method finds the next step from the gradient and the Hessian matrix. Usually the Newton method converges faster than the gradient descent method, but it requires to solve a linear system of the Hessian matrix. After we have a descent step for , we can compute the descent step for by . We call the gradient in space by the projected or reduced gradient, and the Hessian matrix in space by the projected or

reduced Hessian matrix.

There are some termination criteria for various methods for the unconstrained problem . When the gradient descent method is used, the termination criteria is usually that the gradient magnitude is less than some small tolerance.

When the termination criteria for is satisfied, we have reached an optimum for the equality constrained problem . From our discussion in Section 3.2.1, if its Lagrange multipliers for all inequality constraints in are non-negative, it must be also an optimum for the original problem . Otherwise there is a negative Lagrange multiplier with , we can construct a descent direction that moves off the -th constraint. We remove from the current working active set.

When the termination criteria for is not satisfied, we may take a descent step to reduce the objective function. After the step, the solution should be still feasible to the original problem. Denoting the descent direction by , we find the biggest feasible step size such that

37

where is the step size restricted by the -th constraints not in the current working set . Then we do a line search for to minimize the objective function. If this step size touches some constraints not in , we select one of them and add it to .

Based on what we've just described, we have the following active set method for convex quadratic programming [38, 39].

Algorithm 3.1 Classic Active Set Method for Convex Quadratic Programming Initialize , a feasible solution of ;

Initialize the working active set ; for

Compute a descent direction for the equality constrained problem corresponding to ; if satisfy the termination criteria for

then

Compute the Lagrange multipliers at for ; if

then

Terminate with as the optimum; else

Select the most negative from , and denote it by ; _;

end if

else // doesn't satisfy the termination criteria for Find the biggest feasible step size along direction ; Find the optimum step size by line search;

if _{touches some new blocking constraints with indices}

38

Assuming we use the gradient descent method to find the optimum for the equality constrained problems, we can analyze the termination of the active set method from two aspects. First, it takes finite steps to reach an optimum solution for the current working active set. After each descent step, if it doesn't touch any blocking constraints, it decreases along the line search directions. According to [37], gradient descent takes steps for unconstrained quadratic programming where is the initial gap. That means, we will take at most iterations before we either reach the optimum or touch some new blocking constraint. Since the number of new blocking constraints with which we can expand is bounded by , the number of inequality constraints, it takes at most iterations to reach an optimum for a working active set.

Second, when we reach an optimum for the equality constrained problem corresponding to the current working active set , this set won't appear again in future iterations. After an optimum is reached for the current , if there is no negative Lagrange multiplier for the inequality constraints, we have reached the optimum of the original problem. Otherwise there is a negative Lagrange multiplier, we remove its index from . At the next iteration, we will always find a feasible step that is also strictly descent as shown in (3.6). That is, holds. Since all future steps are not ascending, it is impossible for us to reach the same working active set again. Otherwise, we will find a better optimum for the same and reach a contradiction. Since the number of possible working active set is finite, the algorithm must terminate in a finite number of iterations.