In 1984, Karmarkar proved that an Interior-Point Method (IPM) can solve LPs in polynomial time. The two decades that followed the publication of Karmarkar’s paper have seen a very intense effort by the optimization research community to study theoretical and practical properties of IPMs. One of the early discoveries was that IPMs can be viewed as modifications of Newton’s method that are able to handle inequality constraints. Some of the most important contributions were made by Nesterov and Nemirovski who showed that the IPM machinery can be applied to a much larger class of problems than just LPs [54]. Convex quadratic programming problems, for example, can be solved in polynomial time, as well as many other convex optimization problems using IPMs. For most instances of conic optimization problems we discuss in Chapter 9 and 10, IPMs are by far the best available methods.
Here, we will describe a variant of IPMs for convex quadratic program- ming. For the QP problem in (7.1) we can write the optimality conditions in matrix form as follows:
F (x, y, s) = ATy− Qx + s − c Ax− b XSe = 0 0 0 , (x, s)≥ 0. (7.8)
Above, X and S are diagonal matrices with the entries of the x and s vectors, respectively, on the diagonal, i.e., Xii= xi, and Xij = 0, i6= j, and similarly
for S. Also, as before, e is an n-dimensional vector of ones.
The system of equations F (x, y, s) = 0 has n+m+n variables and exactly the same number of constraints, i.e., it is a “square” system. Because of the nonlinear equations xisi = 0 we cannot solve this system using linear system
solution methods such as Gaussian elimination. But, since the system is square we can apply Newton’s method. In fact, without the nonnegativity constraints, finding (x, y, s) satisfying these optimality conditions would be a straightforward exercise by applying Newton’s method.
The existence of nonnegativity constraints creates a difficulty. The exis- tence and the number of inequality constraints are among the most impor- tant factors that contribute to the difficulty of the solution of any optimiza- tion problem. Interior-point approaches use the following strategy to handle these inequality constraints: One first identifies an initial solution (x0, y0, s0)
that satisfies the first two (linear) blocks of equations in F (x, y, s) = 0 but not necessarily the third block XSe = 0, and also satisfies the nonnegativity constraints strictly, i.e., x0 > 0 and s0 > 0. Notice that a point satisfying
some inequality constraints strictly lies in the interior of the region defined by these inequalities–rather than being on the boundary. This is the reason why the method we are discussing is called an interior-point method.
Once we find such an (x0, y0, s0) we try to generate new points (xk, yk, sk) that also satisfy these same conditions and get progressively closer to satis- fying the third block of equations. This is achieved via careful application of a modified Newton’s method.
Let us start by defining two sets related to the conditions (7.8):
F := {(x, y, s) : Ax = b, ATy− Qx + s = c, x ≥ 0, s ≥ 0} (7.9) is the set of feasible points, or simply the feasible set. Note that, we are using a primal-dual feasibility concept here. More precisely, since x variables come from the primal QP and (y, s) come from the dual QP, we impose both primal and dual feasibility conditions in the definition ofF. If (x, y, s) ∈ F also satisfy x > 0 and s > 0 we say that (x, y, s) is a strictly feasible solution and define
Fo := {(x, y, s) : Ax = b, ATy− Qx + s = c, x > 0, s > 0} (7.10) to be the strictly feasible set. In mathematical terms, Fo is the relative
interior of the setF.
IPMs we discuss here will generate iterates (xk, yk, sk) that all lie inFo. Since we are generating iterates for both the primal and dual problems, this version of IPMs are often called primal-dual interior-point methods. Using this approach, we will obtain solutions for both the primal and dual problems at the end of the solution procedure. Solving the dual may appear to be a waste of time since we are only interested in the solution of the primal problem. However, years of computational experience demonstrated that primal-dual IPMs lead to the most efficient and robust implementations of the interior-point approach. Intuitively speaking, this happens because having some partial information on the dual problem in the form of the dual iterates (yk, sk) helps us make better and faster improvements on the
iterates of the primal problem.
Iterative optimization algorithms have two essential components: • a measure that can be used to evaluate and compare the quality of
alternative solutions and search directions
• a method to generate a better solution, with respect to the measure just mentioned, from a non-optimal solution.
As we stated before, IPMs rely on Newton’s method to generate new estimates of the solutions. Let us discuss this more in depth. Ignore the inequality constraints in (7.8) for a moment, and focus on the nonlinear system of equations F (x, y, s) = 0. Assume that we have a current estimate (xk, yk, sk) of the optimal solution to the problem. The Newton step from
this point is determined by solving the following system of linear equations: J(xk, yk, sk) ∆xk ∆yk ∆sk = −F (x k, yk, sk), (7.11)
where J(xk, yk, sk) is the Jacobian of the function F and [∆xk, ∆yk, ∆sk]T is the search direction. First, we observe that
J(xk, yk, sk) = −Q AT I A 0 0 Sk 0 Xk (7.12)
where, Xkand Skare diagonal matrices with the components of the vectors
xk and sk along their diagonals. Furthermore, if (xk, yk, sk)∈ Fo, then
F (xk, yk, sk) = 0 0 XkSke (7.13)
and the Newton equation reduces to
−Q AT I A 0 0 Sk 0 Xk ∆xk ∆yk ∆sk = 0 0 −XkSke . (7.14)
Exercise 7.4 Consider the quadratic programming problem given in Exer- cise 7.3 and the current primal-dual estimate of the solution xk=1
3,13,13
T
, yk = 1,12T, and sk = 12,12, 2T. Is (xk, yk, sk) ∈ F? How about Fo? Form and solve the Newton equation for this problem at (xk, yk, sk).
In the standard Newton method, once a Newton step is determined in this manner, it is added to the current iterate to obtain the new iterate. In our case, this action may not be permissible, since the Newton step may take us to a new point that does not necessarily satisfy the nonnegativity constraints x ≥ 0 and s ≥ 0. In our modification of Newton’s method, we want to avoid such violations and therefore will seek a step-size parameter αk ∈ (0, 1] such that xk+ αk∆xk > 0 and sk+ αk∆sk > 0. Note that
the largest possible value of αk satisfying these restrictions can be found
using a procedure similar to the ratio test in the simplex method. Once we determine the step-size parameter, we choose the next iterate as
(xk+1, yk+1, sk+1) = (xk, yk, sk) + αk(∆xk, ∆yk, ∆sk).
If a value of αk results in a next iterate (xk+1, yk+1, sk+1) that is also inFo,
we say that this value of αk is permissible.
Exercise 7.5 What is the largest permissable stepsize αk for the Newton
direction you found in Exercise 7.4?
A naive modification of Newton’s method as we described above is, un- fortunately, not very good in practice since the permissible values of αk
eventually become too small and the progress toward the optimal solution stalls. Therefore, one needs to modify the search direction as well as ad- justing the step size along the direction. The usual Newton search direction obtained from (7.14) is called the pure Newton direction. We will consider modifications of pure Newton directions called centered Newton directions. To describe such directions, we first need to discuss the concept of the central path.