The conjugate gradient method is an iterative method which allows us to ap- proximately solve the system of linear equations where the size of a problem is too
large and therefore slow for a direct method implementation, [28]. General Conjugate Gradient (CG) algorithm (without constraints) can be modified to accommodate the case of nonnegative constraints as explained by Hestenes [28]. Here we give a brief explanation of CG method with nonnegative constraints adjusted to DL2 SVM no- tation. DL2 SVM’s problem as given in 4.1 subject to 4.2 is the problem of finding a minimum point x(∗) of f(x) such that each component of x(∗) has a nonnegative value.
Lets first assume that x(0) is the initial guess for solution vector x(∗) and that
x(0) = 0. Starting with this initial guess the algorithm searches for the solution with the help of a certain metric that tells if the guess is closer to the solution or not. The metric used here is the residual vector r which becomes smaller as the algorithm gets closer to the unique solution vectorx(∗). Fig. 9 shows the comparison of a steepest (gradient) descent and CG descent. As one can see, conjugate gradient method converges faster, it only takes at mostn steps (assuming no round-off errors) where n is the size of the matrix (here n = 2). However, every iteration of gradient descent method is cheaper than that of conjugate gradient’s.
Before we present the algorithm it is important to introduce the set I and the residual vectorr. Let I be the set of all indexes i ≤m such that x(ik) = 0. Then, at a point x(k) residual vector can be calculated as
r(k) =−f0(x(k)) =1−Ax(k). (4.11)
As one can see, the residual vector r is in fact the negative gradient of f(x) at point
x(k). The algorithm can be explained through the following steps:
Step 1: Select an initial point x(∗) =x(0) that satisfies the nonnegativity con- straints, e.g. x(0) = 0. The residual vectorr(∗)=r(0) used as the first search direction is calculated as r(0) =1−Ax(0).
Step 2: Let I be the set of all indexes i≤m such that
xi(∗)= 0, ri(∗)≤0 ∀i∈I. (4.12)
If residual valueri(∗) = 0,∀i /∈I(or equivalently r
(∗) i
< τ, whereτ is some small stop- ping criteria) thenx(∗) is the solution of the optimization problem and the algorithm terminates.
Step 3: Set the conjugate direction vector p(0) =¯r(0), where¯r(0) is the vector having ¯ ri(0) = 0 :i∈I ri(∗) :otherwise (4.13)
Step 4: Start with k = 0 and perform the standard CG step by computing:
s(k) =A·p(k), (4.14) a(k)= p
(k)T ·r(k)
where scalar valuea(k) is used to adjust the solution and residual vector as follows:
x(k+1) =x(k)+a(k)·pk, (4.16) r(k+1) =r(k)−a(k)·s(k). (4.17)
Step 5: If x(k+1) lies outside of the feasible region, i.e., some x(k+1)
i violate the nonnegativity constraint then go to Step 6 of the algorithm.
Otherwise, if residual valueri(k+1)= 0,∀i /∈I (or equivalently r (k+1) i < τ), reset x(∗) and r(∗) as follows: x(∗)=x(k+1), (4.18) r(∗)=r(k+1) =1−Ax(k+1), (4.19)
setk =k+ 1 and go to Step 2.
Else, set the new residual¯r(k+1) and update the conjugate direction vectorp(k+1) as described bellow: ¯ ri(k+1) = 0 :i∈I ri(k+1) :otherwise (4.20) p(k+1) = ¯r(k+1)− s (kT)·¯r(k+1) p(kT)·s(k) ·p (k). (4.21)
Set k=k+ 1 and go to Step 4.
Step 6: LetJ be the set of indexesj such thatx(jk+1) <0 and define some scalar value ¯a(k) to be the smallest of the ratios:
¯ a(k)=min −x (k) j p(jk) ! ,∀j ∈J. (4.22)
Once we have the value of ¯a(k) we can reset the solution and residual vector as follows:
x(∗) =x(k)+ ¯a(k)·p(k), (4.23) r(∗) =r(k)−a¯(k)·s(k). (4.24)
Redefine set I to be the set of all indexes i≤m such that x(i∗)= 0. If residual vector ri(∗) = 0,∀i /∈I go to Step 2, else go to Step 3.
The algorithm presented above finds the solution vector in a finite number of steps. It starts with a point x(0) that lies in a feasible region (nonnegative). If
x(0) is not a minimum point of f in set S which represents a set of points x whose components are nonnegative, the algorithm locates a pseudo-minimum pointx(1) such that f(x(1)) < f(x(0)). If x(1) 6= x(∗), we locate a new pseudo-minimum point x(2) that minimizes the function on feasible area, such thatf(x(2))< f(x(1)). Ifx(2) 6=x(∗) the process is repeated again until the minimum point has been found. Since there are only a finite number of pseudo-minimum points of F on S, this procedure will eventually terminate when minimum point x(∗) has been found.
Conjugate gradient method, in the absence of round-off errors, produces the exact solution after a finite number of iterations, which is not larger than the size of the matrix. However, the conjugate method is unstable with even small errors, so the exact solution is never really obtained. The error developed in calculating the direction can be detrimental to the convergence. To overcome this drawback, Fletcher and Reeves [33] suggested to revert to the direction of steepest descent after every n or (n + 1) iterations. To complicate things, accumulated roundoff error in the recursive formulation of the residual 4.17 may yield a false zero residual or a value that is within a predefined stopping criterion. This problem could be resolved by restarting with equation 4.11.
To summarize, the CG method monotonically improves the approximation vector until the exact solution is reached within some tolerance. The speed of improvement depends on the condition number of matrix A, larger the number slower the conver- gence.