problem, smooth or nonsmooth. The fact that GSS methods search in multiple direc- tions helps reduce the sensitivity of these algorithms to nonsmoothness. Moreover, the sampling done in GSS can frequently step over loci of nonsmoothness, especially if the nonsmoothness has no particular structure.
We also note that many users are satisfied even when a stationary point cannot be guaranteed. Finding a stationary point of an objective like that in Figure 1.2 might not be what is really wanted. Instead, appreciable improvement in the objective function, at a reasonable cost of time and human effort, is sufficient for many applications.
This said, an inescapable conclusion of this section is that currently there are no theoretical guarantees that GSS methods will succeed on nonsmooth problems. In both the examples set forth above and in our practical experience, there are situations where GSS methods do, in fact, fail to find a stationary point.
Nevertheless, GSS methods, like many optimization algorithms, perform better in practice than their supporting theory might suggest.
7. Some Background on Constrained Optimization. The constrained opti- mization problem of interest is
minimize f(x) subject to c(x)≤0. (7.1)
Here c:Rn →Rm, andf and c are assumed to be continuously differentiable. The inequalityc(x)≤0 is interpreted to hold term by term; i.e.,c(i)(x)≤0,i= 1, . . . , m.
The set of points satisfying the constraints in (7.1) is called thefeasible set and is denoted by Ω:
Ω ={x | c(x)≤0}. A pointxisfeasibleifx∈Ω.
Afeasible iteratesalgorithm ensures (and may require) that each iterate produced is feasible. Aninfeasible iterates approach, on the other hand, allowsxk ∈Ω for some iterations, and only requires feasibility in the limit. In the case of simulation-based optimization, it sometimes is necessary to use methods that produce only feasible iterates since the simulations involved may make no provision for computing outputs if the inputsxare outside the feasible region.
Historically, direct search methods took a feasible iterates approach. Most of the early development of direct search methods for problems with constraints predated the development of infeasible iterates methods such as augmented Lagrangian algorithms [135, 215]. But recall, too, that a common situation in practice is that one wants “improvement” rather than full-blown optimality. Since the user’s goal may be only one or two correct digits, satisfying feasibility “in the limit” is not acceptable. Fur- thermore, in the case of real physical experiments, violating certain constraints (say, setting a temperature to less than absolute zero) may lead to nonphysical, meaning- less results (see [35] for a discussion of one such example in the context of chemical plant management).
Our discussion focuses primarily on methods that use a feasible iterates approach; methods that adopt an infeasible iterates approach are presented, briefly, in sec- tion 8.3.
7.1. The Effects of Constraints. In the unconstrained case, it suffices for one of the search directions to lie within 90◦ of the direction of steepest descent, as dis- cussed in section 2.2. However, if there are constraints present, the situation is more complicated because the constraints may prevent the search from taking a feasible step that would decrease the objective function. Obtaining a direction that yields improvement in the objective while not violating the constraints limits the choices of suitable search directions.
This is illustrated in Figure 7.1(a)–(b). In (a), the red arrow indicates the di- rection of steepest descent from the pointxindicated by the magenta dot. The blue directions form a positive basis and thus would be an acceptable set of search direc- tions (a generating set) for a GSS method in the unconstrained case. In (b) a single constraint, which restricts the solution to the lower half of the figure, is introduced. The boundary of the feasible region Ω is indicated with a black line. To obtain a direction that is within 90◦ of the direction of steepest descentand which does not carry the search out of the feasible region, the directions are constrained to those that fall in the shaded region. Unfortunately, none of the search directions are acceptable in this case—either they are not directions of descent or they take the search outside of the feasible region.
Equally annoying is the situation in Figure 7.1(c). Now x is in the interior of the feasible region Ω, but close to the boundary. The direction pointing toward 10 o’clock is a descent direction, but the search can only take a very short step along this direction if the step is to remain feasible. The longest step possible is indicated by the dot shown in green. As a consequence, only a small improvement in the objective may be realized, much smaller than is possible while remaining feasible. This is solely because of the interaction between the choice of search directions and the geometry
−∇ f(x)
x
(a) A descent direction in the generating set for the unconstrained case.
Ω −∇ f(x)
x
(b) With the introduc- tion of a constraint, no
feasible descent direc- tions in this set.
Ω −∇ f(x)
x
(c) Feasible step along the one descent direction is too short.
Fig. 7.1 How constraints can limit acceptable search directions.
−∇ f(x) x Ω −∇ f(x) x Ω −∇ f(x) x
Fig. 7.2 At least one (feasible) direction of descent along which a sufficiently long step can be taken.
of the nearby portion of the boundary. These short steps, in turn, could cause an unwary algorithm to converge to a point on the boundary that is not a constrained local minimizer. The situation depicted in (c) is similar to the problem of bad step lengths discussed in connection with line search methods in section 2.2; i.e., the steps are short even though the search may be far from any minimizer.
A moment’s reflection reveals that the difficulty derives from the fact that the generating set does not include steps parallel to the boundary. What is needed are directions that allow the search to move along any nearby part of the boundary, taking feasible steps that are sufficiently long. In fact, the remedy for the constraint illustrated in Figure 7.1 is surprisingly simple, as can be seen in Figure 7.2.
7.2. The Geometry of Cones. Unconstrained and equality constrained opti-