An optimisation algorithm is a numerical method or algorithm for finding a value x
such that the result of f(x) is as small (minima) or as large (maxima) as possible, for a given objective function f, possibly with some constraints on x. Here, x can be a scalar or vector of continuous or discrete values. If x is continuous, then the study of the algorithm is part of numerical analysis.
Optimisations practical application often involves writing software (codes) that contains the functional representation (i.e. f(x)) of the problem. This is then linked to the opti- misation algorithm, often supplied in a software library, to create a executable which is typically run on a computer or cluster. The optimisation algorithm during its execution invokes the objective function, often many thousands of times, each time adjusting the input parametersxwhich define the problem subject to any constraints placed on them. Typically, the optimisation algorithm uses the results of prior objective function instan- tiations to determine if the result is converging or diverging from the optimal, adjusting the parameters accordingly. In a successful run the optimisation finishes once it finds a combination of input parameters to the objective function which produces optimal results to within a specified tolerance. Often optimisations fail because the problem subject is extremely complex making it difficult to determine appropriate constraints and tolerances that often results in the algorithm not converging or converging on a less than optimum result. In numerical optimisation programming terms the mathematical model is referred to and handled as the objective function because this provides a suit- able model for development of optimisation algorithm in isolation from the specifics of the mathematical model.
The aim of optimisation is to find results that best meets a specified criteria. For example, within the field of wing design, engineers may wish to find the optimal air flow over a wing that creates the most lift but, with the minimal drag to reduce fuel consumption. The engineer would select a set of parameters for a particular wing model, such as the wings geometry, mesh cell size, material weight and air resistance. The engineer would then produce a mathematical model (objective function) that calculates the air flow. This takes as input information regarding the placement of the particular mesh cell and the applied parameters. Then to simulate the whole wing system this
object function must be passed to a selected optimisation algorithm along with a set of optimisation boundary parameters (constraints) that control the ranges of boundary parameters to try with the optimisation function and the desired tolerances for finding the minimum or maximum optimal result.
The way in which the optimisation algorithm finds the optimal point depends on its type and constraints. The selection of all these variables affects the quality of produced results. Many optimisation algorithms take a non-linear discovery approach whereby it may need to try several different boundary parameters before it can determine if it is headed in the desired direction. This is especially true in multidimensional problems where for instance, in a wing design its geometry and construction material all effect the flow of air.
Optimisation can be a very computationally challenging process as the algorithm may have to instantiate the objective function many thousands of times. This is often sped up by the employment of multiple computers with each machine processing a different objective function in parallel. In addition, objective function themselves can can often be parallelised benefitting both non-linear and linear optimisation.
5.2.1 Nelder-Mead algorithm: Amoeba
The description of the Amoeba algorithm contained in this section is an adaption of the work presented in [111]. Consider the unconstrained optimisation problem of maximising a nonlinear function f(x) for x ∈ <n. A well-known class of methods for solving this
problem is direct search, which does not rely on derivative information (either explicitly or implicitly), but employs only function evaluations. One of the most widely used direct search methods for nonlinear unconstrained optimisation problems is the Nelder- Mead downhill simplex algorithm [112]. It is extremely economical in the number of function evaluations per iteration, and is often able to find reasonably good solutions quickly. On the other hand, the theoretical underpinnings of the algorithm, such as its convergence properties, are less than satisfactory. Nonetheless, its geometric naturalness, working simplicity makes it an idea candidate for study. In this chapter, we focus on one implementation of the Nelder-Mead algorithm as described in the popular handbook Numerical Recipes [113], where it is called the amoeba algorithm.
The amoeba algorithm maintains at each iteration a non-degenerate simplex, a geometric figure in n dimensions of non-zero volume that is the convex hull of n+ 1 vertices,
x0, x1, . . . , xn, and their respective function values. In each iteration, new points are
computed, along with their function values, to form a new simplex. The algorithm terminates when the function values at the vertices of the simplex satisfy a predetermined condition.
1. Order: Order and re-label then+ 1 vertices as x0, x1, . . . , xn, such that f(xo)≥
f(x1)≥. . .≥f(xn). Since we want to maximise, we refer toxo as the best vertex
or point, toxnas the worst point, and toxn−1 as the next-worst point. Letxrefer
to the centroid of then best points in the vertex (i.e., all vertices except forxn):
x= (Pn−1
i=0 xi)/n
2. Reflect: Compute the reflection point xr,
xr=x+α(x−xn) (5.1)
Evaluate f(xr). If f(xo) ≥ f(xr) > f(xn), accept the reflected point xr and
terminate the iteration.
3. Expand: Iff(xr)> f(x0), then compute the expansion point xe,
xe=xr+β(xr−x) (5.2)
Iff(xe)> f(xr) accept xe and terminate the iteration; otherwise (i.e., if f(xr)≥
f(xe) ) accept xe and terminate the iteration.
4. Contract: Iff(xr)< f(xn−1), perform a contraction betweenxand xn
xe=x+ς(x−xn) (5.3)
Iff(xe)≥f(xn) acceptxe and terminate the iteration.
5. Shrink Simplex: Evaluate f at then new vertices fori= 1, . . . , n.
xi =x0+η(xi−x0) (5.4)
For the four coefficients, the standard values reported in the literature are: α = 1, β= 2, ς = 0.5, η = 0.5.