Essentials of Optimization

When solving an optimization problem over a set or given functional, it is assumed that the global optimum (usually in convex optimization) or local optimum (usually in nonlinear optimization) is reachable from the given initial conditions. One way to ensure this is by the careful design of the functional. A setC ⊂Rnis considered affine if a line segment connecting any two points in the set

C also lies in the set. Extending this to geometric regions defined by the combination of surfaces (of dimensionRn−1inn-dimensional space), a geometric region is convex if the set of points inside the region and on the defining surfaces are affine.

Given a convex region inRn, an optimization problem’s solution requires finding a point or set

of points that maximize or minimize as specified a stated objective given constraints. For a linear programming (LP) problem, consider a vector fieldvand state spacex_∈Rnand a convex polytope defined by

Ax≤b (3.69)

bensures that the pointxis interior to the defining planes. The linear problem is formally stated as min x v T_x subj. toAx_≤b Fx=h (3.70)

This states, given a vector fieldv, find the point interior to the convex polytope defined by (3.69) constrained by the equalityFx=hthat is in the most ‘negative’ direction of the vector field. When searching for a unique solution, an inherent limitation to LP is that if the vector fieldv is aligned with a row ofA, there may be a set of pointsxthat satisfy the LP and lie on the face of the polytope defined by the corresponding row ofA.

A general quadratic programming (QP) problem is formally stated as

min

x x

T_Qx+vT_x+a subj. toAx_≤b

Fx=h (3.71)

where a is a scalar, and the quadratic term is xTQx and describes a paraboloid, and since the objective is to find the minimum ofx, the matrixQis defined to be positive semi-definite meaning

|Q| ≥ 0. While this is the most common approach used (and works well in most cases), it lacks a

degree of elegance. This is because while the quadratic term minimizes the norm ofx, the termvT_x minimizesxin the vector fieldvand these objectives are linearly constrained inside of a polytope defined by the union of half-spaces Ax _≤ bwhich can be subject to the same multiple solution scenario as the LP problem (consider a plane that restricts further minimization of the weighted normxTQxand is aligned withv). The use of quadratic constraints are captured in the following definition.

The second order cone programming (SOCP) problem is formally stated as min x v T_x subj. to_||Aix+bi||2≤cTi x+ai, i= 1, . . . , m Fx=h (3.72)

where a norm cone is defined by the setC =_{(x, tsocp)_{| ||}x_{|| ≤}tsocp} ⊂Rn+1and the inequality constraint in (3.72) is a norm cone. Simply stated, the norm cone lies in a space one dimension higher than the state space x and the norm of x is bounded by a scalar (here tsocp) on the last dimension. First it must be noted that when ci = 0 this reduces to a quadratically constrained quadratic program (QCQP), and if Ai = 0∀i then this reduces to a general LP. Geometrically, the SOCP can be thought of asmintersecting ellipsoids in Rnwhose shapes and projections are determined by Ai and are subjected to linear constraints defined by the combination of Ai,bi,ci and norm cone bounded by a combination of bi, ai. Given the overlapping ellipsoids and hyper- planes, minimization ofvTxwithin the combined convex region finds the optimum point over the ellipsoids. It is interesting to note that a constant vector fieldvover a set of overlapping ellipsoids (with bounded norm) will find a unique minimum due to the natural curvature of the ellipsoids [73]. To solve these optimization problems, if it is known that a global unique solution exists and the functions are C1 _{differentiable, then it can be solved for directly using Lagrange multipli-}

ers. For example, given a function f(x) and constraint functionsgopt(x) = 0, then by defining

h(x, λ) = f(x) +λgopt(x), then differentiating, then the extremal point is satisfied at the unique

point_∇h(x∗, λ∗_{) = 0. However, often it is the case that the functions and constraints cannot be}

easily solved in this manner, and if so it is not computationally feasible. In these scenarios, nu- merical methods can be used to find the optimal points. Simplistically, this is done using gradient descent methods, where the perturbation∆xwhich decreasesf(x)the most while respecting con- straints is used to update the estimated optimal state xk+1 = xk+γ∆x. This is done until the value converges for a given step sizeγ. There are a host of algorithms to perform this, including but not limited to Dantzig’s simplex algorithm, Newton’s method, the Gauss-Newton algorithm (sim-

plification of Newton’s method), and interior point method (an application of Newton’s method). Dantzig’s simplex algorithm is useful for LP’s where the minimized vector field is not aligned with a hyper-plane in (3.70). The solution is assumed to be on one of the finite vertices of the defined, closed polytope. Starting on a vertex, the edges are followed until the optimal vertex is reached, comparing the functional evaluation at each consecutive function to determine next best step. New- ton’s method is found by taking a second order Taylor series expansion about the current state (single dimension)xk:f(xk+ ∆x) ≈ f(xk) +f(1)(xk)∆x+1₂f(2)(xk)∆x2. By differentiating with respect to∆x, and setting this to zero, the step size∆x=₋f(1)_(x

k)/f(2)(xk)is obtained and used to find the optimum. Newton’s method generalized to a vectorx_∈Rnbecomes

xn+1 =xn−γ∇2f−1(xn)∇f(xn) (3.73)

whereγ _∈[0,1]allows for a smaller step-size if desired along the search direction

∆x=−∇2_f−1_{(xn)∇f(xn). (3.74)}

Newtons method has the requirement that the function be at least twice differentiable. For func- tionsfi(x)that are only once differentiable the Gauss-Newton method can be used as only the first derivative is required. By defining a least squares problemf(x) = 1₂P_ifi(x)2_{, the Gauss-Newton}

search direction becomes

∆x=− X i ∇fi(x)∇fi(x)T !−1 X i fi(x)∇fi(x) ! (3.75) as developed in [73]. The interior point method is an implementation for convex optimization problems where the objective functions are convex and at least twice differentiable. The value in the interior point method is that it reduces an optimization problem with linear equalities and inequalities to just a problem with linear equalities. This is commonly done using barrier method, where a barrier function (e.g. differentiable logarithmic barrier) is used to represent the inequality as a part of the objective function, where states very close to the barrier are heavily penalized [73].