Numerical Methods
• Consider an unconstrained NP problem defined as: min
• Numerical solution methods follow an iterative update scheme: , where is a search direction and is the step size, such that the function value decreases at each step: • Iterative method: 1. Initialize: chose 2. Check termination: 0 3. Find a suitable search direction that obeys the descent condition: 0
4. Search along to find where attains min value 5. Return to step 2
The Iterative Method
• The general iterative method is a two‐step process:
– Finding the suitable search direction along which the function value locally decreases and any constraints are obeyed.
– Performing line search along to find 1 such that 1 attains its minimum value. • Note, – A necessary condition for to be a descent direction is that: 0. – If is a descent direction, then we are assured that at least for small positive values of , .
Line Search Problem
• Assuming a suitable search direction has been determined, we seek to determine a step length , where an optimal value of
minimizes 1 . Since both and are known, the projected function value along depends on alone and is expressed as:
• The problem of choosing to minimize 1 along thus amounts to a single‐variable functional minimization problem, known as the line search problem, defined as:
min α
Example: Quadratic Function
• Consider minimizing a quadratic function: 1 2 , • Let be a given descent direction; then, the line search problem reduces to solving the following minimization problem: min • A solution is found by setting ′ 0, and is given as: • An update then follows as: 1 .Computer Methods for Line Search Problem
• Interval reduction methods – Golden search – Fibonacci search • Approximate search methods – Arjimo’s rule • Quadrature curve fittingInterval Reduction Methods
• The interval reduction methods find the minimum of a unimodal function in two steps: – Bracketing the minimum to an interval – Reducing the interval of uncertainty to desired accuracy • Fibonacci’s method • Golden section method • The bracketing step aims to find a three‐point pattern, such that for 1, 2, 3, 1 2 3 . The bracketing algorithm can be started from any point in the domain of , though a good guess will reduce the number of steps involved.The Bracketing Algorithm
• The bracketing algorithm: – Initialize: choose 1, ∆, (e.g., 1.618) – Set 2 1 ∆; evaluate 1, 2 – If 1 2, set 0 ← 1, 1 ← 2, 2 ← 0, ∆ ∆ – Set ∆ ∆, 3 2 ∆; evaluate 3– If 2 3, set 1 ← 2, 2 ← 3, 1 ← 2, 2 ← 3; then go to step 3
Fibonacci’s Method
• Fibonacci’s Method. The Fibonacci’s method uses Fibonacci numbers to achieve maximum interval reduction in a given number of steps. The Fibonacci number sequence is generated as: 0 1 1, 1 2, 2. • Fibonacci numbers have some interesting properties, among them: – The ratio lim →∞ 1 5 1 2 ≅ 0.618034 is known as the golden ratio. – Using Fibonacci numbers, the number of interval reductionsrequired to achieve a desired accuracy is the smallest such that 1/ , and can be specified in advance.
– For given 1 and , we have 2 1 1, 3 1 2, 4 2 3, etc.
Fibonacci Algorithm
• Initialize: specify 1, 4 1 4 1 , , : 1 • Compute 1 1; 2 1 1 1 1 4, evaluate 2 • For 1, … , 1 • Introduce 3 1 1 4, evaluate 3 – If 2 3, set 4 ← 1, 1 ← 3 – Else set 1 ← 2, 2 ← 3, 2 ← 3 • Set 1 1Golden Section Method
• The golden section method uses the golden ratio: 0.618034 in each iteration to define intermediate points. • Initialize: specify 1, 4 1 4 1 , , : • Compute 2 1 1 4, evaluate 2 • For 1, … , 1 – Introduce 3 1 1 4, evaluate 3 – If 2 3, set 4 ← 1, 1 ← 3 – Else set 1 ← 2, 2 ← 3, 2 ← 3Integrated Bracketing and Golden Section Algorithm
• Initialize: specify 1, ∆, 0.618034, • Set 2 1 ∆; evaluate 1, 2
– If 1 2, set 0 ← 1, 1 ← 2, 2 ← 0, ∆ ∆
• Set ∆ ∆, 4 2 ∆; evaluate 4
– If 2 4, set 1 ← 2, 2 ← 4, 1 ← 2, 2 ← 4; then go to step 3
• Introduce 3 1 1 4, evaluate 3
– If 2 3, set 4 ← 1, 1 ← 3 – Else set 1 ← 2, 2 ← 3, 2 ← 3
Approximate Search Methods
• Gradient Related Condition. The search direction is gradient related if
, 0. This condition aids in convergence.
• Sufficient Descent Condition. The sufficient descent condition guards
against becoming too close to . The condition is normally stated as: 0 for a small . Alternatively, it may be specified
as: 2, 0.
• Sufficient Decrease Condition. The sufficient decrease condition on
ensures that a nontrivial reduction in the function value is obtained at each step. The condition is stated as:
Approximate Line Search
• Alternate sufficient decrease condition (Arjimo’s rule): 0 ′ 0 , 0 1 • Curvature condition: ′ ′ 0 , 0 1 • These two conditions together with are known as Wolfe conditions, which are commonly used by all line search algorithmsApproximate Line Search
• Bracketing the Minimum. In the bracketing step we seek an interval
, such that 0 and 0. Since for any descent direction, 0 0, therefore, 0 serves as initial lower bound on . To find an upper bound, increasing values, e.g., 1,2, …, are tried. Assume that for some 0, 0 and
0; then, serves as an upper bound.
• Estimating the Minimum. Once the minimum has been bracketed
to a small interval, a quadratic or cubic polynomial approximation is used to find the minimizer. If the polynomial minimizer satisfies Wolfe’s condition for the desired value (say 0.5 and the
sufficient decrease condition for the desired value (say 0.2 , it is taken as the function minimizer, otherwise is used to replace one of the or , and the polynomial approximation step repeated.
Quadratic Curve Fitting
• Assuming that the interval , contains the minimum of a unimodal function, , a quadratic approximation of the function: 0 1 2 2 is obtained using three points , , , where the mid‐point may be used for • The quadratic coefficients 0, 1, 2 are obtained as: 2 1 1 1 2 0 1 2 2 • The minimum is given as: 2 1 2Example: Approximate Search
• We wish to approximately solve the following minimization problem:
min 2 where 2 , 0 1, 0 1
• Use Arjimo’s rule with: 0.2, and 0.1, 0.2, …, to bracket the minimum. The Matlab commands are: >> f=@(x) x.*x+exp(‐x); mu=0.2; al=0:.1:1; >> feval(f,al) 1.0000 0.9148 0.8587 0.8308 0.8303 0.8565 0.9088 0.9866 1.0893 1.2166 1.3679 >> 1‐mu*al 1.0000 0.9800 0.9600 0.9400 0.9200 0.9000 0.8800 0.8600 0.8400 0.8200 0.8000 • Then, the minimum is bracketed as: 0, 0.5
Example: Approximate Search
• Quadratic approximation: >> al=0; am=0.25; au=0.5; >> a2 = ((f(au)‐f(al))/(au‐al)‐(f(am)‐f(al))/(am‐al))/(au‐am); >> a1 = (f(am)‐f(al))/(am‐al)‐a2*(al+am); >> xmin = ‐a1/a2/2 = 0.3531 • The exact solution is given as: 0.3517Computer Methods for Finding the Search Direction
• Gradient based methods – Steepest descent method – Conjugate gradient method – Quasi Newton methods • Hessian based methods – Newton’s methodComputer Methods for Finding Search Direction
• Steepest descent method: • Conjugate gradient methods: ; • Newton’s method: • Quasi‐Newton methods: – Direct Hessian update: ∆ – Inverse Hessian update: ∆ , • Trust region methods:Steepest Descent Method
• The steepest descent method is characterized by: , leading to the following update rule: 1 ∙
• The step size to minimize 1 along can be analytically or numerically determined • Example: quadratic function 1 2 , Then, 1 ∙ ; Define Then, 1 ;
