General Algorithm

Newton’s method is very well established and one of the most popular methods for solving nonlinear equations and systems (cf [42,43,49,77,78,100]). In this section, we describe how this method works. Suppose we need to solve the scalar nonlinear equation F (u) = 0. By using a Taylor expansion with an initial guess u0 we can write, [11],

F (u0+ δ) = F (u0) + δF⁰(u0) +δ²

Now we can extend the Newton method for solving a system of n nonlinear equations with n unknowns straightforwardly from the scalar Newton method. We assume F : Rⁿ → Rⁿ is a differentiable function and that all the partial derivatives of F with respect to u exist. The iterative approach converging on a solution assumes that we have some current approximation u^k, k = 0, 1, 2, .... We can obtain the following formula, analogous to (4.52), for an updated

In order to solve the nonlinear system (4.35), we are using Equation (4.53) to generate a sequence of approximations u¹, u², ..., u^k; where (^∂F_∂u(u^k)) is the n× n Jacobian matrix of partial derivatives,

Jij = ∂Fi

∂uj

, i = 1, 2, . . . , n, j = 1, 2, . . . , n, evaluated at the current solution u^k. We can write this matrix in full as:

J(u^k) =

Solving Equation (4.53) requires the inversion of an n× n matrix at every Newton iteration (in practice the solution of an n× n linear system at each Newton iteration). For implementing the system (4.53), we divide the iterative method into two steps as follows:

57 4.3. Iterative Methods for Nonlinear Systems

– We find the vector δ by solving the n× n linear system

J(u^k) δ =−F(u^k). (4.55)

We can solve the system (4.55) for δ by using a standard numerical method for a system of n linear equations.

– Following this we use the equation

u^k+1= u^k+ δ, (4.56)

in order to update the solution u^k. Then we apply Equation (4.55) and Equation (4.56) iteratively until we meet some suitable convergence or failure criteria, for example, until the following convergence condition is satisfied:

kF(u^k+1)k < T ol, (4.57)

where T ol is a user provided value.

The Newton method is extremely popular due to its local quadratic convergence. However, this performance requires a good initial guess and precise convergence properties are problem dependent (see [42, 78, 100]).

There are a wide variety of Newton-type methods, some of which will be reviewed here. In the monograph [42] Deuflhard divided Newton methods into various classes. The first one is that described above, termed the exact Newton method for general nonlinear problems. In this form, the linear system requires the exact numerical solution of the linear equations (4.55). However, the drawback of this method is that at each step we require to solve a system of linear equations by applying a direct method. This can be costly especially when the number of unknowns is large.

Consequently, we will consider a second class of Newton methods: the inexact Newton methods.

These methods solve the linear system approximately, or approximate the Jacobian (one can approximate the Jacobian in different ways), or both. For instance, when F(u) = 0 represents a system of nonlinear equations, which is obtained from the discretisation of a PDE, the Jacobian matrix is often sparse. As a result, sparse iterative solvers may be applied to produce a Newton method that is more efficient applying a dense direct solver. The Newton iteration is defined to be the outer iteration, and the linear iterative steps applied to obtain an approximate update are defined to be the inner iteration. The full scheme then consists of an approximate inner solve at each Newton step k in the inexact Newton method.

Chapter 4. Solution Algorithms for Algebraic Systems 58

There are various types of inexact Newton methods. For example, an inexact Newton method that computes an approximate solution to the Newton equations in a form such that

J(u^k) δ^k = −F (u^k) + r^k, k = 0, 1, . . . , (4.58)

u^k+1 = x^k+ δ^k, (4.59)

k r^k k

k F (u^k)k ≤ η^k, (4.60)

where r^k is the residual,{η^k} is the (decreasing) forcing sequence which is used to control the level of accuracy; we assume that this sequence is uniformly less than one. If ηk = 0, this rep-resents the exact Newton’s method. In this method, for example, we solve the linear equation approximately for the Newton step, rather than approximate the Jacobian (cf [41, 42, 46]).

Another type of inexact Newton method is the simplified Newton method [42]. This method is defined by keeping the Jacobian fixed (frozen) during the full iterations:

J(u⁰) δ^k = −F (u^k), k = 0, 1, . . . (4.61)

u^k+1 = x^k+ δ^k. (4.62)

Since we use the old Jacobian approximation the computational expense per iteration is reduced at the potential cost of growing the number of iterations and probably reducing the convergence domain of the Newton iterations.

A further enhancement to Newton’s method is the so-called globalised Newton method. In this thesis, we apply Newton’s method to linearise the nonlinear system of equations that arise from the finite difference discretisation of nonlinear PDEs. We are interested in finding optimal ways to solve these linear systems. For boundary value PDE problems, an initial approximation is often not given, and it may be hard to obtain a good initial approximation. However, if the convergence relies on the quality of the initial solution, it may be that the initial approxima-tion is not close enough to guarantee convergence and consequently, even the exact Newton’s method diverges. In this circumstance, a global Newton method may be more suitable. This method is able to compensate for an initial poor approximation to the solution by using damp-ing strategies. As an example, a global Newton method is defined usdamp-ing a dampdamp-ing parameter λ^k, (see [7, 42, 43]).

The global Newton method is given as,

J δ^k = −F (u^k), (4.63)

u^k+1 = u^k+ λ^kδ^k , k = 0, 1, . . . (4.64)

59 4.3. Iterative Methods for Nonlinear Systems

where the parameter λ^k > 0 is chosen ideally such that

kF (u^k+1)k ≤ kF (u^k)k. (4.65)

This cannot be guaranteed, and a separate algorithm for choosing λ^k must be used at each Newton iteration.

We now present a final type of inexact Newton method which are Newton-like methods. In this approach, we replace the Jacobian by some approximation such that

M (u) δ^k = −F (u^k), k = 0, 1, . . . (4.66)

u^k+1 = u^k+ δ^k, (4.67)

or replace the Jacobian by some fixed Jacobian J(v) with v6= u⁰. For example, we may consider

”sparsing” a large Jacobian, e.g. ignoring fill-in during factorization, to allow the use of a direct sparse solve for the Newton-like corrections and thus decrease the work per iteration.