4.1
Well-posed and stable problems
Numerical algorithms deal mostly with well-posed and stable problems. A problem is well posed if
• The solution exists and is unique
• The solution has a continuous dependence on input data (a small change in the input causes a small change in the output)
Most physical problems are well posed, except atcritical points, where any infinitesimal variation in one of the input parameters of the system can cause a large change in the output and therefore in the behavior of the system. This is calledchaos.
Consider the case of dropping a ball on a triangular-shaped mountain. Let the input of the problem be the horizontal position where the drop occurs and the output the horizontal position of the ball after a fixed amount of time. Almost anywhere the ball is dropped, it will roll down the mountain following deterministic and classical laws of physics, thus the position is calculable and a continuous function of the input position. This is true everywhere, except when the ball is dropped on top of the peak of the mountain. In this case, a minor infinitesimal variation to
the right or to the left can make the ball roll to the right or to the left, respectively. Therefore this is not a well posed problem.
A problem is said to bestableif the solution is not just continuous but also weakly sensitive to input data. This means that the change of the output (in percent) is smaller than the change in the input (in percent).
Numerical algorithms work best with stable problems.
We can quantify this as follows. Letx be an input andybe the output of a function:
y= f(x) (4.1)
We define the condition number of f inxas
cond(f,x)≡ |dy/y| |dx/x| =|x f
0
(x)/f(x)| (4.2) (the latter equality only holds if f is differentiable inx).
A problem with a low condition number is said to be well-conditioned, while a problem with a high condition number is said to be ill- conditioned. XXX
We say that a problem characterized by a function f is well conditioned in a domain D if the condition number is less than1 for every input in the domain. We also say that a problem is stable if it is well conditioned. In this book, we are mostly concerned with stable (well-conditioned) problems. If a problem is well-conditioned in for all input in a domain, it is also stable.
4.2
Approximations and error analysis
Consider a physical quantity, for example, the length of a nail. Given one nail, we can measure its length by choosing a measuring instrument. Whatever instrument we choose, we will be able to measure the length of the nail within the resolution of the instrument. For example, with a tape measure with a resolution of1mm, we will only be able to determine the
length of the nail within1mm of resolution. Repeated measurements per- formed at different times, by different people, using different instruments may bring different results. We can choose a more precise instrument, but it would not change the fact that different measures will bring different values compatible with the resolution of the instrument. Eventually one will have to face the fact that there may not be such a thing as the length of a nail. For example, the length varies with the temperature and the details of how the measurement is performed. In fact, a nail (as every- thing else) is made out of atoms, which are made of protons, neutrons, and electrons, which determine an electromagnetic cloud that fluctuates in space and time and depends on the surrounding objects and interacts with the instrument of measure. The length of the nail is the result of a measure.
For each measure there is a result, but the results of multiple measure- ments are not identical. The results of many measurements performed with the same resolution can be summarized in a distribution of results. This distribution will have a mean ¯xand a standard deviationδx, which we call uncertainty. From now on, unless otherwise specified, we assume that the distribution of results is Gaussian so that ¯xcan be interpreted as the mean andδx as the standard deviation.
Now let us consider a system that, given an inputx, produces the output
y; x and y are physical quantities that we can measure, although only with a finite resolution. We can model the system with a function f such thaty= f(x)and, in general, f is not known.
We have to make various approximations:
• We can replace the “true” value for the input with our best estimate, ¯x, and its associated uncertainty,δx.
• We can replace the “true” value for the output with our best estimate, ¯
y, and its associated uncertainty,δy.
• Even if we know there is a “true” function f describing the system, our implementation for the function is always an approximation, ¯f. In fact, we may not have a single approximation but a series of approxi-
mations of increasing precision, fn, which become more and more ac- curate (usually) asn increases. If we are lucky, up to precision errors, asnincreases, our approximations will become closer and closer to f, but this will take an infinite amount of time. We have to stop at some finiten.
With the preceding definition, we can define the following types of errors: • Data error: the difference betweenxand ¯x.
• Computational error: the difference between ¯f(x¯) and y. Computa- tional error includes two parts systematic error and statistical error. • Statistical error: due to the fact that, often, the computation of ¯f(x) =
limn→∞fn(x)is too computationally expensive and we must approxi- mate ¯f(x)with fn(x). This error can be estimated and controlled. • Systematic error: due to the fact that ¯f(x) = limn→∞fn(x) 6= f(x).
This is for two reasons: modeling errors (we do not know f(x)) and rounding errors (we do not implement f(x) with arbitrary precision arithmetics).
• Total error: defined as the computational error + the propagated data error and in a formula:
δy=|f(x¯)−fn(x¯)|+|fn0(x¯)|δx (4.3) The first term is the computational error (we use fn instead of the true
f), and the second term is the propagated data error (δx, the uncer- tainty inx, propagates through fn).
4.2.1
Error propagation
When a variable x has a finite Gaussian uncertainty δx, how does the uncertainty propagate through a function f? Assuming the uncertainty is small, we can always expand using a Taylor series:
And because we interpretδyas the width of the distributiony, it should be positive:
δy=|f0(x)|δx (4.5) We have used this formula before for the propagated data error. For functions of two variablesz= f(x,y)and assuming the uncertainties inx
andyare independent,
δz= s ∂f(x,y) ∂x 2 δx2+ ∂f(x,y) ∂y 2 δy2 (4.6) which for simple arithmetic operations reduces to
z=x+y δz=pδx2+δy2
z=x−y δz=pδx2+δy2
z=x∗y δz=|x∗y|p(δx/x)2+ (δy/y)2
z=x/y δz=|x/y|p(δx/x)2+ (δy/y)2
Notice that whenz=x−yapproaches zero, the uncertainty inzis larger than the uncertainty inxandyand can overwhelm the result. Also notice that ifz=x/yandyis small compared tox, then the uncertainty inzcan be large. Bottom line: try to avoid differences between numbers that are in proximity of each other and try to avoid dividing by small numbers.
4.2.2
buckinghamBuckingham is a Python library that implements error propagation and unit conversion. It defines a single class calledNumber, and a number ob- ject has value, an uncertainty, and a dimensionality (e.g., length, volume, mass).
Here is an example:
1 >>> from buckingham import * 2 >>> globals().update(allunits()) 3 >>> L = (4 + pm(0.5)) * meter 4 >>> v = 5 * meter/second 5 >>> t = L/v
7 (8.00 +/- 1.00)/10 8 >>> print t.units() 9 second
10 >>> print t.convert('hour') 11 (2.222 +/- 0.278)/10^4
Notice how adding an uncertainty to a numeric value with+
pm(...) or adding units to a numeric value (integer or floating point) transforms the float number into a Number object. A Number object be-
haves like a floating point but propagates its uncertainty and its units. Internally, all units are converted to the International System, unless an explicit conversion is specified.
4.3
Standard strategies
Here are some strategies that are normally employed in numerical algo- rithms:
• Approximate a continuous system with a discrete system • Replace integrals with sums
• Replace derivatives with finite differences • Replace nonlinear with linear + corrections • Transform a problem into a different one • Approach the true result by iterations
Here are some examples of each of the strategies.
4.3.1
Approximate continuous with discrete
Consider a ball in a one-dimensional box of size L, and letx be the posi- tion of the ball in the box. Instead of treatingx as a continuous variable, we can assume a finite resolution of h = L/n (where h is the minimum distance we can distinguish without instruments andn is the maximum number of distinct discrete points we can discriminate), and setx ≡ hi, where i is an integer in between 0 andn; x = 0 wheni = 0 and x = L
wheni=n.
4.3.2
Replace derivatives with finite differences
Computing df(x)/dx analytically is only possible when the function f
is expressed in simple analytical terms. Computing it analytically is not possible when f(x)is itself implemented as a numerical algorithm. Here is an example:
1 def f(x):
2 (s,t) = (1.0,1.0)
3 for i in xrange(1,10): (s, t) = (s+t, t*x/i) 4 returns
What is the derivative of f(x)?
The most common ways to define a derivative are the right derivative df+(x)
dx =h→lim0
f(x+h)−f(x)
h (4.7)
the left derivative
df−(x)
dx =h→lim0
f(x)−f(x−h)
h (4.8)
and the average of the two df(x) dx = 1 2 df+(x) dx + df−(x) dx = lim h→0 f(x+h)−f(x−h) 2h (4.9)
If the function is differentiable inx, then, by definition of “differentiable,” the left and right definitions are equal, and the three prior definitions are equivalent. We can pick one or the other, and the difference will be a systematic error.
If the limit exists, then it means that df(x)
dx =
f(x+h)− f(x−h)
2h +O(h) (4.10)
The three definitions are equivalent for functions that are differentiable in
x, and the latter is preferable because it is more symmetric.
Notice that even more definitions are possible as long as they agree in the limith→0. Definitions that converge faster ashgoes to zero are referred to as “improvement.”
We can easily implement the concept of a numerical derivative in code by creating afunctional Dthat takes a function f and returns the function
df(x)
dx (a functional is a function that returns another function):
Listing4.1: in file:nlib.py
1 def D(f,h=1e-6): # first derivative of f 2 return lambda x,f=f,h=h: (f(x+h)-f(x-h))/2/h
We can do the same with the second derivative: d2f(x)
dx2 =
f(x+h)−2f(x)−f(x−h)
h2 +O(h) (4.11)
Listing4.2: in file:nlib.py
1 def DD(f,h=1e-6): # second derivative of f
2 return lambda x,f=f,h=h: (f(x+h)-2.0*f(x)+f(x-h))/(h*h)
Here is an example:
Listing4.3: in file:nlib.py
1 >>> def f(x):return x*x-5.0*x 2 >>> print f(0) 3 0.0 4 >>> f1 = D(f) # first derivative 5 >>> print f1(0) 6 -5.0 7 >>> f2 = DD(f) # second derivative 8 >>> print f2(0) 9 2.00000... 10 >>> f2 = D(f1) # second derivative 11 >>> print f2(0) 12 1.99999...
Notice how composing the first derivative twice or computing the second derivative directly yields a similar result.
ment them, but they are rarely needed.
4.3.3
Replace nonlinear with linear
Suppose we are interested in the values of f(x) =sin(x)for values ofx
between 0 and 0.1:
1 >>> from math importsin
2 >>> points = [0.01*i for i in xrange(0,11)] 3 >>> for x in points:
4 ... print x, sin(x), "%.2f"% (abs(x-sin(x))/sin(x)*100) 5 0.01 0.009999833... 0.00 6 0.02 0.019998666... 0.01 7 0.03 0.029995500... 0.02 8 0.04 0.039989334... 0.03 9 0.05 0.049979169... 0.04 10 0.06 0.059964006... 0.06 11 0.07 0.069942847... 0.08 12 0.08 0.079914693... 0.11 13 0.09 0.089878549... 0.14 14 0.1 0.0998334166... 0.17
Here the first column is the value of x, the second column is the corre- sponding sin(x), and the third column is the relative difference (in per- cent) betweenxand sin(x). The difference is always less than 20%; there- fore, if we are happy with this precision, then we can replace sin(x)with
x.
This works because any function f(x) can be expanded using a Taylor series. The first order of the Taylor expansion is linear. For values of
x sufficiently close to the expansion point, the function can therefore be approximated with its Taylor expansion.
Expanding on the previous example, consider the following code:
1 >>> from math importsin
2 >>> points = [0.01*i for i in xrange(0,11)] 3 >>> for x in points:
4 ... s = x - x*x*x/6
5 ... print x, math.sin(x), s, ``%.6f'' % (abs(s-sin(x))/(sin(x))*100) 6 0.01 0.009999833... 0.009999... 0.000000
7 0.02 0.019998666... 0.019998... 0.000000 8 0.03 0.029995500... 0.029995... 0.000001 9 0.04 0.039989334... 0.039989... 0.000002
10 0.05 0.049979169... 0.049979... 0.000005 11 0.06 0.059964006... 0.059964... 0.000011 12 0.07 0.069942847... 0.069942... 0.000020 13 0.08 0.079914693... 0.079914... 0.000034 14 0.09 0.089878549... 0.089878... 0.000055 15 0.1 0.0998334166... 0.099833... 0.000083
Notice that the third column s = x−x3/6 is very close to sin(x). In fact, the difference is less than one part in10,000(fourth column). There- fore, for x ∈ [−1,+1], it is possible to replace the sin(x) function with thex−x3/6 polynomial. Here we just went one step further in the Tay- lor expansion, replacing the first order with the third order. The error committed in this approximation is very small.
4.3.4
Transform a problem into a different one
Continuing with the previous example, the polynomial approximation for the sin function works whenxis smaller than 1 but fails whenxis greater than or equal to 1. In this case, we can use the following relations to reduce the computation of sin(x)for large x to sin(x) for 0< x < 1. In particular, we can use
sin(x) =−sin(−x)whenx<0 (4.12) to reduce the domain tox∈[0,∞]. We can then use
sin(x) =sin(x−2kπ) k∈N (4.13) to reduce the domain tox∈[0, 2π)
sin(x) =−sin(2π−x) (4.14) to reduce the domain tox∈[0,π)
to reduce the domain tox∈[0,π/2), and
sin(x) = q
1−sin(π/2−x)2 (4.16)
to reduce the domain tox∈[0,π/4), where the latter is a subset of[0, 1).
4.3.5
Approximate the true result via iteration
The approximations sin(x) ' x and sin(x) ' x−x3/6 came from lin- earizing the function sin(x) and adding a correction to the previous ap- proximation, respectively. In general, we can iterate the process of finding corrections and approximating the true result.
Here is an example of a general iterative algorithm:
1 result=guess 2 loop:
3 compute correction 4 result=result+correction
5 if result sufficiently close to true result: 6 return result
For the sin function:
1 def mysin(x): 2 (s,t) = (0.0,x)
3 for i in xrange(3,10,2): (s, t) = (s+t, -t*x*x/i/(i-1)) 4 returns
Where do these formulas come from? How do we decide how many iterations we need? We address these problems in the next section.
4.3.6
Taylor series
A function f(x):R→Ris said to be areal analyticalin ¯xif it is continuous inx=x¯and all its derivatives exist and are continuous inx =x¯.
local power series: f(x) = f(x¯) + f(0)(x¯)(x−x¯) +· · ·+ f (k)(x¯) n! (x−x¯) k+R k (4.17)
The remainderRk can be proven to be (Taylor’s theorem):
Rk=
f(k+1)(ξ)
(k+1)! (x−x¯)
k+1 (4.18)
where ξ is a point in betweenx and ¯x. Therefore, if f(k+1)exists and is limited within a neighborhoodD={xfor|x−x¯|<e}, then
|Rk|< maxx∈Df (k+1) |(x−x¯) k+1| (4.19)
If we stop the Taylor expansion at a finite value ofk, the preceding formula gives us the statistical error part of the computational error.
Some Taylor series are very easy to compute: Exponential for ¯x=0: f(x) =ex (4.20) f(1)(x) =ex (4.21) . . .=. . . (4.22) f(k)(x) =ex (4.23) ex=1+x+1 2x 2+· · ·+ 1 k!x k+. . . (4.24) Sin for ¯x=0:
f(x) =sin(x) (4.25) f(1)(x) =cos(x) (4.26) f(2)(x) =−sin(x) (4.27) f(3)(x) =−cos(x) (4.28) . . .=. . . (4.29) sin(x) =x− 1 3!x 3+· · ·+ (−1)n (2k+1)!x (2k+1)+. . . (4.30)
We can show the effects of the various terms: Listing4.4: in file:nlib.py
1 >>> X = [0.03*i for i in xrange(200)] 2 >>> c = Canvas(title='sin(x) approximations')
3 >>> c.plot([(x,math.sin(x)) for x in X],legend='sin(x)') 4 <...>
5 >>> c.plot([(x,x) for x in X[:100]],legend='Taylor 1st') 6 <...>
7 >>> c.plot([(x,x-x**3/6) for x in X[:100]],legend='Taylor 5th') 8 <...>
9 >>> c.plot([(x,x-x**3/6+x**5/120) for x in X[:100]],legend='Taylor 5th') 10 <...>
11 >>> c.save('images/sin.png')
Notice that we can very well expand in Taylor around any other point, for example, ¯x=π/2, and we get
sin(x) =1−1 2(x− π 2) 2+· · ·+ (−1)n (2k)! (x− π 2) (2k)+. . . (4.31)
and a plot would show:
Listing4.5: in file:nlib.py
1 >>> a = math.pi/2
2 >>> X = [0.03*i for i in xrange(200)] 3 >>> c = Canvas(title='sin(x) approximations')
4 >>> c.plot([(x,math.sin(x)) for x in X],legend='sin(x)') 5 <...>
6 >>> c.plot([(x,1-(x-a)**2/2) for x in X[:150]],legend='Taylor 2nd') 7 <...>
Figure4.1: The figure shows the sin function and its approximation using the Taylor
expansion aroundx=0 at different orders.
8 >>> c.plot([(x,1-(x-a)**2/2+(x-a)**4/24) for x in X[:150]], legend='Taylor 4th') 9 <...>
10 >>> c.plot([(x,1-(x-a)**2/2+(x-a)**4/24-(x-a)**6/720) for x in X[:150]],legend='
Taylor 6th')
11 <...>
12 >>> c.save('images/sin2.png')
Similarly we can expand the cos function around ¯x =0. Not accidentally, we would get the same coefficients as the Taylor expansion of the sin function around ¯x=π/2. In fact, sin(x) =cos(x−π/2):
Figure4.2: The figure shows the sin function and its approximation using the Taylor
expansion aroundx=π/2 at different orders.
f(x) =cos(x) (4.32) f(1)(x) =−sin(x) (4.33) f(2)(x) =−cos(x) (4.34) f(3)(x) =sin(x) (4.35) . . .=. . . (4.36) cos(x) =1−1 2x 2+· · ·+(−1)n (2k)! x (2k)+. . . (4.37)
With a simple replacement, it is easy to prove that
eix=cos(x) +isin(x) (4.38) which will be useful when we talk about Fourier and Laplace transforms. Now let’s consider thekth term in Taylor expansion ofex. It can be rear-
ranged as a function of the previous(k−1)−thterm: Tk(x) = 1 k!x n= x k 1 (k−1)!x k−1= x kTk−1(x) (4.39)
For x<0, the terms in the sign have alternating sign and are decreasing in magnitude; therefore, forx <0,Rk <Tk+1(1). This allows for an easy implementation of the Taylor expansion and its stopping condition:
Listing4.6: in file:nlib.py
1 def myexp(x,precision=1e-6,max_steps=40): 2 if x==0: 3 return1.0 4 elif x>0: 5 return1.0/myexp(-x,precision,max_steps) 6 else: 7 t = s = 1.0 # first term 8 for k in xrange(1,max_steps): 9 t = t*x/k # next term 10 s = s + t # add next term 11 if abs(t)<precision: return s 12 raiseArithmeticError('no convergence')
This code presents all the features of many of the algorithms we see later in the chapter:
• It deals with the special casee0=1.
• It reduces difficult problems to easier problems (exponential of a posi-