Regularization Methods
4.7 The L-Curve
and assume again that bexact satisfies the model (4.13). Then, for a fixed value of k,
min discrete Picard condition is obvious: α must be larger than 1 in order to ensure that the TSVD and Tikhonov solutions produce similar results.
4.7 The L-Curve
In the previous sections we have used the SVD extensively to define and study the behavior of two important regularization methods. We continue our use of the SVD as an analysis tool, and we take a closer look at the solution norm and the corre-sponding residual norm. These norms play a central role in the practical treatment of discrete ill-posed problems, because they can always be computed no matter which regularization method is used; they do not require the computation of the SVD or any other decomposition of the matrix.
The norm of the TSVD solution and associated residual vary monotonically with k, as can be seen from the expressions
xk22= the column space of A. Similarly, the norm of the Tikhonov solution and its residual are given by monotonically with λ, we introduce the notation
ξ =xλ22 and ρ =A xλ− b22.
Downloaded 07/26/14 to 129.107.136.153. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php
72 Chapter 4. Computational Aspects: Regularization Methods Then we can show that
ξ≡ d ξ d λ =−4
λ
n i =1
(1− ϕ[λ]i )!
ϕ[λ]i "2(uiTb)2 σi2 ,
ρ≡ d ρ d λ = 4
λ
n i =1
(1− ϕ[λ]i )2ϕ[λ]i (uiTb)2=−λ2ξ.
We see that ξ< 0 and ρ> 0 for all λ, confirming the monotonicity of the norms as functions of λ. Also, from the relation ρ=−λ2ξwe obtain d ξ/d ρ =−λ−2, showing that that the squared solution normxλ22 is a monotonically decreasing function of the squared residual normA xλ− b22. Since the square root function is monotonic, the same is true for the norms themselves.
There is also a simple relationship between the second derivatives of ξ and ρ.
Specifically, we have
ρ≡ d2ρ d λ2 = d
d λ
!−λ2ξ"
=−2 λ ξ− λ2ξ,
in which ξ≡ d2ξ/d λ2. Consider now the curve obtained by plotting ξ versus ρ, with λ as the parameter. The curvature cλ of this curve, as a function of λ (see any book on analytical geometry for its definition), is given by
cλ= ρξ− ρξ
((ρ)2+ (ξ)2)3/2 = 2 λ (ξ)2 ((ρ)2+ (ξ)2)3/2,
where we have inserted the above relation for ρ. Hence, cλ > 0 for all λ, and therefore the curve (ρ, ξ) is convex.
The curve is of interest because it shows how the regularized solution changes as the regularization parameter λ changes. We can say more about this curve; for example, it is easy to see that
0≤ xλ2≤ A−1b2 and 0≤ A xλ− b2≤ b2. Also, since any point (ρ, ξ) on the curve is a solution to the problem
ρ = minA x − b22 subject to x22≤ ξ
in (4.12), it follows that the curve defines a boundary between two regions of the first quadrant: it is impossible to choose a vector x such that the point (A x −b22,x22) lies below the curve; only points on or above the curve can be attained. The same is, of course, true for the curve (A x − b2,x2). But when either of these curves is plotted in linear scale, it is difficult to inspect its features because of the large range of values for the two norms.
As shown in [40], the features become more pronounced (and easier to inspect) when the curve is plotted in double-logarithmic scale. The associated curve
(12log ρ , 12log ξ ) = ( logA xλ− b2, logxλ2)
Downloaded 07/26/14 to 129.107.136.153. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php
4.7. The L-Curve 73
10−1 100 101
100 101 102 103
Residual norm || A x
λ − b ||
2
Solution norm || x λ || 2
λ = 1 λ = 0.1
λ = 1e−4 λ = 1e−5
Figure 4.10. The L-curve for Tikhonov regularization: A log-log plot of the solution normxλ2 versus the residual normA xλ− b2 with λ as the parameter.
Notice the vertical and horizontal parts, corresponding to solutions that are under-and over-smoothed, respectively.
is referred to as the L-curve for Tikhonov regularization, and it plays a central role in the analysis of discrete ill-posed problems. Figure 4.10 shows an example of a typical L-curve.
We have already seen that the Tikhonov solution behaves quantitatively differ-ently for small and large values of λ. The same is true for the different parts of the L-curve. When λ is large, then xλis dominated by SVD coefficients whose main con-tribution is from the exact right-hand side bexact—the solution is oversmoothed. A careful analysis (see [34]) shows that for large values of λ we have that
xλ2≈ xexact2(a constant) and A xλ− b2increases with λ.
For small values of λ the Tikhonov solution is dominated by the perturbation errors coming from the inverted noise—the solution is undersmoothed, and we have that
xλ2increases with λ−1 and A xλ− b2≈ e2 (a constant).
The conclusion is that the L-curve has two distinctly different parts, namely, a part that is approximately horizontal, and a part that is approximately vertical. We return to these two parts of the L-curve in Section 5.3.
The “corner” that separates these two parts is located roughly at the point ( loge2, logxexact2).
Toward the right, for λ → ∞, the L-curve starts to bend down as the increasing amount of regularization forces the solution norm toward zero. Likewise, toward the
Downloaded 07/26/14 to 129.107.136.153. Redistribution subject to SIAM license or copyright; see http://www.siam.org/journals/ojsa.php
74 Chapter 4. Computational Aspects: Regularization Methods top left and above the vertical part, the L-curve will eventually become less steep as λ→ 0 and xλ2→ A−1b2 (this part is not visible in Figure 4.10).
This curve is found in many books and papers (perhaps starting with the pi-oneering book by Lawson and Hanson [51]), and a careful analysis of many of its properties can be found in [30]. It can also be used to determine a suitable value of the regularization parameter (this use of the L-curve was not mentioned in [51]); we return to this aspect in Chapter 5.