Comparing Different Types of Approximators for Choosing the Parameters in the Regularization of Ill-Posed Problems

ELSEVIER

Available online at www.sciencedirect.com

computers

&

==,=.c- (~o,.=cT.

mathematics

with appli©ations Computers and Mathematics with Applications 48 (2004) 1779-1790

www.elsevier .com/locat e/camwa

C o m p a r i n g Different T y p e s of A p p r o x i m a t o r s

for Choosing the P a r a m e t e r s in the

Regularization of Ill-Posed P r o b l e m s

J . W . H I L G E R S

Signature Research Inc.

P.O. Box 346, Calumet, MI 49913, U.S.A. B . S . B E R T R A M *

Department of Math. Sciences Michigan Technological University

Houghton, MI 49931, U.S.A.

(Received revised and accepted February 2003)

A b s t r a c t - - R e g u l a r i z e d approximations to the solutions of ill-posed problems typically vary from over-smoothed, inaccurate reconstructions to under-smoothed and unstable solutions as the regular- ization parameter varies about its optimal value.

It thus makes sense to compare two (or more) regularized approximations, and seek the parameter values where the distance between the approximations is a minimum.

This paper advances the theory and practice of this methodology. The method appears to work very well and be very stable, particularly in the presence of extreme error levels. ~) 2004 Elsevier Ltd. All rights reserved.

K e y w o r d s - - I l l - p o s e d problems, Regularization, Parameter choice.

1. I N T R O D U C T I O N

Solving first kind Fredholm integral equations with Hilbert-Schmidt kernels and the standard L2 ( m e a n i n g L2[a, hi) t o p o l o g y is well known to be difficult since t h e p r o b l e m is ill-posed, in t h a t

the inverse o p e r a t o r , or generalized inverse o p e r a t o r , is u n b o u n d e d . T h e p r a c t i c a l consequence of t h i s is t h a t a n y linear s y s t e m t h a t discretizes t h e integral e q u a t i o n is necessarily ill-conditioned, usually t o t h e p o i n t where d i r e c t a t t e m p t s to solve t h e s y s t e m b y conventional m e t h o d s either fail t o execute or p r o d u c e solutions t h a t are useless due to e x t r e m e instabilities.

T h e r e m e d y is t o regularize t h e p r o b l e m in some way (see [1-24]). One c o m m o n a n d quite general way to do this [2] is t o use c o n s t r a i n e d least squares (CLS). For s i m p l i c i t y write the integral e q u a t i o n s c h e m a t i c a l l y as K f = g, where K : L2 --* L2, g e R ( K ) (the range of K ) a n d f e L 2 is t h e solution to be recovered. T h e p r o b l e m as e n c o u n t e r e d in a p p l i c a t i o n s will t h e n be

K f = ~ = g + c , (1)

*The second author wishes to thank Signature Research Inc. for hosting her sabbatical.

1780 J.W. HILGERS AND B. S. BEI~TP~AM

where e represents additive error. T h e least squares approach to solving (1) is to seek the solution to

minimizer lfK f - glI ,

any solution of which must solve the normal equation, K * K f = K*O, which is even more ill-posed than the original (1). In the _foregoing, K has a generally nonelosed range, the norm is Lz (as are all norms in this paper) and the adjoint is generated by the usual L2 inner product.

It is shown in [2] t h a t a necessary and sufficient condition for a unique least squares solution of minimal norm (LSSMN) to exist to the problem K f = g, is t h a t

g E R ( K ) + R ( K ) ± = D

(Kt),

(2) in which case the LSSMN is designated as K t g . The above condition translates into a smoothness condition on the data. If ~ in (1) contains additive error it cannot be assumed to satisfy such a condition. Furthermore, regardless of the content of 9, if the practical problem is to be attacked numerically, the round-off error inherent in finite length arithmetic will effectively destabilize the problem.

To counter this instability, a constraint is added resulting in the CLS problem, minimize/~D(C) I l K f - ~Ii, subject to !lLflt < r,

where L is the regularization operator, usually a differential operator, which penalizes unstable variation in the solutions. D ( L ) is the domain of L and r controls the a m o u n t of regularization. This problem is equivalent to the variational problem,

minimizefeD(L) { H K f - 0tl 2 +

~flLfll2},

(3)

where a is a Lagrange multiplier which is inversely related to r. It turns out any solution to (3) must solve,

( K * K + a Z * L ) f = K ' g , (4)

which is just the normal equation for (3). Assuming the null spaces of K and L contain only the zero vector in common, the operator in (4) is invertible.

2. T H E P A R A M E T E R C H O I C E P R O B L E M

T h e obvious concern in solving (4) is fixing the value of c~. If c~ is t o o large, the reconstruction is over-smoothed and lacking detail. Too small a value of a means the inherent instabilities in the original normal equation become apparent and radical fluctuations develop in the computed solution f . This question has been considered by m a n y researchers over at least the last four decades (all cited papers discuss this topic in one form or another). T h e present paper advances one more m e t h o d which appears superior in certain instances.

The method, which will be denoted C R E F , meaning cross referencing different regularized solutions, is a means for solving the parameter choice problem for essentially two (or more) regularized solutions simultaneously. It is based on the conviction t h a t two computed quantities which approach the same thing, in this case an approximation to the LSSMN, or K t g , will approach one another.

THEOREM 1. Let

]~ = (K*K + a I ) - 1 K * ~ (5)

be the solution to (4) with n = I. Further suppose g and e are as in (1) and [[e[] < [Ig[I. Then

there is an optimal value of a for which I[Ktg - fan attains a minimum, and this a is given

implicitly by,

Ktg - ~ , ~ F - 2 K . - \ ~ g ? = 0 , (6)

where (,) is the L2 inner product, P~ = (K*K + aI) and the left side of (6) is just d/da[[Ktg -

/ 11

PROOF. For large a, ]~ ..~ (1/a)K*g and F~2K*9 ~ (1/a2)K*9. Then for a large the derivative

above is approximated by

K i g - K * O , K*~ = - ~ (KIg, K*.~} - ~ IIK*~[[ 2,

where the second term can be neglected for a large. Now since K K t = RP~-R-y (the projection on

R(K)) and g E R(K) it follows that

d Ktg ] 2 1 1 ( K K l g , ~ ) = 1

-~a - ~ -~ (Kig, K*g} = ~-~ ~5 (tIgII 2 + (g,e)).

But I(g,e)[ <_ ][gNI[e[[ so if []e[I < []gH, it must follow that [[g[i2 + {g,e) > 0. Since [IKtg - ]~[[

]lKigII as a --* oc (since ] , ~ 0) and the derivative is positive, a minimum less that []KtgIt must

be attained and it must satisfy (6). |

The theorem merely says that if Ilel[/I]gl] < 1 then something is gained by using (5) to compute

an approximate solution. If the derivative were negative as a ~ oo it would be possible for the optimal parameter to be infinite and the optimal regularized approximate to be trivially ]oo = 0. The theorem may be extended to more general regularization operators. As an example con- sider (3) and (4) where L is an mth-order linear differential operator with m-dimensional null

space. In this case, take D(L) to be the set

H = { h : L h E L 2 , h (m-i) abscont, h(V)(0)=0, f o r v = 0 , 1 , . . . m - 1 } .

The set H with a different norm is a Sobolev space and also a reproducing kernel Hilbert space (see [23]). Define G to be the integral operator corresponding to the Green's function for the equation

L h = 5 , h(')(0) = 0, for v=O, 1 , . . . m - 1 .

Then L maps H one-to-one and onto L2[0,1] and G = L -1. Thus, by letting f = Gh, it

follows h = L f and (3) becomes for h • L2[0, 1]

minimize { I[KGh - .~[[2 + aiih[[2 } . (7)

The substitution has recast the problem back to an optimization over all of L2 and the solution

to (7) is h~ = (A*A + aI)-lA*{7, where A = KG and the corresponding solution to the problem

in H is ]~ = Gha. One can then repeat the argument of the theorem. Equation (6) now appears

as

where ~ = A*A + aI. The derivative of the norm squared error now becomes, for large a,

1 {(Ktg, G A . g ) + ( K ~ g , GA.e) }

1 7 8 2 J . W . H I L G E R S A N D B. S. B E R T R A M

or ignoring the leading coefficient and using A = KG,

{ (O*Ktg, a*K*g} + (G*Ktg, G*K *e}}. (8)

T h e conclusion of the t h e o r e m now follows if the first t e r m in (8) is positive and the error vector, e, now satisfies,

(a*K*g,O*K*g)

Ita*X*~ll <

IIG*K*gll

'

(9)

in which case the derivative would necessarily be positive for a sufficiently large.

W i t h o u t the G* the first t e r m in (8) is positive as shown in the proof of the theorem. It seems to be generically (but not invariably) true t h a t applying G* to b o t h vectors preserves the positivity.

To see this, e x p a n d Ktg and K*g in the eigensystem for GG*. Thus, there exists {¢¢} C L2[0, 1]

and {Ai} E 7~, such t h a t GG*¢~ = )~i¢~, Ai > 0 and Ai decreases to zero as i goes to co. T h e n

expand,

i i

and {Ktg, K'g} = E i 7~5i = ligll 1 > 0. But t h e n

{e*K*g,a*K* g} = ~ 7~5iA~, (10)

i

which is likely to preserve positivity since Ai > O. T h e foregoing is combined into the following.

COROLLARY 1. Assuming G* preserves the positivity of the inner product in (10) and the error

vector e satisfies (9), then the conclusion of the theorem holds for the problem in H as given in (3),(4).

Returning to T h e o r e m 1, it is possible to estimate the optimal p a r a m e t e r , c~0, and the m i n i m u m square error,

M S E = K t g - f~o 2.

It is expected t h a t a0 and MSE are increasing functions of lieN, and estimates of these dependen- cies are useful in w h a t follows. Estimates similar to those derived below a p p e a r in [2], but there the assumptions, approach taken, and the m e t h o d of imposing smoothness on G are somewhat different t h a n in the current paper.

T h e dependence of a0 and MSE on lien will be a p p r o x i m a t e d by analyzing the upper bound,

K * g -

L <_

I l K * g - f< ll + f<~ - -P'~ ,

(11)

where f~ = (KK* + aI)-lK*g, the noise free form of (5), and the right side of (11) will be

expressed in a complete eigensystem for the kernel, K . Thus, {vn}, {u,~}, and {An} are such t h a t

Kvn = AnUn, K* u,~ = Anv,~, An > O, Jtn ~ O,

In t e r m s of the complete system (2) is equivalent to,

g D

(Kt),

iff

(g,

Un} 2 n = l ~

B o t h t e r m s on the right side of (11) must be estimated. integer N >_ 1,

n = l

a s Tt--+ oo.

- - ~ 0 0 .

It w a s s h o w n in [I0] that for a n y

where

LN

(a) is a line whose slope increases with N and whose intercept decreases to zero as N goes to co. Thus,

[]Kfg - fail ~_ IW(~)= ¢infN>_ILN(C~),

which generally is a concave down function of a with ~(0) = 0. The true functional dependence is determined by g and the spectrum of K. For the second term in (11), it is shown in [2,10] that

fa - ]a <-HF~IK*H'IeH

~--

~ .

Combining these results with (11) Theorem 2 follows.

THEOREM 2.

Given previous notation and definitions, (11) becomes

K t g -

9~, < ¢ ( , ) + ~ , f o r , > 0 (12)

and the

arg min and min of the

right side of (11) will be

taken as

approximations to so and

MSE, respectively.

It is worth noting t h a t if g satisfies stronger smoothness conditions than just (2),an algebraic form for • (a) follows.

THEOREM 3. I f then

(g' u~) ~

A2+2---

~

< 0%

for ~ > O,

I]Kfg- f~ll 2 < D~ v,

Thus, (12) becomes,

where D = A2+2 v .

K t g _

<

II tl

- 2v/-5, a > 0 . PROOF. (See [10].) 3. T H E C R E F M E T H O D O L O G Y

So, to repeat, the idea of C R E F is that as two (or more) different regularized approximates

approach

Kig,

they very probably become near one another. Thus, if f~ and h~ are two such

regularizers, one would examine

D(a,~)

= [[f, - h e [ [ 2, (13) and search for the values of the two variables for which D is a minimum. Of course, the minimum will not invariably exist. By the triangle inequality and the assumption that HeN is sufficiently small so t h a t the square errors for f~ and h5 individually have minima as in Theorem 1 and the corollary, it follows t h a t

and it is clear that D ( ~ , ~ ) has an upper bound which has a minimum at (c~0,~0), the two parameter values where ]~ and h~ independently attain their minima. Whether D also displays a minimum depends on assumed differences in the behaviors of the two solutions for values of c~ and ;3 above and below their corresponding optimal values. To be more precise, the following is a list of properties, some known, some assumed, concerning ]~ and h~ t h a t tend to validate the CREF methodology.

1784 J . W . H I L G E R S A N D B, S. B E R T R A M

2. For tt~]! sufficiently small, llKtg - J~ll has a m i n i m u m for an o p t i m a l alpha, s o > 0. This is proven in T h e o r e m 1 and the corollary.

3. II]~ll - + ~ a s s - + 0 + i f ~ ~

D(Kt).

This is proven various places for the L = I case and

equivalent cases [2].

4. so --+ 0 as II~II --+ 0. A p p r o x i m a t o r s of ao in T h e o r e m s 2 a n d 3 do this, but this is not proven here.

5. MSE --+ 0 as

t1~11

+ o. This is proved in T h e o r e m 3.

6. For values below optimal, when b o t h solutions are under-smoothed, the instabilities in the two solutions are highly disordered and, consequently, very different from each other, resulting in very large values of D ( s , fl). This is assumed.

7. For p a r a m e t e r values above optimal, the over-smoothed case, the two solutions, though stable~ are still different enough to permit the m i n i m u m in D. This is assumed.

Condition 6 above has always proven a safe assumption, even when regularization operators are quite similar [17]. T h e randomness in the instabilities results in very large values of the normed difference (13). Condition 7 is more uncertain. It has always held in the cases considered in the present work as well as [17]. Empirically, it appears Conditions 6 and 7 are very good bets when Condition 2 holds, for b o t h (or all) regularized families.

It is quite easy to give conditions for a m i n i m u m in the linearized version of (13). It is easy to c o m p u t e Frechet derivatives for f~ and h~ and replace t h e m with their linearized approximates in (13). Indeed, if f~ is given as the solution to (4)

It, = (K*.K + s L * L ) -1 K * 9 , it then follows,

5f~ = - F , ~ L LF~ K g, -1 • - 1 . -

where F , = K * K + s L * L , or in the case when L = I ,

5f,~

= - F j 2 K * 9 ,

which (apart from the negative sign) is just the right factor in the inner p r o d u c t (6). Substituting the linearizations into (13) yields,

D ( s , Z) ~ i l f . o + ~ f . o ( S - s 0 ) - hao - ~ h Z o ( 9 - Z0)li 2

= Ibf~o - h : o t ? + 2 ( f . o - h~o, ~ f . o A S - ~ h p o ~ 5 >

÷ tlSf~oAa - 5h:~oA/~ll 2

= constant + AAc~ - BA/~ + C A a 2 + E A ~ 2 - F A a A / ~ ,

(14)

where,

• /~S ~ - S - - S o , • A - - ( / - o -

hZo,Sf,~o},

® B = { f - o -

hzo, Shoo),

• c = llSt~oH 2 • E = j : h ~ o l l ~ . F - - <~f~o Shoo>

Differentiating (14) with respect to s a n d / ~ a n d setting each equation to zero yields

Of course, system (15) is just a statement t h a t the difference vector inside the linearized D ( a , ~)

is orthogonal to the vectors along the two lines, namely 5f,~ o and 5hZo. The determinant of the

matrix in (15) is

- y 2 --

[llaS oll 2 Ilah oll 2

- (5f o,5h o) J _> 0,

C E

where equality can hold only if 5f~ o o( ~h~o. Thus, ruling this out, (15) has a unique solution

for a a n d ft.

Furthermore, it is easy to c o m p u t e the eigenvalues of the matrix in (15) as

( c + E ) + - E ) 2 + 4 F 2

A+ = 2 > 0,

where again equality holds for A_ iff 5fa o and ~h~s0 are linearly dependent. Thus, the critical point obtained from (15) is indicative of a minimum as expected. Geometrically this is also obvious from the nature of the problem. We seek the minimum distance between two lines, and this must yield a unique solution, unless the lines are parallel, meaning ~fao oc 5h~o The foregoing is combined into the following.

PROPOSITION 4. For ao,t3o > 0 and assuming 5f~ o and 5h~o are linearly independent, sys-

tem (15), which solves the minimization problem for the linearized version of (13), has a unique solution indicating a critical point that corresponds to a local minimum.

T h a t 5f~o and 5h~o are linearly independent corresponds to Condition 6 above. The two approximate solutions will generally evolve in different directions when their parameters are decreased, given the nonalignment of the Frechet derivatives as assumed in the theorem.

There is no assumption t h a t a0 and /3o in Proposition 4 are the exact minimizers of the individual regularized solution errors that are discussed in Theorem 1 and the corollary. Of course, in real applications there is no way to know these values as they access the unknown, true solution.

In simulations where K i g is known, it is quite common for the minimizers of (13) and the

true, individual optimal parameters of Theorem 1 and the corollary to vary by several orders

of magnitude. This is determined by the frequency content of K t g as well as the space, H, in

which the regularizer is sought. Indeed, it may not follow t h a t f~ --* K t g as a --~ 0, even in the

noiseless (c = 0) case. If the problem is solved in the space H of Section 1 above, even if e = 0

with g E R ( K ) , it m a y not be true that g E D ( A I) where A = KG. Indeed, it is shown in [10]

that g E D ( A t) iff RP-R-K~(K)g E K ( H ) . And even if g E D ( A t) in general, K t g = PN(K)±(G~)

while GAIg = G~, where ~ is an element of N ( A ) j- (see [10]). Thus, f~0 and h~o could be

approximating functions which differ by a projection.

4. A D V A N T A G E S O F C R E F

There are two basic reasons for advocating CREF as a method of preference for approximately solving ill-posed problems, particularly when the kernel has a spectrum which decreases suffi- ciently fast (see [13]).

1786 J . W . HILGERS AND B. S. BERTRAM 70.00 60.00 50.00 40.00 30.00 20.00 t0.00 0.00 -10.00 -20.00 -1.00 I ' I ~ T ' T - - gbar: amp=10 A - . . . i k -0.50 0.00 0.50 1.00 X Figure 1. 50.00 40.00 30.00 20.00 10.00 0.00 -10.00 -20.00 -1.0( t gbar: amp=lO ... parabola -0.50 o.oo 0.50 1.00 X Figure 2.

Secondly, for t h e p o o r l y b e h a v e d kernels described above, c~0 becomes a v e r y u n s t a b l e function of t h e error vector, e, in (1), t y p i c a l l y f l u c t u a t i n g over several o r d e r s of m a g n i t u d e for different e vectors, even when

Ilell

is fixed. T h i s t o o is discussed in

[13]

as arising from large gaps between singular values of K t h a t occur near t h e value of a0. T h i s m a k e s a t t e m p t i n g t o e s t i m a t e C~o d i r e c t l y difficult, w i t h a t e n d e n c y t o w a r d serious u n d e r s h o o t i n g . However, it is i m p o r t a n t to note t h a t t h e o p t i m a l regularizer, f~o, is v e r y s t a b l e even when Cto is not.

A d e t a i l e d c o m p a r i s o n of t h e C R E F and G C V m e t h o d s was c o n d u c t e d in [11]. T h e r e it was found t h a t w i t h kernels whose s p e c t r u m is decreasing sufficiently fast, t h e G C V function of a t h a t is t o be minimized, n a m e l y [1, w i t h M = N]

N

E +

is n o t a r o b u s t c a n d i d a t e for t h e a r g m i n o p e r a t i o n . This is b e c a u s e t h e curve t e n d s to have m u l t i p l e " p l a t e a u s " , often each w i t h its own local m i n i m u m . T h e g l o b a l m i n i m u m can fluctuate over several orders of m a g n i t u d e , even w i t h error vectors of t h e s a m e length. In [13] it was shown t h a t

SSE(a) = IIf~ - f0il 2,

where f0 is the exact solution, has this property if the kernel's eigenvalues decrease approximately

two orders of magnitude with each index near the value of the optimal parameter, O~op t. Thus, it

is not surprising that (16) would also display this behavior.

d 2

In [11], the regularization operators ~-~ and the identity, along with the truncated SVD and conjugate gradients (where the regularization parameters are the number of terms retained and the number of iterations, respectively) were all used in a CREF comparison to the argmin of (16). Two versions of GCV were compared; the true argmin of (16) and the largest local minimizer

1788 J . W . HILGERS A N D B. S. B E R T R A M

of (16). The latter computed a, termed GCV2, turned out to be much more stable, and provided

a much improved regularized solution, which often compared favorably to those obtained by CREF.

The comparisons were run for the same kernel and two true solutions, much like the examples described below, and three noise levels. Overall GCV did not perform well though GCV2 did better. The main point is in the few cases where GCV2 provided a more nearly optimal parameter

t h a n C R E F , the latter's r e c o n s t r u c t i o n w a s a l m o s t as good.

5. N U M E R I C A L E X A M P L E S

The test eases were discretized versions of (1),

where R" is a sinc kernel f ( x ) = 10 sin(2Ox)/Qrx), g = K f o where f0 is the object or true solution, and c is a pseudo-Caussian noise vector. This noise vector is multiplied by an amplitude, AMP,

t h a t was varied from .001 to 10.0. These values of AMP resulted in IHI/I]gH ratios that varied

from approximately 5 . 1 0 -5 to 0.5, respectively. This latter value represents an extreme error level that approaches the limit in the hypotheses of Theorem 1. Figures 1 and 2 display 9 = 9 + e for the two f0 (see below) and AMP=10.

The integral in (17) was discretized by a quadrature sum using the trapezoid rule based on a uniform partition of [-1, 1] into 100 subintervals. Two object functions were used as examples, the uneven rectangles show in Figures 3 and 4 and the parabola y = (x + 1) 2 shown in Figures 5 and 6. These two f0 choices, taken with the regularization operators, represent cases where f0 belongs to the domain of the regularization operator, or fails to do so because of discontinuity or failure to satisfy the requisite boundary condition(s).

Three regularizing operators were used: the identity, the first derivative, and the second deriva- tive (or Laplacian). These regularizers were utilized two at a time yielding six approximate so- lutions for each object and noise level. Each computer simulation produced two approximate solutions: one for each regularizer used. Sometimes these solutions were coincident (to graphical accuracy) and sometimes they were "close" to each other but visually distinguishable. In gen- eral, it was observed t h a t solutions were coincident with the derivative--second derivative pair of regularizers.

Figures 3 and 4 display the results using a noise level of 10 with the uneven rectangles as the object. In Figure 3, the approximate solutions are coincident with the regularizers being the derivative and second derivative operators. Clearly, while the reconstructions are not perfect, they convey valuable information about the object. In Figure 4, the identity and derivative operators are utilized. The results are quite similar, with the identity/derivative curve being slightly better.

Figures 5 and 6 are reconstructions for the parabola using the identity and second derivative operators in the C R E F procedure. Clearly, the solution produced using the lower noise level is the better of the two. However, once again, both give information about the object function.

6. C O N C L U S I O N S

CREF, the process of minimizing (13) over the regularization parameters ~ and [3, has worked in all cases run to date. The resulting regularizers, f~ and h~, have always proved to be good, stable, reconstructions, even when the minimization process was a rather coarse search, and even

when the noise to signal ratio Ilell/tlg[I, was as large as 0.54.

Although work is continuing, to date CREF has provided stable reconstructions of high acuity to a degree exceeding any other method we have tested.

R E F E R E N C E S

1. G.H. Golub, M. Heath and G. Wahba, Generalized cross-validation as method for choosing a good ridge parameter, Technometrics 21 (2), (1979).

2. C.W. Groetsch, The theory of Tikhonov regularization for Fredholm equations of the first kind, Research

Notes in Mathematics 105, P i t m a n Advanced Published, (1977).

3. V.A. Morozov, Methods for Solving Incorrectly Posed Problems, Springer-Verlag, (1984).

4. A.N. Tikhonov, Solution of incorrectly formulated problems and the regularization method, Soviet Math.

Dokl., (1963).

5. C.K. Rushforth, Chapter 1. Signal restoration, functional analysis and Fredholm integral equations of the first kind, In Image Recovery: Theory and Application, (Edited by H. Stark), Academic Press, New York, (1987).

6. A.N. Tikhonov and V.Y. Arsenin, Solutions of Ill-posed Problems, V.H. Winston and Sons, (1977). 7. A.M. Thompson et al., A study of methods of choosing the smoothing parameter in image restoration by

regularization, IEEE Trans. on P A M I 13 (4), 326-339, (April 1991).

1790 J . W . H I L G E R S A N D B. S. B E R T R A M

9. A.K. Katsaggeloe, Iterative image restoration algorithms, Optical Eng. 28 (7), 735-738, (July 1989). 10. 3.W. Hilgers, Non-iterative methods for solving operator equations of the first kind, TSR 1413, Math. Res.

Center, Univ. of Wisconsin-Madison~ (1974).

11. E.M. Wilcheck, A comparison of cross-referencing and generalized cross-validation in the optimal regulariza- tion of first kind equations, Masters Thesis, Michigan Tech. University, (1992).

12. S.J. Reeves and R.M. Mersereau, Optimal estimation of the regularization parameter and stabilizing func- tional for regularized image restoration, Optical Engineering 29 (5), 446-454, (1990).

13. J.W. Hilgers and W.R. Reynolds, Instabilities in the optimization parameter relating to image recovery problems, J. Opt. Soc. Am. A 9 (8)~ 1273-1279, (August 1992).

14. B. Bertram, On the use of wavelet expansions and the conjugate gradient method for solving first kind integral equations, In Integral Methods in Science and Engineering, (Edited by B. Bertram, C. Constanda, and A. Struthers), pp. 62-66, Chapman Hall/CRC, Boca Raton, LA, (2000).

15. B. Bertram and H. Cheng, On the use of the conjugate gradient method for the numerical solution of first kind integral equations in two variables, In Integ~ul Methods in Science and Engineering 2000, (Edited by C. Constanda), Birkhauser, Boston, MA, (2002).

16. H. Cheng and B. Bertram, On the stopping criteria for conjugate gradient solutions of first kind integral equations in two variables, In Integral Method8 in Science and Engineering 2000, (Edited by C. Constanda), pp. 57-62, Birkhauser, Boston, MA, (2002).

17. J. Hilgers, B. Bertram and W. Reynolds, Cross-referencing different regularized families of approximate solutions to ill-posed problems for solving the parameter choice problem in generalized CLS, In Integral

Methods in Science and Engineering 2000, (Edited by C. Constanda), pp. 105-110, Birkhauser, Boston, MA,

(2002).

18. L. Desbat and D. Girard~ Tim ~Minimum Reconstruction Error" choice of reconstriction parameters: Some more efficient methods and the application to deconvolution problems, S I A M J. of Sci. Comp. 16 (5), 1387- 1403, (1995).

19. M. Hanke and T. Raus, A general heuristic for choosing the regularization pararaeter in ill-posed problems,

SIAM g. Sci. Comput 17 (4), 956-972, (1996).

20. A. Frommer and P. Manse, Fast CG-based methods for Tikhonov-Phillips regularization, SIAM. J. Sci.

Comput. 20 (15), 1831-1850, (1999).

21. M.E. Kilmer and D.P. O'Leary, Choosing regulaxization parameters in iterative methods for ill-posed prob- lems, S I A M Y. Matrix Anal. Appl. 22 (4), 1204-1221, (2001).

22. D.P. O'Leary; Near-optimal parameters for Tikhonov and other regularization methods, Sei. J. Sci. Comput 23 (4), 1161-1171, (2001).

23. S. Saitoh, Integral transformations, reproducing kernels and their applications, In Pitman Research Notes is

Mathematics Series (369), Addison Wesley Longman, London, (1997).