Fourier Smoothing

(g)

L L

1.0

1.0

1.0

1.0

– M M

X

FIGURE 6.6 Illustrations of convolution and FSD: (a) A spectrum consisting of two overlapping absorption bands whose peaks are clearly resolved. (b) A boxcar convolution function. (c) MPA smoothed form of (a) having resolved peaks. (d) Fourier transform of (c). (e) Exponential apodization function, exp(2β|x|). (f) Apodized Fourier spectrum accenting the high-frequency coefficients. (g) The self-deconvolved spectrum showing clearly resolved peaks. Parts a, b, and g are WD representations; d, e, and f are FD.

picked up. New regression coefficients are calculated and the polynomial is used to compute the second center point. The fitting and calculating process slides to the right one point at a time until the right end of the spectrum is reached. Seldom would you use a polynomial of a degree higher than five (quintic) for NIR data. Note, although trivial, that no smoothing will be imposed on the spectral data if the number of points in the interval is one less than the degree of the polynomial. That is, a polynomial of degree n will fit(n − 1) points perfectly. Of course, the polynomial method could be used to recompute all points in the interval, but this is seldom done due to the uncertainty of fit in the regression for points removed from the center point of the interval. Therefore, (N− 1)/2 points, where N is an odd number of points over which the polynomial is fitted, are usually dropped just as in the case of the MPA smoothing method. Polynomial smoothing, like MPA smoothing, works on one interval at a time, moving to the right one point at a time.

6.4.1.4 Fourier Smoothing

Fourier smoothing achieves the same objective as MPA and polynomial methods but without the loss of end points. The process seems rather circuitous at first, but a closer look at the results makes the process more appealing. Referring to Figure 6.8, the spectrum f(λ) is transformed to the

(a)

FIGURE 6.7 MPA smoothing: (a1) section of the spectrum in Figure 6.9a, (a2) smoothed by a 9-point MPA smooth, (b) a Lorentzian band with a peak height of 0.75 and FWHH of 10 nm, (c) the MPA convolving function g(x), and (d) convolution of Part b.

FD giving the Fourier spectrum b1. This spectrum is then multiplied by an apodization function (a boxcar function is shown in b2). Finally, the Fourier transform of the truncated Fourier spectrum is computed, yielding the smoothed spectrum.

The effect of smoothing spectral data can be appreciated by looking at Figure 6.9 and Table 6.3.

Figure 6.9a is a spectrum of wool scanned over the range 1150 to 2500 nm. One data point at 1800 nm has been changed from a recorded value of 0.323 to 0.5. log(1/R) units. This change appears as a noise spike on the original spectrum in Figure 6.9a. In the 25-point MPA smooth spectrum the spike has been diminished significantly, but in the process the values of the surrounding data points have been drastically elevated (Figure 6.7a and Figure 6.9b). In addition, 12 points on either end of the spectrum have been dropped as a result of the convolution process. On the other hand, Fourier smoothing has a much less severe effect on surrounding data and end points are not dropped from the spectrum.

The two protein bands (2055 and 2077 nm) and the water band (1941 nm) in Figure 6.9a are marked for reference later. Two things should be noted from Table 6.3. First, MPA smoothing degraded the water band from 0.7168 to 0.7115 as the convolution interval increased from 1 (no smoothing) to 25 points. At the same time the peak of the water band shifted from 1941 to 1943 due to a smoothing moment caused by the asymmetry of the water band. Note this point: If a band is symmetrical there will be no shift of the peak no matter how large the convolution interval. This one fact makes it impossible to determine a priori the optimum convolution interval. Consequently, unless you know a lot about the characteristics of the noise in your spectrometer plus detailed

g (λ)

f (λ)

f (λ)^∗ g (λ) f (λ)^∗ g (λ)

L X

Fourier domain

Wavelength domain

Ᏺ Ᏺ

(a)

(b1)

(b2)

(b3)

FIGURE 6.8 Convolution smoothing in the WD versus smoothing via the FD. MPA smoothing involves the convolution of two functions in wavelength space, but only multiplication of two functions in Fourier space.

information concerning the FWHH of the absorption bands, judgments concerning smoothing are purely empirical.

The effect of boxcar, polynomial, and Fourier smoothing can be seen by expanding the region around the noise spike at 1800 nm in Figure 6.9b–d. Boxcar smoothing (Figure 6.9b) spreads the effect of the noise spike equally over the convolution interval. Polynomial smoothing (Figure 6.9c) spreads the effect over the region of fit but in the process produces features that fall below the normal spectrum. Both boxcar and polynomial smoothing had an effect only within the convolution and region of fit, respectively. Fourier smoothing (Figure 6.9d), on the other hand, produces smoothing with very little effect on the neighboring points. In this case, 39 coefficients produced the best smoothing without excessively altering neighboring points. It is only when the number of coefficients approaches 100 that the effect on neighboring points becomes a problem.

6.4.2 DECONVOLUTION VIAFOURIERSPACE

FSD is the process of removing the effect of the convolution function g(k + λ) (Figure 6.6b) from the spectrum in Figure 6.6c. Another way of putting it is to say FSD allows you to take Figure 6.6c with its unresolved absorption bands and get Figure 6.6a, the original spectrum with its resolved bands. When you scan a spectrum with your machine, you may end up with a spectrum that looks very much like Figure 6.6c. You do not have spectrum (a) as a reference point. However, you can

1.100 1.390 1.680 1.970

FIGURE 6.9 Spectra of wool: (a) Original spectrum with a noise spike at 1800 nm, MPA (25 pts) smoothed spectrum, and Fourier (50 pair) smoothed spectrum. (b) MPA smoothing of the noise spike. (c) Polynomial smoothing of the noise spike. (d) Fourier smoothing of the noise spike.

TABLE 6.3

Effect of MPA and Polynomial Smoothing on Position and Peak Value of Water Band in Wool

MPA smoothing Polynomial smoothing^a No. of points λmax log(1/R) No. of points λmax log(1/R)

1 1941 0.7168 l 1941 0.7168

5 1941 0.7168 5 1941 0.7168

9 1942 0.7158 9 1941 0.7168

17 1942 0.7147 17 1941 0.7168

25 1943 0.7115 25 1941 0.7168

aFourier smoothing gave the same results but without loss of end points.

see a shoulder on spectrum (c) that alerts you to a “hidden” absorber. You would like to know the center wavelength of that absorber. FSD will allow you to make that determination.

The process is a rather simple and fun to do. First, you take the Fourier transform of the spec-trum (c) in Figure 6.6. Then you multiply the resulting Fourier coefficients D, point by point, by an

1.000

FIGURE 6.10 FSD (a) of the NIR spectra of whole-kernel and ground wheat along with the derivative spectrum of whole kernel wheat (b). Tick marks on the whole-kernel spectrum in (a) correspond to the tick marks on the negative peaks of the derivative spectrum in (b). Note the ease with which the deconvolved spectra in (a) compare with the original spectra.

exponential e^βz(Figure 6.6e) where z is the Fourier index over the range from 0 to L. The value ofβ determines the degree of self-deconvolution. The Fourier transform of the result yields the self-deconvolved spectrum in (g) with its clearly resolved peaks.

It is even more fun to look at the effects achieved with FSD of real spectra as in Figure 6.10.

Figure 6.10a gives the spectra of ground and whole-kernel wheat along with their FSD spectra.

The FSD spectrum of kernel wheat has tick marks on absorption bands that correspond to the tick marks on the negative peaks of the second derivative spectrum of kernel wheat (Figure 6.10b). As can be seen, all the bands that appear in the FSD spectrum also appear in the derivative spectrum.

The principal difference is that the derivative spectrum is in a form not easily related to the original log(1/R) spectrum.

How do we know if we have over-deconvolved? Well, we don’t. Neither do we know if our derivative transform produces spurious bands. Both FSD and the derivatization of recorded data are pretty much an art, not a science. One just has to try it and develop a feel for its effect on the data. But here are some guidelines: (a) FSD will produce negative lobes when you over-deconvolve a spectrum. If these lobes go below zero, you have probably over-deconvolved. (b) Look at the narrowest bands. If lobe production around the narrow bands look as if they should not be there, then you probably have over-deconvolved. (c) Finally, you can compare deconvolved spectra with derivative spectra. Until you get a feel for FSD you may want to use this security blanket.

6.4.3 C^OMPUTINGDERIVATIVES VIAF^OURIERS^PACE

NIR researchers are aware that certain advantages may be achieved with derivative spectra. Removal of the offset and the dominant linear term from the spectral data is observed in the computation of derivatives and moving from log(1/R) to derivatives was one of the first attempts to correct for particle-size variations.

There are basically three different approaches to take in computing derivatives from experiment-ally acquired discrete data (a) by convolution, (b) by polynomial regression, and (c) via the FD.

The first method, namely MPA convolution, is derived from an expansion of the Taylor series. The

Taylor series expansion of a function y= F(λ) at (λi+ λ) about λiis

F(λi+ λ) = Fi+ F_i ( λ) + F_i ( λ)²

2! +F_i ( λ)³

3! + · · · (6.14)

where Fiis the ordinate corresponding toλiand (λi+ λ) is the region of convergence. The function at (λi− λ) is similarly given by

F(λi− λ) = Fi− Fi ( λ) + F_i ( λ)²

2! −F_i ( λ)³

3! + · · · (6.15)

The first four terms of Equation (6.14) and Equation (6.15) are added to get an expression for F_i , yielding

F(λi+ λ) + F(λi− λ) = 2F_i + Fi( λ)² (6.16) Therefore

F_i = F(λi+ λ) − 2Fi+ Fi(λi− λ)

x² (6.17)

Equation (6.17) is called the first central difference [15] approximation of the second derivative of the function F(λ) at λi, and the solution is simply an algebraic expression. Therefore, the second derivative of the absorbance d²A/dλ²_i can be approximated by the following expression

d²A

dλ²_i = (A(λi+ λ)) − 2Aλi + (A(_{λi− λ}))

λ² (6.18)

where λ is a small wavelength interval, say, 1 or 2 nm. Equation (6.18) is the mathematical representation of a simplified form (A−2B +C) extensively referred to in the literature [7]. As such, both are only an approximation of the true derivative.

Derivatives are inherently noisy. Choosing to do derivatives with only three points, as is implied by Equation (6.18), will result in noisy data. For example, if one were to compute the derivative of the noise spike in Figure 6.9, the results would be

d²A

dλ²_i = (V1799) − 2(V1800) + (V1801) (6.19) or

(0.3134) − (2)(0.50) + (0.3130) = 1.6264 (6.20) a value that is not the derivative at 1800 nm, assuming the noise spike were not present. In practice, smoothing is incorporated into the derivative calculation by requiring values of A, B, and C in the simplified equation to be the computed averages of a segment of points [7].

Polynomial smoothing requires that an odd set of points be fitted to a polynomial and the center point calculated from the polynomial. Likewise, the derivative spectrum may also be computed from that same polynomial. For example, Equation (6.13) can be used to compute the second derivative of F(λ) by noting that

F (λ) = a1+ 2a2λ + · · · + nanλⁿ⁻¹ (6.21) The procedure is repeated for each group at a time, dropping one point on the left and picking up one at the right each time. As was the case with polynomial smoothing, derivative computations with polynomials can be enhanced by considering it a convolution process and using the convoluting numbers provided by Savitzky and Golay [6].