Trend Removal

It was important to remove the trend prior to the calculations described above. When looking for turbulence signatures, one is typically investigating deviations from short-time/space scale phenomena; and hence, larger-scale (typically) non-turbulent phenomena should be excluded.

Figure 75 and Figure 76 illustrate the results of the trend removal process. This is a more extreme case, showing a fairly significant variation in the radiance as a function of time, as seen in Figure 75. Recall that the sub-window used for trend removal is 512 points long, so even after the trend has been removed, some moderate-scale variations can still be seen (Figure 76). Figure 77 and Figure 78 show estimates of the probability distributions (histograms) for the original and trend removed data, respectively. In those figures, the red curve is a smoothed version of the histogram data, and the black curve is a Gaussian distribution with the mean and standard deviation coming from the histogram data. The bimodal structure seen in Figure 77 is due to the flattening of the radiance data at both ends, as seen in Figure 75. After the trend removal the data looks relatively consistent with a zero-mean Gaussian distribution, as seen in Figure 78.

Figure 75. Radiance time series.

94

Figure 76. Same data as shown in Figure 75, after trend removal.

Figure 77. Probability distribution (histogram) for the data shown in Figure 75. The red curve is a smoothed version of the underlying histogram, and the black curve is a Gaussian distribution fit to the histogram.

Figure 78. Probability distribution for the data shown in Figure 76, i.e. the trend-removed data.

95

The importance of removing a trend is clear from Figure 79, which shows the correlation function for the original (black curve) and trend removed (red curve) data. Note that the correlation function is normalized so that the value at zero lag is always one, hence the apparent jump in the red curve. It can be seen that the trend creates an artificial correlation in the data.

Figure 79. Correlation functions with and without trend removal.

Mathematically, this can be seen as follows. The normalized correlation function is given by

(5.1.1) where as an approximation,

(5.1.2) Assume for simplicity that the mean X¯ is zero, and that for not-too-large values of i, the data can be approximated by a linear function of i,

(5.1.3) This approximation means that

(5.1.4) where

(5.1.5) and

(5.1.6)

96

Equation 5.1.1 implies that the correlation function in this case is a linear function of τ. We can further analyze this via a simple simulation. From the original radiance time series, a linear fit was calculated. We created a time series by adding the linear fit to a Gaussian noise sequence with mean and standard deviation calculated from the original time series. These quantities are shown in Figure 80, where the black curve is the original data (i.e. as seen in Figure 75), the red line is the linear fit to the original data, and the blue curve is the simulated data. Figure 81 shows the results of the analysis. The black and red curves are those from Figure 79, whereas the blue curve is the correlation function calculated from the simulated data. It can be seen that the correlation functions for both the original and simulated data are quite similar, and both are linear functions. This provides a good illustration of how trends can produce artificial correlations, and hence why it is important to remove such trends.

Figure 80. Black curve is radiance for wavenumber 961; Red curve is fitted linear trend; Blue curve is fitted linear trend with Gaussian noise (mean and std. deviation from radiance data).

Figure 81. Black and red curves are as seen in Figure 79; blue curve is the correlation function for the simulated data (blue curve) from Figure 80.

Figure 82 and Figure 83 show the spectra calculated from the original and de-trended data, respectively. As mentioned above, the spectra were fit to a quadratic function plus noise (blue curves) and a von Karman function plus noise. As with the correlation functions, there is a dramatic difference at the low frequencies between the spectrum from the original and de-trended

97

data. This behavior is expected since removing a trend acts as a high-pass filter. Figure 84 is a direct comparison between the original and de-trended spectra. Figure 85 illustrates the original and de-trended spectra from a different wavenumber and from a different day. In this case, as can be seen in Figure 86 and Figure 87, there is not too much difference between the original and de-trended data, i.e. there is not much of a trend. From looking at numerous cases, it appears that trends are more prevalent in the lower wavenumber data.

Figure 82. Power spectrum of radiance time series (black). The blue curve is the fit of a quadratic function plus noise to the data, and the red curve illustrates using a von Karman model plus noise.

Figure 83. Same as Figure 82, but with trend-removed radiance data.

98

Figure 84. The effect of trend removal on the power spectra. Wavenumber 961 shown.

Figure 85. The effect of trend removal on the power spectra. Wavenumber 2354 shown. (Note that this is from a different day as the data shown above.)

Figure 86. Probability plot for non-trend removed data used in Figure 85.

99

Figure 87. Probability plot for trend removed data used in Figure 85.

Another type of trend that was observed is a sinusoidal one, as seen in Figure 88. Figure 89 shows the result of the quadratic trend removal process, as described above. Another approach to trend removal is using Fourier methods; an example of which is as follows. A discrete cosine transform of the data was calculated and all the coefficients past the first 60 (arbitrary number), were set to zero. The inverse cosine transform was then applied to this new set of coefficients to generate a filtered time series. The results of this calculation are shown in Figure 90, where the red curve is the original time series, and the black curve is the Fourier-filtered data. The difference of these curves, the Fourier de-trended data, is shown in Figure 91, and a comparison of the two power spectra is given in Figure 92. Returning to the quadratic trend removal, Figure 93 illustrates the correlation functions for the original and quadratic de-trended data (seen in Figure 88 and Figure 89, respectively).

Figure 88. Original radiance data showing larger-scale sinusoidal trend.

100

Figure 89. De-trended radiance data showing residual, smaller-scale sinusoidal trend. (Using quadratic trend removal, as above.)

Figure 90. Using Fourier method for trend removal. Red curve is the original data as seen in Figure 88; black curve is trend from Fourier method.

101

Figure 91. De-trended data – difference between red and black curves from Figure 90.

Figure 92. Spectra from original (black curve) and Fourier de-trended data (red curve).

Figure 93. Correlation functions before and after quadratic trend removal, for the data shown in Figure 88 (original) and Figure 89 (de-trended).

102

As with the linear trend case, a closed-form correlation function can be computed for a sinusoidal trend. We use Equation 5.1.2 with

(5.1.7) and

(5.1.8)

where T is the period of the sinusoidal trend. Using , we

have

(5.1.9) and

(5.1.10)

The correlation function, given by Equation 5.1.1, can then be calculated. The parameters a and T were estimated by eye from the data shown in Figure 88 as 0.13 and 1800, respectively. Figure 94 gives the calculated correlation function. A comparison with the correlation function for the original data (cf. the black curve in Figure 93) shows a good match. This match indicates that the sinusoidal trend dominates the correlation function of the original data.

Figure 94. Correlation function for a cosine trend

Note that the cosine transform is the interferogram for the data, and hence the method just described is simply the application of a filter in the interferogram domain. Since performing Fourier transforms introduces artifacts, a preferable method would be to apply the filter in the original interferogram data and then transform to get a trend-removed radiance time series. For non-stationary data, a method preferable to the Fourier method is the application of a wavelet filter. In summary, we have seen how important trend removal is in investigating small-scale structures in the data, as the trends can dominate the statistical measures, i.e. correlation functions and spectra. Of course, the question arises, “What is the trend that should be removed?”

Obviously, this is an arbitrary process: Is a low-order polynomial or sinusoidal trend appropriate?

103

Over what sub-window lengths should the trend be calculated? Unfortunately, there is no one-size-fits all method, and it is then left to the analyst to decide. Typically, the statistical measures are calculated and then the analyst’s intuition is brought to bear. Often, this process reveals artifacts in the data or in the pre-processing methods. This same issue is discussed in a different context in the next section: fitting models to the radiance spectra.