Abstract
The term “image fusion” covers multiple techniques used to combine the geometric detail of a high-resolution panchromatic image and the color information of a low-resolution multispectral image to produce a final image with the highest possible spatial information content while still preserving good spectral information quality.
During the last twenty years, many methods such as Principal Component Analysis (PCA), Multiplicative Transform, Brovey Transform, and IHSTransform have been developed producing good quality fused images. Despite the quite good visual results, many research papers have reported the limitations of the above fusion techniques. The most signifi-cant problem is color distortion. Another common problem is that the fusion quality often depends upon the operator’s fusion experience and upon the data set being fused.
In this study, we compare the efficiency of nine fusion techniques and more specifically the efficiency of IHS, Modified IHS, PCA, Pansharp, Wavelet, LMM (Local Mean Matching), LMVM (Local Mean and Variance Matching), Brovey, and Multiplicative fusion techniques for the fusion of QuickBird data. The suitability of these fusion tech-niques for various applications depends on the spectral and spatial quality of the fused images.
In order to quantitatively measure the quality of the fused images, we have made the following controls. First, we have examined the visual qualitative result. Then, we examined the correlation between the original multispectral and the fused images and all the statistical parameters of the histograms of the various frequency bands. Finally, we performed an unsu-pervised classification, and we compared the resulting images.
All the fusion techniques improve the resolution and the visual result. The resampling method practically has no effect on the final visual result. The LMVM, the LMM, the Pansharp, and the Wavelet merging technique do not change the statistical parameters of the original images. The Modified IHSprovokes minor changes to the statistical parameters than the classical IHSor than the PCA. After all the controls, the LMVM, the LMM, the Pansharp, and the Modified IHSalgorithm seem to gather the more advan-tages in fusion panchromatic and multispectral data.
Introduction
Almost all the high-resolution (SPOT, Landsat, IRS, Ikonos, QuickBird, and Orbview) collect a high spatial resolution panchromatic (PAN) image and multiple (usually four) multispectral (MS) images with significant lower spatial
Comparison of Nine Fusion Techniques
for Very High Resolution Data
Konstantinos G. Nikolakopoulos
resolution. The PANimages are characterized by very high spatial information content well-suited for intermediate scale mapping applications and urban analysis. The MSimages provide the essential spectral information for smaller scale thematic mapping applications such as land-use surveys. It is of interest to note that most satellites do not collect high-resolution MSimages directly, to meet this requirement for high-spatial and high-spectral resolutions.
There is a limitation to the data volume that a satellite sensor can store on board and then transmit to a ground receiving station. Usually the size of the PANimage is many times larger than the size of the MSimages. The size of the PAN of Landsat ETMis four times greater than the size of an ETM MSimage. The PANimage for Ikonos, QuickBird, SPOT5, and Orbview is sixteen times larger than the respective MSimages. As a result, if a sensor collected high-resolution multispectral data, it could acquire fewer images during every pass.
Considering these limitations, it is clear that the most effective solution for providing high-spatial-resolution and high-spectral-resolution remote sensing images is to develop effective image fusion techniques.
The principal interest of fusing multi-resolution image data is to create composite images of enhanced interpretabil-ity (Welch and Ehlers, 1987; Kaczynski et al., 1995). The images should have the highest possible spatial information content while still preserving good spectral information quality (Cliché et al., 1985). Some authors stress the idea that the merging method used should not distort the spectral characteristics of the original MSdata, ensuring that targets, which are spectrally separable in the original data, are still separable in the merged data set (Chavez et al., 1991). Such products not only allow a more accurate delineation of ground features, making them more useful for various applications (Vrabel, 1996), but also are more easily inter-pretable in terms of their original spectral signatures. Gar-guet-Duport et al. (1996) demonstrated that spectral informa-tion preservainforma-tion is particularly well suited in the case of vegetation analysis, and its usefulness in urban mapping applications. Going one-step further, some authors even suggest that fused products with maximal spectral informa-tion preservainforma-tion could ideally simulate MSimages acquired at higher spatial resolutions (Vrabel, 1996; Wald et al., 1997).
Different merging methods have been proposed in the literature; using Principal Component Analysis (Chavez et al., 1991), Intensity-Hue-Saturation (IHS) transforms (Haydn et al., 1982; Carper et al., 1990), Brovey Transform
University of Athens, Department of Geology & Geoenvironment, Remote Sensing Laboratory, 1, Iroon Polytechniou Street, 15127 Melissia, Greece ([email protected])
Photogrammetric Engineering & Remote Sensing Vol. 74, No. 5, May 2008, pp. 647–659.
0099-1112/08/7405–0647/$3.00/0 © 2008 American Society for Photogrammetry and Remote Sensing
(Gillespie et al., 1987), Multiplicative Transform (Crippen, 1989), Wavelet Transform (Li et al., 1995; Garguet-Duport et al., 1996; Yocky, 1995 and 1996; Zhou et al., 1998; Fanelli et al., 2001), a statistics based fusion, currently implemented in the PCI Geomatica®software as special
module, named Pansharp, (Zhang, 2002), Back Propagated Neural Networks (Del Carmen-Valdes and Inamura, 2001), High Pass Filters (HPF) (Schowengerdt, 1980), Smoothing Filters (Liu, 2000), or Local Mean Matching Method (De Béthune et al., 1998). Recently, a Modified IHSwas proposed (Siddiqui, 2003). Some of these methods have been compared for the fusion of ETM PAN and MSdata (Vaiopoulos et al., 2001; Nikolakopoulos, 2003).
In this paper, nine different fusion algorithms, IHS, Modified IHS, PCA, Pansharp, Wavelet, LMM(Local Mean Matching), LMVM(Local Mean and Variance Matching), Brovey and Multiplicative, were applied to QuickBird data set in order to assess the quality of the fused products.
The data set corresponds to a (1000 pixel 1000 pixel) portion of a QuickBird PANimage (0.7 m resolution) and four QuickBird MSchannels (2.8 m resolution). The final fused images have 0.7 m spatial resolution, and they are produced by one of the nine different fusion techniques that were previously mentioned. The nearest neighbor and the cubic convolution resampling methods were applied.
In order to quantitatively measure the quality of the fused images, we have made the following controls. First, we have examined the visual qualitative result. Then, we examined the correlation between the original MSand the fused images and all the statistical parameters of the histograms of the various frequency bands. Finally, we performed an unsuper-vised classification, and we compared the result images.
Fusion Techniques Used in this Study
Since the launch of SPOT1(providing high-resolution (10 m) PANimages and low-resolution (20 m) MSimages) in 1986, many fusion algorithms have been presented. Some are very simple, based on algebra functions, and others are quite sophisticated based on wavelet theory.
The simplest fusion technique used in this study is the Multiplicative. This method applies a simple multi-plicative algorithm, which integrates the two-raster images. As it is computationally simple, it is generally the fastest method and requires the least system resources. However, the resulting merged image does not retain the radiometry of the input multispectral image. Instead, the intensity component is increased, making this technique good for highlighting urban features (which tend to be higher reflecting components in an image).
The basic procedure of the Brovey Transform first multiplies each MSband by the high-resolution PANband, and then divides each product by the sum of the MSbands. The Brovey Transform was developed to visually increase contrast in the low and high ends of an images histogram (i.e., to provide contrast in shadows, water, and high reflectance areas such as urban features). Consequently, the Brovey Transform should not be used if preserving the original scene radiometry is important. However, it is good for producing RGBimages with a higher degree of contrast in the low and high ends of the image histogram and for producing “visually appealing” images. Since the Brovey Transform is intended to produce RGB images, only three bands at a time should be merged from the input multi-spectral scene (ERDASImagine®, 2003).
In the IHSfusion, an RGBcolor composite of bands (or band derivatives) such as ratio is transformed into Intensity-Hue-Saturation color space. The intensity component is replaced by the PAN image, the hue and saturation bands are
resampled to the high-resolution pixel size using a nearest neighbor, bilinear, or cubic convolution technique, and the scene is reverse transformed. The technique integrally merges the two data sets (Parcharidis et al., 2001; ENVI, 2004)
The PCAtransform converts intercorrelated MSbands into a new set of uncorrelated components. The first compo-nent also resembles a PANimage. It is, therefore, replaced by a high-resolution PANfor the fusion. The PANimage is fused into the low-resolution MSbands by performing a reverse PCAtransform (Zhang, 2004). Modified IHSwas proposed by (Siddiqui, 2003). The modified IHSmethod is a vast improvement over traditional IHSfor fusing satellite imagery that differs noticeably in spectral response. The modified HISmethod was designed to produce an output that approximates the spectral characteristics of the input MSbands while preserving the spatial integrity of the PAN data. The technique works by assessing the spectral overlap between each MSband and the high-resolution PANband and weighting the merge based on these relative wave-lengths. Therefore, it works best when merging images (and bands) where there is significant overlap of the wavelengths. As such, it may not produce good results when merging SAR imagery with optical imagery, for example.
The LMM(Local Mean Matching) and the LMVM(Local Mean and Variance Matching), methods were specifically designed in order to minimize the difference between the fused image and the low-resolution MSchannels (De Béthune et al., 1997), hence to preserve most of the original spectral information of the low-resolution channels. These filters apply normalization functions (Joly, 1986) at a local scale within the images in order to match the local mean and/or local mean and variance values of the PANimage with those of the original low-resolution spectral channel. The small residual differences remaining correspond to the high spatial information stemming from the high-resolution PANimage. This type of filtering drastically increases the correlation between the fused product and the low-resolution channel. By adjusting the filtering window sizes, we can control the amount of spectral information preserved in the fused product.
The wavelet algorithm used is a modification of the work of (King et al., 2001) with extensive input from (Lemeshewsky, 1999 and 2002). Fusing information from several sensors into one composite image can take place on four levels: signal, pixel, feature, and symbolic. This algorithm works at the pixel level. The results of pixel level fusion are primarily for presentation to a human observer/analyst (Rockinger and Fechner, 1998). However, in the case of PAN/multi-spectral image sharpening, it must be considered that computer-based analysis (e.g., supervised classification) could be a logical follow-on. Thus, it is vital that the algorithm preserve the spectral fidelity of the input dataset. In this wavelet fusion, the high-resolution image is first decomposed through several iterations until a low-resolution low-pass image is generated plus all the corresponding high-pass images derived during the recursive decomposition. This low-resolution, low-pass image, derived from the original PAN image, can be replaced with the low-resolution MSimage, and the whole wavelet decomposition process reversed using the high-pass images derived during the decomposition to reconstruct a high-resolution multi-spectral image. The approximation component of the high spectral resolution image and the horizontal, vertical, and diagonal components of the high spatial resolution image are fused into a new output image. If all of the above calcula-tions are done in a mathematically rigorously way (histom-atch and resample before substitution, etc.) one can derive aMSimage that has the high-pass (high frequency) details from PANimage (ERDASImagine, 2003).
A new statistics based fusion called Pansharp was presented by (Zhang, 2002). This statistics-based fusion technique solves the two major problems in image fusion: color distortion and operator (or dataset) dependency. It is different from existing image fusion techniques in two principle ways: (a) it utilizes the least squares technique to find the best fit between the grey values of the image bands being fused and to adjust the contribution of individual bands to the fusion result to reduce the color distortion, and (b) It employs a set of statistic approaches to estimate the grey value relationship between all the input bands to eliminate the problem of dataset dependency (i.e., reduce the influence of dataset variation) and to automate the fusion process.
As the Brovey transform and the IHStransform produce RGBimages with only three bands, we have used different RGBcombinations (3,2,1; 4,3,2; 4,2,1) of the original MS bands. The modified HISalgorithm has solved this problem. Since this transform relies on the RGB to IHSalgorithm that can only process three bands at a time, we have to select in groups of three to get the bands processed that we need in the output image. So in order to merge a four-band QuickBird scene, we had to merge 3,2,1 and 4,3,2; thereby, covering all four bands in two iterations. The final fused image has four bands the same as the original MSimage.
Results
Visual ComparisonAs already mentioned in this study, quality was evaluated both qualitatively and quantitatively. Visual comparison of all the possible band combinations of the fused images was used for the qualitative assessment, since it is the most simple but effective tool for showing the major advantages and disadvantages of a method. Natural color and color-infrared RGBcomposites are presented for the original MS and all the fused images in order to facilitate the visual evaluation.
The resolution of all the fused images is improved in comparison with the original MSdata (Figure 1a and 1b), and it is comparable to the resolution of the original PAN data (Figure 2).
As we can observe at Figures 3 and 4, the Multiplicative algorithm improves the spatial resolution of the MSdata. We can now distinguish the cars and locate the edges of the buildings. We can also distinguish the trees as they are characteristically observed in the middle of the image. The Figure 1. (a) A RGB 3,2,1 combination of the original multispectral image with 2.8 m resolution, and
(b) A RGB 4,3,2 combination of the original multispectral image with 2.8 m resolution. A color version of this figure is available at the ASPRSwebsite: www.asprs.org.
Figure 2. The original panchromatic image with 0.7 m resolution.
Figure 4. An infrared color RGB (4,3,2) combina-tion fused image with the Multiplicative trans-form. A color version of this figure is available at the ASPRSwebsite: www.asprs.org.
algorithm changes the colors of the 3,2,1 RGBcombination (Figure 3). The color of the trees has changed from green-brown to blue. In contrary it doesn’t change the colors of the 4,3,2 RGBcombination (Figure 4), but it makes the colors darker.
The Brovey transform (Figures 5 and 6) has similar results. It ameliorates the resolution but also changes the colors. It is remarkable that the red color of the trees has changed to blue in the 3,2,1 RGBcombination (Figure 5). Similarly in the 4,3,2 RGB combination, the vegetation is represented with a red-violet color instead of red (Figure 6).
The classical IHStransform (Figures 7 and 8) improves the resolution of the fused image. It causes the same effects Figure 3. A natural color RGB(3,2,1)
combina-tion fused image with the Multiplicative trans-form. The resolution of the fused image is improved in comparison to the original multispec-tral data. There is a significant change at the colors. A color version of this figure is available at the ASPRSwebsite: www.asprs.org.
Figure 5. Fused image with the Brovey transform (3,2,1 RGB combination). The algorithm amelio-rates the resolution but causes some changes to the colors. It is remarkable that the red color of the trees has changed to blue. A color version of this figure is available at the ASPRSwebsite: www.asprs.org.
Figure 6. Fused image with the Brovey transform (4,3,2 RGBcombination). The algorithm ameliorates the resolution but causes some changes to the colors. The vegetation is represented with a red-violet color instead of red. A color version of this figure is available at the ASPRSwebsite: www.asprs.org.
Figure 8. Fused image with the IHStransform (4,3,2 RGB combination). The red color of the vegetation has reduced sharpness in comparison to the original MSimage. A color version of this figure is available at the ASPRSwebsite: www.asprs.org.
at the image colors with the Brovey transform when the 3,2,1 RGBcombination is used (Figure 7). It is also remarkable that the red color of the vegetation has reduced sharpness in the 4,3,2 RGBcombination (Figure 8).
In Figures 9 and 10, the LMMfused image (natural color and color infrared respectively) are presented. The colors of the LMMare a little lighter than the colors of the original MSimage in the 3,2,1 RGBcombination (Figure 9). Generally,
the LMMalgorithm doesn’t change the original colors in all the possible RGBcombinations.
The LMVMfusion algorithm has similar results in improving the resolution as the LMMalgorithm. As we can observe in Figures 11 and 12, the trees and the cars can be distinguished. The colors of the original image remain invariable with all the RGB combinations when the LMVMalgorithm is used for the fusion. Figure 7. Fused image with the IHStransform
(3,2,1 RGB combination). It is remarkable that the red color of the vegetation has changed to blue. A color version of this figure is available at the ASPRSwebsite: www.asprs.org.
Figure 9. Fused image with the LMM transform (3,2,1 RGBcombination). The colors of the LMM are somewhat lighter than the colors of the original MSimage. A color version of this figure is available at the ASPRSwebsite: www.asprs.org.
Figure 10. Fused image with the LMM transform (4,3,2 RGBcombination). No significant changes in comparison to the original MS image. A color version of this figure is available at the ASPRS website: www.asprs.org.
Figure 12. Fused image with the LMVMtransform (4,3,2 RGB combination). The trees and the cars can be easily distinguished. The colors of the original image remain invariable. A color version of this figure is available at the ASPRSwebsite: www.asprs.org.
The modified IHStransform (Figures 13 and 14) also improves the resolution of the fused image. It changes the image colors when the 3,2,1 RGBcombination is used (Figure 13). The green color of the vegetation has changed to blue. In contrary, when the bands 4, 3, and 2 are used in an infrared combination the algorithm preserve the colors of the original image as is shown in Figure 14.
The PCAfusion algorithm (Figures 15 and 16) facilitates the target detection as it ameliorates the resolution. It also preserves the original colors in all the possible combina-tions. The colors seem to be a little lighter in comparison with the colors of the original MSimage.
The Pansharp transform (Figures 17 and 18) also preserves the original colors in all the possible band combinations. Figure 11. Fused image with the LMVMtransform
(3,2,1 RGB combination). The trees and the cars can be easily distinguished. The colors of the original image remain invariable. A color version of this figure is available at the ASPRSwebsite: www.asprs.org.
Figure 13. Fused image with the modified IHS transform (3,2,1 RGBcombination). The green color of the vegetation has changed to blue. A color version of this figure is available at the ASPRSwebsite: www.asprs.org.
Figure 14. Fused image with the modified IHS transform (4,3,2 RGB combination). The algorithm preserves the colors of the original image. A color version of this figure is available at the ASPRSwebsite: www.asprs.org.
Figure 16. Fused image with the PCAtransform (4,3,2 RGB combination). The colors seem to be a somewhat lighter in comparison with the colors of the original MSimage. A color version of this figure is available at the ASPRSwebsite: www.asprs.org.
Generally, it gives very good visual results, but the colors are little lighter in comparison with the colors of the original MSimage.
The Wavelet transform fusion technique (Figures 19 and 20) keeps exactly the same colors with the original MSimage. But it seems to present some distortions problems (Figure 21). These distortion problems do not
appear when we fused ETM PANand MSdata (Nikolakopoulos, 2004).
As mentioned earlier, we have used both the nearest neighborhood and the cubic convolution resampling methods during the fusion of the PANwith the MSdata. During the first visual comparison we did not notice any Figure 15. Fused image with the PCAtransform
(3,2,1 RGB combination). The PCAfusion algorithm facilitates the target detection as it ameliorates the resolution. The colors seem to be a some-what lighter in comparison with the colors of the original MSimage. A color version of this figure is available at the ASPRS website: www.asprs.org.
Figure 17. Fused image with the Pansharp transform (3,2,1 RGB combination). In this RGB combination, the colors are somewhat lighter in comparison with the colors of the original MS image. A color version of this figure is available at the ASPRSwebsite: www.asprs.org.
Figure 18. Fused image with the Pansharp transform (4,3,2 RGBcombination). In this RGB combination the colors are somewhat lighter in comparison with the colors of the original MS image. A color version of this figure is available at the ASPRSwebsite: www.asprs.org.
Figure 20. Fused image with the Wavelet transform (4,3,2 RGB combination). The fusion algorithm preserves the original colors exactly, but seems to present some distortions problems. A color version of this figure is available at the ASPRSwebsite: www.asprs.org.
significant difference in the fused images according to the resampling method. In Figure 22, the fused images with the LMVMand the Brovey algorithm are presented at a scale 1:1 000 instead of the suggested scale for mapping with Quick-Bird orthorectified images that ranges between 1:2 500 and 1:3 000. As it can be observed, the resampling method does not provoke any change at all to the fused images. Only with the use of the IHSalgorithm and with a 4,2,1 RGB combination, the resampling method seems to affect the result image. As we can see in Figure 23a, the cubic convolution resampling method gives a very poor result in comparison with the nearest neighbor resampling method
that gives quite good results (Figure 23b). It should be men-tioned here that it is difficult to explain this phenomenon; Figure 19. Fused image with the Wavelet
transform (3,2,1 RGB combination). The fusion algorithm preserves the original colors exactly, but seems to present some distortions problems. A color version of this figure is available at the ASPRSwebsite: www.asprs.org.
Figure 21. Distortions problems provoked by the Wavelet transform fusion technique. A color version of this figure is available at the ASPRSwebsite: www.asprs.org.
Figure 22. The fused images with the LMVMand the Brovey algorithm at a scale 1:1 000 instead of the suggested scale for mapping with QuickBird orthorecti-fied images that ranges between 1:2 500 and 1:3 000. As can be observed, the resampling method does not provoke any change at all to the fused images. A color version of this figure is available at the ASPRSwebsite: www.asprs.org.
the same procedure (of fusion with the IHSalgorithm using the 4,2,1 RGB combination) was repeated three times using three different software (ERDASImagine®8.7,ERDASImagine®
9.0, and ENVI4.0). Statistical Control
Correlation Coefficient
The closeness between two images can be quantified in terms of the correlation function. The correlation coefficient ranges from 1 to 1. A correlation coefficient value of 1 indicates that the two images are highly correlated, i.e., very close to one another. A correlation coefficient of 1 indi-cates that the two images are exactly opposite to each other.
Each band of the original MSimage has been correlated with the respective nine fused bands. Correlation coeffi-cients were computed with the results shown in Figure 24. Also, the correlation between each band of the MSimage before and after the application of the fusion techniques was computed (Figure 25). The best spectral information is available in the MSimage, and hence, the fused image bands should have a correlation closer to that between the MS image bands. The spectral quality of the fused image is good if the correlation values are closer to each other.
The LMVMand the LMMfusion techniques create bands with a very high correlation of approximately 0.90. It is remarkable that the second and the third bands present almost equal correlation coefficients. The first and the fourth bands present reverse results. The first LMVMband has a very high correlation (0.90) with the first original MSband and the fourth LMVMband having a little lower correlation (0.85) with the fourth original MSband. The respective coefficients for the LMMbands are 0.85 and 0.91.
The IHSand the Brovey techniques create bands with the lowest correlation. The bands created with the Pansharp and the PCAalgorithms present a medium correlation that ranges between 0.77 and 0.79.
The correlation of Modified IHSbands present high fluctuation. The first band has the second lower value of 0.65, and the second band presents a medium value of 0.79. The values of the third and the fourth band are high 0.86
and 0.83, respectively. In contrary, the correlation of all the Multiplicative bands is high and quite stable around 0.88.
Additional correlation between each band of the MS image before and after the application of the fusion tech-niques was computed and the results are interesting:
• The Multiplicative algorithm increases the correlation between the MSbands (Figure 25). It is remarkable that the correlation between the first and the fourth band passes from 0.78 (original MSbands) to 0.96 (fused MSbands).
• The Brovey and the IHSalgorithms notably decrease the correlation between the first and the third band. The correlation value passes from 0.96 (original MSbands) to 093 and 0.94 (fused MSbands), respectively. Figure 23. Fused image with the IHStransform (4,2,1 RGBcombination): (a) the cubic
convolution resampling method gives a very poor result in comparison with (b) the nearest neighbor resampling method that gives quite good results. A color version of this figure is available at the ASPRSwebsite: www.asprs.org.
Figure 24. Each band of the original MSimage has been correlated with the respective nine fused bands. Correlation coefficients were computed and presented in this diagram. A color version of this figure is available at the ASPRSwebsite: www.asprs.org.
(a) (b)
Figure 26. The minimum value of all the bands of the original MSand the fused images. A color version of this figure is available at the ASPRSwebsite: www.asprs.org.
• The Pansharp, the LMM, the LMVM, and the Wavelet algorithm increase the correlation value between the first and the fourth band. The correlation value between the first and the third band presents small fluctuation of 0.010. • The Modified IHSresults in important decreases to the
correla-tion values of all the bands after the fusion. In contrary, the PCAalgorithm results in increases to all the correlation values.
Histogram Statistics
For all the images, the statistical parameters of the histogram and especially the minimum value (Figure 26), the maximum value (Figure 27), and the standard deviation (Figure 28) were studied. The statistical control is necessary in order to examine spectral information preservation. Thus, the mini-mum, mean, and maximum values are usually examined. As the value of the standard deviation is correlated with the possibility to recognize different unities, it is necessary to examine closely any significant decreases that may be the result of the fusion. In general, the preservation of the spectral fidelity of the original MSdata is extremely impor-tant when the researcher wants to proceed to further process-ing of the data. The use of band ratios (e.g., vegetation
indexes) or the classification using spectral libraries pre-suppose the spectral and consequently statistical integrity of the fused data.
The LMM, the LMVM, the Pansharp, and the Wavelet merging technique do not induce major changes to the statistical parame-ters of the original images. The Modified IHSresults in minor changes to the statistical parameters more than the classical IHS or than the PCA. The Brovey and the Multiplicative algorithms result in the major changes to the statistical parameters.
All the possible combination of the Brovey transform algorithm eliminated the minimum values of all the MSbands, and the maximum values of all the MSbands are also
Figure 25. The correlation between each band of the MSimage before and after the application of the fusion techniques was computed. Those correlation coefficients are presented in this diagram. A color version of this figure is available at the ASPRSwebsite:
www.asprs.org. Figure 27. The maximum value of all the bands of the
original MS and the fused images. A color version of this figure is available at the ASPRSwebsite: www.asprs.org.
Figure 28. The standard deviation value of all the bands of the original MSand the fused images. A color version of this figure is available at the ASPRS website:
www.asprs.org.
decreased. There is also an important decrease to the values of the mean value and of the standard deviation with all the RGB band combinations. The resampling method does not appear to affect the statistics when we use the Brovey algorithm.
The Multiplicative algorithm stretches the histogram of all the MSbands, i.e., the minimum values became 0 and the maximum 2,047. All the other statistics values and especially the mean and the standard deviation decrease significantly. The choice of the resampling method does not have any important effect to the statistics.
The LMMtechnique results an increase to the maximum values of all the MSbands and decreases the minimum values to 1 or 2. The mean values of all the MSbands present very small changes that range between 1 and 3 units. The standard deviation values of the first three bands increase a little, and the value of the fourth band decrease by 0.7.
The LMVMalgorithm technique increases the maximum value of the first band and decreases the maximum values of the other three MSbands. It also decreases the minimum values of the third and the fourth band to 1. The minimum value of the first band is decreased from 106 to 19, and the minimum value of the second band is decreased from 100 to 24. The mean values of all the MSbands present very small changes lower than 1 unit. The standard deviation values stay almost invariable. We discovered that the LMM and LMVMalgorithms present exactly the same statistics irrespective of the resampling method that we used.
The Wavelet technique provokes a small decrease to the maximum values of the second, the third and the fourth MS bands. The image produced by the wavelet method presents exactly the same minimum values with the original MSimage for all the bands. The standard deviation values do not appear to change. For example, the standard deviation of the second band decreases from 110.5 (original MSimage) to 110.3 (fused image).
The Pansharp algorithm decreases both the minimum and the maximum values of all the MSbands. The decrease of the maximum value of the first band is small, while the decrease of the rest three bands is quite large. The values of the standard deviation present very small changes that range between 0.8 and 2.1 units.
All the possible combinations of the IHSalgorithm stretch the histogram of all the MSbands. The values of all the spectral bands range from 1 to 2,047. There is also an increase to the values of the standard deviation when we use the 3,2,1 RGBband combination. The use of the cubic convolution resampling method with the 4,3,2 and the 4,2,1 RGBcombinations provokes a very important decrease to the resulting values of the standard deviation. In contrary, the use of the nearest neighborhood resampling method with the 4,3,2 and the 4,2,1 RGBcombinations provokes an increase to the values of the standard deviation.
We note that different RGBcombinations of the initial MS bands provoke different results to the statistics of the fused bands. For example, the mean value of the second MSband changes from 254.2 (original image) to 292.7 (3,2,1 RGB combi-nation) or to 266.2 (4,3,2 RGBcombination) or to 280 (4,2,1 RGB combination) using the nearest neighbor resampling method.
The Modified IHS also provokes a small increase to the values of the standard deviation of the first and the fourth MSbands. In contrary, there is small decrease to the standard deviation of the second and the third MSbands. The Modi-fied IHSalgorithm decreases the minimum values of the first two bands independently of the resampling method used, and there is also a small decrease to all the mean values.
The PCAalgorithm stretches the histogram of all the MS bands. It decreases the minimum values, and increases the maximum values. It also increases all of the mean and the standard deviation values of all the MSbands independently of the resampling method used.
Unsupervised Classification Control
After the visual and the statistical comparison, the nine fusion techniques were evaluated by classifying the fused image bands using an unsupervised clustering algorithm. The fused images were classified in the following three basic categories: buildings, roads, and vegetation; these three are the dominant categories of the image. Further, as in the original multispectral image, the vehicles cannot be recog-nized; it was decided not to discriminate them from the road.
The Iterative Self-Organizing Data Analysis Technique (ISODATA) (Tou and Gonzalez, 1974) clustering method has been used (ERDAS, 2003). It is iterative in that it repeat-edly performs an entire classification (outputting a thematic raster layer) and re-calculates statistics. “Self-Organizing” refers to the way in which it locates the clusters that are inherent in the data. The ISODATAclustering method uses the minimum spectral distance formula to form clusters. It begins with either arbitrary cluster means or the means of an existing signature set, and each time the clustering repeats, the means of these clusters are shifted. The new cluster means are used for the next iteration. The ISODATAutility repeats the cluster-ing of the image until either (a) a maximum number of itera-tions have been performed, or (b) a maximum percentage of unchanged pixels have been reached between two iterations.
It is important to note that this control does not indicate classification accuracy, as the classification is not compared with any ground truth. This control is used to indicate the classification changes in the fused image bands compared to the corresponding MSbands.
In all the images, the result was not as good as expected because parts of the original image are in the shadow (Figure 1). As expected, the classification of the fused image with the Multiplicative algorithm gave the worst results (Figure 29). The Pansharp (Figure 30) and the Modified IHSimages gave the best results in classification. The 4,3,2 RGB combina-tion of the IHS(Figure 31) and the Brovey gave quite good results; all the other images gave similar quite good results.
Figure 29. The unsupervised classification result of the fused image with the Multiplicative algorithm which provided the worst results.
Figure 30. The unsupervised classification result of the fused image with the Pansharp algorithm which provided the best results.
Conclusions
In this paper, nine different fusion algorithms, IHS, Modified IHS, PCA, Pansharp, Wavelet, LMM(Local Mean Matching), LMVM(Local Mean and Variance Matching), Brovey, and Multiplicative, were compared for the fusion of QuickBird PANand MSdata.
All the fusion techniques improve the resolution and the visual result. Even the Multiplicative that is the simplest fusion technique ameliorate the detection of small targets like cars and trees, and facilitate the mapping of the buildings.
The LMVM, the LMM, the Pansharp, the Wavelet, and the PCAfusion techniques preserve, in general, the original colors in all the possible RGBband combinations. Small differences are detected in the tonality and the sharpness of the colors. In contrast the IHS, the Brovey, and the Multi-plicative fusion techniques cause changes in the colors of the original images and make the photo-interpretation more difficult. Thus, the color of the vegetation changes from green to blue when the blue band is used in natural color combinations. The influence of the modified IHSof the original colors depends on the different RGBband combina-tions. The wavelet algorithm causes small distortion prob-lems. In general, with the use of the 4,3,2 RGBcombination, all the fusion algorithms resulted in minor changes to the colors of the original MSimage.
For the eight of the nine compared algorithms, there was not any significant difference in the fused images according to the resampling method. Only with the use of the IHSalgorithm and with a 4,2,1 RGBcombination, the resampling method seems to affect the result image. In that case, the cubic convo-lution resampling method gives the least desirable result.
The correlation coefficient was computed for different band combinations of the fused images. First, each band of the original MSimage has been correlated with the respective nine fused bands. The correlation of the fused bands should be close to that of the original MSimage to ensure good spectral quality. The values in Figure 24 indicate that the LMVMand the LMMmethods produce the best correlation result. The Brovey and the IHSmethods gave the worst results. The correlation of the IHSbands presents a high fluctuation.
Also, the correlation between each band of the MSimage before and after the application of the fusion techniques was computed. Using the Pansharp, the LMM, the LMVM, the PCA, and the Wavelet algorithm, the correlation value between the first and the fourth band was increased proving the good spectral quality of the fused image.
The LMVM, the LMM, the Pansharp, and the Wavelet, more or less, do not change the statistical parameters of the original MSbands. The Modified IHSprovokes minor changes to the statistical parameters than the classical IHSor the PCA. The Multiplicative algorithm stretches the histogram of all the MSbands and decreases the standard deviation values. The Brovey algorithm causes an important decrease to the values of the mean value and of the stan-dard deviation with all the RGBband combination. It is remarkable that the LMMand LMVMalgorithms present exactly the same statistics irrespective of the resampling method used.
During the unsupervised classification, the eight of the nine algorithms give quite good results. Only the multiplicative algorithm gave unacceptable results.
After all the controls, the LMVM, the Pansharp, and the LMMalgorithms seem to gather the more advantages in fusion panchromatic and multispectral data. These algorithms are proposed if the researcher want to pro-ceed to further processing using for example vegetation indexes or to perform classification using the spectral signatures.
References
Carper, W.J., T.M. Lillesand, and R.W. Kiefer, 1990. The use of Intensity-Hue-Saturation Transform for merging SPOT panchro-matic and multi-spectral image data, Photogrammetric Engineering & Remote Sensing, 56(4):459–467.
Figure 31. The unsupervised classification result of the fused image with the IHSalgorithm.
Chavez, P.S. Jr., S.C. Sides, and J.A. Anderson, 1991. Comparison of three different methods to merge multiresolution and multispec-tral data: Landsat TM and SPOT panchromatic, Photogrammetric Engineering & Remote Sensing, 57(3):295–303.
Cliche, G., F. Bonn, and P. Teillet, 1985. Integration of the SPOT panchromatic channel into its multispectral mode for image sharpness enhancement, Photogrammetric Engineering & Remote Sensing, 51(3):311–316.
Crippen, R.E., 1989. A simple spatial filtering routine for the cosmetic removal of scan-line noise from Landsat TM P-tape imagery, Photogrammetric Engineering & Remote Sensing, 55(3):327–331.
De Bethune, S., F. Muller, and M. Binard, 1997. Adaptive intensity matching filters: A new tool for multiresolution data fusion, Proceedings of the AGARD Conference 595, Multi-Sensor Systems and Data Fusion for Telecommunications, Remote Sensing and Radar, Lisbon, 29 September-02 October, pp. 28:1–28:15. De Bethune, S., F. Muller, and J.P. Donnay, 1998. Fusion of
multi-spectral and panchromatic images by local mean and variance matching filtering techniques, Proceedings of the Second International Conference – Fusion of Earth Data Merging Point Measurements, Raster Maps and Remotely Sensed Images (Thierry Ranchin and Lucien Wald, editors), EARSeL-SEE-Ecole des Mines de Paris, 28–30 January, pp. 31–36.
Del Carmen-Valdes, M., and M. Inamura, 2001. Improvement of remotely sensed low spectral resolution images by backpro-pagated neural networks using data fusion techniques, Inter-national Journal of Remote Sensing, 22:629–642.
ENVI, 2004. ENVI Users Guide, Version 4.1, Research Systems, Inc., 1150 p.
ERDAS, 2003. ERDAS Field Guide, Seventh edition, Leica Geosys-tems, Atlanta, Georgia, 698 p.
Fanelli, A., A. Leo, and M. Ferri, 2001. Remote sensing images data fusion: A wavelet transform approach for urban analysis, Proceedings of the IEEE/ISPRS Joint Workshop on Remote Sensing and Data Fusion over Urban Areas, Rome, 08–09 November, pp. 112–116.
Garguet-Duport, B., J. Girel, J. Chassery, and G. Pautou, 1996. The use of multiresolution analysis and wavelets transform for merging SPOT panchromatic and multispectral image data, Photogrammetric Engineering & Remote Sensing,
62(10):1057–1066.
Gillespie, A.R., A.B. Kahle, and R.E. Walker, 1987. Color enhance-ment of highly correlated images-II: Channel ratio and “chro-maticity” transformation techniques, Remote Sensing of Environment 22:343–365.
Haydn, R., G.W. Dalke, J. Henkel, and J.E. Bare, 1982. Applications of the IHS color transform to the processing of multisensor data and image enhancement, Proceedings of the International Symposium of Remote Sensing of Arid and Semiarid Lands, Cairo, Egypt, pp. 599–616.
Joly, G., 1986. Traitements des fichiers images, Collection Télédétec-tion satellitaire, No. 3, Paradigme, Caen, 137 p.
Kaczynski, R., J.P., Donnay, and F. Muller, 1995. Satellite image maps of Warsaw in the scale 1:25,000, EARSeL, Advances in Remote Sensing, Topography from Space, 4(2):100–103. King, R., and J. Wang, 2001. A wavelet based algorithm for pan
sharpening Landsat-7 imagery, Proceedings of the Geoscience and Remote Sensing Symposium, IGARSS, 2:849–851. Lemeshewsky, G.P, 1999. Multispectral multisensor image fusion
using wavelet transforms, in visual image processing VIII (S.K. Park and R. Juday, editors), Proceedings of SPIE, 3716:214–222. Lemeshewsky, G.P, 2002. Multispectral image sharpening using a
shift-invariant wavelet transform and adaptive processing of multiresolution edges in visual information processing XI
(Z. Rahman and R.A. Schowengerdt, editors), Proceedings of SPIE, Vol. 4736.
Li, H., B.S. Manjunath, and S.K. Mitra, 1995. Multisensor image fusion using the wavelet transform, Graphical Models and Image Processing, 57:234–245.
Liu, J.G., 2000. Smoothing filter-based intensity modulation: A spectral preserve image fusion technique for improving spatial details, International Journal of Remote Sensing, 21(18):3461–3472. Nikolakopoulos, K., 2003. Comparative study of fusing ETM data
with five different techniques for the broader area of Pyrgos, Greece, 08–12 September, Barcelona, Spain, Proceedings of SPIE, 5238:84–95.
Nikolakopoulos, K., 2004. Pansharp vs wavelet vs PCA fusion technique for use with Landsat ETM panchromatic and multispectral data, 13–16 September, Gran Canaria, Spain, Proceedings of SPIE, 5573:30–40.
Parcharidis, I., N. Konstantinos, S. Konstantinos, and I. Baskoutas, 2001. Synergistic use of optical and radar data for active faults and corresponding displaced landforms detection in Kozani Basin (Greece), Geocarto International, A Multi-disciplinary Journal of Remote Sensing and GIS,16(3):17–23.
Rockinger, O., and T. Fechner, 1998. Pixel-level image fusion, Signal Processing, Sensor Fusion and Target Recognition (I. Kadar, editor), Proceedings of SPIE, 3374:378–388. Siddiqui, Y., 2003. The modified IHS method for fusing satellite
imagery, Proceedings of the ASPRS 2003 Annual Conference, May, Anchorage, Alaska, unpaginated CD-ROM.
Schowengerdt, R.A., 1980. Reconstruction of multispatial, multi-spectral image data using Spatial frequency content, Photogram-metric Engineering & Remote Sensing, 46(10)1325–1334. Tou, J.T., and R.C. Gonzalez, 1974. Pattern Recognition Principles,
Reading, Massachusetts, Addison-Wesley Publishing Company. Vaiopoulos, D., N. Konstantinos, and G.A. Skianis, 2001, A
com-parative study of resolution merge techniques and their effi-ciency in processing images of urban areas, Proceedings of the IEEE/ISPRS Joint Workshop on Remote Sensing and Data Fusion over Urban Areas, Rome, Italy, 08–09 November, pp. 270–274. Vrabel, J., 1996. Multispectral imagery band sharpening study,
Photogrammetric Engineering & Remote Sensing, 62(9):1075–1083. Wald, L., T. Ranchin, and M. Mangolini, 1997. Fusion of satellite
images of different spatial resolutions: Assessing the quality of resulting images, Photogrammetric Engineering & Remote Sensing, 63(6):691–699.
Welch, R., and M. Ehlers, 1987. Merging multiresolution SPOT HRV and Landsat TM data, Photogrammetric Engineering & Remote Sensing, 53(3):301–303.
Yocky, D.A., 1995. Image merging and data fusion by means of the two-dimensional wavelet transform, Journal of the Optical Society of America, 12(9):1834–1845.
Yocky, D.A., 1996. Multiresolution wavelet decomposition image merger of Landsat Thematic Mapper and SPOT panchromatic data, Photogrammetric Engineering & Remote Sensing, 62(9):1067–1074. Zhang, Y., 2004. Understanding image fusion, Photogrammetric
Engineering & Remote Sensing, 70(6):657–661.
Zhang, Y., 2002. A new automatic approach for effectively fusing Landsat 7 as well as Ikonos images, Proceedings of IEEE/ IGARSS’02, Toronto, Canada, 4:2429–2431.
Zhou, J., D.L. Civco, and J.A. Silander, 1998. A wavelet transform method to merge Landsat TM and SPOT panchromatic data, International Journal of Remote Sensing 19:743–757.
(Received 10 May 2006; accepted 24 May 2006; revised 01 November 2006)