Merging panchromatic multispectral images for enhanced image analysis

Rochester Institute of Technology

RIT Scholar Works

Theses

Thesis/Dissertation Collections

8-1-1990

Merging panchromatic multispectral images for

enhanced image analysis

Curtis K. Munechika

Follow this and additional works at:

http://scholarworks.rit.edu/theses

This Thesis is brought to you for free and open access by the Thesis/Dissertation Collections at RIT Scholar Works. It has been accepted for inclusion in Theses by an authorized administrator of RIT Scholar Works. For more information, please contactritscholarworks@rit.edu.

Recommended Citation

MERGING PANCHROMATIC AND

MULTISPECTRAL IMAGES

FOR

ENHANCED IMAGE ANALYSIS

by:

CURTIS K. MUNECHIKA, Captain, USAF

A thesis submittedinpartial fulfillment of the requirements for the degree of Master of Science

in the Center for Imaging Scienceinthe College of Graphic Arts and Photography at the

Rochester Institute of Technology

August 1990

Signature of the Author ....lo.(...1JLLrtLLiS:>...JKL>...l.lMUlI..u.IO.JJ:e:.Jo...cL1JbiL.Do.ka..a _

Accepted by _

Center for Imaging Science College of Graphic Arts and Photography

Rochester Institute of Technology Rochester, New York

CERTIFICATE OF APPROVAL

M.S. DEGREE THESIS

The M.S. degree thesis of Curtis Munechika has been examined and approved by the thesis committee as satisfactory for the thesis requirement

for the Master of Science degree

Dr. John R. Schott, Thesis Advisor

Lt. Col. Phil Datema

Mr. Carl Salvaggio

d3

406

/0

Center for Imaging Science

College of Graphic Arts and Photography Rochester Institute of Technology

Rochester, New York

THESIS RELEASE PERMISSION FORM

Title of Thesis: Merging Panchromatic and Multispectral Images for Enhanced Image Analysis

I, Curtis K. Munechika grant pennission to the Wallace Memorial Library of the Rochester Institute of Technology to reproduce this thesis in whole or in part

provided any reproduction will not beof commercial use or for profit.

MERGING PANCHROMATIC AND

MULTISPECTRAL IMAGES FOR

ENHANCED IMAGE ANALYSIS

by

Curtis K. Munechika

Submittedto theCenter for_ImagingScience in

partialfulfillmentoftherequirementsforthe

MasterofScience degreeatthe

Rochester Instituteof_Technology

ABSTRACT

This study evaluates severalmethods that enhance the spatial resolution of multispectral

images using afinerresolutionpanchromatic image. Theresultant_hybrid,highresolution, multispectral data set has increased visible interpretation and improved classification

accuracy, while _preserving the _radiometry of the original multispectral images. These

ACKNOWLEDGEMENTS

Thiswork could nothave been completed without the supportof_manypeople. I would first like to thank theUnited StatesAir Force for_allowingmethe _opportunity to _study at

theCenterof_Imaging Science. Theexperiencehas been_{exceptionally}rewarding.

Iwouldalsoliketo thank_mycommittee membersforalltheirassistance:

toDr. JohnSchott,for providing theoverall _{guidance,focus,} and an occasionalkick in the pants;

to Lt Col Phil _{Datema, for his time, patience,} and support in maintaining a _long distance workingrelationship; and

toCarl _{Salvaggio, for answering} allof_mynot-so-quick questions.

Specialthanks alsoto: Jim Warnickwhofirst startedthis _study andwas_{my best sounding} board; toDr. Roger Eastonwhose_companyattheoddesthours was_appreciated; to Steve Schultzwhorescued_myworkfroma_terminallyillcomputer; andtoLaura_Abplanalpand

Leslie Gentherwhohelped in puttingthis together.

Lastly I wish to acknowledge _my comrades-in-arms fortheir _{encouragement, assistance,}

and abuse: Ranjit_{Bhaskar, Betsy Fry,}GopalSundaramoorthy, _WendyRosenblum, Steve

DEDICATION

Thisworkis dedicated to_{my family...}

To myparents,for_instillingmy desiretolearn (and fortheir_{babysitting);}

To mybrother,for his insightandtime (as wellasuseofhis computer);

and_especially

to

Table of Contents

ListofFigures ListofTables

1.0 Introduction 1

1.1 _{Spatial, Spectral,} Radiometric Resolution 2

1.1.1 Resolution and Classification _Accuracy 3

1.1.2 Resolution Trade-offs 5

1.2 Overview on _Merging Images 7

1.2.1 _Merging for _Display 7

1.2.2 _{Merging by}Separate Manipulationof

Spatial Information 9

1.2.3 _Merging and _{Maintaining Radiometry} 10

2.0 _Merging Methods and Modifications 12

2.1 DIRS Method for _Merging 12

2.1.1 Results _Using the DIRS _Merging Method 13

2.1.2 Concerns on the DIRS _Merging Method 13

2.2 General Modifications to the DIRS Method 17

2.2.1 _Creating a TM Panchromatic Image 17

2.2.2 Interpolated TM Input Images 24

2.2.3 A Technique for Radiometric Post-_{Correction 25}

2.3 DIRS Enhancement 1 Method for Merging 27

2.4 DIRS Enhancement 2 Method for _Merging 29

2.4.1 A Modification to DIRS Enhancement 2 Method 36

2.5 Price's Method for _Merging 37

2.5.1 Case 1 - Correlated Input Images 37

2.5.2 Case 2

-Weakly Correlated Bands 40

2.6 Price's Modification ~ A Method

of_Handling

Weakly Correlated Bands 42

3.0 Experimental Approach 46

3.1 Selection and Preparation of Test _Imagery 46

3.1.1 Registration of Images 47

3.1.2 Blurred Image Sets 49

3.1.3 Figures of Images 50

3.2 Classification of Test _Imagery 54

3.2.1 _Training Samples for Classification 54

3.2.2 _Measuring Classification _Accuracy 55

3.2.3 _Testingthe SignificanceofOverall Classification _Accuracy 56

3.3 Radiometric Error Analysis 57

3.4 _Running the Methods and Modifications 59

3.4.1 _Running DIRS Enhancement 2 59

3.5 Experimental Procedure 61

3.5.1 Pan 1

-Merging with Coarser Resolution Data Sets 61 3.5.2 Part 2

-Merging withCoarser Resolution DataSets with a

Lower Registration Error 64

3.5.3 Part 3 _Merging with Original Resolution Data Sets 64

4.0 Results 65

4.1 The New Synthetic Panchromatic Image 65

4.2 _Computing the Error in Reflectance 70

4.3 Results for Part 1

-Merging with Coarser Resolution Data Sets 75 4.3.1 Observations from Part 1

-MergingwithCoarser

4.4 Results for Part 2 _Mergingwith CoarserResolutionDataSets

with a Lower Registration Error 80

4.4.1 Observations from Part 2 _Mergingwith Coarser

Resolution Data Sets withaLowerRegistration Error 84

4.5 Results for Part3

-Merging withOriginal Resolution DataSets 85 4.5.1 Observationsfrom Part3

-Merging with

Original Resolution Data Sets 90

4.6 Other Comparisons Between _Merging Methods 90

4.6.1 Visual Comparisons 90

4.6.2 Implementation Comparisons 99

5.0 Conclusions 101

6.0 Recommendations 103

6.1 Improvements to the Price Methods 103

6.2 Improvements to the DIRS Methods 104

6.2.1 Recommendations in _Processing Correlated Bands 104 6.2.2 Recommendations in _{Handling Weakly} Correlated Bands 106 6.2.3 Improvements to the Interpolated Inputs 107 6.2.4 Improvements to the DIRS Enhancement 2 Method 107

6.3 General Considerations 109

Appendix A Classification Results for Original TM Data

Appendix B Classification Results for Hybrid Data_(30m)using

Blurred TMandSPOT Data

AppendixC- Radiometric Results fortheHybrid Data

(30m)using Blurred TMandSPOT Data

AppendixD Classification Results for Hybrid Data_(30m)using

BlurredTM andRe-registered SPOT Data

Appendix E~ Radiometric Resultsfor_theHybrid Data

(30m)using Blurred TMandRe-registered SPOT Data

Appendix F Classification Results for Hybrid Data(10m)using

TMandRe-registered SPOT Data

AppendixG - Statistical Tests

AppendixH~

Input ParameterstoComputethe_WeightingFactors

foraSynthetic Panchromatic Image

Appendix I

List of Figures

Figure 1-1 Pradine's_MergingMethod

Figure2-1 Superpixeland subpixels

Figure2-2 Reflectance SpectraforSelectedUrbanTargets Figure 2-3 Reflectance SpectraforSelectedSoilTargets Figure2-4 ReflectanceSpectra for SelectedWaterTargets

Figure2-5 Reflectance SpectraforSelected TreesTargets

Figure 2-6 Reflectance SpectraforSelected Grass Targets Figure 2-7 Pixel Replication

Figure 2-8 InterpolatedInput

Figure2-9 DIRS NearestNeighbor Correction Routine Figure2-10 DIRS Enhancement1 Correction Routine

Figure2-_{1 1 Connectionsbetweenthe}

center subpixel

Figure2-12 Flow DiagramofDIRS Enhancement2_MergingMethod Figure 2-_{13 Linear RegressionofTM Bands}

andthePanchromatic Image Figure 2-14 Price's_MergingMethod- Case 1

Figure2-15 Creationof a_Look-upTable _(LUT)

Figure 3-1 Original TM (30m_GIFOV)Bands 5,3,2 in RGB

Figure3-2 Blurred TM (90m_GIFOV)Bands 5,3,2 in RGB Figure 3-3 OriginalSPOT (10m_GIFOV)

Figure4-_{1 PlotofTM 1 DCversusEstimatedReflectance}

Figure 4-2 PlotofTM2 DCversusEstimated Reflectance

Figure 4-3 PlotofTM3 DCversusEstimated Reflectance

Figure4-4 PlotofTM4 DCversusEstimated Reflectance

Figure4-5 ComparisonofCoarser Resolution Images

Figure 4-6 ComparisonofOriginal ResolutionImages

Figure 4-7 Replicated TM Bands 5,3,2 (30mGIFOV) (magnification_4x)

Figure4-8 Interpolated TM Bands 5,3,2 (30m_GIFOV)(magnification_4x) Figure 4-9 DIRS MethodwithInterpolated Input (magnification_4x)

Figure4-10 DIRSMethodwithInterpolatedInputand_Post-fixing(magnification_4x) Figure4-11 DIRSEnhancement 1 with_Post-fixing (magnification_4x)

Figure4-12 DIRSEnhancement2with_TM4,TM5modification(magnification _4x)

Figure4-13 Price's Method _(LUT)(magnification_4x)

List of Tables

Table 1-1

Table 3-1 1

Table3-2 Table3-3

Table 3-4 ]

Table3-5 I Table 4-1

Table 4-2

Table 4-3

Table 4-4 !

Table4-5 I

Table 4-6

Table 4-7a

Table 4-7b I

Table4-7 ! Table 4-8a I

Table4-8b :

Table4-8c

Table 4-8 I

Table 4-9a

Table 4-9b

Table 4-9c !

AComparisonofMultispectral Scanners_(MSS) and

Thematic Mapper_(TM)

SceneAcquisition andEphemeris Data

Scene Statistics

Transformation Coefficients

New Transformation Coefficients SummaryTableof_MergingMethods

Summaryof_WeightingFactors usedtoGenerate theTM Panchromatic Image

SummaryoftheHistogram StatisticsforthePanchromatic Images

SummaryoftheLinearCoefficientsusedfor Adjustment

SummaryoftheError Differences Selected Control Points

Averaged DCandEstimated Reflectance

SummaryofClassification Breakdown _byPercentage (30m_hybrid) SummaryofClassification_Accuracy with

Independent DataSet 1 (30m_hybrid)

SummaryTable for 30m HybridResults

SummaryofClassification Breakdown_byPercentage

(re-registered 30mhybrid)

SummaryofClassification_Accuracywith

IndependentData Set 1 (re-registered 30mhybrid)

SummaryofClassification_Accuracywith

aRandomData Set 1 (re-registered 30mhybrid) SummaryTable for Selected 30m Hybrid Sets usinga

Re-registered SPOT imageasInput

SummaryofClassificationBreakdownbyPercentage (10mhybrid) SummaryofClassification_Accuracywith a

Random DataSet 1 (10m_hybrid)

SummaryofClassification_Accuracywith

Table 4-9d _SummaryofClassification_Accuracywith Independent Data Set 2 (10m_hybrid)

1.0 INTRODUCTION

Spatial resolution is an importantparameter in image interpretation. _However, to

capture multispectral images or images within a narrower spectral _bandpass, spatial

resolution is often diminished. In _general, if a _sensing system has fine spectral discrimination,thenit_{is physically}difficulttoalsohave fine spatial resolution.

The emphasis of this _{study is} to enhance the spatial resolution of multispectral

images using data from another sensor. In particular, this _study evaluates several

techniques tomerge the medium resolutionThematic Mapper_(TM) multispectral images

with a high resolution panchromatic image captured from the SPOT satellite. The

performance ofeach _merging technique is measured _by classification _accuracy on the hybrid _images, as well as on how well the hybrid images maintain the radiometric and

spectralinformationoftheTM data.

The results of this _study show that these _merging techniques can produce high

resolution multispectral images for enhanced image analysis. _{Not only is} visible

interpretation clearly improved, butclassification _{accuracy is}alsoincreased. In addition,

theresultanthybridimages_adequatelymaintainthe TMradiometry.

Fortheremainderofthis_introductorychapter, section 1.1 discusses some concepts

on resolution in theremote _sensing arena. It covers the effects that different types of

1.1 _{Spatial, Spectral,} and Radiometric Resolution

Ingeneral_terms,resolution can bethoughtof as the_abilityofa systemtodistinguish

fine detail. Most often, we think of resolution in terms of "spatial _resolution", orhow

well we can resolve the spatial detail in an image. As such, there are _many ways to

measure or_quantify the spatial resolution of an _imaging system (i.e. _by themodulation

transfer_function, ground resolvable_distance,etc).

For _simplicity, the ground instantaneous field-of-view _(GIFOV) is used in this

study. TheGIFOV is theprojection ofthe_limitingdetector aperture ontothe ground. Itis

similar to the ground sample distance and has units of length. Thus the smaller the GIFOV, the smaller the _{sampling distance} on the ground, and the finer the spatial

resolution ofthe system.

However, in the field of remote sensing, and _especially with multispectral _data,

thereareotherformsofresolutionthatare_equally asimportant.

"Radiometric resolution"

is determined_by thenumberof effective _grey levels that

are availableto the systemtorepresentthescene brightness. Forexample, an8-bitsystem

would be ableto record an image in 256 levelsofbrightness. Its radiometric resolution

wouldbe finerand more precisethan a6-bitsystem which can_onlywork with64 levels.

"Spectral resolution"

can be thought of as the width of the bandpass over which

radiance is measured. The more narrow this wavelength _interval, the finer the spectral

resolution. _Intuitively, thefineryourspectral _resolution,themore spectralbandswe could

more spectralinformation willbeavailable.

1.1.1 Resolution and Classification _Accuracy

Theeffect ofthese three types of resolution on_classifyingimageshas been studied,

particularlysince theintroductionoftheThematic Mapper.

The ThematicMapper_(TM) was launchedaboardLandsat-4 on_{July 16,} 1982. In

comparison with the older Multispectral Scanners (MSS), TM provided finer spatial

resolution, narrower and more _optimally placed spectral _bands, and finer radiometric

precision. Acomparison chartis shownbelow.

Table 1-1

AComparisonofMultispectral Scanners_(MSS)and Thematic_Mappej(TM)

MSS TM

GIFOV

(m) 80

30 (bands_1-5,7)

120 (band₆₎

Spectral

Bandpass

(micron)

0.5 - 0.6 0.6- 0.7 0.7-0.8 0.8-1.1

0.45- 0.52 0.52- 0.60 0.63- 0.69 0.76- 0.90 1.55

- 1 .75 10.40 - 12.50

2.08- 2.35

Quantization Levels 64 (6Ms) 256 (8_bits)

determine their effect on classification accuracy. Even before the TM was _launched,

Sadowski et al _[77] used simulated data to find that classification _accuracy would be

significantlyenhanced with theadditionalTM spectral bandsandtheincreasednumber of quantization levels. _{However, they} alsodiscovered thatclassification _{accuracy actually}

decreased as spatial resolution became finer. These results were corroborated _by

Morgenstern et al _{[77], Latty} and Hoffer _[81], Markham and Townshend [81], and

Williamset al[84].

The reason behind the apparent lack of effect that finer spatial resolution has on

classification is becauseoftwo_offsettingeffects. _Improving spatial resolution will sharpen

boundaries and reduce theamountofmixed pixels in an image. This reduction in mixed

pixels contributesto an increase inclassification accuracy. _{However, improving} spatial

resolution also increases the within-class spectral variance _causing a decrease in

classification _{accuracy [Landgrebe77,} and Irons et al 85]. Thus _depending on the input

image, increasing spatial resolution_mayor_maynot aidincomputerclassification - _even

though the advantages of increased spatial resolution appearobvious when _conducting

manual photointerpretation.

Many of these studies concluded that the commonly-used per-pixel Gaussian

maximumlikelihood_(GML)classifierdidnot_effectivelyusetheadditionalinformationthat

increased spatial resolution provides. _However, in these cases, only spectral bands were

usedasinputto theGMLclassifier.

To better use this information in high spatial resolution _images, current classifiers

have been_followingeitheroftwo_paths,one _usingthetextural features found in theimage

and the other _using context to support classification. For two of the more well-known

papers on _using textureand context forclassification, referto Haralick _[79] and _Gurney

Using these types of classifiers which _apply spatial _information, classification

accuracies_{have increased}with morespatial detail [DiZenzoet al _87,HjortandMohn 84, Warnick et al 89]. Rosenblum _[90] found that the optimal set of input features for

classification ofhighresolutionairphotos containbothspectral andtexturalinformation.

Generally, if a classifier can _effectively use all _{spatial, spectral,} and radiometric

information, classification accuracies can be improved with finer spatial, spectral, and

radiometric resolution.

1.1.2 Resolution Trade-offs

Unfortunately, as_{in manyrelationships,} therearetrade-offsbetween thedifferenttypes

of resolution. Forexample, ifwe wished tohave a systemwith finespatialresolution, it

would be very difficult for this system to _{additionally have} fine spectral resolution. To improvethespatial resolution of our_scanningsystem, we wouldrequirea smallerGIFOV. A smaller GIFOV means the _{energy reaching} the sensor has originated from a smaller

ground area and ifother parametersremain constant-- _this

meansless energy reaching the sensor. Lower levels of_energy means a lower signal level available to the sensor,

resulting in a lower signal-to-noise ratio (SNR). To produce useful output, the sensor mustbe ator above athresholdSNRtodistinguisha signalfromthenoise. Tocompensate thislowerSNR due toa smallerGIFOVwe couldbroaden thespectral bandpassto allow

more_energythrough

-i.e., spectral resolution is sacrificedto compensateforthe gainin

spatial resolution.

Conversely, ifwe wished ourdata tobe captured within a narrower spectral _band,

again _{less energy}wouldbe incidentonthedetector. _Increasing theGIFOV_(losing spatial

acceptableSNRismaintained.

Another trade-off mentioned _by Green _[88] is a simple data _handling problem.

Supposewehad 8Mbitsofdigitalstorage. Wecould _{approximately}store either a:

1000x 1000pixel _image,8_bits/pixel, 1 spectral_band;

or

380x 380pixel_images, 8 _bits/pixel,7 spectral_bands;

or

1000x 1000pixel_images,4bits/pixel,2 spectralbands.

The first system provides high spatial resolution but limited spectral information. The

second example provides a significant amount of spectral _data, but at a low spatial

resolution. The third system provides some spectral resolution athigh spatial _resolution, but at much lower radiometric precision than the first two systems. Other similar data

handlingconstraintsthatcoulddrive informationtrade-offsarelimitations in transmission

and acquisitiontime.

All these system trade-offsare dependent on thecurrent state oftechnology. With

timeandtechnological_advances,information trade-offswillbecomelesssevere. Butgiven

existing data and_existingremote _sensing systems, we can overcomethese current trade

offs _bycombining data from two _{complementary} systems. The resultant hybrid data set

would contain the "best"

1.2 Historical Overview on _Merging Images

Toimprovethe_relatively coarse spatial resolution of multispectral_images,theidea

of _{merging data} from different sensors has been attempted. _Early studies reported

successful _merging with the LandsatMSS images (80 m GEFOV). Radar images from

airborne systems and the Shuttle _Imaging Radar

(SIR-A) have been merged with MSS

images toenhance geologicalinterpretation _{[Daily 79;} Chavez 83]. LauerandTodd [81]

combined MSS and RBV images to obtain a hybrid _product, while Schowengerdt _[82]

mergedMSSwiththeHeat_{Capacity Mapping}Mission_(HCCM)data.

With theintroduction oftheTMand SPOTsensor systemswhich have finerspatial

and spectral _resolutions, further studies on _merging werereported. All ofthese _merging

techniques, as well as those done with.MSS images, could _basically be classifiedinto 3

categories.

1.2.1 _Merging for _Display

Price _[87] coined the first category of _merging techniques as the "ad hoc"

approaches. Since the _primary concern ofthese methods was to optimizeimage _display,

some unusual operations were done on the data. _However, theresultanthybrid images

appeared tohave increased spatial resolution. Welshet al _[87] summarizedthese ad hoc

methods intotwoequations:

M'i = _aj x (m- x p)1/2

+ _{b; (1-1)}

or

where:

M|'

= _thedigital_count

(DC)fora pixelinthei-th bandofthemerged_image;

M-= DC for_the

correspondingpixelinthei-thmultispectral_image;

P = DC for _the

correspondingpanchromatic referenceimagepixel;

(>l ,a>2 = weightingfactors;

a_i ,

bj

= _{operatorwhich couldbe}

addition, subtraction, multiplication, ratio,etc.

Using these methods and simulated SPOT data, Cliche _[85] integrated the

panchromatic channel into the multispectral channels to _{significantly improve} visible

interpretation. Chavez _[84] added edge enhanced _(simulated) SPOT data toTMimages.

Chavez _[86] also mergedTMdatawithadigitizedpanchromatic photograph. Hashim _[88]

simply

"overlaid"

registeredTMandMSS imagestoobtain ahybridproduct.

Currently, many Geographical Information Systems (GIS) which contain layersof

information intheir_database,implementsome sortofdatacombination for_display to the

user [Welsh _85; Walsh et al 87]. Since a high priority of a GIS is to optimize image

display, "ad hoc"

approaches to _merging are _frequently used. One of the more simpler

methodsforanenhancedRGB _displayistoplacethehighresolution panchromatic image

intothe green _channel, and two lowresolution multispectral bands into thered and blue

channels. Since the green channel contributes the most to the _intensity component, the

1.2.2 _{Merging by} separate manipulation of spatial _information

Amultispectralimagecanbethoughtof as_havinga spectral component and a spatial

component. Thesecondtypeof_mergingalgorithmfirsttries toseparate thesecomponents,

then manipulates the spatial component to obtain _spatially enhanced images without

touchingthe spectralinformation.

One _way to separate spatial/spectral information is _{by looking} at the spatial

frequenciesoftheimage. An_{image,according}to_{Schowengerdt,}can alsobe consideredto

bethe sum of alow spatial_frequency component and ahighspatial _frequencycomponent

[Schowengerdt 80].

image = lowpass

(image ₎ + highpass ₍image _{) (1-3)}

The primaryassumptionin this techniqueisthat thespectralinformation iscontained

inthelowpasscomponent andthatthe spatialinformation is inthehighpass component. If

theedges,orthehigh spatialfrequenciesoftheimagestobemerged are_correlated, thenthe

highpass component of the finer spatial resolution image could be substituted for the

highpasscomponent ofthe lowerspatial resolution image. _Assumingthat the _majorityof

the spectralinformation iscontained in the low _frequencycomponent, then theresultant

hybridimagewould maintain itsspectral content while_{gaining improved}spatial resolution.

Schowengerdt used this technique to reconstruct compressed MSS images _by

extrapolating the edge information from the high resolution bands to the low resolution

(compressed) bands. Tomet al _[85] usedavariationofthis technique to sharpenTM band

Another way to separate spatial and spectral information is based on the

intensity-hue-saturation _(IHS) color transformation [Hayden et al 82]. The IHS transformation

allows spatial_information, contained in the_{intensitycomponent,} tobe treated _separately

from the spectral information which is embedded in the hue and saturation components.

The user can then manipulate the spatial information via the _intensity component while

maintainingoverall colorbalanceoftheoriginal scene.

With the IHS _method, three_carefully selected multispectral bands are transformed

into the IHS domain. The digital counts _(DC) of the _intensity component can then be

modified_usingthehighresolution_{(panchromatic)}referenceimage DCs. Themodifieddata

isthen transformedbackintothered-green-blue _(RGB)colordomain anddisplayed.

UsingIHS methods,Carperet al _[87] andWelsh and Ehlers _[87] obtained_visually

superiorresults than thoseobtained _{using Cliche's (adhoc)}methods. Care hasto be taken

though,toensurethat thehighspatialresolution referenceimage is_highly correlatedto the

otherinput images. _Having high correlation between two images means that the linear

relationship between the_{images is very} strong.

1.2.3 _Merging and _{Maintaining Radiometry}

All of these methods _up to now produced _visually enhanced images. Spatial

resolution appearedto takeonthereference _{(panchromatic)}imageresolutionwhile_retaining

mostofthe spectralinformation. However,in allcases, thespecific radiometric values of

themultispectralimageswerelost. Thethird typeof_mergingalgorithmissimilarto thead

hocapproach,butismorestatistically-basedand attemptstomaintain radiometricintegrity.

One method suggested _by Pradines [86] to _keep radiometric _quality of the

P1 P2

P3 P4

XP1 XP2

XP3 XP4

where _XP(J) = _{X x} P(J)

PI +P2+ P3+P4 , J

= 1..4

Figure 1-1. Pradine's_MergingMethod

3y using this method, the aggregate of the four hybrid pixels will return the

radiometry oftheoriginalmultispectralimage. _However,thehigh spatial resolution image

(panchromatic _channel)mustbe correlatedwith theindividual multispectral images. For

thosebands not correlatedwiththereference_image, this_mergingalgorithmcannotbeused.

Two other _merging techniques which _try to maintain the _radiometry of the

multispectral images are the Price _[87] and the DIRS methods [Warnick 89]. These

2.0 MERGING METHODS AND MODIFICATIONS

This_studyevaluates the third typeof_mergingtechniques

-those techniqueswhich

attempt topreserve the _integrity ofthe multispectral data. The two _primary techniques

undertest are the DIRS and the Price methods. In _addition, several modifications and

enhancements have been incorporated to address some ofthe known short-comings of

these techniques. Thesemodified versions are alsoincludedforcomparison.

2.1 The DIRS Method

The Digital_Imaging andRemote_{Sensing Laboratory (DIRS)}conducted a

proof-of-concept _study on _merging multi-date-multi-sensor-multi-resolution images forenhanced

image analysis _[Warnick, et _al, 1989]. _{Specifically they} merged a high resolution

panchromatic image (SPOT-1 panchromatic_{channel, 10m GIFOV)}with medium resolution

multispectral images(Landsat-5,TM bands _{1-5, 7,} 30m GIFOV).

The DIRSmethod canbe summarizedinthe_followingsteps:

(1) The_{SPOT image is geometrically}registeredto theTM images.

(2) Amediumresolutionpanchromaticimage is createdfroma weighted

average ofTM bands 1 through 4. This synthetic image approximates the same spectral

characteristics asthehighresolution SPOTpanchromaticchannel.

(3) The histogram of the SPOT panchromatic image is then _linearly

adjustedto the histogramofthe syntheticTMpanchromatic image. This transformation

will, to thefirstorder, accountforthe_differingatmospheric and sensor effectsbetweenthe

(4) The images are then merged to create a high resolution, multiband

hybrid_{image. The merging}algorithmis:

DCHybridMultibandO) =

DCSPOTPan

L

1

\UL.SynTMPan/

where:

DCHybrid

multiband_image;

DCSPOT

DCjj^O)istheDCinthei-th bandoftheoriginalmultispectral_image; and

---'CsynTMPan -s^ie digitalcountinthesyntheticTMpanchromaticimage.

The DIRS method is applied on apixel-by-pixel _basis, and therefore each ofthe

aboveterms alsohas apixel location. These locationsareleftoffforeasierreading.

2.1.1 Results _Using the DIRS _Merging Method

In addition to the visible improvement of the _images, the overall classification

accuracywas_{approximately 10}percentagepointshigherforthehybrid datasetthan forthe

originalTMimages.

2.1.2 Concerns on the DIRS _Merging Method

Because the DIRS method was developed only as a proof-of- _concept

study, the

authors identifiedseveralareasfor furtherstudy.

The synthetic, medium resolution panchromatic imagewas produced as a weighted

average ofTM bands 1 through_{4. The weighting factor for}each band is determined _by

findingthecommon area oftheTM spectral response curve andthe SPOTpanchromatic

spectral response curve. Thecommon area under the two curves isthen divided _by the

totalareaundertheTMcurve considered. Theresultant quotientisthe_{weighting factor for}

that TM band. _{By using integration} to determine areas, the _{weighting factor} can be

representedas:

I

w;'

=

^ (2-2)

TMRSR,i(X) dX

1

where:

i refersto theTM bandnumber, i= ₁ ..4;

w-'

isthe_{weighting factor for}thei-th_band;

TMRSR

scene; and

SPOTRSR(X) istherelative spectral response curve oftheSPOTpanchromatic

channel.

Once the _{weighting factors} are determined for the 4 TM bands, the factors are

w

W;'

1 "

4

X

i=l

ThesyntheticTMpanchromaticimagecan nowbecreated as:

(2-3)

TMsynPan =

W; X _{i (2'4)}

i=l

However, itwasfoundthat the syntheticTMpanchromaticimage wasnot sensitive

in the 700 to 750 nm _range, while the SPOT panchromatic sensor is still _relatively

responsive. Since the _reflectivity of vegetation is active in _{this region, the} SPOT

panchromaticimagerespondsto thisvegetation reflectanceinformationwhile thesynthetic

TMpanchromaticdidnot.

Tocorrectthis_discrepancy, thespectralresponse curvefor TM band4 was

"shifted"

from a bandpass of _{(760, 900)} nm down to _{(710, 850)} nm. This shift increased the

weighting factor for TM band 4, thereby increasing the _{responsivity in} this region and

providinga

"truer"

approximation oftheSPOTpanchromatic image. However,theauthors

suggestedthatanother method of_producinga syntheticTMpanchromatic imageshould be

explored.

(2) Concernson theweak correlationbetweenthe SPOTpanchromatic channel andTM

bands_{4, 5,} and7:

The mergingalgorithm used in theDIRS method as_previously shown in equation

DCHybridMultiband(i) = _DCSpOT

Pan '

L.^

\U<~SynTMPan

where i represents the TMbands 1 through 5and 7. Theconcern iswhen the images for

TM bands4, 5, and7 are merged with the SPOTpanchromatic image. These bands are

only weakly correlated with the SPOT panchromatic channel and should _{probably be}

mergedinadifferent fashion.

(3) Concernson Radiometry:

The high resolution, multiband hybrid images can be thoughtof as TM multiband

images that_{have been spatially}enhanced. Theenhancementistheresult oftheintegration

oftheSPOTpanchromaticimage,buttheresultantmeanradiances should still bethesame

asthe original TM data. As illustrated_below, theoriginal TM images haveone pixel to

describea30m_by 30marea,whilethehybrid image has9pixels todescribethe same area.

Forconsistent _{nomenclature, the} 30m _by 30m area will be referredto as a _{"superpixel";}

and a 10m_by 10mpixel as a "subpixel".

30m

TMpixel

"superpixel"

(a)

1 2 3

4 5 6

7 8 9

Nine highresolution

"subpixels"

(b)

Figure 2-1 Superpixeland subpixels

Assuminglinearmodeling, the_followingcondition mustholdtomaintain radiometric

integrity.

9

DC =

t

E

j=_i

However,no such check was made or enforcedin theDIRSmethod.

2.2 General Modifications to the DIRS Method

Thissectiondescribes threegeneral modificationsto theDIRS methodwhichaddress

some ofthe concerns raised in the previous section. These modifications are grouped

together because _they donot alterthe _merging algorithm. _{Instead, they} affect eitherthe

input imagesto the_mergingalgorithm,orthehybridoutputimages.

2.2.1 _Creating a TM panchromatic image

In the DIRS study, the _{weighting factors for} each TM band were obtained _by

computing the _overlapping areas ofthe SPOTpanchromatic and the TM bands spectral

response curves. _However,the syntheticTMpanchromaticimage hadalackof_sensitivity

in the 700-750 nm region. _Consequently the TM band 4 spectral response curve was

"shifted"

toprovide a stronger_{weighting factor}and more_{responsivity in}thatbandpass.

Anothermethod to obtain the _weightingfactors has been used _by Suits [Suitset al

Suit's approach,new_{TM weighting factors} werecomputed_usinga multivariateregression on estimated sensor signals.

EstimatingSensor Signals

Given the reflectance spectrum of a _target, some atmospheric parameters, and an atmospheric model (such as _LOWTRAN) we can estimate the radiance that reaches the sensor in a bandpass of interest. This radiance parameter is designated _L^ , and is a

function of wavelength, X. With

L^

function, $(X),to gettheeffective radianceseen _bythe sensor. Thiseffectiveradiance, _Ls

iscomputed as:

(

Ls = ^ (2-6)

P(X) dX

Jo

Theoutput signal (in_DCs) can nowbeestimated_usingLs andtheknown gain and offset of each sensor.

DCs = Ls-i

-offseti

(2-7)

garni

where: DQ = _estimatedDCofTM band i;

L = effective radiance seenbythei-th band; and

S-l

gainj, offset;

These simulatedsignals are thenregressed to_determinethe

weighting factors. This

regression canberepresentedas:

DCspot-1

DCspot-2

DCspoT-3

DCspQT-n _

DCxMl-1 DC-TM2-1 DCTM3-1 DCtM4-1

DCtmI-2 DCtM2-2 DCjM3-2 DCTM4-2

DCtmI-3 DCtM2-3 DCTM3-3 DCTM4-3

_ DCTMl-n DCTM2-n DCTM3-n DCTM4-n

CO! co2 (03 C04. El 2 3 -nJ (2-8)

where: DC istheestimated signal ofthesubscriptedsensor;

com , m = 1..4 is the _weightingfactors forTM1, TM2, TM3,_andTM4

respectively;

n is thenumberofsamplesin the_regression; and

e istheerrorvector whichis minimized when _{solving for}co.

To compute the simulated signals from _{TM1, TM2, TM3,} TM4 and _SPOT,

LOWTRAN7 (an atmospheric propagation_model)was modified. The firstmodification to

LOWTRAN 7 was to allow the program to access a target reflectance spectrum.

LOWTRAN 7 in itsoriginalformuses_onlyone reflectancevalue (calledthesurface albedo

or_'SALB')forall wavelengths. This limitation impliesthatalltargetreflectance spectra are

uniform and flat. The modified version now allows LOWTRAN 7 to access a file

containingreflectanceinformation. Areflectance valueis then accessed(orcomputed via

interpolation)foreachwavelengthrun_byLOWTRAN.

Thesecond modification was availablefrompreviouswork_by CarlSalvaggio. His

modification integrated the sensor's spectral response curves with the radiance values

equation (2-7).

Having modified LOWTRAN 7 to produce simulated sensor _DCs, 25 target

reflectance spectra were chosen. These 25 targetswere comprisedof5 majorclasses

-urban, soil, water, trees, and grass with 5 samples each. These spectra are plotted in

figures 2-2through2-6onthe_followingpages.

^ 40

30

-20

-10

-1 ₁ 1 ₁ 1 ₁ ' r~

0.3 0.5 0.7 0.9 1.1

wavelength (um)

asphaltave concrete ave

slate ave gravel

roofasphalt

ou

-4n-

.-

30-

20-/

VS*-""

*&r<

y'"

10

-0 - _'

1 * 4?

clayave

loam_dryave

loamwet ave

sand ave

soil ave

0.3 0.5 0.7 0.9 1.1

wavelength (um)

Figure 2-3 Reflectance SpectraofSelected Soil Targets

0.2

waterl

water2

water3

water4

water ave

0.4 0.6

wavelength (um)

0.3 0.9

wavelength (um)

ash

beech

maple

oak

pine

r

1.1

Figure2-5 ReflectanceSpectraofSelected Tree Targets

clover

coarse grass

grassave

orchardgrass

swampgrass

0.7

wavelength

The modified LOWTRAN 7 was then run with these 25 reflectance spectra at 3

differentatmospheres of_{varying haze. These 75} samples werethenregressedvia equation

(2-8) and new _{weighting factors for}TM1, TM2, TM3 and TM4 were obtained. These

results are described in section 4. 1. For further information on the LOWTRAN input

2.2.2 Interpolated Input Images

In the original DIRS method, each TM 30m superpixel was replicated into nine

smaller 10mpixels tomatchtheSPOT'Spixel resolution(see figure 2-7).

512

pixels

512

pixels

->

K

30m pixels

-> 3 9 15 21 1536 pixels Y 1536 pixels h-(a)

10mpixels

(b)

->

3 3 3 9 9

3 3 3 9 9

3 3 3 9 9

15 15 15 21 21

15 15 15 21 21

Figure 2-7 PixelReplication

(a) 512 _by512originalTMimagewith 30mpixels.

(b) Afterreplication, theimage isnow 1536_by 1536with 10mpixels. Thecovered ground areaisthesame.

ThesereplicatedTMimageswere thenused asinputto the_mergingalgorithm. In

hopesof_reducingthe subsequent

"blocky"

appearancein thehybrid images (caused inpart

input images. This simple _filtering technique has been used to reduce the _blocky

appearance when _{enlarging images} [Bernstein _79], and when _merging multi-resolution

images [Chavez 84, 86]. _Thus, given a replicated_image as in Figure _2-7, the resultant

input image isshown_{in Figure 2-8 below.}

1 1 1

1 1 1

1 1 1

3 3 5 7 9

3 3 5 7 9

7 7 9 11 13

11 11 13 15 17

15 15 17 19 21

(a) 3_{by3 averaging kernel (b)} resultantinterpolated input

Figure 2-8 Interpolated Input

By using these averaged TM _inputs, the _merging algorithm does not _change, althoughthe term_DCj^ri) nolonger hasthesame value overtheentiresuperpixel.

Forthisstudy,thesereplicatedimagesthatwere smoothed_bythe_{averaging filter}are referredtoasthe"interpolated" inputimages. This termisusedtodistinguishtheseimages

fromotherimagesthatwillbe"blurred", "smoothed", or "averaged".

2.2.3 A Technique for Radiometric Post-correction

Although the interpolation of the input data reduces the _blocky appearance, the

the original data radiometric _integrity is lost. In addition, the DIRS method does not

check or enforcewhethertheaverageDCwithintheSPOTpanchromaticsuperpixelequals

the_{corresponding}superpixelin thesyntheticTMpanchromaticimage. Again, theprecise

radiometric valueislost.

Ratherthanalterthe_{basic merging}algorithm oftheDIRS method (or anymethod),

radiometric _integrity canbe restored at the superpixel level _{by multiplying} each hybrid

superpixel (a block of 9 10m _{pixels) by} a correction factor. This correction factor is definedas:

fjp _ TMj

. 30m

(2-9)

(i)

where:

TM;.30m

= _{DC fromone original}30_{m pixel}fromTMband i

Hybrid(j)i. 10m

= _{DC fromthe}j-th 10m pixel (outof₉₎fromthehybrid

superpixelthatcorrespondsto

TMi

Thispost-fix operationcorrectsfor differences intheSPOTpanchromaticand thesynthetic

TMpanchromatic images,as wellas theradiometric changeto theinterpolated TM input

images. Thiscorrection, however, re-introducessomeoftheblockappearances sincethe

2.3 The DIRS Enhancement 1 _Merging Method

In a simple efforttoreduce the "blocky" appearance ofthe hybrid data, the DIRS

report suggested _implementing a nearest neighbor-type of correction scheme. In this

correction_{routine,each subpixel was comparedtoitsown superpixel'scenter valueandto}

its neighbors center value(see Figure 2-9).

::::^

\

\

-y

-O-^:::: >::^

..^7. \J

~

comersubpixel

side subpixel

center subpixel

Figure2-9 DIRS nearest neighbor correction routine.

Comer subpixels compared values with the diagonal neighbor, and side subpixels

compared to the adjacent neighbor superpixel's center value. If the subpixel's DC was

closer toitsown center pixel's _value,then the_mergingalgorithm remained thesame as in

equation(2-1):

DCHybridMultibandO) = _DCsPOT Pan

DCtmG)

|

Ifthe subpixel's DCwas closerto theneighbor's _DC,than the neighbor's ratiowas

usedinthe_merging algorithm:

T\r Ji\ - T\r

I

\

^T-HybridMultiband(l) = _^SPOT

Pan

-f-7; s

\UL,NeighborSyn TM PanI (2-10)

The DIRS Enhancement 1 merging methodfollows this same correction _scheme,

except that the comer pixels also compare theirvalues against its adjacent superpixel's

center values (notjust the diagonal superpixel). Thecomparison scheme forone corner

subpixel looksasbelow:

::::^ m

\

/

/

K

^

*>;;;!

Figure2-10 DIRS Enhancement 1 Correction Routine

Thesuperpixel whose center pixelvaluewas closestto thesubpixel value contributed

theratiointhe_mergingalgorithm.

subpixelsis because thecorrelationbetween adjacent pixelsismuch higher thanbetween

diagonalpixels[Ahem 86]. This bettercorrelation meansthereisan increasedchance that

a comer subpixel willfind a_better,more accurate matchfor_substitutingratios. _Correcting

thecomer subpixels_obviouslyreducestheoverallblockappearance.

Despitethesechanges,theDIRS Enhancement 1 method stillhasseverallimitations.

These include:

- The

centersubpixel of a superpixel isnever changed andwill always use itsown

ratio.

- Each

and_every non-center subpixel is checked,regardless if itresides in a

non-varyingsuperpixel, orifthe _neighboringsuperpixelshavea highvariance.

- TM bands

4,5, and 7 are still handled as though _they are correlated to the SPOT

panchromaticimage.

-Radiometry isstill not_preciselypreserved.

2.4 DIRS Enhancement 2 _Merging Method

DIRS Enhancement 2 is an attempttocorrect some ofthedeficiencies in previous

DIRS methods. This method is described below and can be followed in the flow chart

diagramin Figure2-_{12attheend ofthis} section.

The basic premise ofthis method is that the high resolution SPOT panchromatic

imageprovidesthe spatialinformationon whichsegmentationand computationaldecisions

images. The SPOTpanchromatic image is handled in 3 _by 3 blockswhich correspond to

thereplicatedTMinput imagesuperpixels.

Thestepsforeach3_by3panchromaticblockare outlined as follows:

1. Check to see ifthe SPOT block is "mixed"

or "pure". A block is considered

mixed ifthe variance _among its nine pixels is higher than an established threshold. A

"pure"

block wouldbeone with a variancelowerthan this threshold.

a) If the superpixel block is pure, then the DIRS original _merging

algorithm (section_2.1)with a post-fix correction routine (section_2.2.3)isrun tocreatethe

corresponding hybrid block. This merging operation will be referred to as the "pure

merge"

inthisalgorithm.

b) If the block has a high variance, then the algorithm continues

processingthemixedblock.

2. Check ifall the _{neighboring blocks} are also mixed. Ifall are mixed, then the

algorithmdefaults to do the pure merge (the original DIRS _method) over the superpixel

block of interest. If some of the _neighboring superpixel blocks are _"pure", then the

algorithm continues.

3. Connectthepanchromatic superpixel. Eachsubpixel within thepan superpixel is

compared toitsadjacent subpixels. Ifthedifference betweenthe2subpixels is lowerthan

a set_threshold, then _they areconsidered"connected". Iftwo subpixelsare_connected,then

each subpixel is allowed to compare to the others adjacent subpixels for further

connections. Thus anetwork ofthesuperpixelcan be constructed. An exploded view of

Figure 2-11. Connectionsbetween thecentersubpixel

4. Findthe subpixels within the mixedSPOT block that can be matched to a pure

neighboring superpixel. A match can occur when the subpixel DC and the _neighboring superpixel's mean value are within a set threshold. Rather than limit the subpixels to

compare_onlywiththose _neighboringsuperpixels_they cantouch (as in DIRS Enhancement

1), this method also allows the subpixels to compare to _any of the eight _neighboring

superpixels providedthat: _(a) the _neighboringsuperpixel ispure; and_(b) thesubpixelcan

be "connected" to thesuperpixel. Ifthe subpixelhas morethan one superpixel to whichit

couldbe_matched, then thesuperpixel whose mean valueistheclosesttothe subpixelvalue ischosen. Thesematched subpixels are thenreferencedas

"pure"

subpixels.

5. Createthe _{corresponding TM hybrid}subpixels _bymergingthesepure subpixels

usingthe matching superpixel's ratio andcorrection factor. This merge is similarto the

6. Derive a newTM value forthe _{remaining (mixed) subpixels}

by removing the

hybridvaluescomputed inthe_{step before.} Thiscanbe_describedas:

9-(TM0

-hybridiO)

NewTMi = -J^.

(2.U)

where:

NewTM;

remaining (mixed)subpixels within

thesuperpixel ofinterestin TM bandi;

TMj

superpixel ofinterestin TM bandi;

n = _{number of pure}

subpixels;

hybrid;(j) = _{those hybrid}

subpixels computed fromthe pure

sub-pixels;

Ifall the subpixels werefound to be_pure, (n = _{9), then this}

step is skipped to _step

10.

7. Checkto seeifthenewTMvalues are valid. ThenewTMvalues cannot be less

than 0or greaterthan 255.

a) Ifthevaluesare_invalid,then the leastpurehybrid subpixel isremoved

and reclassified as a mixed subpixel. The leastpurehybridsubpixelis thesubpixel which

hasthelargest differencebetween itselfand_{its matching}superpixel mean. The leastpure

hybridsubpixel will continuetoberemoveduntilthereare no purehybridsubpixelsleft,or

new valid valuesfor TMare computed. Ifthereare no pure hybrid subpixelsleft, then the

algorithmdefaults todoa pure merge onthesuperpixel.

TM DCs. _Using the new valid TM DCs and the _{weighting factors developed} in section

2.1.1,thenew syntheticTMpanchromaticcanbedescribedas:

NewTMSynPan =

j

wi x _{NewTM; (2-12)}

i=_l

9. The _remaining subpixels are now merged _using the new TM DC and the new

synthetic

TM^

DCHybrid(i)

I

(2-13)

\!JUNewTMSynPan/

10. _{Radiometrically}correctthesemixed hybridsubpixels generated_{in step 9 using}

thepost-fix operation.

11. Check to see if this final hybrid block (pure and mixed _subpixels) is

"reasonable". Some checksfor_beingreasonable are: _(a) ifthehybrid band iscorrelated

withtheSPOTpanchromatic_image,theorder ofthe9 hybridsubpixels should bethesame

as the order of the original SPOT subpixels; (b) the standard _{deviation among} the

subpixels of the hybrid superpixel should be within a threshold factor of the standard

deviationoftheoriginalSPOTsuperpixel.

12. Ifthehybrid superpixelis foundtobe unreasonable, then theleastpure subpixel

is removed and the algorithm returns to _{step 6.} Ifthe hybrid superpixel is found to be

Note thatwith thisnewmethod, the_{followingimprovements}toDIRS Enhancement

1 weremade:

(1) The center subpixel is not constrained to _only use its superpixel ratio and

correction factor. If itcan match toa_{connecting superpixel,} then thecenter subpixel can

usethe_matching superpixel's parameters. In _addition, theother subpixels can also match

with_anyoftheeight_neighboringsuperpixels providedthata connection exists between the

twoandthe superpixelis pure.

(2) Rather than _running comparisons for all _{subpixels, only} those SPOT pan

superpixel blocks thatareidentified as

"mixed"

(havea largevariance) will beprocessed.

In addition,only those_neighboringsuperpixelsthatare purewillbeable tocontribute their

ratioandcorrection factor. Thesechanges should _helpin speed andin avoiding improper

substitutionsforsuperpixelparameters.

(3) Radiometry is preserved. Those subpixels that are identified as 'pure'

(are

matched to a pure _neighboring superpixel) produce the pure hybrid subpixels. The

remainingsubpixels are thencomputedtomaintaintheradiometric value oftheoriginalTM

value. Insimplistic_{terms, it}canbe describedas:

known_answer =

pure_valuessubpix + mixed_valuessubpix

Givenaknown answer(original TMvalue) andthepuresubpixel component_{(determined),}

themixedcomponentis_simplywhat'sleft.

However, DIRS Enhancement 2 still does not handle the _weakly correlated TM

CD

O*- _No

JL Connectthesubpixels

Findthepure subpixels

-thosewhichare closetoa

pure neighbor's mean value

>0

Mergepuresubpixels_by

usingthepure neighbor's

ratio and correctionfactor

Removetheleast

purehybridsubpixel

DerivenewTM DCs for

remainingmixed sub-pixels

2.4.1 A Modification to the DIRS Enhancement 2 Method

This small modification is a quick fix to handle the TM bands that are _weakly

correlated to thepanchromaticimage. Forthis_{modification,} the_onlychange to theDIRS Enhancement 2 method is to the "puremerge"

operation forthe_weaklycorrelated bands.

Insteadof pure_mergingthese_{bands using}theDIRSmethod:

DCHybridMultibandO) =

DCsPOTPan '

\DCsynTMPan)

thehybridvalue now_simplytakes thevalue oftheoriginalTM superpixel:

DGHybrid

Thus if a subpixel is matched to a _neighboring pure superpixel, the hybrid subpixel for

these_weaklycorrelatedbandswould contain theTMvalueforthat_matchingsuperpixel.

Allother_(correlated)bands aremanipulatedintheidenticalfashion asbefore.

Thesemethodsdescribedsofar insection2arevariationsofthe_{basic DIRS merging}

algorithm. Theremainderofthissectiondescribes Price's mergingmethod and some ofits

2.5 Price's _Merging Technique

Price'stechnique_{for merging}multispectral and panchromaticimagesis splitintotwo

cases. The firstcase is formultispectral bands thatare correlatedwith the panchromatic

image. Thesecond caseis_{for weakly}correlatedbands.

2.5.1 Case 1 -- Correlated Input Images

Becausetheimagesarecorrelated, the_relationship canbe describedaslinear. On an

individual band _basis, the linear coefficients can _{be determined using} a simple linear

regression. This is illustrated in Figure 2-13, where the DCs of an averaged SPOT

panchromaticimageare plotted againsttheDCsofaThematic Mapper(TM) band.

TM TM = _{( a} x Pan ₎ + b

i i ave i

Pan ave

Figure2-13. Linear RegressionofTM bandsandthePanchromaticImage

where: TM- = DCofasuperpixelintheTM i-thband;

Panave

oftheSPOTpanchromaticimage;

a-,

After solving forajandb- we cancompute ahighresolution estimate ofthei-th band

using theoriginalhighresolutionpanchromatic channel:

fMj

^ Paniom + _{b; (2-14)}

The mergingalgorithmthenusesthehighresolutionestimatetocreatethehybrid image:

Hybrid, - LLi (2-15)

1_{Mi_ave}

where: Hybrid- = DC_ofthehybrid i-thband;

TMj = DC from_theoriginal TMi-th_band;

TMi = DC from_thehighresolutionestimateofTM i-thband;

TMi.ave = average oftheDCs inthe_{correspondingTM;}image

correspondingto the

TM^

Forexample, themergingofTMmultispectral images (30m_GIFOV) witha SPOT

T

Original

Panchromatic

30m

1

PI P2 P3

P4 P5 P6

P7 P8 P9

Original TM

Tl T2 T3

T4

*

TJ T8

?

High Resolution EstimateofTM

HI H2 H3

H4 H5 H6

H7 H8 H9

Hybrid.

Figure 2-14 Price's_MergingMethod - Case 1

where:

H(J); = TM, T(J)j

TMj.ave

, J = 1...9

and _TMi.ave =

i

T(J)-y j= i

Notethe _similarity toPradines' method,except that Price is_using an estimate ofa

multispectral band insteadofthepanchromaticdatadirectlyforthemergingoperation. In

multispectral _band, while Pradine has the sum of the hybrid DCs equal the original

multispectralvalue. _Finally,Price suggests a methodtohandlemultispectral bandsthatare

weaklycorrelatedwiththepanchromaticchannel.

2.5.2 Case 2

-Weakly Correlated Bands

Becausesomebandsare not_linearlyrelated(ievisible andnearinfrared channels),a

more general _relationship was used _by Price. After registering the _images, Price again

averaged the panchromatic channel to the same spatial resolution as the multispectral

images. Pricethen computedtheexpected _(mean)valueforthe_weaklycorrelatedband for

eachgivenvalueinthe 30mpanchromaticimage.

Forexample,Price wouldfind allthepixelsin the30mpanchromaticimage thathad

aDCofX (0to 255). He would then find the average oftheDCs ofthe _{corresponding}

pixelsinthe_weaklycorrelated band. Thusa simplelook-up-table _(LUT)can begenerated.

This process is depicted in Figure 2-15 on the _following page. _Using this _LUT, a high

resolution estimate of the multispectral band can be computed from the original high

resolution panchromatic image. This high resolution estimate is now used as in the

correlatedcase,andismergedwiththeoriginal multispectraldata.

2.5.3 Price's Results

Pricetested hisprocedure_{using SPOT}simulationdatawitha 10mpanchromatic and

three 20m multispectral channels. He then averaged the channels to obtain 20m

panchromatic and 40m multispectral channels. Applying the above procedures to the

averaged_data,Priceproducedhybrid 20mvalues whichwere thencomparedto theoriginal

0 0 0

0 0 2

1 1 2

Average Pan Image (30m)

Average Pan DC

8 8 8

7 9 16

20 24 14

WeaklyCorrelatedTM Band (30m)

CorrespondingDCs from

WeaklyCorrelatedTM Band

0

1 2

8, 8, 8, 7, 9, 20, 24, ...

16, 14, ...

255

Computethemean ofthe

corresponding TM DCs

tocreatetheLUT

Ave Pan DC Mean TM DC

0

1 2

255

8 22

15

Multispectralchannels 1 and2were_highlycorrelatedwith the panchromatic_channel,

and produced residual errors of aroundtwodigitalcounts. This isquite accurate sincethe

standarddeviationoftheoriginaldatawas around20 DCs.

For the thirdchannel which is not_{correlated, the procedure was}

only "moderately

successful". Thepredicted values accounted _{only for} about 75% ofthe variance in the

original_data,as comparedto99%inthecorrelated case.

Price didnottest theeffecthistechniquehadon classification accuracy.

2.6 Price Modification -- A Method _of

Handling Weakly Correlated Input

Images

One ofthe biggest concerns in all ofthe_merging techniques to date has been the

mergingof multispectral imagesthatare_weaklycorrelated withthereference panchromatic

channel. In our _case, TM bands 4, 5, and 7 are not _strongly correlated with the

panchromatic channel.

This modification to Price'stechnique implementsa method _byTom et al [85]. In

hispaper,Tomimprovedthespatial resolutionoftheTM band 6 from 120mto30m_using

theotherTMbands. His techniqueis basedon an adaptive multibandleastsquares method

which computes an optimal image estimate. His approach relies onthe assumption that

registeredTMdataare correlated acrossthebands in small local areas. _Byusingthelocal

correlation_property, Tomusedvisible and IRbands topredictthe thermal IR image data

(band ₆₎in 30m resolution.The thermalimageestimate wasformed _byaweightedlinear

image.

By modifying Tom's technique, we could obtain high resolution estimates ofTM

bands _4,5, and 7. To compute the high resolution TM band 7 _estimate, the adaptive

multiband enhancement procedure wouldtake thesegeneral steps:

(1) Average filterthepanchromatic channeltoobtain30mresolution.

(2) Using theaveraged 30 m panchromatic image and the originalTM 1, 2, and 3

images as_input,we can implementtheadaptiveleastsquares methodtogeneratethelinear

prediction coefficientsfor band 7 at each pixellocation. Fora givenlocationofthe_sliding

3_by 3window,wecan representthisas:

TM7i TM72 TM73

TM79 _

1 _{TMh TM2] TM3i}

Pan-1 _{TM12 TM22 TM32 Pan2}

1 _{TM13 TM23 TM33 Pan3}

1 _{TM19 TM2g TM39 Pan9}

~b0" "ei"

bi e2

b2 + _{e_3}

b3

-b4_ _e9_

(2-16)

The least square solution is computed _{by solving for}theset of coefficients (thevector _b)

thatminimizes theerror vector(e). Notethat each pixellocation will have its own set of

coefficients. _{One way}tohandle thisdata isto_{have matching}coefficient

"images"

foreach

input/referenceimage.

(3) Obtain 10m resolution estimates ofTM bands 1, 2,and3_byprevious methods.

(i.e.,PriceorDIRS _methods)

(4) Generateanoptimalestimateof_{TM band 7 using}thepredictioncoefficients and

the high resolution input images (panchromatic, Hybrid bands 1, 2, 3). For each 10m

TM7 =

(br

Hybi)+(b2- _Hyb2)+(b3- _Hyb3)+

(b4-Pan,om)

where

Hybj

Hyb2

Hyb3

Price's techniqueforcorrelatedbands.

(5) Mergethishighresolution estimate ofTM band 7 withtheoriginal _{band 7 using}

Price'smethod with correlatedimages(section_2.5.1)toobtain a 10m

Hyb7

After obtaining Hyb7, we can usethis band as another inputreference to compute

Hyb5. As before, avector b is determined, this time with one more dimension for the

additionalinputreferenceimage.

TM5i

TM52 TM53

TM5g

1 TMh TM2i TM3i

1 _{TM12 TM22 TM32}

1 _{TM13 TM23 TM33}

TM7i Pam

TM72 Pan2 TM73 Pan3

1 _{TM19 TM2g TM3g} TM79 Pan9

"bo" -, bi ei e2 b, ₊ e3 b3 b4

Lbs.

With this vector _b, the high resolution estimate for TM5 can be computed similar to

equation(2-17):

TM5 =

(by

(b4- _Hyb7)+

(b5Paniom)

(2-19)

The hybrid image for TM5 is then obtainedby _merging this highresolution estimate of

TM5with theoriginalTM5 image_{using Price's}methodwith correlatedimages. Oncethe

hybrid for TM5 (Hyb5)is found, this band as well as

Hyb7

Theorder of_{computing band 7, 5,} and then4ischosen becausethis is theorder of

decreasing band correlation with the panchromatic image. Tom et al _{[85] additionally}

showedthat _increasing thenumber ofinputreference bandsdecreased thehighresolution

estimate error. _Therefore,band 4 the weakest correlatedband~ is

computedlastto use

the2additional referencebands.

Section 2 described the two_primarymethods of_mergingthatareinvestigated inthis

study the DIRS and the Price methods. In addition, several modifications and

enhancements have also been introduced. The _following section describes how these

3.0 EXPERIMENTAL APPROACH

In comparing these _{mergingmethods, several variations ofthe input images were}

used. Thesevariations aredescribed insection 3.1. Thetwo_primarytestsforevaluation

-classification _accuracy and radiometric error

-are described in sections 3.2 and 3.3

respectively. _Finally,theprocedureforthis _studyispresentedin section3.4.

3.1 Selection and Preparation of Test _Imagery

The SPOTandTM imagesselected forthis _studyare theidentical imagesusedinthe

DIRS proof-of-concept _{study [Warnick 89].} The images were selectedfrom a scene of

greater_{Rochester, NY,}