An Image fusion algorithm for spatially enhancing spectral mixture maps

Rochester Institute of Technology

RIT Scholar Works

Theses

Thesis/Dissertation Collections

8-1-1996

An Image fusion algorithm for spatially enhancing

spectral mixture maps

Harry Gross

Follow this and additional works at:

http://scholarworks.rit.edu/theses

This Dissertation is brought to you for free and open access by the Thesis/Dissertation Collections at RIT Scholar Works. It has been accepted for inclusion in Theses by an authorized administrator of RIT Scholar Works. For more information, please [email protected].

Recommended Citation

AN IMAGE FUSION ALGORITHM FOR SPATIALLY

ENHANCING SPECTRAL MIXTURE MAPS

by

Harry N. Gross

Major, USAF

B.S. United States Air Force Academy (1983)

S.M. Massachusetts Institute of Technology (1985)

A dissertation submitted in partial fulfillment

of the requirements for the degree ofPh.D.

in the Chester F. Carlson Center for Imaging

Science in the College of Science of the

Rochester Institute of Technology

August 1996

Signature of the Author

_

Accepted by

H_e_n_ry_E_._R_h_o_d_y

_

CHESTERF.CARLSON

CENTER FOR IMAGING SCIENCE

COLLEGE OF SCIENCE

ROCHESTER INSTITUTE OF TECHNOLOGY

ROCHESTER, NEW YORK

CERTIFICATE OF APPROVAL

Ph.D. DEGREE DISSERTATION

The Ph.D. Degree Dissertation ofHany N. Gross

has been examined and approved by the dissertation

committee as satisfactory for the dissertation

requirement for the Ph.D. degree in Imaging Science

Dr. John

R.

Schott, Dissertation Advisor

Dr. Harvey

E.

Rhody

Dr. Robert T. Gray

DISSERTATION RELEASE PERMISSION

ROCHESTER INSTITUTE OF TECHNOLOGY

COLLEGE OF SCIENCE

CHESTER F CARLSON CENTER FOR IMAGING SCIENCE

Title of Thesis:

An Image Fusion Algorithm for Spatially Enhancing Spectral

Mixture Maps

I,

Harry

N. Gross, hereby grant permission to the Wallace memorial Library of R.I. T. to

reproduce my thesis

in

whole or in part. Any reproduction

will

not be for commercial use

or profit.

Signature:

_

An

Image

Fusion

Algorithm

for

Spatially Enhancing

Spectral Mixture Maps

by

Harry

N.

Gross

ABSTRACT

An image fusion algorithm, basedupon spectral mixture analysis, is presented. The algorithm

combineslowspatialresolutionmulti/hyperspectraldatawithhighspatial resolution_{sharpeningimage(s)}

to createhigh resolution materialmaps. Spectral

(un)mixing

estimates thepercentage of each material

(called endmembers) within eachlowresolution pixel. Theoutputs of_unmixingareendmemberfraction

images (material maps) at the spatial resolution of the multispectral system. This research includes

developing

animproved_unmixingalgorithmbasedupon stepwise regression. Inthesecond stage ofthe

process, theunmixing solution is sharpenedwithdata from another sensorto generate high resolution

material maps.

Sharpening

is implementedas anonlinear optimization_usingthesametypeofmodel as

unmixing.

Quantifiable results are obtained through the use of _{synthetically} generated imagery. Without

synthetic_images, alargeamount ofground truthwouldberequiredinorderto measurethe _accuracyof

thematerialmaps.Multipleband sharpening iseasilyaccommodated

by

the algorithm,andtheresults are

demonstratedat multiple scales. The analysis includes an examination ofthe effects of constraints and

texture variation on the material maps. The results show stepwise _unmixing is an improvement over

traditional_unmixingalgorithms.Theresultsalso_{indicate sharpening improves}thematerial maps.

The motivation for this research is to take advantage of the next generation of

multi/hyperspectral sensors.Althoughthehyperspectralimageswillbeof modesttolowresolution,

fusing

them with high resolution _{sharpening images} will produce a higher spatial resolution land cover or

Acknowledgments

Iamindebtedto_myadvisor,John

Schott,

andtheothermembersof_mydissertationcommittee.

Your guidance at each _step of this research was _gratefully accepted. Each member ofthe committee

contributed a different perspective to the problems at hand I thankyou for

helping

me maintain an

aggressiveresearch schedule.

Ialsorecognizetheindirectcontributions of_myfellowstudents. Thankyou_{for showing interest}

in my research,

listening

asItried tounderstand,and_offeringadvice whenIneededit. Icounted on _my

fellow Air Force officers, theotherPh D. students, andanyonewhohappenedtowalkintothecomputer

lab. I_particularlythank theDIRSstaff andstudentsfortheirassistance.Youshowed mehowtogetthings

done.

Finally,Ithank_my

family

Theemail connectionmade each

day

more

interesting

and gave mea

mechanismforquicktechnicalsupport. Most

importantly,

IthankAmy, Lauren,andBradley. Yourlove.

Table

of

Contents

LIST OF FIGURES VHI

LIST OF TABLES IX

1.INTRODUCTION 1

1.1Image Fusion 1

1.2Spatialvs. Spectral Resolution 2

1.3Correlation 6

1.3.1BandCorrelation 6

1.3.2 MaterialCorrelation 7

1.4 Mixed Pixels 8

1.5EndResult 9

1.5.1 High Resolution DigitalCounts 9

1.5.2High Resolution Material Maps 10

1.6Synthetic Imagery 11

1.7Outline 12

2.ALTERNATE FUSION TECHNIQUES 14

2.1 ImageFusion Paradigm 14

2.1.1 Transformations. 15

2.1.2Combinations 16

2.2 ModelingBandCorrelation 16

2.2.1 CoordinateTransformations 17

2.2.2 Multiresolution Decomposition 18

2.2.3 RatioMethods 19

2.3 AlgorithmSummary 23

3. PROPOSEDALGORITHM 25

3.1 SpectralMixture Analysis 26

3.2Sharpening ₃₁

3.3Constraint Conditions ₃₃

4.ALGORITHMDEVELOPMENT ₃₈

4.1OptimizationandtheLeast Squares Problem 38

4.1.1

Necessary

Conditions ₃₉

4.1.2 TheGeneralLSProblem ₄₀

4.1.3Analysis of Variance 44

4.1.4 Subset Selection ₄₈

4.1.5

Handling

Constraints 57

4.2 Unmixing: Over-DeterminedLeastSquares ₆₀

4.3 Sharpening: Under-DeterminedLeast Squares ₆₃

5. RESULTS 67

5.1Experimental Design ₆₇

5.1.1Synthetic

Imagery

Characteristics ₅₇

5.1.2 Data Sets ₅₉

5.2.1 BandSelection 7S

5.2.2 Scale 7(5

5.2.3 Variation 7<*

5.3UnmtxingResults 78

5.3.1Scale 78

5.3.2_{Benefit of}

Unmixing

EachPixel 79

5.3.3 Variation 81

5.4FusionResults 82

5.4.1SingleBand 84

5.4.2 MultipleBand. 84

5.4.3_{Effect of Texture Variation} 85

5.4.4Spatial Error Distribution 86

5.5Summary 87

5.6 Applicationto a

"Real"

Image 89

6.CONCLUSIONS 93

6.1Contributions 93

6.2 Limitations 94

6.3Recommendations 95

7. APPENDICES 98

AppendixA: AnalyticalSolutiontoEqualityConstrained Over DeterminedLeastSquares

Problem 98

AppenddcB:Gradient ProjectionAlgorithm 99

AppendixC: SpectralLibraries 103

AppendkD:DataSets 108

AppendixE:Statistical SignificanceofFusion Results Ill

List

Of Figures

Figure 1: Spectralvs. Spatial Resolution 3

Figure 2: ImageCube 4

Figure3: Spectral

Responsivity

ofTMandSPOTPanchromaticBands 6

Figure4: Material Correlation 7

Figure5: Basic TypesofMixtures 9

Figure 6: ImageFusionConcept

-Combining

DigitalCounts 10

Figure 7: Notional Fraction Images(Material_Maps) 11

Figure8: General Image Fusion Process 15

Figure9: Example

Look-Up

Table for

Fusing

Weakly

Correlated Bands 22

Figure10: Image FusionDataFlow

-Creating

HighResolution MaterialMaps 26

Figure11: There Are

Many

HighResolutionUnknownsinaSuperpixel 32

Figure12: ThreeMaterialMixturesinTwo SpectralBands 35

Figure13: Gaussian Distributed Endmembers 36

Figure14: Mixture

Requiring

Negative Fractions 36

Figure15: Three TypesofLeastSquares Problems 42

Figure 16: Geometrical InterpretationofResidual 45

Figure17: If VG is Not Parallelto

VF,

theFunction is Not Minimized 56

Figure 18: AtaMinimum,VF isaLinear Combinationof

Vg;

56

Figure 19: Least Distance

Programming

(LDP)

Problem Illustration 58

Figure20:

Solving

theGeneralLeastSquares Problem 60

Figure21: M-7 SensorBand Passes 68

Figure22: Band 4 (460- 620

nm)ofSynthetic Test Image 69

Figure23:

Creating

SIG Data Sets 70

Figure 24:

Perfectiy

UnmixedMaterial Maps (4 m/p) 71

Figure25:

Comparing

MapsatDifferent Scales 73

Figure 26: Single Band

Sharpening

74

Figure27: MultipleBand

Sharpening

75

Figure28:

Sharpening

atDifferent Scales 76

Figure29: EffectofTextureVariationon

Sharpening

77

Figure30:

Unmixing

atDifferent Scales 79

Figure31: Traditionalvs.Stepwise(Per

Pixel)

Unmixing

₈₀

Figure32:

Unmixing

andReplicationtoaHigher Spatial Resolution 81

Figure33: EffectofTextureVariationon

Unmixing

₈₂

Figure34: Unmixed_(16m)vs. Sharpened_(4m)MaterialMaps 83

Figure 35: FusionwithaSingle

Sharpening

Band 84

Figure 36: FusionwithMultiple

Sharpening

Bands 85

Figure37: EffectofTextureonImageFusionAlgorithm ₈₅

Figure38: SpatialErrorDistribution: High ResolutionMapsvs.Truth 86

Figure 39: Real M-7

Image,

lmResolution ₈₉

Figure40: 10mMaterialMaps

Using

Stepwise

Unmixing

₉₀

Figure 41: 2m MaterialMaps

Using

Stepwise

Unmixing

₉₁

Figure42:

Unmixing

aReal M-7Image ₉₂

Figure43: Projected Gradient ₉₉

Figure 44: M-7Spectral Reflectance Curves ₁₀₄

Figure 45: M-7

Sharpening Library

₁₀₇

List

of

Tables

Table 1: Spreadsheet FormofHigh Resolution

Sharpening

Problem 33

Table2: Basic ANOVA Table 46

Table3: Extra SumofSquares ANOVA Table 50

Table4: Percent Improvementof

Sharpening

OverReplication 86

Table5: M-7Spectral Bands_(\tm) 103

Table6: M-7Spectral Bands_(nm) 104

Table 7: NewSpectral

Library

Reflectance Values for GrassandTrees 105

Table8:

Sharpening

Bands_(nm) 106

Table9: Reflectance Values for

Sharpening

Library

106

Table 10: New

Sharpening

Library

ReflectanceValues forGrassandTrees 107

Table 11: Detailed Results 110

Table12: Statistics from Single BandFusionDataSets 112

1. Introduction

1.1

Image

Fusion

Analystsuseremote_{sensing images}togain informationabout a targetorlandareathatcannotbe

obtained

by

direct measurement.

They

usethe information in the imagesto infer characteristics ofthe

objects in question. For example, analysts maybe interested in_crop health, landuse, or mapping. The

particularobjects_{may have been imagedmany times,}withdifferentsensors.Clearly,an analystinterested

in the most accurate description would want to include as _many images as possible as part of the

"evidence'

thatisexamined.

Using

data fromall available sensorsinthe _studywould enable athorough

analysis.

Thedata available_mayincludeimagestakenfrom bothsatellitesand aircraft.Theactual sensor

platform, ofcourse, affectstheimagecharacteristics.However,

knowing

thesecharacteristics, theanalyst

canaccountfor_{any differences in}theacquisition parameters. The specific sensors _may alsodiffer. For

example, thedetectormaterials (which dictatethe_underlyingspectral _{sensitivity) may}not be thesame.

Thecombinationof adetectormaterial with afilterora_{diffraction grating}definesthe spectral response.

Thedetector size, optical pathcharacteristics, and thesensor altitude combineto determinethe ground

sampledistance correspondingtoanimagepixel.This isthespatial response.

Image fusion mergesimagesofdifferentspatialandspectralresolutionstocreate ahighspatial

resolution multispectral combination. Spatialresolutionisthesize of apixel projected ontothe ground.

Spectralresolutioncorrespondstothespectral widthofthedetector/filter inthesensor

Many

imagefusionalgorithms combinethevariousimagesatthedigital countlevel. The result

is a set ofmultispectral, high spatial resolution images. _However, these images must often be further

processed to create maps ofthe materials in the scene. This dissertation presents an image fusion

1.2 Spatial

vs.

Spectral Resolution

Withpassive_{sensors, thedetector simply}measurestheincidentenergy. Thisenergy,represented

by

theradianceatthe

detector,

isafunctionoftheradiancefrom different sources. Thevarioustypes of

radiance _{include energy} that is reflected from the target, self-emitted from the target, as well as

backgroundand atmospheric energy. Alargepart of remotesensing ismodelingalltheseradianceterms

inordertoestimatefeaturesofthetarget.

At the sensor, the digital counts forthe image are _typicallytaken as a linear function ofthe

detected radiance,

dc

=_gain

\\\

_radiance

dA dQ, dk

+

bias

. (1)

Equation

(1)

shows the sensor will integrate the energy within a differential area, solid angle, and

wavelength. The differentialarea,dA. is a pixel. The differentialsolidangle, dQ. istheprojectionofthe

pixel, through the optical path, to the ground. The combination dA dQ is the spatial resolution. The

differentialwavelength,dl. correspondsto thespectral_bandwidth,andisthespectralresolution. Inorder

to have confidence in the measured _radiance, the signal to noise ratio

(SNR)

at the detector must be

adequate. This requirement on SNR is what determines the detector size, and results in the tradeoff

betweenspatialandspectral resolution.

Asillustratedin Figure1,ifanarrowfilter isusedtogive ahighspectral _resolution,theamount

of electromagnetic radiationthat makes itthrough thefilterwill _{consequently be}small. Becauseofthe

relativelysmallnumberofphotons, thedetectorsizemustbemadelarge inordertomaintain SNR. The

largedetectorsize, whenprojected through thesensor_optics, resultsin alarge spot size onthe ground.

Detector

Spectral filter

Narrow

Filter

Highspectral Lowspatial

Wide Filter

Lowspectral

Highspatial

Figure1: Spectralvs. SpatialResolution

Conversely,

ifa wide spectralfilter isused,wehave lowspectral resolution,but alargenumber

of photons can reachthe detector.

Therefore,

the detectorcanbe smallerand, thus, the ground spot is small.Alowspectralresolutionimpliesahighspatial resolution.

Thistradeoffbetweenspectraland spatialresolution meansthatanalystswillalwaysbepresented

with a_varietyofimagedatawith whichtoperformimage fusion.

One_mayconsider_anyscene as

having

areflectance which canbewrittenas afunctionof spatial

locationandwavelength, e.g.,r(x,y,X). Image data isoften represented as a cube. The face ofthe cube

shows howtheobjects in the scene_{vary spatially} - _an

"image."

The depth ofthe cube corresponds to

differentwavelengths. Slicesofthecuberepresent image bands. Whilethe_{underlying data}forthecube

arecontinuous,thesensor samplesthecubeinxandytomake_pixels,andindepthtoformbands. Figure

2 shows animagecube.Becauseoftheperspective_{view, the}spectralresponse ofthematerials_alongthe

Figure 2: Image Cube

Sensorssamplethis

imaging

space withdifferentcharacteristics.Fortheimage fusionproblem.

data fromatleasttwodifferentsensors aremerged. Thesensor withlower spatialresolution, but higher

spectral resolution is the spectral sensor. The sensor with high spatial resolution but low spectral

resolutionisthespatial sensor. Thespectral sensorimagecube_typicallywill_{have many}slices,orspectral

bands, tooffsetitspoor spatial resolution.Thosesliceswouldbethinnerthan the slice(s)fromthespatial

sensor. Onthe other_hand, a spatial sensor would havemoredata values_(pixels) ineach band thana

correspondingareafromthespectralsensor.

Typically, sensors with a few spectral bands are called multispectral. Their bandwidths are

usuallyontheorder of100run.Hyperspectralsensors

imply

dozensor more narrow_{(i.e. 10 run)}spectral

bands. Panchromatic refers to image bands with several hundred nm bandwidths. In image fusion

applications,panchromaticdata isusuallyusedto_spatially

"sharpen"

multi orhyperspectral image data.

Forthiswork, thedistinction between multispectral and hyperspectral isnot _usually relevant, and both

terms areused interchangeably. Clearly, some applications need hyperspectral sensors with medium to

lowresolution. Other applications require highspatial resolution panchromatic data. Image fusion uses

Imagery

comes froma _variety of sources. _{Two commonly} available commercial satellites are

LandsatandSPOT. Landsat isa series of satelliteslaunched

by

theUnited States inthe 1970sand 1980s.

The latestvehicles, numbers 4 and _5, were launchedin 1982 and 1984 respectively. A large

library

of

Landsat imagesexists.Onesensor onLandsat4and5 iscalledtheThematicMapper(TM)andhasseven

spectralbands. Sixofthebands have 30meterspatialresolutionandcontain data invisibleandinfrared

spectral regions. Theseventhband

(actually

bandnumber_six)givesthermal informationata 120 meter

pixel size.

The Systeme Pour l'Observation del la Terre (SPOT) is a French satellite system that has

launchedthreevehiclesin_{1986, 1990,}and 1993.SPOT has3spectralbands inthevisibleandNIRregion

with20meterpixels. Italsohasapanchromaticbandwitha10 meter resolution. Oneexample ofimage

fusion is to combine the Landsat 30 meter spectral data with a SPOT 10 meter panchromatic image.

Another,

almost_{trivial, fusion}challengeistocombineSPOTspectralbandswiththeSPOTpanchromatic.

Future sensors will improve the resolution in different ways. One class ofsensors will have

higherspatial resolution. Severalcompaniesare_planninghighresolutioncommercial satelliteswith_only

a few spectral bands

(typically

visible or near-infrared).

They

will have resolutions of a few meters.

Russian, Canadian,

and Japanese satellites will also provide _commercially available

imagery

(Foley.

1994).

At the other extreme are

imaging

spectrometers such as AVTRIS andMODIS. The Airborne

Visible/Infrared

Imaging

Spectrometer(AVJJUS)isaNASAsensor whichflieson anER-1 _(U-2)aircraft.

The sensor has _224, 10 run spectral bands at wavelengths from 0.4 to 2.5 \im. The nominal spatial

resolutionis 20 meters(Vane etal., 1993, JohnsonandGreen, 1995). _{NASA is planning} tolaunch the

Moderate-Resolution

Imaging

Spectroradiometer(MODIS)sensorinthelate 1990's. Thissatellite, a part

ofthe Earth

Observing

System program, will have low spatial resolution _{(250-1000 meters)} with 36

spectral_{bands covering}wavelengthsfrom0.4to 14.5 urn_(NASA, 1995). Future fusion possibilitiesare

combining AVIRISwithdigitizedair-photos or highresolutioncommercial satellite

data,

or

increasing

1.3

Correlation

Fusion is possible because of the large amount of correlation in images. The bands of

multispectral images tend to have some degree of correlation. (This is one _way the data can be

compressed.)Also, thepanchromatic image band spectrum_typicallyoverlapswith some ofthe spectral

bands.This bandcorrelationis_veryimportantin

determining

howwellthefusionalgorithms work.

1.3.1

Band

Correlation

AnanalysisofTM data(e.g., forcompression_purposes) showstherearelessthan sevenbands

worthofinformation inthe data. Digital counts _{in any}givenband are correlatedwith digital counts in

others. In

fact,

Crist and Cicone ₍₁₉₈₄₎ estimate the TM data can

largely

be represented in 3 or 4

dimensions. Fortypical scene objects,

knowing

digital counts in onebandmakes thedigital counts in

another

fairly

predictable.

Figure3 showsnormalizedreflective TMandSPOTpanchromaticbands and their_overlapping

sensitivities(Markhamand_Barker, 1985).

Clearly

intheoverlappingregions, the digitalcounts will be

highly

correlated.The SPOTpanchromaticband is

highly

correlatedwithTM bands2and3.

Spectral

Responsivity

c o a

n

a o

> N

5

^

8o O O I

Ol _y CD t

Wavelength(microns)

-TM-1

-TM-5

-TM-2

-TM-7

-TM-3

SPOTPan

-TM-4

Becauseofthis_correlation,itwouldbe easytofuse SPOTpanchromatic withTMvisiblebands.

However,using SPOTtoimprovetheTM infrared bands

(especially

bands5and

7)

is moreproblematic.

Image fusion algorithms address this

by

_creating models to predict how the highresolution data will

appearinbandsthatcorrespondtothespectral sensor.Thevariousmethodsfor accomplishingthiswillbe

reviewedinthenext chapter

1.3.2

Material

Correlation

Analternate_waytocapitalizeonthe correlationwithinanimage istouse material reflectance

curves. Spectral Mixture Analysis models the total radiance measured at the sensor as a linear

combination of radiance _{(reflectance)} from a number ofmaterials (Adams et al., 1986, Smith et al.,

1990a,Adamsetal., 1993). Spectralunmixingistheprocessthat takesdigitalcounts and calculatesthe

percentage of eachmaterial withinthepixel. Ifthematerialsinthescene areidentified, theirreflectance

curvescouldbeusedas abasis for_{mapping between bands.}

Band 1 Band 2

WAVELENGTH

Figure 4: MaterialCorrelation

Figure4shows a sketch oftwomaterial reflectancecurves.Ifthematerial were

known,

onecould

predictitsreflectanceinalternatebands. Inthis example, eventhoughvegetationis darker than soil in

1.4 Mixed Pixels

The region onthe groundimaged

by

a singlepixel_maycontainavarietyofmaterials. In some

respects,the typesof materialsdependupontheapplication Forexample, apixel _maybe 100%forest, or

classified as a mixture ofdeciduous and coniferous trees._{Farmland may be} classifiedas agricultural vs

urban, corn vs. _wheat, or

healthy

vs. stressed.

By

_makingthe material classes more specific, almost all

pixels become "mixed."

Conversely,

if the classes are general, many of the pixels _{may be}

pure."

However, even withgeneral classes, pixelsthat_{lie along}theboundaries will encompass morethan one

material. A mixed pixel is a pixel _containing morethan one type ofmaterial of interest. Obviously.

whetheror notapixelismixeddependsupontheapplication

One can envision _manykinds of material combinations. For convenience, distinctions will be

made _amongthree typesof mixtures. An intrinsicmixtureis definedas onewhose constituent materials

interactat amicroscopiclevel. Photons striking intrinsic mixtures _may encounter multiple interactions.

Thus, theaverage reflectanceis

likely

tobeacomplex combinationofthe individualmaterial properties.

Unmixing

intrinsicmixtures requires anonlinearmodelandisnot addressedinthiswork.

Aggregate and areal mixtures, on the other hand, are characterized

by

linear interactions

betweenthe materials and

incoming

photons.

They

consistofdistinctmaterials,butare mixedat various

spatial scales. Aggregate mixtures combine on a macroscopic level. The total reflectance is a spatial

average ofthe constituent materials, but their components are not _spatially separable at the sensor

resolution.Arealmixtures alsocombine_linearly,buttheir components are _spatiallyresolvable, especially

Wwffi&M

i

Intrinsic Aggregate Areal

Figure5: Basic TypesofMixtures

Envisionalinearmixturewhichisnot_spatiallyresolvable

by

thelowresolution spectral sensor.

Tothis sensor, the material isan aggregate mixture. Onecould unmix the pixel to determine subpixel

composition,butcouldnot_{spatially locate}theendmembers._However, toahighspatial resolutionsensor.

themixturecouldbeareal. Thesecond sensor couldbeusedtosharpenthematerial maps obtainedfrom

thelowresolutionunmixing.

Spatially

separatingarealmixturesisthemotivationfor

developing

image fusionalgorithms.

1.5 End

Result

1.5.1

High Resolution Digital Counts

Onegoal ofimage fusionis tomerge a Low (spatial)Resolution Multi-Spectral

(LRXS)

set of

imageswith aHigh(spatial) ResolutionPanchromatic _(HRP) image to create aHigh Resolution

' 1

=

Mill

1

1

+

Tl I 1

i

i

M

r-LRXS

LowResolution

Multi-Spectral

+

HRP

+ HighResolution Panchromatic

-HRXS

High Resolution

Multi-Spectral

(Hybrid)

Figure 6: ImageFusion Concept

-Combining

DigitalCounts

The largepixels oftheLRXS image are called superpixels.

Assuming

the imagesare _properly

registered,asuperpixelcorrespondstoa number of smallersubpixelsintheHRP image.

Since the images come from different sensors, there _will, in general, not be a simple

correspondencebetweenthepixelsinthe twoimages.

However,

fusionalgorithms depend on_accurately

registering the separate images. This requires _accounting for distortions due to

differing

acquisition

parameters,aswellas_reseatingand

interpolating

toaccountforthedifferentpixel sizes. Thisprojectdoes

not examinetheeffects ofregistration Itassumesthatregistrationisaccomplishedwith_verysmallerror.

For example, pixel _oversampling via interpolation allows registration to be doneto subpixel _accuracy

usinginteractivecontrolpoint selection.

Fusionatthe digitalcount level worksbestfor strongly correlatedbands. In weakly correlated

bands, the performance deteriorates. It is especially difficulttofuse mixed pixels _{in weakly} correlated

bands.

1.5.2 High Resolution Material Maps

Alternatively, the analyst

frequently

desires material maps ofthe area. One purpose of_image

fusion istoimproveclassificationperformance. The hypothesisisthattheHRXS imagewould givebetter

classification results than_{just using} the HRP image.

Classifying directly

from the LRXS data would

The fusionprocedure proposedin this _studyapplies spectral _unmixingto the LRXS imagesto

derivematerialmaps.

Then,

theHRP imagesareusedtosharpenthematerial mapstoahigherresolution.

Theendproductisanimage_(map)foreachmaterial. The

intensity

intheseimagesismadeproportional

tothefractionofthematerial present.Figure7illustratesfraction images.

Truth Water Grass Trees

Figure 7: Notional FractionImages(Material

Maps)

1.6 Synthetic

Imagery

Sincethisproposedfusionalgorithm aimstocreatehighresolutionmaterial maps, _quantifying

thealgorithmperformanceis difficult. Intheabsenceof agreatdealofgroundtruthdata, materialmaps

are often evaluated on anecdotal evidence.

To ensure accurate knowledge ofthe _underlying materials

during

algorithm development and

testing, syntheticimagegeneration_(SIG)toolsare usedtocreatetheimagery. SIGcontrols alltheimage

parameters, making it easiertoanalyze algorithm performance. It also aids algorithm development

by

regulating variation. The image fusion algorithm is developed

incrementally

by including increasing

amounts ofrealisminthesyntheticimagery. The natureanddegreeof errorisobservedateach stage of

development,

and adjustments madetothe algorithm to reduce the errors. Algorithm design with SIG

imagery

isusefultomeasureandimproveeven_widelyacceptedalgorithmslikespectralunmixing,where

quantitativeperformancehas been difficulttodocument.

ASIG developed imagecanbeusedtocontrolthevariouserror sources thatare

likely

toimpair

endmembers, and _variability in _topography and illumination

By

controlling the introduction ofthese

errors,therobustness ofthealgorithm canbestudiedandimprovedupon.

Thealgorithmdevelopmentworkcapitalizesontwobenefitsof syntheticimagegeneration(SIG).

SIGscenes canbe builtwith_{varying degrees}of complexity._{The ability}toincrementallyincreaserealism

givesfeedbacktoalgorithmdesigners.

They

_maytest theirdesignsunder

increasingly

difficultconditions.

and _modify the designs to

incrementally

improve robustness.

Secondly,

since SIG is _entirely computer

createdfromadefined datasource, the_underlying'truth" isknown. Algorithmscanbeevaluated under

variousconditions,wheretheirperformance canbequantified and comparedtoalternatetechniques.

Only

afterthealgorithmisshowntowork under simulatedconditionsis itthen testedon realimagery.

1.7 Outline

Theobjective ofthis research was to

develop

an alternate image fusion algorithmbased upon

spectral unmixing. Spectral unmixingtransforms hyperspectraldata fromtheimagedomaintomaterial

maps.To_date, spectral_unmixingproducts_{have only}beengeneratedatlowspatial resolution.

Fusing

the

material maps with high resolution _{sharpening image(s)} yields a more _spatially accurate classification

map.Quantifiableresultsareavailablebecause SIG

imagery

isused.

A secondary objective was toimprove the spectral _unmixing algorithm. _{Traditional unmixing}

calculatesmaterial mapsforan entirescene. _However,thesamematerials are notpresentinallthepixels

withintheimage.Analgorithmispresented which selectsthematerialstobeunmixed ona pixel

by

pixel

basis.

This document isorganizedasfollows. Section twoprovides backgroundreference on alternate

image fusion techniques. These would establish a baseline _{for comparing image fusion} performance.

Sectionthree

briefly

describestheproposedalgorithm asacombinationof_unmixingandsharpening. The

fourthsectioncontainsthemathematicalfoundationrequiredtodesignand

develop

codetoimplementthe

algorithm. Section five contains quantified results _using synthetic test imagery. The results show

more accuraterepresentationofthegroundtruth. The lastsectionsummarizesthecontributions made

by

2.

Alternate

Fusion

Techniques

2.1 Image

Fusion Paradigm

Thegrowthin remote_{sensing is}a _relativelyrecentphenomenon.

Only

in thelast decade orso

have the_{data become generally}available and affordable. Inaddition, advances incomputer_technology

have just _recently placed the required _{digital image processing} power in low cost workstations. As a

result,manyofthe_{image processing}toolsarenew andnotwelltested. Thealgorithmshave beenapplied

to _only a small number ofimages. As the _{literature shows,} sometimes the results are scene or image

dependent(Braun, 1992). Sensor designscontinueto improve,andthe trendistowardsbetterresolution.

bothspatialandspectral.Allthesefactorscombineto createa_{rapidlychanging field}with_many creative

and successfulideas.

Image fusion combines images of different spatial and spectral resolutions to make a

multispectral combination. The most difficult aspect of image fusion is _accounting for the different

spectral responses inthebands to be fused. Figure8 isa block diagram representation ofthe generic

fusionprocess.Twosteps areinvolved.First,thesensorbandsare aligned_usingatransformation.

Second,

thedataarecombinedinsomemanner.Insomemethods, aninversetransformationisrequiredas athird

1

1

LRXS

1

^^m

Hi

\

\

)

HRP

TRANSFORM SUBSTITUTE

1

/ - Rotation

- I_L-mearinpjir r . rllVA.i

- Linear Combination -Nonlinear

- Reflectance

Figure8: General ImageFusionProcess

2.1.1

Transformations

Severaltypesoftransformationsareusedintheliterature. Thegoal ofthe transformationstageis

to account for the different spectral sensitivities ofthe low resolution and panchromatic bands. Some

algorithms, called Component Substitution (COS), use a coordinate transformation to rotate the

multispectral data so that one ofthe new axes lies in the same direction as the panchromatic band.

Shettigara₍₁₉₉₂₎presents a generalizedCOStechnique.

Anotherpopulartransformation is a linearregression. Price (1987), Munechika et al. (1993).

andBraun

(1992)

uselinearregressionmodelstopredict multispectral digital counts asfunctions ofthe

panchromatic and other spectral bands. These high resolution estimates are used in the ratio method

technique.

Multiresolution decomposition techniques like wavelets are used

by

Ranchin et al. (1993).

IversonandLersch(1994),andRanchinandWald

(1996)

toestablishrelationshipsbetweenpanchromatic

andmultispectraldataatvarious resolutions. Thesedatapointsareusedas_trainingdata fora nonlinear

modelthatrelatesthebands.

Finally,

this project proposes to use spectral mixture analysis as the transformation model.

Spectral _mixing(or unmixing) generates estimates ofthefractionsofmaterials in each pixel_using the

material reflectancecurves(Smithetal., 1990a). Sincetheaverage spectral response of each materialis

known,it is easyto transformfromone spectralbandtoanother.Furthermore, sincethetransformationis

done_accordingto theobjectsinthe scene,contrast reversalsinuncorrelatedbandsare recognized.

2.1.2

Combinations

Afterthespectralbandsare_transformed,thedatamustbecombined. The literaturecontains_only

afewcombination methods. The COS algorithmsuse a substitutionofpanchromaticdata foroneofthe

transformed spectral bands. Most ofthe other methods use a linear combination of multispectral and

panchromaticdata. Awaveletreconstructioncanbe done

by

applyingthetransformation to thedetail in

the image (the wavelet) ratherthan the entireimage. In this methodology, thewaveletsare orthogonal

whilefilteredversionsofthepanchromaticimagearecorrelated.

The

following

sectionsummarizesthemost commonfusionalgorithms.

2.2

Modeling

Band

Correlation

Munechika ₍₁₉₉₀₎ distinguishes three classes offusion algorithms. The first class is called

"fusion forvisual

display."

Thesealgorithms are_primarilyconcerned with_making an imagethat looks

good toa human interpreter. Simple histogram manipulation and contrast _{stretching may fit into} this

category.Thesemethodsareeasy,andtend togive reasonableresults,whichexplains_whyimage fusion is

so popular.

They

require no transformation other than scaling. One example is to substitute the high

resolutionpanchromatic data intothecomputerCRT

display

green channel. Multispectral red andblue

areleftunchanged. Sincethehumanvisual systemismost sensitivetogreen,thisgives a_pleasingresult.

The second classistermed "fusion

by

separate manipulation ofthespatial information."These

aretheCOSalgorithms. Inthesetechniques,thehighresolutionpanchromaticdata isassumedtolie ina

particular direction in a specifiedimage space. The multispectral data aretransformed into that image

back intotheoriginalspatial_{domain. Chavez}et al. ₍₁₉₉₁₎ andBraun(1992)compared threealgorithms

ofthisclass.

2.2.1

Coordinate Transformations

2.2.1.1

High

Pass

Filter

(HPF)

The High Pass Filter

(HPF)

isthe mostobvious_way toseparately manipulatethe spatial data.

Schowengerdt

(1980)

suggests an imagecanberepresented asthesum ofalowpass filtered image anda

highpassfiltered

image, i.e.,

PAN

=

LPAN+HPAN

(2)

Ifthehighresolutiondatacontainstheedgesnot visibleinthelowresolution set.this technique_{may be}

usedtoreplacethose_missingedges. Schowengerdt'sHPFusesthecorresponding multispectral data for

thelowpassimage,

HRXS

=

LRXS

₊

K

HPAN

(3)

whereK is selectedto_{appropriately}weightthecombination oflowresolutionmultispectral andfiltered

panchromatic images. Filberti et al. (1994) uses HPF to fuse color aerial _photography with AVIRIS

hyperspectralimagery.

2.2.1.2

Intensity

Hue Saturation

(IHS)

The

Intensity

Hue Saturation_(IHS)techniqueis describedin bothChavezetal.

(1991)

andBraun

(1992). Someotherrecent applicationsaredescribed inCarperet al.

(1990)

andEhlers (1991). The IHS

technique can _only be used for three bands of data. The transformation is similar to a color space

manipulation. Thethreelowresolutionbandsofdataare treatedas colors(for example, red, green, and

blue). _{Thus, they} can _equivalentlybe represented

by

an _intensity, a hue, and a saturation.

Intensity

is

similar to lightness It would have a scale from black to white. The hue is the dominant _color, which

IHSspace.The

intensity

image is_removed,and replacedwiththescaled panchromaticimage. Thishybrid

isthentransformed backtoRGB. The IHS algorithm assumesthepanchromaticand

intensity

imagesare

similar.

2.2.1.3

Principal Components

Analysis

(PCA)

The Principal Components Analysis

(PCA)

is awell-known transformation in Linear Algebra

andisusedin manycontrolsystemformulations. Thetransformationtakesavectorof correlateddataand

changes it into orthogonal components. These components are uncorrelated with each other. Richards

(1986)usesPCAtoanalyze multispectralimages.Intheimagefusionapplication. PCA isused asaCOS

algorithmtoseparatethefirstprincipal component. This firstcomponent should containthedatathat are

commontoall the

bands,

and is

likely

tobe similarto thepanchromatic image. First, themultispectral

dataaretransformedintoprincipalcomponent space.Thefirstprincipalcomponentimage isremoved and

replaced with the scaled panchromatic image. The hybrid is then transformed back into multispectral

space.Image_{fusion using}PCA is described in Chavezetal._(1991),Braun(1992),andShettigara(1992).

2.2.2

Multiresolution

Decomposition

Multiresolutiondecomposition canbeusedto

develop

a_{relationship between}the panchromatic

and spectraldata. Forexample,an orthonormal waveletdecompositionis doneonbothimagestogenerate

resolution(Laplacian)pyramids.The levelsofthesepyramids representsubsampleddetailimagesthatare

uncorrelatedacrosstherespectivebands.Thepanchromaticimagehasone extralayer inthepyramiddue

toits higherspatial resolution. Anonlinear modelrelatesthepanchromatic and spectraldigitalcounts at

each of the subsampled layers. Once this model is trained, it is applied to the high resolution

panchromaticimagetopredicta highresolution spectral image. Ranchinet al.

(1993)

and Iverson and

Lersch

(1994)

usethis methodtofuse SPOTmultispectraland SPOT panchromatic. The 2:1 resolution

ratio _{is especially} suitedtomultiresolutiondecomposition. Ranchinand Wald

(1996)

usethe methodto

2.2.3 Ratio

Methods

Anunfortunatecharacteristic ofCOStechniquesistheappearanceofeffectsofone multispectral

bandintoanother band dueto the coordinatetransformation. _Therefore, Munechika has labeled athird

category"fusion for radiometricintegrity." Withthese algorithms, primeimportance has been given to

properlyallocatingeachband'sdigitalcountsinthehybridimage. Several authorshave recognizedthat

computer segmentation algorithms will perform further operations on the digital counts. Therefore.

radiometric_accuracyiscritical.Thisthird_categoryoftechniquesis basedupontheratiomethod.

Theratio methodisastraightforwardapproachto_parcelingthelowresolution_energy intohigh

resolution pixels. Itworksbestonbandsthatare

highly

correlated. Onesimpleratio methodisattributed

toPradines (1986). He uses the

following

equation to merge the SPOT spectralbands with the SPOT

panchromaticband:

HRXS

=

LRXS

^

₍₄₎

^HRP

superpixel

whereHRXS isthedesiredHigh Resolution Multi-Spectral digital count, LRXSisthedigitalcountfrom

the Low ResolutionMulti-Spectral superpixel, andHRP is the digital count from the High Resolution

Panchromatic subpixel. Recall a subpixel refers to the small pixels in the high resolution image. A

superpixel correspondsto acollection ofsubpixelsthatisequivalentin sizeto the lowresolution pixels.

Pradines doesnobandtransformation ashismethodwasdesignedtofusethefirsttwoSPOTmultispectral

bandswiththe

highly

correlatedSPOTpanchromaticband.

Price₍₁₉₈₇₎proposesatwostage process. Heusesa ratioforthe_stronglycorrelatedbandsand a

Look-Up

Table

(LUT)

forthe_weaklycorrelatedbands. Hisratio equationissimilartoPradines'

HRXS,

=

LRXS,

HRXS'

(5)

Insteadof

directly

_using

HRP,

Price uses a regression routineto estimate thehigh resolution data. His

LRXS^afAN^+b,

(6)

The lowresolution dataandthe averagedPAN dataare used atthe coarse resolution to derive a set of

regressioncoefficients.Thosecoefficients areusedathighresolutiontopredicttheHRXS'term.

=aiPAN +

b1.

(7)

Thislinearregression compensatesforthespectraldifferencesbetweenthePANandlowresolutionbands.

Interestingly, Filberti et al. ₍₁₉₉₄₎ show that with proper choice ofA', their HPF (3) is very

similartoPrice'sratiomethod(5). Substituting.

LRXS

K

= ₍₈₎

LpAN

into(3)gives

HRXS

=

LRXS

₊

===-HPAN

'-PAN

=

LRXS\+-1-HPAN

*-PAN

_

LRXS_Tj

w 1

-~~j

\f-PAN

+"PAN

J

*-PAN

HRXS

=

^^-

PAN

'-'PAN

Ifthebandsare

highly

correlated.Price'sHRXS'willbe_verysimilartoPAN.

Furthermore,

when

Priceaveragesoverthesuperpixel,it isequivalentto_takingalowpassfilteredversion ofthePAN image.

Before

discussing

Price'sLUTforthe_weaklycorrelated _bands, we canshow the modifications

made

by

Munechika

(1990)

to the ratio equation. Munechika constructs a low resolution synthetic

panchromaticbandas

SYNPAN

=

y/.LRXS,

(10)

bands

He uses simulations to derive the coefficients

{\|/}

for combining TM bands 1-4 with the SPOT

panchromaticband. MunechikaranaLOWTRANatmospheric modelfor5reflectancecurves

(objects)

in

each of5 scenesin 3 differentatmospheric conditions. These 75 simulation results were combined with

the sensor models to create a set of coefficients that best describe the bands' spectral relationship.

Munechikausesanempirical modelfor histransformation.Munechika'sratioequationis

HRXS,=

LRXS'

HRP

(ID

SYNPAN

Munechikamakes another modification.Definetheratio

D

LRXS

R= ₍₁₂₎

SYNPAN

Typically,

hecalculatesthehybrid digitalcount as

HRXS

=R-HRP ₍₁₃₎

whereRistheratiofortheparticular superpixel.

However,

Munechikanotesina mixed pixel, someofthe

subpixelsare more

likely

tobesimilartoa_neighboringsuperpixelthan to thecurrent one. Heproceeds as

follows. ComparetheHRXS digitalcountto theSYNPAN(orPAN average) digital countin thecurrent

andadjacent superpixels.

Identify

thesuperpixelwiththevalueclosesttoHRXS. Usetheratiofromthat

superpixel in equation (13). In this approach, the hybrid average is no longer guaranteedto equal the

LRXS. Munechika found_tradingthislossof radiometric

integrity

improvedclassificationperformance.

Alltheseratio methods are similar. Pradinesusesa sumratherthan an average. Price uses an

unweighted estimateofthe data. Munechika creates a syntheticpan band

by

_{empirically modeling} the

specific sensor relationships. His SYNPAN includes a regression weighted

by

band. Munechika also

modifiestheratiosformixedpixels.Braun₍₁₉₉₂₎foundthePriceandMunechikaalgorithmshadsimilar

performance.

Developing

ahighresolutionhybrid imageinthe_weaklycorrelatedbands ismuch moredifficult.

The LRXSandHRPimagesare not

linearly

related.

Price₍₁₉₈₇₎suggests a

Look-Up

Table(LUT)approach.ThetablerelatesthePAN digitalcounts

to theLRXSdigitalcounts.Foreachvalue ofPAN avg(0-255for TM 8-bit

data),

Pricerecords each value

theHRPdigitalcount and extracts adigitalcountforHRXS'.Thisvalueisusedinhisratioequation(5).

Figure9showsanexample

look-up

table.

Pan

ava

DC

LRXSj

DC

0

8,8,7,9

1

20,24

2

16,14,15

255

Ava

LRXSj

DC

8

22

15

Figure9: Example

Look-Up

Table for

Fusing

Weakly

Correlated Bands

Munechikaet al.

(1993)

useadifferent approachtosolveforthe_weakly correlatedbands.

They

postulate a _{linear relationship between} the _weakly correlated bands, the panchromatic band, and the

previously predicted bands. At low resolution,digital countsfrom a neighborhood of6 superpixels are

usedtocalculatethecoefficientsinthe

following

regression

LRXSm

=a0+a,SYNPAN+

a^LRXS,

+...

(14)

where m refers tobandm

(weakly

correlated) and / refers toband /

(strongly

correlated and _previously

predicted).

These regression coefficients

{a}

are calculated_usingeach ofthe_previouslypredictedbands. If

theregressionerroris largerthanathreshold,anothertermisaddedto theequation

LRXSm

=a0+a.SYNPAN+_a2LRXSi +

a3LRXSj

+...

(15)

wherebandsiand

j

areboth_previouslypredictedbands.

Additionalterms

(bands)

are added until the regression error is reduced below the threshold.

Adding

termsuntiltheerrorcriterionissatisfied_{is why}the techniqueiscalledextendedregression. Once

theerror criterionissatisfied,thecoefficients areusedwiththehighresolutiondatatopredictthehybrid

image

HRXSm

=a0+a,

HRP

+

a2HRXS,

+...

where_HRXStis knownviatheratio method(1

1)

forthe_stronglycorrelatedband.

Oneproblem withextendedregression is ittends togive _noisyresults when applied touniform

areas of animage. Braun

(1992)

describesamodificationcalledglobal regression. Theglobal regression

algorithm recognizesthat local neighborhoods_may not bethe properdomain in which to perform the

regression. _Instead,theregressionisdonewith all pixelsthatcontainthesameclassof ground cover.

First an unsupervised classifier uses the low resolution multispectral and the high resolution

panchromaticimagestomakea class_mapof allthepixelsintheimage. Allpixelsareclassified asgrass.

trees,water,urban,etc. Whentheregression equation₍₁₅₎isapplied, _onlypixelsthat

belong

to thesame

ground coverclass areusedtocalculatethecoefficients. _But. thosepixels are not restrictedtobe inthe

localneighborhood;theycan comefromanywhereintheimage. Thusthenameglobal regression refersto

the regression

being

performed on pixels spaced _globally throughout the image. As _{before, previously}

predicted bands are

incrementally

added to the regression equation until the resultant error decreases

belowa threshold. Oncethecoefficientsare determined at low resolution, they are applied to the high

resolution data with equation (16). Both ratio methods and their extended and global regression

modifications are soundfusion algorithms.

They

usealineartransformation and a ratio tocombine the

data.

2.3 Algorithm

Summary

Chavezetal. ₍₁₉₉₁₎comparethreealgorithmsthat_separatelymanipulatethespatialinformation.

They

find the Intensity-Hue-Saturation transformation gives poor results since the panchromatic band

does not correspondwell to the

intensity

image. PrincipalComponents Analysis works bettersince the

panchromaticimage is more similarto thefirst principal component. Chavezet al. find the High Pass

Filteralgorithm worksbest.

Braun(1992)findsthealgorithmsthatmaintainradiometric

integrity

workbetterthanthosethat

separatelymanipulatethe spatialdata. Theratiomethodisthefoundation ofthisapproach. Braun finds

Braun finds extended regression works best overall when scenes contain high _frequency data (urban

areas).Theglobal regression worksbestwhenthesceneis lowor medium

frequency

(agriculturalareas).

In summary, image fusion works well _{estimating bands} that are strongly correlated with the

panchromaticdata. Theratio method combinedwithglobalregressiongivesthebestoverall performance.

Theresults arebestforscenes withmedium andhighspatial

frequency

content

Fusion doesnot workaswell onimageswithlow

frequency

information inthe_weaklycorrelated

bands.The

Intensity

Hue Saturationalgorithmisthemostrestrictive sinceitcan_{only be}appliedto three

bandsofdata. Onaverage,theHigh Pass FilterandPrincipal Componentstechniquesdonot performas

wellastheratiomethod.

Algorithms for image fusionwork_reasonablywell.

They

canpredictthehighresolutionspectral

response_{for many}pixels.Theperformance

degrades, however,

when_{predicting weakly}correlatedbands

orwhenthe pixels aremixed. Toaddressthese shortcomings, thealgorithms add complexity. The next

section presents analternate model basedupon spectral mixtureanalysis.

Recasting

_{image fusion} as a

spectral_mixingproblem usesthematerial reflectance curvestoensure allbands are_properly predicted.

The limitationsof spectral_mixingare addressedviaaprioriknowledge,constrainingthepossible solution

set, andproper algorithmdesign. The next sectiondescribesa_{framework for applying} spectral mixture

3.

Proposed

Algorithm

Typical imagefusionalgorithms generatehighresolutionimagesofdigitalcountsinthevarious

spectral bands. _However, digital counts do not reveal what kind of object is in the scene Spectral

unmixingmapsthe objects,butto_{date has only been done}atlowresolution.Thedesiredoutput productis

oftenahigh resolution material map. Suchaproductwould

identify

materials, andlocate them tohigh

accuracy. The contribution ofthis research is a method of_using spectral _mixing tools at high spatial

resolution.

By integrating

spectral_mixingandimage_fusion,ahighresolution material_mapisattained.

The proposed image fusion algorithm takes advantage of the spectral correlation between

materials

by

first

identifying

the materials via spectral mixture analysis. This results in low spatial

resolution material maps. Thesemaps arethensharpenedviaanonlinear optimization algorithm(Gross

and

Schott,

1996a). Thissection summarizestheprocedure.

The overall image fusion algorithm is a two stage process depicted in Figure 10. A set of

multi/hyperspectralimagesisrepresented

by

animagecube. _{Spectral unmixing}operatesontheimagesin

thecubetocreateaset of materialmaps atthe_(low)spatial resolution ofthemulti/hyperspectralimages.

Image fusionisaccomplished

by

_usingoneor more_{sharpening bands}toincreasethespatial resolution of

ImageCube Low Resolution

Material Maps Concrete

Trees Asphalt

Low

Resolution

High Resolution

Concrete Trees

Asphalt

Sharpening

Band(s)

High Resolution

MaterialMaps

Figure 10: Image Fusion DataFlow

-Creating

High ResolutionMaterial Maps

3. 1 Spectral

Mixture Analysis

Spectral Mixture Analysis (Adams et al., 1986. Smith et al.. 1990a. Adams et al.,

1993)

was

developedtoaddresstheproblem with_classifyingmixed pixels.Theauthors recognizedthelargescaleof

low resolution images creates situations where the measured digital counts are due to _many materials

within the _{corresponding} ground areas. Multispectral and hyperspectral

imaging

sensors provide the

abilitytoextract spectralsignaturesofindividualand mixtures of materials.

The first step in the algorithm generates material maps from the spectral sensor data. The

spectral mixture algorithm transforms digital counts, or radiance, into fractions of the constituent

materials,called endmembers. _{In many}ways,spectralmixture analysisissimilartoprincipalcomponents

analysis. "A

key

difference isthespectral mixture analysisdefinesafixedreference

(endmember)

thatis

(Mertesetal., _{1993). The primary}outputs of spectral mixture analysis are"fraction

images"

inwhichthe

intensity

canbe made proportionalto thepercentageofthatparticular endmember. This gives a spatial

mappingoftheendmembers.

As the mathematics will show, the spectral mixtureanalysis algorithm relies on

having

more

spectralbandsthanendmembers. _Therefore,thistechniquehasbeensuccessful_{in analyzing} multispectral

or hyperspectral images. To date,

however,

it has _{only been} applied to images with high spectral

resolution,but lowspatial resolution. _{Spectral mixing has only been}usedtomakelowresolution material

maps.

Spectral mixture analysis has been used with _{many different} sensors and has mapped _many

different materials.ThematicMapper_(TM) dataare usedto _{map desert}vegetation(Smith et al.. _1990b)

andtoevaluate sedimentinsurface waters(Merteset al.. 1993). AVTRISdataare usedtoseparategreen

vegetation, nonphotosynthetic_vegetation, and soils (Robertsetal., 1993), todistinguishsoil, grass, and

bedrock (Mustard, 1993), and tocalibrate apparent surface reflectance (Farrand et al., 1994). Spectral

mixture analysis ofTM dataand synthetic aperture radar data isusedto separatevegetationfrom rock

(Evans and

Smith,

1991). Analysis ofthermal infrared images is used to evaluate desert rock types

(Gillespie, 1992). Spectral mixtureanalysis isalso used to determinethe optical components ofinland

tropicalwaters(NovoandShimabukuro. 1994).Finally,image_processingandlinearsystems analysisare

usedtoimprovespectral_{mixing fraction images (Wu}and_{Schowengerdt,} 1993).

Themathematical modelforspectral_{mixing is}straightforward. Inaparticularpixel, forthe 1th

spectral

band,

let

dc,

=

gain, radiance,

+

bias,

(17)

wheredcaredigitalcountsrecorded

by

thedetector,gain andbiasareband dependentsensorvalues,and

radiance is the effective radiance at the sensor. Note that radiance is affected

by

the detector s

responsivity,

P(k),

radiance,

=

Spectralmixture analysis assumesthe radiance_reachingthedetectorcanbemodeledas a linear

sum of radianceduetondifferentendmembers

n

radiance(X)=

Le(X)fe

(W)

where _Le(k) is the spectral radiance ofthe particular endmember and _fe is the fraction of the e'

endmemberinapixel.

Combining,

th

n

radiance,

=

j

'2X(>0/ePA)<fc

=ZJ4WP,(^we

radiance,

=

^L,efe

(20) e=l

where

L^

istheeffective radiance ofthee*

endmember seen

by

thei*spectralbandofthedetector.

Thetermsgain, andbias,accountforsensoreffects._Typically,

imaging

systemsuse preflight or

onboard calibrationtocalculatetheselinearcorrectionterms. Again andbiasarereported

by

_{band along}

withtheimage data.

Assuming

the termsareconstantthroughout the _image, digitalcounts can_easilybe

convertedtoradiance.

Onecan correctAVTRISdata foratmosphericeffects(e.g.. Greenetal., 1993,Gao,etal., 1993).

If thedigitalcounts arecorrected,we_maythencasttheproblemintermsof apparentreflectanceofthee*

endmemberseen

by

the 1th

spectralband, R,,t

dc,

=

gain,

reflectance,+

bias,

(21)

reflectance,

=

R,Je

(22)

e=l

Thespectral_mixingcanbe done intermsof radiance(Equation

20)

or, if corrected, intermsof

reflectance (Equation 22). The choice of units depends upon the application. A reference

library

of

datathemselves,theunits couldbe inradiance ordigitalcounts. Theequationsthatfollowwillbewritten

intermsof_reflectance,buttheselectionisarbitrary.

Insummary,thespectral_mixingequation canbewrittenas

LRXS,=ttRlJ.

+_*, <23>

=i

where

LRXSj

are the i* band digital counts from the low resolution multispectral data converted to

reflectance _{units, Ruc} are the endmember reflectances from a stored library,

f

are the unknown

endmember_fractions, and e, are residualerrors_{arising from}errorsintheendmember_library, as well as

unmodeledhigherorder effects.

Solving

equation

(23)

resultsinn valuesoff,ateach pixel. The materialmaps are n different

plotswhich_mapanfeintoadigitalcount

(intensity)

range. Agreaterfractionof an endmembermakesthe

correspondingpixelbrighter.

Toimplement ₍₂₃₎ on animage of unknownendmembers, one searches a

library

of candidate

endmembers_{(reflectance spectra)}forthenendmembersthatwillminimizetheresidualerror. _Typically,a

library

of candidate endmembers isgeneratedforabroadnumber of materials. _{There may be L} n

potential endmembersinthelibrary. Various setsof n endmembers arechosen, andfractions generated

using equation (23). Almost all implementations ofspectral _mixinguse the same endmembers for the

entire scene. Inthis research, thealgorithmisextendedtounmix on a pixel

by

pixelbasis.

Accurate spectral _unmixing requires careful implementation (Adams et al., 1993).

Unmixing

spectral curvesisanill-posedproblem(Price, 1994).There isnounique set ofmaterialsthatcombineto

matchthe lowresolutiondata. Incolorscience, this samecharacteristic is knownas metamerism. Asa

result,theentire_unmixingprocessmustbeguided

by

significant userknowledge. Smithetal.

(1990a)

use

aniterativeapproach,stoppingwhenthenumberof endmembersisreasonable andtheestimation residual

isreducedtothelevelofimagenoise. Aprioriknowledge isessentialtomakethenumber ofendmembers

inthe

library

reasonable.

combinationofrealeffects andan errortermto accountforsome ofthedataspread. Ashade endmember

helpsreducetheerrorand preservesthe topographicinformationforplotting.

The

dimensionality

of remote _{sensing data is typically} much lessthanthe numberofspectral

bands in the sensor because ofcorrelation in the data. However, the additional bands give increased

confidence inthe unmixed fractions(Sabol etal., 1992). It alsoallows the residual errortobe used to

deducecharacteristicsof unmodeledendmembers(Robertsetal.. 1993).

Endmembers do nothavetocorrespondtoreal objects. Smithetal. _(1990a) describea_two-step

processinwhichthedigitalcounts arefirst relatedtoendmembers,andthentheendmembers are related

to reference endmembers. The reference endmembersare real objects whose spectra are measured in a

controlled environment.

Aligning

the endmembers allows for_calibrating the digital counts interms of

reflectance spectra.

Using

anatmosphericcorrection algorithm shouldgive good resultswithout_resorting

toreferenceendmembers.

Insome cases, the reflectance spectra do not combine

linearly

togenerate the radiance at the

sensor. Forexample, the_relationshipbetweenleafspectraand radiance isnonlinear duetothe layered

nature oftreecanopies. Nonlinearspectral_mixingmodelshavebeendevelopedtorelateleafspectra and

radiance (Roberts et al., 1990, Borel et al., 1991, and Borel and

Gerstl,

1994). However, in many

applications, one is _only concerned with

identifying

the canopies, not the _underlying leaves. In this

situation,thenonlinearmodelsare not needed since spectral_mixingcanbedonewithcanopy,ratherthan

leaf,

endmembers. Intrinsicmixturesare not suitableforlinearspectral unmixing.

Ifthescenecontainsmaterialsforwhichthereflectance spectra areunknown, forexample tree

canopies, itmaybepossibletoinferreflectance spectrafromthedata. Boardman et al.

(1995)

present a

method of _reducing the endmembers into those of interest and

"background."

Their methodology

identifies the "purest pixels"

by

assuming that combinations of endmembers result in spectrathat are

numerically intermediateto the spectraof pure endmembers. This is thesame assumption made in the

spectral mixtureanalysisalgorithm. Ifthepixeldigital counts are mappedinan m-band spectral _space,

Boardmanetal. notethat theextrema ofthisregion correspondtopurepixels.The/w-band coordinates of

thesepure pixels canbeusedasderivedreflectance spectrafortheendmembers.

Insummary, spectral reflectances of real world objects exhibit tremendousvariability.Temporal

effects are _especially evident in organic materials, illumination and view angle are _only partially

compensatedfor

by

theshadeendmember.Therestoftheeffectsresultinmore variationfromthe

library

endmember spectra. This variation ismanifested as errorsinthe material maps. Additionally, itis _very

difficult to _quantifythe results ofa spectral _unmixing algorithm since one requires a detailed ground

truth.

Tocontroltheenvironmentforthe algorithmdevelopmentreportedhere. SIGtools areusedto

completely define the images. Because the data are _{synthetically} generated, the _underlying unmixed

solutionis known.

Additionally,

perfectlyunmixedimagescanbecreatedtousein_testingthe_sharpening

process. This separatesthe _unmixing and _sharpeningsteps, andlets theeffects of eachbe

individually

analyzed.

3.2

Sharpening

Givena set oflowresolutionfractions produced

by

the _unmixingprocessdescribed above, the

next_{step is}to_{spatially locate}thefractionstotheresolutionofthe_{sharpening band(s). The}

difficulty

here

isthatthereare_manymore unknownsthanequations.Letsbethenumberofhighresolution subpixelsin

a single hyperspectral superpixel, and constrain the subpixels to contain the same n materials as the

superpixel.

Then,

therearesequationsandn*s unknowns.

Sharpening

isan underdetermined _problem

requiringanoptimization algorithm. Figure 1 1 showsasuperpixel, its correspondingsubpixels, andthe

ft1

iV

f.

f2

fs

Superpixel

w=₃

endmembers

>

fi6

f26

f36

5=₉

Subpixels

n*s=₂₁

unknown

fractions

Figure11: ThereAre

Many

High Resolution Unknowns inaSuperpixel

The sharpeningmodel takes the spectral_{unmixing form. The function} tobe minimized is the

residual errorinthesuperpixeldigitalcounts (calibrated intoreflectanceunitsasinthespectral _mixing

algorithm),

w;

=*;_// +s,,

j

=_\...s

(24)

where,

HRPj

isthej*elementinoftheHighResolution_{Panchromatic image corresponding}tosubpixelj.

R'pan,e containsthe

library

valuesforthereflectancesofthe appropriate endmembers in the _sharpening

band(s),andfeJ

isthehighresolutionfractionforthee*

endmemberinthej*subpixelfractionvector.

We desire the values offeJ that minimize equation ₍₂₄₎ while _{maintaining consistency} with the low

resolutionresults.

Consistency

requiresforeachmaterial,thehighresolutionfractionsmustaveragetothe

lowresolution

fraction,

-/.'=/., e=l..n.

(25)

7=1

It isconvenienttoviewthisproblemin"spreadsheet"form. Foran exampleof s=₄

subpixelsin

a superpixel, and n = ₃

endmembers from low resolution _{unmixing, the} n*s = ₁₂

representedinthecenterofthe spreadsheet_{(Table 1). The consistency}constraints requiretherowssumto

the appropriatevalues. Tosolvetheoptimizationproblem the twelvefractionsareadjusted tominimize

thesquared errorinthedigitalcounts,subjectto the_consistencyconstraints.

Subpixel 1 Subpixel2 Subpixel3 Subpixel 4

Consistency

Constraints

Endmember 1 -

^

ft2 ^ ft* f,4

4*f,

Endmember 2 fi1 : f22 fi>

-ft ,:::

4*f2

Endmember 3

:,:

f3l f*

::,,

&

fs4 _4*f,

Table1: SpreadsheetFormofHigh Resolution

Sharpening

Problem

Ifall s highresolution digital counts are _identical, thepixel isan aggregate mixture, and no

sharpening ispossible.Ifthepanchromaticdigitalcounts

differ,

the_sharpeningalgorithmadjuststhehigh

resolutionfractionstominimizetheresidual errorin(24).

3.3

Constraint

Conditions

The _preceding discussion presented spectral _unmixing and _sharpening as unconstrained

problems. However, the _underlying pixel represents a true mixture of one or more materials. Mixture

problemsare solved

by

_constrainingthecoefficients.

The unmixing literature contains examples _using each of the constraint strategies. Some

investigators use unconstrained _unmixing and screen out unrealistic solutions. Others use partial

constraints

by

requiringthefractionsto sumtounity. The

fully

constrained approach isusedless often.

Applying inequality

constraints makes thealgorithm more complex. The

fully

constrained case is also

likely

tobe susceptibletospectral variationbetween differentmaterial samples andbetweenthe samples

andthespectral

library

Definethreedifferentconstraint conditions. The first is unconstrained,where thefractionscan

takeon_anyvalueneededtominimizetheresidual error.

However,

theunconstrained_sharpeningproblem

The second condition is called _partially constrained. Here, the fractions within a pixel are

required to sum to unity. The sum is taken over all the materials in the appropriate pixel. At low

resolutionthereisoneconstraint

n

/.=!

(26)

e=l

while athighresolution, thisprovides5_equalityconstraints.

n

/.'=l,

J=_{l...S (27)}

e=l

However,only(5

-1)oftheconstraintsare_{independent. If equality}constraints wereincludedintheTable

1 spreadsheet,theywould require eachcolumn sumtoone.

A

fully

constrainedsetwouldalso require eachindividual fractiontoliebetweenzeroand one.

Thereare2*

inequality

constraints atlowresolution

0</e<l,

(28)

and2*n*s

inequality

constraints athighresolution

0<//<

1,7

=_1...*.

(29)

The

inequality

constraintsare not all_independent, butan optimization algorithm_onlyapplies theactive

inequality

constraints

during

_any search. _{Problems containing inequality constraints,} such as