Original citation:
Wilson, Roland, 1949- and Li, Chang-Tsun (1999) Multiresolution random fields and their
application to image analysis. University of Warwick. Department of Computer Science.
(Department of Computer Science Research Report). (Unpublished) CS-RR-361
Permanent WRAP url:
http://wrap.warwick.ac.uk/61088
Copyright and reuse:
The Warwick Research Archive Portal (WRAP) makes this work by researchers of the
University of Warwick available open access under the following conditions. Copyright ©
and all moral rights to the version of the paper presented here belong to the individual
author(s) and/or other copyright owners. To the extent reasonable and practicable the
material made available in WRAP has been checked for eligibility before being made
available.
Copies of full items can be used for personal research or study, educational, or
not-for-profit purposes without prior permission or charge. Provided that the authors, title and
full bibliographic details are credited, a hyperlink and/or URL is given for the original
metadata page and the content is not changed in any way.
A note on versions:
The version presented in WRAP is the published version or, version of record, and may
be cited as it appears here.For more information, please contact the WRAP Team at:
Appliation to Image Analysis
Roland Wilson
and Chang-Tsun Liy
Department of of Computer Siene,
University of Warwik,
Coventry CV4 7AL, UK
yDepartment of Eletrial Engineering, CCIT, Taiwan, ROC.
August 20, 1999
Abstrat
Inthispaper,a newlassofRandom Field,denedon a multires-olution arraystruture, is dened. Theseombine earlier, tree based
modelswith the more onventional MRF models. The fundamental statistial propertiesof thesemodelsareinvestigatedanditisproved thatthey an avoid someof theobvious limitationsof their
predees-sors, interms of modellingrealistiimage strutures. Predition and estimationfrom noisydataarethenonsideredanda newproedure: Multiresolution Maximum a Posteriori estimation, is dened. These
ideasare thenapplied to theproblemof analysingimages ontaining a number of regions. It is shown that the model forms an exellent basisforthe segmentationof suh images.
Keywords: Markov randomelds, image segmentation,
Bayesian estimation.
Amongthestatistialapproahestoimagemodelling,MarkovRandomFields
(MRF's)havebeenaroundaboutthelongest[28℄,[6℄,[14℄. Reently,however,
theyhavegainedsigniantattention [8℄ [10℄[18℄ [22℄[23℄ [25℄,espeiallyin
thesegmentationofregionsofmoreorlessuniformolourortexture. For
ex-ample,Gemanetal. [9℄usedthe Kolmogorov-Smirnovnon-parametri
mea-sure ofdierenebetweenthedistributionsofspatialfeaturesextrated from
pairs of bloks of pixel gray levels, with MAP estimation of the boundary.
Panjwanietal. [23℄ adopted anMRFmodeltoharaterise texturedolour
images in terms of spatial interation within and between olour planes. In
a tehnique whih is similarto that used in[27℄, Bouman and Shapiro used
sequential maximum a posteriori (SMAP) estimation in onjuntion with a
multi-salerandomeld(MSRF)[5℄,whihisasequene ofrandomeldsat
dierentsales. OtherworkexploringthemultiresolutionapproahtoMRF's
is desribed in [7℄, [1℄,[21℄,[16℄ and [24℄. The last two of these papers point
out the diÆulties in preserving the Markovian properties, whih require a
loality onstraint, within a sampling sheme whih impliesthe violation of
that onstraint. On the other hand, there is ampleevidene that
multireso-lutionproessingan leadtohighlyeÆientalgorithmsinmanyareas -from
image restoration tooptial ow.
TheupsurgeininterestinMRF'shasbeenpromptedbytheworkofBesag
[2℄andtheGemans[10,9℄,largelybeauseofthenewapproahestoBayesian
or Maximum a Posteriori(MAP) estimation. Although expensive
omputa-tionally,these algorithmsprovidearelativelysimpleway toapproahan
op-timal estimateusing the priniples of stohasti sampling,orMarkov Chain
Monte Carlo methods [11℄, asthey are sometimes alled. Alternatively, itis
oftenpossibletoget adequate resultswith deterministiproedures, suhas
Besag's IteratedConditionalModes(ICM)algorithm[3℄. Inanyevent,MAP
estimation based onnon-ausal MRF's suers fromseveral drawbaks when
applied to images. Apart from the onsiderable omputational burden, the
simpler models have `lowenergy' states whihrepresent uniform olourings,
so that if an MCMC algorithm is allowed to run long enough, it will tend
to produe results whih reetthis, espeiallyif the data are noisy. On the
other hand,deterministialgorithmssuh asICM anbeometrapped in
lo-alminima,alsoanundesirableproperty. Whiletherehas beenalotofwork
showing the eÆay of multiresolution methods, these have often been
are pronetothe blokingartefatsassoiatedwiththe samplingsheme.
At-tempts to avoid these problems tend to remove the link to the model, to a
greater orlesser extent, unfortunately.
The work desribed in this paper presents a new attempt to marry the
ideas of the MRF and multiple resolutions. As suh, it is more losely
re-lated to the work of Bouman than the other multiresolution approahes to
the problem. It also starts from a quadtree proess, in whih the passage
from oarse to ne sales an be desribed as a Markov Chain. It diers
fundamentally, however, in the proess by whih eah sale or resolution is
onditioned on its immediate predeessor in sale: whereas in the simple
model, a pixel ata given sale depends onlyon itsanestors insale, in the
new model, it is also diretly dependent on its neighbours at its own sale.
In eet, this makes the model a ross between a onventional MRF,
ap-plied at asingle resolution and the models proposed by Bouman and others
[7℄, [1℄, [21℄. This allows it to model image strutures, suh as multiple
re-gions havingsmoothboundaries, inamuhmorerealistiway thanprevious
stohasti image models. The next setion presents the bakground theory
of the new model. This is followed by a desription of its appliation to
the problemof segmenting images intohomogoneousregions fromunertain
data. The paper is onluded with a disussion of the new model and its
possibleextensions.
2 Multiresolution Random Fields
The featureof aMarkov RandomFieldwhihmakesitattrativein
applia-tions isthat the state of a given site depends expliitly onlyon interations
with its neighbours [10℄. We modelan image as a sequene of MRF's,
on-forming to a quadtree struture, with a nominal top level 0 and N levels
below that, level k having 2 k
2
k
sites for a nite image of size 2 N
2
N
pixels. Note that we order levels from 0 at the top of the tree to N at the
image level. The neighbourhood struture we impose onsists of n pixelsin
anisotropineighbourhood,suhasthestandardrstandseondorderMRF
models [2℄:
N ijk
=f(i l;j m;k);l 2
+m
2
R
2
Typially, we have hosen a radius R = 1 or R = 2, implying the 4 or 8
neighbours ommonly used in image proessing [12℄. In addition, we dene
the parent set, on level k 1, whih in the simplest ase onsists of the so
alled quadtree father
P ijk
=(bi=2;bj=2;k 1) (2)
whereb:denotesthe oorofarealnumber. Inmoreomplexases, wehave
used 8 neighbours on the same level and 4 onthe `father' level. The ruial
pointaboutany oftheseneighbourhoodsystemsisthat they implyausality
in sale: inotherwords, theproess atlevelk is onditionedonthat atlevel
k 1. This has importantonsequenes for the properties whihsuh elds
andisplay. Forexample,itfollowsimmediatelythattheeldisnotMarkov.
We shall only onsider the ase where pairwise interations are involved, so
that the onditionalprobability dening the RF an be written
P(X ijk
=mjfX
pqr
=n;(p;q;r)2N ijk [P ijk g)= Y (p;q;r)2N ijk [P ijk (X ijk =mjX pqr =n) (3)
where m;1 m M is the label at (i;j;k), is a normalising fator and
the pairwiseinterations are given by
(X ijk
=mjX
pqr
=n)=e a r +b r Æ mn (4) where a k ;b k
are onstants and Æ mn
is the Kroneker-Æ. Equivalently, the
model an be expressed in terms of Gibbs potentials [15℄:
U(! k j! k 1 )= X i;j V kjk 1 (! ijk j! i=2;j=2;k 1 )+ X (l ;m;k)2N ijk V k;k (! ijk ;! l mk ) (5)
The model enompasses both the quadtree model, for whih b k
=0and the
onventional`at'MRF,forwhihb k 1
=0. Notethatthemodelisbasedon
dierenes between labels: it speies that adjaent sites are more likely to
havethe samelabelthannot. Beauseof theprodutform,theonditioning
of X ijk
depends only on the number of labels of dierent lasses among its
neighbours. The onventional 2 D disrete MRFwas muh studiedin the
ontext of the Ising model for magneti spin, where it was shown that the
modelin2 Dhas asigniantfeature,namelythe ourrene ofritiality.
In thebinaryase, itwasestablishedthatfor largevaluesof theratiop=(1
p),wherep istheonditionalprobabilityP(X ij
=kjX mn
=k;(m;n)2N ij
to a predominant olouring of the lattie [15℄. The interesting feature of
the new model, whih is not shared by a onventional MRF, is that the
`lowenergy' (high probability) states of the model are not in general single
oloured: thefathersexertaforeontheirhildrenwhihdisouragesuniform
labellings. Forexample,ifthere exists onlevelk aregion ofonstantolour,
of arear 2
,then the4r 2
pixelsonlevelk+1whihrepresentthe sameregion
would inur apenaltyof order4r 2
fordisagreeing withtheir fathers,via (4);
for large r, this will greatly outweigh that inurred by the approximately
4r boundary pixels whih disagree with one or two of their neighbours on
levelk+1. This isquitedierent fromthe situationinaonventional MRF,
where any olouring whih is nonuniform inurs a ost proportional to the
boundary length, so that the only way to generate signiantly nonuniform
labellings is to `inrease the temperature', with the result that the typial
sample will have a random textured appearane. Although with the above
asymmetrial neighbourhoods, the eld isnot Markovian, if weonsider the
state of, say, level k of the eld onditionedupon that atlevelk 1
P(X k =! k jX k 1 =! k 1 )= 1 Z k e U(! k j! k 1 ) (6) where ! i
is the onguration of the wholeimage at level i and Z k
is the
so-alled partitionfuntion,then the Markovian/Gibbsian propertyis restored.
In eet, the onguration at level k 1 ats as an `external eld' for the
lattie at level k. Unlike a typial suh eld, this one varies at half the
sampling rate of the image,imposing the oarse struture on its hildren at
thenextlevel,whihaddsdetails,espeiallyintheviinityoftheboundaries.
This is demonstrated in the followingtheorem:
Proposition 1 Let P(! k
j! k 1
) dene a onditional Markov Random Field
through the potential funtion in (5) above, where the indexes i=2;j=2 are
taken asintegersandthe potentialsare, withoutloss of generality, zero ifthe
two states are identialand satisfy
V kjk 1
(njm) jN ijk
jmax n6=m
V k;k
(m;n)>0; m6=n (7)
Then the onguration! k
whih maximises P(! k
j! k 1
), for a given
ongu-ration ! k 1
, isgiven by
In other words, the most likely onguration on level k;! k
, is just a opy of
that on the level above.
Proof: To prove the assertion, note that eah site (i;j;k 1) on level
k 1 has four hildren on level k, eah of whih has the same state in
on-guration ! k
. Similarly, eah neighbour of eah of these sites will have a
state equal to that of its father on level k 1. Suppose that all of these
sites have the same state, m, say. Then any hange to another state at
(2i+q;2j+r;k); q;r2 [0;1℄ must inrease the potentialat that site, sine
itwillintrodueadisagreementbetween itselfand itsfather andneighbours.
It follows that only sites on the boundary between two regions with
dier-ent states need be onsidered. Any suh site (i;j;k 1) an have at most
jN i;j;k 1
j neighbours on level k 1 in a dierent lass. Similarly, eah hild
of suh a node an have at most jN i;j;k
j 2 neighbours in a dierent lass,
sine it will have at least 2 siblings whih are also in lass m. Hene the
maximum redution in potential from hanging suh a site ours when all
the neighbours havethe same state n 6=m and all 4hildren are hangedto
n fromm:
4U =4V
kjk 1 (njm)
X
(s;t;k)2N s ijk
V k;k
(n;m) (9)
where N s ijk
is theset of neighbours of (i;j;k)orany ofits3siblings. Butfor
any isotropi neighbourhood struture onlevelk
jN s ijk
j<4jN ijk
j 4 (10)
Hene if the potentialssatisfy (7), then
4U >4V k;k
(n;m)>0 (11)
So the hange in potential is positive and the state is less likely. Sine this
isthe worst ase, intermsof hangeinpotential,itfollowsthat the state! k is indeed the most likelyone, given !
k 1 .
Although the result is a simple one, its signiane should not be
over-looked: itshowsthatthestrutureobtainedatoarsesaleswillbepreserved
as details are added. Without suh struture preservation, there would
in-deed be littlepoint inhaving the multiresolution model.
Witha tighter onstraint on the father-hild potential, we an establish
k k 1
thepotential funtionin (5)above,wheretheindexesi=2;j=2are takenas
in-tegers andthe potentials are,without loss of generality,zero ifthe two states
are identialand satisfy
V kjk 1 2jN ijk jV k;k
>2klog 2
(12)
Then the opy onguration ! k
dened in (8) has a probability
P(! k
j! k 1
)>1 (13)
Proof: Toprovetheresult,itsuÆes tonotethatanyongurationdiering
from ! k
has a probability bounded below by that of a Bernouilli proess,
whose probabilities are dened by the potentials. This proess is dened as
follows:
ijk
=
0 if! ijk
=!
i=2;j=2;k 1
1 else
(14)
Nowthe assoiated potentialsare taken from the originalproess:
V kjk 1 =V kjk 1 ; V k;k =V k;k : (15)
Witheahonguration! k
isaorresponding`error'onguration k
. Eah0
in k
hasamaximumpotentialRV k;k
,whereR=jN ijk
jistheneighbourhood
size; this ours if all of its neighbours are dierent, while eah 1 arries a
ost noless than V kjk 1
jN ijk
jV kjk
, sine itmust at the very least disagree
with its father. It follows that the probabiltity of any onguration having
m 0'sis bounded belowby
P k (m) 1 Z e mRV k ;k (2 2k m)(V k jk 1
RV k ;k
)
(16)
whih an be expressed as
P k
(m)p m q 2 2k m (17) where p= e V k jk 1
2RV k ;k
1+e V
k jk 1 2RV
k ;k
(18)
Hene the probabilityof the ongurationwith 0 errors isjust
P k
(0)
1
(1+e 2RV
k ;k V
k jk 1 )
2 2k
2 2k
e 2RV
k ;k V
k jk 1
> (20)
then P k
(0)>1 , fromwhihthe result follows immediately.
It is perhaps obvious from the onstrution that this is a weak bound,
sinethevastmajorityoferrorongurationsatuallyinreasetheboundary
length. That has littleimpat, however, on the general onlusion that the
spread of the distribution around ! k
an be tightly ontrolled by the hoie
of interationpotentials.
Toillustratehowthefather levelaets theproess, gure3(a) shows an
array of 44 sampleimages generated from the same 1616 father image
usingdierentombinationsoffather-hildandneighbourinterations. Eah
3232 image within this array is the result of 2000 iterations, more than
suÆient for an array of this size to approah the stationary distribution.
Note that the image at the top left shows omplete randomness, as both
interations are 0 in this ase, while moving fromtop to bottom the
neigh-bour interation strength inreases and from left to right the father-hild
interation inreases. This shows rather learly the eet of the father over
a wide range of interation strengths: there is a doublingof the interation
potential for eah step to the right or downwards. Note that in this and
the othersample images, the image boundary pixelsare xed as`blak' (0),
so that, in the absene of father-hild interations, for the higher levels of
neighbourinteration,thestationarydistributionwillhaveamaximumwhen
all pixelsare blak. It an alsobe seen that allof the images inthe top row
have random, isolated pixels. This is beause, in the absene of neighbour
interations, the proess is Bernouilli, with eah pixel's olour being hosen
independently; as we move right along the row, however, the probability of
its olourbeing dierent fromits father's dereases geometrially.
Although the 4 neighbour eld is simple, it is rather limited in its
ability to produe smooth boundaries and so we also examined the use of
the 8 neighbourhood. Using the 8 neighbours as onditioning elements
produestheresultshowningure3(b). Inomparisonwith3(a),theregions
are notieablysmoother, with the gaps in the nononvex shape being lled,
but the same general trends an be observed as the interation strengths
inrease. For example, with greater potentials to the father level, small
regionstendtosurvivebetter. Thisisevidentintheimagesontherightside
of the array. Anotherobvious defet of the above models is that boundaries
problem is to make the inuene of the father level k 1 zero for any site
at a boundary (ie. having a neighbour of a dierentolour). This is simply
aomplished by setting the father-hild potential to 0 whenever the father
has aneighbour in adierent lass
V
ijkji=2;j=2;k 1
(njm)=0; if !
i=2;j=2;k 1 6=!
l mk 1
;forsome(l;m;k 1)2N
i=2;j=2;k 1 (21)
This isequivalenttoextendingthe set ofneighbours of(i;j;k)onlevelk 1
toalarger set ofpixels. Asa demonstrationof theeet of the twohanges,
it is simple to show that the most likely onguration on level k, ! k
, tends
to remove ornerspropagated from level k 1:
Proposition 2 Let ! k 1
be a onguration of level k 1 of a MMRF and
let ! k
be the orresponding most likely onguration on level k, based on
pairwise potentialsusing 8 neighbours andsuh thatfatherson boundaries
have interation potential 0with their hildren. Let (i;j;k 1) be a pixel on
level k 1, suh that
! ijk 1
6=m; ! pqk 1
=m; (p;q)2N 2 ijk 1
(22)
where m 2 [0;1℄ and the neighbourhood N 2 ijk 1
= f(i+d;j;k 1);(i;j +
e;k 1);(i+d;j +e;k 1), d = 1or d = 1 and e = 1or e = 1, are
neighbours of (i;j;k 1)within a blok of size 22 pixels. Suppose thatthe
onguration ! k
has the property that
!
2p+r;2q+s;k
=m; r;s 2[0;1℄; (p;q;k 1)2N 2 ijk 1
(23)
ie. thehildrenofthe3neighbourshavethesamelassastheirfathers. Then
!
2i+t;2j+u;k
=m; t= 1+d
2
; u= 1+e
2
(24)
In other words, the `orner' pixel (2i+t;2j+u;k) is more likely to hange
its labelthan to retain its father's lass. The eet isillustrated in gure 1
Proof: First, observethat sineeah of thepixelswithin the 22 blok on
levelk 1hasaneighbour inadierentlass,the interationpotentialswith
their hildren are 0. It follows that only the interations onlevel k need be
corner
at level
k-1
corner at
level k
m
m
m
m
n
n
n
Figure1: The`ornereet' inan8-neighbour MMRFwithboundaryeet.
The orner pixel at level k 1 is rened at level k to redue the boundary
energy.
Nextonsider the pixel(2i+t;2j+u;k)onlevelk. Ofits 8neighbours,
at most 3 have a labelother than m in onguration! k
. It follows that the
minimum energy labelfor (2i+t;2j+u;k) must be m.
Anoteworthy onsequene ofthis isthat the approximationof astraight
edge doesbeome less `jaggy' as the resolution inreases. As an illustration
of the eet of these modiations, gure 4 shows a sample from a binary
proess usingthe 8-neighbours onlevels5k 8,but a quadtree onlevels
0 k 4. By hoosing the model parameters appropriately as funtions
of the level, dierent ombinations of struture an be obtained. In this
ase, the oarsestruture representing the bottomlevelofthe purequadtree
proess isrenedby the 8 neighbourMMRF, resultingina single,smooth
`blob' representing the objet. Similar results were obtained after 10;000
iterations, indiating that 4 is in equilibrium. A seond example, using a
lower orrelation oeÆient between neighbours, is shown in gure 6. Note
that the large sale struture in this ase is puntuated by some smaller
`objets'. In both ases, however, the parameters of the eld are
super-ritialand the boundarysites ateahlevel arexed at0,sothe most likely
imageisblak. Thesampleautoorrelationfromasetof20imagesprodued
in a similar way is shown in gure 7; not surprisingly, this reets the large
9Neighbours 0.0093 0.0101 0.0178 0.0898
Father 0.0092 0.0104 0.0171 0.0303
Table 1: Predition error rates from level k 1 to k using all 9 neighbours
or opying the father level, forgure 4.
2.1 Predition and Estimation
The above observations suggest that a very high level of ompression ould
be ahieved in representing a sample from suh a proess. In eet, the
onguration !
k
, whih depends only on the level above, is an exellent
preditor for ! k
. This is illustrated by the dierene pyramid in gure 5,
whih shows the absoluteerror
k =k! k ! k k (25)
Indeed, itfollows fromproposition 1abovethat, underthe onditions of the
proposition, ! k
is the Maximum Likelihood prediition of level k from level
k 1. In moregeneralases,however, samplingwillbeneessary tond the
best preditor
^ !
kjk 1
=argmax ! k P(! k j! k 1 ) (26)
Figure 8 shows the result of simulated annealing over 200 iterations, with
a logarithmi shedule, to predit eah level of the 8 neighbour pyramid
of gure 4 from itsfather. Whilethe result is visuallyquite onvining, the
tableoferrorrates1showsthatthereisnoimprovementoverthe`opy'from
the father.
Apart fromits importanein ommuniations appliations, the question
of preditabilityalsorelates diretly tothe Gibbs distribution, whih iswell
knowntomaximiseentropyforagivenexpetedenergy[15℄. Theappropriate
entropy measure isthe entropy onditioned onthe neighbours
H k
(P)=
X ! ijk ;! lmn ;(l ;m;n)2N ijk [P ijk P(! ijk ;! l mn ) log 2 1 P(! ijk j! l mn ) (27)
Table 2 shows the sampleentropies fromthe bottom 5levelsof the pyramid
in4,onditionedboth onthe 8neighboursandononlythe father. Although
alltheentropiesaresmall,representingtheveryhighdegreeofpreditability
8 Neighbours 0.0026 0.0044 0.0037 0.0080 0.0293
Father 0.0758 0.0816 0.1206 0.1967 0.3005
Table 2: Conditional entropies in bits/pixel for various levels of the 8
neighbour MMRF.
level, over the predition from the father. More signiantly, it should be
noted that the entropy per pixel seems to be headed for 0 as the resolution
inreases. The reason for this is lear from the predition error images in
gure 5: the errors are onned to a narrow region around the edge of the
region. Sine the edge is tending to a smooth shape, it will be of length
O(2 N
), asN inreases, resultingin anentropy whihtends to zero
lim k!1
H k
(P)=0 (28)
Theothermainappliationofthemodelisinestimationfromnoisydata.
At its simplest, we might onsider the problem of estimating the image at
one level, X k
, say, from noisy data Y k
. If we know the image X k 1
, we an
use the ConditionalMaximum A Posteriori (CMAP) estimator
^ X k
=argmax X k P(X k jY k ;X k 1 ) (29)
where, from(6) above,
P(X k jY k ;X k 1 )= P(Y k jX k )P(X k jX k 1 ) P(Y k jX k 1 ) ; (30)
the rsttermontherightbeingthe likelihood. Whilethe CMAPestimateis
simple, thereare fewpratialappliationswhere theimages X k
;0k N
are available. Theobviousalternativeistheunonditioned MAPestimateof
all the levels
f ^ X i
;ikg=arg max fXi;ikg
P(X k
;X k 1
;:::::;X 0
jY k
) (31)
and so onfor X i
;i<k, whih an beexpressed via (6)as
P(X k
;X k 1
k 1 selet X
i
;i < k to simultaneously maximise the posterior with respet to
all k +1 images. The obvious weakness of this approah is that the data
Y k
onstrain X i
;i < k quite strongly and this ought to be built into the
estimation proedure. Thisisthe goalofthe multiresolutionMAP(MMAP)
estimator. For the above problem, we start again from the left side of (32),
but nowexpand as
P(X k
;X k 1
;::::;X m
jY k
)=P(X m jY k ) k Y i=m+1 P(X i jX i 1 ;Y k ) (33)
where we have used the Markov property of the sequene X i
and we may
start the estimation on a level other than 0. The interesting feature of this
estimatoristhat ithas asequentialstruture: rst estimateX m
, thenX m+1 and so on, up toX
k , using P(X i jX i 1 ;Y k )= P(Y k jX i )P(X i jX i 1 ) P(Y k jX i 1 ) (34)
Thedenominator,forxedX i 1
,isaonstant,butthereisagapbetweenthe
data Y k
and X i
. A short ut to the solutionis touse the opy onguration
X kji
, whih forparameters whih are superritialandsatisfy the onditions
of Theorem 1 represents a good preditor for X k
. This allows sampling to
beperformedonX i
. Inthe binary ase, anequivalent proedureistodene
data Y i
;i < k, by simply averaging over the blok of 2 k i
2
k i
pixels on
level k orresponding to eah pixel (p;q;i) on level i. It is not hard to see
that in this ase, we an replae (33) by
P(X k
;X k 1
;::::;X m
jY k
;Y k 1
;::::;Y m
)=P(X m jY m ) k Y i=m+1 P(X i jX i 1 ;Y i ) (35)
As a simple example, onsider gure 9, whih shows the pyramid obtained
from the image at the bottom level of gure 4, orrupted by additive white
Gaussian noise of unit variane. The pyramid, whih represents the `raw'
data for the estimator, was obtained using simple blok averaging, as in a
quadtree: eah pixelonlevelk is the averageof its4 hildrenonlevelk+1.
The estimation problem is to reonstrut the original binary pyramid from
these data. The data are onditionallynormal
p(Y ijk
jl)=N(l;v k
k k+1 8
use is anextensionof the stohasti(Gibbs Sampling) methodsdesribed in
[10℄, whih takes aount of the onditioning of level k by its father k 1.
Thus the MMAP estimateis dened by
^ X
ijk
=argmax m
P(X ijk
=mjY ijk
; ^ X
pqr
;(p;q;r)2N ijk
[P ijk
) (37)
where Y l mn
are the noisy data. There are two distint proedures for
im-plementing the estimator, whih we have dubbed MMAP and sequential
(SMMAP) inthesequel. TheMMAPmethodvisitseahlevelkoneah
iter-ation,thussamplingdierentsales`simultaneously'. Thesequentialmethod
iterates on one level at a time, with level k only being estimated after level
k 1 onverges, making it analogous tothe onditional MAP proedure.
The estimates were initialised by thresholding level 4 in the data
pyra-mid and thereafter using onditioned Gibbs sampling. After 100 iterations
ateahlevel,theresultingure10wasobtained. Theerrorratesatthe
var-ious levelsforeahestimatorare showninTable3below. Althoughonlyfew
iterations were used, the results at high resolutions are signiantly better
thanwereobtainedbysimplyopyingtheinitiallevelorusingaonventional
8 neighbour MRFestimator. Of the MMAP estimators, the MMAP
algo-rithm performedbetter than either CMAP or SMMAP. Closer examination
shows that the MMRF estimate gives a more or less onstant error rate of
50% per boundary pixel, aross a range of sales. This is beause the data
are unertain in these areas - averaging aross the boundary does not
im-prove this. Figure11 shows the number of sites hangingon eah iteration,
for eah of the 4 levels. It shows that after an initial burst at eah level,
oupyinga fewiterations, the samplersettlesdown toa steadystate where
only a handful of sites hange on any iteration. The next image 12 shows
that even with arandominitialonguration, the samplerquikly onverges
to a point where only a few sites hange on eah iteration. Note that one
iterationhere refers toa san-order visit toeverypixel ona given level.
We onlude this example by making the following observations:
1. Both MMAP estimators perform well on this problem - better than
those based onsimple MRFor low-resolution thresholding.
2. The relatively worse performane of the SMMAPalgorithmshows the
eet of onstraining the estimate at level k by a single realisationat
CMAP 0.0044 0.0084 0.0164 0.0332 0.0547
MMAP 0.0036 0.0074 0.0149 0.0332 0.0547
SMMAP 0.0047 0.0083 0.0156 0.0292 0.0547
8-neighbour MRF 0.0126 0.0094 0.0171 0.0332 0.0508
Copy from lev. 3 0.0155 0.0128 0.0178 0.0302 0.0547
Table 3: Error rates in MMAP estimates at various levels, ompared with
rate from 8-neighbour MRF and fromthresholding atlevel4 and opying.
3. Both are fast,requiringof the order of20 iterationsto obtain
satisfa-tory estimates.
4. However, sine the ost of an iteration at high resolution far
out-weighs that at low resolution, there is a omputational advantage to
the SMMAP approah beause it allows a tailoring of the annealing
shedule to eah level separately.
5. A further advantage of SMMAP isthat the estimateat level k an be
used toinitialise modelparameter estimation atlevelk+1, eg. using
the sampling method desribed in [17℄.
A more realisti ase is the image shown in gure 13, whih again is a
binary image with addedwhite Gaussiannoise at astandard deviation of1,
ie. equal to the dierene between blak and white. The estimation error
at the highestresolution, using the MMAP algorithmin this ase was1:3%,
better than most results reported on omparable problems in the literature
[5℄,[17℄,[27℄. The resulting estimate is shown in gure 14; apart from the
orners, where the modeldoesnot tthe data, theestimateis visuallyquite
good.
Inmoregeneralases,themeasurementsarenotalltheresultofaveraging
noisy binaryimage data,of the formof g. 9. Instead, letthe databegiven
by the pyramid fY i
;m i kg, where
P(Y k
;Y k 1
;::::;Y m
jX k
;X k 1
;::::;X m
)= k Y
i=m P(Y
i jX
i
) (38)
In otherwords,the data onlevel iare the resultof applyinganindependent
f ^ X i
;m ikg=arg max
fX i ;mikg P(X k ;X k 1
;::::;X m
jY k
;Y k 1
;::::;Y m
) (39)
Nowthe posteriorinthis aseiseasilyobtainedwith thehelpof(6)and(38)
P(X k ;X k 1 :::;X m jY k ;Y k 1
;:::;Y m )= P(X m )P(Y m jX m ) Q k i=m+1 P(X i jX i 1 )P(Y i jX i ) P(Y k ;Y k 1
;:::;Y m
) (40)
where the denominator is a onstant. This has signiant impliations for
how wemay obtain aMAP estimate: itleads us diretlyto the MMAP and
SMMAPalgorithmsdesribed above. Initialisationatlevelmisreadilydone
if we assume that the father-hild potential V mjm 1
= 0; alternatively, ML
estimation an be used at that level (asin the binary example of g. 9). In
the MMAP estimate, (40) an be used to sample simultaneously from the
posteriordistribution,whileinSMMAP,samplingatlevelkonlystartswhen
that on level k 1 terminates. This implies that, while SMMAP may give
anexellentapproximationto the MAPestimate, it isnot MAP, butin this
ase, MAP=MMAP.
2.2 Hidden Models
Whilethereare fewsegmentation tasksinwhihthisdisretemodeldiretly
reetsimageintensityorolour,itisveryusefulasahiddenmodel: thestate
of the site (i;j;k) ontrols the parameters of a loal image model dening
the harateristis within the region of 2 k
2
k
pixels assoiated with that
site. In that ase, there will be a measurement vetor Y ijk
assoiated with
the site, whih depends onthe label,ie
p(Y ijk
jX ijk
=m) 6=p(Y ijk
jX ijk
=n); ifm6=n (41)
The measurement vetor might represent a histogram of intensity or olour
or some suitable texture measure, for example. The Maximum Likelihood
(ML) estimatorfor the labelis then
^ X
ijk
=argmax m
p(Y ijk
jX ijk
=m) (42)
whih is simple, but ignores the prior probability. The MMAP estimatorin
inSMAP, weomputetheestimates sequentially oversale,startingatsome
oarsesalek min
,forwhihweusethe onventionalMAPestimate, obtained
by asimulated annealing proess
^ X
ijk
=argmax m P(mjY ijk ;fY pqk ; ^ X pqk
;(p;q;k)2N ijk
g) (43)
In eet, weare assumingindependene oflevelk min
fromlevelk min
1. At
subsequent levels, the labelling ^ X
ijk
is used toondition that atlevelk+1:
^ X
ijk
=argmax m P(mjY ijk ;fY pqr ; ^ X pqr
;(p;q;r)2N ijk
[P ijk
g) (44)
This gives the estimation a ausal diretionthrough sale, whilst using the
non-ausal, iterativeproess ofannealing ateahsale. Moreover, the`opy'
onguration isused as the initiallabelling atlevel k.
In many pratial appliations, using pairwise potentials and dierene
measurements, we end up with a normal model for the likelihoods, of the
general form p(Y ijk Y pqr j! k ;! k 1
)=N((! ijk ;! pqr );(! ijk ;! pqr
)); (p;q;r)2N ijk
[P ijk (45)
wherethenormalmeanandovarianeparametersdependonlyonthelasses
at the two sites. These parameters an be estimated on-line, given the
ur-rent lassiation atlevel k. This illustratesanother advantage of using the
multiresolutionapproah: althoughthe equilibriumdistribution willin
prin-iplebeapproahedfromanyinitialonguration,inpratie,itwillhappen
sooner if the initialonguration is lose to equilibrium. As with the prior,
the posterior distribution of ! k
willbe a Gibbs distribution onditioned on
the onguration on level k 1 and so sampling methods an be used to
loatethe maximum.
2.3 Appliation to Texture Segmentation
Initsappliationtotexturesegmentation,themodelishidden,witheahsite
on level k representing a square region of nominal size 2 N k
2
N k
pixels,
fromwhihtexture measurementsare taken, asin[9℄. In fat,windows with
a 50% overlap are used to redue estimation artefats. It is onvenient to
speify themodelintermsoftheGibbs potentials. Theinterationpotential
dening the MRFat level k in the tree is based onpairwiseinterations:
V ijk
(mjn) =a+bkY ijk Y pqr k 2 Æ mn
; (p;q;r)2N ijk
[P ijk
words, there isa ostbased onfeature similarity assoiatedwith sites inthe
same lass. Samplingis then basedon the orrespondingGibbs distribution
P(m)/e U(m)
T
(47)
wherethepositionindieshavebeensuppressedandT isthesaleparameter,
or temperature, whih is varied using a logarithmi annealing shedule [10℄
From these denitions, the SMMAP algorithmbeomes:
For level kk min
;k N
1. Sample at every site on level k using measurements Y ijk
and a
loga-rithmi annealing shedule, untilno hange is deteted over a number
I k
of iterations over the image at that sale.
2. Use labels on level k as the initiallabelling on level k+1, by opying
labels from fathers to hildren in the quadtree and to ondition the
simulation onlevel k+1.
The initiallabelling atlevelk min
is random.
While the above algorithm provides a general framework for
segmenta-tion, its eetiveness depends ritially on the texture desriptors used. We
havefourloalmeasurements,whiharebasedonthe`deterministi+stohasti'
deomposition,whihisageneralisationoftheWolddeompositionofsignals
[13℄. The four omponents are:
1. The dierene between the average gray level inthe bloks.
2. Two measures assoiated with the deterministiomponent, based on
an aÆnedeformationmodel
f s
( ~ )=f
s 0
(A 1
( ~
))~ +
s (
~
) (48)
where f s
(:) represents the path of an image entred at site s, site
s 0
= (l;m;k) is a 4-neighbour of site s = (i;j;k), A is that 2 2
nonsingular linear o-ordinate transformand ~ that translation whih
together give the best tintermsof total deformationenergy between
the two pathes. These are identied using the method desribed in
[13℄,whihmakesuse of loalFourier spetra alulated atthe
appro-priate sale using the Multiresolution Fourier Transform (MFT) [26℄.
(a) ThedeformationtermkA Ik representstheamountof`warping'
required tomaththe given path using itsneighbour.
(b) The error term k s
( ~ )k
2
is the average residual error in the
ap-proximation.
3. A measure for the stohasti omponent, based on dierenes in the
spetralenergydensitiesestimatedateahsiteviatheMFT,j ^ f(
~ ;~!;)j
2 , where ^ f( ~ ;~!;)= 1 p Z
d~xf(~x)w( ~ x ~ )e
|om:~~ x
(49)
isthe(ontinuous)MFTatspatialo-ordinate ~
,frequeny ~!andsale
[26℄, whihisapproximated byasampledversioninpratie. Thisis
similar to many texture lassiation methodsbased onloalspetra,
Gaborlters or autoovarianeestimates [27℄.
Eahofthesemeasuresissaledbytheorresponding(within-lassor
between-lass) sample variane and the four are added with appropriately hosen
weights togive the nal interation energy. Onlythe gray level diereneis
used for the father interation, however.
The neighbour onditional probabilities are estimated diretly from the
data during the samplingproess, as are the within-lass and between-lass
varianes. At levels k >k min
, the priors take into aount the lassiation
on the previous level,k 1: the prior probability that ahildhas the same
lass as itsfather isapproximated by
P(X ijk
=X
i=2;j=2;k 1
)=1
d
i=2;j=2;k 1
; (50)
where (i=2;j=2=k 1) is the father site, < 1 is a onstant and d s
is the
shortest distane between site s and a site having a dierent lass, ie. it
represents distane to the boundary. In the experiments reported below,
=0:5, implyingthat fathers have noeet atthe boundary,whihensures
that boundaries are not biased by the quadtree.
Inaddition,alineproesshasbeenintroduedtoinreasetheaurayof
the segmentationsusinganassumption ofsmoothnessof theboundary,sine
texture measurements require aminimum samplesize, whih wehave found
inpratietoorrespondtoasamplingintervalof44pixelswiththeabove
texture measures. The line proess is also based on pairwise interations
proessing is also a simulation designed to nd the Bayesian estimate, but
ours after the regions have been identied on a given level. Only region
sites having neighbours whih belong to a dierent lass are identied as
potentiallyboundary-ontainingandthe proessisrunonthosealone. From
these sites, a subset is seleted by stohasti labelling, using a potential
funtionwhihpenalisesurvatureinthe linejoiningtheestimatedentroids
of the putative boundarysegment ineahblok. The potentialhas the form
V(Y;Z)=(sinY 3
+sinZ 3
) 2 X
i=1 (Y
i Z
i )
2
(51)
wherethersttwovetoromponentsrepresenttheentroidposition(X 1
;X 2
)
and thethird omponentisthe angulardierenebetween theboundary
an-gleat(X 1
;X 2
)and thelinejoiningthe two entroids, asillustrated ingure
2. The entroid position and boundary angle at a site are estimated using
the MFT-based tehnique rst desribed in [26℄. In this way, both texture
and boundary features an be omputed within the same framework. Full
detailsan befoundin[19℄. Asummaryofthe boundarylabellingalgorithm
follows:
At eahtemperatureT:
1. For eah site i2B
2. Calulate the potential V(Y i
;Y j
); j 2N B;i
3. Sample fromthe Gibbs distribution todetermine the label i
Alogarithmiannealing sheduleisagainfollowedforthe boundary
proess-ing, whih runs after the region proessing is omplete at a given level. In
the presentsheme, noinformationispropagatedfrom`boundaryfathers'to
their hildren and sites in the boundary set B are labelledas eitherB or B.
This is asigniantly dierent modelfromthe lassi lineshemes based on
pixel labelling(eg. [10℄, asit isdesigned tofull adierentrole.
The experiments we have used to test the model demonstrate itsability
to segment textured images of various types, as an beseen from gures 15
and 16. Ingure15,therenementofthesegmentationthroughthe MMAP
proedure is evident, as is the improvement due to the boundary proess.
θ
1
2
θ
l
Figure2: `Distane' between boundary segments
Table 4: Segmentationerror ratesand numberofiterationsperpixel(# i/p
) for imageof g. 15.
level C Region Proess Boundary Proess
k Error rate (%) # i/p Error rate (%) # i/p
3 8 7.053 0.189 3.079 0.018
4 6 1.640 0.074 1.059 0.009
5 4 1.265 0.349 0.485 0.016
6 3 0.716 2.191 0.365 0.045
Total # i/p 2.803 0.088
error rate drops to less than 1% with the boundary proess at the highest
resolution andthisisahieved atanormalisednumberofiterationsperpixel
of only 2. This gure is the sum of ontributions from the various levels,
eahweightedbythe numberofpixelsonthat level. Notethatthealgorithm
terminates2levelsabovetheimagelevelbeausethisisthehighestresolution
for whihwe an obtain meaningfultexture and boundaryestimates.
In the seond gure, a summary of the high resolution segmentations is
shown, for several ombinationsof twoor moretextures. It shouldbenoted
that noadditionalinformationonthe numberoftexturedregionsisrequired
bythealgorithm-itisompletelyunsupervised. Thesepituresillustratethe
eetiveness ofthe overall tehniqueandthe utilityofthe boundaryproess,
whih both improves the subjetive quality and lowers the mislassiation
rates tobeamong the best reported inthe literature - typiallyof the order
of 1 2%. The test images were 256256 pixels, with the textures taken
from Brodatz'sbook. Beause of the multiresolutionestimation, the overall
numberofiterationsrequiredtoattainonvergenewaslow-intheexamples
shown ingure 16,the numberof iterations/pixel wasof the order of 4.
We have ompared these results with those presented by a number of
authors, inluding [16℄ [4℄,[5℄, [17℄,[27℄ and [20℄. The results presented here
3 Conlusions
In this paper, we have presented a new model for image analysis, whih
ombinesthenotionsofmultipleresolutionsandMRF'stoprovideapowerful
way ofdesribing imagestruture statistially. Correspondingly,a new form
ofMAPestimator-theMultiresolutionMAPestimator-waspresented. The
model wasillustrated with examplesof image segmentation, in whih it has
been shown to be among the most eetive methods yet desribed for the
task. The advantages of the new modelmay be summarised as:
1. Byonditioningthe MRFat levelk by that onlevelk 1,uniform
la-bellings are nolongerthe `ground' state of the model. This avoids one
ofthemostobviousweaknesses ofonventional MRFmodels. Byusing
an appropriateneighbourhoodand onditioning the father-hild
inter-ationsonthe preseneof boundaries,itispossibletotrade o
bound-ary smoothness against the degree of struture preservation. This is
a ompletely new feature of the model, whih it does not share with
previous image models.
2. Thenalstateatlevelk,aswellasonditioningtheMRFatlevelk+1,
an be used as aninitial state at level k+1, simplyby opying labels
from fathers totheir four quadtree hildren. Although the nal MAP
estimate should be independent of the initialstate, the time taken to
get there is aeted by the initialisation. Using the labelling in this
way speeds up omputation.
3. ByappropriatelyombiningthespatiallyinvariantMRFstruturewith
the quadtree,the blokingandnon-stationarityartefatsofthatmodel
are greatly redued.
4. Themodelparameters,whihgenerallyareunknown, anbeestimated
for the higher resolutions by using the segmentation obtained at the
oarser sales.
5. Againbeauseof the onditioningby oarsesales, the resultsare not
before it an be onsidered omplete. For example, the line proess whih
wasusedtoimprovetheestimateoftheboundary,doesnotinteratwiththe
region labelling. Similarly,thesegmentation modelhas not been testedwith
other image features. Work is urrently underway toaddress these issues.
Aknowledgments
The authors wish to thank Chung-Cheng Institute of Tehnology in Taiwan
for supporting this study.
Referenes
[1℄ M.Basseville,A.Benveniste,K.C.Chou,S.A.Golden,R.Nikoukah,and
A.S. Willsky. Modelling and estimation of multiresolution stohasti
proesses. IEEE Transations on Information Theory, 38(2):766{784,
1992.
[2℄ J. Besag. Spatial Interation and the statistial Analysis of Lattie
Systems. Journal of the Royal Statistial Soiety (Series B), 36:192{
236, 1974.
[3℄ J. Besag. On the StatistialAnalysis of Dirty Pitures. Journal of the
Royal Statistial Soiety (Series B), 48(3):259{302,1986.
[4℄ C. A. Bouman and B. Liu. Multiple Resolution Segmentation of T
ex-tured Images. IEEE Transations on Pattern Analysis and Mahine
Intelligene,13(2):99{113,1991.
[5℄ C. A. Bouman and M. Shapiro. A MultisaleRandom Field Model for
BayesianImageSegmentation. IEEETransationsonImageProessing,
3:162{176, 1994.
[6℄ R.ChellappaandR.L.Kashyap. TextureSynthesisusing2-Dnonausal
autoregressive models. IEEE Transations on Aoustis, Speeh, Signal
thesis, Department of Computer Siene, The University of Warwik,
UK, September 1988.
[8℄ F. S. Cohen and Z.Fan. Maximum LikelihoodUnsupervised Textured
Image Segmentation. Computer Vision, Graphis, and Image
Proess-ing, 54:239{251, 1992.
[9℄ D. Geman, C. GraÆgne S. Geman, and P Dong. Boundary Detetion
by Constrained Optimization. IEEE Transations on Pattern Analysis
and Mahine Intelligene, 12:609{628,1990.
[10℄ S.GemanandD.Geman.Stohastirelaxation,Gibbsdistribution,and
Bayesianrestorationof images. IEEE Transations on PatternAnalysis
and Mahine Intelligene, 6:721{741,1984.
[11℄ W. R. Gilks, S. Rihardson, and D. J. Spiegelhalter. Markov Chain
Monte Carlo in Pratie. Chapman and Hall,1996.
[12℄ R. C. Gonzalez and R. E. Woods. Digital Image Proessing. Addison
Wesley, 1992.
[13℄ T. I. Hsu and R. G. Wilson. A Two-omponent Model of Texture
for Analysis and Synthesis. IEEE Transations on Image Proessing,
7(10):1466{1476, 1998.
[14℄ A. K. Jain. Fundamentals of Digital Image Proessing. Prentie Hall,
1989.
[15℄ R. Kinderman and J. L. Snell. Markov Random Fields and Their
Ap-pliations . AmerianMath. Soiety, 1980.
[16℄ S. Krishnamahari and R. Chellappa. Multiresolution Gauss-Markov
Random Field Models for Texture Segmentation. IEEE Transations
on Image Proessing,6(2):251{267,1997.
[17℄ S. Lakshmanan and H. Derin. Simultaneous Parameter Estimation
and Segmentation of Gibbs Random Fields Using Simulated
Anneal-ing. IEEE Transations on Pattern Analysis and Mahine Intelligene,
ology For Parametri Image Models. IEEE Transations on Pattern
Analysis and Mahine Intelligene, 17:211{217, 1995.
[19℄ Chang-Tsun Li. Unsupervised Image Segmentation Using
Multiresolu-tion Markov Random Fields". PhD thesis, University of Warwik, UK,
1998.
[20℄ C. S.Lu,P.C. Chung, and C. F.Chen. Unsupervised Texture
Segmen-tation via wavelet Transform. PR, 30(5):729{742,1997.
[21℄ M.R. Luettgen, W.C. Karl, and A. Willsky. Multisale representations
of Markov Random Fields. IEEESP, 41:3377{3395,1993.
[22℄ B.S.ManjunathandR.Chellappa.UnsupervisedTextureSegmentation
Using Markov Random Field Models. IEEE Transations on Pattern
Analysis and Mahine Intelligene, 13:478{482, 1991.
[23℄ D. K. Panjwani and G.Healey. Markov Random FieldModels For
Un-supervised SegmentationOfTextured olorImages. IEEE Transations
on Pattern Analysis and Mahine Intelligene,17:939{954, 1995.
[24℄ P.PerezandF.Heitz.RestritionofaMarkovRandomFieldonaGraph
and MultiresolutionStatistialImage Modeling. IEEE Transations on
Information Theory, 42(1):180{190,1996.
[25℄ R. Szeliski. Bayesian Modelling of Unertainty in Low-level Vision.
Kluwer AademiPublishers, 1989.
[26℄ R. Wilson, A. Calway, and E.R.S. Pearson. A Generalised Wavelet
Transform forFourier Analysis: The MultiresolutionFourierTransform
and Its Appliationto Image and Audio Signal Analysis. IEEE T
rans-ations on Information Theory,38(2), Marh 1992.
[27℄ R. Wilson and Mihael Spann. Image Segmentation and Unertainty.
Researh Studies Press, 1988.
[28℄ J.W.Woods.Two-dimensionalDisreteMarkovianFields.IEEET
(a)
(b)
Figure 3: Illustratingthe eet of father-hild interations. Eah of the 16
sub-images is sampledat32*32 pixelsusing (a)4-neighbours and the father
as the onditioning elements and (b) using 8-neighbours and father. From
top to bottom, the neighbour interation energy inreases, while from left
dierene between eah level of the pyramid and the `opy' from the level
above. With superritial parameters, the MRF ats to rene the existing
The number of sales is 8, with the bottom level image being 128 by 128
pixels. The neighbourhoodsize was 8forthe bottom 3levelsof the pyramid
and 500 iterations were run at eah of these sales. The top 4 levels were
20
40
60
20
40
60
400
600
800
20
40
60
Figure 7: Sample autoorrelation from 20samples of a binary MMRF. The
number of sales is 8, with the bottom level image being 128 by 128 pixels.
Theneighbourhoodsizewas8forthebottom3levelsofthe pyramidand500
iterations were run at eah of these sales. The top 4 levels were generated
50
100
150
200
20
40
60
80
100
Figure 11: Plot of the number of hanges on eah iteration of the sampler,
for levels8 (top urve)up to5(bottom urve).
20
40
60
80
100
20
40
60
80
100
Figure 12: Numberof hanges oneah iteration,for random initial
