Inferen e in Loopy Gaussian Graphi al
Models
?
K.H. Plarre and P.R. Kumar a;
a
Department of Ele tri al and Computer Engineering, and Coordinated S ien e
Laboratory, University of Illinois at Urbana-Champaign
Abstra t
We onsider message passing for probabilisti inferen e in undire ted Gaussian
graphi almodels.Weshowthatforsingly onne tedgraphs,messagepassingyields
an algorithm that is equivalent to the appli ation of Gaussian elimination to the
solutionofaparti ularsystemofequations.Thisrelationprovidesanaturalwayof
extendingmessage passingto arbitrary graphswithloopsby rst studyingtheop-
erationsrequiredbyGaussianelimination.Wethusobtaina nitetime onvergent
algorithm that solves the inferen e problem exa tly and whose omplexity grows
graduallywith the\distan e"of thegraphto a tree. Thisalgorithm an be imple-
mented ina distributedfashionat nodes throughmessage passing,asfor example
insensornetworks.
Key words: Graphi al models,message passing,loopygraphs, probabilisti
inferen e,Gaussianinferen e
?
ThismaterialisbaseduponworkpartiallysupportedbyUSAROunderContra t
Nos. DAAD19-00-1-0466 and DAAD19-01010-465 , DARPA under Contra t Nos.
N00014-01-1-0576 and F33615-01-C-1905, AFOSR under Contra t No. F49620-02-
1-0217, DARPA/AFOSR unfer Contra t No. F49620-02-1-0325 , and NSF under
Contra t No. NSF ANI 02-21357. Any opinions, ndings, and on lusions or re -
ommendations expressed in this publi ation are those of the authors and do not
ne essarilyre e t theviewsof theabove agen ies.
Universityof Illinois at Urbana-Champaign,CSL, 1308 West Main St., Urbana,
Illinois,61801.
Email addresses: plarre ontrol. sl.uiu .edu(K.H. Plarre),
prkumar ontrol. sl.uiu .edu(P.R. Kumar).
URL: http://bla k1. sl.uiu .edu/
~
prkumar (P.R. Kumar).
Given a set of random variables, the problem of probabilisti inferen e an
be astas one of omputingthe posterior probabilityof asubset of variables,
given the values of another subset; see for example [9℄. When the number of
variables is large, inferen e requires integration over high dimensional spa es
and an easily be ome intra table.
In some ases, there are several onditional independen e relationships be-
tween setsof randomvariables.The olle tionof allsu h onditionalindepen-
den erelationsgivesrisetoafa torizationofthejointprobabilitydistribution
intoaprodu toffun tions,ea hofwhi hdependsonasubsetofthevariables.
Thisfa torization ansigni antlyredu ethe omplexityofinferen e.Thefo-
us of this paper is on eÆ ient probabilisti inferen e for su h olle tions of
randomvariables.
In graphi al models, all su h onditional independen e relations of a set of
randomvariablesare en odedinagraph.Ea hnode inthegraphrepresentsa
randomvariableandtheindependen erelationsareen odedintheedges.The
graph an be dire ted (e.g., Bayesian networks) or undire ted (e.g., Markov
random elds).Inthispaperwefo usonundire tedgraphs.TheHammersley-
Cli ordtheorem(e.g.,[3,12,9℄)providesthe onne tionbetweenindependen e
and fa torization: A stri tly positiveprobability distributionsatis es all on-
ditionalindependen erelationsimpliedbythegraphG,ifandonlyifitfa tors
a ording to the maximal liques of G. When the underlying graph is singly
onne ted(thereisatmostonepathbetweenanypairofnodes,i.e.,itisatree
ora forest), eÆ ientalgorithmsexistthat solvethe inferen e problem;[12,5℄.
Inparti ular, message-passing algorithms(e.g.,[12℄) asso iate the nodes with
individual pro essors that an perform \lo al" omputations and ommuni-
atewithea hotherthrough \messages."Themessages onverge aftera nite
number of steps, after whi h ea h node has orre tly omputed its own pos-
terior distribution.
When the graph has loops, the same message passings heme an beapplied,
but the algorithm is not guaranteed to onverge, and if it does, it will, in
general,not onverge tothe orre tposteriordistributions;see [12,9℄.Despite
this, message-passing algorithmshavebeen appliedwith enormous su ess in
loopygraphi almodels,asawayofapproximatingtheposteriordistributions;
see, for example [6℄.
Gaussian)theproblemofprobabilisti inferen eredu es tothatof omputing
the posterior mean and ovarian e. The problem is linear [10℄, and the algo-
rithmtakesasimpli edform[16℄.In fa t,the posteriormean anbeobtained
asthe solution toasystem oflinear equations,while the posterior ovarian e
an be obtained as the inverse of a matrix. Re ently, message passing algo-
rithms have been studied on Gaussian graphi al models with loops { on the
turbo-de oding graph with Gaussian nodes in [13℄, and for arbitrary graphs
withGaussiannodesin[16℄.Inboth asesitisshown thatwhenmessagepass-
ing onverges, the omputed posteriormeans are orre t, but the ovarian es
are wrong.Also, both papers providesuÆ ient onditions for onvergen e. In
[15,14℄aniterative\EmbeddedTrees"(ET)algorithmfor omputing thepos-
terior means and ovarian es is presented. At ea h step, a modi ed inferen e
problem is solved on a spanning subtree of the graph G, and the omputed
meanand ovarian eisusedinthesubsequentiteration.Itisshown thatifthe
algorithm onverges, itdoes so geometri ally,to the exa t means and ovari-
an es. In [14℄, the ET algorithmis studied. The relation of the ET algorithm
todi erent iterativemethodsfor thesolutionof systemsof linearequationsis
found. In parti ular, itis shown that the ET algorithm an beused to derive
a nite time algorithm that omputes the posterior means in time O(dNE)
and the ovarian esin time O(dNE 2
). Here N is the numberof nodes inthe
graph G, E is the minimum number of edges that need to be removed from
G torevealan embedded tree, and d is the dimension of the nodalvariables.
It isimportanttomention that the ET algorithm an be appliedtoarbitrary
graphi al models, not only Gaussian, providing a re ursive algorithmfor in-
feren e.
We onsidermessagepassingforgeneralGaussiangraphi almodels,thosewith
loops,orwithoutloops(i.e., atree). Wefo usons alarrandomvariables,but
the results presented an easily be generalized to random ve tors. We show
that when the graph is a tree, the resulting algorithm is equivalent to the
appli ation of Gaussian elimination to the solution of a parti ular system of
equations.Itis known thatina rootedtree (atree inwhi ha parti ularnode
hasbeen hosentobetheroot),inferen e anbeperformedintwosteps:A rst
\ ne-to- oarseKalman lteringstep,"equivalenttoGaussianelimination,(in
the message passing ontext, this step orresponds to messages sent starting
from the leaves towards the root), and a ba ksubstitution or \Rau h-Tung-
Striebel smoothing step;" see [4,17,11℄. We show how message passing solves
theinferen eproblembysendingmessagesdire tlyinthe nonrootedtree,and
how this is equivalent to Gaussianeliminationin many rooted trees. In fa t,
the inferen e problem an be solved by onsidering N rooted trees, ea h one
rooted ata di erent node, and applying only the Kalman lter step on ea h
of them. Messagepassing performs these omputations simultaneously inthe
nonrooted tree.This observation not onlyfa ilitatesthe understanding ofthe
passinginanaturalway tohandlegraphswith loops. Thus weobtaina nite
timealgorithm onverginginO(NL)stepstothe orre tposteriormeans,and
an algorithm that omputes the posterior means and varian es in O(NL 2
)
steps, where L is the number of of nodes that are not isolated inG T (the
subgraph of G remainingafter the edges of T havebeen deleted), where T is
aspanningsubtree ofG.Althoughobtained throughadi erentapproa h,the
omplexity ofour algorithmsisequivalenttothatof the algorithmspresented
in[14℄, when d=1.
InSe tions2and3wepresentthedetailsofthegraphi almodelswe onsider,
the assumptions we make, and introdu e some notation. Se tion 4 presents
expressions for the messages in Gaussian graphi al models. In Se tion 5 we
show that when G is a tree, the system of equations to be solved has a very
simple stru ture, afa t that ismentioned in[17℄, and in Se tion7 we exploit
this stru ture to solve the system through Gaussian elimination, and show
that the algorithm we obtain is equivalent to message passing. We in lude
a brief review of Gaussian elimination in Se tion 6. In Se tions 8 and 9 we
use this equivalen e toextend the messagepassing algorithmto ompute the
posterior means and varian es, respe tively, in graphs with loops. Se tion 10
presents the proposed algorithms as pseudo- ode. We in lude a entralized
versionthat anbeimplementedonasingle pro essor,and adistributedone,
that an be implemented in, for example a sensor network, where ea h node
is, infa t, aseparate physi alunit. In Se tion 11westudy the omplexityof
the proposed algorithms, and we on lude with some remarks inSe tion 12.
2 The setting
We follow losely the setting and notation in [15℄ and [16℄. We onsider a
Gaussiansto hasti pro ess given by an(unobserved) IR N
valuedstate ve tor
x
~
N(0; 1
),withprobabilitydensityfun tionp(x)/ exp n
1
2 x
T
x o
where
= T
> 0. We asso iate with x an undire ted graph G = (V
G
;E
G
) whi h
ontains N nodes, ea h indexed by an x
i
, and with an edge onne ting x
i
and x
j
for i 6= j if and only if [℄
i;j
, the i;j-th entry of , is nonzero. The
mannerinwhi hand Ghavebeen de ned makesxMarkov withrespe t to
G, i.e., for any three subsets S
1
:= fx
i
1
;:::;x
i
n1 g, S
2
:= fx
j
1
;:::;x
j
n2 g, and
S
3 :=fx
k1
;:::;x
kn3
g,ifremovingtheverti esinS
2
fromG ompletelyisolates
S
1
from S
3
, then the variables in S
1
are mutually onditionally independent
of the variables inS
3
, given S
2 .
For simpli ity (only) we suppose that ea h node in G orresponds to a om-
ponent of x, and not to a subve tor as in the more general ase. To ea h x
i
h
h
h h
h
h h
x x
x
x x
x
x
x
2
2 6
x
1
y
1
y
3 y
4 x
4
x
5
y
5
y
7 x
6
x
7 x
3
Fig.1.ExampleofaGaussiansto hasti pro essde nedonaloopygraph.Variables
arerepresentedby ir les. Observed variables are oloredbla k.
orresponds anoisyobservation y
i
su h thatthe observation ve tory satis es
y=Cx+v,v
~
N(0;R ).WeappendtoG,N morenodesindexedbyy
1
;:::;y
N ,
and edges (x
i
;y
i
) for 1 i N. We all this graph as
G = (V
G
;E
G ). In
G,
sin e ea h y
i
is onne ted only to x
i
, it implies that the random variables
fy
i g
N
i=1
are onditionally independent given x, whi h in turn means that C
and R are diagonal matri es. Figure 1 shows an example of su h a graphi al
model,wheretheobservednodes,thenodes orrespondingtoy
i
for1iN,
are olored bla k. We allthis a Gaussiangraphi almodel.
The goal of the inferen e problem is to determine the onditional marginals
p(x
i
jy),i.e., the posterior probability of ea h x
i
given the observations, when
,C andRaregiven.LetP := 1
.Asisstandard [16℄,we onsider =P 1
rather than P as given, sin e the joint probability distribution of x is given
by the oeÆ ientsof . Sin e the jointdistribution isGaussian,the posterior
distribution is alsoGaussian, and it suÆ es to determine the posterior mean
^
x and ovarian e
^
P.It is known that x^and
^
P satisfy 1
^ x =
^
PC T
R 1
y;
^
P = h
P 1
+C T
R 1
C i
1
:
(1)
Note that, sin e the onditional error varian es are the diagonal elements of
^
P, solving (1) is, in fa t, more general than solving the inferen e problem as
statedabove,be ausethe ompleteposterior ovarian eis omputed,andnot
justitsdiagonalelements.Here weare interested onlyin omputing the diag-
onal elements of
^
P.
1
From the expression for y, we see that, onditioned on x, y is a Gaus-
sian random ve tor with mean Cx and ovarian e matrix R . The on-
ditional density of x given y is given by p(xjy) = p(yjx)p(x)=p(y) /
exp
1
2
(y Cx) T
R 1
(y Cx)+x T
P 1
x
. Completing the square in this ex-
pressionwe ndthat onditioned ony,xis aGaussianrandomve tor with ovari-
an e
^
P =
P 1
+C T
R 1
C
1
and meanx^=
^
PC T
R 1
y.
(1) The nodes of
G orrespond tothe statevariablesx
i
and observations y
i .
To simplify the notation, we sometimes use indi es toindi ate the state
nodes,e.g.,\nodei"means\nodex
i
",and\edge(i;j)"referstothe edge
joiningx
i
and x
j
.We namethe observations asy
i .
(2) N
i
denotes the set of indi esof allstate variables that are neighborsof i
in G, i.e., N
i
=f1 j Nj(i;j) 2E
G g. N
i;j
is the set of indi es of all
state variables that are neighbors of x
i
, ex ept x
j
,i.e., N
i;j
=N
i nfjg.
(3) If M is a matrix, [M℄
i;j
and [M℄
i;
denote the (i;j)-th omponent and
thei-th rowofM,respe tively.Likewise[w℄
i
denotes thei-th omponent
of ve tor w.
(4) To simplify the presentation, we de ne the following matri es: V
x;x :=
^
P 1
, V
x;y
:= CR
1
and V
y;y := R
1
. Also, we let b := V
x;y
y. The
equation that de nes the posteriormean in this notationis
V
x;x
^ x=b:
(5) Weonly onsider the appli ation of message passing ontrees. We some-
times distinguish a parti ular node x
r
as the root node, and relabel the
variables in breadth rst order, starting from the root (see Figure 2).
Notethat ingeneral,this labeling isnot unique.The parti ularbreadth-
rst-order labeling hosen does not a e t our on lusions. Let Q
r be a
permutation matrix that maps the original indi es to the new ones, and
r
a fun tionsu h that
r
(i) isthe index of x
i
in the breadth rst order
labeling when x
r
is the root node. In Figure 2, for example,
r
(4) = 1,
r
(2)=2,
r
(3)=3,
r
(5)=4,
r
(1)=5,
r
(6)=6 and
r
(7)=7.
(6) When applying message passing in the rooted tree, we use
i
to denote
the parentof node i.
(7) As usual in graphi al models, we use the notation u = v, to denote
thatuisproportionaltov.Here,uandv ouldbeany pairof ompatible
quantities (ve tors, fun tions, et .).
(8) In most guresthe observed nodes are not shown expli itly. Thisis only
for larity; wealways onsider graphi al models onthe variablesfx
i g
N
i=1
and fy
i g
N
i=1 .
We begin by studying the message passing algorithm when G is a tree and
nd its relationto Gaussianelimination. Then we use this relation toextend
the algorithmto handlegraphs Gwith loops.
h
h h
h
h h h
%
%
%
%
%
e
e
e
e
e
A
A
A
A
A
A 1
x
3 x
2
x
4
x
6 x
5
x
7 h
h
h h
h
h h
?
-
x
1 x2
x
3
x
7 x
5 x6
x
4
Rootnode
(b) (a)
Fig. 2. Example of relabeling pro ess. The tree in (a) shows the original labeling.
Afternode x
4
hasbeen hosenastheroot andthevariableshavebeenrelabeledin
breadth rst order,thetree in(b)is obtained.
3 Parametrization
Inorder touse messagepassingtosolve the inferen eproblem,it isne essary
to have a fa torization of the joint distribution as a produ t of lo al fun -
tions or \kernels" (fun tions of the maximal liques in the graph G).In gen-
eral, the fa torization isnot unique. Ea h hoi e of lo alfun tions provides a
parametrizationofthejointdistribution.Tobevalid,aparametrizationneeds
to satisfy ertain onditions, whi h we study in this se tion. We begin with
anexpression for the joint density fun tion:
p(x;y)= exp 8
>
>
<
>
>
: 1
2 0
B
x
y 1
C
A T
0
B
V
x;x V
x;y
V T
x;y V
y;y 1
C
A 0
B
x
y 1
C
A 9
>
>
=
>
>
;
: (2)
Our goal is to onstru t a message passing algorithm for loopy graphs as a
modi ationofthemessagepassingalgorithmfortrees.Webeginbyassuming
that V
x;x
orresponds to a tree 2
. Noting that a tree is a pairwisegraph, i.e.,
a graph in whi h the largest lique has size 2, we know that the joint distri-
bution fa tors into a produ t of lo alkernels, ea h of whi h depends only on
2variables
p(x;y)= Y
(i;j)2E
G
i;j (x
i
;x
j )
N
Y
k=1
k;k (x
k
;y
k ):
2
We say that a matrix V
x;x
\ orresponds" to a graph G if the stru ture of V
x;x
respe ts the onne tivity of G, i.e., if [V
x;x
℄
i;j
6= 0 only ifi =j or, for i6= j, x
i is
onne tedto x
j inG.
i;j (x
i
;x
j
) = exp 8
>
>
<
>
>
: 1
2 0
B
x
i
x
j 1
C
A T
0
B
Z
x
i
;x
i
x
i
;x
j Z
x
i
;x
j
x
i
;x
j
Z xi;xj
x
i
;x
j Z
xj;xj
x
i
;x
j 1
C
A 0
B
x
i
x
j 1
C
A 9
>
>
=
>
>
;
;
i;i (x
i
;y
i
) = exp 8
>
>
<
>
>
: 1
2 0
B
x
i
y
i 1
C
A T
0
B
Z
x
i
;x
i
x
i
;y
i Z
x
i
;y
i
x
i
;y
i
Z xi;yi
xi;yi Z
yi;yi
xi;yi 1
C
A 0
B
x
i
y
i 1
C
A 9
>
>
=
>
>
; :
(3)
For (3) to onstitute a orre t parametrization of the joint distribution (2),
the following onditions have tobemet
[V
x;x
℄
i;j
= 8
>
>
>
>
>
>
>
>
>
<
>
>
>
>
>
>
>
>
>
: Z
x
i
;x
i
x
i
;y
i +
P
k2N
i Z
x
i
;x
i
x
k
;x
i
if i=j;
Z x
i
;x
j
x
i
;x
j
if j 2N
i
;
0 else;
V
x;y
=diag(Z x
1
;y
1
x
1
;y
1
;:::;Z x
N
;y
N
x
N
;y
N );
V
y;y
=diag(Z y
1
;y
1
x
1
;y
1
;:::;Z y
N
;y
N
x
N
;y
N ):
(4)
Note that for ea h pair of indi es, i and j, Z x
i
;x
j
x
i
;x
j
=Z x
j
;x
i
x
j
;x
i
. Likewise Z x
i
;x
i
x
i
;x
j
=
Z x
i
;x
i
x
j
;x
i
, and Z x
j
;x
j
x
i
;x
j
=Z x
j
;x
j
x
j
;x
i .
Given , C and R it is an easy task to nd a valid parametrization, Z x
i
;x
j
x
i
;x
j ,
Z x
i
;x
i
x
i
;x
j , Z
x
j
;x
j
x
i
;x
j ,Z
x
i
;y
i
x
i
;y
i , Z
x
i
;x
i
x
i
;y
i , Z
y
i
;y
i
x
i
;y
i
for 1i;j N, that satis es (4).
4 The messages
In message passing, ea h node in G sends one or more messages to ea h of
its neighbors. The message that x
i
sends to x
j
ontains the parameters of a
fun tion of x
j
. In a Gaussian graphi al modelthe messages are parametriza-
tions of Gaussian distribution fun tions and an be therefore be represented
by theirmeanand inverse ovarian e,i.e,\pre ision."Wedenotethe fun tion
orresponding to the message from x
i to x
j as m
i;j (x
j
), and its mean and
inverse ovarian e as
i;j
and P
i;j
, respe tively.
When G is a tree, the order in whi h the messages are sent is irrelevant to
the nal result. For our purposes, it is onvenient to s hedule the algorithm
h
h
h
h
h h
h
-
H
H
H
H
H
H
H
H
H
H
H
H
H
H
*
H H H Y H
H
H j
* H H H Y H
H
H j
* 2
x
3
x
4
x
5
6
x
7 m
4;5
m
5;4 m
4;2
m
3;4 m
2;1
m
6;5
m
5;7
m
7;5 m
5;6
m
4;3 m
2;4 m
1;2
x
1
Fig.3.Completesetof messagestransmittedduringappli ationofmessage passing
to solve theinferen e problemforea hnode inthegraph.
a ording tothe following rule:
\Node i is allowed to send a message to node j only after it has re eived
messages fromall itsneighbors, ex ept j."
Notethat this rule annotbeimplemented if G has loops. However, for trees
su hapro edure anbeinitiatedattheleavesofthetreeandprogressinwards
into the tree, as an bededu ed fromFigure 3.
The inferen e problem we want to solve is the omputation of the posterior
distributionsofallthe statenodesinG,given theobservations. Thisproblem
is solved by sending messages between ea h pair of neighboring nodes. For
example,Figure3shows atree and the ompleteset of messages ne essary to
solve the inferen e problem.
Solvingtheinferen eproblemforonlyonestatevariablex
r
requiresthetrans-
missionofonlyasubsetofthemessages(butea hmessageis omputedexa tly
asin the general ase).We onsider this simpler problem rst. It is herethat
the relation to Gaussian elimination be omes lear. For larity, we onsider
the tree rootedat x
r
and relabelthe variablesin breadth rst order, starting
from the root, as depi ted in Figure 2. The messages that are ne essary to
nd the posterior distribution of the root node are the ones owing upwards
towards the root.An exampleof message passingonthe rooted tree isshown
inFigure 4.
We nowpresentthe generalmessage passingalgorithm.The mean and pre i-
sion of the message that a node i sends toits neighborj are given by
P
i;j
= Z x
j
;x
j
x
i
;x
j (Z
x
i
;x
j
x
i
;x
j )
2
Z x
i
;x
i
x
i
;x
j +P
i;j
0
;
i;j
= Z
x
i
;x
j
x
i
;x
j P
i;j
0
i;j
0
Pi;j(Z x
i
;x
i
x
i
;x
j +P
i;j
0 )
;
P i;j
0
= Z x
i
;x
i
x
i
;y
i +
P
k2N
i;j P
k;i
;
P i;j
0
i;j
0
= Z
x
i
;y
i
x
i
;y
i y
i +
P
k2N
i;j P
k;i
k;i :
When message passing has nished, node i omputes its posterior mean,
i ,
and pre ision, P
i
,a ording to the followingequations:
P
i
=Z x
i
;x
i
x
i
;y
i +
P
k2N
i P
k;i
;
P
i
i
= Z
x
i
;y
i
x
i
;y
i y
i +
P
k2N
i P
k;i
k;i :
Notethat inthe rooted treeonly the rootnode (x
1
in thenew labeling) om-
putes
1
and P
1
. In the rest of this se tion we suppose that message passing
is performed inthe rooted tree. Toavoid onfusing notation, we assumethat
V
x;x
and the parametrization of the joint distribution are given with respe t
tothe new labeling.
InSe tion7we ompare messagepassingtoGaussianelimination.Forthis we
need detailedexpressions for the messages. Expandingthe expression forP i;j
0
h
h
h
h
h
h
h
h h
h h
h
h h
6
I 6
6 6
A
A
A
A
A A
A A A K
A
A
A
A
A A
A A A K 3
2
5
1
m
5;2
3 2
5
1
m4;1
m
3;1
m
5;2 m
3;1
6 6
Firststepofmessagepassing Se ondstepofmessagepassing m
2;1
4
7
4
7 m
6;4
m
7;4
m
6;4
m
7;4
Fig. 4. Example of message passing pro ess in the rooted tree, under the hosen
s hedule.Notethatthetree shownhereisequivalentto thetree inFigure3,rooted
at x
4
and relabeled in breadth rst order. Note also that only a subset of the
messagesshown inFigure 3are transmittedhere.
P i;j
0
= Z x
i
;x
i
x
i
;y
i +
P
k2Ni;j P
k;i
= Z x
i
;x
i
x
i
;y
i +
P
k2Ni;j
"
Z x
i
;x
i
x
k
;x
i (
Z x
k
;x
i
x
k
;x
i )
2
Z x
k
;x
k
x
k
;x
i +P
k ;i
0
#
= Z
x
i
;x
i
x
i
;y
i +
P
k2N
i;j Z
x
i
;x
i
x
k
;x
i
!
P
k2N
i;j (Z
x
k
;x
i
x
k
;x
i )
2
Z x
k
;x
k
x
k
;x
i +P
k ;i
0
= [V
x;x
℄
i;i Z
x
i
;x
i
xi;xj
P
k2N
i;j (Z
x
k
;x
i
x
k
;x
i )
2
Z x
k
;x
k
x
k
;x
i +P
k ;i
0
;
(5)
where j isthe parent of i. De ningR
i;j :=Z
xi;xi
x
i
;x
j +P
i;j
0
we nd
R
i;j
=[V
x;x
℄
i;i
X
k2N
i;j
Z x
k
;x
i
x
k
;x
i
2
R
k;i
: (6)
The initial ondition for this re ursion is R
l ;
l
= [V
x;x
℄
l ;l
for l a leaf node. In
the same fashion we an see that
P i;j
0
i;j
0
= Z
x
i
;y
i
x
i
;y
i y
i +
P
k2Ni;j P
k;i
k;i
= Z
xi;yi
xi;yi y
i P
k2N
i;j Z
x
k
;x
i
x
k
;x
i
Z x
k
;x
k
x
k
;x
i +P
k ;i
0 P
k;i
0
k;i
0 :
(7)
Letting S
i;j :=P
i;j
0
i;j
0
we an write
S
i;j
=[b℄
i
X
k2N
i;j Z
x
k
;x
i
x
k
;x
i
R
k;i S
k;i
; (8)
withinitial onditionS
l ;
l
= Z
x
l
;y
l
x
l
;y
l y
l
=[b℄
l
forea hrootnodel.Thedetailed
expression for the pre isionatthe rootnode is
P
1
= Z x
1
;x
1
x1;y1 +
P
k2N
1 P
k;1
= Z x
1
;x
1
x
1
;y
1 +
P
k2N1
"
Z x
1
;x
1
x
k
;x
1 (
Z x
k
;x
1
x
k
;x
1 )
2
Z x
k
;x
k
x
k
;x
1 +P
k ;1
0
#
= Z
x
1
;x
1
x
1
;y
1 +
P
k2N1 Z
x
1
;x
1
x
k
;x
1
!
P
k2N1 (Z
x
k
;x
1
x
k
;x
1 )
2
Z x
k
;x
k
x
k
;x
1 +P
k ;1
0
= [V
x;x
℄
1;1 P
k2N
1 (Z
x
k
;x
1
x
k
;x
1 )
2
R
k ;1 :
(9)
2 3 4
10 9 8
7 6
5
2 1 3 4 5 6 7 8 9 10
1 2 3 4 5 6 7 8 9 10
(b) (a)
1
Fig. 5. Figure (b) shows the stru ture of V
x;x
orresponding to the tree in Figure
(a). Nonzeroelementsareshown asbla ksquares.
Likewise,the value of P
1
1
isgiven by
P
1
1
= Z
x1;y1
x1;y1 y
1 +
P
k2N
1 P
k;1
k;1
= [b℄
1 P
k2N
1 Z
x
k
;x
1
x
k
;x
1
R
k ;1 S
k;1 :
(10)
These expressions willbeused later.
5 Stru ture of V
x;x
InSe tion7 weapply Gaussianeliminationtosolvethe inferen eproblemfor
the root node in the relabeled rooted tree. For this we rst need to nd the
stru ture 3
ofV
x;x
.AsinSe tion4,we simplifythe problemby assumingthat
V
x;x
isgiven with respe t to the relabeledvariables.
Re allthat inaGaussiangraphi almodel[V
x;x
℄
i;j
an be nonzeroonly ifx
i is
onne ted by an edge to x
j
in G; see (4). In other words, V
x;x
has the same
stru ture as the adja en y matrix of G (ex ept forthe diagonalterms).
Thebreadth rstorderlabelinggivesV
x;x
averysimplestru ture. Toseethis,
onsider forexamplethe relabeledrooted treeshown inFigure5(a).Consider
a xed rowi of V
x;x
.Entry [V
x;x
℄
i;j
an benonzero onlyif x
j
is neighborof x
i
inG.We an seethat the onlyneighborx
j ofx
i
withj <iis theparentof x
i
in the rooted tree. Likewise the neighbors of x
i
with index higher than i are
the hildren of x
i
. This means that row i of V
x;x
ontains nonzero entries in
3
By \stru ture"we meanthepositionsof thenonzero entries.
i i
and the hildren of x
i
. It is also important to note that in ea h row i there
is a unique nonzero element to the left of the diagonal, orresponding to the
unique parent of i.By the symmetry of V
x;x
, it follows that inea h olumn i
there isaunique nonzero element abovethe diagonal,again orrespondingto
the parent of x
i
. This stru ture of V
x;x
for the tree in Figure5(a) is shown in
Figure5(b).
6 Review of Gaussian elimination
In this se tion we review brie y Gaussian elimination and introdu e some
terminology whi h will be helpful in des ribing how Gaussian elimination is
appliedin the ontextof graphi al models.
Consider the system of equations
Ax=b: (11)
Suppose that we want to transform this system into an equivalent one (one
that has the same solution)
A 0
x=b 0
;
su h that [ A 0
℄
i;j
= 0 for some xed i 6=j. We willsay that element [A℄
i;j has
been \eliminated." We do soby adding a s aledversion of row j to row i. In
this ase we say that \row j has been used to eliminate element [ A℄
i;j
" or,
sometimes that \element[ A℄
j;j
has been used toeliminateelement [A℄
i;j ." To
formalize,we premultiply(11) by a matrix W de ned in the followingway:
[W℄
k;l
= 8
>
>
>
>
>
>
>
>
<
>
>
>
>
>
>
>
>
:
1 if k =l;
[ A℄
i;j
[A℄
j;j
if i6=j and (k;l)=(i;j);
0 else:
We all [W℄
i;j
the \elimination fa tor." Then, A 0
= WA and b 0
= Wb. To
makethenotationsimpler,wedonotmentionthematrixW expli itly.Infa t
we onsider that the i th row of matrix A and the i th element of b are
[ A℄
i;
[A℄
i;
[A℄
i;j
[A℄
j;j [A℄
j;
;
[ b℄
i
[b℄
i [A℄
i;j
[A℄
j;j [b℄
j :
This update operation isknown as\row operation."
When the values of some of the variables in (11) are known, they an be
substituted into the equations, thus redu ing the order of the system to be
solved. This pro edure is alled \ba ksubstitution." See [8℄ for a fuller more
detailedexplanation of Gaussian elimination.
7 Gaussian elimination for trees
In this se tion we still assume that inferen e has to be made only for a par-
ti ular node x
r
. After relabeling the variables, the problem an be stated as
one of nding [^x
r
℄
1
and [V 1
r
℄
1;1
, where
V
r
^ x
r
= b
r
;
V
r
:=Q
r V
x;x Q
T
r
;
b
r
:=Q
r b:
As in Se tion 4, we assume that V
x;x
and the parametrization of the joint
distributionare given intermsof the breadth- rst-order labeling.This allows
us to use V
x;x
dire tly and drop the subindex r. So, we have to nd [^x ℄
1 and
h
V 1
x;x i
1;1 , with
V
x;x
^ x=b:
We use Gaussian elimination to solve this problem. In order to do this, we
eliminate the elements in the upper triangle of V
x;x
using row operations.
After this pro ess, the system is transformed into
V 0
x;x
^ x=b
0
;
where V
x;x
isa lower triangular matrix. Let d
i
denote the i-th diagonal entry
of V 0
x;x
,and [b 0
℄
i
the i-th omponentof b 0
. Then
[^x℄
1
= [b
0
℄
1
d
1
;
h
V 1
x;x i
1;1
= 1
d
1
;
are, in fa t,the orre t posterior meanand varian eof the rootnode.
We nowdeterminethe order inwhi hthe elements intheuppertrianglehave
tobeeliminated.Considering thetree shown inFigure5(a) and thestru ture
of V
x;x
inFigure5(b), it iseasy to inferthe following rules:
(1) If i is a leaf node and j >i then [V
x;x
℄
i;j
= 0. This means that the rows
orresponding toleaf nodes need not be hanged.
(2) An element in row j is eliminated using the diagonal term in the same
olumn. Note that the elimination of the term [V
x;x
℄
j;i
= Z x
i
;x
j
xi;xj
is done
using thediagonalterm inthe rowi, whereiis a hildof j in therooted
tree.
(3) Sin eea h olumnofV
x;x
ontainsonlyonenonzero elementintheupper
triangle,ea hrowis usedonly on einthe eliminationpro ess (whi h,in
fa t, orresponds toone \message" sent by a node toits parent).
(4) Theelementsinrowj ontainedintheuppertriangleareexa tly[V
x;x
℄
j;i ,
foria hildofj.Thesetermsareeliminatedusingtherows orresponding
tothe hildrenofj (whi h orrespondstothe hildrensendingamessage
totheir parent).
(5) On e all terms inrow j inthe upper triangle have been eliminated, row
j an be used to eliminatethe unique nonzero element ontained in the
uppertriangle,in olumnj.Thiselementis[V
x;x
℄
k;j
,wherekistheparent
of j.
Theseobservationsimplythattheeliminationpro ess anbedonebyfollowing
exa tly the same order as message passing, i.e., start at the leaf nodes, then
the parents of the leaves, and so on, until the root is rea hed. The omplete
eliminationpro ess forthe tree inFigure5isshown inFigure6.The orderin
whi h elimination is done givesa re ursive expression for the diagonal terms
fd
i g
N
i=1
. Fori6=1,
d
i
=[V
x;x
℄
i;i
X
k2N
i;j
Z x
k
;x
i
x
k
;x
i
2
d
k
; (12)
where j is the parent of i in the rooted tree. The initial onditions for the
Rows used
Elements to eliminate
Step 2
Rows used
Elements to eliminate
Step 3 Step 4
Elements to eliminate
Rows used Rows used
Elements to eliminate
Final matrix Step 1
Fig. 6.The omplete sequen e of row operationsto ndthe posterior mean of the
root node ofthetree inFigure 5(a).Ea h stephereshows tworow operations.The
a tionofoneelementonanotherisshownasanarrow. Nonzeroelementsareshown
asbla ksquares.
re ursion are given by d
l
=[V
x;x
℄
l ;l
forea h leaf node l.For i=1 we have
d
1
=[V
x;x
℄
1;1 X
k2N1
Z x
k
;x
1
x
k
;x
1
2
d
k
: (13)
Comparing(12) and (13) to(6) and (9)wesee that for ea h i2,d
i
=R
i;
i ,
and d
1
=P
1 .
To nd an expression for [b 0
℄
1
, note that the fa tor used inthe eliminationof
the term [V
x;x
℄
j;i is
Z x
i
;x
j
x
i
;x
j
d
i
!
. Again, the order in whi h eliminationis done
gives are ursion for [b℄
i
, for 2iN:
[b 0
℄
i
=[b℄
i
X
k2N
i;j Z
x
k
;xi
x
k
;xi
d
k [b
0
℄
k
; (14)
with initial onditions [b 0
℄
l
=[b℄
l
for ea hleaf l.For[b 0
℄
1
we have
[b 0
℄
1
=[b℄
1 X
k2N
1 Z
x
k
;x
i
x
k
;x
i
d
k [b
0
℄
k
: (15)
Comparing (14) and (15) to (8) and (10), we nd that [b 0
℄
i
=S
i;
i
for i 2,
and [b 0
℄
1
=P
1
1 .
Wehavethusshown thatthe omputationsperformedby messagepassingare
equivalent to the appli ation of Gaussian eliminationto solve the asso iated
system of equations.
Notethatinthe previousanalysis wehave xed the rootnode and performed
message passing to nd the posterior distribution of only the root node. A
priori, solving the inferen e problem for all nodes would mean having to run
messagepassingonN rootedtreesor,equivalently,usingGaussianelimination
to nd [x^
r
℄
1
in ea h of the N systems of equations given by
V
r
^ x
r
=b
r
: (16)
However, a riti al observation isthat node
r
(i) willsend the same message
to
r
(j)inanyrootedtreeinwhi h
r
(j)isparentof
r
(i).So,thesemessages
do not have tobe re omputed for every hoi e of r. In parti ular, if b 0
r is the
right hand side after Gaussianelimination. We have P i;j
0
i;j
0
=[b 0
r
℄
r (i)
inany
rooted tree in whi h
r
(j) isthe parent of
r (i).
Based on this observation, message passing an be applied dire tly on the
non-rooted tree G. InG there are noparent- hildrelationships.Node i sends
amessage to ea h of its neighbors (see Figure3). When i sends a messageto
j, j a ts as the parent of i in a rooted tree, and the nodes in N
i;j
a t as its
hildren.Forexample,P i;j
0
i;j
0
an beinterpreted as:\This wouldbethe right
hand side in position
r
(i), after Gaussian elimination, in any rooted tree in
whi h
r
(j)is parent of
r (i)."
This s heme allows messagepassing tosolve the N inferen e problems simul-
taneously.Themessages thatanodex
r
re eivesfromitsneighborsareexa tly
the same as those itwould re eive inthe relabeled tree rooted atx
r .
in loopy graphs
The fa t that Gaussian elimination an be used on any matrix, suggests ex-
tending message passing to \loopy graphs," i.e., graphs with y les, by rst
studyingtheoperationsperformedbyGaussianelimination.Inthisse tionwe
extendthemessagepassingalgorithmto omputetheposteriormeanswhenG
hasloops.Aswewillsee,the algorithm an bedividedintothreestages:Mes-
sage passing, solution of a smaller system of equations,and ba ksubstitution
of apartial solution.
8.1 Stage 1: Modi ed message passing
LetV
x;x
orrespond toaloopy graph.We beginby ndingade ompositionof
V
x;x
inthe followingway:
(1) Let G = (V
G
;E
G
) be the loopy graph and T = (V
G
;E
T
) be a spanning
subtree of G.
(2) LetE :=E
G nE
T
bethe set of extra edges. Let V =f
1
;:::;
L
g denote
thesetofendpoints(i.e.,verti es)ofedgesinE.We allthenodesindexed
by V \spe ial nodes." Note thatthe numberof spe ial nodes isL (re all
that L isthe number of non-isolatednodes inG T).
(3) LetK 2IR NN
besu h that
[K℄
i;j :=
8
>
>
>
>
>
>
>
>
>
<
>
>
>
>
>
>
>
>
>
: P
(k;i)2E Z
x
i
;x
i
x
k
;x
i
if i=j 2V;
Z x
i
;x
j
x
i
;x
j
if i;j 2V and i6=j;
0 else:
Then V tree
x;x :=V
x;x
K orresponds toa tree.
Notethatinthis ase, N
i
isthe setofneighbors ofiinT,andN
i;j
=N
i nfjg.
For a given root node r, let V
r :=Q
r V
tree
x;x Q
T
r , K
r :=Q
r KQ
T
r
, and b
r := Q
r b.
The task isnow to nd [x^
r
℄
1
in ea h of the systems
(V
r +K
r )^x
r
=b
r
; 1r N:
Let V 0
r , K
0
r and b
0
r
be the matri es afterGaussian elimination. Weknow that
message passing in T will provide ea h node x
r
with information about the
rst row of V
r
and the rst element of b
r
. We need to reate new messages
that allowx
r
to ompute the nonzero elements in the rst row of K 0
r .
Consideringtheobservationsattheendoflastse tion,weletthenewmessage
fromitoj inthenonrootedtreebe:\These wouldbethevaluesofthenonzero
elementsinrow
r
(i)ofK 0
r if
r
(j)weretheparentof
r
(i)intherootedtree."
To formalize, we let ea h node i re ord a ve tor
i;i 2 IR
L
, and ea h edge
(i;j)2E
T
a ve tor
i;j 2IR
L
, su h that:
[
i;i
℄
l
=[K℄
i;
l
. Theseve tors are xed throughoutthe algorithm.
i;j
ontains the nonzero elements in row
r
(i) of K 0
r
in a rooted tree in
whi h
r
(j) isparentof
r (i).
From these de nitions we see that the ve tors
i;j
satisfythe re ursion
i;j
=
i;i
X
k2N
i;j Z
x
k
;x
i
x
k
;x
i
Z x
k
;x
k
x
k
;xi +P
k;i
0
k;i :
When messagepassinghas nished,ea hnode i an ompute [V 0
i
℄
1;1
=P
i and
[b 0
i
℄
1
= P
i
i
. Likewise i an ompute the the perturbation terms in the rst
row of K 0
i as
i
=
i;i X
k2N
i Z
x
k
;x
i
x
k
;x
i
Z x
k
;x
k
x
k
;xi +P
k;i
0
k;i :
Theinterpretationof
i
isthat [
i
℄
l
=[K 0
i
℄
1;
l
.Weknow that(V 0
i +K
0
i )^x
i
=b 0
i .
The rst equation of this system, rewritten in the originallabelingis
P
i [^x℄
i +
L
X
l =1 [
i
℄
l [^x℄
l
=P
i
i
: (17)
Sin e this is true for ea h 1 i N, equation (17), in fa t, de nes a new
system of equations for x.^
Now, onsider only the equations orresponding to the nodes indexed by V.
Forea h of these nodes we have
(P
l +[
l
℄
l )[^x ℄
l +
L
X
k=1
k6=l [
l
℄
k [^x℄
k
=P
l
l
; (18)
for 1 l L. Equation (18) orresponds to a subsystem of equations that
an be solved independently.So, whatmessage passingdoes inthis ase is to
After message passing between special nodes 1 2 3 4 5 6 7 1
2 3 4 6 7 5
1 1
1
1 2 3 4 5 6 7 1
2 3 4 6 7 5
1 1
1 1
1 1
1 After Backsubstitution
1 2 3 4 5 6 7 1
2 3 4 6 7 5
Structure of new system (Matrix H)
Special nodes
Special nodes
Fig. 7. Stru ture of new system of equations de ned by equation (17). The pro-
ess for solvingthisnew system is donein two steps: Solution of a subsystemand
ba ksubstitution.
redu ethe system of equations oforder N to one of order L.
8.2 Stage 2: Solution of subsystem of equations
LetH 2IR NN
and 2IR N
su h that for ea h 1iN:
[H℄
i;
:=[V 0
i +K
0
i
℄
1;
;
[ ℄
i
:=P
i
i :
The stru ture of H is shown in Figure7. The system of equations de ned by
(17) an be rewritten as
Hx^= :
The system of equations de ned by equation (18) an be solved by standard
Gaussian elimination, whi h we write here as message passing, now between
only the spe ial nodes. For 1 l L, let
l
2 IR , and h
l 2 IR
L
be su h
that
l :=P
l
l
and [h
l
℄
k
:=[H℄
l
;
k
for 1 k L. Weimplement Gaussian
eliminationto solve (18) in the following way:
(1) Forl =1to L do
(a) Node
l
sends h
l and
l
toea h node inV nf
l g.
(b) Ea h node
k
2Vnf
l
g updatesitsparameters inthe followingway
k
k
l [h
k
℄
l
[h
l
℄
l
;
h
k h
k h
l [h
k
℄
l
[h
l
℄
l :
l
l
[^x℄
l
=
l
[h
l
℄
l
: (19)
Notethat this algorithm orresponds to a\message passing version" of stan-
dard Gaussianelimination.
8.3 Stage 3: Ba ksubstitution
After the values of the variables [^x℄
l
, 1 l L are omputed, they an be
ba ksubstituted intothe system de ned by (17). The value of ea h [^x ℄
i ,i2= V
an be found from
[^x℄
i
= 1
P
i [ ℄
i L
X
l =1 [
i
℄
l [^x℄
l
!
:
9 Extendedmessagepassingalgorithmfor omputingtheposterior
varian es in loopy graphs
Inthisse tionwe onstru ta nitetime onvergentmessagepassingalgorithm
to ompute the varian e of ea h estimator [^x℄
i
, 1 i N, i.e the diagonal
terms of the ovarian e matrix
^
P.
In the rooted tree, when solving for [^x
r
℄
1
, we are really nding a matrix W
r
su hthatW
r V
r
islowertriangular,andthenreadingout[W
r V
r
℄
1;1
and[W
r b
r
℄
1
(see Se tion 6). The new messages introdu ed in the previous se tion allow
us to ompute [W
r K
r
℄
1;
l
for 1 l L. We now introdu e new messages to
ompute the rst row of I 0
r
:= W
r
I = W
r
(the e e t of the row operations
on the identity matrix). Computing [I 0
r
℄
1;
for ea h node r would allow us to
ompute the omplete posterior ovarian e
^
P, but doing so would in rease
ex essively the omplexity of the algorithm. Sin e we are interested only in
the diagonal terms of
^
P, we need to ompute only the rows [I 0
r
℄
1;
for ea h
spe ial node r 2V.
Let1l L be xed. To ompute I 0
l
1;i
,weobserve that
h
^
P i
l
;i
= 1
P
l h
I 0
l i
1;i
;
h
^
P i
i;
l
= 1
P
i [I
0
i
℄
1;
l :
Bythe symmetry of
^
P we have that
h
I 0
l i
1;i
= P
l
P
i [I
0
i
℄
1;
l :
In view of this, we introdu enew messages toprovide ea h node x
r
with the
informationne essary to ompute ave tor
r 2IR
L
, su h that, [
r
℄
l
=[I 0
r
℄
1;
l .
Firstwe nd expressions forthe desired elementsof [I 0
r
℄
1;
and thenwe trans-
late them tomessages.
In the rooted tree, if the (unique) path joining the root node and x
l
is 1 =
i
1
;i
2
;:::;i
m
=
l , then
[I 0
r
℄
1;
l
= m 1
Y
k=1 2
4 Z
x
i
k
;x
i
k +1
xi
k
;xi
k +1
d
i
k +1 3
5
:
To use this expression inthe non-rooted tree T,we de ne d
i;j
2IR through
d
i;j
=[V 0
r
℄
i;i
=Z x
i
;x
i
x
i
;x
j +P
i;j
0
;
and interpret it as \the i th diagonal element of I 0
r
in any rooted tree in
whi h
r
(j) is parent of
r
(i)." If the path joining x
l
and x
r
in T is
l
=
j
1
;j
2
;:::;j
n
=r then
[I 0
r
℄
1;
l
= n 1
Y
k=1 2
4 Z
x
j
k
;x
j
k +1
x
j
k
;x
j
k +1
d
j
k
;j
k +1 3
5
In order to ompute all the required produ ts, we de ne ve tors
i;i ,
i;j ,and
i 2IR
L
,su h that
[
i;i
℄
k
= 8
>
>
>
<
>
>
>
:
1if i=
k
;
0else;
and
i;j
is anew message from x
i to x
j
, satisfying the re ursion
i;j
=
i;i
X
k2N
i;j Z
x
k
;x
i
x
k
;x
i
d
k;i
k;i :
1 2 3 4 5 6 7 1
2 3 4 6 7 5
Structure of new system (Matrix H)
1 2 3 4 5 6 7 1
2 3 4 6 7 5
Special nodes
Special nodes
Structure of matrix J
1
1
1
? ? ? ? ?
?
?
?
?
?
?
?
?
?
?
?
?
? ? ? ? ? ?
?
1
Fig.8. Stru tureof matri es H and J for omputingthe posteriorvarian es. Irrel-
evantentries inJ aremarked withquestionmarks.
When message passing has nished, ea h node i omputes ve tor
i
in the
following way
i
=
i;i X
k2Ni Z
x
k
;xi
x
k
;x
i
d
k;i
k;i :
The interpretation is that [
i
℄
l
= [I 0
i
℄
1;
l
. With the information olle ted by
ea h node, we an onstru t a new system of equations, just as we did for
omputingthe mean,althoughinthis ase weperformrowoperations,not to
nd the solution of the system, but to nd the diagonalelements of
^
P.
We de ne H and asbefore, and amatrix J 2IR LN
su hthat
[J℄
i;j :=
8
>
>
>
>
>
>
>
>
<
>
>
>
>
>
>
>
>
:
1 if i=j 2= V;
[
j
℄
l
if i=
l 2V;
0 else:
NotethattheelementsinJ thathavebeende nedaszero,arereallyunknown,
but they are unimportant for our purposes. The stru ture of J is shown in
Figure 8. When solving the subsystem of equations, the e e t of the row
operationsonthe rows ofJ havetobere orded. Sin e informationabout the
matrix J isdistributedamongall nodes,we needto transmittheinformation
about the row operations performed by the spe ial nodes to ea h node in
G. For this, ea h spe ial node re ords a the elimination fa tor used in ea h
operationinStep1-bof thealgorithm.Tobespe i ,weletea hspe ialnode
k
re ord a ve tor f
k 2 IR
L
su h that after message passing, f
k
ontains the
elimination fa tors used by node
k
during the solution of the subsystem of
equations. For doing this, we initialize f
k
(0;0;:::;0) T
, and add to the
[f
k
℄
l
[h
k
℄
l
[ h
l
℄
l :
After messagebetween spe ial nodes has been nished, ea hspe ial node x
l
sends f
l , [h
l
℄
l
, and P
l
to every other node in G. Ea h node i then de nes a
ve tor v
i :
[v
i
℄
l
= P
l
P
i [
i
℄
l
;
so that [v
i
℄
l
= h
I 0
l i
1;i
. After this, node i reprodu es the row operations per-
formedduringthesolutionofthe subsystemofequationsand ndstheire e t
on v
i
(we give the detailsof the algorithmin the next se tion). Finally,node
x
i
an ompute its posteriorvarian eas
h
^
P i
i;i
= 8
>
>
>
<
>
>
>
: [v
i
℄
l
if i=
l 2V;
1
P
i
1 T
i v
i
if i2= V:
In the next se tion we provide pseudo odes for the algorithmspresented.
10 The omplete algorithms for omputing the means and vari-
an es in loopy graphs
To summarize the omplete pro edure, we provide the algorithmin toto. We
present the algorithm that omputes the posterior means and varian es. If
onlythe meansare desired,the part of the ode that omputes the varian es
an be omitted. We present two versions of the algorithm:a entralized ver-
sionthat anbeimplementedonasinglepro essor,andadistributedonethat
anbeimplementedinadistributedfashionon, forexample,asensornetwork
where ea h node has ameasurementy
i .
Before applying the algorithmit is ne essary to nd a spanning subtree T of
G,and the set of spe ial nodes V. In the entralized version of the algorithm,
this an be done by spe ifying a root node x
r
, and applying Dijkstra's algo-
rithm to nd the shortest path from every node to x
r
; see, for example, [2℄.
In the distributed implementation of the algorithm,ea h node an run Dijk-
stra'salgorithmto nd T.Sin eevery nodeexe utesthe same(deterministi )
algorithm, ea h one nds the same spanning subtree T. Another approa h,
is to exe ute a distributed Bellman-Ford algorithm before starting message
a detailed presentation of Dijkstra's, Bellman-Ford, and other algorithms to
solve the shortest path problem,see [2℄.
To order the omputationsin the distributed algorithmwe require that ea h
node has aunique identifyingnumber. Thisis spe iallyimportantwhen solv-
ingthe subsystem of equations.
The presentation of the algorithms follows. The algorithms are presented as
pseudo- ode.
10.1 Centralized algorithm
We now present a entralized algorithmthat an be implemented ona single
pro essor.
Begin entralized algorithm
Input: A graph G=(V
G
;E
G
)on N nodes fx
i g
N
i=1 .
A prespe i edrootnode r.
For ea h 1iN,a Gaussiankernel:
i;i (x
i
;y
i
) (equation3).
For ea h (i;j)2E
G
, a Gaussiankernel:
i;j (x
i
;x
j
) (equation3).
A ve tor of observations, y2IR N
.
Output: The posteriormean [^x ℄
i
,and varian e h
^
P i
i;i
of ea hx
i .
Find a spanning subtree T =(V
G
;E
T
) of G,for example by running
Dijkstra's algorithmon Gwith root r.
De ne:
E :=E
G nE
T
;
V :=non-isolated verti es inG T :
For ea h node i2f1;:::;Ng de ne:
N
i
:=f1j N : (i;j)2E
T g;
N
i;j :=N
i;j
nfjg;
[
i;i
℄
l :=
8
>
>
>
<
>
>
>
:
1 if i=
l
;
0 else:
[
i;i
℄
j :=
>
>
>
>
>
>
>
>
>
<
>
>
>
>
>
>
>
>
>
: P
(k;i)2E Z
xi;xi
x
k
;xi
if i=j 2V;
Z xi;xj
x
i
;x
j
if i;j 2V and i6=j;
0 else:
Repeat
Forea hnode i2f1;:::;Ng
Forea h node j 2N
i
Ifall messages from nodes inN
i;j
have been re eived,
ompute:
P i;j
0
=Z x
i
;x
i
x
i
;y
i +
P
k2N
i;j P
k;i
;
P i;j
0
i;j
0
= Z
x
i
;y
i
x
i
;y
i y
i +
P
k2Ni;j P
k;i
k;i
;
P
i;j
=Z x
j
;x
j
x
i
;x
j (Z
x
i
;x
j
x
i
;x
j )
2
Z x
i
;x
i
x
i
;x
j +P
i;j
0
;
P
i;j
i;j
= Z
x
i
;x
j
x
i
;x
j
Z x
i
;x
i
x
i
;x
j +P
i;j
0 P
i;j
0
i;j
0
;
i;j
=
i;i
P
k2N
i;j Z
x
k
;x
i
x
k
;x
i
Z x
k
;x
k
x
k
;x
i +P
k ;i
0
k;i
;
i;j
=
i;i P
k2Ni;j Z
x
k
;x
i
x
k
;x
i
Z x
k
;x
k
x
k
;x
i +P
k ;i
0
k;i :
Untilevery node has re eived messages fromall itsneighbors.
For ea h node i2f1;:::;Ng ompute:
P
i
= Z x
i
;x
i
x
i
;y
i +
P
k2N
i P
k;i
;
P
i
i
= Z
x
i
;y
i
x
i
;y
i y
i +
P
k2N
i P
k;i
k;i
;
i
=
i;i P
k2N
i Z
x
k
;x
i
x
k
;x
i
Z x
k
;x
k
x
k
;x
i +P
k ;i
0
k;i
;
i
=
i;i P
k2N
i Z
x
k
;x
i
x
k
;x
i
Z x
k
;x
k
x
k
;x
i +P
k ;i
0
k;i :
For ea h node
l
, l2f1;:::;Lg de ne h
l
;f
l 2IR
L
, and
l
2IR through:
f
l
:=(0;0;:::;0) T
;
l
:=P
l
l
;
[ h
l
℄
k :=
8
>
>
>
<
>
>
>
: P
l +[
l
℄
l
if k=l;
[
l
℄
k
if k6=l:
For l=1 toL do