Formulation of the Problem

take on. Probability density functions (PDFs) and probability mass functions (PMFs) are denoted bypand subindexed by the corresponding r.v.

3.2.2

Network Distributed Source Coding

We study the design of optimal quantizers for network distributed coding of noisy

sources with side information. Figure3.3 depicts a network with several lossy encoders

communicating with several lossy decoders. Letm,n∈Z⫹ represent the number of encoders and decoders,respectively. LetX,Y⫽(Yj)n_j⫽1andZ⫽(Zi)m_i⫽1be r.v.’s defined on a common probability space, statisticallydependent in general, taking values in arbitrary, possibly different alphabets, respectively,X,Y, andZ. For eachi⫽1, . . . ,m, Zirepresents anobservationstatistically related to somesource data X of interest, available only at encoderi. For instance,Zimight be an image corrupted by noise, a feature extracted from the source data such as a projection or a norm, the pair consist- ing of the source data itself together with some additional encoder side information, or any type of correlated r.v., andXmight also be of the form(Xi)mi⫽1, whereXicould play the role of the source data from whichZi is originated. For each j⫽1, . . . ,n, Yj represents some side information, for example, previously decoded data, or an additional, local noisy observation, available at decoderjonly. For eachi, a quantizer

Noisy

Observation i _{Source Data j} Estimated

Side Information j Joint Decoder j Separate Encoder i Z1 Y₁ Reconstruction

Quantization Lossless Coding

Lossless Encoder Lossless Encoder Lossless Encoder Lossless Decoder Joint Decoder 1 Separate Encoder 1

Separate Encoder m Joint Decoder n

Q_1j Q_{i j} Q_{m j} (q ·_j, y_j) Y_j Z_m q_i·(z_i) Y_n Q_i1 Z_i Q_{i j} Q_{i n} Xˆ1 Xˆ_j Xˆ_n xˆ_j FIGURE 3.3

Distributed quantization of noisy sources with side information in a network withmencoders and

ndecoders (mandnneed not be equal).Xˆis the reconstruction of a r.v.Xjointly distributed with

qi·(zi)⫽(qij(zi))nj⫽1, which can also be regarded as a family of quantizers, is applied to the observationZi, obtaining the quantization indicesQi·⫽(Qij)n_j⫽1. Each quantiza- tion indexQij is losslessly encoded at encoderiand transmitted to decoderj, where it is losslessly decoded. We shall see that both encoding and decoding of quantiza- tion indices may be joint or separate. For each j, all quantization indices received by decoderj,Q_·j⫽(Qij)mi⫽1, and the side informationYj are used jointly to estimate the unseen source dataX. LetXˆj represent this estimate, obtained with a measurable functionxˆj(q·j,yj), calledreconstruction function, in an alphabetXˆj possibly differ- ent fromX. DefineQ⫽(Qij)mi⫽,n1,j⫽1,Xˆ⫽(Xˆj)nj⫽1, andxˆ(q,y)⫽(xˆj(q·j,yj))nj⫽1.Qand

ˆ

X will denote the alphabets ofQ andXˆ, respectively.Partially connected networks in which encoderidoesnot communicate with decoderj can easily be handled by redefiningQi·,Q·j andQnot to includeQij.

3.2.3

Cost, Distortion, and Rate Measures

We shall now extend the concept of rate measure presented in [37] by means of a much more general definition, accompanied by a characteristic property that will play a major role in the extension of the Lloyd algorithm for optimized quantizer design, called update property. Even though the definition of rate measure and its update property may seem rather abstract and complex initially, we shall see that it possesses

great generality and is applicable to a wide range of problems.The termscost measure,

rate measure, anddistortion measure, formally equivalent by definition, will be used to emphasize different connotations in the context of applications,usually a Lagrangian cost, a transmission rate, and the distortion introduced by lossy decoding, respectively. Acost measureis a nonnegative measurable function of the formc(q,x,xˆ,y,z), possibly defined in terms of the joint probability distribution of(Q,X,Y,Z).1Itsasso- ciated expected costis defined asC⫽Ec(Q,X,Xˆ,Y,Z). Furthermore, a cost measure

will be required to satisfy the followingupdate property. For any modification of the

joint probability distribution of(Q,X,Y,Z)preserving the marginal distribution of

(X,Y,Z), there must exist an induced cost measurec⬘, consistent with the original

definition almost surely (a.s.) but expressed in terms of the modified distribution,satis-

fying Ec⬘(Q,X,Y,Z)C, where the expectation is taken with respect to the original

distribution.

An important example of rate measure is that corresponding to asymmetric SW

coding of a quantization indexQgiven side informationY. The achievable rate in this

case has been shown to be H(Q|Y), when the two alphabets of the random variables

involved are finite [39], in the sense that any rate greater than H(Q|Y)would allow

arbitrarily low probability of decoding error, but any rate less than H(Q|Y)would not.

1_{Rigorously, a cost measure may take as arguments probability distributions (measures) and proba-}

bility functions; that is, it is a function of a function. We shall become immediately less formal and call the evaluation of the cost measure for a particular probability distribution or function, cost measure as well.

3.2

Formulation of the Problem

67

In [6], the validity of this result was generalized to countable alphabets, but it was still assumed that H(Y)⬍⬁. We show in [31, App. A] (also [35]) that the asymmetric SW result remains true under the assumptions in this chapter, namely, for anyQin a countable alphabet and anyYin an arbitrary alphabet, possibly continuous, regardless of the finiteness of H(Y).

Concordantly, we define the rate measurer(q,y)⫽⫺logpQ|Y(q|y)to model the use of anidealSW coder, in the sense that both the decoding error probability and the rate redundancy are negligible. Note that the very same rate measure would also model a conditional coder with access to the side information at the encoder. We check that the SW rate measure defined satisfies the update property. The rate mea- sure corresponding to any other PMFp⬘Q|Y would ber⬘(q,y)⫽⫺logpQ⬘|Y(q,y). The modified associated rate would then beR⬘⫽Er⬘(Q,Y), wherer⬘is defined in terms of the new PMF but the expectation is taken with respect to the original one. Clearly, R⬘⫺R⫽D(pQ|Yp⬘Q|Y)0. It can be seen that all rate measures defined in [37] satisfy the update property, thereby making the corresponding coding settings applicable to this framework.

If the alphabets ofX,Y,Z, and Xˆ are equal to some common normed vector space, then an example of a distortion measure isd(x,xˆ,y,z)⫽␣x⫺xˆ 2⫹␤y⫺ ˆ

x2_{⫹␥z⫺xˆ} 2_{, for any␣,␤,␥∈[0,⬁). An example of a cost measure suitable for dis-} tributed source coding,the main focus of this work,isc(q,x,xˆ,y)⫽d(x,xˆ)⫹␭r(q,y), where␭is a nonnegative real number determining the rate-distortion trade-off in the Lagrangian costC⫽D⫹␭R.

3.2.4

Optimal Quantizers and Reconstruction Functions

Given a suitable cost measure c(q,x,xˆ,y,z), we address the problem of finding

quantizersqi·(zi)and reconstruction functionsxˆj(q·j,yj)to minimize the associated

expected costC. The choice of the cost measure leads to a particular noisy distributed

source coding system, including a model for lossless coding.

3.2.5

Example: Quantization of Side Information

Even though the focus of this work is the application represented in Figure3.3, the

generality of this formulation allows many others. For example, consider the cod- ing problem represented in Figure 3.4, proposed in [3]. A r. v. Z is quantized. The quantization index Q is coded at rate R1⫽H(Q) and used as side information for a SW coder of a discrete random vector X. Hence, the additional rate required is R2⫽H(X|Q). We wish to find the quantizerq(z)minimizingC⫽R2⫹␭R1. It can be shown thatr1(q)⫽⫺logpQ(q)andr2(x,q)⫽⫺logpX|Q(x|q)are well-defined rate measures, using an argument similar to that for⫺logpQ|Y(q|y). Therefore, this prob- lem is a particular case of our formulation. In fact, this is a noisy WZ, or a statistical inference problem in whichR2plays the role of distortion, since minimizing H(X|Q) is equivalent to minimizing I(Z;X)⫺I(Q;X), nonnegative by the data processing inequality, zero if and only ifQis a sufficient statistic.

q (z) _EncoderEntropy _DecoderEntropy Z

Slepian–Wolf Decoder

X _X

Separate Lossless Encoder Joint Lossless Decoder

Lossy Side Information Encoder

R1 R2 Slepian–Wolf Encoder Q Q FIGURE 3.4

Quantization of side information.

Aside from particularizations and variations of distributed coding problems such as quantization of side information, quantization with side information at the encoder, broadcast with side information and an extension of the Blahut–Arimoto algorithm to noisy WZ coding, we find that our theoretical framework also unifies apparently unrelated problems such as the bottleneck method and Gauss mixture modeling. An extensive list of examples appears in [31].