Theoretical foundations of distributed coding

Consider{(X_i, Y_i)}n_i=1as the sequence of independent drawings of a pair of correlated random variables X, Y from a given distribution p(x, y). The sequences X and Y are statistically dependent and the dependency between them can be described by the conditional probability mass function P (X|Y ). Let us first consider the joint encoding and joint decoding of the correlated sources X and Y as shown in Fig. 2.5(a). The question to ask is: what is the minimum encoding rate R required to perfectly recover (without any errors) the two sources X and Y at the joint decoder ? The answer to this question derived from information theory is the joint entropy R = H(X, Y ) = H(Y ) + H(X|Y ) = H(X) + H(Y |X), where H(.) is the entropy function. For example, this can be achieved by encoding Y at a rate R_Y = H(Y ) and based on the complete knowledge of Y at both the encoder and the decoder, the source X is encoded at a rate R_X = H(X|Y ) where H(X|Y ) is the conditional entropy of X given Y . Next, we consider the distributed compression of sources X and Y as shown in Fig. 2.5(b), where the correlated sources X and Y are separately encoded but the decoding process is performed jointly. Certainly, encoding the sources X and Y at rates R_X  H(X) and R_Y  H(Y ) guarantees error free reconstruction at the joint decoder. However, for the correlated sources X and Y , the total rate R  H(X) + H(Y ) is greater than the joint entropy H(X, Y ). The real question is: is it possible to perfectly recover the sequences at the decoder, by encoding them at a total rate smaller than the sum of individual entropies ?

In 1970’s, Slepian and Wolf studied this distributed coding problem and showed that a total rate R =

H(X, Y ) is sufficient to perfectly reconstruct the sources. In particular, they showed that there is no loss in

coding efficiency in separate encoding compared to joint encoding, as long as joint decoding is performed. More specifically, for the two correlated sources X and Y the achievable rate region is given by

R_X  H(X|Y ), R_Y  H(Y |X), R_X+ R_Y  H(X, Y ). (2.14) The achievable rate region or the Slepian-Wolf region for two correlated sources X and Y is shown in Fig. 2.6. The corner points U and V in the Slepian-Wolf region represent a special case of the distributed

RX RY H(X|Y) H(Y|X) H(Y) H(X) RX + RY = H(X,Y) U V S RX RY H(X|Y) H(Y|X) H(Y) H(X) RX + RY = H(X,Y) RX RY H(X|Y) H(Y|X) H(Y) H(X) RX + RY = H(X,Y) U U V S S

Figure 2.6: Achievable Slepian-Wolf rate region for the two correlated sources X and Y .

source coding, often referred as coding with side information or asymmetric coding. For this special case, Pradhan et al. [74] proposed a practical and constructive design approach to realize Slepian-Wolf theorem based on channel codes. We illustrate the relationship between the channel codes and the Slepian-Wolf theorem for the corner point U in Fig. 2.7, where the source X is compressed based on the knowledge of Y that is available at the decoder. In a channel coding framework, in order to recover the sequence X from the noisy observation Y one has to do channel coding for X. The encoder therefore transmits parity bits that are used by the decoder to perfectly recover X from its noisy observation Y . From the Slepian-Wolf coding point of view the correlated sources X and Y can be modeled by a virtual dependency channel with

X as the channel input and Y as the noisy output. The virtual dependency channel motivates the authors

in [74] to employ channel coding techniques in DSC. Therefore, the error between sources X and Y can be corrected by applying channel coding to the source X. Hence, the encoder shown in Fig. 2.7 is actually a channel encoder that generates the parity bits from source X. The transmitted parity bits are later used by the decoder to reconstruct X from Y .

Therefore, it is clear that the Slepian-Wolf encoding problem is actually a channel coding problem, and the Slepian-Wolf bounds can be achieved by using capacity approaching channel codes. Practical DSC schemes have been first designed by establishing a relation between the Slepian-Wolf theorem and channel coding [74]. They brought channel coding ideas in the Slepian-Wolf source coding problem and they used Trellis channel code to yield a practical Slepian-Wolf system. This further motivated a lot of researchers to use better channel codes. In this context, systems with better compression performance were developed using Turbo codes [75, 76]. Liveris et al. [77] introduced the encoding strategy with LDPC codes and they showed that the LDPC code is a better alternative to the Turbo code. However, both the LDPC and Turbo codes reach the channel capacity asymptotically, and therefore it requires large block sizes in order of 105 to reach the Slepian-Wolf bound. In practice, the source lengths are usually small in the order of few hundreds to few thousands. Recently, M. Grangetto et al. [78] built an asymmetric distributed coding scheme using the Arithmetic codes that provide a good performance at small block lengths compared to LDPC and Turbo codes. For more details, we refer the reader to the overview articles [72, 73].

In 1976, Wyner and Ziv extended the problem of coding with side information to the lossy coding scenario [71]. It is the problem of lossy compression of one source (e.g., source X) when the other source (e.g., source

Y ) is available at the decoder, as shown in Fig. 2.8. Wyner and Ziv considered an average distortion D

between the original X and the decoded ˆX versions, and they computed the minimum rate RW Z(D) required to encode X under the constraint that the average distortion between X and ˆX is E{d(X, ˆX)}  D. In

this scenario, Wyner and Ziv proved that the transmission rate increases, when the statistical dependency between sources is exploited only at the decoder compared to the case where the dependency is exploited at both the encoder and the decoder. Mathematically, the Wyner-Ziv theorem is stated by the following equation,

2.5 Distributed coding 19 Source X Source Y X _Decoder Y Xˆ ) , ( ~pxy Parity bits Virtual Channel Encoder

Figure 2.7: Relationship between Slepian-Wolf coding and channel coding. The correlation between sources can be modeled by a virtual channel with X as the channel input and Y as the output. The source X can be perfectly recovered from the noisy observation Y using parity bits.

Source X _encoderLossy

Source Y X Decoder Y Xˆ ) (D Rwz ) , ( ~pxy

Figure 2.8: Wyner-Ziv coding scheme: Lossy compression of source X with side information Y available at the decoder [71].

where RW Z_{(D) and R}

X|Y(D) are rate-distortion functions with the average distortion D. RW Z(D) repre-

sents the minimum encoding rate for X when Y is available only at the decoder, and R_X|Y(D) represents the minimum encoding rate for X when Y is simultaneously available at both the encoder and the decoder. They also showed that the equality sign in Eq. (2.15) holds when X and Y are jointly Gaussian. Finally, as a special case when D = 0, Eq. (2.15) becomes RW Z_{(0) = R}

X|Y(0), i.e., the Slepian-Wolf theorem is a

special case of the Wyner-Ziv theorem. This shows that it is possible to reconstruct X with an arbitrarily small probability of error, when the side information Y is available only at the decoder. For more details, we refer the reader to [71].