4Zero-error Distributed Source
Theorem 4. 9 If for all graphs G in some family G
x∈z
2⫺lx⭐␣ ᭙z∈⍀(G) (4.28)
is sufficient for the existence of a VLZE codewith given codeword lengths|(x)|⫽lx, then for anyG∈G, one can construct a codesatisfying
R()⫺Rmin(G,PX)⭐1⫺log␣.
Tuncel [15] also showed that Equation (4.28) with ␣⫽12 and ␣⫽14 are indeed sufficient conditions for codes to exist for the classes of graphs with three and four cliques,respectively. Previously,Yan and Berger [17] had shown the sufficiency of␣⫽1 for the class of graphs with only two cliques.
4.5.3
An Exponential-time Optimal VLZE Code Design Algorithm
In this section, we provide the algorithm proposed by Koulgi et al. [7] for optimal VLZE code design. The algorithm is inevitably exponential-time. Before describing the algorithm,we extend the notion of an induced subgraph by also inducing probabilities
of vertices. To that end, for a graphG⫽(X,E)andPX, define for any subsetX⬘
P(X⬘)⫽ x∈X⬘ PX(x) and PX|X⬘(x)⫽ PX(x) P(X⬘).
Also define the weighted codeword length of the subgraphG(X⬘)as
L(X⬘)⫽P(X⬘)Rmin1 (G(X⬘),PX|X⬘).
Then, for any codeand codewordi, we have
L(⫺1(i∗))⫽L(⫺1(i0∗))⫹L(⫺1(i1∗))⫹P(⫺1(i∗))⫺P(⫺1(i)) (4.29)
wherei∗denotes the set of all codewords that are prefixed byi, provided that none
of the sets above are empty. If, on the other hand, is the optimal code achieving
R1min(G,PX), Equation (4.29) can be recast as L(⫺1(i∗))⫽ min
D⊆⫺1(i∗)⫺⫺1(i){L(D)⫹L(
⫺1(i∗)⫺⫺1(i)⫺D)}
⫹P(⫺1(i∗))⫺P(⫺1(i)). (4.30) Note that the vertices in the set ⫺1(i) must be isolated in G(⫺1(i)), because otherwise they would violate the prefix condition of VLZE codes. Using this observa- tion together with (4.30) suggests an iterative algorithm whereby given anyX⬘⊆X, isolated verticesI(X⬘)inG(X⬘)are identified and
L(X⬘)⫽ min D⊆X⬘⫺I(X⬘) L(D)⫹L(X⬘⫺I(X⬘)⫺D) ⫹P(X⬘)⫺P(I(X⬘))
is computed, with the terminating condition thatL(X⬘)⫽0 ifI(X⬘)⫽X⬘.
In fact, it is not necessary to go over all subsetsD⊆X⬘⫺I(X⬘). As was argued in [7], it suffices to consider only those Dthat induce a dominating 2-partition, that is, where every vertex inDis connected to some vertex inX⬘⫺I(X⬘)⫺Dand vice versa. This follows by the observation that (i) if a vertex in ⫺1(0∗)⫺⫺1(0)is not connected to any vertex in⫺1(1∗), then it can instead be assigned the codeword 1, and (ii) if a vertex in⫺1(0)is not connected to any vertex in⫺1(1∗), then it can instead be assigned the null codeword since it must be isolated. Since both actions reduce the rate,cannot be optimal.
4.6
Conclusions
109
{a, b, c, d}
{b} | {a, c, d} {d} | {a, b, c}
{a, c} | {d} {a, c} | {b} {a} | {b} {c} | {d} {a} | {d} {b} | {c} {b} | {d} {a, b} | {c, d} {a, d} | {b, c} {a, c} | {b, d}
b a d c L522pb L50 L50 L50 L50 L50 L50 L50L50 L50L50 L50L50 L50 L50 L50 L50 L50 L512pb L512pd L522pd L52 L52 L511pb1pd L5pa1pb L5pc1pd L5pa1pd L5pb1pc L5pb1pd FIGURE 4.6
The recursive algorithm, which is terminated within two levels for this example. Values indicated on branches of the tree represent values ofL(X⬘)returned for each dominating 2-partition.
In Figure 4.6, this recursive algorithm is demonstrated using a very small per- fect graph with verticesX⫽{a,b,c,d}and associated probabilities{pa,pb,pc,pd}. Considering the simplest case, ifpa⫽pb⫽pc⫽pd⫽0.25, one obtains
L(X)⫽R1min(G,PX)⫽1.5
achieved by(a)⫽(c)⫽0,(b)⫽10,(d)⫽11. In this case, the rate can be shown to coincide with Rmin(G,PX)⫽H(G,PX)⫽H(G,PX). On the other hand, if pa⫽ 1 6,pb⫽ 1 3,pc⫽ 1 6,pd⫽ 1
3, the optimal code (a)⫽10,(b)⫽0,(c)⫽10,(d)⫽11 achievesR1min(G,PX)⫽53≈1.667, whereas
Rmin(G,PX)⫽log23≈1.585.
4.6
CONCLUSIONS
As we discussed in this chapter, the behavior of distributed source coding with zero error is significantly different from the better-known vanishingly small error regime. Unfortunately, it is also considerably less understood. Single-letter characterization of achievable rates seems impossible because of the connection of zero-error coding to the notoriously difficult problem of graph capacity.The task of designing optimal VLZE
codes is also difficult in that unless P⫽NP,optimalVLZE code design takes exponential
time.
What is discovered so far is not insignificant,however. In particular,as we argued,the fact that complementary graph entropy characterizes the asymptotically achievable minimum rate for VLZE coding is instrumental in proving that there is no rate loss in
certain network extensions of the main DSC problem. Also, single-letter bounds on complementary graph entropy, which are known to be tight for perfect graphs, are directly inherited for the minimum rate. On the code design front,the exponential time algorithm outlined in Section 4.5.3 is still useful if the size of the graph is small. Koulgi et al. [7] also proposed approximate polynomial time algorithms. Also, as Theorem 4.9 states, one can upper bound the redundancy of optimal codes for a class of graphs if the Kraft sum on each clique can be shown to be sufficient for the class.
REFERENCES
[1] Alon, N., and Orlitsky, A. A lower bound on the expected length of one-to-one codes.IEEE Transactions on Information Theory, 40(5):1670–1672, 1994.
[2] Alon,N.,and Orlitsky, A. Source coding and graph entropies.IEEE Transactions on Information Theory, 42(5):1329–1339, 1996.
[3] Berge, C.Graphs and Hypergraphs. North-Holland, Amsterdam, 1973.
[4] Csiszár, I., and Körner, J.Information Theory, Coding Theorems for Discrete Memoryless Systems, Academic Press, New York, 1982.
[5] Csiszár, I., Körner, J., Lovász, L., Marton, K., and Simonyi, G. Entropy splitting for antiblocking corners and perfect graphs.Combinatorica, 10(1), 1990.
[6] Gargano, L., Körner, J., and Vaccaro, U. Capacities: From information theory to extremal set theory.Journal of Combinatorial Theory Series A, 68(2):296–316, 1994.
[7] Koulgi, P., Tuncel, E., Regunathan, S., and Rose, K. On zero-error source coding with decoder side information.IEEE Transactions on Information Theory, 49(1):99–111, 2003.
[8] Koulgi, P.,Tuncel, E., Regunathan, S., and Rose, K. On zero-error coding of correlated sources.
IEEE Transactions on Information Theory, 49(11):2856–2873, 2003.
[9] Körner, J. Coding of an information source having ambiguous alphabet and the entropy of graphs.Trans. of the 6th Prague Conference on Information Theory, etc.,Academia, Prague, 411–425, 1973.
[10] Körner, J., and Longo, G . Two-step encoding of finite memoryless sources.IEEE Transactions on Information Theory, 19(6):778–782, 1973.
[11] Körner, J., and Marton, K. Graphs that split entropies.SIAM J. Discrete Math., 1, 71–79, 1988. [12] Marton,K. On the Shannon capacity of probabilistic graphs.Journal of Combinatorial Theory,
Series B, 57(2):183–195, 1993.
[13] Simonyi, G. On Witsenhausen’s zero-error rate for multiple sources.IEEE Transactions on Information Theory, 49(12):3258–3261, 2003.
[14] Slepian, D., and Wolf, J. K. Noiseless coding of correlated information sources.IEEE Transac- tions on Information Theory, 19(4):471–480, 1973.
[15] Tuncel, E. Kraft inequality and zero-error source coding with decoder side information.IEEE Transactions on Information Theory, 53(12):4810–4816, 2007.
[16] Witsenhausen, H. S. The zero-error side information problem and chromatic numbers.IEEE Transactions on Information Theory, 22(5):592–593, 1976.
[17] Yan, Y., and Berger, T. Zero-error instantaneous coding of correlated sources with length constraints is NP-complete.IEEE Transactions on Information Theory, 52(4):1705–1708, 2006.