Coding Theory
4.1 Encoding and decoding
The situation is this: we have a source alphabet S= {s1,...,sm} and a code alphabet A= {a1,...,an}, which is also the input alphabet of some channel.
We would like to transmit text written in the source alphabet, but our channel accepts only code alphabetic characters. Therefore, we aim to associate a code alphabet word to represent each source alphabet word that we might wish to send.
In many real situations, it is not really necessary to represent each member of S+, the set of all source words, by a code word, a member of A+. For instance, if the source text is a chunk of ordinary English prose, we can be reasonably certain that we will not have to transmit nonsense words like “zrdfle”
or “cccm.” However, it does not seem that there is any great advantage to be had by omitting part of S+from consideration, and there is some disadvantage—the discussion gets complicated, quarrels break out, anxieties flourish.
Definitions An encoding function is a functionφ : S+→ A+. We say that such a function defines, or determines, a code. The code determined byφ is said to be unambiguous if and only ifφ is one-to-one (injective). Otherwise, the code is ambiguous.
A valid decoder-recognizer (VDR) for the code determined byφ is an al-gorithm which accepts as input anyw ∈ A+, and produces as output either the message “does not represent a source word” if, indeed,w is not in the range of φ, or, if w ∈ ranφ, some v ∈ S+such thatφ(v) = w.
The code determined byφ is uniquely decodable if and only if it is unam-biguous and there exists a VDR for it.
Some remarks are in order.
4.1.1 Note that the definitions above do not really say what a code is. It is something determined by an encoding function, but what? It might be more satisfying logically to identify the code with the encoding function which deter-mines it, but, unfortunately, that would lead to syntactic constructions that clash
with common usage. The definition above stands without apology, but the uses of the word “code” may increase in the future.
4.1.2 A VDR is, as its name indicates, an algorithm that either correctly codes a code word, or correctly recognizes that the code word cannot be de-coded.
We shall be quite informal about describing VDRs, and extremely cavalier about proving that a given algorithm is a VDR for a given code. For instance, suppose that S= A = {0,1}, and that φ is described by: φ doubles each 0 and leaves 1 as is. [Thus, for instance,φ(1010) = 100100.] Then the following describes a VDR for this code: givenv ∈ {0,1}+, scanv, and if any maximal block of consecutive 0’s of odd length is found inv, report “does not represent a source word”; otherwise, halve each maximal block of consecutive 0’s inv, and output the resulting wordw. We leave it to the reader to divine what is meant by “maximal block of consecutive 0’s.”
The point is that we do not make a fuss about how you scan or how you find a maximal block of consecutive 0’s in v and determine its length. Any implementation of the algorithm described would have to handle these and other matters, but the details are not our concern here. Also, it is possible to prove formally that this algorithm is a VDR for the given code, and that the code is uniquely decodable, but a bit of thought will convince anybody that these things are true, so that writing out formal proofs becomes an empty exercise, as well as being no fun. It can be of practical value to attempt proofs of algorithm validity and unique decodability, especially when these matters are in doubt, but we shall not be at all conscientious about such proofs.
4.1.3 In modern naive set theory, it is proven that for any non-empty S and A, there are uncountably many injective functions from S+ into A+. The codes determined by two different such functions cannot have the same VDR. It is also proven that there are but countably many algorithms expressible in any natural language. It follows that there are quite a few, in fact, uncountably many codes with no VDR. We certainly want nothing to do with such codes, but don’t worry—there is very little danger of encountering such a code.
In most of the codes actually in use in real life, the encoding function is defined in a particularly straightforward way.
Definition An encoding scheme for a source alphabet S= {s1,...,sm} in terms of a code alphabet A is a list of productions,
s1→w1
...
sm→wm,
in whichw1,...,wm∈ A+. For short, we will say that such a list is a scheme for S→ A.
4.1 Encoding and decoding 73
Each encoding scheme gives rise to an encoding functionφ : S+→ A+ by concatenation. The concatenation of a sequence of words is just the word obtained by writing them down in order, with no separating spaces, commas, or other marks. Given an encoding scheme, as above, and a wordv ∈ S+, we let φ(v) be the concatenation of the sequence of the wi,1 ≤ i ≤ m, corresponding, according to the scheme, to the source letters occurring inv. For example, if S= {a,b,c}, A = {0,1}, and the scheme is
a→ 01 b→ 10 c→ 111, thenφ(acbba) = 01111101001.
It is sometimes useful to be more formal; given an encoding scheme, we could define the corresponding encoding function by induction on the length of the source word. Forv ∈ S+, let lgth(v) stand for the length of v, the number of letters appearing inv. If lgth(v) = 1, then v ∈ S, so v = si for some i , and we set φ(v) = wi, wherewi is the code word on the right-hand side of the production si→ wi in the scheme. If lgth(v) > 1, then v = usifor some si ∈ S, and some u∈ S+ with lgth(u) = lgth(v) − 1; φ(u) has already been defined, so we set φ(v) = φ(u)wi.
The formality of this definition ofφ is unnecessary for most purposes, but it is advisable to keep it in mind. It provides a form for proving by induction statements about the code determined byφ. Sometimes the other obvious in-ductive definition ofφ, in which source words are formed by adding letters on the left rather than on the right, is more convenient.
4.1.4 When an encoding scheme is given, and thereby an encoding function, the term “the code” can refer to (i) the encoding scheme, (ii) the listw1,...,wm
of code words appearing in the scheme, or (iii) the set{w1,...,wm}.
4.1.5 Theorem Every code determined by an encoding scheme has a VDR.
Proof: Given a scheme si→ wi, i= 1,...,m, and a word w ∈ A+, look among all concatenations of thewi, with length of the concatenation equal to lgth(w).
[There are surely systematic ways to go about forming all such concatena-tions—but it would be tiresome to dwell upon those ways here.] If none of them matchw, report “does not represent a source word.” If one of them matches w, decode in the obvious way, by replacing eachwi in the concatenation by some corresponding si in the encoding scheme. It is left to you to convince yourself that this prescription constitutes a VDR for the given code.
The algorithm plan described above is a very bad one, extremely slow and inefficient, and should never be used. It is of interest only because it works whatever the encoding scheme.
4.1.6 Given an encoding scheme si → wi, i= 1,...,m, reading-left-to-right with reference to this scheme is the following algorithm: givenw ∈ A+, scan
from left to right alongw until you recognize some wi as an initial segment ofw. If the wi you recognize is also wj for some i= j (i.e., if the same code word represents different source letters according to the encoding scheme, heaven forfend), then decide for which i you have recognizedwi by some rule – for instance, you could let i be the smallest of the eligible indices.
If nowihas been recognized as an initial segment ofw, after scanning over max1≤i≤mlgth(wi) letters of w, or if you come to the end of w after scanning fewer letters, without recognizing somewi, report “does not represent a source word.” Otherwise, having recognizedwi, jot down si on your decoder pad, to the right of any source letters already recorded, peel (delete) the segmentwi
fromw, and begin the process anew, with the smaller word replacing w. If there is nothing left afterwi is peeled fromw, stop, and declare that the decoding is complete. Reading-right-to-left is described similarly.
For example, suppose that S= {a,b,c}, A = {0,1}, and the scheme is a→ 0
b→ 01 c→ 001.
(This is a particularly stupid scheme, for all practical purposes. Note that 001 represents both ab and c in the code defined by this scheme.) With reference to this scheme, what will be the outcome of applying the reading-left-to-right algorithm to 001? Answer: the output will be “does not represent a source word.” (Surprised?) It follows that reading-left-to-right is not a VDR for this code. However, reading-right-to-left is a VDR for this code. Verification, or proof, of this assertion is left to you. (Take a look at Exercise 4.2.2.)
4.1.7 If the wordswi appearing in an encoding scheme are all of the same length, the code is said to be a fixed-length or block code, and the common length of the wi is said to be the length of the code. Otherwise, the code is said to be a variable-length code.
4.1.8 If A= {0,1}, or some other two-element set, the code is said to be binary.
Exercises 4.1
1. Let S be the set of all English words, let A be the set of letters a,b,...,z, and let the encoding scheme be defined by a very complete unabridged dic-tionary—the O.E.D. will do. Ignore capitalizations. Show that the code defined by this scheme is ambiguous.
2. Suppose that S= A and φ : S+→ S+is defined by φ(w) =
w, if lgth(w) is odd, ww, if lgth(w) is even.
Show thatφ is not given by an encoding scheme. Describe a VDR for this code. Is this code uniquely decodable? Justify your answer.
4.2 Prefix-condition codes and the Kraft-McMillan inequality 75
3. Same questions as in 2, except thatφ(w) =
w, if lgth(w) is even, ww, if lgth(w) is odd.
4. Suppose that S= {a,b,c}, A = {0,1}, and consider the scheme a→ 0, b → 010, c → 0110.
Show that neither reading-left-to-right nor reading-right-to-left provides a VDR for this code. Describe a VDR for this code—make it a better one than the clunker described in the proof of Theorem 4.1.5. Is this code uniquely decodable? Justify your answer.
5. Give an encoding scheme for a uniquely decodable code for which reading-left-to-right is a VDR, but reading-right-to-left is not.
6. Suppose that S= {a,b,c}, A = {0,1}, and the encoding scheme is a→ 010, b → 0100, c → 0010.
Is the code defined by this scheme uniquely decodable? Justify your an-swer.
7. Give an encoding scheme for the code described in 4.1.2.