Chapter 1 Introduction

Chapter 1 Introduction

1. Shannon’s Information Theory 2. Source Coding theorem

3. Channel Coding Theory

4. Information Capacity Theorem 5. Introduction to Error Control

Coding

Appendix A : Historical Notes

1. Shannon's Information Theory

The theory provides answers to two fundamental questions (among others):

(a)What is the irreducible complexity below which a signal cannot be compressed?

(b)What is the ultimate transmission rate for reliable communication over a noisy channel?

2. Source Coding Theorem (Shannon's first theorem)

The theorem can be stated as follows:

Given a discrete memoryless source of entropy ^{H (S )} , the average code-word length ^L for any distortionless source coding is bounded as

)

(S

H

L ≥

§ This theorem provides the mathematical tool for assessing data compaction, i.e.

lossless data compression, of data generated by a discrete memoryless source.

§ The entropy of a source is a function of the probabilities of the source symbols that constitute the alphabet of the source.

§ Entropy of Discrete Memoryless Source Assume that the source output is modeled as a discrete random variable,

S

, which takes on symbols from a fixed finite alphabet

} { s

, s

, , s

= L

S

With probabilities

∑

=

−

=

=

=

0 k

k k

k

) p , k 0,1,2, , K 1 p 1 s

P(S L with

Define the amount of information gain

after observing the event ^{S = s}

as the

logarithmic function

bits p ) ( 1 log )

I(s

k 2

k

=

the entropy of the source is defined as the mean of ^{I ( s}

⁾ over source alphabet

S given by

∑

∑

−

=

−

=

=

=

=

1 K

0

k k

2 k 1 K

0 k

k k

k

p log 1

p

s I p

s I E H

bits )

( ) (

)]

( [ )

(S

§ The entropy is a measure of the average information content per source symbol.

§ The source coding theorem is also

known as the "noiseless coding

theorem" in the sense that it establishes

the condition for error-free encoding to

be possible.

3. Channel Coding Theorem (Shannon's 2

theorem)

§ The channel coding theorem for a discrete memoryless channel is stated in two parts as follows:

(a)Let a discrete memoryless source with an alphabet

have entropy ^{H (S )} and produce symbols once every ^T

seconds. Let a discrete memoryless channel have capacity ^C and be used once every

seconds. Then if

C

S

T

C T

H (S ) ≤

There exists a coding scheme for

which the source output can be

transmitted over the channel and be

reconstructed with an arbitrarily

small probability of error.

(b) Conversely, if

C

S

T

C T

H (S ) >

It is not possible to transmit information over the channel and reconstruct it with an arbitrarily small probability of error.

§ The theorem specifies the channel capacity ^C as a fundamental limit on the rate at which the transmission of reliable error-free message can take place over a discrete memoryless channel.

§ Mutual Information

Channel

x y

X

alphabet alphabet Y

Channel input x (selected from ^X ) with entropy H . ^{X = { x}

^{, x}

^{,..., x}

^}

Channel output y (selected from

) ,

} ,...,

,

{

= y y y

Y

How can we measure the uncertainty

about x after observing y?

Define the conditional entropy of X selected from ^X , given that

as

∑

=

=

= ^{J 1}

0

j j k

2 k

j

k P x y

log 1 y

x P y

Y

H ]

)

| [ (

)

| ( )

| ( X

The mean of entropy ^{H X ( | y}

⁾ over the output alphabet

is given by

∑ ∑

∑

=

−

=

−

=

=

=

=

1 - K

0 k

1 J

0

j j k

2 k

j 1

K

0 k

k k

y x P log 1 y

, x P

y P y Y H

H

)]

| [ (

) (

) ( )

| ( )

Y

|

(X X

The conditional entropy represents the amount of uncertainty remaining the channel input after the channel output has been observed.

The difference ^{H ( X ) − H ( X | Y )} is called the mutual information, which represents the uncertainty about the channel input that is resolved by observing the channel output.

Denoting the mutual information by

)

; ( X Y

I , we may thus write

)

| ( )

( )

;

( X Y H X H X Y

I = −

Note that ^{I ( X ; Y ) = I ( Y ; X )} .

§ Channel Capacity

The mutual information can be expressed by

∑∑

−

=

=

0 k

1 J

0

j k

k j 2

k

j

P y

y x log P

y , x P

I ]

) (

)

| [ (

) (

)

; ( X Y

The input probability distribution

)}

(

{ P x

is obviously independent of the channel. We can then maximize ^{I ( X ; Y )} with respect to

Hence we define the channel capacity

C as

)

;

)}

(

(

{

max I X Y

C

xj

=

The channel capacity ^C is measured in bits per channel use.

§ The Channel Coding Theorem is also

known as “noisy coding theorem”.

4. Information Capacity Theorem

(also known as Shannon-Hartley law or Shannon's 3

theorem)

It can be stated as follows:

The information capacity of a continuous channel of bandwidth ^B Hz, perturbed by additive white Gaussian noise of power spectral density ^N ₂

and limited in bandwidth to ^B , is given by

d bits/secon

)

( N B

1 P log B C

0

2

+

=

where ^P is the average transmitted power.

This theorem implies that, for given

average transmitted power ^P and

channel bandwidth ^B , we can transmit

information at the rate ^C bits per

second, with arbitrarily small

probability of error by employing

sufficiently complex encoding systems.

Shannon limit:

For an ideal system that transmits data at a rate ^R

^{= C}

Thus ^{P = E}

^{• R}

^{= E}

^{• C} Then ^C _B ^log

^{( 1} _N ^{E C} _B ⁾

o b

• +

=

B N c

E

o

b

= 2 − 1

∴

For infinite bandwidth, the ^{E /}

^N

approaches the limiting value

N dB E N

E

o b o B

b

) lim ( ) ln 2 0 . 693 1 . 6

( = = = = −

∞

∞ →

This value is called Shannon limit.

Appendix A. Historical Notes

§ The seeds of Shannon's information theory appeared first in a Bell Labs classified memorandum dated 1 September 1945, “A Mathematical Theory of Cryptography” (revised form was published in BSTJ, 1949.)

§ Shannon's work on cryptography problem at Bell Labs brought him into contact with two scholars, R.V.L.

Hartley and H. Nyquist, who were major consultants to the cryptography research.

§ Nyquist (1924) has shown that a certain bandwidth was necessary in order to send telegraph signals at a definite rate.

§ Hartley (1928) had attempted to

quantify of information as the logarithm

of the number of possible message built

from a pool of symbols.

§ Shannon was aware of Norbert Wiener's work on cybernetics (he cited Wiener's book in his two 1948 articles on information theory). Wiener had recognized that the communication of information was a problem in statistics.

§ Shannon had taken a mathematics course from Norbert Wiener when he was studying at MIT, and Shannon had access to the "Yellow Peril" report (Wiener's book before publication) while he worked on cryptographic research.

Major Publications of Shannon

§ A Mathematical Theory of Communication, I, II, published in BSTJ, 1948, 379-423, 623-656.

§ Communication Theory of Secrecy

Systems, published in BSTJ, 1949,

656-715.

§ Paperback edition of "The Mathematics Theory of Communication" was published by University of Illinois Press, 1963. 32,000 copies have been sold up to 1964.

§ In a review of the accomplishments of Bell Labs. in communication science, Millman in his edited book (1984), “A history of engineering and science in the Bell system: Communication Science (1925-1980)” says: “Probably the most specular development in communication mathematics to take place at Bell Laboratories was the formulation in the 1940's of information theory by C. E.

Shannon.”

§ In his 1948 articles on information

theory, Shannon credits J.W. Tukey (a

professor at Princeton Univ.) with

suggesting the word "bit" as a measure

of information.

§ In 1984 J.L. Massey wrote a paper

“Information Theory: the Copernican System of Communications.” Published in IEEE Communications Magazine, vol.

22, No. 12, pp. 26-28.

The Shannon’s theory of information is the scientific basis of communications in the same sense that the Copernicus’

heliocentric theory is the scientific basis of astronomy.

§ IEEE Transaction on Information Theory Oct. 1998. Vol.44 No.6.

Commemorate Issue (1948-1998)

5. Introduction to Error Control Coding 5.1 Purpose of Error Control Coding

§ In data communications, coding is used for controlling transmission errors induced by channel noise or other impairments, such as fading and interference, so that error-free communication can be achieved.

§ In data storage system, coding is used for controlling storage errors (during retrieval) caused by storage medium defects, dust particles and radiation so that error-free storage can be achieved.

Information source

Source encoder

Channel encoder

Modulation (writing unit)

Transmission channel (storage medium)

Demodulator (read unit) Channel

decoder

Destination Source

decoder

noise interference

5.2 Coding Principle

§ Coding is achieved by adding properly designed redundant digits (bits) to each message. These redundant digits (bits) are used for detecting and/or correcting transmission (or storage) errors.

5.3 Types of Codings

§ Block and Convolutional coding.

§ Block coding:

A message of k digits is mapped into a structured sequence of n digits, called a codeword.

k digits n digits

rate code n:

k

length code

: n

length message

: k

The mapping operation is called encoding. Each encoding operation is independent of the past encoding, i.e.

memoryless. The collection of all

codeword is called a "block code".

§ Convolutional Coding:

« An information sequence is divided into (short) blocks of k digits each.

Each k-digit message is encoded into an n-digit coded block. The n-digit coded block depends not only the corresponding k-digit message block but also on m (

) previous message blocks. That is, the encoder has memory of order m.

« The encoder has k inputs and n outputs.

« An information is encoded into a coded sequence. The collection of all possible code sequences is called an (n, k, m) convolutional code.

« Normally, ¹ ₂ ^≤ _≤ ^k _{n ≤} ^{≤ 8} _{9 n} ^{k : code rate}

5.4 Type of Errors and Channels

§ Random errors and burst errors.

Type of Channels

§ Random error channels:

Deep space channel, satellite channels, line of sight transmission channel, etc.

§ Burst error channels:

Radio links, terrestrial microwave links,

wire and cable transmission channels,

etc.

5.5 Decoding

Suppose a codeword corresponding to a message is transmitted over a noisy channel.

§ Let ^r be the received sequence. Based on ^r , the encoding rules and the noise characteristics of the channel, the receiver (or decoder) makes a decision which message was actually transmitted.

This decision making operation is called

“decoding”. The device which performing the decoding operation is called a “decoder”.

§ Two types of decoding:

Hard-decision decoding and soft-decision decoding.

§ Hard-decision decoding:

When binary coding is used, the

modulator has only binary inputs. If

binary demodulator output quantization

is used, the decoder has only binary inputs. In this case, the demodulator is said to make hard decisions.

Decoding based on hard decision made by the demodulator is called “hard decision decoding”.

§ Soft-decision decoding:

If the output of demodulator consists of more than two quantization levels or is left unquantized, the demodulator is said to make soft decisions.

Decoding based on soft decision made by demodulator is called soft-decision decoding.

§ Hard-decision decoding is much easier

to implement than soft-decision

decoding. However, soft-decision

decoding offers much better

performance.

5.6 Some Channel Models

§ Binary Symmetric Channel (BSC)

1-p

1-p

p p

0 0

1 1

Input Output

p: transition probability

When BPSK modulation is used over the AWGN channel with optimum coherent detection and binary output quantization, the bit-error probability for equally likely signal is given by

) (

N

Q 2E p =

∫

≡

2 y

dy 2p e

Q(x) 1

2

where

E : bit energy

2 N

: power spectral density of AWGN

§ Binary-input, 8-ary Output Discrete Channel

0

0

1

1

Input Output

2 3 4 5 6 7

5.7 Optimum Decoding

§ Suppose the codeword ^c corresponding to a certain message

is transmitted. Let ^r be the corresponding output of the demodulator.

§ The decoder produces an estimate ^{m ˆ} of the message based on ^r .

§ An optimum decoding rule is that minimize the probability of a decoding error. That is, ^{P ( c ˆ ≠ c | r )} is minimized.

Or, equivalently, maximizing ^{P ( c ˆ = c | r )} .

§ The decoding error is minimized for a given ^r by choosing ^cˆ to be a codeword ^c that maximizes

) (

) ( ) ) (

( P r

c P c

| r r P

| c

P =

That is, ^cˆ is chosen to be the most likely codeword, given that ^r is received.

5.8 Maximum a posteriori decoding (MAP decoding)

§ In general, we have

) (

) ( ) ) (

( P r

c P c

| r r P

| c

P =

If knowledge or an estimation of

is

used for decoding, the technique is

called MAP decoding.

5.9 Maximum Likelihood Decoding (MLD)

§ Suppose all the messages are equally likely. An optimum decoding can be done as follows:

For every codeword ^c

, compute the conditional probability

) (r |c_j P

The codeword ^c

with the largest conditional probability is chosen as the estimate ^cˆ for the transmitted codeword ^c .

This decoding rule is called the

Maximum Likelihood decoding

(MLD).

§ MLD for a BSC

Let ^{a = ( a}

^{, a}

^{, L , a}

⁾ and ^{b = ( b}

^{, b}

^{, L , b}

⁾ be two binary sequences of ⁿ components.

The Hamming distance between

and

b , denoted as ^d

^{( a , b )} , is defined as the number of places where

and ^b differ.

In coding for a BSC, every codeword and every received sequence are binary sequences.

Suppose some codeword is transmitted and the received sequence is

) ( r

, r

, , r

r = L

For a codeword

, the conditional probability ^{P ( r | c}

⁾ is

) ( )

(

( )

)

( r | c

p

1 - p

P =

For ^{p <} ₂ ¹ ,

is a monotonically decreasing function of ^d

^{( r , c}

⁾ .

Then ^{P ( r | c}

^{) > P ( r | c}

⁾ if and only if

) (

)

(

H

r , c d r , c

d <

The MLD is completed by the following steps:

(i) Compute ^d

^{( r , c}

⁾ for all ^c

(ii) ^c

is taken as the transmitted codeword if ^d

^{( r , c}

^{) < d}

^{( r , c}

⁾ for ^{j ≠ i}

(iii) Decode

into message

That is, the received vector ^r is decoded into the closest codeword.

This is also called the minimum distance

(nearest neighbor) decoding.

5.10 Bounded Distance Decoding

§ Given a received word ^r , a t-error correcting, bounded distance decoder selects that codeword ^c which minimizes ^d

^{( c r , )} if and only if there exists

such that ^d

^{( r , c ) ≤ t} . If no such

c exists, then a decoder failure is declared.

¬ All practical bounded distance decoder use some form of syndrome decoding.

¬ The bounded distance decoding is

usually an “incomplete decoding” since

it decodes only those received words

lying in a radius-t sphere about a

codeword.