MIT 6.02 DRAFT Lecture Notes Last update: September 23, 2012
C
HAPTER
5
Coping
with
Bit
Errors
using
Error
Correction
Codes
Recallourmaingoalin designingdigitalcommunication networks: tosend information reliablyandefficiently betweennodes. Meetingthat goalrequirestheuseof techniques tocombatbiterrors,whichareinevitableinbothcommmunicationchannelsandstorage media(storagemaybeviewedas“communicationacrosstime”;youstoresomethingnow andusuallywanttobeabletoretrieveitlater).
Thekeyideawewillapplytoachievereliablecommunicationistheadditionofredun dancytothetransmitteddata,toimprovetheprobabilitythattheoriginalmessagecanbe reconstructedfromthepossiblycorrupteddatathatisreceived.Thesenderhasanencoder
whosejobistotakethemessageandprocessittoproducethecodedbitsthatarethensent overthechannel. Thereceiverhasadecoderwhosejobistotakethereceived(coded)bits andtoproduceitsbestestimateofthemessage. Theencoder-decoderprocedurestogether constitute channel coding; goodchannelcodesprovide error correction capabilitiesthat reducethebiterrorrate(i.e.,theprobabilityofabiterror).
Withproperdesign,fullerrorcorrectionmaybepossible,providedonlyasmallnum beroferrorshasoccurred.Evenwhentoomanyerrorshaveoccurredtopermitcorrection, it maybe possible to perform error detection. Error detection providesa way forthe re ceiver to tell(withhigh probability) ifthe messagewas decodedcorrectly or not. Error detection usually worksby the senderandreceiver using a different code fromthe one usedtocorrecterrors;commonexamplesincludethecyclicredundancycheck(CRC)orhash functions.Thesecodestaken-bitmessagesandproduceacompact“signature”ofthatmes
sagethatismuchsmallerthanthemessage(e.g.,thepopularCRC-32schemeproducesa 32-bitsignatureof anarbitrarilylong message). The sendercomputesandtransmitsthe signaturealongwiththemessagebits,usuallyappendingittotheendofthemessage. The receiver,afterrunningthedecodertocorrecterrors,thencomputesthesignatureoverits estimateofthemessage bitsandcomparesthatsignatureto itsestimateofthesignature bits in thereceived data. If thecomputed andestimated signaturesare not equal, then thereceiver considersthemessage to haveoneor morebit errors; otherwise,it assumes thatthemessagehasbeenreceivedcorrectly. Thislatterassumptionisprobabilistic:there issomenon-zero(thoughverysmall,forgoodsignatures)probabilitythat theestimated
andcomputedsignaturesmatch,butthereceiver’sdecodedmessageisdifferentfromthe sender’s.Ifthesignaturesdon’tmatch,thereceiverandsendermayusesomehigher-layer protocoltoarrangeforthemessagetoberetransmitted;wewillstudysuchschemeslater. WewillnotstudyerrordetectioncodeslikeCRCorhashfunctionsinthiscourse.
Ourplan forthischapter isas follows. To start, we willassume a binary symmetric channel (BSC). In a BSC, the probability of any given bit “flipping” (a 0 sent over the
channelisreceivedasa 1,orviceversa)is",independentofallotherbits. Then,wewill
discussandanalyzeanelementaryredundancyschemecalledarepetitioncode,whichwill simply makencopiesof any given bit. The repetition code hasa code rate of 1/n—that
is,forevery usefulmessagebit,we enduptransmittingntotalbits. The overheadofthe
repetitioncode ofratenis 1 -1/n, whichisratherhighfortheerrorcorrectingpowerof
thecode.Wewillthenturntothekeyideasthatallowustobuildpowerfulcodescapable of correctingerrorswithout suchahighoverhead(orequivalently,capable ofcorrecting farmoreerrorsatagivencoderatecomparedtotherepetitioncode).
Therearetwobig,inter-relatedideasusedinessentiallyallerrorcorrectioncodes. The firstisthenotionof embedding, wherethemessages onewishestosendare placedin a geometricallypleasingwayin alargerspacesothat thedistancebetweenanytwovalid pointsintheembeddingislargeenoughto enablethecorrection anddetectionoferrors. Thesecondbigideaistouse parity calculations,whicharelinearfunctionsover thebits we wish to send, to generatethe redundancy in the bitsthat are actually sent. Wewill study examples of embeddings and parity calculations in the context of two classes of codes: linear block codes (which are aninstance of the broad class of algebraic codes)
and convolutional codes (which are perhaps thesimplest instance of thebroad class of
graphical codes).
Westartwithabriefdiscussionofbiterrors.
⌅
5.1
Bit
Errors
and
BSC
ABSCischaracterizedbyoneparameter,",whichwecanassumetobe< 1/2,theproba
bilityof abiterror. Itisanaturaldiscretizationofanoisemodeloversignals(acommon model for noise, as we will see in Chapter 9, is additive Gaussian noise, which is also a single-parameter model fully characterized by thevariance, (2). Wecan determine "
empirically by noting that if we send N bits over the channel, the expected number of
erroneouslyreceivedbitsisN ·". Bysendingalong knownbitpatternandcountingthe fractionorerroneouslyreceivedbits,wecanestimate",thankstothelawoflargenumbers. Inpractice,evenwhentheBSCisareasonableerrormodel,therangeof"couldberather large, between 10-2 (orevenhigher)allthewayto 10-10 oreven 10-12. Avalueof "of about 10-2 meansthatmessageslongerthana100bitswillseeatleastoneerroronaver
age;giventhatthetypicalunitofcommunicationoverachannel(a“packet”)isgenerally between500bitsand12000bits(ormore,insomenetworks),suchanerrorrateistoohigh. But is " of 10-12 small enough that we don’t need to bother about doing any error
correction? Theansweroftendependsonthedatarateof thechannel. Ifthechannelhas arateof10Gigabits/s(availabletodayevenoncommodityserver-classcomputers),then the“low”"of 10-12meansthatthereceiverwillseeoneerrorevery10secondsonaverage
49 SECTION5.2. THESIMPLESTCODE:REPETITION
thesituation,theapplicationsusingthechannelmayfinderrorsoccurringtoofrequently. Ontheotherhand,an"of 10-12maybefineoveracommunicationchannelrunningat10
Megabits/s,aslongasthereissomewaytodetecterrorswhentheyoccur.
In the BSC model, a transmitted bit b (0 or 1) is interpreted by the receiver as 1-b
with probability " and as b with probability 1-". In this model, each bit is corrupted
independently and with equal probability (which makes this an “iid” random process, for “independentand identically distributed”). We call" thebit-flip probabilityor the“error probability”, andsometimes abusenotationandcall itthe“bit errorrate” (itisn’treally a “rate”, but the term is stillused in the literature). Given a packetof size S bits, it is
straightforward to calculate the probabilityof the entire packetbeing received correctly whensentoveraBSCwithbit-flipprobability":
S
P(packetreceivedcorrectly) = (1-") .
Thepacketerrorprobability,i.e.,theprobabilityofthepacketbeingincorrect,is1minus thisquantity,becauseapacketiscorrectifandonlyifallitsbitsarecorrect.
Hence,
S
P(packeterror) = 1-(1-") . (5.1)
When"<<1, asimplefirst-order approximation ofthePERispossible because (1 + N
x) ⇡ 1 +N x when|x|<< 1. Thatapproximation gives the pleasingresult that, when
"<<1,
P(packeterror)⇡ 1-(1-S") =S". (5.2)
TheBSCisperhapsthesimplestdiscretechannelmodelthatisrealistic,butreal-world channelsexhibit more complexbehaviors. For example, over many wireless and wired channels as well as on storage media (like CDs, DVDs, and disks), errors can occur in
bursts. Thatis,theprobabilityofanygivenbitbeingreceivedwronglydependsonrecent history:theprobabilityishigherifthebitsintherecentpastwerereceivedincorrectly.Our goalistodeveloptechniquestomitigatetheeffectsofboththeBSCandbursterrors.We’ll startwithtechniquesthatworkwelloveraBSCandthendiscusshowtodealwithbursts.
⌅
5.2
The
Simplest
Code:
Repetition
Ingeneral,achannelcodeprovidesawaytomapmessagewordstocodewords(analogous to a source code, excepthere the purposeis notcompressionbut rather theaddition of redundancyforerrorcorrectionordetection).Inarepetitioncode,eachbitbisencodedas ncopiesofb,andtheresultisdelivered. Ifweconsiderbitbtobethemessageword,then
thecorrespondingcodewordisbn (i.e.,bb...b,ntimes). Inthisexample,thereareonlytwo
possiblemessagewords(0and1)andtwocorrespondingcodewords.Therepetitioncode isabsurdlysimple,yetit’sinstructiveandsometimesevenusefulinpractice!
But how well does it correct errors? To answerthis question, we will write out the probabilityofovercomingchannelerrorsfortheBSCerrormodelwiththerepetitioncode. Thatis,ifthechannelindependentlycorruptseachbitwithprobability",whatistheprob abilitythatthereceiverdecodesthereceivedcodewordcorrectlytoproducethemessage wordthatwassent?
Figure5-1:Probabilityofadecodingerrorwiththerepetitioncodethatreplaceseachbitbwithncopiesof b.Thecoderateis1/n.
maximumlikelihooddecoding: givenareceivedcodeword,r,whichissomen-bitcombina
tionof0’sand1’s, thedecodershouldproduce themostlikely message thatcouldhave causedr to bereceived. SincetheBSC errorprobability,", issmallerthan 1/2,themost
likely optionisthe codewordthat hasthemostnumberof bits incommonwith r. This
decodingruleresultsintheminimumprobabilityoferrorwhenallmessagesareequally likely.
Hence,thedecodingprocessisasfollows.First,countthenumberof1’sinr.Ifthereare
morethann/21’s,thendecodethemessageas1.Iftherearemorethann/20’s,thendecode
themessageas0. Whennisodd,eachcodewordwillbedecodedunambiguously. When niseven,andhasanequalnumberof0’sand1’s,thedecodercan’treallytellwhetherthe
message wasa0 or 1, andthebestit cando istomakeanarbitrary decision. (Wehave assumedthattheaprioriprobabilityofsendingamessage0isthesameasthatofsending a1.)
Wecanwritetheprobabilityofdecodingerrorfortherepetitioncodeasfollows,taking caretowritethelimitsofthesummationcorrectly:
( Pn (n)"i(1 -")n-i ifnodd i=dn 2e i P(decodingerror) = Pn (5.3) n (n)"i(1 -")n-i + 1( n )"n/2(1 -")n/2 ifneven i=2+1 i 2 n/2
Thenotation(n) denotesthenumberofwaysof selectingiobjects(inthiscase, bitposi i
tions)fromnobjects.
Whenniseven,weaddatermattheendtoaccountforthefactthatthedecoderhasa
fifty-fiftychanceofguessingcorrectlywhenitreceivesacodewordwithanequalnumber of0’sand1’s.
51 SECTION5.3. EMBEDDINGSANDHAMMINGDISTANCE
Figure5-1showstheprobabilityofdecodingerrorasafunctionoftherepetitionfactor,
n,fortherepetitioncode,computedusingEquation(5.3). They-axisisonalogscale,and
theprobability of erroris more or less a straightline withnegative slope (ifyou ignore theflatpieces),whichmeansthatthedecoding errorprobabilitydecreasesexponentially with the code rate. It is also worth noting that the error probability is the same when
n= 2`aswhenn= 2`-1.Thereason,ofcourse,isthatthedecoderobtainsnoadditional
informationthatitalreadydidn’tknowfromany 2`-1ofthereceivedbits.
Despitetheexponential reductioninthe probabilityof decodingerror asnincreases,
therepetitioncodeisextremelyinefficientin termsoftheoverheaditincurs,fora given rate,1/n. Assuch,itisusedonlyin situationswhenone isnotconcernedwith theover
headofcommunicationorstorage(i.e.,theresourcesconsumed),and/oroneisunableto implementamorecomplexdecoderinthesystem.
Wenowturntodevelopingmoresophisticatedcodes. Therearetwobigrelatedideas:
embedding messagesinto spacesina waythat achievesstructuralseparation andparity(linear) computationsoverthemessagebits.
⌅
5.3
Embeddings
and
Hamming
Distance
Let’s start our investigation into error correction by examining the situations in which errordetection andcorrection are possible. For simplicity, we will focus on single-error correction(SEC)here. Bythatwemeancodesthatareguaranteedtoproducethecorrect messageword,givenareceivedcodewordwithzerooronebiterrorsinit. Ifthereceived codewordhasmorethanonebiterror,thenwecanmakenoguarantees(themethodmight returnthecorrectmessageword,butthereisatleastoneinstancewhereitwillreturnthe wronganswer).
n
Thereare 2 possiblen-bitstrings. DefinetheHammingdistance(HD)betweentwon
-bitwords, w1 andw2, asthenumberof bitpositionsin whichthemessages differ. Thus 0HD(w1, w2)n.
Supposethat HD(w1, w2) = 1. Consider whathappensif we transmit w1 and there’s
a single bit error that inconveniently occursat theone bit position in which w1 andw2
differ. Fromthereceiver’spointofview itjustreceivedw2—the receivercan’tdetectthe
difference betweenreceivingw1 with a unfortunately placedbit error andreceivingw2.
Inthiscase,wecannotguaranteethatallsinglebiterrorswillbecorrectedifwechoosea codewherew1andw2arebothvalidcodewords.
WhathappensifweincreasetheHammingdistancebetweenanytwovalidcodewords to2?Moreformally,let’srestrictourselvestoonlysendingsomesubsetS ={w1, w2, ..., ws}
ofthe 2n possiblewordssuchthat
HD(wi, wj ) 2forallwi, wj 2Swherei6=j (5.4)
Thusifthetransmissionofwi iscorruptedbyasingleerror,theresultisnotanelement
of S and hencecan bedetected as anerroneousreceptionbythe receiver, whichknows
whichmessagesareelementsofS.AsimpleexampleisshowninFigure5-2:00and11are
validcodewords,andthereceptions01and10aresurelyerroneous.
WedefinetheminimumHammingdistanceofacodeastheminimumHammingdistance betweenanytwocodewordsinthecode. Fromthediscussionabove,itshouldbeeasyto
Figure5-2: CodewordsseparatedbyaHammingdistanceof2canbeusedtodetectsinglebiterrors. The codewordsareshadedineachpicture.Thepictureontheleftisa(2,1)repetitioncode,whichmaps1-bit messagesto2-bitcodewords. Thecodeonthe rightis a(3,2)code,whichmaps2-bitmessagesto3-bit codewords.
seewhathappensifweuseacodewhoseminimumHammingdistanceisD.Westatethe
propertyformally:
Theorem 5.1 Acode with a minimum Hammingdistance ofD candetect any error patternof D-1 orfewer errors. Moreover,there isatleastoneerrorpatternwith Derrorsthatcannotbe detectedreliably.
Hence,ifourgoalistodetecterrors,wecanuseanembeddingofthesetofmessageswe wishtotransmitintoabiggerspace,sothattheminimumHammingdistancebetweenany twocodewordsinthebiggerspaceisatleastonemorethanthenumberoferrorswewish todetect.(Wewilldiscusshowtoproducesuchembeddingsinthesubsequentsections.)
Butwhatabouttheproblemofcorrectingerrors? Let’sgobacktoFigure5-2,withS = {00,11}. Supposethereceivedsequenceis 01. Thereceivercantellthatasingleerrorhas
occurred,butitcan’ttellwhetherthecorrectdatasentwas00or11—boththosepossible patternsareequallylikelyundertheBSCerrormodel.
Ah, but we can extend our approach by producing an embedding with more space betweenvalidcodewords! SupposewelimitourselectionofmessagesinS evenfurther,
asfollows:
HD(wi, wj ) 3 forallwi, wj 2Swherei6= j (5.5)
Howdoesithelp toincreasetheminimumHamming distanceto3? Let’sdefine one morepieceofnotation: letEwi bethesetofmessagesresultingfromcorrupting wi witha
singleerror.Forexample,E000 = {001,010,100}.NotethatHD(wi,anelementofEwi) = 1.
Witha minimumHamming distanceof 3between thevalidcodewords, observe that there is no intersectionbetween Ewi and Ewj when i6= j. Why is that? Suppose there
was a message wk that was in both Ewi and Ewj. We know that HD(wi, wk) = 1 and
HD(wj , wk) = 1,whichimpliesthatwi andwj differinatmosttwobitsandconsequently
HD(wi, wj ) 2. (ThisresultisanapplicationofTheorem5.2below,whichstatesthatthe
Hammingdistancesatisfiesthetriangleinequality.)Thatcontradictsourspecificationthat theirminimumHammingdistancebe3. SotheEwi don’tintersect.
53 SECTION5.3. EMBEDDINGSANDHAMMINGDISTANCE
ofS(noerrors),orisamemberofsomeparticularEwi(oneerror),inwhichcasethereceiver
can deduce the original message was wi. Here’s a simpleexample: let S = {000,111}.
SoE000 = {001,010,100} andE111 = {110, 101,011}(note thatE000 doesn’tintersect E111).
Supposethereceivedsequenceis 101. The receiver cantelltherehasbeen asingleerror
because 101 2/S. Moreoveritcan deducethat theoriginal message wasmostlikely 111
because 101 2E111.
Wecanformallystatesomepropertiesfromtheabovediscussion,andspecifythe error-correctingpowerofacodewhoseminimumHammingdistanceisD.
Theorem 5.2 TheHammingdistancebetweenn-bitwordssatisfies thetriangleinequality. That is,HD(x,y) +HD(y,z) HD(x,z).
Theorem 5.3 ForaBSCerrormodelwithbiterrorprobability< 1/2,themaximumlikelihoodde
codingstrategyistomapanyreceivedwordtothevalidcodewordwithsmallestHammingdistance fromthereceivedone(tiesmaybebrokenarbitrarily).
Theorem 5.4 Acodewitha minimumHamming distanceofDcancorrectanyerror patternof
bD2-1c orfewer errors. Moreover, there isat leastoneerror pattern withbD-21c + 1errors that
cannotbecorrectedreliably.
Equation(5.5)gives usaway ofdeterminingif single-biterrorcorrection canalways beperformedona proposedsetS oftransmissionmessages—wecouldwriteaprogram
tocomputetheHammingdistancebetweenallpairsofmessagesinS andverifythatthe
minimum Hamming distancewas at least 3. We can also easily generalize this idea to check ifa codecan alwayscorrect more errors. And we can usetheobservations made aboveto decodeanyreceivedword: justfind theclosestvalidcodewordtothereceived one,andthenusetheknownmappingbetweeneachdistinctmessageandthecodeword toproducethemessage.Themessagewillbethecorrectoneiftheactualnumberoferrors isnolargerthanthenumber forwhicherrorcorrection isguaranteed. Thecheckforthe nearest codeword maybe exponential in thenumber of message bits we would like to send,makingitareasonableapproachonlyifthenumberofbitsissmall.
Buthowdowegoaboutfindingagoodembedding(i.e.,goodcodewords)? Thistask isn’tstraightforward,asthefollowingexampleshows. Supposewewanttoreliablysend 4-bitmessagessothat thereceiver cancorrectallsingle-biterrorsinthereceivedwords. Clearly,weneedtofindasetofcodewordsSwith 24elements.Whatshouldthemembers
ofS be?
The answerisn’t obvious. Once again, we couldwrite a program to search through possiblesetsofn-bitcodewordsuntilitfindsasetof size16 withaminimumHamming
distanceof3. Atediousandexhaustivesearchforour 4-bitmessageexampleshowsthat
theminimumnis7,andoneexampleofS is:
0000000 1100001 1100110 0000111 0101010 1001011 1001100 0101101 1010010 0110011 0110100 1010101 1111000 0011001 0011110 1111111
But such exhaustive searches are impractical when we want to send even modestly longermessages. Sowe’dlike some constructivetechnique forbuildingS. Much ofthe
theory and practice of coding is devoted to finding such constructions and developing efficientencodinganddecodingstrategies.
Broadlyspeaking,therearetwoclassesofcodeconstructions,eachwith anenormous number of instances. The first isthe class of algebraic block codes. The second isthe classof graphical codes.Wewillstudytwosimpleexamplesof linear block codes,which themselvesare a sub-class of algebraicblock codes: rectangularparitycodes andHam mingcodes.WealsonotethattherepetitioncodediscussedinSection5.2isanexampleof alinearblockcode.
Inthenexttwochapters, we willstudy convolutional codes,a sub-classof graphical codes.
⌅
5.4
Linear
Block
Codes
and
Parity
Calculations
Linearblockcodesareexamplesofalgebraicblockcodes,whichtakethesetofk-bitmes
sageswewishtosend(thereare 2k ofthem)andproduceasetof 2k codewords,eachnbits
long(n k)usingalgebraicoperationsover theblock. The word“block”referstothefact
thatanylongbitstreamcanbebrokenupintok-bitblocks,whichareeachthenexpanded
toproducen-bitcodewordsthataresent.
Suchcodesarealsocalled (n,k)codes,wherekmessagebitsarecombinedtoproduce ncodebits(soeachcodewordhasn-k“redundancy”bits). Often,we usethenotation
(n,k,d),wheredreferstotheminimumHammingdistanceoftheblockcode.Therateofa
blockcodeisdefinedask/n;thelargertherate,thelesstheredundancyoverheadincurred
bythecode.
Alinearcode(whetherablockcodeornot)producescodewordsfrommessagebitsby restrictingthealgebraicoperationstolinearfunctionsoverthemessagebits. Bylinear, we meanthatanygivenbitinavalidcodewordiscomputedastheweightedsumofone or moreoriginalmessagebits.
Linearcodes, aswe willsee, are bothpowerfulandefficientto implement. They are widely used in practice. In fact, all the codes we will study—including convolutional codes—arelinear, as are mostof thecodes widely usedin practice. Wealready looked attheproperties of asimple linearblockcode: the repetitioncode we discussedin Sec tion5.2isalinearblockcodewithparameters (n,1, n).
Animportantandpopularclassoflinearcodesarebinarylinearcodes.Thecomputations inthecase ofabinary codeusearithmeticmodulo2,whichhasaspecialname: algebra in a Galois Fieldof order 2, alsodenoted F2. Afield mustdefine rulesfor addition and
multiplication,andtheirinverses. AdditioninF2isaccordingtothefollowingrules: 0 + 0 = 1 + 1 = 0; 1 + 0 = 0 + 1 = 1. Multiplicationisasusual:0· 0 = 0· 1 = 1· 0 = 0; 1· 1 = 1.
Weleaveyoutofigureouttheadditiveandmultiplicativeinversesof 0and 1. Ourfocus
inthisbookwillbeonlinearcodesoverF2,buttherearenaturalgeneralizationstofields
ofhigherorder(inparticular,ReedSolomoncodes,whichareoverGaloisFieldsoforder
2q).
Alinearcodeischaracterizedbythefollowingtheorem,whichisbothanecessaryand asufficientconditionforacodetobelinear:
55 SECTION5.5. RECTANGULARPARITYSECCODE
Ausefulcorollaryofthistheoremisthattheall-zeroescodewordhastobeinanylinear code,becauseitresultsfromaddingacodewordtoitself.
For example, the block code defined by codewords 000,101,011 is not a linear code,
because 101 + 011 = 110 is nota codeword. But if we add 110 to the set, we get a lin
ear code because the sum of any two codewords is now another codeword. The code
000,101,011,110 hasaminimumHamming distanceof2 (thatis, thesmallestHamming
distancebetweenanytwocodewordsin2),andcanbeusedtodetectallsingle-biterrors thatoccurduringthetransmissionofacodeword. Youcanalsoverifythattheminimum Hammingdistanceofthiscodeisequaltothesmallestnumberof1’sinanon-zero code-word. Infact,that’sageneralpropertyofalllinearblockcodes,whichwestateformally below:
Theorem 5.6 Definetheweightofacodewordasthenumberof1’sintheword. Then,themini mumHammingdistanceofalinearblockcodeisequaltotheweightofthenon-zerocodewordwith thesmallestweight.
Toseewhy,usethepropertythattheHammingdistancebetweenanytwobit-strings of equal length isequal to the weight of their sum. Hence, theminimum value of the Hammingdistanceoverallpairsofcodewords,c1andc2,isequaltotheminimumvalue
oftheweightofthecodewordc1+ c2.Becausethecodeislinear,c1+ c2isalsoacodeword,
completingtheproof.
The rest of thissection showshow to construct linearblock codes over F2. For sim
plicity,andwithoutmuchlossofgenerality, wewillfocusoncorrectingsingle-biterrors. i.e., on single-errorcorrection(SEC) codes.. Wewill showtwo waysof buildingtheset S
oftransmissionmessagestohavesingle-errorcorrectioncapability,andwilldescribehow thereceivercanperformerrorcorrectiononthe(possiblycorrupted)receivedmessages.
WewillstartwiththerectangularparitycodeinSection5.5,andthendiscussthecleverer andmoreefficientHammingcodeinSection5.7.
⌅
5.5
Rectangular
Parity
SEC
Code
Wedefinetheparityofbitsx1, x2, . . . , xn as (x1 + x2 + . . .+ xn),wheretheadditionisper
formedmodulo2(it’sthesameastakingtheexclusiveORofthenbits).Theparityiseven
whenthesumis0(i.e.,thenumberofonesiseven),andoddotherwise.
Letparity(s) denote the parityof allthe bitsin thebit-string s. We’ll usea dot, ·, to indicatetheconcatenation(sequentialjoining)oftwomessagesoramessageandabit. For anymessage M (asequenceof one ormore bits),letw = M·parity(M). You shouldbe
ableto confirmthat parity(w) = 0. This code, which adds aparity bitto each message,
isalsocalledtheevenparitycode,becausethenumberofonesineachcodewordiseven. Evenparityletsusdetectsingleerrorsbecausethesetofcodewords,{w},eachdefinedas M·parity(M),hasaHammingdistanceof2.
IfwetransmitwwhenwewanttosendsomemessageM,thenthereceivercantakethe
receivedword,r,andcomputeparity(r) todetermineifasingleerrorhasoccurred. The
receiver’sparitycalculationreturns1ifanoddnumberofthebitsinthereceivedmessage hasbeencorrupted. When thereceiver’s paritycalculationreturns a1, wesaytherehas beenaparityerror.
d11 d12 d13 d14 p row(1)
d21 d22 d23 d24 p row(2)
p col(1) p col(2) p col(3) p col(4)
Figure5-3:A2⇥4arrangementforan8-bitmessagewithrowandcolumnparity.
0 1 1 0 0 1 1 0 1 1 1 0 1 1 (a) 1 0 0 1 1 0 0 1 0 1 1 0 1 0 (b) 0 1 1 1 1 1 1 1 0 1 1 0 0 0 (c)
Figure5-4:Examplereceived8-bitmessages.Which,ifany,haveoneerror?Which,ifany,havetwo?
ThissectiondescribesasimpleapproachtobuildinganSECcodebyconstructingmul tipleparitybits,eachovervarioussubsetsofthemessagebits,andthenusingtheresulting patternofparityerrors(ornon-errors)tohelppinpointwhichbitwascorrupted.
Rectangular code construction: Supposewewanttosendak-bitmessageM.Shapethe kbitsintoarectangulararraywithrrowsandccolumns,i.e.,k= rc.Forexample,ifk= 8,
thearraycouldbe 2 ⇥ 4 or 4 ⇥ 2 (oreven 8 ⇥ 1 or 1 ⇥ 8,thoughthosearelessinteresting).
Labeleachdatabitwithasubscriptgivingitsrowandcolumn: thefirstbitwouldbed11,
thelastbitdrc.SeeFigure5-3.
Definep row(i) tobetheparityofallthebitsinrowiofthearrayandletRbeallthe
rowparitybitscollectedintoasequence:
R = [p row(1),prow(2), . . . ,p row(r)]
Similarly,definepcol(j) tobetheparityofallthebitsincolumnj ofthearrayandletC
beallthecolumnparitybitscollectedintoasequence:
C = [pcol(1),p col(2), . . . ,pcol(c)]
Figure5-3showswhatwehaveinmindwhenk = 8.
Letw = M·R·C,i.e., thetransmitted codewordconsistsof theoriginal messageM,
followed bytherowparitybitsR in roworder,followed bythecolumnparitybits C in
columnorder. Thelengthofwisn = rc+ r+ c. Thiscodeislinearbecausealltheparity
bitsarelinearfunctionsofthemessagebits. Therateofthecodeisrc/(rc+ r+ c).
Wenowprovethattherectangularparitycodecancorrectallsingle-biterrors.
Proof of single-error correction property: Thisrectangularcode isanSEC codeforall valuesofr andc. Wewillshowthatitcancorrectallsinglebiterrorsbyshowingthatits
minimumHammingdistanceis3(i.e.,theHammingdistancebetweenanytwocodewords isatleast3). Considertwodifferentuncodedmessages,Mi andMj. Therearethreecases
todiscuss:
• If Mi and Mj differ by a single bit, then the row and column parity calculations
57 SECTION5.5. RECTANGULARPARITYSECCODE
wi andwj ,willdifferbythreebits: thedifferentdatabit,thedifferentrowparitybit,
andthedifferentcolumnparitybit. SointhiscaseHD(wi, wj ) = 3.
• If Mi and Mj differ by two bits, then either (1) the differing bits are in the same
row, inwhich casetherowparitycalculationisunchangedbut twocolumnparity calculationswilldiffer,(2)thedifferingbitsareinthesamecolumn,inwhichcasethe columnparitycalculationisunchangedbut tworowparitycalculationswilldiffer, or(3)thedifferingbitsareindifferentrowsandcolumns,inwhichcasetherewillbe tworowandtwocolumnparitycalculationsthatdiffer.SointhiscaseHD(wi, wj ) 4.
• IfMi andMj differby threeor more bits, thenHD(wi, wj ) 3 becausewi and wj
containMi andMj respectively.
HencewecanconcludethatHD(wi, wj ) 3andoursimple“rectangular”codewillbe
abletocorrectallsingle-biterrors.
Decoding the rectangular code: How can the receiver’s decoder correctly deduce M
fromthereceivedw,whichmayormaynothaveasinglebiterror? (Ifwhasmorethan
oneerror,thenthedecoderdoesnothavetoproduceacorrectanswer.)
Uponreceivingapossiblycorruptedw,thereceivercheckstheparityfortherowsand
columnsbycomputingthesumoftheappropriatedatabitsandthecorrespondingparity bit(allarithmeticinF2).Bydefinition,thissumwillbe1ifthereisaparityerror.Then:
• Iftherearenoparityerrors,thentherehasnotbeenasingleerror,sothereceivercan
usethedatabitsas-isforM. ThissituationisshowninFigure5-4(a).
• Ifthereissingleroworcolumnparityerror,thenthecorrespondingparitybitisin
error.Butthedatabitsareokayandcanbeusedas-isforM.Thissituationisshown
inFigure5-4(c),whichhasaparityerroronlyinthefourthcolumn.
• If thereisone row andone columnparity error, thenthe databit in that rowand
columnhasanerror.Thedecoderrepairstheerrorbyflippingthatdatabitandthen usesthe repaireddata bitsfor M. Thissituation isshownin Figure 5-4(b), where
thereareparityerrorsinthefirstrowandfourthcolumnindicatingthatd14should
beflippedtobea0.
• Other combinationsof rowand column parityerrors indicatethat multipleerrors
have occurred. There’s no “right” action the receiver can undertake because it doesn’thavesufficientinformationtodeterminewhichbitsareinerror. Acommon approachistousethedatabitsas-isforM.Iftheyhappentobeinerror,thatwillbe
detectedbytheerrordetectioncode(mentionednearthebeginningofthischapter). Thisrecipewillproducethemostlikelymessage,M,fromthereceivedcodewordifthere
hasbeen atmosta singletransmission error(and ifthe biterror probabilityisless than
1/2). p
Intherectangularcodethenumberofparitybitsgrowsatleastasfastas k(itiseasyto
verifythatthesmallestnumberofparitybitsoccurswhenthenumberofrows,r,andthe
Figure5-5: Acodewordinsystematicformforablockcode. Anylinearcodecanbetransformedintoan equivalentsystematiccode.
orresource,we’reinterestedindevotingasmuchofitaspossibletosendingmessagebits, not parity bits. Are there other SEC codes that have better code rates than our simple rectangularcode? Anaturalquestionto askis: howlittleredundancycanwegetawaywith andstillmanagetocorrecterrors?
TheHammingcodeusesacleverconstructionthatusestheintuitiondevelopedwhile answeringthequestionmentionedabove.Weanswerthisquestionnext.
⌅
5.6
How
many
parity
bits
are
needed
in
an
SEC
code?
Let’s think about what we’re trying to accomplish with an SEC code: the correction of transmissions that have a single error. For a transmitted message of length n there are n + 1situationsthe receiver hasto distinguish between: no errors anda single error in
aspecified positionalongthestringof nreceivedbits. Then,dependingon thedetected
situation,thereceivercanmake,ifnecessary,theappropriatecorrection.
Ourfirstobservation, which wewill statehere without proof, isthat anylinearcode canbetransformedintoanequivalentsystematic code. Asystematiccodeisone where every n-bit codeword can be represented as the original k-bitmessage followed by the n-kparitybits(itactuallydoesn’tmatterhowtheoriginalmessagebits andparitybits
areinterspersed). Figure5-5showsacodewordinsystematicform.
So,given a systematic code, how manyparity bitsdo we absolutelyneed? Weneed to choose nso that singleerror correction ispossible. Since there are n-k parity bits,
eachcombinationofthese bitsmustrepresentsomeerrorconditionthat wemustbeable to correct(or inferthat therewere no errors). There are 2n-k possibledistinct paritybit
combinations,whichmeans thatwe candistinguishatmostthatmany errorconditions. Wethereforearriveattheconstraint
n-k
n + 1 2 (5.6)
i.e., there have to be enough parity bits to distinguishall corrective actions that might needto betaken(includingnoaction). Givenk, wecandeterminen-k, thenumber of
paritybits neededto satisfythisconstraint. Taking thelog(tobase 2)of bothsides, we canseethat thenumberofparitybits must growatleastlogarithmicallywiththenumber ofmessagebits. Notallcodesachievethisminimum(e.g.,therectangularcodedoesn’t), buttheHammingcode,whichwedescribenext,does.
59 SECTION5.7. HAMMINGCODES
d1 p1 p2 p3 d2 d3 d4 d1 p1 p2 p3 d2 d3 d4 p4 d9 d11 d10 d6 d7 d5 d8
(a)(7,4)code (b) (15,11)code
Figure5-6:VenndiagramsofHammingcodesshowingwhichdatabitsareprotectedbyeachparitybit.
lowerboundonthenumberofparitybitsneededtocorrectt >1errors.
⌅
5.7
Hamming
Codes
Intuitively, it makes sense that for a code to be efficient, each parity bit should protect asmany databitsas possible. Bysymmetry, we’dexpecteachparitybit todo thesame amountof“work”inthesensethateachparitybitwouldprotectthesamenumberofdata bits.Ifsomeparitybitisshirkingitsduties,it’slikelywe’llneedalargernumberofparity bitsinorder toensurethateachpossiblesingleerrorwillproduceauniquecombination ofparityerrors(it’stheuniquecombinationsthatthereceiverusestodeducewhichbit,if any,hadanerror).
The class of Hamming single error correcting codes is noteworthy because they are particularlyefficientintheuseofparitybits:thenumberofparitybitsusedbyHamming codesgrowslogarithmicallywiththesizeofthecodeword.Figure5-6showstwoexamples oftheclass: the(7,4)and(15,11)Hammingcodes. The(7,4)Hammingcodeuses3parity bitstoprotect4databits;3ofthe4databitsareinvolvedineachparitycomputation.The (15,11)Hammingcodeuses4paritybitstoprotect11databits,and7ofthe11databitsare usedineachparitycomputation(thesepropertieswillbecomeapparentwhenwediscuss thelogicbehindtheconstructionoftheHammingcodeinSection5.7.1).
Lookingatthediagrams,whichshowthedatabitsinvolvedineachparitycomputation, youshouldconvinceyourselfthateachpossiblesingleerror(don’tforgeterrorsinoneof theparitybits!) resultsinauniquecombinationofparityerrors. Let’sworkthroughthe argumentforthe(7,4)Hammingcode.Herearetheparity-checkcomputationsperformed bythereceiver:
E1 = (d1+d2+d4+p1)
E2 = (d1 +d3 +d4 +p2)
whereeachEi iscalledasyndromebitbecauseithelpsthereceiverdiagnosethe“illness”
(errors) in the received data. For each combination of syndrome bits, we can look for thebits ineach codewordthat appear inall theEi computations that produced1; these
bitsarepotentialcandidatesforhavinganerrorsinceanyofthemcouldhavecausedthe observedparityerrors.NoweliminatefromthecandidatesthosebitsthatappearinanyEi
computationsthatproduced0sincethosecalculationsprovethosebitsdidn’thaveerrors. We’llbeleftwitheithernobits(noerrorsoccurred)oronebit(thebitwiththesingleerror). Forexample, if E1 = 1, E2 = 0andE3 = 1, we noticethatbits d2 andd4 bothappear
inthecomputations forE1 andE3. However, d4 appearsin thecomputation forE2 and
shouldbeeliminated,leavingd2asthesolecandidateasthebitwiththeerror.
Anotherexample: supposeE1= 1,E2= 0andE3= 0.Anyofthebitsappearinginthe
computation forE1 couldhavecaused theobservedparityerror. Eliminating thosethat
appearinthecomputationsforE2 andE3,we’releftwithp1,whichmustbethebitwith
theerror.
Applyingthisreasoningtoeachpossiblecombinationofparityerrors,we canmakea tablethat showstheappropriatecorrectiveactionforeachcombinationofthesyndrome bits: E3E2E1 000 001 010 011 100 101 110 111 CorrectiveAction noerrors
p1hasanerror,fliptocorrect
p2hasanerror,fliptocorrect
d1hasanerror,fliptocorrect
p3hasanerror,fliptocorrect
d2hasanerror,fliptocorrect
d3hasanerror,fliptocorrect
d4hasanerror,fliptocorrect
⌅ 5.7.1 Is There a Logic to the Hamming Code Construction?
Sofarsogood,buttheallocationofdatabitstoparity-bitcomputationsmayseemrather arbitraryandit’snotclearhowtobuildthecorrectiveactiontableexceptbyinspection.
TheclevernessofHammingcodesisrevealedifweorderthedataandparitybitsina certainwayandassigneachbitanindex,startingwith1:
index binaryindex 1 001 2 010 3 011 4 100 5 101 6 110 7 111 (7,4)code p1 p2 d1 p3 d2 d3 d4
Thistablewasconstructedbyfirstallocatingtheparitybitstoindicesthatarepowers oftwo(e.g.,1,2,4,...). Thenthedatabitsareallocatedtotheso-farunassignedindicies, startingwith thesmallest index. It’s easyto seehow to extend thisconstructionto any numberofdatabits,rememberingtoaddadditionalparitybitsatindicesthatareapower oftwo.
Allocatingthedatabitstoparitycomputationsisaccomplishedbylookingattheirre spectiveindicesinthetableabove.Notethatwe’retalkingabouttheindexinthetable,not thesubscriptofthebit. Specifically,di isincludedinthecomputationofpj if(andonlyif)
61 SECTION5.7. HAMMINGCODES
thelogicalANDofbinaryindex(di) andbinaryindex(pj ) isnon-zero.Putanotherway,di
isincludedinthecomputationofpj if,andonlyif,index(pj ) contributestoindex(di) when
writingthelatterassumsofpowersof2.
Sothecomputationofp1(withanindexof1)includesalldatabitswithoddindices:d1,
d2andd4. Andthecomputationofp2 (withanindexof2)includesd1,d3 andd4. Finally,
thecomputationofp3 (withanindexof4)includesd2,d3 andd4. Youshouldverifythat
thesecalculationsmatchtheEi equationsgivenabove.
Iftheparity/syndromecomputationsareconstructedthisway,itturnsoutthatE3E2E1,
treatedasabinarynumber,givestheindexofthebitthatshouldbecorrected. Forexam ple,if E3E2E1 = 101, thenwe should correctthemessage bitwith index5, i.e.,d2. This
correctiveactionisexactlytheonedescribedintheearliertablewebuiltbyinspection. TheHammingcode’ssyndromecalculationandsubsequentcorrectiveactioncanbeef ficiently implementedusingdigitallogicandsothese codesare widelyusedincontexts wheresingleerrorcorrectionneedstobefast,e.g.,correctionofmemoryerrorswhenfetch ingdatafromDRAM.
⌅
Acknowledgments
ManythankstoKatrinaLaCurtsandYuryPolyanskiyforcarefullyreadingthesenotesand makingseveralusefulcomments,andtoSigtryggurKjartanssonfordetectinganerror.
---⌅
Problems
and
Questions
1. ProvethattheHammingdistancesatisfiesthetriangleinequality.Thatis,showthat HD(x, y) +HD(y, z) HD(x, z)foranythreen-bitbinarywords.
2. Considerthefollowingrectangularlinearblockcode:
D0 D1 D2 D3 D4 | P0
D5 D6 D7 D8 D9 | P1
D10 D11 D12 D13 D14 | P2
P3 P4 P5 P6 P7 |
Here,D0–D14aredatabits,P0–P2arerowparitybitsandP3–P7arecolumnparity
bits.Whataren,k,anddforthislinearcode?
3. ConsiderarectangularparitycodeasdescribedinSection5.5.BenBitdiddlewould like usethiscode ata varietyof differentcoderatesandexperiment withthem on somechannel.
(a) Isit possibletoobtainaratelowerthan1/3 withthiscode? Explainyour an swer.
(b) Supposeheisinterestedincoderateslike 1/2, 2/3,3/4, etc.;i.e., ingenerala
`
rateof `+1,forsomeinteger`>1.Isitalwayspossibletopicktheparametersof
thecode(i.e,theblocksizeandthenumberofrowsandcolumnsoverwhichto
`
constructtheparitybits)sothatanysuchcoderateoftheforml+1isachievable? Explainyouranswer.
4. Two-BitCommunications(TBC), a slightlysuspect network provider, usesthefol lowinglinearblockcodeoveritschannels.AllarithmeticisinF2.
P0=D0, P1= (D0+D1), P2 =D1.
(a) Whatarenandkforthiscode?
(b) Supposewewanttoperformsyndromedecodingoverthereceivedbits. Write outthethreesyndromeequationsforE0, E1, E2.
(c) Fortheeightpossiblesyndromevalues,determinewhaterrorcanbedetected (none, error in aparticular data or paritybit, or multipleerrors). Make your choiceusingmaximumlikelihood decoding, assumingasmall biterror prob ability(i.e.,thesmallestnumberoferrorsthat’sconsistentwiththegivensyn drome).
(d) Supposethat thethe5-bitblocks arriveatthereceiver inthefollowing order:
D0, D1, P0, P1, P2. If11011arrives,whatwilltheTBCreceiverreportasthere
ceiveddataaftererrorcorrectionhasbeenperformed?Explainyouranswer. (e) TBCwouldliketoimprovethecoderatewhilestillmaintainingsingle-biterror
correction. Theirengineerwouldliketoreducethenumberofparitybitsby1. GivetheformulasforP0andP1thatwillaccomplishthisgoal,orbrieflyexplain
63 SECTION5.7. HAMMINGCODES
5. PairwiseCommunicationshasdevelopedalinearblockcodeoverF2withthreedata
andthreeparitybits,whichitcallsthepairwisecode:
P1 = D1+ D2 (EachDi isadatabit;eachPi isaparitybit.)
P2 = D2+ D3
P3 = D3 + D1
(a) Fillinthevaluesofthefollowingthreeattributesofthiscode: (i)Coderate=
(ii)Numberof1sinaminimum-weightnon-zerocodeword= (iii)MinimumHammingdistanceofthecode=
6. Considerthesame “pairwisecode”as inthepreviousproblem. Thereceiver com putes three syndrome bits from the (possiblycorrupted) received data andparity bits: E1 = D1+ D2+ P1, E2= D2+ D3+ P2,andE3= D3+ D1+ P3. The receiver
performs maximumlikelihood decoding using thesyndrome bits. For thecombi nationsofsyndromebitsinthetablebelow,statewhatthemaximum-likelihoodde coderbelieveshasoccured:noerrors,asingleerrorinaspecificbit(statewhichone), ormultipleerrors.
E3E2E1 Errorpattern[Noerrors/Errorinbit...(specifybit)/Multipleerrors]
0 0 0 0 0 1 0 1 0 0 1 1 1 0 0 1 0 1 1 1 0 1 1 1
7. AlyssaP.Hackerextendstheaforementionedpairwisecodebyaddinganoverallpar ity bit. That is, she computes P4 = P3i=1(Di + Pi), andappends P4 to each origi
nal codeword to produce thenew set of codewords. What improvement in error correction or detectioncapabilities, ifany, does Alyssa’sextendedcode show over Pairwise’soriginalcode?Explainyouranswer.
8. Foreachofthesetsofcodewordsbelow,determinewhetherthecodeisalinearblock codeoverF2ornot.Alsogivetherateofeachcode.
(a) {000,001,010,011}.
(b) {000,011,110,101}.
(c) {111,100,001,010}.
(d) {00000,01111,10100,11011}. (e) {00000}.
9. Foranylinearblockcodeover F2 withminimumHammingdistanceatleast 2t+ 1
betweencodewords,showthat:
✓
n◆ ✓n◆ ✓n◆
2n-k 1 + + + . . . .
1 2 t
Hint:Howmanyerrorscansuchacodealwayscorrect?
10. For each (n, k, d) combination below, statewhether a linear block codewith those
parametersexistsor not. Pleaseprovideabriefexplanationforeachcase: ifsucha codeexists,giveanexample;ifnot,youmayrelyonasuitablenecessarycondition.
(a) (31,26,3): Yes / No
(b) (32,27,3): Yes / No
(c) (43,42,2): Yes / No
(d) (27,18,3): Yes / No
(e) (11,5,5): Yes / No
11. UsingtheHammingcodeconstructionforthe (7,4) code,constructtheparityequa
tions forthe (15,11) code. How manyequationsdoesthiscodehave? How many
messagebitscontributetoeachparitybit?
12. ProveTheorems5.2and5.3.(Don’tworrytoomuchifyoucan’tprovethelatter;we willgivetheproofinthenextchapter.)
13. The weight of a codeword in a linearblock code over F2 is the number of 1’s in
theword. Showthatanylinearblockcode musteither: (1) haveonlyeven weight codewords,or(2)haveanequalnumberofevenandoddweightcodewords.
Hint:Proofbycontradiction.
14. ThereareN peopleinaroom,eachwearingahatcoloredredorblue,standingina
lineinorderofincreasingheight. Eachpersoncanseeonlythehatsofthepeoplein front,anddoesnotknowthecolorofhisorherownhat.Theyplayagameasateam, whoserulesaresimple. Eachpersongets tosay oneword: “red” or “blue”. Ifthe wordtheysaycorrectlyguesses thecolor oftheirhat,theteamgets1point; ifthey guesswrong,0points. Beforethegamebegins, theycanget togethertoagreeon a protocol(i.e.,whatwordtheywillsayunderwhatconditions).Oncetheydetermine theprotocol,theystoptalking,formtheline,andaregiventheirhatsatrandom. Can youdevelop aprotocol thatwill maximizetheirscore? What score doesyour protocolachieve?
MIT OpenCourseWare
http://ocw.mit.edu
6.02
Introduction to EECS II: Digital Communication Systems
Fall 2012
