Leonid A.Levin
ThesearethenotesforthecourseCS-172ItaughtintheFallof1986atUCBerkeley.Thegoalwastointroduce
theundergraduatestobasicconceptsofTheoryofComputationandtoprovoketheirinterestinfurtherstudy. The
model-dependent eects were systematically ignored. Concrete computational problems were considered only as
illustrationsofgeneralprinciples. Thenotes(preparedbythestudentsandrevisedbyme)areskeletal: theydohave
(terse)proofs, butexercises,references, intuitivecommentsand examplesare missingor inadequate. Thebetter is
Englishinaparagraphthesmallerwasmycontributionandthegreatercautionisneeded.
Thenotescanbeusedbyaninstructordesigningacourseorbystudentswhoeitherknowthematerialandwantto
refreshthememoryorareexceptionallybrightandhaveaccesstoaninstructorforquestions.Eachsubsectiontakes
aboutaweekofthecourse. Idistributethesenotestogeneratesomediscussiononthe contentoftheintroductory
theorycourse. I ammostinterested inyour comments,bothgeneral andonparticular account details. My home
addressis: 460 CommonwealthAvenue,Newton, MA 02459-1333; tel.: (617) 332-9492; e-mail: Lnd (@bu.edu). I
keepupdatingthesenotes: takeafreshcopyfromhttp://www.cs.bu.edu/fac/lnd/toc/(z.psorz.dvi).
Acknowledgments. I amgrateful to the University of California at Berkeley, itsMacKey Professorship fund
and ManuelBlumwho made possiblefor metoteachthis course. Theopportunity toattend lecturesofM. Blum
andRichardKarpandmanyideasofmycolleaguesatBUandMITwereverybenecialfor mylectures. Iamalso
gratefultotheCaliforniaInstituteofTechnologyforasemesterwithlightteachingloadinastimulatingenvironment
enablingmetorewritethestudents'notes. NSFgrants#DCR-8304498,DCR-8607492,CCR-9015276alsosupported
thework. Andmostofall Iamgrateful tothe studentswho notonlyhaveoriginally writtenthesenotes,butalso
inuencedthelecturesalotbyprovidingveryintelligentreactionsandcriticism.
Contents
1 Modelsof Computations;PolynomialTime &Church'sThesis. 2
1.1 DeterministicComputation. . . 2
1.2 RigidModels. . . 3
1.3 PointerMachines.. . . 4
1.4 Simulation. . . 5
2 Universal Algorithm;Diagonal Results. 6 2.1 UniversalTuringMachine. . . 6
2.2 Uncomputability;GodelTheorem. . . 7
2.3 Intractability;CompressionandSpeed-upTheorems. . . 8
3 Games;Alternation;ExhaustiveSearch;Time v. Space. 9 3.1 HowtoWin. . . 9
3.2 ExponentiallyHardGames. . . 10
3.3 Reductions;Non-DeterministicandAlternating TM;TimeandSpace. . . 11
3.4 FastandLeanComputations. . . 12
4 Nondeterminism;InvertingFunctions;Reductions. 13 4.1 ExampleofaNarrowComputation: InvertingaFunction. . . 13
4.2 ComplexityofNPProblems. . . 14
4.3 AnNP-CompleteProblem: Tiling. . . 15
5 Randomnessin Computing. 16 5.1 AMonte-CarloPrimalityTester. . . 16
5.2 RandomizedAlgorithmsandRandomInputs. . . 17
5.3 RandomnessandComplexity. . . 18
5.4 Pseudo-randomness. . . 19
5.5 Cryptography.. . . 20
1.1 Deterministic Computation.
Sections1,2study deterministiccomputations. Non-deterministicaspects ofcomputations(inputs,
interac-tion,errors,randomization,etc.) arecrucialandchallenginginadvancedtheoryandpractice. Deningthem
asan extension of deterministic computations is simple. The latter, however, while simplerconceptually,
requireelaborate modelsfor denition. These models maybe sophisticatedif we needa precise
measure-mentofallrequiredresources. However,ifweonlyneedtodenewhatiscomputableandgetaveryrough
magnitudeoftheneededresources,allreasonablemodelsturnoutequivalent,eventothesimplestones. We
willpaysignicantattentiontothissurprisingandimportantfact. Thesimplestmodelsaremostusefulfor
provingnegativeresultsandthestrongestones forpositiveresults.
Westart with terminologycommon to allmodels, gradually makingit morespecic to the models we
actuallystudy. Computationsconsistofeventsandcanberepresentedasgraphs,whereedgesbetweenevents
reectvariousrelations. Nodesandedgeswillhaveattributescalledlabels,states,values,colors,parameters,
etc. Werequiredierentlabelsforanytwoedgeswiththesamesource. Edgesofonetype,calledcausal,run
fromeacheventxto alleventsessentialfortheoccurrenceorattributes ofx. Theyformadirectedacyclic
graph(though cyclesaresometimesaddedarticiallytomarktheinputparts ofthecomputation).
Wewill study only synchronous computations. Their nodes have atime parameter. It reectslogical
steps,notnecessarilyaprecisevalueofanyphysicalclock. Causaledgesonlyrunbetweeneventswithclose
(typically,consecutive) valuesoftime. Oneeventamong thecausesof anode iscalled itsparent. Pointer
edges connect the parent of each event to all its other possible causes. Pointers reect the connection
betweensimultaneouseventsthatallowsthemtointeractandhaveajointeect. Thesubgraphofeventsat
aparticularvalueoftime(withpointersandattributes)isaninstantmemorycongurationofthemodel.
Eachnon-terminalcongurationhasactivenodes/edgesaroundwhich itmaychange. Themodelswith
onlyasingleactiveareaat anystepof thecomputationaresequential. Othersarecalledparallel.
Complexity.
ThefollowingmeasuresofcomputingresourcesofamachineAoninputxwillbeusedthroughoutthecourse:
Time: ThegreatestdepthD
A(x)
ofcausalchainsisthenumberofcomputationsteps. ThevolumeV
A(x)
isthecombinednumberofactiveedgesduringallsteps. TimeT
A(x)
isused (dependingon thecontext)as
eitherdepth orvolume,whichcoincide forsequentialmodels.
Space: S
A(x) orS
A
(x)ofasynchronouscomputationisthegreatest(overtime)sizeofitscongurations.
Sometimesexcludedarenodesunchangedsincetheinput.
Notethat timecomplexityisrobustonlyuptoaconstantfactor: amachinecanbemodiedintoanew
onewithalargeralphabetoflabels,representingseverallocationsinone. Itwouldproduceidenticalresults
in afraction of time and space (provided that the time limitsare suÆcientfor the transformation of the
inputintoandoutputfrom thenewalphabet).
GrowthRate Notations: f(x)=O(g(x)) 1
() g(x)=(f(x)) () sup
x f(x)
g(x) <1.
o;!:f(x)=o(g(x)) () g(x)=!(f(x)) () lim
x!1 f(x)
g(x) =0.
:f(x)=(g(x)) () (f(x)=O(g(x))andg(x)=O(f(x))).
Here are a few examples of frequently appearing growth rates: negligible (logn) O(1)
; moderate n (1)
(calledpolynomialorP,likeinP-time);infeasible2 n
(1)
;anotherinfeasible: n!=(n=e) n
p
t+2n;t2[1;2].
The reasonfor ruling outexponential(andneglecting logarithmic)rates isthat the known Universe is
toosmall toaccommodateexponents. Beingabout15 billionsyearsold, itisat most15billionlightyears
10 61
Plank Unitswide. A systemof R 1:5
particlespacked in R PlankUnits radius collapsesrapidly,
beittheUniverseoraneutronstar. Sothenumberofparticlesis<10 91:5
2 304
4 4
4
5!!.
1.2 Rigid Models.
Rigid computations haveanothernode parameter: location orcell. Combinedwithtime, it designatesthe
event uniquely. Locations have structure or proximity edges between them. They (or their short chains)
indicate allneighbors ofanodetowhichpointersmaybedirected.
CellularAutomata (CA).
CAare aparallelrigidmodel. Itssequentialrestrictionis theTuring Machine (TM). Thecongurationof
CA is a (possibly multi-dimensional) grid with axed (independent of the grid size) number of states to
labeltheevents. Thestatesinclude,amongothervalues,pointerstothegridneighbors. Ateachstepofthe
computation, thestateofeach cellcanchangeas prescribedbyatransition functionof thepreviousstates
ofthecellanditspointed-toneighbors. TheinitialstateofthecellsistheinputfortheCA.Allsubsequent
statesare determinedbythetransitionfunction (alsocalledprogram).
Anexampleofapossibleapplicationof CAisaVLSI (verylargescaleintegration)chiprepresentedas
agridofcellsconnectedbywires (chainsofcells) ofdierentlengths. Thepropagationofsignalsalongthe
wires issimulatedbychangingthestateof thewirecellsstepbystep. Theclockintervalcanbeset tothe
timethesignalspropagatethroughthelongestwire. Thiswaydelaysimplicitlyaect thesimulation.
An example: the Game of Life (GL).
Consideraplanegridofcells,eachhavinga1-bitstate(dead/alive)andpointerstothe8naturalneighbors.
Thecellremainsdeadoraliveifthenumberi ofitsliveneighborsis 2. Itbecomes(orstays)aliveifi=3.
Inallothercasesitdies(ofoverpopulationorsolitude).
AsimulationofamachineM
1 byM
2
isacorrespondencebetweenmemorycongurationsofM
1 andM
2
whichispreservedduringthecomputation(maybewithsometimedilation). Suchconstructionsshowthat
thecomputation of M
1
on any inputx canbe performed by M
2
aswell. GL cansimulate anyCA (seea
sketchofaningeniousproofinthelastsectionof[Berlekamp, ConwayGuy82])inthis formalsense:
Wexspace andtimeperiodsa;b. Cells(i;j)ofGL aremapped tocell(bi=ac;bj=ac)of CAM
(com-pressingaablocks). WerepresentcellstatesofM bystatesofaablocksofGL. This correspondence
ispreservedafteranynumbertstepsofM andbtstepsofGLregardlessofthestartingconguration.
Turing Machines.
Consideraninnite(say,totheright)chainortapeofcellswithtwoadjacentneighborseach. Eachstateof
acellhasapointertooneneighbor. TheinputtothisCAisanarrayofrightwardcellsfollowedattheright
byleftwardblanks. A cellchangesstate, ifand onlyifit and itsneighbor\face",i.e. point toeach other.
Thetransitionfunctionpreventsthecellsfromeverturning\back-to-back." Wecanusethese1-pointerCA
asadenition oftheTM.The pairofactivecellscanbeviewedastheTM'smovinghead(the cellwhich
just changedthepointer direction)andthetapesymbolitworkson.
AnothertypeofCArepresentsaTMwithseveralnon-communicatingheadswhichcanemergefromand
disappearintotherstcell(which, thus, controlsthenumberof activecells). Thecomputationhaltswhen
allheadsvanish. Thismodel hasconvenienttheoreticalfeatures. E.g.withlinear(inT)number(jpj 2
T)of
statechanges(volume)onecansolvetheBoundedHaltingProblemH(p;x;T): ndoutwhetherthemachine
withaprogrampstopsonaninputxwithin volumeT ofcomputation.
Problem: Findamethodtotransformanygivenmulti-headTMAintoanotheroneBsuchthatthevalue
oftheoutputofB(x)(asabinaryinteger)andthevolumesofcomputationofA(x)andofB(x)areallequal
withinaconstantfactor(forallinputsx).
Hint: B may keep a eld to simulate A and maintain (in other elds) two binary counters h for the
ThememorycongurationofaPointerMachine(PM),calledpointergraph,isanitedirectedlabeledgraph.
Onenodeismarkedasrootandhasdirectedpathstoallothernodes. Thenodesactasautomatachanging
thegraphlocally. Edges(pointers)are labeledwith colors from anite alphabet xed for allinputs. The
pointerscomingoutofanodemusthavedierentcolors(whichboundstheoutdegree).
Nodes see the conguration of their out-neighborhood of constant (2 suÆces)depth. Some colors (as
wellas theircarryingedgesandthenodesseeing them)arecalledactiveandarenotusedininput/output.
Active pointers must have inverses; their source nodes must have active loop-edges. An active node can
create/deletepointers toneighborsandcreate newnodeswithpointerstoand fromit. It doesthat,based
onitsneighborhood,accordingtotheprogramwhichisthesameforallnodes. Nodeswithnopointersvanish.
Nodes withno path from theroot arepermanentlyinvisible and eectivelyremoved. Thecomputation is
initiatedby insertinganactiveloop-edge into theroot. Whennoactivepointersremain, the graphisthe
output.
The subgraph of active pointers is kept connected and acyclic: A node cannot set or remove active
pointersto othernodeswithactiveloops. Activeloopsmustberemovedifnoincidentactiveedgesexist.
Itisconvenientto assumePMnodestoactin twostages: Atpullingstageone,foreach2-pointerpath
colored x;y, creates anew pointer witha special colorxy. Then the node removes/recolorspointers and
createsnewnodesbasedonwhatcolorsitspointershaveand whichoftheirsinksareidentical.
Problem:DesignaPMtransformingtheinputgraphintothesameonewithanextrapointerfromeach
nodetotheroot. Hint: Nodeswithnopathtotherootcanneverbeactivated. Theyshouldbecopiedwith
pointers, copiesconnectedto theroot, thentheoriginalinputremoved.
PointerMachinescanbeeither Parallel, PPM [Barzdin'Kalnin's74] orSequential. Thelatterdierby
therestrictionthat onlynodespointedto bytherootcanbeactive.
AKolmogorov or Kolmogorov-Uspenskii Machine (KM) [KolmogorovUspenskii 58], is aspecial caseof
PointerMachine[Shonhage80]withtherestrictionthatallpointershaveinverses. Thisimpliesthebounded
in/out-degreeofthegraphwhichwefurtherassumetobeconstant.
FixedConnectionMachine(FCM)isavariantofthePKMwiththerestrictionthatpointersoncecreated
cannot beremoved,only re-colored. Sowhenthe memorylimitsarereached,thestructure ofthe machine
cannotbealteredandthecomputationcan becontinuedonlybychangingthecolorsofthepointers.
PPMisthemostpowerfulmodel weconsider: itcansimulate theothersin thesamespace/time. E.g.,
cellularautomata makeasimplespecialcaseofaPPMwhichrestrictsthePointerGraphtobeagrid.
ExampleProblem. Designamachineofeachmodel(TM,CA,KM,PPM)whichdeterminesifaninput
stringx is adouble (i.e. hasaform ww, w2fa;bg
). Analyze time and space. KM/PPM takesinput x
intheformofcolorsofedgesinachainofnodes,withrootlinkedto bothends. ThePPMnodesalsohave
pointersto theroot. BelowarehintsforTM,PM,CA.ThespaceisO(jxj) in allthreecases.
Turingand PointerMachines. TMusesextrasymbolsA;B. Firstndthemiddleofwwby
capital-izingthelettersatbothendsonebyone. Thencompareletterbyletterthetwohalves,loweringthecaseof
thecomparedletters. Thecomplexityis: T(x)=O(jxj 2
). PMalgorithmissimilartotheTM's,exceptthat
theroot keepsandupdates thepointersto thebordersbetweentheupperand lowercasesubstrings. This
allowscommutingbetweenthesesubstringsinconstanttime. So,thecomplexityis: T(x)=O(jxj).
CellularAutomata. Thecomputationbeginsfromtheleftmostcellsendingrighttwosignals.Reaching
theendtherstsignalturnsback. Thesecondsignalpropagatesthreetimesslowerthantherst. Theymeet
inthemiddleofwwanddisappear. Whilealive,thesecondsignalcopiestheinputeld iofeachcellintoa
specialelda. Theasymbolswilltrytomoverightwheneverthenextcell'saeldisblank. Sothechainof
thesesymbolsalternatingwithblankswillstartmovingrightfromthemiddleofww. Whentheyreachthe
endtheywillpushtheblanksoutandpackthemselvesbackinto acopyoftheleft halfofwwshiftedright.
Whenanasymboldoesnothaveablankat therightto moveto, itcomparesitself withthe ield ofthe
samecell. Theyshouldbeidentical, ifthewwform iscorrect. Otherwiseasignalisgeneratedwhichhalts
1.4 Simulation.
Weconsideredseveraltypesofmachines(modelsof computation). Wewill seenowthatallthese machines
canbesimulatedbythesimplestofthem: theTuringMachine. Inotherwords,thesepowerfulmachinescan
computeonlythosefunctions computablebyaTM.
Church-Turing Thesis is a generalization of this conclusion: TMs can compute every function
com-putable in any thinkable physical model of computation. This is not a mathematical result because the
notionofmodelis notformallyspecied. Butthe longhistoryof investigations ofwaystodesignreal and
ideal computingdevices makesit veryconvincing. Moreover,this Thesis hasastrongerPolynomial Time
versionwhich states that if any model computesa function in polynomialvolume, TM can dothe same.
BothformsoftheThesisplayasignicantroleinfoundationsofComputerScience.
PKM Simulation of PPM. Forconvenience, we assumeeach PPM node hasanedge into root. Now,
for each node, wereconnect its incoming (in unlimited number) PPM edges, 2perleaf, to a bidirectional
binarytreewithnewPKMcolorsl;r;u. Thenumberofedgesincreasesatmost4times. Thenodessimulate
PPM'spullingstagebyextendingtheirtreesto doubledepth. Tosimulatethere-coloringstage,eachnode
getsabinarynameformedbythel;rcolorsonitspaththroughtheroottree. Thenitbroadcastsitsname
down its own tree. When each node thus receives identities and pointers of its PPM neighbors,it stores
them bycreatingalittleauxiliarychain(actingasTM). Thenit computesandimplementsthe actionsof
theoriginalPPMandrebalancesitstree. ThissimulationofaPPMsteptakespolylogarithmictime.
TM Simulation of PPM. To simulate the above PKM by a TM, we rst represent its memory
con-guration on the TM tape as the list of all pointers, sorted by the source names (described above) and
then by color. The PKMprogram is reected in the TM'stransition table. Now theTM cansimulate a
PKM'spullingstageasfollows: Itcreatesacopyofeachpointerand sortscopiesbytheirsinks. Noweach
pointer, located at source, has its copy near its sink. So both components of 2-pointer paths are nearby:
thespecial double-coloredpointers can be createdandmovedto their sourcesby sorting. There-coloring
stage is straightforward,as all relevant pointers havethesamesource and arelocated together. Whenno
activeedges remain in theroot, the Turing machine stops and its tape representsthe PKM output. Ifa
PPMcomputesafunctionf(x)int(x)steps,usings(x)nodes,thesimulatingTMusesspaceS=O(slogs),
(O(logs)bitsforeachofO(s)pointers)andtimeT =O(S 2
)t,asTM sortingtakesquadratictime.
FCMSimulationofPPM [Ofman65]. FCMrepresentsthepointergraphinthesamewayastheabove
TMandusesasimilarprocedure.Allstepsarestraightforwardtodolocallyinparallelpolylogarithmictime
exceptsortingpointers. Weneedtocreateaxedconnectionsortingnetworkwhichtakesanarbitraryarray
ofintegersasinputandoutputsitsorted. Sophisticatednetworkstakelogarithmictime. Butweneedonlya
simplerpolylogarithmicmethod, Merge-SortwithparallelBatcher-Merge: Arrayswith twoormoreentries
are separated in two halves and each sorted recursively. The Batcher-Merge combines twosorted lists in
parallel logarithmictime.
Batcher Merge: Callarrayentries i-th partnerswhen theiraddressesdieronlyin i-thbit. Operations
on partners canbeimplemented ona Shue Exchange graphof 2 k
nodes. Each node has pointers to its
k-thpartner andtoandfromitsshiftnodeobtainedbymovingitsrstaddressbittotheend.
A bitonic cycle is the combination of two sorted arrays (one may be shorter), connected by
max-to-max and min-to-min entries. We merge sorted arraysby appending the reversed second one to the rst,
consideringthelastandrstentriesasneighbors,andsortingtheresultingbitoniccycle.
The sorting of a 2 k
long bitonic cycle proceeds by comparing each entry with its k-th partner (i.e.
diametricopposite onthecycle)andswitching ifwronglyordered. Each half becomes thenabitoniccycle
2.1 Universal Turing Machine.
Therstcomputerswerehardware-programmable. Tochangethefunction computed,onehadtoreconnect
thewiresorevenbuildanewcomputer. JohnvonNeumannsuggestedusingTuring'sUniversalAlgorithm.
Thefunction computed canbethenspeciedby just givingitsdescription(program)aspartof theinput
ratherthanbychangingthehardware. Thiswasaradicalidea,sincein theclassicalmathematicsuniversal
functionsdonotexist(aswewill see).
Let R bethe classof allTM-computable total(dened forallinputs) and partial (which may diverge)
functions. Surprisingly,there isauniversalfunction uin R . Itsimulates anyother f 2R intime c 2
T and
space S+c, where S;T >jxj are space andtime ofcomputing f(x) and c is itsprogram length. This u
expectstheprexmofitsinputmxtolistthecommandsofaTuringMachineM anditsinitialheadstate.
Then u(mx) operates in cycles. Each cycle simulates one stepof M(x). Let after i stepsof M(x), l
i be
the left (from the head)part of its tape; r
i
be the rest of the tapeand s
i
be thehead's state. Thetape
congurationofu(mx)aftericyclesist
i =l
i ms
i r
i
. Thenulooksupmtondthecommandcorresponding
to the state s
i
and the rst character of r
i
and modies t
i
accordingly. WhenM(x) halts, u(mx) erases
ms
i
from thetapeandhaltstoo. UniversalMulti-headTMworkssimilarlybut canalso determinein time
O(t(x))whetherithaltsint steps(givenx;t(x) andanappropriateprogram). Wenowdescribeindetaila
simplerbut slightlysloweruniversalTM.
Thetransition tableattherightdenesasmall(11states+
6symbols)TMU byIkenowhichcansimulateanyotherTMM
overf0;1gtapealphabet withthefollowingstipulations(which
stillallowM tosimulateanyotherTMs): Thedirectionofhead
shiftis afunction of thenewpost-transitionstate(lowercase
-left, uppercase-right). Andsois, forM only,thedigittyped.
Thetapeisinnitetotherightonly:theleftstatesintheleftmost
cellremainthere. ForM only,thenewstateisthetapebitread
plus afunction of theold state. Inthe U table the statesand
tapedigitsareshownonlywhenchanged;exceptthat theprime
isalwaysshown. Thehaltandexternalinput/outputcommands
arespecialstatesforM;forU theyareshownas=.
1' 0' *' 1 0 *
A f f e0
B C C e1
fC c b* a* C
c = C E' ' '
a b' C E' ' ' '
b a' D ' ' '
d ' ' D ' '
D e' d' {
E ' ' e' = { '
e B A = ' ' '
U's tapeconsist of segments: each isa0/1 stringpreceded witha*. Somesymbolsareprimed. Each
nite segment describesa transition performed by onestate ofM and neverchanges (except for primes).
Therightmost (innite)segmentisalwaysacopyofM'stape,initiallywithU'sheadat thesamelocation
in the stateC. Each transition is representedas STW, where W is the symbol to write, T the direction
L/R to turn, represented as 0/1, S the new state (when 0 is read). S is represented as 1 k
, if the next
state is k segments to the left, or 0 k
(if to the right). Initially, primed are the digits of S in the
seg-ment correspondingto the initial stateof M andall digitsto their left. An exampleof theconguration:
0
0 0
0 0
0 0
1 0
0 0
0
0 0
0 0
0 0
0 0
01011:::00 head 00.
U rstreads thedigitof anM's cellchangingthestate fromC or f toa=b, puts a*there, movesleft
to the primedstatesegmentS, nds from it thenew statesegment and movesthere. With only 10head
states, U can't nd the new segment at once. So, it (alternating the states c=C or d=D) keeps priming
nearestunprimed*and1sofS (orunpriming0s). WhenSisexhaustedthetargetsegment,jSjstarsaway,
is reached. Then U reads (changing state from e to A=B) therightmost symbol W of the new segment,
copiesit at the*in theM area, goesback,reads thenextsymbolT,returnsto thejust overwritten(and
rst unprimed)cell of M areaand turns left orright. As CA, M and U have in each cell three standard
2.2 Uncomputability; Godel Theorem.
Universal and CompleteFunctions.
Notations: Let us choosea special mark and after its k-th occurrence, breakany stringx into Prex
k (x)
and SuÆx
k
(x). Let f +
(x) be f(Prex
k
(x) x) and f (x) be f(SuÆx
k
(x)). Wesay uk-simulates f i for
some p =Prex
k
(q);q 6= p and all x, u(px) = f(x). The prex can be intuitively viewed as a program
whichsimulatingfunction uappliesto thesuÆx(input). Wealsoconsider asymmetric variantofrelation
\k-simulate"whichmakessomeproofseasier. Namely,uk-intersectsf iu(px)=f(px)forsomeprex
k p
andallx. E.g.,lengthpreservingfunctionscanintersectbutnotsimulateone another.
WecalluniversalforaclassF,anyuwhichk-simulatesallfunctionsinF foraxedk. WhenFcontains
f ;f +
foreachf 2F,universalityisequivalentto[orimplies,ifonlyf +
2F]completeness: uk-intersects
allf 2F. Indeed,uk-simulatesf iitk-intersectsf ; u2k-intersectsf ifitk-simulatesf +
.
Anegationofa(partialortotal)functionf isthetotalpredicate:f whichyields1if(x)=0andyields
0otherwise. Obviously,noclosedunder negationclassoffunctionscontainsacompleteone. So,thereisno
universalfunctionintheclassofall(computableornot)predicates. ThisisthewellknownCantorTheorem
thatthesetofallsets ofstrings(aswellasthesets ofallpartial functions,realsetc.) isnotcountable.
GodelTheorem.
There is no complete function among the total computable(recursive) ones, as this class is closed under
negation. Sotheuniversalin Rfunction u(andu
2
=(umod2)) hasnototalcomputableextensions.
FormalproofsystemsarecomputablefunctionsA(P)whichcheckifP isanacceptableproofandoutput
the proven statement. ` s means s = A(P) for some P. A is rich i it allowscomputable translations
s
x
of statements \u
2
(x) = 0," provable whenever true, and refutable (` :s
x
), wheneveru
2
(x) =1. A is
consistentiat mostoneofanysuchpairs
x ;:s
x
isprovable,andcompleteiat leastoneofthemalways
(even when u(x) diverges) is. A rich consistent and complete formal system cannot exist, since it would
providean obvioustotalextensionu
A ofu
2
(byexhaustivesearchforP toproveorrefutes
x
). Thisisthe
famousGodel'sTheorem which wasoneoftheshockingsurprisesofthe scienceofourcentury. (HereA is
anextensionoftheformalPeanoArithmetic;weskipthedetailsofitsformalizationandproofofrichness.) 2
Recursive Functions. Another byproduct is that the Halting (of u(x)) Problem would yield a total
extension of u and, thus, is not computable. This is the source of many other uncomputability results.
Another sourceis an elegantFixed PointTheorem by S. Kleene: any totalcomputable transformation A
of programs (prexes) maps someprogram into an equivalent one. Indeed, the complete/universal u(px)
intersects computableu(A(p)x). This implies(exercise), e.g.,that the onlycomputableinvariant (i.e. the
sameonprogramscomputingthesamefunctions)propertyofprogramsisconstant(Rice-Uspenskii).
Computable(partialandtotal)functionsarealsocalledrecursive(duetoanalternativedenition). Their
ranges(and,equivalently,domains)arecalled (recursively) enumerableorr.e. sets. An r.e.setwithanr.e.
complementiscalled recursive(asisitsyes/nocharacteristicfunction) ordecidable. Afunctionis recursive
iitsgraphisr.e. Anr.e. graphofatotalfunctionisrecursive. Eachinniter.e.setistherangeofa1-to-1
totalrecursivefunction (\enumerating"it,hencethenamer.e.).
Wecanreduce membership problemof aset A to theoneof aset B bynding arecursivefunction f
s.t. x2 A () f(x) 2B. ThenA is called m-(or many-to-1-)reducible to B. A morecomplexTuring
reductionis given by analgorithm which, starting from input x,interacts with B by generating stringss
and receiving answersto s 2?B questions. Eventuallyit stops and tells ifx 2 A. R.e. sets (likeHalting
Problem)towhichallr.e.setscanbem-reducedarecalledr.e.-complete. Onecanshowasetr.e.-complete
(and,thus,undecidable)byreducingtheHaltingProblemtoit. SoJu.Matijasevichprovedr.e.-completeness
ofDiophantineEquationsProblem: givenamultivariatepolynomialofdegree4withintegercoeÆcients,nd
ifithasintegerroots. Theabove(andrelated)conceptsandfactsarebroadlyusedinTheoryofAlgorithms
andshould belearnedfrom anystandardtext,e.g.,[Rogers67].
2
Acloserlookatthisproofrevealsthe secondfamous Godeltheorem: the consistencyitselfisan exampleofunprovable
:s
x
. ConsistencyCofAisexpressibleinAasdivergenceofthesearchforcontradiction. u
2
intersects1 u
A
forsomeprex
a. Cimpliesthatu
A
extendsu
2
,and,thus,u
2 (a);u
A
(a)bothdiverge. So,C):s
a
.ThisproofcanbeformalizedinAwhich
The t-restriction u
t
of u aborts and outputs 1 if u(x) does not halt within t(x) steps, i.e. u
t
computes
thet-Bounded Halting Problem (t-BHP).It remainscomplete forthe classoffunctions computablewithin
o(t(x)) stepswhich isclosed undernegation. So,u
t
doesnotbelongto theclass, i.e.requires time(t(x))
[Tseitin 55]. E.g. 2 jxj
-BHP requiresexponentialtime. For similar reasonsanyfunction which agrees with
t-BHPonadense (i.e.havingstringswitheach prex)subsetcannotbecomputedin o(t(x))stepseither.
Ontheother hand,weknow that forsometrivialinput programsthe BHTcanbe answeredbyafast
algorithm. ThefollowingRabin's CompressionTheorem providesanotherpredicate P
f
(x) forwhich there
is only a nite number of such trivial inputs. The theorem is stated for the volume of computation for
Multi-HeadTuringMachine.ItcanbereformulatedintermsoftimeofPointerMachineandspace(or,with
smalleraccuracy,time)ofregularTuring Machine.
Denition: Afunction f(x)isconstructibleifsomealgorithm F computesit,in binary,within volume
O(f(x)), i.e. V
F(x)
=O(f(x)).
Herearetwoexamples: 2 jxj
isconstructible,asV
n
=O(nlogn)2 n
. 2 jxj
+h(x),whereh(x)is0or1,
dependingonwhetherU(x)haltswithin3 jxj
stepsisnotconstructible.
Theorem: Foranyconstructiblefunctionf,thereexistsafunctionP
f
suchthatforallfunctionsT,the
followingtwostatementsareequivalent:
1. There existsanalgorithmAsuchthatforallinputsx,A(x)computesP
f
(x)in volumeT(x).
2. t isconstructibleandf(x)=O(T(x)).
Lett-bounded Kolmogorov Complexity K
t
(i=x) ofi givenx bethe lengthofthe shortestprogrampfor
theUniversalMulti-HeadTuringMachinetransformingxintoiwith<tvolumeofcomputation. LetP
f (x)
bethe smallest i, with 2K
t
(i=x)> log(f(x)=t) for all t. P
f
can be computed in volume f by generating
allioflowcomplexity,sortingthemandtakingtherstmissing. It satisestheTheorem, sincecomputing
i=P
f
(x)fastercausesaviolationofcomplexitybounddeningit. P
f
canbemadeapredicate: see[Rabin
59].
Speed-upTheorem.
This theorem will be formulated for exponential speed-up, but it remains true if log is replaced by any
computableunbounded monotonefunction[Blum67].
Theorem: There exists atotal computablepredicate P such that for any algorithm computing P(x)
withrunningtimeT(x),thereexistsanotheralgorithmcomputingP(x)withrunningtimeO(logT(x)).
This procedure may continue any constant number of steps. In other words, there is no even nearly
optimalalgorithmforthepredicateP.
So,thecomplexityofsomepredicatesP cannotbecharacterizedbyasingleconstructiblefunctionf,asin
CompressionTheorem. However,theCompressionTheoremcanbegeneralizedbyremovingtherequirement
that f is constructible (itstill must becomputable or enumerablefrom below). In this form it isgeneral
enoughsothat everycomputablepredicate(or function) P satises thestatementof thetheorem withan
appropriate computable function f. There is no contradiction with Blum's Speed-up Theorem, sincethe
complexityf cannotbereached(f isnotconstructibleitself). Seeareviewin[Seiferas,Meyer].
Rabin's predicate has an optimal algorithm. Blum's does not. In general, one can't tell whether a
3 Games; Alternation; Exhaustive Search; Time v. Space.
3.1 How to Win.
Inthis sectionweconsider amoreinterestingprovably intractableproblem: playinggames withfull
infor-mation, twoplayersand zero sum. Wewill see that even for somesimple gamesthere cannot be amuch
moreeÆcientalgorithm,thanexhaustivesearchthroughallpossiblecongurations.
Therules of an n-player game G are set by a family f of information functions and a transition rule
r. Each playeri 2I at each stepparticipates in transformingaconguration (gameposition) x 2S into
the new congurationr(x;m); m : I ! M by choosing amove m
i
= m(i) basedonly on his knowledge
f
i
(x) ofx. Thegameproceeds until aterminal congurationst 2T S isreached. Thenf
i
(t) istheloss
(or gain) of the i-th player. We consider games with two-players. We identify S and M with the set of
integersandtakef(T)=T =I =f1g. Our gameswill havezero sum P
f
i
(t)=0and fullinformation:
f
i
(x) = ix;r(x;m) = r 0
(x;m
a(x)
), where a(x) points to the active player. We take a(x) = sign(x) and
r 0
(x;m)2fm; a(x)g.ThepredicateR (x;y)checksifr(x;y)=yi.e.whethertheruleacceptsthetransition
fromthecongurationxtoy (a legal move)orterminatesthegame.
Anexampleofsuchgamesischess. Examplesofgameswithoutfull informationarecardgames, where
onlypartf
i
(x)(player'sownhand)ofthepositionxisknown. EachplayermayhaveastrategySproviding
a next position y = S(x) for each position x. Once players choose strategies, the game (its moves) is
completelydetermined. Astrategyiswinning(non-losing)ifitguaranteesvictory(resp. survival)whatever
theopponentdoes,evenifheknowsS. Apositioniswinning,labeledW
i
,iftheactiveplayerhasani-step
winning strategy,losing (L
i
)ifhis opponentdoes,and neutral(N)ifbothhavenon-losingstrategies. The
lattercanbeexcludedbyaddingatimecounterdecrementedeachstep.
Solvingagame,meansdeterminingthewinningsideforeachposition. Theabilityisclosetotheability
to ndagood movein amodiedgame. Indeed,modifyagame R into R 0
byadding apreliminary stage
to it. Atthisstage theplayerAoersastartingpositionforR and heropponentB chooseswhichside to
play. Then AmayeitherstartplayingRoroeranew(lowerin someorder) startingposition. Obviously,
B winsifhecandeterminethewinningsideofeveryposition. IfhecannotwhileAcan,shewins.
Conversely, anygame Rcanbemodiedinto achoosing game R 0
in which thelistof alllegalmovesis
easytocomputeforanyposition. SolvingsuchgamesisobviouslysuÆcientforchoosingtherightmove. A
positionof R 0
consistsof aposition x ofR and asegmenty ofanother position extended with\?"s tothe
boardsize. Theactivesidereplacesone? with thenextbit ofy. When?saregone,hechecksR (x;y)and
ifthetransition islegal,replacesx withy,emptiesyand switchingtheactiveside(i.e.thesign).
Games may be categorized by thediÆculty to compute R . We will consider only R with < 2 jxj
time
(mostlymuchfaster). Ourboards alsoneverchangesize: R (x;y))jxj=jyj.
Theorem. Each positionofany fullinformation game iswinning,losing orneutral.
(Thistheorem [Neumann, Morgenstern44] fails forgames with partial information: either player may
lose if his strategy is known to the adversary. E.g.: 1. Blackjack (21); 2. Each player picks a bit; their
equalitydeterminesthe winner.) Thebestmovecanbefoundby playingall strategiesagainsteach other.
There are2 n
positions oflengthn, (2 n
) 2
n
=2 n2
n
strategiesand 2 n2
n+1
pairsofthem. Fora5-bitgame
thatis2 320
. TheproofofthisTheoremgivesamuchfaster(but stillexponentialtime!) strategy.
Proof: Make a graph of all jxj-bit positions and legal moves; set i = 0. Repeat: label L
i
all nodes
withoutmoves(ifnone,exit);labelW
i
allnodeswithamovetoL
i
;removealllabelednodesandadd1toi.
Forallpositions,thisprocedureguaranteesexistenceofthelegalmoves: W
i !L
i
;N !N (unlabeled);
L
i !W
i 1
. Anystrategy choosing thesemovesisobviouslyoptimal. Themovesguaranteednottoexistare
N !L;L!N;L!L;L
i !W
i ;W
i !L
<i
. Such labelingiscalledconsistentandisunique.
Indices are bounded by the number 2 jxj
of game congurations, since the number of Ns shrinks each
step. Thereare<2 2jxj
moves. Thetime todetermine thelegalmovesis<2 3jxj
. Thealgorithm trieseach
legalmoveineachrelabelingstep. Thus,itstotalrunningtimeis2 3jxj+1
: extremelyslow(2 313
fora13-byte
game)butstillmuchfasterthantheprevious(doubleexponential)algorithm.
Problem: theMatchGame. Consider3boxeswith3matcheseach: ! ! ! ! ! ! ! ! ! .
Asimpleexampleofafull informationgameisLinear Chess, playedonanitelinearboard. Eachpiece is
loyaltooneoftwosides: W(weak)orS(shy). ItisassignedagenderMorFandarankfromasetofranks
;this setdoesn'tdependon theboardsize. All theW's arealwaysontheleftside andalltheS'sonthe
right. All cellsofthe board arelled. Changesoccuronlyat theactiveborder whereW andS meet(and
ght). ThewinnerofaghtisdeterminedbythefollowingGenderRules:
1. IfSandWareofthesamesex,W(beingweaker)loses.
2. IfSandWareofdierentsexes,S getsconfusedand loses.
The party of a winning piece A replaces the loser's piece B by its own piece C. The choice of C is
restrictedbythetableofruleslistingallallowedtriples(ABC).Wewillseethatthisgamecannotbesolved
(i.e. consistent labeling computed) in a subexponential time. We rst prove that (see [Chandra, Kozen,
Stockmeyer1981])foranarticialgame. ThenwereducethisHalting GametoLinearChess.
ForExp-TimeCompletenessofregular(but nn)Chess,Go andCheckerssee: [Fraenkel,Lichtenstein
1981],[Robson1983,1984].
ExptimeCompleteHalting Game.
WeuseauniversalTuringMachine u(denedas 1-pointer cellularautomata) whichhaltsonlybyitshead
rollingoofthetape'sleftend. BoundedHaltingProblemBHP(x)determinesifu(x)stops(i.e.theleftmost
tapecellpointsleft)in2 jxj
steps. Thisrequires(2 jxj
)steps. Wenowconvertuinto theHaltingGame.
The playersare: L claiming u(x) halts in time (and should
havewinning strategy i this is true); His opponent S. The
board hasfour parts: thediagram,the inputx to u,positive
integersp(position)andt (timeintheexecutionofu(x)):
p t A t+1
x B
1 B
0 B
+1 t
p 1 p p+1
ThediagramshowsthestatesAofcellpattimet+1andB
s
;s2f0;1gofcellsp+s,attimet. A;B
include thepointerdirection;B maybereplacedby \?". Someboard congurationsare illegal: if(1) two
ofB
s
pointawayfrom eachother,or(2)AdiersfromtheresultprescribedbythetransitionrulesforB
s ,
or(3)t=1,while(B
s )6=x
p+s
. (Att=1,u(x)isjust starting,so itstapehastheinput xat theleft,the
headintheinitialstateattheendwithblanksleadingoto theright.) Here aretheGame Rules:
Thegame starts in the congurationshown below. L movesrst replacing the ?'s with symbols that
claimtoreectthestateofcellsp+satsteptofu(x). WhenS moves,he: choosess,copiesB
s
intoAand
llsallB with?'s; addssto pand1tot.
Start:
p=0 t=2 jxj
inputx ? ? ?
Lputs:
a t+1
b c d t
Sputs:
d t
? ? ? t 1
NotethatLmaylie(i.ellin\?" distortingtheactualcomputationofu(x)),aslongasheisconsistent
withtheabove\local"rules. AllS candoistocheckthetwoconsecutiveboard congurations. Hecannot
refertopastmovesortoactualcomputationofu(x)asanevidenceofL's violation.
Strategy: Ifu(x)doesindeedhaltwithin2 jxj
steps,thentheinitialcongurationistruetothecomputation
ofu(x). ThenL hasanobvious(though hardto compute) winning strategy: hejust tellsthe truth, what
actuallyhappensduring thecomputation. Hewill be alwaysconsistent; S willlose whent=1andcannot
decrease anymore. If theinitial congurationisfalse thenS canwin exploitingthat L must lie. IfL lies
once,S canforceLtolieallthewaydownto t=1. How?
Ifthe upperboxa of alegalconguration is falsethen thelowerboxesb;c;dcannot all be true,since
therulesofudetermineauniquelyfromthem. If S guessescorrectlywhich ofb,c,ordisfalseandbrings
itto thetoponhismove,then Lisforcedto keeponlying. Attime t=1allchipsaredown: thelyingof
Lis exposedsincethecongurationdoesn'tmatch theactualinputstringx,i.e. isillegal. Inotherwords,
Lcan't consistentlyfoolS allthetime: eventuallyheiscaught.
Solvingthis game (i.e. tellingwinning congurations from losing) amountsto answering whether the
initialcongurationiscorrect,i.e.whetheru(x)haltsin2 jxj
steps,whichrequires(jxj)steps. ThisHalting
3.3 Reductions; Non-Deterministic and Alternating TM; Time and Space.
Toreduce(seedenition insec.2.2)HaltinggametoLinearChessweintroduceafewconcepts.
Anon-deterministicTuringMachine(NTM)isaTMthatsometimesoersa(restricted)transitionchoice,
madebyadriver. A deterministic(ordinary)TMM acceptsastringx ifM(x)=yes;anon-deterministic
TM M doesifthereexistsadriverds.t.M
d
(x)=yes.
NTM represent singleplayergames, puzzles, e.g.Rubik's Cubewith a simpletransition rule. We can
computethewinning strategyinexponentialtime(exhaustingallpositions).
Home Work: Is there a P-time winning strategy foreverysuch game? Nobody knows. Alternatively,
show it requires exponentialtime. Grade A for thecourse and, probably, a seniorfaculty position at the
universityofyourchoice willbeawardedforasolution.
ThealternatingTM(ATM)isavariationoftheNTMdrivenbytwoalternatingplayerslandr. Astring
isacceptedifthereislsuchthatforanyr:M
l;r
(x)=yes. OurgamescouldbeviewedasATMusingasmall
spacebutuptoanexponentialtimeandreturningtheresultof thegame. It promptslandralternatingly
to choose theirmoves(in several stepsifthemoveis speciedbyseveral bits)and computestheresulting
position,untilawinneremerges. Acceptedstringsdescribewinningpositions.
Linear ChessSimulation ofTM-Game. WerstsimulateourHaltingGamebyL-Chess,avariantof
Linear Chess. It hasthe sameboard: Weak||||| |||||Shy and, likeregularchess,6 ranks. Unlike
LinearChess,whereonlythevanquishedpieceisreplaced,inL-chessthewinningpiecemayalsobereplaced
by(\promotedto")apiece ofthe sameside; thegender bit isset to thesidebit ofthe previousstep,and
anarbitrarytable ratherthanthesimple\GenderRule"determinesthewinningpiece.
ThesimulationisachievedsimplybyrepresentingtheHaltingGameasanATMcomputationsimulated
bytheuniversalTM(using\="commandsforplayers'input). TheUTMisviewedasanarrayof1-pointer
cellularautomata: Weakcellsasrightward,Shyleftward. TheTMheadisset tomoveupontermination to
theend ofthetape, sothat noloser piecesare left. TotransformL-Chess toLinearChessit isleft (asan
exercise)tomodifygenders,extendranks,andreplaceeachtransition byseveral,sothatthewinning piece
isdetermined bytheGenderRuleandis notreplaced. So, anyfastalgorithm tosolveLinearChess,could
beused tosolveanygame. SinceHaltingGamerequiresexponentialtime,sodoestheLinearChess.
A QuestionStillOpen: Space-Time Trade-o.
Deterministic linear space computations are games where any position has at most one (and easily
com-putable)legal move. Weknownogeneralnon-linearlowerbound orsubexponentialupperbound fortime
to determinetheiroutcome. This isthespace-time trade-o problem. You metwith such trade-os using
techniqueslikedynamicprogramming: savingtimeattheexpenseofspace.
Recallthat onaparallelmachine: timeisthenumberofstepsuntil
thelastprocessorhalts;spaceistheamountofmemoryused;volume
isthecombinednumberofstepsofallprocessors. \Small"willrefer
to values bounded by apolynomialof the input length; \large" to
exponential. Letuscallcomputationsnarrowifeithertimeorspace
are polynomial, and compact if both (and, thus, volume too) are.
Sec. 3.4 reduces thecomputations with largetime and smallspace
to(parallel)oneswithlargespaceandsmalltime, andviceversa.
-space
?
time large
time,
small
space
small time,largespace
narrowcomputations
Can every narrow computation be converted into a compact one? This is equivalent to the
existenceof aP-time algorithm forsolvinganyfast game, i.e. agame witha P-timetransition rule anda
counter decremented at each move,limitingthe number of movesto apolynomial. Thesec. 3.1 algorithm
canbeimplementedin parallelP-timeforsuchgames. Conversely,anynarrowlycomputablepredicatemay
beexpressedasonedeterminingthewinning sideofafastgame (similartotheHaltingGame). Thus,fast
games (i.e. compact alternating computations)correspondto narrowdeterministic computations; general
games(i.e. narrowalternatingcomputations)correspondto largedeterministicones.
ARelated Question: Doallexponentialvolumealgorithms (e.g.,onesolvingLinearChess)allowan
Basedon[Chandra,Stockmeyer1976],wenowreducethecomputationswithlargetimeandsmallspaceto
parallelones(PPMimplementedbysortingnetworks)withlargespaceandsmalltime,andvice versa.
Parallelization. SupposeProfessorhasinhisoÆceaprogram(foraSmallMachinewithlinearspaceand
exponentialtime)solvingthenextexamproblems. Youbreakintohis oÆceshortlybeforethetestandget
thetape. Yourtimeis verylimited,insuÆcientto run theprogram-but wait! -alsoin theoÆce yound
thepasswordthat gains you accessto areallyBig Machine, onewith essentiallyunlimitedspace resources
(exponential number of parallel processors, memory, etc.). How canyou, the devious student, exploit in
smalltimethisvast resourcetosolvetheexam?
GeneratesimultaneouslyallpossiblecongurationsoftheSmallMachinewithlinearmemoryspace,each
asaseparateprocess. TherewillbeM=n O(n)
ofthesecongurations/processes,astherearen O(n)
possible
graphswithnnodesandO(n)edges. Eachconguration/processxcomputesapointerx!x 0
,wherexand
x 0
aresuccessivecongurationsofProfessor'sSmallMachine.Now,eachprocessgetsacopyofitssuccessor's
successorpointer:x!x 0
!x 00
leadstox!x 00
. Next,thesinglesteppointersareerasedandtheprocedure
repeatsforthe4-steppointers,8-steppointers,etc.
IftheProfessor'sSmallMachinehalts, thenitcannotrepeatacongurationandmuststabilizein time
< M. So, our pointer-compressing procedure will take at most logM = O(nlogn) steps, which is only
P-time. Onceitiscomplete,weneedonlytaketheinputcongurationforthetest,anditwillhaveapointer
totheanswerconguration. Thevolume(timespace)ofcomputationisstillvast. Thereisnowayknown
toreduce volumetoapolynomial,but youseehowwecantradespacefortime.
Computing in Tight Space: a Pebble Game.
ConsidernowalargearrayM ofinteractingautomatarunningin parallelforaP-time.
Wewant tocompute in polynomial spaceits output, howeverlong
it takes. Assume M to be a xed connection network (which, we
know, can simulate any PPM). The directed, acyclicgraph at the
rightservesas aspace-time diagram of the operationof M: rows
representtimesteps. Eachnodestoresan(event),i.e. thestateofa
particularautomatonat aparticulartime. Eacheventisafunction
of itsparents i.e.the events oftheprevious stepin theautomaton
andits(O(1),say,3) neighbors. Here a
t ;b
t ;c
t andd
t
representthe
statesat timetofb anditsneighborswhich determineb
t+1 .
-space
?
time
(input) ...
.
.
.
.
.
.
... a
t
... b
t ... c
t ... d
t ...
... b
t+1
...
.
.
.
.
.
.
... (output) ...
Wewill computeM's output(in the central automatonO) assumingthe computationtime n (i.e. the
depthof thegraph) issmall. Eachnodecanbedescribedbyatriplet(i;t;s),whereiisthepositionofthe
automaton;sitsstate;t thetime. Thelengthofthistripletissmall: jsj=O(1);jtj=O(log (n)). Howlong
i's mayweneed? An automatoncanonly berelevantifit hastimeto propagateitsinformation to O, i.e.
isnlinksawayfromit. Thereareonly3 n
ofthose. So: jij=O(log (3 n
))=O(n).
Wethereforecanstoreeacheventinasmallspace. Butitdoesn'thelpifweneedtostorealargenumber
ofevents. ToknowhowmanyeventsweneedtostoreweconsiderthePebbleGamewiththefollowingrules:
Thegoalisto pebble(put apebblein)amarkednodeOofadigraph;onecanonlypebbleanodeifallits
parentsarepebbled; therearekpebbles,theycanberemovedandreused.
Note that input nodes haveno parentsand can alwaysbe pebbled. You canwin with k = O(dn) (d:
degreeofthegraph;n: itsdepth). Theproofisbyinduction. Supposeyoucanpebbleanynodeatlevelt0
with1+(d 1)tpebbles. Thenyoucanpebbleanynodeatlevelt+1with(d 1)t+d=1+(d 1)(t+1)
pebbles. Youjust pebbleeachofthenode'sparents,leaveapebblethere andreuse therestofthepebbles
for thenextparent. Finallyyouput apebble in thenode itself. However,thetime needed topebble this
graphmaybelarge(youmayhavetotraversealldescendingpaths).
Pebbling anodecorrespondsto computinganeventin ourdiagram. Eacheventcanbecomputedonce
itsparents'are. kisactuallythenumberofeventswemayhavetostoresimultaneously. Sincewecanpebble
4 Nondeterminism; Inverting Functions; Reductions.
4.1 Example of a Narrow Computation: Inverting a Function.
ConsideraP-timefunctionF. Forconvenience,assumejF(x)j=jxj,(oftenitisenoughifjxjandjF(x)jare
boundedbypolynomialsofeachother). InvertingF meansgiveny,ndx2F 1
(y),i.e.suchthatF(x)=y.
TheremaybemultiplesolutionsifF ismany-to-one;weneedtondonlyone. How?
Wemaytryall possiblex forF(x)=y. AssumeF runsin lineartime onaPointer Machine. Whatis
thecostofinvertingF? Thespaceusedisjxj+jyj+space
F
(x)=O(jxj). ButtimeisO(jxj2 jxj
): absolutely
infeasible. Nomethod is currentlyprovenmuch better in theworstcase. And neithercanwe provesome
invertingproblemstorequiresuper-lineartime. ThisisthesadpresentstateofComputerScience!
AnExample: Factoring. LetF(x
1 ;x
2 )=x
1 x
2
bethetheproductofintegersx
1 ;x
2 jx
1 j=jx
2
j. jF(x)j
isalmost jxj. Forsimplicity, assumex
1 ;x
2
areprimes. A fastalgorithminsec. 5.1determinesifaninteger
isprime. Ifnot,nofactorisgiven,onlyitsexistence. ToinvertF meansto factorF(x). Howmanyprimes
wemight haveto check? The densityof n-bitprime numbersisapproximately1=(nln2). So,factoringby
exhaustivesearch takesexponentialtime! Infact,eventhebestknownalgorithmsforthis ancientproblem
runin time about2 p
jyj
, despite centuriesof eorts by mostbrilliant people. The task is now commonly
believedinfeasibleandthesecurityof manyfamouscryptographicschemesdependsonthisunprovenfaith.
One-WayFunctions: x F
!yarethoseeasytocompute(x7!y)andhardtoinvert(y7!x)formostx.
EventheirexistenceissortofareligiousbeliefinComputerTheory. Itisunproven,thoughmanyfunctions
seem to beone-way. Some functions, however, are proven to be one-way, IFF one-wayfunctions EXIST.
Manytheoriesandapplicationsarebasedonthishypotheticalexistence.
Searchand NP Problems.
Letuscomparetheinvertingproblemswithanothertype: thesearchproblems. Theyare,givenx,tondw
satisfyingagivenpredicateP(x;w)computablein timejxj O(1)
. Anyinverting problemisasearchproblem
and anysearchproblem canberestatedasaninvertingproblem. E.g., ndingaHamiltoniancycle C ina
graphG, canbe statedasinvertingaf(G;C), whichoutputs G;0:::0ifC isin fact a Hamiltoniancycle
ofG. Otherwise,f(G;C)=0:::0. There aretwoparts toasearch problem,(a) decisionproblem: doesw
(calledwitness)exists,and(b)aconstructiveproblem: actuallyndw.
Atimebound forsolvingoneofthese typesofproblemsgivesasimilarboundfortheother.
SupposeaP-timeA(x)ndswsatisfyingP(x;w)(if wexists). IfAdoesnotproducewwithinthetime
limitthenitdoesnotexist. Sowecanusethe\witness"algorithmto solvethedecisionproblem.
Onthe other hand,an algorithm deciding ifthe witness exists can be used to nd it. Assume, given
x and apredicate P, A quickly determines ifthere is w satisfyingP(x;w). Considerapredicate P 0
(y;w)
statingthaty=x;z&P(x;zw). \Decision"answersforP 0
couldthenbeusedto ndwforP(x;w). First,
takez=1andchecktheexistenceofws.t. P 0
(y;w)fory=x;1. Ifitexists thenasolutionwfor P(x;w)
beginswith a1. Ifnot,wcouldonly startwith0. Thendo thesameextendingthefoundz with 1and so
on. Youwill ndwin jwjiterations. Unfortunately,formanyproblemssuch Aisnotknowntoexist.
Thelanguageofaproblem isthe setof allacceptableinputs. For theinverting problemit istherange
of f. For the search problem it is the set of all x s.t. P(x;w) holds for somew. An NP language is the
set of all inputs acceptable by a P-time non-deterministic Turing Machine (sec. 3.4). All three classes of
languages{search,inverseandNP{coincide. WhatNPmachineacceptsxifthesearchproblemwithinput
xandpredicateP issolvable? Thisisjust themachinewhichpromptsthedriverfordigitsofwandchecks
P(x;w). Conversely,which P correspondsto anon-deterministicTM M? P(x;w)just checksifM accepts
x,whenthedriverchoosesthestatesreectingthedigitsofw.
Interestingly,polynomialspaceboundeddeterministicandnon-deterministicTMshaveequivalentpower.
ItiseasytomodifyTMtohaveauniqueacceptingconguration. Anyacceptablestringwillbeacceptedin
times2 s
, wheresis thespacebound. Thenweneed tocheckA(x;w;s;k): whethertheTM canbedriven
fromthecongurationxtowintime<2 k
andspaces. Forthisweneedforeveryz,tocheckA(x;z;s;k 1)
andA(z;w;s;k 1), whichtakesspacet
k t
k 1
+jzj. So,t
k
=O(sk)=O(s 2
)[Savitch1970].
Wedeterminedthat inversion,search,andNP typesofproblemsareequivalent. Nobodyknowswhetherall
suchproblemsaresolvablein P-time(i.e.belongtoP).Thisquestion(calledP=?NP)isprobablythemost
famous one in TheoreticalComputer Science. All such problems aresolvablein exponentialtimebut it is
unknownwhether anybetteralgorithm generallyexists. FormanyproblemsthetaskofndinganeÆcient
algorithmmayseemhopeless,whilesimilarorslightlyrelaxedproblemscanbesolved. Examples:
1. Linear Programming: Givenanintegernm matrixA, ndarationalvectorx with Ax>0. Note
that ifAcontainsk-bitcoeÆcientsandx existsthenanxwithO(nk)bitnumbersalsoexists.
Solution: TheDantzig'sSimplex algorithm ndsx quicklyformostA. Some A, however,take
expo-nentialtime. Afterlongfrustratingeorts,aworstcaseP-timeEllipsoidAlgorithmwasnallyfound
in [YudinNemirovsky1976].
2. Primalitytest: Determinewhetheragivenintegerphasafactor?
Solution: Abad(exponentialtime)wayistotryall2 jpj
possibleintegerfactorsofp. Moresophisticated
algorithms, however,runfast(seesection5.1).
3. Graph Isomorphism: Problem: Given two graphsG
1 andG
2 , is G
1
isomorphicto G
2
? i.e. Canthe
verticesof G
1
bere-numberedsothat itbecomesequalG
2 ?
Solution: Checkingalln!enumerationsofverticesisnotpractical(forn=100,thisexceedsthenumber
ofparticlesintheknownUniverse). [Luks1980]foundanO(n d
)stepsalgorithmwheredisthedegree.
Thisis aP-timeford=O(1).
ManyotherproblemshavebeenbattledfordecadesorcenturiesandnoP-timesolutionhasbeenfound.
Evenmodicationsofthepreviousthreeexampleshavenoknownanswers:
1. Linear Programming: All knownsolutions producerational x. Noreasonablealgorithm is known to
ndintegerx.
2. Factoring: Given aninteger,nd afactor. Canbedonein aboutexponentialtime n p
n
. Seemsvery
hard: Centuriesofquestforfastalgorithmwereunsuccessful.
3. Sub-graph isomorphism: In a more general case where one graph may be isomorphic to a part of
anothergraph,noP-time solutionhasbeenfound.
Welearnedthe proofsthat Linear Chessand someother games haveexponentialcomplexity. Noneof
theaboveoranyother search/inversion/NPproblem, however,havebeenprovento requiresuper-P-time.
When,therefore,dowestoplookingforaneÆcientsolution?
NP-Completeness theoryisanattempttoanswerthisquestion. SeeresultsbyS.Cook,R.Karp,L.Levin,
andotherssurveyedin[Garey,Johnson][Trakhtenbrot]. AP-timefunctionf reducesoneNP-predicatep
1 (x)
top
2 (x)ip
1 (x)=p
2
(f(x)),forallx. p
2
isNP-completeifallNPproblemscanbereducedtoit. Thus,each
NP-completeproblem isas leastas worst casehardasallotherNP problems. This maybe agoodreason
to give upon fastalgorithms forit. AnyP-time algorithm for oneNP-completeproblem would yield one
forallotherNP(orinversion,orsearch)problems. Nosuchsolutionhasbeendiscoveredyetandthisisleft
asahomework(10yearsdeadline). WhatdowedowhenfacedwithanNP-completeproblem? Sometimes
onecanrestatetheproblem,ndasimilaronewhichiseasierbutstillgivestheinformationwereallywant,
4.3 An NP-Complete Problem: Tiling.
Example: NP-complete Tiling Problem. Invert thefunction which, givena
tiledsquare,outputsitsrstrowandthelistoftilesused. Atileisoneofthe26 4
possiblesquarescontainingaLatinletterat each corner. Twotilesmaybeplaced
nexttoeachother ifthelettersonthemutualsidearethesame. E.g.:
a x
m r
x c
r z
m r
n s
r z
s z
Wenow reduce any NP/search problem P to Tiling. Recall: A search problem is, given x, to nd w
which satises a P-time computableproperty P(x;w). Existence of w is an NP problem since w can be
\guessed"non-deterministicallyandveriedin P-time.
Padding Argument. First,weneedtoreduceittosome\standard"NPproblem. Anobviouscandidate
is the problem [Is there? w : U(v;w)], where U is the universal Turing Machine, simulating P(x;w) for
v =px. A diÆculty isthat U does notrunin P-time. Wemust restrict U to u which stops within some
P-time limit. Howto makethis xed degreelimitsuÆcient forsimulatinganypolynomial(even ofhigher
degree)timeP? LettheTMu(v;w)forv=00:::01pxsimulateaboutjvj 2
stepsofU(px;w)(and,thus,of
P(x;w)). Ifthepadding of0'sinv issuÆcientlylong,uwillhaveenoughtimetosimulateP,eventhough
u runs in quadratic time, while P's time limit may be, say, cube (of a shorter \un-padded" string). So
theNP problem P(x;w) isreducedto u(v;w)by mappinginstancesx into f(x)=0:::01px=v,with jvj
determinedbythetimelimitforP. NoticethatprogrampforP(x;w)isxed.
So,ifsomeNP problemcannot besolvedinP-time thenneithercanbetheu-problem. Equivalently,if
theproblem [isthere? w: u(v;w)]IS solvablein P-time then sois any searchproblem. Wedo notknow
whichofthesealternativesistrue. Itremainstoreduce thesearchproblemutoTiling.
The Reduction. We compute u(v;w) (where v = 00:::01px) by a TM represented as an array of
1-pointercellularautomatathatrunsforjvj 2
stepsandstopsifwdoesNOTsolvethepredicateP. Otherwise
itentersaninniteloop. Aninstancexhasasolutioniu(v;w)runsforeverforsomewandv=0:::01px.
Hereisthespace-timediagramofcomputationofu(v;w). Weset n
to u's time (andspace) jvj 2
. Eachrowin this table representsthe
conguration of u in a dierent moment of time. The solution w
is lledin at the secondstep below aspecialsymbol"?". Suppose
somebody llsin awrongtablethat doesn'treecttheactual
com-putation. Weclaimthat anywrongtablehasfour adjacentsquares
thatcouldn'tpossiblyappearinthecomputationofuonanyinput.
-space:n=jvj 2
?
time
v ?...? #...# (init.cong.)
v w T
1
.
.
.
.
.
.
.
.
.
T
n
Proof. As the inputv and theguessed solutionw are thesamein boththe right and thewrong tables,
therst2lines agree. Theactualcomputation startsonthethird line. Obviously,in therstmismatching
lineatransition ofsomecellfrom thepreviouslineiswrong. Thisis visiblefromthe statein bothlinesof
thiscellandthecellitpointsto,resultinginanimpossiblecombinationoffouradjacentsquares.
Foragivenx,theexistenceofw satisfyingP(x;w)is equivalentto theexistenceof
atablewith theprescribedrstrow,nohaltingstate,andpermissiblepatternsof each
fouradjacentsquares. Conversionofourtable totheTilingProblem:
Thesquaresinthetableareseparatedby\|";thetilesby\...";Breakeverysquarein
thetableinto4parts,eachpartrepresentsacornerof4separatetiles. Ifthe4adjacent
squaresinthetablearepermissible,then thesquareisalsotiledpermissibly.
u v
v x
So,anyP-time algorithmextendinga givenrstline to thewhole table of matching tilesfrom agiven
5.1 A Monte-Carlo Primality Tester.
Thefactoringproblemseemsveryhard. Buttotestanumberforhavingfactorsturnsouttobemucheasier
thanto ndthem. It alsohelps ifwesupply thecomputerwithacoin-ippingdevice. Wenowconsidera
MonteCarloalgorithm,i.e.onethatwithhighprobabilityrejectsanycompositenumber,butneveraprime.
See: [Rabin 1980],[Miller1976],[Solovay,Strassen1977].
Residue Arithmetic. pjx means p is a divisor of x. y = (x modp) denotes the residue of x when
divided by p, i.e. y 2 [0;p 1], pj(x y). x y (mod p) means pj(x y). Residues can be added,
multipliedandsubtractedwiththeresultputbackin therange[0;p 1](byaddinganappropriatemultiple
of p). E.g., x means p x for residues mod p. We usex to mean either x or x. If r and p haveno
common divisors > 1 (are mutually prime), division (x=rmodp) is possible, since x ! (r xmodp) is
one-to-oneon[0;p 1]. Operations+; ;;=obeyallusualarithmeticallaws. gcd(x;y)isthegreatest(and
divisiblebyanyother) commondivisorof x andy. Itcanbefound byEuclid'sAlgorithm: gcd(x;0) =x;
gcd(x;y) = gcd(y;(xmody)), for y > 0. By induction, g = gcd(x;y) = Ax B y, where integers
A=(g=xmody)andB=(g=ymodx)areproduced asabyproductofEuclid'sAlgorithm.
Wewill needto compute(x q
modp) in polynomialtime. Wecannot multiplyx q times, sinceit takes
q>2 jqj
steps. Insteadwecomputeallnumbersx
i =(x
2
i1
modp)=(x 2
i
modp);i<jqj. Thenwerepresent
qin binary,i.e.asasumofpowersof2andmultiply mod ptheneededx
i 's.
Fermat Test. TheLittleFermatTheorem foreveryx2[1;p 1]andprime psays: x (p1)
1 (modp).
Indeed, the sequence (ximodp) is a permutation of i = 1;:::;p 1. So, 1 ( Q i<p (xi))=(p 1)! x p 1 (modp).
Thistestrejectstypicalcompositep. Othercompositepcanbeactuallyfactoredbythefollowingtest:
Square Root Test.
Lemma: Foreachyandprime p,theequation(x 2
modp)=y hasatmostonepairofsolutionsx.
Proof: Let x;x 0
be twosolutions: y x 2
x 0
2
(modp). Then x 2
x 0
2
=(x x 0
)(x+x 0
) 0
(modp). We haveaproductof twointegerswhich iscongruentto 0, i.e.divisibleby p. Thereforepmust
divideat leastoneofthefactors. (Otherwisepiscomposite,andgcd(p;x+x 0
)actually givesusitsfactor).
So,eitherx x 0
0 (mod p)orx+x 0
0 (mod p)i.e.xx 0
(modp). End of proof.
Inparticularx 2
1 (mod p)impliesx1. NotethatthisdoesNOTholdifpiscomposite,sinceits
factorscanbespreadbetween(x x 0
)and(x+x 0
). Theny couldhavemorethanonepairofroots.
Rabin-Miller Test. Letus combinethese testsintoT(x;p) which usesgiven x toprovepis composite.
Letp 1=q2 k
,withoddq. T setsx
0 =(x
q
modp), x
i =(x
2
i 1
modp)=(x q2
i
modp),ik. Ifx
0 =1,
oroneofx
i
is 1,T givesup. Ifx
k
6=1,Fermat Test rejectsp. Otherwisethere isz=x
i
6=1, such that
(z 2
modp)=x
i+1
=1. ThentheSquareRoot Testfactorsp.
First, for each odd composite p, we show that T succeeds with some x, mutually prime with p. If
p=a j
;j>1,thenx=(1+p=a)isgoodforT :(1+p=a) p1
=1+(p=a)(p 1)+(p=a) 2
(p 1)(p 2)=2+:::
1 p=a 6= 1 (modp), since (p=a) 2
0 (modp). So the Fermat Test works. Otherwise p = ab,
gcd(a;b)=1. Takethe greatestiksuchthat x
i
6=1,for somex mutually primewithp. Suchi exists,
since( 1) q
1foroddq. Ifi<kthen(x
i )
2
1 (modp). Takex 0
=1+b(1=bmoda)(x 1). Check:
x 0
1x 0
i
(modb),whilex 0
i x
i
(mod a). Thus,either x
i orx
0
i
isnot1.
Now,T(y;p)succeedsonstepiwithmosty: functiony7!xyis1-1andT cannotfailwithbothy and
xy. This testcan be repeated formany randomlychosenx. Each timeT fails, weare twice moresure
that pisprime. Theprobabilitythat T fails300times onacompositepis<2 300
5.2 Randomized Algorithms and Random Inputs.
Las-Vegas algorithms, unlikeMonte-Carlo,nevergivewrong answers. Unluckycoin-ips just makethem
runlongerthanexpected. Quick-Sortisasimpleexample. Itis aboutas fastasdeterministic sorters,but
ispopulardue to itssimplicity. It sortsanarraya[1::n]of >2numbersbychoosing init arandompivot,
splittingtheremainingarrayintwobycomparingwiththepivot,andcallingitselfrecursivelyoneachhalf.
Foreasyreference,replacethearrayentrieswiththeirpositions1;:::;ninthesortedoutput(noeecton
thealgorithm). Denotet(i)the(random)timeiischosenasapivot. Theniwilleverbecomparedwithji
eithert(i) ort(j)isthesmallestamong t(i);:::;t(j). Thishas2outofjj ij+1chances. So,theexpected
numberofcomparisonsis P
i;j>i
2=j1+j ij=3 n+(n+1) P
n
k =3
2=k=2nlnn O(n):Note, thatthe
expectationof thesumofvariablesisthesumoftheirexpectations (nottrue,say,forproduct).
ThereisalsoaLas-Vegasprimalitytester,butitgoesfarbeyondthescopeofthesenotes.
The above Monte-Carlo and Las-Vegas algorithms require choosing strings at random with uniform
distribution. Wementallypicturethatasippingacoin. (Computersusepseudo-random generatorsrather
thancoinsinhope,rarelysupportedbyproofs,thattheiroutputshaveallthestatisticalpropertiesoftruly
randomcoinipsneededfortheanalysis ofthealgorithm.)
RandomInputs toDeterministicAlgorithmsareanalyzedsimilarlytoalgorithmswhichipcoins
them-selvesandthetwoshouldnotbeconfused. Consideranexample: Someoneisinterestedinknowingwhether
ornot certaingraphs containHamiltonian Cycles. He oersgraphsand pays$100 if weshoweither that
the graph has or that it has not Hamiltonian Cycles. Hamiltonian Cycle problem is NP-Complete, so it
shouldbeveryhardforsome,but notnecessarilyformostgraphs. Infact,ifourpatronchoosesthegraphs
uniformly, a fastalgorithm can earn us the$100 most of the time! Let all graphshaven nodes and, say,
k(n)<nlnn=4edgesandbeequallylikely. Thenwecanusethefollowing(deterministic)algorithm: output
\NoHamiltonianCycles"andcollectthe$100,ifthegraphhasanisolatednode. Otherwise,passonthat
graph and the money. Now, how often do we get our $100. The probability that a given node A of the
graphisisolated is(1 2=n) k
>(1 O(1=n))= p
n. Thus, the probabilitythat none of nnodes isisolated
(andweloseour$100)is O((1 1= p
n) n
)=O(e p
n
)andvanishesfast. Similarcalculationscanbemade
wheneverr=lim2k(n)=(nlnn)<1. Ifr>1,otherfastalgorithmscanactuallyndaHamiltonianCycle.
See: [Johnson1984], [Karp1976], [Gurevich 1985]. Seealso [Venkatesan, Levin] for a proof that another
problemisNP-completeevenonaverage. HowdothisHCalgorithmandtheaboveprimalitytestdier?
Theprimality algorithmworksforallinstances. Ittossesthecoinitself andcanrepeatit foramore
reliableanswer. TheHCalgorithmonlyworksformostinstances(withisolatednodes).
In the HC algorithm, we must trustthe opponent to follow thepresumed random procedure. If he
cheatsandproducesconnectedgraphsoften,ouranalysisbreaksdown.
Symmetry Breaking. Randomnesscomesinto ComputerScience in manyother waysbesidesthose we
considered. Hereisasimpleexample: avoidingconictsforsharedresources.
Severalphilosophersdineatacirculartable. Beforeeachofthemisaplate,andleftofeachplateiseither
aknifeoraforkarrangedsothateachdinerhasaknife onhisrightandaforkonhisleftorviceversa. The
problemisthatneighboringdinersmustsharetheutensils: neighborscannoteatatthesametime. Howcan
thephilosopherscomplete thedinnergiventhat allofthem mustact inthe samewaywithoutany central
organizer? Tryingto grabthe knivesand forksat once may turn theminto ghtingphilosophers. Instead
theycouldeachipacoin,andsitstillifitcomesupheads,otherwisetrytograbtheutensils. Iftwodiners
tryto grabthesameutensil,neithersucceeds. Iftheyrepeatthis procedureenoughtimes,mostlikelyeach
philosopherwilleventuallygetbothaknifeandforkwithoutinterference.
Theobviousdenition ofarandomsequenceisonethathasthesamepropertiesasasequenceofcoinips.
Butthisdenitionleavesthequestion,whataretheseproperties? Kolmogorovresolvedtheseproblemswith
anewdenition of randomsequences: those withno descriptionshorterthantheirfull length. Seesurvey
andhistoryin [Kolmogorov,Uspensky1987],[Li,Vitanyi1993].
KolmogorovComplexity K
A
(x=y)ofthestringxgivenyisthelengthoftheshortestpossibleprogram
pwhich letsalgorithm Atransformy intox: minf(jpj):A(p;y)=xg. There exists aUniversalAlgorithm
U suchthat,K
U
(x)<K
A
(x)+O(1), foreveryalgorithmA. ThisconstantO(1) isboundedbythelength
oftheprogramU needstosimulateA. WeabbreviateK
U
(x=y)asK(x=y)orK(x),foremptyy.
Anexample: ForA:x7!x,K
A
(x)=jxj,soK(x)<K
A
(x)+O(1)<jxj+O(1).
CanwecomputeK(x)? Onecouldtryallprogramsp;jpj<jxj+O(1)andndtheshortestonegenerating
x. This doesnotworkbecausesomeprogramsdiverge, and thehaltingproblem isunsolvable. Infact,no
algorithmcancomputeK orevenanyitslowerboundsexceptO(1).
Consider an old paradox expressed in the following phrase: \The smallest integer which cannot be
uniquelyandclearlydened byanEnglishphraseoflessthantwohundredcharacters." Thereare<128 200
Englishphrasesof<200characters. Sothere mustbeintegersthat cannotbeexpressed bysuchphrases.
Thenthereisthesmallestsuchinteger,butisn'titdescribedbytheabovephrase?
Asimilar argumentprovesthat K isnot computable. Supposean algorithm L(x) 6=O(1) computesa
lowerbound for K(x). Wecan useit to computef(n) that nds x with n < L(x) <K(x), but K(x)<
K
f
(x)+O(1) and K
f
(f(n))jnj, so n<K(f(n)) <jnj+O(1) =logO(n)n: acontradiction. So,K
anditsnon-constantlowerboundsarenotcomputable.
Aniceapplication ofKolmogorovComplexitymeasurestheMutualInformation:
I(x:y)=K(x)+K(y) K(x;y). Ithasmanyuseswhichwecannotconsiderhere.
Deciency ofRandomness.
Therearewaystoestimatecomplexitythatarecorrectin someimportantcases,butnotalways. Onesuch
caseiswhenastringxisgeneratedat random. Letusdened(x)=jxj K(x=jxj); (d(x)> O(1)).
Whatis theprobabilityof d(x)>i, forjxj=n? There are 2 n
stringsx of lengthn. If d(x)>i, then
K(x=jxj)<n i. Thereare<2 n i
programsofsuchlength,generating<2 n i
strings. So,theprobability
ofsuchstringsis<2 n i
=2 n
=2 i
(regardlessofn)! Soevenforn=1;000;000,theprobabilityofd(x)>300
isabsolutelynegligible(providedxwasindeedgeneratedbyfaircoinips). Wecalld(x)thedeciency of
randomnessofastringxwithrespecttotheuniformprobabilitydistributionsonallstringsofthatlength.
If d(x) is small then x satises all other reasonable properties of random strings. Indeed, consider a
property \x 62P"with enumerableS =:P of negligibleprobability. LetS
n
be the number of stringsof
lengthn, violating P and log(S
n )=s
n
. Whatcan be saidabout thecomplexity of allstringsin S? S is
enumerableandsparse (hasonlyS
n
strings). Togeneratex,usingthealgorithm enumeratingS,oneneeds
only the position i of x in the enumerationof S. We knowthat i S
n
2
n
and, thus, jij s
n n.
Thenthedeciencyofrandomnessd(x)>n s
n
islarge. Everyx whichviolatesP will, thus,alsoviolate
the\smalldeciencyofrandomness"requirement. Inparticular,thesmalldeciencyofrandomnessimplies
unpredictabilityofrandomstrings: Acompactalgorithmwith frequentpredictionwouldassurelarged(x).
Sadly,therandomnesscannotbedetected: wesaw,K anditslowerbounds arenotcomputable.
RecticationofDistributions. Werarelyhaveasourceofrandomnesswithpreciselyknowndistribution.
ButthereareveryeÆcientwaystoconvert\imperfect"randomsourcesintoperfectones. Assume,wehave
a\junk-random"sequence with weirdunknown distribution. Weonly knowthat its longenough(m bits)
segmentshavemin-entropy>k+i,i.e. probability<1=2 k +i
,givenallpreviousbits. (Withoutsuch mwe
would notknowa segment neededto extract even onenot fully predictable bit.) Wecan fold X into an
nmmatrix. Norelationisrequiredbetweenn;m;i;k,butusefulare smallm;i;kandhuge n=o(2 k
=i).
Wealsoneedasmallmi matrixZ, independentofX andreallyuniformlyrandom(orrandomToeplitz,
i.e. withrestrictionZ
a+1;b+1 =Z
a;b
). Thentheni product XZ hasuniformdistribution withaccuracy
p
