Recall that computability theory is concerned with questions such as whether a prob- lem is solvable at all, assuming one is given unlimited amounts of space and time. In contrast, complexity theory is concerned with questions such as whether a problem can be solved within certain limited computing resources, typically space or time. Whereas computability theory is concerned with unsolvable problems and the boundary between solvable and unsolvable problems, complexity theory analyzes the set of solvable prob- lems.
To address such questions, one must have a precise definition of space and time costs. Granted that, complexity theory asks questions such as:
• Which problems can be solved within a certain limit of time or space, and which cannot?
• Are there resource limits within which a known combinatorial problem definitely cannot be solved?
• Are there problems which inherently need more resources than others?
• What characteristics of problems cause the need for certain amounts of resources?
• What is the class of problems solvable within certain resource limits, and what are the basic properties of this class?
• Given a problem, what is the complexity of its best algorithm?
• Do best algorithms always exist?
• Does adding more resources allow one to solve more problems?
1.5.1
Polynomial time
Similarly to the situation in computability theory, one might fear that one single definition of resource accounting would not suffice, and in fact different models exist giving rise to different theories of complexity. Specifically, the class of problems solvable within certain sharp limits may vary from model to model.
However, we will see that many computation models define precisely the same class ptimeof problems decidable within time bounded by some polynomial function of the length of the input. Many researchers identify the class of computationally tractable
problems with those that lie inptime, thereby suggesting what could well be calledCook’s thesis, after Stephen C. Cook, a pathbreaking researcher in computational complexity:
1. All reasonable formalizations of the intuitive notion of tractable computability are equivalent (they can simulate each other within a polynomially bounded overhead in time);
2. Polynomial-time Turing machine computability is a reasonable formalization of tractable computability.
Note the close similarity with the Church-Turing thesis: “Turing machine computability” has been replaced by “polynomial time Turing machine computability,” and “effectively computable” by “tractable computability.” A stronger form of the first part is sometimes called theInvariance Thesis [15].
Cook’s thesis is a useful working assumption but should not be taken as being as solidly founded as the Church-Turing thesis, which concerns computability in a world of unlimited resources. Reasons for a certain skepticism about Cook’s thesis include the facts that an algorithm running in time |x|100, where |x| is the length of the input to the algorithm, can hardly be regarded as computationally tractable; and that there are algorithms (for instance as used in factorizing large integers) that run in a superpolyno- mial time bound such as|x|log log|x|, but with constant factors that are small enough for practical use.
1.5.2
Complexity hierarchies and complete problems
Ideally, one would like to be able to make statements such as “the XXX problem can be solved in time O(n3) (as a function of its input size); and it cannot be solved in time O(n3−ε) for anyε >0.” Alas, such definitive statements can only rarely be proven. There
are a few problems whose exact complexity can be identified, but very few.
Because of this, a major goal of complexity theory is classification of problems by difficulty. This naturally leads to a division of all problems into hierarchies of problem
classes. Standard classes of problems include: logspace, nlogspace, ptime, np-
time, pspace. Each class is characterized by certaincomputational resource bounds. For example, problems inlogspace can be solved with very little storage; those in ptime can be solved with unlimited storage, but only by algorithms running in polynomial time; and those innptimecan be solved by polynomial time algorithms with an extra feature: they are allowed to “guess” from time to time during their computations.
Various combinations of these resources lead to a widely encompassing “backbone” hierarchy:
Surprisingly, it is not known whether any one of the inclusions above is proper: for example, the question ptime = nptime?, often expressed as P = NP?, has been open for decades.
Nonetheless, this hierarchy has proven itself useful for classifying problems. A great many problems have been precisely localised in this hierarchy. A typical example is SAT, the problem of deciding whether a Boolean expression can be made true by assigning truth values to the variables appearing in it. This problem is complete for nptime, meaning the following. First, SAT is innptime: There is a nondeterministic algorithm that solves it and runs in polynomial time. Second, it is “hardest” among all problems in nptime: Ifit were the case that SAT could be solved by aptime algorithm, thenevery problem innptimewould have a deterministic polynomial time solution, andptime = nptimewould be true. This means that two stages of the hierarchy would “collapse.”
The last four chapters of this book concern complete problems for the various com- plexity classes.
1.6
Historical background
At the Paris Conference in 1900 D. Hilbert gave a lecture which was to have profound consequences for the development of Mathematics, particularly Mathematical Logic, and the not yet existing field of Computer Science. Hilbert’s ambitions were high and his belief in the power of mathematical methods was strong, as indicated by the following quote from his lecture:
Occasionally it happens that we seek the solution under insufficient hypotheses or in an incorrect sense, and for this reason do not succeed. The problem then arises: to show the impossibility of the solution under the given hypotheses, or in the sense contemplated. Such proofs of impossibility were effected by the ancients, for instance when they showed the ratio of the hypotenuse to the side of an isosceles triangle is irrational. In later mathematics, the question as to the impossibility of certain solutions plays a preeminent part, and we perceive in this way that old and difficult problems, such as the proof of the axiom of parallels, the squaring of the circle, or the solution of equations of the fifth degree by radicals have finally found fully satisfactory and rigorous solutions, although in another sense than originally intended. It is probably this important fact along with other philosophical reasons that gives rise to the conviction (which every mathematician shares, but which no one has as yet supported by a proof)that every definite mathematical problem
must necessarily be susceptible to an exact settlement, either in the form of an exact answer, or by proof of the impossibility of its solution and therewith the necessary failure of all attempts9.
At the conference Hilbert presented 23 unsolved mathematical problems. One of these, theEntscheidungsproblem (decision problem), was described as follows:10
The Entscheidungsproblem is solved if one knows a procedure which will permit one to decide, using a finite number of operations, on the validity, respectively the satisfiability of a given logical expression.
This problem was part of Hilbert’s program which included an endeavour to formalize number theory in a first-order deductive system. It was hoped that the provable theorems of the system would be precisely the true number-theoretic propositions, and that one could devise a procedure to decide whether or not a given proposition were a theorem of the system.
A negative answer to the Entscheidungsproblem, i.e., a proof that no such procedure exists, must necessarily be grounded in a precise definition of the notion of procedure. However, Hilbert and his school believed that such a universal decision procedure existed, and so had no reason to formalize the notion of a procedure in general terms.
In 1931 G¨odel showed his celebrated Incompleteness Theorem[54] stating, roughly, that for any consistent, sufficiently strong formalization of number theory, there are true propositions which cannot be proved in that formalization. To the experts this result made it seem highly unlikely that the Entscheidungsproblem could have a positive solution.
In 1936 it was shown independently by Church [22, 23] and Turing [170] that the Entscheidungsproblem does not have a positive solution. Further, and just as im- portant in the long run, each author gave a formalization of the notion of procedure (via λ-expressions and Turing machines, respectively), and derived the unsolvability of the Entscheidungsproblem from unsolvability of the Halting problem, which they both showed for their respective formalisms. Similar work on other formalizations, also in 1936, was done by Kleene [98] and Post [141]. Gandy [51] describes this astonishing “confluence of ideas in 1936.”
It is a remarkable fact that the different formalisms all define the same class of number- theoretic functions, the so-called partial recursive functions, and equivalences between
9Our italics; not present in the original.
various formalisms were soon proved by Kleene, Turing, and others. In fact, one can write compilers that turn a program in one formalism into a program in one of the other formalisms that computes the same function, supporting what we have previously called the Church-Turing thesis. It should be noted that this correspondence between the
algorithmsin the various formalisms is a stronger result than the fact that the various formalisms define the same class offunctions.
The initial work in complexity theory in the late 1920’s and early 1930’s was concerned with subclasses of the effectively computable functions, e.g., the primitive recursive func- tions studied by Hilbert [69], Ackermann [1], and others. Subclasses of primitive recursive functions were studied by Kalmar [92] and Grzegorczyk [58]. More programing language oriented versions of these classes were later introduced by Meyer and Ritchie [125].
With the appearance of actual physical computers in the 1950’s, an increasing interest emerged in the resource requirements for algorithms solving various problems, and the field of complexity as it is known today, began around 1960. One of the first to consider the question as to how difficult it is to compute some function was Rabin [145, 146]. Later, Blum [14] introduced a general theory of complexity independent of any specific model of computation.
The first systematic investigation of time and space hierachies is due to Hartmanis, Lewis, and Stearns [65, 64, 109] in the 1960’s, who coined the term “computational complexity” for what we call complexity theory in this book.
Important results concerning the classes of problems solvable in polynomial time and non-deterministic polynomial time were established by Cook [26] and Karp [95] who were among the first to realize the importance of these concepts.