Asymptotic Analysis

Asymptotic analysis [Greene and Knuth, 2007] is an approach to analysing algo- rithms to determine their theoretical space-time complexity. Space and Time Com- plexity are defined as follows:

Time Complexity How the completion time of an algorithm grows with the prob- lem size. That is, what is the relationship between the size of input to a

3

A greedy algorithm is an algorithm that follows the problem solving heuristic of making the locally optimal choice at each stage with the hope of finding a global optimum.

computation and the amount of time it will take that computation to run to completion;

Space Complexity How much memory an algorithm uses with respect to the size of its input data set.

In this section we will refer to time complexity but the general arguments and ap- plication applies to space complexity equally. When measuring complexity we generally measure:

Best Case (Lower Bound) The fastest possible time (smallest amount of mem- ory usage) taken for an algorithm to run to completion;

Worst Case (Upper Bound) The slowest possible time (largest amount of mem- ory usage) taken for an algorithm to run to completion;

Average Case The average time (memory usage) taken for an algorithm to run to completion. This is the most useful measure as the best and worst cases may only occur very infrequently. It may be, for example, that the best case time is very good but the likelihood of the circumstances causing the best case to occur are very low.

Asymptotic analysis provides measures for the best and worst cases. Other meth- ods must be used if we want to compute the average case.

Asymptotic analysis provides general measures that are independent of any specific implementation issues. For example, the measure should be independent of the speed of any actual processor as all processors run at different speeds and these speeds change from year to year. It achieves generality by ignoring constant differences in the estimates (constant differences in speed can be overcome by waiting for faster processors to arrive). Under this scheme a measure of 10n2† is considered to be equal to n2. That is, we ignore the constant multiplier which we assume to be insignificant. Here ‘equal’ is defined to mean that it belongs to the same complexity class. In computational complexity theory, a complexity class is a set of problems of related resource-based complexity. Once this principle is accepted it becomes clear that we can also ignore all lower order powers in an expression. For example for any constants a, b, c, d:

a × n3+ b × n2+ c × n + d

†

< a × n3+ b × n3+ c × n3+ d × n3 = (a + b + c + d) × n3 = n3(ignoring constants)

More formally, if two algorithms belong to two different complexity classes then there exists input sets of some size such that one of the algorithms will always be faster than the other regardless of the speed of the processor [Greene and Knuth, 2007]. The main complexity classes of interest to us, listed from best to worst are4:

C or 1 Constant time required, regardless of input set size;

log n Logarithmic time. Here log is assumed to be log₂. However, as different logarithmic bases differ only by constant amounts, the base is not important; n Linear. Algorithm size is directly proportionally to input set size;

n log n known simply as “nLog n”. Algorithms in this complexity class are viewed as acceptably fast;

nC _{Polynomially bounded. There are separate complexity classes for each value} of C. (n2is faster than n3is faster than n4et cetera);

Cn Exponential. Here C is some constant. These algorithms display unacceptable (exponential) rates of growth of time requirements as the data set increases. There are other complexity classes worse than exponential. All the algorithms worse than polynomially bound, known as non-polynomially bound are considered bad5.

Formal Definitions

The three main bounds measured by asymptotic analysis are formally defined as follows [Knuth, 1976]:

Upper bound or worst case analysis: O(−)6:

O(f (x)) = {g(x) | ∃ x0 > 0, c > 0 ∀ x > x0: g(x) < c × f (x)} (3.18)

4

A full list is available at: https://complexityzoo.uwaterloo.ca/Complexity_ Zoo

5

To paraphrase the rule from the αβ search algorithm if it is bad enough to belong to a non- polynomial bound complexity class then there is probably not much point spending any time working out just how bad it is.

Lower bound or best case analysis: Ω(−)

Ω(f (x)) = {g(x) | ∃ x0 > 0, c > 0 ∀ x > x0 : g(x) > c × f (x)} (3.19) Bound above and below: Θ(−)

Θ(f (x)) = {g(x) | ∃ x0 > 0, c1> 0, c2 > 0 ∀ x > x0 : c1×f (x) < g(x) > c2×f (x)} (3.20) As we can see O(−) provides an upper bound or worst case analysis, Ω(−) pro- vides a best case analysis and Θ(−) is both measures combined into one complex- ity class. Of these three measures Θ is the most useful as it combines both other measures in the same complexity class. Providing a Θ value is not always pos- sible as it requires proving that both the upper and lower bounds are in the same complexity class.

Issues

While asymptotic analysis has proven to be very useful in practice there are issues we need to beware of. First even if we produce best and worst case measures for an algorithm this does not tell us how likely the best or worst case scenarios are. For example, Quicksort is O(n2) [Hoare, 1961] but the expected completion time is actually n log n as the worst case scenarios giving n2are so unlikely.

Another issue is the ignoring of constants. If the constants being ignored are sufficiently large then they can still have a large impact on the algorithm comple- tion time. Sufficiently large is determined by the problem input set size we are dealing with so it varies from situation to situation. It is good practice to back up asymptotic analysis with some benchmarks that demonstrate how the algorithm translates on actual processor architectures.

We follow this procedure by providing Θ(−) measurements for our new algo- rithms in chapter five and then following this up with benchmarks in chapter six to indicate how well they operate on current processor architectures.