Data Structures and Algorithms

(1)

Data Structures and Algorithms

Escola Politècnica Superior d’Alcoi

Computational Complexity

(2)

Introduction

Resources consumptions: spatial and temporal cost Costs characterization

Cost boundaries: best, worst and average cases

Asymptotic notation: computational costs hierarchy Additional aspects in computational complexity

Introduction

(4)

Algorithm analysis is a very important activity in the development process, especially in constrained resource environments (in time or memory).

Algorithm analysis is required in order to:

• Compare two or more different algorithms.

• Foresee the behavior of an algorithm in extreme conditions.

• Adjust the algorithm parameters to get the best results.

Introduction

(5)

Algorithm analysis can be carried out in two different ways:

• empirically (experimental, a posteriori), or

• theoretically (a priori)

Introduction Introduction

Not “exact” results Flexible, “cheap”,

machine independent theoretically

We need same quality in the implementations, real data, machine

dependency

“Easy”, “real” results empirically

Disadvantages Advantages

Type

(6)

Efficiency: capacity of solving the proposed problem using a low consumption of computational resources.

Two fundamental factors of efficiency:

• Spatial cost or quantity of memory required

• Temporal cost or time required to solve the problem

In order to solve a given problem, an algorithm or program A will be better than another B if A solves the problem in less time and/or uses less quantity of memory than B.

Generally there is a tradeoff between time and space, so a good algorithm is the one that solves the problem with a good commitment.

We will focus our attention mainly on temporal cost.

Resources

Resources consumption consumption : : spatial spatial and and temporal temporal costs costs

(7)

Sometimes, just time or memory are not only suitable in order to appreciate the quality of a program. Let’s consider three simple programs to calculate 10²:

A simple

A simple example example

Be the times required to carried out a product, sum, and successor.

(8)

Let’s assume that A1, A2, A3 are executed in four different computers with different characteristics (different times for successor, sum and product execution).

A simple

A simple example example

Which program is the best, A1, A2 or A3 ?

For each computer there’s a best one.

(9)

Our “simple problem” was too restrictive. In a normal situation we would like to calculate no only 10² but, in general, n², where n is a program data.

With this new point of view, the specific problem of calculating 10² is considered as an instance of the general problem of calculating n².

Given a problem, in general not all of its instances are of the same size. The size of an instance is an integer number that measures how big this instance is.

In our problem of calculating n², a natural measurement of the size is precisely n.

So, the size of the instance 10² is 10 and the size of the instance 2345678² is 2345678.

Problems, instances and size of an instance

(10)

We can easily rewrite A1, A2, A3 in order to calculate n² instead of 10².

Cost Cost as a function as a function of of the the size size of of the the problem problem

Size=n; temporal costs (μs):

Cost still depends on

(11)

Intuitively, it seems clear that A1 is the best program and A3 the worst. But, even fixing the computer characteristics, we will see that our intuition is not clearly kept.

For example, if we fix

Cost Cost as a as a function function of of the the size size is is still still not not suitable suitable

Which is the better, A1, A2, A3 ? For each instance there’s a best one.

A good cost characterization should allow to establish the program “quality”

independently of the computer and of the particular sizes of the instances to process.

(12)

In general, we should appreciate a relative behavior of A1, A2, A3 as follows:

Asymptotic cost Asymptotic cost

Calculus of n²: relative costs of A1, A2, A3

Temporal cost

Size

A good computational characterization of a program:

Cost functional dependency with the size of input – for large sizes !

(13)

• Generally, programs are useful to solve problems of large sizes of input (if they are small, we could solve them manually

• Considering large sizes of input, we can carry out simple approximations that simplify considerably cost analysis

• Programs no longer depend on specific execution time values of the

different elementary instructions used (if they don’t depend on the size of input), and don’t depend on sizes of input of specific instances of the

program to solve

Advantages in the asymptotic characterization Advantages in the asymptotic characterization

of the computational cost in function of the size of input

(14)

A STEP is the execution of a code segment which processing time doesn’t depend on the size of input of the considered program, or it is bounded by a constant.

Computational cost of a program: number of STEPS as a function of the size of input of the program

Elements to which 1 STEP can be assigned:

• Assignment, arithmetic or logical operations, comparison, access to a vector or matrix element, etc.

• Any finite sequence of the previous instructions which length doesn’t depend on the size of input.

Elements to which 1 STEP cannot be assigned, but a number of STEPS as a function of the size of input:

• Assignment of structured variables (ex: arrays) which number of elements depends on the size of input

• Loops (recursion) which number of iteration (calls) depends on the size of input

Simplification of cost analysis: the

Simplification of cost analysis: the “ “step step” ” concept concept

(15)

Cost analysis (STEPS) of n

²²

programs programs

(16)

Another example on cost analysis with STEPS Another example on cost analysis with STEPS

(printing the elements of an array in reverse order) (printing the elements of an array in reverse order)

STEP: Accesses to v; Size of input = n; T (n) = n+n=2n STEPS

(17)

More examples on cost analysis with STEPS More examples on cost analysis with STEPS

(searching the closest element to the average) (searching the closest element to the average)

STEP: Accesses to A; Size of input = n; T (n) = ??? STEPS

(18)

More examples on cost analysis with STEPS More examples on cost analysis with STEPS

(statistical mode for a series of ages: data input)

(19)

More examples on cost analysis with STEPS More examples on cost analysis with STEPS

(inefficient calculus of statistical mode) (inefficient calculus of statistical mode)

STEP: if comparisons; Size of input = n; T

_mod

(n) = ??? STEPS

(20)

More examples on cost analysis with STEPS More examples on cost analysis with STEPS

(more efficient calculus of statistical mode) (more efficient calculus of statistical mode)

STEP: if comparisons; Size of input = n; T

_mod

(n) = ??? STEPS

(21)

More examples on cost analysis with STEPS More examples on cost analysis with STEPS

( ( efficient efficient calculus of statistical mode) calculus of statistical mode)

STEP: if comparisons; Size of input = n; T

_mod

(n) = ??? STEPS

(22)

Costs for the different mode approaches Costs for the different mode approaches

Time (seconds)

Size of input (number of ages processed)

(23)

Costs for the different mode approaches Costs for the different mode approaches

(efficient

(efficient aproachesaproaches))

Time (seconds)

Size of input (number of ages processed)

(24)

More examples on cost analysis with STEPS More examples on cost analysis with STEPS

( ( weird weird functions) functions)

STEP: +=; Size of input = n; T

_main

(n) = ??? STEPS

(25)

Sometimes, cost is not (only) a function Sometimes, cost is not (only) a function

of the size of input of the size of input

In the examples viewed so far, all the “instances” of a problem had the same computational cost. But this is not always the case. In the following example, which is the cost of the function busca(n) ?

(26)

Sometimes, cost is not (only) a function Sometimes, cost is not (only) a function

of the size of input of the size of input

STEP: if comparison; size of input=n; T

_busca

(n) = ???

Depends on the array contents and the particular value to look for !

(27)

Cost extremes: best, worst and average cases Cost extremes: best, worst and average cases

Temproalcost

Size of input

(28)

Sometimes it is difficult to calculate the

Sometimes it is difficult to calculate the “ “exact exact” ” costs for a size of input. Example:

costs for a size of input. Example:

(29)

Sometimes it is difficult to calculate the

Sometimes it is difficult to calculate the “ “exact exact” ” costs for a size of input. Example:

costs for a size of input. Example:

STEP: accesses to X. The total number varies in different ways with the

instancies of the size of input depending if that is multiple or not of 3

(30)

Superior and inferior cost boundaries Superior and inferior cost boundaries

Generally, it is enough to determine the asymptotic boundaries, i.e. for large size of inputs.

Temproalcost

Size of input

(31)

Essential aspects in computational costs determination Essential aspects in computational costs determination

• Cost cannot be expressed with just one value. It has to be expressed as a cost function the size of input of the problem to solve.

• Comparison of cost functions has to be insensitive to “implementation

constants”, as particular execution times of individual operations (which costs is independent of the size of input)

• The independency from the constants is achieved by only considering the asymptotic behavior of the function cost (i.e., for large sizes of input)

• Frequently, for a given size of input, there are several instancies with different costs, therefore the cost cannot be properly expressed as a function of the size of input.

• In such cases, it is convenient to determine superior and inferior boundaries of the cost function, i.e., in the best and worst cases.

• There are even situations in which the boundaries for best and worst cases are complex functions. Generally, it will be enough to determine simple functions that bound superiorly and inferiorly the costs of all the instancies for large sizes of input.

(32)

Asymptotic notation Asymptotic notation

Abstraction of constants associated to STEP concept, notion of “large sizes of input” and

“asymptotic boundaries”

Be a function from natural numbers to the positive real. It is defined:

From the definition of we have:

(33)

Examples

(34)

Examples

(35)

Examples of costs in asymptotic notation Examples of costs in asymptotic notation

Mode: (inefficient) (efficient)

busca:

Worst case:

Best case:

Average case:

raro:

Worst case:

Best case:

Average case:

(36)

Asymptotic notation: some useful properties Asymptotic notation: some useful properties

Order relation among order functions:

Order of sum of functions: dominant function

(37)

Asymptotic notation: some useful properties Asymptotic notation: some useful properties

Limit Rule

For two arbitrary functions and

•

if then

• if then but and

•

if then but

For example, and . Which is the relative order ?

(38)

Asymptotic notation: some useful properties Asymptotic notation: some useful properties

Some sums of series

(39)

Asymptotic notation: some useful properties Asymptotic notation: some useful properties

Order of polynomial functions

(40)

Asymptotic notation: some useful properties Asymptotic notation: some useful properties

Order of exponential, logarithmic etc. functions

(41)

Asymptotic notation: computational costs hierarchy Asymptotic notation: computational costs hierarchy

Some usual order relations:

(42)

Asymptotic notation: computational costs hierarchy Asymptotic notation: computational costs hierarchy

sublinear, linear and superlinear

(43)

Asymptotic notation: computational costs hierarchy Asymptotic notation: computational costs hierarchy

superlinear, polynomial and exponential

(44)

Computational costs hierarchy: practical consequences Computational costs hierarchy: practical consequences

Maximum size of a problem, n, that can be solved by several algorithms and computers with different processing performance:

By increasing the calculus time and/or the computer performance, an algorithm ...

• logarithmic one increases enormously the size of input of tackled problems

• linear ones (or nlogn) achieves linear increments (or almost) of the tackled sizes

• quadratic (polynomial) ones achieve proportional moderate improvements

• exponential ones only achieve negligible additive improvements

(45)

Computational costs hierarchy: practical consequences Computational costs hierarchy: practical consequences

Execution times of a machine that executes 10⁹ steps by second (~ 1 GHz), as a function of the algorithm cost and the size of input n:

(46)

Other concepts of interest in computational complexity Other concepts of interest in computational complexity

Spatial Complexity

• Memory address concept: storage space where one or more data is located, which extension is independent of the size of input of the instancies of the

considered problem

• Spatial cost of a program or algorithm: number of memory addresses required for its execution

• All the concepts studied in temporal cost analysis are directly applicable to spatial cost

Data Structures and Algorithms