Lecture 00 Array Stack Queue

(1)

Subhash Suri UC Santa Barbara

Divide and Conquer

• A general paradigm for algorithm design; inspired

by emperors and colonizers.

• Three-step process:

1. Divide

the problem into smaller problems.

2. Conquer

by solving these problems.

3. Combine

these results together.

• Examples: Binary Search, Merge sort, Quicksort

(2)

Merge Sort: Illustration

2 5 4 6 1 3 2 6

6 3 2 1 6 5 4 2

1 2 2 3 4 5 6 6 5 2 4 6 1 3 2 6

5 2 4 6 1 3 2 6

5 2 4 6 1 3 2 6

(3)

Merge Sort

• Sort an

unordered

array of numbers

A

.

Merge-Sort

(A, p, q)

1. if

p

≥

q

return

A

;

2. r

=

b

(p

+

q)/2

c

3. Merge-Sort

(

A, p, r

)

4. Merge-Sort

(

A, r

+ 1

, q

)

5. MERGE

(A, p, q, r)

(4)

Merge sort:

• Divide: Split array of size n into two subarrays of size n/2

• _{Conquer: Two recursive calls on subarrays of size n/2}

• Combine: Merge two sorted subarrays to create sorted array of size n

Let T(n) = running time of merge sort on array of size n

T(n) = Tdivide(n) + Tconquer(n) + Tcombine(n)

T(n) = θ(1) + [T(n/2) + T(n/2)] + θ(n) T(n) = 2 T(n/2) + θ(n)

• Number of recursive subproblems = 2

• _{Size of each subproblem = n/2}

• _{Time for all the non-recursive steps =}θ_(n)

Later we’ll see how to solve these kinds of recurrence equations to determine T(n)

(5)

Binary Search

• Search for

x

in a

sorted

array

A

.

Binary-Search

(A, p, q, x)

1. if

p > q

return -1;

2. r

=

b

(p

+

q)/2

c

3. if

x

=

A

[

r

]

return

r

4. else if

x < A[r]

Binary-Search

(A, p, r, x)

5. else Binary-Search

(A, r

+ 1, q, x)

(6)

Binary search: Find(27)

0 1 2 3 4 5 6 7 8 9 10 13 18 24 27 34 41 49 55 60 66 72

27 < 41 0 1 2 3 4

13 18 24 27 34 27 > 24

3 4 27 34 found

Let T(n) = running time of binary search on array of size n T(n) = Tdivide(n) + Tconquer(n) + Tcombine(n)

T(n) = θ(1) + T(n/2) + 0 T(n) = T(n/2) + θ(1)

• _{Time for all the non-recursive steps =}θ₍₁₎

(7)

Multiplying Numbers

• We want to multiply two

n

-bit numbers. Cost is

number of elementary bit steps.

• Grade school method has

Θ(

n

2

)

cost.:

xxxxxxxx xxxxxxxx xxxxxxxxx xxxxxxxxx

xxxxxxxxx

xxxxxxxxxxxxxxxx ..

.

(8)

Polynomial multiplication and Large integer multiplication:

Given two polynomials:

P = 5x3_{+ 7x}2_{+ 6x + 2}_⇒_{[5, 7, 6, 2] = [p}₃_{, p}₂_{, p}₁_{, p}₀_]

Q = x3_{– 8x}2_{+ 9x – 1}_⇒_{[1, –8, 9, –1] = [q}₃_{, q}₂_{, q}₁_{, q}₀_]

P*Q = (5x3 + 7x2 + 6x + 2)(x3 – 8x2 + 9x – 1)

Alternatively, given two large integers:

P = 9863 = 9x3_{+ 8x}2_{+ 6x + 3 where x = 10}

Q = 7245 = 7x3_{+ 2x}2_{+ 4x + 5 where x = 10}

P*Q = (9863)(7245) = (9x3 + 8x2 + 6x + 3)(7x3 + 2x2 + 4x + 5)

where x = 10

(9)

Why Bother?

• Doesn’t hardware provide multiply? It is fast,

optimized, and free. So, why bother?

• True for numbers that fit in one computer word.

But what if numbers are very large.

• Cryptography (encryption, digital signatures)

uses big number “keys.” Typically 256 to 1024

bits long!

• n

2

multiplication too slow for such large numbers.

• Karatsuba’s (1962) divide-and-conquer scheme

(10)

Divide-and-conquer Algorithm 1 for Polynomial multiplication:

P = 5x3_{+ 7x}2_{+ 6x + 2 = (5x + 7)x}2_{+ (6x + 2) = Ax}2_{+ B}

Q = x3_{– 8x}2_{+ 9x – 1 = (x – 8)x}2_{+ (9x – 1) = Cx}2_{+ D}

A = 5x + 7 ⇒ [5, 7] B = 6x + 2 ⇒_{[6, 2]}

C = x – 8 ⇒_{[1, –8]}

D = 9x – 1 ⇒ [9, –1]

Let n = number of terms in P and Q P = Axn/2_{+ B}

Q = Cxn/2_{+ D}

A, B, C, D are polynomials with n/2 terms

P*Q = (Axn/2 + B)(Cxn/2 + D) = (AC)xn + (BC + AD)xn/2 + (BD)

Four recursive subproblems: A*C

B*C

A*D

B*D

(11)

Let T(n) = running time for the preceding Algorithm 1

T(n) = 4 T(n/2) + θ(n)

• _{Time for all the non-recursive steps =}θ_(n):

o adding n-term polynomials

o xn and xn/2 which are just shifts (not products) o also note the final product P*Q has < 2n terms

Again, later we’ll see how to solve these kinds of recurrence equations to determine T(n)

(12)

Karatsuba’s Algorithm

• Let

X

and

Y

be two

n

-bit numbers. Write

X

=

a b

Y

=

c d

• a, b, c, d

are

n/

2 bit numbers. (Assume

n

= 2

k

.)

XY

= (

a

2

n/2

+

b

)(

c

2

n/2

+

d

)

(13)

An Example

• X

= 4729

Y

= 1326

.

• a

= 47;

b

= 29

c

= 13;

d

= 26

.

• ac

= 47

∗

13 = 611

• ad

= 47

∗

26 = 1222

• bc

= 29

∗

13 = 377

• bd

= 29

∗

26 = 754

• XY

= 6110000 + 159900 + 754

(14)

Karatsuba’s Algorithm

• This is D&C: Solve 4 problems, each of size

n/2

;

then perform

O(n)

shifts to multiply the terms by

2

n

and

2

n/2

.

• We can write the recurrence as

T

(n) = 4T

(n/2) +

O

(n)

(15)

Divide-and-conquer Algorithm 2 for Polynomial multiplication

(Karatsuba’s algorithm)

:

As before,

P*Q = (Axn/2 + B)(Cxn/2 + D) = (AC)xn + (BC + AD)xn/2 + (BD)

This time we write (BC + AD) = (A+B)(C+D) – (AC) – (BD)

So only three recursive calls: A*C

B*D

(A+B)*(C+D)

Again stop the recursion when n = 1

Let T(n) = running time for the preceding Algorithm 2

T(n) = 3 T(n/2) + θ(n)

• Size of each subproblem = n/2

• _{Time for all the non-recursive steps =}θ_(n)

Again, later we will see how to solve these kinds of recurrence equations to determine T(n)

(16)

Majority element problem:

Given array A of size n, does there exist any value M that appears more than n/2 times in array A?

Majority element algorithm:

• Phase 1: Use divide-and-conquer to find candidate value M

• Phase 2: Check if M really is a majority element, θ(n) time, simple loop

Phase 1 details:

• Divide:

o Group the elements of A into n/2 pairs

o If n is odd, there is one unpaired element, x

 Check if this x is majority element of A

 If so, then return x, but otherwise discard x

o Compare each pair (y, z)

 If (y==z) keep y and discard z  If (y!=z) discard both y and z

o So we keep ≤ n/2 elements

• Conquer: One recursive call on subarray of size ≤ n/2

(17)

Example:

A = [7, 7, 5, 2, 5, 5, 4, 5, 5, 5, 7]

(7, 7) (5, 2) (5, 5) (4, 5) (5, 5) (7) A = [7, 5, 5]

(7, 5) (5) ⇒_{return 5 (candidate, also majority)}

Example:

A = [1, 2, 3, 1, 2, 3, 1, 2, 9, 9]

(1, 2) (3, 1) (2, 3) (1, 2) (9, 9)

A = [9] ⇒ return 9 (candidate, but not majority)

Let T(n) = running time of Phase 1 on array of size n

T(n) = T(n/2) + θ_(n)

• _{Size of each subproblem = n/2 [worst-case]} • _{Time for all the non-recursive steps =}θ_(n)

(18)

Recurrence Solving: Review

• T

(

n

) = 2

T

(

n/

2) +

cn

, with

T

(1) = 1.

• By term expansion.

T

(

n

) = 2

T

(

n/

2) +

cn

= 2

¡

2 T

(

n/

2

) +

cn/

2 ¢

+

cn

= 2

2

T

(

n/

2

) + 2

cn

= 2

2

¡

2 T

(

n/

2

3

) +

cn/

2

¢

+ 2

cn

= 2

3

T

(

n/

2

3

) + 3

cn

...

= 2

i

T

(

n/

2

i

) +

icn

• Set

i

= log

₂

n

. Use

T

(1) = 1.

(19)

The Tree View

• T

(

n

) = 2

T

(

n/

2) +

cn

, with

T

(1) = 1.

T(n/4) _T(n/4) cn/8 cn/8 cn/8 cn/4 cn/4 T(n/8) cn/2 cn/2 T(n/2) T(n/2) T(n) cn T(n/4)

cn/4 T(n/4)cn/4

cn/8 cn/8 cn/8 cn/8 cn/8 cn Total Cost cn

2(cn/2) = cn

4(cn/4) = cn

8(cn/8) =

• #

leaves

=

n

;

#

levels

= log

n

.

(20)

Solving By Induction

• Recurrence:

T

(

n

) = 2

T

(

n/

2) +

cn

.

• Base case:

T

(1) = 1

.

• Claim:

T

(

n

) =

cn

log

n

+

cn

.

T

(

n

) = 2

T

(

n/

2) +

cn

= 2 (

c

(

n/

2) log(

n/

2) +

cn/

2) +

cn

=

cn

(log

n

−

1 + 1) +

cn

(21)

More Examples

• T

(

n

) = 4

T

(

n/

2) +

cn

,

T

(1) = 1

.

Level Work

0

1

2

3

i

cn

4i cn /₂i

= 2 cni 4cn/2 = 2cn

16cn/4 = 4cn n/4

n/2

n

(22)

More Examples

Level Work

0

1

2

3

i

cn

4i cn /₂i

= 2 cni 4cn/2 = 2cn

16cn/4 = 4cn n/4

n/2

n

n/8

• Stops when

n/

2

i

= 1, and

i

= log

n

.

(23)

By Term Expansion

T

(

n

) = 4

T

(

n/

2) +

cn

= 4

2

T

(

n/

2

) + 2

cn

+

cn

= 4

3

T

(

n/

2

3

) + 2

2

cn

+ 2

cn

+

cn

...

= 4

i

T

(

n/

2

i

) +

cn

¡

2

i−1

+ 2

i−2

+

. . .

+ 2 + 1

¢

= 4

i

T

(

n/

2

i

) + 2

i

cn

• Terminates when

2

i

=

n

, or

i

= log

n

.

•

4

i

= 2

i

×

2

i

=

n

×

n

=

n

2

.

(24)

More Examples

T

(

n

) = 2

T

(

n/

4) +

√

n,

T

(1) = 1

.

T

(n) = 2T

(n/4) +

√

n

= 2

³

2T

(n/4

2

) +

p

n/4

´

+

√

n

= 2

2

T

(n/4

2

) + 2

√

n

= 2

2

³

2T

(n/4

3

) +

p

n/4

2

´

+ 2

√

n

= 2

3

T

(n/4

3

) + 3

√

n

...

(25)

More Examples

• Terminates when

4

i

=

n

, or when

i

= log

₄

n

=

log2 n

log₂ 4

=

12

log

n

.

T

(

n

) = 2

12 logn

+

√

n

log

4

n

(26)

Master Method

T

(n) =

aT

³

_n

b

´

+

f

(n)

n n/b n/b2 a children a a a a a a

a f(n/b )2 2

a f(n/b ) af(n/b)

i i

f(n)

(27)

Master Method

n

n/b

n/b2

a children

a

a a

a

a a

a f(n/b )2 2

a f(n/b ) af(n/b)

i i

f(n)

Total Cost

• #

children multiply by factor

a

at each level.

• Number of leaves is

a

logbn

=

n

logb a

. Verify by

(28)

Master Method

• By recursion tree, we get

T

(

n

) = Θ(

n

logb a

) +

log

_X

_b n−1

i=0

a

i

f

³

_n

b

i

´

• Let

f

(n) = Θ(n

p

log

k

n)

, where

p, k

≥

0 .

• Important:

a

≥

1 and

b >

1 are constants.

• Case I:

p <

log

_b

a.

n

logb a

grows faster than

f

(

n

).

(29)

Master Method

• By recursion tree, we get

T

(

n

) = Θ(

n

logb a

) +

log

_X

_b n−1

i=0

a

i

f

³

_n

b

i

´

• Let

f

(n) = Θ(n

p

log

k

n)

, where

p, k

≥

0 .

• Case II:

p

= log

_b

a.

Both terms have same growth rates.

(30)

Master Method

• By recursion tree, we get

T

(

n

) = Θ(

n

logb a

) +

log

_X

_b n−1

i=0

a

i

f

³

_n

b

i

´

• Let

f

(n) = Θ(n

p

log

k

n)

, where

p, k

≥

0 .

• Case III:

p >

log

_b

a.

n

logb a

is slower than

f

(

n

).

(31)

Applying Master Method

• Merge Sort:

T

(

n

) = 2

T

(

n/

2) + Θ(

n

).

a

=

b

= 2,

p

= 1, and

k

= 0. So

log

_b

a

= 1, and

p

= log

_b

a

. Case II applies, giving us

T

(n) = Θ(n

log

n)

• Binary Search:

T

(

n

) =

T

(

n/

2) + Θ(1).

a

= 1

, b

= 2

, p

= 0, and

k

= 0. So

log

_b

a

= 0, and

p

= log

_b

a

. Case II applies, giving us

(32)

Applying Master Method

• T

(

n

) = 2

T

(

n/

2) + Θ(

n

log

n

).

a

=

b

= 2,

p

= 1, and

k

= 1.

p

= 1 = log

_b

a

, and Case

II applies.

T

(n) = Θ(n

log

2

n)

• T

(

n

) = 7

T

(

n/

2) + Θ(

n

2

).

a

= 7

, b

= 2,

p

= 2, and

log

_b

2 = log 7

>

2. Case I

applied, and we get

(33)

Applying Master Method

• T

(

n

) = 4

T

(

n/

2) + Θ(

n

2

√

n

).

a

= 4

, b

= 2,

p

= 2

.

5, and

k

= 0. So

log

_b

a

= 2, and

p >

log

_b

a

. Case III applies, giving us

T

(n) = Θ(n

2

√

n)

• T

(

n

) = 2

T

(

n/

2) + Θ

³

n

logn

´

.

(34)

Matrix Multiplication

• Multiply two

n

×

n

matrices:

C

=

A

×

B

.

• Standard method:

C

_ij

=

P

n_k₌₁

A

_ik

×

B

_kj

.

• This takes

O(n)

time per element of

C

, for the

total cost of

O(n

3

)

to compute

C

.

• This method, known since Gauss’s time, seems

hard to improve.

• A very surprising discovery by Strassen (1969)

broke the

n

3

asymptotic barrier.

(35)

Divide and Conquer

• Let

A, B

be two

n

×

n

matrices. We want to

compute the

n

×

n

matrix

C

=

AB

.

A

=

µ

a

₁₁

a

₁₂

a

₂₁

a

₂₂

¶

B

=

µ

b

₁₁

b

₁₂

b

₂₁

b

₂₂

¶

C

=

µ

c

₁₁

c

₁₂

c

₂₁

c

₂₂

¶

(36)

Divide and Conquer

• The product matrix can be written as:

c

₁₁

=

a

₁₁

b

₁₁

+

a

₁₂

b

₂₁

c

₁₂

=

a

₁₁

b

₁₂

+

a

₁₂

b

₂₂

c

₂₁

=

a

₂₁

b

₁₁

+

a

₂₂

b

₂₁

c

₂₂

=

a

₂₁

b

₁₂

+

a

₂₂

b

₂₂

• Recurrence for this D&C algorithm is

T

(

n

) = 8

T

(

n/

2) +

O

(

n

2

).

(37)

Strassen’s Algorithm

• Strassen chose these submatrices to multiply:

P

₁

= (

a

₁₁

+

a

₂₂

)(

b

₁₁

+

b

₂₂

)

P

₂

= (

a

₂₁

+

a

₂₂

)

b

₁₁

P

₃

=

a

₁₁

(

b

₁₂

−

b

₂₂

)

P

₄

=

a

₂₂

(

b

₂₁

−

b

₁₁

)

P

₅

= (

a

₁₁

+

a

₁₂

)

b

₂₂

P

₆

= (

a

₂₁

−

a

₁₁

)(

b

₁₁

+

b

₁₂

)

(38)

Strassen’s Algorithm

• Then,

c

₁₁

=

P

₁

+

P

₄

−

P

₅

+

P

₇

c

₁₂

=

P

₃

+

P

₅

c

₂₁

=

P

₂

+

P

₄

c

₂₂

=

P

₁

+

P

₃

−

P

₂

+

P

₆

• Recurrence for this algorithm is

(39)

Strassen’s Algorithm

• The recurrence

T

(

n

) = 7

T

(

n/

2) +

O

(

n

2

)

.

solves to

T

(

n

) =

O

(

n

log2 7

) =

O

(

n

2.81

).

• Ever since other researchers have tried other

products to beat this bound.

• E.g. Victor Pan discovered a way to multiply two

70 ×

70 matrices using

143 ,

640 multiplications.

• Using more advanced methods, the current best

algorithm for multiplying two

n

×

n

matrices runs

(40)

Quick Sort Algorithm

• Simple, fast, widely used in practice.

• Can be done “in place;” no extra space.

• General Form:

1. Partition:

Divide into two subarrays,

L

and

R

;

elements in

L

are all

smaller

than those in

R

.

2. Recurse:

Sort

L

and

R

recursively.

3. Combine:

Append

R

to the end of

L

.

(41)

Partition

• Partition returns the index of the cell containing

the pivot in the reorganized array.

11 4 9 7 3 10 2 ₆ 13 21 8

(42)

Quick Sort Algorithm

• QuickSort

(

A, p, q

)

sorts the subarray

A

[

p

· · ·

q

]

.

• Initial call with

p

= 0

and

q

=

n

−

1. QuickSort(A, p, q

)

if

p

≥

q

then return

i

←

random(

p, q

)

(43)

Analysis of QuickSort

• Lucky Case:

Each Partition splits array in halves.

We get

T

(n) = 2T

(n/2) + Θ(n) = Θ(n

log

n)

.

• Unlucky Case:

Each partition gives unbalanced

split. We get

T

(

n

) =

T

(

n

−

1) + Θ(

n

) = Θ(

n

2

).

• In worst case, Quick Sort as bad as BubbleSort.

The worst-case occurs when the list is already

sorted, and the last element chosen as pivot.

• But, while BubbleSort

always

performs poorly on

certain inputs, because of

random pivot,

(44)

Analyzing QuickSort

• T

(

n

)

: runtime of randomized QuickSort.

• Assume all elements are distinct.

• Recurrence for

T

(n)

depends on two subproblem

sizes, which depend on random partition element.

• If pivot is

i

smallest

element, then exactly

(

i

−

1)

items in

L

and

(

n

−

i

)

in

R

. Call it an

i

-split.

• What’s the probability of

i

-split?

(45)

Solving the Recurrence

T

(

n

) =

n

X

i=1

1 n

(runtime with

i

-split) +

n

+ 1

=

1 n

n

X

i=1

(

T

(

i

−

1) +

T

(

n

−

i

)) +

n

+ 1

=

2 n

n

X

i=1

T

(

i

−

1) +

n

+ 1

=

2 n

n

_X

−1

i=0

(46)

Solving the Recurrence

• Multiply both sides by

n

. Subtract the same

formula for

n

−

1 .

nT

(

n

) = 2

n

_X

−1

i=0

T

(

i

) +

n

2

+

n

(

n

−

1)

T

(

n

−

1) = 2

n

_X

−2

i=0

(47)

Solving the Recurrence

nT

(

n

) = (

n

+ 1)

T

(

n

−

1) + 2

n

T

(

n

)

n

+ 1

=

T

(

n

−

1)

n

+

2 n

+ 1

=

T

(

n

−

2)

n

−

1 +

2 n

+

2 n

+ 1

...

=

T

(2)

3 +

n

X

i=3

2 i

= Θ(1) + 2 ln

n

(48)

Median Finding

• Median

of

n

items is the item with rank

n/

2 .

• Rank of an item is its position in the list if the

items were sorted in ascending order.

• Rank

i

item also called

ith statistic

.

• Example:

{16

,

5 ,

30 ,

8 ,

55}.

• Popular statistics are

quantiles

: items of rank

n/4, n/2,

3n/4

.

(49)

Median Finding

• After spending

O

(

n

log

n

)

time on sorting, any

rank can be found in

O

(

n

)

time.

(50)

Min and Max Finding

• We can find items of rank 1 or

n

in

O

(

n

)

time.

minimum

(A)

min

←

A

[0]

for

i

= 1

to

n

−

1 do

if

min

> A

[

i

]

then

min

←

A

[

i

];

return

min

• The algorithm

minimum

finds the smallest

(rank 1) item in

O

(

n

)

time.

(51)

Both Min and Max

• Find both min and max using

3 n/

2 comparisons.

MIN-MAX

(A)

if

|

A

|

= 1, then return

min = max =

A

[0]

Divide

A

into two equal subsets

A

₁

, A

₂

(min

₁

,

max

₁

) :=

MIN-MAX

(

A

₁

)

(min

₂

,

max

₂

) :=

MIN-MAX

(

A

₂

)

if

min

₁

≤

min

₂

then return

min = min

₁

else return

min = min

₂

(52)

Both Min and Max

• The recurrence for this algorithm is

T

(

n

) = 2

T

(

n/

2) + 2

.

(53)

Finding Item of Rank

k

• Direct extension of min/max finding to rank

k

item will take

Θ(

kn

)

time.

• In particular, finding the median will take

Ω(

n

2

)

time, which is worse than sorting.

• Median can be used as a perfect pivot for

(deterministic) quick sort.

• But only if found faster than sorting itself.

• We present a linear time algorithm for selecting

(54)

Linear Time Selection

SELECT

(k

)

1. Divide items into

b

n/

5 c

groups of 5 each.

2. Find the

median

of each group (using sorting).

3. Recursively find

median

of

b

n/5

c

group medians.

4. Partition using median-of-median as pivot.

5. Let low side have

s

, and high side have

n

−

s

items.

6. If

k

≤

s

, call

select

(

k

)

on low side; otherwise, call

(55)

Illustration

• Divide items into bn/5c groups of 5 items each.

• Find the median of each group (using sorting).

• Use SELECT to recursively find the median of the bn/5c group medians.

Group 1 Group 2 Group 3 Group 4 Group 5 Group 6

medians

x = median of

(56)

Illustration

• Partition the input by using this median-of-median as pivot.

• Suppose low side of the partition has s elements, and high side has n − s elements.

• If k ≤ s, recursively call SELECT(k) on low side; otherwise, recursively call SELECT(k − s) on high side.