SortingAnalysis

Goal of Sorting

• _{Remove the most inversions with the least}

number of operations (comparisons / swaps)

Inversions

• _{How many places off an element is in an array} • _{Sorting is just removing all inversions from a list} • _{Can count inversions one element at a time}

Counting Inversions

• Unsorted List = 5 1 2 6 8 3 9 4 7

• _{Start at ‘5’}

– Is 5 > 1? Yes

– Is 5 > 2? Yes

– Is 5 > 6? No

– Is 5 > 8? No

– Is 5 > 3? Yes

– Is 5 > 9? No

– Is 5 > 4? Yes

– Is 5 > 7? No

• _{Whenever an element is greater than an element after it}

Counting Inversions

• Unsorted List = 5 1 2 6 8 3 9 4 7

• _{Continue counting inversions for each of the following}

elements

• Sum up the number of inversions for each element

Counting Inversions

Unsorted List = 5 1 2 6 8 3 9 4 7

5 Inversions 1 Inversions

5>1 1 x

5>2 1 1>2 0

5>6 0 1>6 0

5>8 0 1>8 0

5>3 1 1>3 0

5>9 0 1>9 0

5>4 1 1>4 0

5>7 0 1>7 0

Counting Inversions

Unsorted List = 5 1 2 6 8 3 9 4 7

2 Inversions 6 Inversion

x x

x x

2>6 0 x

2>8 0 6>8 0

2>3 0 6>3 1

Counting Inversions

Unsorted List = 5 1 2 6 8 3 9 4 7

8 Inversions 3 Inversions

x x

x x

x x

x x

8>3 1 x

8>9 0 3>9 0

8>4 1 3>4 0

8>7 1 3>7 0

Counting Inversions

Unsorted List = 5 1 2 6 8 3 9 4 7

9 Inversion 4 Inversions

x x

x x

x x

x x

x x

x x

Counting Inversions

Unsorted List = 5 1 2 6 8 3 9 4 7

• Total Inversions = sum of all individual element inversions

Inversion Questions

• _{What is the greatest number of inversions a list} can have?

• _{What is the average number of inversions a list} can have?

• _{How many inversions can be removed in one} operation

Sorting Analysis

• _{Selection Sort}

Selection Sort

• _{Brief reminder of selection sort}

• What is the worst case complexity of selection

sort? (O)

• _{What is the average case complexity of}

selection sort (Θ)

• What is the best case complexity of selection

Selection Sort (Worst Case)

• _{for(int i=0; i<unsortedArray.length-1; i++)}

– _{for(int j=i+1; j<unsortedArray.length; j++)} • _{Basic operation}

• In the worst case how many times will the

basic operation execute?

Selection Sort (Worst Case)

• _{Outer loop – executes n-1 times}

• Inner loop - executes for each outer loop

• Inner loop

– _{i=0 – n-1 times} – _{i=1 – n-2 times} – _{i=2 – n-3 times} – _…

Selection Sort (Worst Case)

• _{Inner loop}

– _{Average of (n-1) + (n-2) + …n-(n-1)} – _{Average = n/2 times}

– _{Ex: n=5}

• J=1  4 operations

• _{J=2  3 operations} • J=3  2 operations

Selection Sort (Worst Case)

• _{Outer loop = (n-1)}

– _{Inner loop = n/2}

• _{Multiply outer * inner}

• (n-1) * (n/2)

• (n*n – n) / 2

Selection Sort (Average Case)

• Do list permutations matter?

• Will the algorithm use a different number of

comparisons based on the order of the list?

• for(int i=0; i<unsortedArray.length-1; i++)

– _{for(int j=i+1; j<unsortedArray.length; j++)} • Basic operation

• No

Selection Sort (Best Case)

Insertion Sort

• _{Brief reminder of insertion sort}

• What is the worst case complexity of Insertion

sort? (O)

• _{What is the average case complexity of}

Insertion sort? (Θ)

• What is the best case complexity of Insertion

Insertion Sort (Worst Case)

• for(int i=1; i<unsortedArray.length; i++)

– _{for(int j=i-1; j>=0 && !done; j--)} • _{Basic operation}

• if(unsortedArray[i] > unsortedArray[j]) – done = true;

• In the worst case how many times will the

basic operation execute?

Insertion Sort (Worst Case)

• Outer loop – executes n-1 times

• Inner loop - executes for each outer loop until

the correct position is found

• Inner loop

– _{Assume worst case}

– _{Assume loop will never finish early}

– _{Inner loop will execute a varying number of times} • _{Same as selection sort}

Insertion Sort (Worst Case)

• _{Outer loop = (n-1)}

– _{Inner loop = n/2}

• _{Multiply outer * inner}

• (n-1) * (n/2)

• (n*n – n) / 2

Insertion Sort (Average Case)

• Do list permutations matter?

• for(int i=1; i<unsortedArray.length; i++)

– _{for(int j=i-1; j>=0 && !done; j--)} • Basic operation

• if(unsortedArray[i] > unsortedArray[j]) – done = true;

• Yes – may only have to do one comparison in

Insertion Sort (Average Case)

• Assume all list permutations are equally likely

• The inner loop worst case = all comparisons

– _{Average of all times run = n/2}

• The inner loop best case = one comparison

• What is the average number of times?

• _{On average the inner loops will only run half}

of the worst case time

Insertion Sort (Average Case)

• Outer loop = (n-1)

– _{Inner loop = n/4}

• _{Multiply outer * inner}

• (n-1) * (n/4)

• (n*n – n) / 4

Insertion Sort (Best Case)

• _{As mentioned in the average case}

• Least number of inner loop comparisons = 1

• Outer loop = n-1

• Inner loop = 1

Quicksort

• _{Brief reminder of Quicksort}

• What is the worst case complexity of

Quicksort? (O)

• _{What is the average case complexity of}

Quicksort? (Θ)

• What is the best case complexity of Quicksort?

Analysis of quicksort

• Assume all input elements are distinct.

• In practice, there are better

partitioning algorithms for when duplicate input elements may exist.

• Let T(n) = worst-case running time on

Worst-case of quicksort

• _{Input sorted or reverse sorted.}

• _{Partition around min or max element.}

• _{One side of partition always has no elements.}

Worst-case recursion tree

Worst-case recursion tree

cn

T(0) T(n–1)

Worst-case recursion tree

cn

T(0) c(n–1)

Worst-case recursion tree

T(n) = T(0) + T(n–1) + cn

cn

T(0) c(n–1)

Worst-case recursion tree

T(n) = T(0) + T(n–1) + cn

T(0) c(n–2)

T(0)

(1)

cn

T(0) c(n–1)

Worst-case recursion tree

T(n) = T(0) + T(n–1) + cn

T(0) c(n–2)

T(0)

(1)



cn

(1) c(n–1)

Worst-case recursion tree

T(n) = T(0) + T(n–1) + cn

(1) c(n–2)

(1)

(1)

T(n) = (n) + (n2)

= (n2)

h = n



Best-case analysis

If we’re lucky, PARTITION splits the array evenly:

T(n) = 2T(n/2) + (n) = (n lg n)

What if the split is always ₁₀1 :₁₀9 ?

   

)

( ₁₀9

10

1 n T n n

T n

Analysis of “almost-worst” case

Analysis of “almost-worst” case

cn

 

T ₁₀1 _T

 

Analysis of “almost-worst” case

 

T ₁₀₀1 T

 

9 T

 

9 T

 

Analysis of “almost-worst” case

(1)

(1)

… …

log_10/9n

…

O(n) leaves

O(n) leaves

log₁₀n

Analysis of “almost-worst” case

(1)

(1)

… …

log_10/9n

T(n) £ cn log n + O(n)

…

cn log n £

O(n) leaves

O(n) leaves

(n lg n)

Quicksort (Average Case)

• _nlog(n)

• _{Not going to prove this rigorously}

• _{Think of the good and bad splits in the} recursion tree

– _{Even a bad split gives the best case}

– Always having the worst split gives the worst case – On average the splits will be better than the

Merge Sort

• Brief reminder of Merge Sort

• What is the worst case complexity of Merge

Sort? (O)

• What is the average case complexity of Merge

Sort? (Θ)

• What is the best case complexity of Merge

Merge Sort (Worst Case)

• _{Always breaks the lists up into two equal}

(possibly 1-off) sub lists

• This creates a balanced binary recursion tree

• _{Height of tree = log(n)}

• Work done merging = size of list

Merge Sort (Worst Case)

• _{T(n) =}

– _{(# of sub problems) * (size of sub problems)} – _{+ work done dividing and merging}

• T(n) = 2(n/2) + n

– _{Technically T(n) = T(ceiling(n/2)) + T(floor(n/2)) + (n – 1)}

Merge Sort (Worst Case)

• _{T(n) = 2T(n/2) + n} • _{F(n) = n}

• Log_ba = log

22 = 1

• Compare n to n

• f(n) = n(log_ba)

– _{n = n}

• Case 2 T(n) = Θ(n^(log_ba) * log(n))

• T(n) = Θ(nlog(n))

Merge Sort (Average Case)

• _{No different than the worst case}

• Algorithm is the same regardless of the input

Merge Sort (Best Case)

• No different than the worst case

• Algorithm is the same regardless of the input

Decision Tree

• Represent Comparisons as a decision tree

• Each node represents a comparison

• Each number represents an element number

• All leaves represent all possible list

permutations

Decision Tree

List of Size 3

Decision Tree

List of Size 3

0:1

0:2 0:2

Decision Tree

List of Size 3

0:1

0:2 0:2

element[0] <= element[1] _{element[0] > element[1]}

1:2 1:2

e[0] <= e[2] e[0] > e[2] e[0] <= e[2] e[0] > e[2]

Decision Tree

List of Size 3

0:1

0:2 0:2

element[0] <= element[1] _{element[0] > element[1]}

1:2 1:2

e[0] <= e[2]

e[0] > e[2] e[0] <= e[2] e[0] > e[2]

e[2] < e[0] <= e[1] e[1] < e[0] <= e[2]

e[0] <= e[2] <=

e[1] e[0] <= e[2] < e[1] e[1] <= e[2] <= _e[0] e[2] < e[1] < e[0]

e[1] <=

e[2] e[1] > e[2] e[1] <= e[2]

Decision Tree - Example

List of Size 3

0:1

0:2 0:2

element[0] <= element[1] _{element[0] > element[1]}

1:2 1:2

e[0] <= e[2]

e[0] > e[2] e[0] <= e[2] e[0] > e[2]

e[2] < e[0] <= e[1] e[1] < e[0] <= e[2]

e[0] <= e[2] <=

e[1] e[0] <= e[2] < e[1] e[1] <= e[2] <= _e[0] e[2] < e[1] < e[0]

e[1] <=

e[2] e[1] > e[2] e[1] <= e[2]

Decision Tree - Example

List of Size 3

0:1

0:2 0:2

element[0] <= element[1] _{element[0] > element[1]}

1:2 1:2

e[0] <= e[2]

e[0] > e[2] e[0] <= e[2] e[0] > e[2]

e[2] < e[0] <= e[1] e[1] < e[0] <= e[2]

e[0] <= e[2] <=

e[1] e[0] <= e[2] < e[1] e[1] <= e[2] <= _e[0] e[2] < e[1] < e[0]

e[1] <=

e[2] e[1] > e[2] e[1] <= e[2]

Decision Tree - Example

List of Size 3

0:1

0:2 0:2

element[0] <= element[1] _{element[0] > element[1]}

1:2 1:2

e[0] <= e[2]

e[0] > e[2] e[0] <= e[2] e[0] > e[2]

e[2] < e[0] <= e[1] e[1] < e[0] <= e[2]

e[0] <= e[2] <=

e[1] e[0] <= e[2] < e[1] e[1] <= e[2] <= _e[0] e[2] < e[1] < e[0]

e[1] <=

e[2] e[1] > e[2] e[1] <= e[2]

e[1] > e[2]

Decision Tree - Example

List of Size 3

0:1

0:2 0:2

element[0] <= element[1] _{element[0] > element[1]}

1:2 1:2

e[0] <= e[2]

e[0] > e[2] e[0] <= e[2] e[0] > e[2]

e[2] < e[0] <= e[1] e[1] < e[0] <= e[2]

e[0] <= e[2] <=

e[1] e[0] <= e[2] < e[1] e[1] <= e[2] <= _e[0] e[2] < e[1] < e[0]

e[1] <=

e[2] e[1] > e[2] e[1] <= e[2]

e[1] > e[2]

Decision Tree - Example

List of Size 3

0:1

0:2 0:2

element[0] <= element[1] _{element[0] > element[1]}

1:2 1:2

e[0] <= e[2]

e[0] > e[2] e[0] <= e[2] e[0] > e[2]

e[2] < e[0] <= e[1] e[1] < e[0] <= e[2]

e[0] <= e[2] <=

e[1] e[0] <= e[2] < e[1] e[1] <= e[2] <= _e[0] e[2] < e[1] < e[0]

e[1] <=

e[2] e[1] > e[2] e[1] <= e[2]

e[1] > e[2]

6 > 2

Decision Tree - Example

List of Size 3

0:1

0:2 0:2

element[0] <= element[1] _{element[0] > element[1]}

1:2 1:2

e[0] <= e[2]

e[0] > e[2] e[0] <= e[2] e[0] > e[2]

e[2] < e[0] <= e[1] e[1] < e[0] <= e[2]

e[0] <= e[2] <=

e[1] e[0] <= e[2] < e[1] e[1] <= e[2] <= _e[0] e[2] < e[1] < e[0]

e[1] <=

e[2] e[1] > e[2] e[1] <= e[2]

e[1] > e[2]

6 > 2

Decision Tree - Example

List of Size 3

0:1

0:2 0:2

element[0] <= element[1] _{element[0] > element[1]}

1:2 1:2

e[0] <= e[2]

e[0] > e[2] e[0] <= e[2] e[0] > e[2]

e[2] < e[0] <= e[1] e[1] < e[0] <= e[2]

e[0] <= e[2] <=

e[1] e[0] <= e[2] < e[1] e[1] <= e[2] <= _e[0] e[2] < e[1] < e[0]

e[1] <=

e[2] e[1] > e[2] e[1] <= e[2]

e[1] > e[2]

6 > 2 6 > 3 2 <= 3

Proof Sorting

O

nlog(n)

– _{n! possible combinations}

• _{n! possible combinations}

– _{At least n! leaves in the decision tree}

• At height h a binary tree has < 2h leaves

• n! < 2h

• log(n!) < h

Counting Sort

• _{Lecture5_mit6046jf05_demaine_lec05_01.pdf}

Counting Sort Code

Counting Sort (Worst Case)

• _{Must loop through the array exactly once}

• Must loop through the secondary array exactly

once

• _{Must loop through the array again}

Counting Sort (Worst Case)

• _{Must loop through the array exactly once}

– _O(n)

• _{Must loop through the secondary array exactly}

once

– _{O(k), where k is the range of the values}

• _{Must loop through the array again}

– O(n)

• _{May have to copy the array back}

Counting Sort (Worst Case)

• _{O(n) + O(k) + O(n) = O(2n + k)}  O(n)

– _{Although if k is large, k will dominate n} • _{This is very possible}

• If need to copy and initialize new array

• O(n) + O(n) + O(k) + O(n) + O(n)

When to Use Counting Sort

• _{When values are integers}

• When range is known

Properties of Sorting

• _Stability

– _{Preserved order of equal elements} – _{Show by example}

• In place

– _{If the sort can be done with the input array and}

does not require extra space (auxiliary array)

• In place if it does not require extra space

Which sorts are in place?

• _{Selection sort}

• Insertion sort

• Quicksort

• Merge sort

Which sorts are in place?

• _{Selection sort} • Insertion sort

• Quicksort

– _{Log(n) space version exists (recursive calls)} – _{No additional array required}

– _{Technically not in place since it is not O(1)}

• _Mergesort

– _{In place version exists}

Stability Example

• _{StableExample.xlxs}

Importance of Stability

• _{Sorting by multiple columns}

• Important for database queries

– _{ORDER BY user_id ASC, order_id ASC} • Will get all users in order

• Will get all orders for each user in order

• _{No stability means the query will return results}

out of order

Which sorts are stable?

• _{Selection sort}

• Insertion sort

• Quicksort

• Merge sort

Which sorts are stable?

• _{Selection sort}

• Insertion sort

• Quicksort

• _Mergesort

Sorting

Algorithm Best Case

Ω Average CaseΘ Worst CaseO In-Place Stable Selection Sort n2 _n2 _n2 _{Yes No}

Insertion Sort n n2 _n2 _{Yes Yes}

Quicksort nlog(n) nlog(n) n2 _{No No}

Merge Sort nlog(n) nlog(n) nlog(n) No Yes

Counting Sort* n n n No Yes