by on 12/28/21. Re-use and distribution is strictly not permitted, except for Open Access articles. Chapter 1

(1)

Chapter 1 Algorithms

In the subject of computer science the word algorithm, in general, refers to a method that can be used by a computer for the solution of a problem. As such the word algorithm does not simply mean a process, technique or method. Strictly speaking, it is a finite collection of well- defined computational techniques each of which is clear and unambiguous, capable of being performed with finite effort and terminates in a finite amount of time. Before an algorithm comes to play, a problem to be solved on computer is first converted into a mathematical problem called mathematical model. An algorithm is different from a program (computer program). Although algorithm is an easily understood procedure giving a set of logical instructions for solving the problem at hand, a program is written in a computer language giving the instructions that the computer understands.

1.1. Some Sorting Algorithms

We begin our study of algorithms by considering some sorting algorithms.

Here we are given a sequence of non-negative integers a1, a2, . . . , an, not necessarily all distinct, the aim is to obtain a permutation of the a’s to get the numbers in a non-decreasing (or non-increasing) order i.e., to obtain a sequence a₁, a2, . . . , anwith{a1, a2, . . . , an} = {a1, a2, . . . , an} and a₁ ≤ a2≤ · · · ≤ an. We restrict ourselves to obtaining a non-decreasing sequence.

1 An Elementary Approach to Design and Analysis of Algorithms Downloaded from www.worldscientific.com by 134.122.89.123 on 12/28/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

(2)

1.1.1. Bubble sort

Among the sorting algorithms, this is most perhaps the easiest. In this algorithm a_n is first compared with a_n₋₁and if a_n < an−1, a_n and a_n₋₁ are interchanged. Then a_nis compared with a_n₋₂and if a_n< an−2, a_nand a_n₋₂are interchanged and the process stops when we come to an a_i such that a_i < an. The process is then followed with a_n₋₁, then a_n₋₂etc.

Algorithm 1.1. Bubble Sort Algorithm 1. For i = 1 to n − 1, do

2. for j = n to i + 1 3. if aj < aj−1, then

4. interchange (swap) aj ↔ aj−1

We will better explain this algorithm through two examples: one a sequence with all terms distinct and the other when all the terms are not necessarily distinct. We may not restrict ourselves to sequences of numbers alone but may consider sequences over any ordered set.

Example 1.1. Using bubble sort, rearrange the sequence Patiala, Ernakulam, Karnal, Amritsar, Venus, Shimla in alphabetical order.

Solution. This is a sequence of six terms all distinct. So i takes the values from 1 to 5. We express the sequence as a column denoting each word by the first letter of the word. We shall fully describe the internal loops for every i (refer to Table 1.1).

Observe that in the case i = 1, there is no change for j = 5, in the case of i = 2, there is no change for j = 6, 5, 4 and for i = 3, there is no change for j = 6, 5. At this stage the sequence is already sorted and, so, for i = 4 and 5, no changes are required. Hence the final sorted sequence is A, E, K, P, S, V or

Amritsar, Ernekulam, Karnal, Patiala, Shimla, Venus

Example 1.2. Using bubble sort, rearrange the sequence 2, 3, 6, 1, 0, 4, 8, 0 in non-decreasing order.

An Elementary Approach to Design and Analysis of Algorithms Downloaded from www.worldscientific.com by 134.122.89.123 on 12/28/21. Re-use and distribution is strictly not permitted, except for Open Access articles.

(3)

Table 1.1

For i = 1 For i= 2 For i = 3

P P P P A A A

E E E A P E E

K K A E E P K

A A K K K K P

V S S S S S S

S V V V V V V

−−−→j= 6 −−−→

j= 4 −−−→

j= 3 −−−→

j= 2 −−−→

j= 3 −−−→

j= 4

Table 1.2(a)

For i= 1 For i= 2

2 2 2 2 2 2 0 0 0 0

3 3 3 3 3 0 2 2 2 2

6 6 6 6 0 3 3 3 3 0

1 1 1 0 6 6 6 6 0 3

0 0 0 1 1 1 1 0 6 6

4 4 0 0 0 0 0 1 1 1

8 0 4 4 4 4 4 4 4 4

0 8 8 8 8 8 8 8 8 8

−−−→j= 8 −−−→

j= 7 −−−→

j= 5 −−−→

j= 4 −−−→

j= 3 −−−→

j= 2 −−−→

j= 6 −−−→

j= 5 −−−→

j= 4 −−−→

j= 3

Table 1.2(b)

For i= 3 For i= 4

0 0 0 0 0

2 2 2 1 1

3 3 1 2 2

6 1 3 3 3

1 6 6 6 4

4 4 4 4 6

8 8 8 8 8

−−−→j= 6 −−−→

j= 5 −−−→

j= 4 −−−→

j= 7

Solution. This is a sequence of eight terms. So i takes values from 1 to 7.

We express the sequence as a column and fully describe the internal loops for every i (as given in Tables 1.2(a) and 1.2(b)).

(4)

Observe that the array obtained in the last column of Table 1.2(b) is already sorted and, so, no changes are needed for i = 5, 6, 7. Also on the way, no changes were needed for i = 1 and j = 6, for i = 2 and j = 8, 7, for i = 3, j = 8, 7 and for i = 4, j = 8, 6, 5. Hence the sorted sequence is 0, 0, 1, 2, 3, 4, 6, 8.

1.1.2. Insertion sort

In this sorting method, we start with the index i = 2 and compare the keys a[1], a[2]. If a[2] < a[1], we interchange (swap) a[1] ↔ a[2]. Otherwise, leave these as they are. Here we are writing a[i ] for ai. Thus a1, a2 are in sorted order. Suppose that a1, a2, . . . , ai are in sorted order. Then, we consider ai+1. If ai+1 < ai, swap a[i ] ↔ a[i + 1]. Then compare the present a[i ] with a[i − 1] and continue the above process of swapping till we arrive at a position j such that the key value a[ j − 1] < a[ j]. Thus the terms a1, a2. . . , ai+1 are in sorted order. The process ends when we finally arrive at i = n and that we have put an in its proper place. Thus the algorithm may be stated as follows.

Algorithm 1.2. Insertion Sort Algorithm 1. For i = 2 to n do

2. move a[i] forward to the position j ≤ 2 in a step by step manner such that

3. a[i ] < a[k] for j ≤ k < i, and 4. either a[ j − 1] ≤ a[i] or j = 1 More fully, it may be stated as Algorithm 1.2.

1. For i = 2 to n do begin 2. for j = i to 2

3. while a[ j ]< a[ j − 1] (provided j ≥ 2 ) do begin 4. interchange a[ j ]↔ a[ j − 1]

5. end 6. end

(5)

We explain the procedure through the two examples considered for bubble sort.

Example 1.3. Sort the sequence of Example 1.1 using insertion sort.

Solution. The given sequence is P, E, K, A, V, S. We have P, E, K, A, V, Sⁱ→ E, P, K, A, V, S⁼² ⁱ→ E, K, P, A, V, S⁼³

i→ (E, K, A, P, V, S → E, A, K, P, V, S → A, E, K, P, V, S)=4 i→ A, E, K, P, V, S=5 ⁱ→ A, E, K, P, S, V.⁼⁶

In the step (or pass) i = 4, we have to move A to its proper place and it first moves past P, then past K and finally past E . These steps for the pass i = 4 have been placed in parenthesis.

Example 1.4. Sort the sequence of Example 1.2 using insertion sort.

Solution. The given sequence is sorted as follows (observe that no change is needed for the passes 1, 2, 3):

2, 3, 6, 1, 0, 4, 8, 0ⁱ→ (2, 3, 1, 6, 0, 4, 8, 0 → 2, 1, 3, 6, 0, 4, 8, 0⁼⁴

→ 1, 2, 3, 6, 0, 4, 8, 0)ⁱ→ (1, 2, 3, 0, 6, 4, 8, 0 → 1, 2, 0, 3, 6, 4, 8, 0⁼⁵

→ 1, 0, 2, 3, 6, 4, 8, 0 → 0, 1, 2, 3, 6, 4, 8, 0)ⁱ⁼⁶→ 0, 1, 2, 3, 4, 6, 8, 0

i=8

→ (0, 1, 2, 3, 4, 6, 0, 8 → 0, 1, 2, 3, 4, 0, 6, 8 → 0, 1, 2, 3, 0, 4, 6, 8

→ 0, 1, 2, 0, 3, 4, 6, 8 → 0, 1, 0, 2, 3, 4, 6, 8 → 0, 0, 1, 2, 3, 4, 6, 8).

Observe that no changes are needed for the passes i = 2, 3, 7. The finally sorted sequence is 0, 0, 1, 2, 3, 4, 6, 8 as before.

1.1.3. Selection sort

In the selection sort algorithm, we look for the smallest element (key) in a[1], a[2], . . . , a[n], say it is a[i]. We interchange a[1] ↔ a[i]. Then in the revised sequence, we look for the smallest element, which we again call a[i ] and this time we interchange (swap) a[2] ↔ a[i]. In general, if after j− 1 steps or passes we come to the sequence a[1], . . . , a[ j], . . . , a[n] in which the sequence a[1. . . j] (= (a[1], . . . , a[ j])) is already sorted, then

(6)

we look for the smallest element a[i ] in a[ j. . . n] and swap that a[i] with a[ j ]. We continue doing so until we arrive at sorted sequence a[1. . . n−1].

Then we look for the smaller of the two elements a[n− 1], a[n] and swap that with a[n− 1] finally getting the sorted sequence a[1], a[2], . . . , a[n].

In a formal manner, the algorithm may be stated as follows.

Algorithm 1.3. Selection Sort Algorithm 1. For i = 1 to n − 1 do

2. choose the smallest element a[ j ] in a[i. . . n]

3. swap a[i ]↔ a[ j]

4. call low key a[i ] 5. for j = i + 1 to n − 1 6. repeat steps 2, 3, 4 7. end

We now apply this sorting algorithm to sort the strings of Examples 1.1 and 1.2.

Example 1.5. Using selection sort algorithm, sort the sting of Example 1.1.

Solution. The given string is P, E, K, A, V, S which we may call a[1. . . 6]. In the first step, we look for the smallest element (key) for the whole sequence and find that it is a₄ = A. Swap a4 with a₁ to get the sequence A, E, K, P, V, S. Now we look for the smallest element (low key) in a[2. . . 6] which is a[2] = E. We need to swap a2 with a2 and, so, no change is needed and the sequence after two passes (steps) is again A, E, K, P, V, S. Looking for the low key in the sequence a[3 . . . 6], we find that a3 itself is the low key and swapping does not lead to a new sequence. Now looking for low key in the sequence a[4. . . 6], we find that a4 itself is the low key and swapping does not change this sequence.

Finally, observe that a6 = S < a5 = V, so swap a5 ↔ a6 and we get the sorted sequence A, E, K, P, S, V .

Symbolically, we may represent the above steps as in Table 1.3 and the final sorted sequence appears as the last column in the table.

(7)

Table 1.3

P A A A A A

E E E E E E

K K K K K K

A P P P P P

V V V V V S

S S S S S V

i−−→= 1 −−→

i= 2 −−→

i= 3 −−→

i= 4 −−→

i= 5

Example 1.6. Sort the sequence of Example 1.2 using the selection sort.

Solution. We indicate the steps symbolically as follows.

2, 3, 6, 1, 0, 4, 8, 0ⁱ→ 0, 3, 6, 1, 2, 4, 8, 0⁼¹ ⁱ→ 0, 0, 6, 1, 2, 4, 8, 3⁼²

i=3

→ 0, 0, 1, 6, 2, 4, 8, 3ⁱ→ 0, 0, 1, 2, 6, 4, 8, 3⁼⁴ ⁱ→ 0, 0, 1, 2, 3, 4, 8, 6,⁼⁵

i→ 0, 0, 1, 2, 3, 4, 8, 6=6 ⁱ→ 0, 0, 1, 2, 3, 4, 6, 8.⁼⁷

Hence, the final sorted sequence is 0, 0, 1, 2, 3, 4, 6, 8.

1.1.4. Quick sort

Algorithm 1.4. Quick Sort Algorithm 1. given a sequence a[1. . . n]

2. choose a pivot ( = some kind of a central element)v = a[m]

3. partition a[1. . . n] into two subsequences a[1 . . . p] and a[p + 1 . . . n]

such that a[i ] ≤ v for every i, 1 ≤ i ≤ p and a[ j] > v for every j, j = p + 1, . . . , n

4. quick sort the subsequences a[1. . . p], a[p + 1 . . . n] in place 5. end

Thus in quick sort there are two major steps. One is the choice of a pivot and, then, is the partitioning of the sequence around the pivot.

One of the earliest method of choosing a pivot is to look for the leftmost consecutive pair of unequal entries and to choose the larger of these two entries as a pivot. Suppose that the pivot chosen in this manner is a[m]. (There are two possibilities for the consecutive pair of entries

(8)

am−1, am being distinct. Either am−1 < am in which case am itself is the pivot or am < am−1in which case we swap am−1and amand rename am−1

as amand amas am−1so that am again is the pivot.) Then all the entries of the sequence a[i ] with i < m are either equal or a[1] = · · · = a[m−2] and a[m−1] < a[m −2]. Sorting of the sequence a[1 . . . m −1] is easy. Either a[1. . . m − 1] is already sorted or a[m − 1], a[1], . . . , a[m − 2] is sorted.

Thus the problem reduces to sorting the sequence a[m. . . n]. Hoar (cf. [5]) has given a simple procedure for partitioning a sequence a[m. . . n] around a[m]. It works as under:

Starting left, look for the first i such that a[i ] ≥ v and starting from right look for the first j such that a[ j ] < v. Interchange a[i] ↔ a[ j].

Continue this process till we find a sequence in which there is a k such that a[i ] ≤ v with m + 1 ≤ i ≤ k and a[ j] > v for all j ≥ k + 1. Finally interchange a[m]↔ a[k], if necessary. Thus

for i = m + 1, j = n

if a[i ]≤ v, a[ j] > v, set i = m + 2, j = n − 1 if a[i ]≤ v, a[ j] ≤ v, set i = m + 2, j = n if a[i ]≥ v, a[ j] > v, set i = m + 1, j = n − 1 Algorithm 1.5. Procedure Partition 1

Partition a[m. . . p] around a[m] = v

{partition a[m], . . . , a[p] into two sequences a(m + 1, . . . , k − 1), a(k + 1, . . . , p) with a[i] ≤ v for i = m + 1, . . . , k − 1, a[ j] > v for j = k+ 1, . . . , p, a[k] = v}

begin i = m

j = p a[m]= v

1. repeat i = i + 1until a[i] > v 2. repeat j = j − 1 until a[ j] ≤ v 3. interchange a[i ]↔ a[ j] if i < j 4. repeat 1, 2, 3 until i > j

5. return i interchange a[m]↔ a[i]

end

(9)

Algorithm 1.6. Procedure Partition 2

1. Partition a[m. . . p] so keys a[i] ≤ pivot are on the left and keys a[ j ]≥ pivot are on the right

2. k= pivot index, pivot key a[k] = v 3. begin

4. i = m, j = p 5. repeat

6. i = i + 1 until a[i] > v 7. j = j − 1 until a[ j] ≤ v 8. interchange a[i ]↔ a[ j]

9. return i 10. end (partition)

The two partition procedures are similar, the only difference being that in procedure 2, the pivot index is already decided and occupies its right position while in procedure 1, the partitioning key a[m] is shifted to its right position after all the keys≤ a[m] have been collected together on the left side and all the keys> a[m] are collected on the right side.

There is yet another partition procedure which is also variation of the above two procedures — it is more like procedure 1.

Let the pivot key = v be determined without occupying its rightful position in the sequence. This key is supposed to sit outside the sequence:

either on the left or on the right. For a change, suppose that it sits on the right. Start scanning the sequence from left to right. Keep all the beginning consecutive keys that are≤ v unchanged. Suppose we come to keys a[i ], a[i + 1] such that a[i] > v, a[i + 1] ≤ v. Then interchange a[i ]↔ a[i +1]. Change i = i +1. Compare a[i +1] and a[i +2], . . .. Keep all the keys a[i+1], . . . that are > v intact till we arrive at a situation where a[ j ]> v, a[ j + 1] ≤ v. Then interchange a[ j + 1] ↔ a[i + 1]. Continue the process till all keys≤ v have been collected together on the left side without changing their relative positions and all keys> v on the right side.

For the sake of simplicity, we take array a[1. . . n] to be sorted and pivot x . In case the pivot is not occurring either on the extreme right or on the extreme left but occurs somewhere in the array, the above partitioning procedure may be stated as follows.

(10)

Choose i such that a[ j ] < x for 1 ≤ j < i but a[i] > x. Choose j such that a[k] > x for i ≤ k < j but a[ j] < x. Group together a[i ], . . . , a[ j − 1] and consider it one block (we may think of it as a[i]).

Interchange a[i ], a[ j]. Continue the process till we arrive at an array which looks like

(i) a[k] < x, for 1 ≤ k < i, a[i] = x, a[k] < x, for i < k < j and a[k]> x for j ≤ k ≤ n or

(ii) a[k] < x, for 1 ≤ k < i, a[k] > x, for i ≤ k < j, a[ j] = x and a[k]> x for j < k ≤ n.

In case (i), regarding a[i+ 1], . . . , a[ j − 1] as one block, interchange a[i ], a[i + 1] so that the pivot x appears in the position a[ j]. In case (ii), regarding a[i ], . . . , a[ j − 1] as one block, interchange a[i], a[ j] so that the pivot x appears in the position a[i ].

We illustrate the above partition procedures through a couple of examples.

Example 1.7. Partition the array 65, 90, 75, 70, 45, 60, 85, 50, 80, 55 with pivot x = 70.

Solution. We use procedure partition 1 and the above modified partition leaving the details.

(i) The array swap

65, 90, 75, 70, 45, 60, 85, 50, 80, 55 2↔ 10 65, 55, 75, 70, 45, 60, 85, 50, 80, 90 3↔ 8 65, 55, 50, 70, 45, 60, 85, 75, 80, 90 4↔ 6 65, 55, 50, 60, 45, 70, 85, 75, 80, 90

(ii) The array swap(s)

65, [90], 75, (70), 45, 60, 85,50,80,55 65, [90, 75], (70), 45, 60, 85, 50, 80, 55

65, 45, (70), [90, 75], 60, 85, 50, 80, 55 2↔ 5 65, 45, (70), 60, [90, 75], 85, 50, 80, 55 4↔ 6 65, 45, (70), 60, [90, 75, 85], 50, 80, 55

65, 45, (70), 60, 50, [90, 75, 85], 80, 55 5↔ 8 65, 45, (70), 60, 50, [90, 75, 85, 80], 55

65, 45, (70), 60, 50, 55, [90, 75, 85, 80] 8↔ 10

65, 45, 60, 50, 55, (70), 90, 75, 85, 80 3↔ 4, 4 ↔ 5, 5 ↔ 6

(11)

We have shown the pivot within parentheses and the elements grouped together have been shown within the brackets [ ]. Observe that the total number of swaps involved is 5 (counting the last three swaps as just 1).

Example 1.8. Partition the array 65, 90, 75, 70, 45, 60, 85, 50, 80, 55 with pivot x = 75.

Solution. We apply procedure partition 1 and the modified partition procedure leaving the details.

(i) The array swap

65, 90, 75, 70, 45, 60, 85, 50, 80, 55

65, 55, 75, 70, 45, 60, 85, 50, 80, 90 2↔ 10 65, 55, 50, 70, 45, 60, 85, 75, 80, 90 3↔ 8 65, 55, 50, 70, 45, 60, 75, 85, 80, 90 7↔ 8 Observe that the number of swaps used is 3.

(ii) The array swap(s)

65, 90, (75), 70, 45, 60, 85, 50, 80, 55 65, [90], (75), 70, 45, 60, 85, 50, 80, 55

65, 70, (75), [90], 45, 60, 85, 50, 80, 55 2↔ 4 65, 70, (75), 45, [90], 60, 85, 50, 80, 55 4↔ 5 65, 70, (75), 45, 60, [90], 85, 50, 80, 55 5↔ 6 65, 70, (75), 45, 60, [90, 85], 50, 80, 55

65, 70, (75), 45, 60, 50, [90, 85], 80, 55 6↔ 8 65, 70, (75), 45, 60, 50, [90, 85, 80], 55

65, 70, (75), 45, 60, 50, 55, [90, 85, 80] 7↔ 10

65, 70, 45, 60, 50, 55, (75), 90, 85, 80 3↔ 4, 4 ↔ 5, 5 ↔ 6 The total number of swaps is 6 on counting the last three swaps as just one.

We now consider a couple of examples for Quicksort algorithm.

Example 1.9. Sort the sequence 2, 3, 6, 1, 0, 5, 4, 9, 8, 0 using quick sort.

Solution. Scanning the sequence from left to right we find that a[1] = 2, a[2] = 3 are distinct and a[2] > a[1]. So, the pivot is a[2] = 3.

(12)

Algorithm 1.7. Procedure Quicksort a[m. . . p]

{sort the elements a[m], . . . , a[p] in non-decreasing order}

1. find pivot

2. pivot index= k, pivot key = a[k]

3. partition a[m. . . p] with pivot index k and pivot key = a[k]

4. quick sort

a[m. . . k]

a[k+ 1 . . . p]

5. end

6. end;{quick sort}

We partition the sequence using procedure partition 2 with a[2] as its pivot.

2, 3, 6, 1, 0, 5, 4, 9, 8, 0 → 2, 0, 6, 1, 0, 5, 4, 9, 8, 3

→ 2, 0, 0, 1, 6, 5, 4, 9, 8, 3

Now partition 2, 0, 0, 1 with pivot v = 2 and 6, 5, 4, 9, 8, 3 with pivot v = 6. We then get

2, 0, 0, 1 → 1, 0, 0; 2 6, 5, 4, 9, 8, 3 → 3, 5, 4 9, 8, 6

Now partition 1, 0, 0 with pivot v = 1; 3, 5, 4, with pivot v = 5 and 9, 8, 6 with pivot v = 9. We then get the subsequences in order as 0, 0, 1; 2; 3, 4, 5; 6, 8, 9. Concatenating these subsequences we get the sorted form of the given sequence 0, 0, 1; 2; 3, 4, 5; 6, 8, 9.

Example 1.10. Sort the sequence of Example 1.9 using quick sort with procedure partition 1.

Solution. The given sequence is 2, 3, 6, 1, 0, 5, 4, 9, 8, 0. Partition it using Procedure 1 around 2. Then

2, 3, 6, 1, 0, 5, 4, 9, 8, 0 → 2, 0, 6, 1, 0, 5, 4, 9, 8, 3

→ 2, 0, 0, 1, 6, 5, 4, 9, 8, 3 → 1, 0, 0, 2, 6, 5, 4, 9, 8, 3

(13)

→ 0, 0, 1, 2, 6, 5, 4, 3, 8, 9 → 0, 0, 1, 2, 3, 5, 4, 6, 8, 9

→ 0, 0, 1, 2, 3, 4, 5, 6, 8, 9 → 0, 0, 1, 2, 3, 4, 5, 6, 8, 9

→ 0, 0, 1, 2, 3, 4, 5, 6, 8, 9

which is the final sorted form of the sequence.

Observe that on the way we have used procedure 1 to partition the sequences 1, 0, 0 around 1; 6, 5, 4, 9, 8, 3 around 6 and 3, 5, 4 around 5.

Example 1.11. Sort the sequence of Example 1.9 using quick sort with procedure partition 3.

Solution. The given sequence is 2, 3, 6, 1, 0, 5, 4, 9, 8, 0. Let us con- sider 4 as pivot. The given sequence may then be rewritten as 4|2, 3, 6, 1, 0, 5, 9, 8, 0.

The elements 2, 3 are less than 4 and 6 is greater than 4. So we get the initial blocks 4|2, 3|6, 1, 0, 5, 9, 8, 0.

Then swap block of 1, 0 with 6 to get 4|2, 3, 1, 0|6, 5, 9, 8, 0. Now swap 6 and 0 to get 4|2, 3, 1, 0, 0|5, 9, 8, 6. Now shifting the pivot to its rightful position, we get 2, 3, 1, 0, 0|4, 5, 9, 8, 6.

Use 2 as pivot for the first subsequence and 6 as pivot for the second subsequence, this sequence takes the form

2|3, 1, 0, 0||6|4, 5, 9, 8 → 2|1, 3, 0, 0||6|4, 5, 9, 8

→ 2|1, 0, 3, 0||4, 5, 6||9, 8 → 2|1, 0, 0, 3||4, 5, 6||8, 9 (using 9 as pivot for the last subsequence)

1, 0, 0||2, 3||4, 5, 6||8, 9 → 1|0, 0||2, 3||4, 5, 6||||8, 9

→ 0, 0, 1||2, 3||4, 5, 6||8, 9

Hence the sorted sequence is 0, 0, 1, 2, 3, 4, 5, 6, 8, 9.

1.2. Algorithm Performance/Algorithm Complexity

There are two types of complexity of an algorithm. One of these is the time complexity meaning the running time of an algorithm depending on the size of the data while the other is space complexity meaning the memory space an algorithm needs. It is important to study the complexity

(14)

of an algorithm. More than one algorithms are sometimes (in fact, in general) available for solving a given problem. For example, we have seen four different algorithms for sorting a given sequence of numbers (or elements of an ordered set). Which of the available algorithms should be preferred depends on their performance — the running time required by the algorithm and the memory space required. An algorithm which takes less time on machine and also requires less memory space is, in general, preferred over an algorithm that has more running time and or more memory space for a data of the same size. However, sometimes a simpler written algorithm may be preferred over an algorithm which is very complicatedly written. For all this, we have to be quite precise about what an algorithm is and we need to make clear how the input data and the time and the memory space needed are measured.

Time complexity of an algorithm A is a function f, where f(n) is the maximal number of steps or computations that the algorithm A needs to solve an instance of a problem having an input data of size n. The word “steps” may mean the number of arithmetic operations, comparisons of numbers, etc. It may not, sometimes, be possible to compute the complexity f(n) of an algorithm exactly. In such cases, we have to be content with how the function f(n) grows.

While talking of complexity of an algorithm, we mean time complexity unless stated otherwise.

As an illustration, we compute the complexity of the sorting algorithms considered in the earlier section.

(a) The inner loop of the bubble sort repeats (n − i) times and since i varies from 1 to n − 1, the number of computations involved is n−1

i=1(n − i) = n−1

i=1 i = ⁿ⁽ⁿ⁻¹⁾₂ . Each such computation involved consists of comparison of two numbers followed by swapping of the two numbers, if necessary. Each such computation requires a constant number of time units. Thus, the complexity or running time of the bubble sort is

≤ ^cn⁽ⁿ⁻¹⁾₂ < cn², where c is a constant.

Observe that in the inner loop j varies from n to i + 1, i.e., we start with an, compare it with an−1 and if an < an−1, we swap an with an−1. Then an is compared with an−2 and swap an ↔ an−2 if an < an−2 and we continue the process till we arrive at an i such that ai < an where the

(15)

process stops so far as an is concerned. Then we consider the number an−1

which may now occupy the nth place in the sequence. As the very name suggests, you may compare it with the movement of an air bubble stuck in water bottle.

(b) Insertion sort works precisely in the reverse order to the order in which the bubble sort works. The inner loop in this case moves j from i to 2 i.e., j takes i−1 values and then i takes values from 2 to n. In the internal loop, i − 1 comparisons and swaps are required and each comparison followed by a swap operation, if necessary, requires a constant units of time. The total number of steps is _n

i=2(i − 1) = _n₋₁

i=1i = ⁿ⁽ⁿ⁻¹⁾₂ . Therefore, the running time of the insertion sort is T(n) ≤ ^cn(n−1)₂ < cn², where c is a constant (which is the number of units of time needed for one move consisting of comparison of two numbers and a possible swap).

(c) Finding the smallest element in the sequence a[1. . . n] needs n −1 comparisons. Once this is done, one swap is needed. Once the smallest element is placed in the first position from left, we are left with a sequence of length n− 1. Finding the smallest element of this sequence needs n − 2 comparisons followed by one swap. Thus, for this sorting procedure, a total number of comparisons followed by possible swaps ≤ (n − 1) + (n − 2) + · · · + 1 = ⁿ⁽ⁿ⁻¹⁾₂ . Hence the running time of the selection sort

≤ ^cn⁽ⁿ⁻¹⁾₂ ≤ cn².

We find that the complexity of each of the three sorting algorithms is

≤ cn².

(d) Quick sort works quite differently. In this case, we first choose a pivot and then divide the sequence into two subsequences, say S1and S2, which are such that if v is the pivot, then every entry of S1 is< v while every entry of S2 is ≥ v with v sitting between S1 and S2. Once having done this, quick sort is applied to S1and S2. The process continues till the sequence is completely sorted.

The given sequence of elements is a[m. . . p] with p −m +1 = n. The pivot chosen a[k] is, say, the i th smallest entry in the sequence. The index i could take any one of the n values from 1 to n and every i is equally likely.

So, the probability of choosing i is _n¹. Partitioning of the sequence requires n− 1 comparisons and one swap, if necessary, for every comparison. Each step consisting of a comparison and a swap, if necessary, takes a constant

(16)

amount of time. Therefore, partitioning of the sequence takes at most cn time units, where c is a constant. Therefore, the running time T(n) for quick sort satisfies, for n ≥ 2,

T(n) ≤ cn +1 n

n i=1

{T (i − 1) + T (n − i)}, (1.1)

where T(i − 1), T (n −i) are respectively, the running time on using quick sort for the subsequences S1 and S2. It is clear that T(0) = T (1) = b — a constant. Observe that

n i=1

T(i − 1) = T (0) + T (1) + · · · + T (n − 1)

= T (n − 1) + T (n − 2) + · · · + T (1) + T (0)

=

n i=1

T(n − i).

Therefore (1.1) takes the form T(n) ≤ cn + 2

n

n−1

i=0

T(i) (1.2)

Using induction on n, we prove that T(n) ≤ kn log n, where k = 2b+ 2c. For n = 2,

T(2) ≤ 2c + T (0) + T (1) = 2(c + b) = k < k log 4 = k(2 log 2).

Now, suppose that for any i, 2 ≤ i < n, T (i) ≤ ki log i. Then, by (1.2) and by induction hypothesis,

T(n) ≤ cn + 2 n

n−1

i=0

T(i)

= cn + 4b n + 2

n

n−1

i=2

T(i) ≤ cn + 4b n +2k

n

n−1

i=2

i log i. (1.3) Let f(x) = x logex , where x is a positive real variable. If x > x, then log x ≥ log x and therefore, xlog x > x log x. Thus f (x) is a strictly

(17)

y = f(x)

Fig. 1.1

increasing function of x . Therefore f(x) is Reiman integrable. For any real numbers a, b, 2 ≤ a < b,_b

a f(x)dx equals the area AabB bounded by the lines a A, bB parallel to y-axis, the x-axis and the curve y = f (x) (Fig. 1.1). Also _n−1

i=2 f(i) is the sum of the line segments (i, f (i)) all parallel to the y-axis and lying within the area Aab B with a = 2 and b = n − 1. Therefore,

n−1

i=2

i log i ≤

_n₋₁

2

x log_ex d x ≤

_n

2

x log_ex d x

= x²

2 log_ex | −

_n

2

x² 2 .1

xd x

= n log n

2 − 2 log_e2−

n² 4 − 1

.

(1.4) Observe that ^x₂² log_ex | stands for evaluating ^x₂²log_ex for x = n and x = 2 and taking their difference. From (1.3) and (1.4), we get

T(n) ≤ cn +4b

n + kn log n −kn

2 + (1 − 2 log 2)2k n .

As k = 2b + 2c and n ≥ 2, it follows that cn + ^4b_n − ^kn₂ + ^2k_n −

4k log 2

n ≤ 0. Hence T (n) ≤ kn log_en.

(18)

1.3. Heap Sort

We finally discuss another sorting algorithm called heap sort.

Recall that a tree is called a binary tree if every node has at most two children. Suppose that an array a[1. . . n] is represented as a binary tree T with a[1] as its root and a[i ] going from left to right i.e., a[1] has a[2] as its left child and a[3] as its right child, then a[2] has a[4] as its left child and a[5] as its right child, a[3] has a[6] as its left child and a[7] as its right child and so on. In general for any node a[i ] its left child is a[2i ], and its right child is a[2i + 1] (if one exists).

The binary tree T representing an array a[1. . . n] is called a max heap if for every i ≥ 1, a[i] ≥ a[2i] and a[i] ≥ a[2i + 1]. It is called a min heap if for every i ≥ 1, a[i] ≤ a[2i] and a[i] ≤ a[2i + 1]. Representing an array as a max heap or a min heap is called build heap. Also by a heap we mean a data represented as a binary max heap or a binary min heap.

Henceforth, by a heap we shall mean a max heap unless explicitly mentioned otherwise.

Given any array a[1. . . n] we can always build a heap for it (or heapify it). Once this is done, the element at the root is the largest element of the array. We swap the root element with the last element of the heap and then consider the last element as detached from the data. The remaining n− 1 elements may not be a heap. We build a heap for the remaining n − 1 elements. Swap the element at the root with the last element of the heap formed and, again, considered as detached so that it comes just before the last element detached. Continue the process till we arrive at the data in sorted order.

Formally we have the following heap sort algorithm.

Algorithm 1.8. Procedure Heap Sort 1. given an array a[1. . . n] of length n

2. build heap from a[1. . . n] (again denoted by a[1 . . . n]) 3. for i = n to 2

4. swap a[i ]↔ a[1]

5. detach a[i ]

6. repeat steps 2 to 5 till we arrive at two elements a[1], a[2] (and the elements detached in proper sorted form)

7. end

(19)

Observe that build max heap a[1. . . n] or heapify a[1 . . . n] has to be done n − 1 times. Therefore, the complexity of the heap sort is ≤ c (n× complexity of build heap a[1 . . . n]), where c is a constant. Thus, in order to apply heap sort, we need to give procedure for build max heap a[1. . . n] and also find its complexity.

Given a data a[1. . . 3] of length 3 arranged in the form of a binary tree with root a[1]. If a[1]≥ max{a[2], a[3]}, the data is already in max heap form. Otherwise swap a[1]↔ max{a[2], a[3]} and the data is in max heap form.

Observe that an array a[1. . . n] arranged in the form of a binary tree, the node a[i ] has a[2i ], a[2i + 1] as its children while the node a[i] has a _i

2

as its parent/ancestor (here _i

2

denotes the largest integer≤ ₂ⁱ).

The height of the binary tree is log n. To build a heap out of the given data, we have to compare every node a[i ] with its parent a _i

2

and swap a[i ]↔ a _i

2

if a[i ]> a _i

2

.

Thus, to build a heap, every node needs at most log n comparisons and necessary swaps, if any. There being a total of n nodes, the running time of build heap procedure is≤ c(n log n). Also, for the process of build heap we start with a leaf of the data tree with largest entry. The process build heap can be performed on line (without drawing actual binary tree representing the data array). We explain this process through some examples.

Example 1.12. Build a heap for the array 6, 5, 3, 4, 1, 2, 9, 8, 7.

Solution.

Data swap

a[1] a[2] a[3] a[4] a[5] a[6] a[7] a[8] a[9] a[i ]↔ a _i

2

6 5 3 4 1 2 9 8 7 4, 7

6 5 3 7 1 2 9 8 4 5, 7

6 7 3 5 1 2 9 8 4 6, 7

7 6 3 5 1 2 9 8 4 5, 8

7 6 3 8 1 2 9 5 4 6, 8

7 8 3 6 1 2 9 5 4 7, 8

8 7 3 6 1 2 9 5 4 3, 9

8 7 9 6 1 2 3 5 4 8, 9

9 7 8 6 1 2 3 5 4

(20)

As a _i

2

> a[i] for every i, no further swaps are needed and we have got a heap.

Example 1.13. Build a heap for the array 4, 7, 8, 6, 1, 2, 3, 5.

Solution.

Data swap

a[1] a[2] a[3] a[4] a[5] a[6] a[7] a[8] a[i ]↔ a _i

2

4 7 8 6 1 2 3 5 4, 8

8 7 4 6 1 2 3 5

As a _i

2

> a[i] for every i, no further swaps are needed and we get

a heap: 8 7 4 6 1 2 3 5

Example 1.14. Build a heap for the array 5, 7, 4, 6, 1, 2, 3.

Solution.

Data swap

a[1] a[2] a[3] a[4] a[5] a[6] a[7] a[i ]↔ a _i

2

5 7 4 6 1 2 3 5, 7

7 5 4 6 1 2 3 5, 6

7 6 4 5 1 2 3

As a _i

2

> a[i] for every i, no further swaps are needed and we get a heap:

7 6 4 5 1 2 3

Example 1.15. Build a heap for the array data 3, 6, 4, 5, 1, 2.

Solution.

Data swap

a[1] a[2] a[3] a[4] a[5] a[6] a[i ]↔ a _i

2

3 6 4 5 1 2 3, 4

4 6 3 5 1 2 4, 6

6 4 3 5 1 2 4, 5

6 5 3 4 1 2

No further swaps are needed and, so, the heap is

6 5 3 4 1 2

(21)

Example 1.16. Build a heap for the array data 2, 5, 3, 4, 1.

Solution.

Data swap

a[1] a[2] a[3] a[4] a[5] a[i ]↔ a _i

2

2 5 3 4 1 2, 3

3 5 2 4 1 3, 5

5 3 2 4 1 3, 4

5 4 2 3 1

Example 1.17. Build a heap for the array data 1, 4, 2, 3.

Solution.

Data swap

a[1] a[2] a[3] a[4] a[i ]↔ a[ _i

2

]

1 4 2 3 1, 2

2 4 1 3 2, 4

4 2 1 3 2, 3

4 3 1 2

Example 1.18. Build a heap for the array data 2, 3, 1.

Solution. For this we need only one swap 2↔ 3 and the resulting heap is 3, 2, 1.

We now have available all the intermediate steps to heap sort the array of Example 1.12.

Example 1.19. Heap sort the sequence 6, 5, 3, 4, 1, 2, 9, 8, 7.

Solution. We first build a heap for the data which is already done in Example 1.12.

(22)

Heap swap detach sorted array 9 7 8 6 1 2 3 5 4

4 7 8 6 1 2 3 5 9 4↔ 9 9

4 7 8 6 1 2 3 5 9 build heap cf. Ex. 1.13

8 7 4 6 1 2 3 5

5 7 4 6 1 2 3 8 5↔ 8 8

5 7 4 6 1 2 3 8, 9 build heap cf. Ex. 1.14

7 6 4 5 1 2 3

3 6 4 5 1 2 7 3↔ 7 7

3 6 4 5 1 2 7, 8, 9 build heap cf. Ex. 1.15

6 5 3 4 1 2

2 5 3 4 1 6 2↔ 6 6

2 5 3 4 1 6, 7, 8, 9 build heap cf. Ex. 1.16

5 4 2 3 1

1 4 2 3 5 1↔ 5 5

1 4 2 3 5, 6, 7, 8, 9 build heap cf. Ex. 1.17

4 3 1 2

2 3 1 4 2↔ 4 4

2 3 1 4, 5, 6, 7, 8, 9 build heap cf. Ex. 1.18 3 2 1

1 2 3 1↔ 3 3

1 2 3, 4, 5, 6, 7, 8, 9 build heap

2 1

1 2 1↔ 2 2

1 2, 3, 4, 5, 6, 7, 8, 9

and we get the final sorted sequence

1 2 3 4 5 6 7 8 9

Exercise 1.1.

1. Arrange the sequence of Example 1.18 in non-decreasing order using (a) bubble sort, (b) insertion sort, (c) selection sort and (d) quick sort.

2. Arrange the sequences of Examples 1.2, 1.7 and 1.9 using heap sort.