Potential method and amortized analysis

To accurately analyse the performance of an algorithm, let us denote by Φ() as a function that captures the state of an algorithm or its associated data structure at any stage i. We define amortized work done at step i of an algorithm as w_i + ∆_i where wi is actual number of steps⁵ ∆i = Φ(i)− Φ(i − 1) which is referred to as the difference in potential. Note that the total work done by an algorithm over t steps is W =Pi=t

i=1wi. On the other hand, the total amortized work is

t

X

i=1

(wi+ ∆i) = W + Φ(t)− Φ(0)

If Φ(t)− Φ(0) ≥ 0, amortized work is an upperbound on the actual work.

Example 2.1 For the counter problem, we define the potential funcion of the counter as the number of 1’s of the present value. Then the amortised cost for a sequence of 1’s changing to 0 is 0 plus the cost of a 0 changing to 1 resulting in O(1) amortised cost.

Example 2.2 A stack supports push, pop and empty-stack operations. Define Φ() as the number of elements in the stack. If we begin from an empty stack, Φ(0) = 0. For a sequence of push, pop and empty-stack operations, we can analyze the amortized cost. Amortized cost of push is 2, for pop it is 0 and for empty stack it is negative.

Therefore the bound on amortized cost is O(1) and therefore the cost of n operations is O(n). Note that the worst-case cost of an empty-stack operation can be very high.

Exercise 2.11 Can you define an appropriate potential function for the search data-structure analysis ?

5this may be hard to analyze

Chapter 3

Optimization I :

Brute force and Greedy strategy

A generic definition of an optimization problem involves a set of constraints that defines a subset in some underlying space (like the Euclidean space Rⁿ) called the feasible subset and an objective function that we are trying to maximize or minimize as the case may be over the feasible set. A very important common optimization problem is Linear Programming where there is a finite set of linear constraints and the objective function is also linear - the underlying space is Euclidean. A convex function f satisfies f (λ· x + (1 − λ) · y) ≤ λf(x) + (1 − λ)f(y) where 0 < λ < 1.

A convex programming problem is one where the objective function is convex and so is the feasible set. Convex programming problem over Euclidean real space have a nice property that local optima equals the global optimum. Linear programming falls under this category and there are provably efficient algorithms for linear programming.

Many important real life problems can be formulated as optimization problems and therefore solving them efficiently is one of the most important area of algorithm design.

3.1 Heuristic search approaches

In this section, we will use the knapsack problem as the running example to expose some of the algorithmic ideas. The 0-1 Knapsack problem is defined as follows.

Given a knapsack of capacity C and n objects of volumes {w1, w2. . . wn} and profits{p1, p2. . . pn}, the objective is to choose a subset of n objects that fits into the knapsack and that maximizes the total profit.

In more formal terms, let xibe 1 if object i is present in the subset and 0 otherwise.

The knapsack problem can be stated as

Note that the constraint xi ∈ {0, 1} is not linear. A simplistic approach will be to enumerate all subsets and select the one that satisfies the constraints and maximizes the profits. Any solution that satisfies the capacity constraint is called a feasible solution. The obvious problem with this strategy is the running time which is at least 2ⁿ corresponding to the power-set of n objects.

We can imagine that the solution space is generated by a binary tree where we start from the root with an empty set and then move left or right according to selecting x1. At the second level, we again associate the left and right branches with the choice of x2. In this way, the 2ⁿ leaf nodes correspond to each possible subset of the power-set which corresponds to a n length 0-1 vector. For example, a vector 000 . . . 1 corresponds to the subset that only contains xn.

Any intermediate node at level j from the root corresponds to partial choice among the objects x₁, x₂. . . x_j. As we traverse the tree, we keep track of the best feasible solution among the nodes visited - let us denote this by T . At a node v, let S(v) denote the subtree rooted at v. If we can estimate the maximum profit P (v) among the leaf nodes of S(v), we may be able to prune the search. Suppose L(v) and U(v) are the lower and upperbounds of P (v), i.e. L(v)≤ P (v) ≤ U(v). If U(v) < T , then there is no need to explore S(v) as we cannot improve the current best solution, viz., T . In fact, it is enough to work with only the upper-bound of the estimates and L(v) is essentially the current partial solution. As we traverse the tree, we also update U(v) and if it is less than T , we do not search the subtree any further. This method of pruning search is called branch and bound and although it is clear that there it is advantageous to use the strategy, there may not be any provable savings in the worst case.

Exercise 3.1 Construct an instance of a knapsack problem that visits every leaf node, even if you use branch and bound. You can choose any well defined estimation.

Example 3.1

Let the capacity of the knapsack be 15 and the weights and profits are respectively Profits 10 10 12 18

Volume 2 4 6 9

We will use the ratio of profit per volume as an estimation for upperbound. For the above objects the ratios are 5, 2.5, 2 and 2. Initially, T = 0 and U = 5× 15 = 75.

After including x1, the residual capacity is 13 and T = 10. By proceeding this way, we

obtain T = 38 for{x¹, x2, x4}. By exploring further, we come to a stage when we have included x1 and decided against including x2 so that L(v) = 10, and residual capacity is 13. Should we explore the subtree regarding {x3, x₄} ? Since profit per volume of x3 is 2, we can obtain U(v) = 2× 13 + 10 = 36 < L = 38. So we need not search this subtree. By continuing in this fashion, we may be able to prune large portions of the search tree. However, it is not possible to obtain any provable improvements.

Exercise 3.2 Show that if we use a greedy strategy based on profit/volume, i.e., choose the elements in decreasing order of this ratio, then the profit as at least half of the optimal solution. For this claim, you need to make one change, namely, if xk

is the last object chosen, such that x1, x2. . . xk in decreasing order of their ratios that can fit in the knapsack, then eventually choose max{Pi=k

i=1pi, pk+1}. Note that xk+1

is such that Pi=k

i=1w_i ≤ C <Pi=k+1 i=1 w_i.