CS9212 UNIT I

DATA STRUCTURES AND ALGORITHMS COMPLEXITY ANALYSIS & ELEMENTARY DATA STRUCTURES

Asymptotic notations Properties of big oh notation asymptotic notation with several parameters conditional asymptotic notation amortized analysis NP-completeness NP-hard recurrence equations solving recurrence equations arrays linked lists trees. UNIT II HEAP STRUCTURES Min-max heaps Deaps Leftist heaps Binomial heaps Fibonacci heaps Skew heaps Lazy-binomial heaps. UNIT III SEARCH STRUCTURES Binary search trees AVL trees 2-3 trees 2-3-4 trees Red-black trees B-trees splay trees Tries. UNIT IV GREEDY & DIVIDE AND CONQUER Quicksort Strassens matrix multiplication Convex hull - Tree-vertex splitting Job sequencing with deadlines Optimal storage on tapes UNIT V DYNAMIC PROGRAMMING AND BACKTRACKING Multistage graphs 0/1 knapsack using dynamic programming Flow shop scheduling 8queens problem graph coloring knapsack using backtracking REFERENCES: 1. E. Horowitz, S.Sahni and Dinesh Mehta, Fundamentals of Data structures in C++, Galgotia, 1999. 2. E. Horowitz, S.Sahni and S. Rajasekaran, Computer Algorithms / C++, Galgotia, 1999. 3. Adam Drozdex, Data Structures and algorithms in C++, Second Edition, Thomson learning vikas publishing house, 2001. 4. G. Brassard and P. Bratley, Algorithmics: Theory and Practice, Printice Hall, 1988. 5. Thomas H.Corman, Charles E.Leiserson, Ronald L. Rivest, Introduction to Algorithms, Second Edition, PHI 2003.

UNIT I COMPLEXITY ANALYSIS & ELEMENTARY DATA STRUCTURES

Asymptotic Notation Introduction A problem may have numerous algorithmic solutions. In order to choose the best algorithm for a particular task, you need to be able to judge how long a particular solution will take to run. Or, more accurately, you need to be able to judge how long two solutions will take to run, and choose the better of the two. You don't need to know how many minutes and seconds they will take, but you do need some way to compare algorithms against one another. Asymptotic complexity is a way of expressing the main component of the cost of an algorithm, using idealized units of computational work. Consider, for example, the algorithm for sorting a deck of cards, which proceeds by repeatedly searching through the deck for the lowest card. The asymptotic complexity of this algorithm is the square of the number of cards in the deck. This quadratic behavior is the main term in the complexity formula, it says, e.g., if you double the size of the deck, then the work is roughly quadrupled. The exact formula for the cost is more complex, and contains more details than are needed to understand the essential complexity of the algorithm. With our deck of cards, in the worst case, the deck would start out reverse-sorted, so our scans would have to go all the way to the end. The first scan would involve scanning 52 cards, the next would take 51, etc. So the cost formula is 52 + 51 + ..... + 2. generally, letting N be the number of cards, the formula is 2 + ... + N, which equals . But the N^2 term dominates the expression, and this is what is key for comparing algorithm costs. (This is in fact an expensive algorithm; the best sorting algorithms run in sub-quadratic time.) Asymptotically speaking, in the limit as N tends towards infinity, 2 + 3 + .,,,.. + N gets closer and closer to the pure quadratic function (1/2)N^2. And what difference does the constant factor of 1/2 make, at this level of abstraction? So the behavior is said to be O( ). Now let us consider how we would go about comparing the complexity of two algorithms. Let f(n) be the cost, in the worst case, of one algorithm, expressed as a function of the input size n, and g(n) be the cost function for the other algorithm. E.g., for sorting algorithms, f(10) and g(10) would be the maximum number of steps that the algorithms would take on a list of 10 items. If, for all values of n >= 0, f(n) is less than or equal to g(n), then the algorithm with complexity function f is strictly faster. But, generally speaking, our concern for computational cost is for the cases with large inputs; so the comparison of f(n) and g(n) for small values of n is less significant than the "long term" comparison of f(n) and g(n), for n larger than some threshold. Note that we have been speaking about bounds on the performance of algorithms, rather than giving exact speeds. The actual number of steps required to sort our deck of cards (with our naive quadratic algorithm) will depend upon the order in which the cards begin. The actual time to perform each of our steps will depend upon our processor speed, the condition of our processor cache, etc.,
2

etc. It's all very complicated in the concrete details, and moreover not relevant to the essence of the algorithm.

Big-O Notation Definition Big-O is the formal method of expressing the upper bound of an algorithm's running time. It's a measure of the longest amount of time it could possibly take for the algorithm to complete. More formally, for non-negative functions, f(n) and g(n), if there exists an integer and a constant c > 0 such that for all integers , f(n) cg(n), then f(n) is Big O of g(n). This is denoted as "f(n) = O(g(n))". If graphed, g(n) serves as an upper bound to the curve you are analyzing, f(n). Note that if f can take on finite values only (as it should happen normally) then this definition implies that there exists some constant C (potentially larger than c) such that for all values of n, f(n) Cg(n). An appropriate value for C is the maximum of c and Theory Examples So, let's take an example of Big-O. Say that f(n) = 2n + 8, and g(n) = . Can we find a constant , so that 2n + 8 <= ? The number 4 works here, giving us 16 <= 16. For any number n greater than 4, this will still work. Since we're trying to generalize this for large values of n, and small values (1, 2, 3) aren't that important, we can say that f(n) is generally faster than g(n); that is, f(n) is bound by g(n), and will always be less than it. It could then be said that f(n) runs in O( ) time: "f-of-n runs in Big-O of n-squared time". .

To find the upper bound - the Big-O time - assuming we know that f(n) is equal to (exactly) 2n + 8, we can take a few shortcuts. For example, we can remove all constants from the runtime; eventually, at some value of c, they become irrelevant. This makes f(n) = 2n. Also, for convenience of comparison, we remove constant multipliers; in this case, the 2. This makes f(n) = n. It could also be said that f(n) runs in O(n) time; that lets us put a tighter (closer) upper bound onto the estimate. Practical Examples O(n): printing a list of n items to the screen, looking at each item once. O(ln n): also "log n", taking a list of items, cutting it in half repeatedly until there's only one item left. O( ): taking a list of n items, and comparing every item to every other item.

Big-Omega Notation
3

For non-negative functions, f(n) and g(n), if there exists an integer and a constant c > 0 such that for all integers , f(n) cg(n), then f(n) is omega of g(n). This is denoted as "f(n) = (g(n))". This is almost the same definition as Big Oh, except that "f(n) cg(n)", this makes g(n) a lower bound function, instead of an upper bound function. It describes the best that can happen for a given data size. Theta Notation For non-negative functions, f(n) and g(n), f(n) is theta of g(n) if and only if f(n) = O(g(n)) and f(n) = (g(n)). This is denoted as "f(n) = (g(n))". This is basically saying that the function, f(n) is bounded both from the top and bottom by the same function, g(n). Little-O Notation For non-negative functions, f(n) and g(n), f(n) is little o of g(n) if and only if f(n) = O(g(n)), but f(n) (g(n)). This is denoted as "f(n) = o(g(n))". This represents a loose bounding version of Big O. g(n) bounds from the top, but it does not bound the bottom. Little Omega Notation For non-negative functions, f(n) and g(n), f(n) is little omega of g(n) if and only if f(n) = (g(n)), but f(n) (g(n)). This is denoted as "f(n) = (g(n))". Much like Little Oh, this is the equivalent for Big Omega. g(n) is a loose lower boundary of the function f(n); it bounds from the bottom, but not from the top. How asymptotic notation relates to analyzing complexity Temporal comparison is not the only issue in algorithms. There are space issues as well. Generally, a trade off between time and space is noticed in algorithms. Asymptotic notation empowers you to make that trade off. If you think of the amount of time and space your algorithm uses as a function of your data over time or space (time and space are usually analyzed separately), you can analyze how the time and space is handled when you introduce more data to your program. This is important in data structures because you want a structure that behaves efficiently as you increase the amount of data it handles. Keep in mind though that algorithms that are efficient with large amounts of data are not always simple and efficient for small amounts of data. So if you know you are working with only a small amount of data and you have concerns for speed and code space, a trade off can be made for a function that does not behave well for large amounts of data.

Amortized Analysis
Overview This lecture discusses a useful form of analysis, called amortized analysis, for problems in which one must perform a series of operations, and our goal is to analyze the time per operation. The motivation for amortized analysis is that looking at the worst-case time per operation can be too pessimistic if the only way to produce an expensive operation is to set it up with a large number of cheap operations beforehand.

We also introduce the notion of a potential function which can be a useful aid to performing this type of analysis. A potential function is much like a bank account: if we can take our cheap operations (those whose cost is less than our bound) and put our savings from them in a bank account, use our savings to pay for expensive operations (those whose cost is greater than our bound), and somehow guarantee that our account will never go negative, then we will have proven an amortized bound for our procedure.As in the previous lecture, in this lecture we will avoid use of asymptotic notation as much as possible, and focus instead on concrete cost models and bounds. Introduction So far we have been looking at static problems where you are given an input (like an array of n objects) and the goal is to produce an output with some desired property (e.g., the same objects, but sorted). For next few lectures, were going to turn to problems where we have a series of operations, and goal is to analyze the time taken per operation. For example, rather than being given a set of n items up front, we might have a series of n insert, lookup, and remove requests to some database, and we want these operations to be efficient. Today, well talk about a useful kind of analysis, called amortized analysis for problems of this sort. The definition of amortized cost is actually quite simple: Definition : The amortized cost of n operations is the total cost of the operations divided by n. Analyzing the amortized cost, however, will often require some thought if we want to do it well.We will illustrate how this can be done through several examples. Example #1: Implementing a stack as an array Say we want to use an array to implement a stack. We have an array A, with a variable top that points to the top of the stack (so A[top] is the next free cell). This is pretty easy: To implement push(x), we just need to perform: A[top] = x; top++; To implement x=pop(), we just need to perform: top--; x = A[top]; (first checking to see if top==0 of course...) However, what if the array is full and we need to push a new element on? In that case we can
7

allocate a new larger array, copy the old one over, and then go on from there. This is going to be an expensive operation, so a push that requires us to do this is going to cost a lot. But maybe we can amortize the cost over the previous cheap operations that got us to this point. So, on average over the sequence of operations, were not paying too much. To be specific, let us define the following cost model. Cost model: Say that inserting into the array costs 1, taking an element out of the array costs 1, and the cost of resizing the array is the number of elements moved. (Say that all other operations, like incrementing or decrementing top, are free.) Question 1: What if when we resize we just increase the size by 1. Is that a good idea? Answer 1: Not really. If our n operations consist of n pushes then even just considering the array-resizing cost we will incur a total cost of at least 1 + 2 + 3 + 4 + . . . + (n 1) = n(n 1)/2. Thats an amortized cost of (n 1)/2 per operation just in resizing. Question 2: What if instead we decide to double the size of the array when we resize? Answer 2: This is much better. Now, in any sequence of n operations, the total cost for resizing is 1 + 2 + 4 + 8 + . . . + 2i for some 2i < n (if all operations are pushes then 2i will be the largest power of 2 less than n). This sum is at most 2n 1. Adding in the additional cost of n for inserting/removing, we get a total cost < 3n, and so our amortized cost per operation is < 3. Piggy banks and potential functions Here is another way to analyze the process of doubling the array in the above example. Say that every time we perform a push operation, we pay $1 to perform it, and we put $2 into a piggy bank. So, our out-of-pocket cost per push is $3. Any time we need to double the array, from size L to 2L, we pay for it using money in the bank. How do we know there will be enough money ($L) in the bank to pay for it? Because after the last resizing, there were only L/2 elements in the array and so there must have been at least L/2 new pushes since then contributing $2 each. So, we can pay for everything by using an out-of-pocket cost of at most $3 per operation. Putting it another way, by spending $3 per operation, we were able to pay for all the operations plus possibly still have money left over in the bank. This means our amortized cost is at most 3.1 This piggy bank method is often very useful for performing amortized analysis. The piggy bank is also called a potential function, since it is like potential energy that you can use later. The potential function is the amount of money in the bank. In the case above, the potential is 2 times the number of elements in the array after the midpoint. Note that it is very important in this
8

analysis to prove that the bank account doesnt go negative. Otherwise, if the bank account can slowly drift off to negative infinity, the whole proof breaks down. Definition: A potential function is a function of the state of a system, that generally should be non-negative and start at 0, and is used to smooth out analysis of some algorithm or process. Observation: If the potential is non-negative and starts at 0, and at each step the actual cost of our algorithm plus the change in potential is at most c, then after n steps our total cost is at most cn. That is just the same thing we were saying about the piggy bank: our total cost for the n operations is just our total out of pocket cost minus the amount in the bank at the end. Sometimes one may need in an analysis to seed the bank account with some initial positive amount for everything to go through. In that case, the kind of statement one would show is that the total cost for n operations is at most cn plus the initial seed amount. Recap: The motivation for amortized analysis is that a worst-case-per-operation analysis can give overly pessimistic bound if the only way of having an expensive operation is to have a lot of cheap ones before it. Note that this is different from our usual notion of average case analysis: we are not making any assumptions about the inputs being chosen at random, we are just averaging over time. Example #2: a binary counter Imagine we want to store a big binary counter in an array A. All the entries start at 0 and at each step we will be simply incrementing the counter. Lets say our cost model is: whenever we increment the counter, we pay $1 for every bit we need to flip. (So, think of the counter as an 1In fact, if you think about it, we can pay for pop operations using money from the bank too, and even have $1 left over. So as a more refined analysis, our amortized cost is $3 per push and $1 per successful pop (a pop from a nonempty stack). EXAMPLE #3: WHAT IF IT COSTS US 2K TO FLIP THE KTH BIT? array of heavy stone tablets, each with a 0 on one side and a 1 on the other.) For instance, here is a trace of the first few operations and their cost: A[m] A[m-1] ... A[3] A[2] A[1] A[0] cost 0 0 ... 0 0 0 0 $1
9

0 0 0 0 0

0 0 0 0 0

... 0 ... 0 ... 0 ... 0 ... 0

0 0 0 1 1

0 1 1 0 0

1 0 1 0 1

$2 $1 $3 $1 $2

In a sequence of n increments, the worst-case cost per increment is O(log n), since at worst we flip lg(n) + 1 bits. But, what is our amortized cost per increment? The answer is it is at most 2. Here are two proofs. Proof 1: Every time you flip 0 ! 1, pay the actual cost of $1, plus put $1 into a piggy bank. So the total amount spent is $2. In fact, think of each bit as having its own bank (so when you turn the stone tablet from 0 to 1, you put a $1 coin on top of it). Now, every time you flip a 1 ! 0, use the money in the bank (or on top of the tablet) to pay for the flip. Clearly, by design, our bank account cannot go negative. The key point now is that even though different increments can have different numbers of 1 ! 0 flips, each increment has exactly one 0 ! 1 flip. So, we just pay $2 (amortized) per increment. Equivalently, what we are doing in this proof is using a potential function that is equal to the number of 1-bits in the current count. Notice how the bank-account/potential-function allows us to smooth out our payments, making the cost easier to analyze. Proof 2: Here is another way to analyze the amortized cost. First, how often do we flip A[0]? Answer: every time. How often do we flip A[1]? Answer: every other time. How often do we flip A[2]? Answer: every 4th time, and so on. So, the total cost spent on flipping A[0] is n, the total cost spent flipping A[1] is n/2, the total cost flipping A[2] is n/4, etc. Summing these up, the total cost spent flipping all the positions in our n increments is at most 2n. Example #3: What if it costs us 2k to flip the kth bit? Imagine a version of the counter we just discussed in which it costs 2k to flip the bit A[k]. (Suspend disbelief for now well see shortly why this is interesting to consider). Now, in a sequence of n increments, a single increment could cost as much as n, but the claim is the amortized cost is only O(log n) per increment. This is probably easiest to see by the method of Proof 2 above: A[0] gets flipped every time for cost of $1 each (a total of $n). A[1] gets flipped every other time for cost of $2 each (a total of at most $n). A[2] gets flipped every 4th time for cost of $4 each (again,
10

a total of at most $n), and so on up to A[blg nc] which gets flipped once for a cost at most $n. So, the total cost is O(n log n), which is O(log n) amortized per increment.

Example #4: A simple amortized dictionary data structure One of the most common classes of data structures are the dictionary data structures that support fast insert and lookup operations into a set of items. In the next lecture we will look at balanced-tree data structures for this problem in which both inserts and lookups can be done with cost only O(log n) each. Note that a sorted array is good for lookups (binary search takes time only O(log n)) but bad for inserts (they can take linear time), and a linked list is good for inserts (can do them in constant time) but bad for lookups (they can take linear time). Here is a method that is very simple and almost as good as the ones in the next lecture. This method has O(log2 n) search time and O(log n) amortized cost per insert. The idea of this data structure is as follows. We will have a collection of arrays, where array i has size 2i. Each array is either empty or full, and each is in sorted order. However, there will be no relationship between the items in different arrays. The issue of which arrays are full and which are empty is based on the binary representation of the number of items we are storing. For example, if we had 11 items (where 11 = 1 + 2 + 8), the data structure might look like this: A0: [5] A1: [4,8] A2: empty A3: [2, 6, 9, 12, 13, 16, 20, 25] To perform a lookup, we just do binary search in each occupied array. In the worst case, this takes time O(log(n/2) + log(n/4) + log(n/8) + . . . + 1) = O(log2 n). What about inserts? Well do this like mergesort. To insert the number 10, we first create an array of size 1 that just has this single number in it. We now look to see if A0 is empty. If so we make this be A0. If not (like in the above example) we merge our array with A0 to create a new array (which in the above case would now be [5, 10]) and look to see if A1 is empty. If A1 is empty, we make this be A1. If not (like in the above example) we merge this with A1 to create a new array and check to see if A2 is empty, and so on. So, inserting 10 in the example above, we now have: A0: empty
11

A1: empty A2: [4, 5, 8, 10] A3: [2, 6, 9, 12, 13, 16, 20, 25]

Cost model: To be clear about costs, lets say that creating the initial array of size 1 costs 1, and merging two arrays of size m costs 2m (remember, merging sorted arrays can be done in linear time). So, the above insert had cost 1 + 2 + 4. For instance, if we insert again, we just put the new item into A0 at cost 1. If we insert again, we merge the new array with A0 and put the result into A1 at a cost of 1 + 2. EXAMPLE #4: A SIMPLE AMORTIZED DICTIONARY DATA STRUCTUREClaim 7.1 The above data structure has amortized cost O(log n) per insert. Proof: With the cost model defined above, its exactly the same as the binary counter with cost 2k for counter k.

NP-complete problems
Hard problems, easy problems In short, the world is full of search problems, some of which can be solved efciently, while others seem to be very hard. This is depicted in the following table.

This table is worth contemplating. On the right we have problems that can be solved effciently. On the left, we have a bunch of hard nuts that have escaped effcient solution over
12

many decades or centuries. The various problems on the right can be solved by algorithms that are specialized and diverse: dynamic programming, network ow, graph search, greedy. These problems are easy for a variety of different reasons. In stark contrast, the problems on the left are all diffcult for the same reason! At their core, they are all the same problem, just in different disguises! They are all equivalent: each of them can be reduced to any of the others.and back. P and NP It's time to introduce some important concepts. We know what a search problem is: its dening characteristic is that any proposed solution can be quickly checked for correctness, in the sense that there is an efcient checking algorithm C that takes as input the given instance I (the data specifying the problem to be solved), as well as the proposed solution S, and outputs true if and only if S really is a solution to instance I. Moreover the running time of C(I; S) is bounded by a polynomial in jIj, the length of the instance. We denote the class of all search problems by NP. We've seen many examples of NP search problems that are solvable in polynomial time. In such cases, there is an algorithm that takes as input an instance I and has a running time polynomial in jIj. If I has a solution, the algorithm returns such a solution; and if I has no solution, the algorithm correctly reports so. The class of all search problems that can be solved in polynomial time is denoted P. Hence, all the search problems on the right-hand side of the table are in P. Why P and NP? Okay, P must stand for .polynomial.. But why use the initials NP (the common chatroom abbreviation for .no problem.) to describe the class of search problems, some of which are terribly hard? NP stands for .nondeterministic polynomial time,. a term going back to the roots of complexity theory. Intuitively, it means that a solution to any search problem can be found and veried in polynomial time by a special (and quite unrealistic) sort of algorithm, called a nondeterministic algorithm. Such an algorithm has the power of guessing correctly at every step. Incidentally, the original denition of NP (and its most common usage to this day) was not as a class of search problems, but as a class of decision problems: algorithmic questions that can be answered by yes or no. Example: .Is there a truth assignment that satises this Boolean formula?. But this too reects a historical reality: At the time the theory of NPcompleteness
13

was being developed, researchers in the theory of computation were interested in formal languages, a domain in which such decision problems are of central importance. Are there search problems that cannot be solved in polynomial time? In other words, is P not equal to NP? Most algorithms researchers think so. It is hard to believe that exponential search can always be avoided, that a simple trick will crack all these hard problems, famously unsolved for decades and centuries. And there is a good reason for mathematicians to believe that P not equal to NP.the task of nding a proof for a given mathematical assertion is a search problem and is therefore in NP (after all, when a formal proof of a mathematical statement is written out in excruciating detail, it can be checked mechanically, line by line, by an efcient algorithm). So if P = NP, there would be an efcient method to prove any theorem, thus eliminating the need for mathematicians! All in all, there are a variety of reasons why it is widely believed that P not equal to NP. However, proving this has turned out to be extremely difcult, one of the deepest and most important unsolved puzzles of mathematics. Reductions, again Even if we accept that P not equal to NP, what about the specic problems on the left side of the table? On the basis of what evidence do we believe that these particular problems have no efcient algorithm (besides, of course, the historical fact that many clever mathematicians and computer scientists have tried hard and failed to nd any)? Such evidence is provided by reductions, which translate one search problem into another. What they demonstrate is that the problems on the left side of the table are all, in some sense, exactly the same problem, except that they are stated in different languages. What's more, we will also use reductions to show that these problems are the hardest search problems in NP.if even one of them has a polynomial time algorithm, then every problem in NP has a polynomial time algorithm. Thus if we believe that P not equal to NP, then all these search problems are hard. We dened reductions in Chapter 7 and saw many examples of them. Let's now specialize this denition to search problems. A reduction from search problem A to search problem B is a polynomial-time algorithm f that transforms any instance I of A into an instance f(I) of B, together with another polynomial-time algorithm h that maps any solution S of f(I) back into a solution h(S) of I; see the following diagram. If f(I) has no solution, then neither does I. These two translation procedures f and h imply that any algorithm for B can be converted into an algorithm for A by bracketing it between f and h.
14

And now we can nally dene the class of the hardest search problems. A search problem is NP-complete if all other search problems reduce to it. This is a very strong requirement indeed. For a problem to be NP-complete, it must be useful in solving every search problem in the world! It is remarkable that such problems exist. But they do, and the rst column of the table we saw earlier is lled with the most famous examples. In Section 8.3 we shall see how all these problems reduce to one another, and also why all other search problems reduce to them.

The two ways to use reductions So far in this book the purpose of a reduction from a problem A to a problem B has been straightforward and honorable: We know how to solve B efciently, and we want to use this knowledge to solve A. In this chapter, however, reductions from A to B serve a somewhat perverse goal: we know A is hard, and we use the reduction to prove that B is hard as well!
15

If we denote a reduction from A to B by A --> B then we can say that difculty ows in the direction of the arrow, while efcient algorithms move in the opposite direction. It is through this propagation of difculty that we know NP-complete problems are hard: all other search problems reduce to them, and thus each NP-complete problem contains the complexity of all search problems. If even one NP-complete problem is in P, then P = NP. Reductions also have the convenient property that they compose. If A --> B and B --> C, then A ---> C . To see this, observe rst of all that any reduction is completely specied by the pre- and postprocessing functions f and h (see the reduction diagram). If (fAB; hAB) and (fBC; hBC) dene the reductions from A to B and from B to C, respectively, then a reduction from A to C is given by compositions of these functions: fBC o fAB maps an instance of A to an instance of C and hAB hBC sends a solution of C back to a solution of A. This means that once we know a problem A is NP-complete, we can use it to prove that a new search problem B is also NP-complete, simply by reducing A to B. Such a reduction establishes that all problems in NP reduce to B, via A.

NP-Hard Problems
Efficient Problems

A generally-accepted minimum requirement for an algorithm to be considered efficient is that its running time is polynomial: O(nc) for some constant c, where n is the size of the input.1 Researchers recognized early on that not all problems can be solved this quickly, but we had a hard time figuring out exactly which ones could and which ones couldnt. there are several so-called NPhard problems, which most people believe cannot be solved in polynomial time, even though nobody can prove a super-polynomial lower bound. Circuit satisfiability is a good example of a problem that we dont know how to solve in polynomial time. In this problem, the input is a boolean circuit: a collection of AND, OR, and NOT gates connected by wires. We will assume that there are no loops in the circuit (so no delay lines or flipflops). The input to the circuit is a set of m boolean (TRUE/FALSE) values x1, . . . , xm. The output is a single boolean value. Given specific input values, we can calculate the output of the circuit in polynomial (actually, linear) time using depth-first-search, since we can compute the output of a kinput gate in O(k) time.

16

The circuit satisfiability problem asks, given a circuit, whether there is an input that makes the circuit output TRUE, or conversely, whether the circuit always outputs FALSE. Nobody knows how to solve this problem faster than just trying all 2m possible inputs to the circuit, but this requires exponential time. On the other hand, nobody has ever proved that this is the best we can do; maybe theres a clever algorithm that nobody has discovered yet! P, NP, and co-NP A decision problem is a problem whose output is a single boolean value: YES or NO.2 Let me define three classes of decision problems: P is the set of decision problems that can be solved in polynomial time.3 Intuitively, P is the set of problems that can be solved quickly. NP is the set of decision problems with the following property: If the answer is YES, then there is a proof of this fact that can be checked in polynomial time. Intuitively, NP is the set of decision problems where we can verify a YES answer quickly if we have the solution in front of us. co-NP is the opposite of NP. If the answer to a problem in co-NP is NO, then there is a proof of this fact that can be checked in polynomial time. For example, the circuit satisfiability problem is in NP. If the answer is YES, then any set of m input values that produces TRUE output is a proof of this fact; we can check the proof by evaluating the circuit in polynomial time. It is widely believed that circuit satisfiability is not in P or in co-NP, but nobody actually knows. Every decision problem in P is also in NP. If a problem is in P, we can verify YES answers in polynomial time recomputing the answer from scratch! Similarly, any problem in P is also in co-NP. One of the most important open questions in theoretical computer science is whether or not P = NP. Nobody knows. Intuitively, it should be obvious that P 6= NP; the homeworks and exams in this
17

class and others have (I hope) convinced you that problems can be incredibly hard to solve, even when the solutions are obvious in retrospect. But nobody knows how to prove it. A more subtle but still open question is whether NP and co-NP are different. Even if we can verify every YES answer quickly, theres no reason to think that we can also verify NO answers quickly. For example, as far as we know, there is no short proof that a boolean circuit is not satisfiable. It is generally believed that NP != co-NP, but nobody knows how to prove it.

NP-hard, NP-easy, and NP-complete A problem is NP-hard if a polynomial-time algorithm for would imply a polynomial-time algorithm for every problem in NP. In other words:

Intuitively, if we could solve one particular NP-hard problem quickly, then we could quickly solve any problem whose solution is easy to understand, using the solution to that one special problem as a subroutine. NP-hard problems are at least as hard as any problem in NP.4 Saying that a problem is NP-hard is like saying If I own a dog, then it can speak fluent English. You probably dont know whether or not I own a dog, but youre probably pretty sure that I dont own a talking dog. Nobody has a mathematical proof that dogs cant speak Englishthe fact that no one has ever heard a dog speak English is evidence, as are the hundreds of examinations of dogs that lacked the proper mouth shape and braiNPower, but mere evidence is not a mathematical proof. Nevertheless, no sane person would believe me if I said I owned a dog that spoke fluent English. So the statement If I own a dog, then it can speak fluent English has a natural corollary: No one in their right mind should believe that I own a dog! Likewise, if a problem is NP-hard, no one in their right mind should believe it can be solved in polynomial time.

18

Finally, a problem is NP-complete if it is both NP-hard and an element of NP (or NP-easy). Npcomplete problems are the hardest problems in NP. If anyone finds a polynomial-time algorithm for even one NP-complete problem, then that would imply a polynomial-time algorithm for every NP-complete problem. Literally thousands of problems have been shown to be NP-complete, so a polynomial-timealgorithm for one (that is, all) of them seems incredibly unlikely.

It is not immediately clear that any decision problems are NP-hard or NP-complete. NPhardness is already a lot to demand of a problem; insisting that the problem also have a nondeterministic polynomial-time algorithm seems almost completely unreasonable. The following remarkable theorem was first published by Steve Cook in 1971 and independently by Leonid Levin in 1973.5 I wont even sketch the proof, since Ive been (deliberately) vague about the definitions.

What is a recurrence relation?

A recurrence relation, T(n), is a recursive function of integer variable n. Like all recursive functions, it has both recursive case and base case.

Example:

The portion of the definition that does not contain T is called the base case of the recurrence relation; the portion that contains T is called the recurrent or recursive case. Recurrence relations are useful
19

for expressing the running times (i.e., the number of basic operations executed) of recursive algorithms. Forming Recurrence Relations For a given recursive method, the base case and the recursive case of its recurrence relation correspond directly to the base case and the recursive case of the method. Example 1: Write the recurrence relation for the following method. public void f (int n) { if (n > 0) { System.out.println(n); f(n-1); } } The base case is reached when n == 0. The method performs one comparison. Thus, the number of operations when n == 0, T(0), is some constant a. When n > 0, the method performs two basic operations and then calls itself, using ONE recursive call, with a parameter n 1. Therefore the recurrence relation is:

Example 2: Write the recurrence relation for the following method. public int g(int n) { if (n == 1) return 2; else return 3 * g(n / 2) + g( n / 2) + 5; } The base case is reached when n == 1. The method performs one comparison and one return statement. Therefore, T(1), is constant c. When n > 1, the method performs TWO recursive calls, each with the parameter n / 2, and some constant # of basic operations.
20

Hence, the recurrence relation is:

Solving Recurrence Relations To solve a recurrence relation T(n) we need to derive a form of T(n) that is not a recurrence relation. Such a form is called a closed form of the recurrence relation. There are four methods to solve recurrence relations that represent the running time of recursive methods:

Iteration method (unrolling and summing) Substitution method Recursion tree method Master method

In this course, we will only use the Iteration method. Iteration method Steps: Expand the recurrence Express the expansion as a summation by plugging the recurrence back into itself until you see a pattern. Evaluate the summation In evaluating the summation one or more of the following summation formulae may be used: Arithmetic series:

Geometric Series:

21

Geometric Series:(special case)

Harmonic Series:

Others:

Analysis Of Recursive Factorial method

Example1: Form and solve the recurrence relation for the running time of factorial method and hence determine its big-O complexity: long factorial (int n) { if (n == 0) return 1; else return n * factorial (n 1); } T(0) = c T(n) = b + T(n - 1) = b + b + T(n - 2) = b +b +b + T(n - 3) = kb + T(n - k)
22

When k = n, we have: T(n) = nb + T(n - n) = bn + T(0) = bn + c. Therefore method factorial is O(n).

Analysis Of Recursive Selection Sort

public static void selectionSort(int[] x) { selectionSort(x, x.length - 1);} private static void selectionSort(int[] x, int n) { int minPos; if (n > 0) { minPos = findMinPos(x, n); swap(x, minPos, n); selectionSort(x, n - 1); } } private static int findMinPos (int[] x, int n) { int k = n; for(int i = 0; i < n; i++) if(x[i] < x[k]) k = i; return k; } private static void swap(int[] x, int minPos, int n) { int temp=x[n]; x[n]=x[minPos]; x[minPos]=temp;} findMinPos is O(n), and swap is O(1), therefore the recurrence relation for the running time of the selectionSort method is: T(0) = a T(n) = T(n 1) + n + c n>0 = [T(n-2) +(n-1) + c] + n + c = T(n-2) + (n-1) + n + 2c = [T(n-3) + (n-2) + c] +(n-1) + n + 2c= T(n-3) + (n-2) + (n-1) + n + 3c = T(n-4) + (n-3) + (n-2) + (n-1) + n + 4c = = T(n-k) + (n-k + 1) + (n-k + 2) + .+ n + kc When k = n, we have :
23

2 Therefore, Recursive Selection Sort is O(n )

Analysis Of Recursive Binary Search

public int binarySearch (int target, int[] array, int low, int high) { if (low > high) return -1; else { int middle = (low + high)/2; if (array[middle] == target) return middle; else if(array[middle] < target) return binarySearch(target, array, middle + 1, high); else return binarySearch(target, array, low, middle - 1); } } The recurrence relation for the running time of the method is: T(1) = a T(n) = T(n / 2) + b Expanding: T(n) = T(n / 2) + b 2 = [T(n / 4) + b] + b = T (n / 2 ) + 2b 3 = [T(n / 8) + b] + 2b = T(n / 2 ) + 3b = .. k = T( n / 2 ) + kb
24

if n = 1 if n > 1

(one element array)

k k When n / 2 = 1 --> n = 2 --> k = log n, we have: 2 T(n) = T(1) + b log n 2 = a + b log n 2 Therefore, Recursive Binary Search is O(log n)

Analysis Of Recursive Towers of Hanoi Algorithm

public static void hanoi(int n, char from, char to, char temp){ if (n == 1) System.out.println(from + " --------> " + to); else{ hanoi(n - 1, from, temp, to); System.out.println(from + " --------> " + to); hanoi(n - 1, temp, to, from); } } The recurrence relation for the running time of the method hanoi is: T(n) = a T(n) = 2T(n - 1) + b Expanding: T(n) = 2T(n 1) + b = 2[2T(n 2) + b] + b 2 = 2 [2T(n 3) + b] + 2b + b 3 = k = 2 T(n k) + b[2 k- 1 +2 k 2 1 0 +... 2 +2 ] 2 = 2 [2T(n 4) + b] + 2 b + 2b + b 2 = 2 T(n 2) + 2b + b 3 2 = 2 T(n 3) + 2 b + 2b + b 4 3 2 1 0 = 2 T(n 4) + 2 b + 2 b + 2 b + 2 b if n > 1 if n = 1

25

When k = n 1, we have:

n Therefore, The method hanoi is O(2 )

UNIT II HEAP STRUCTURES

Min-Max Heaps and Generalized Priority Queues INTRODUCTION A (single-ended) priority queue is a data type supporting the following operations on an ordered set of values: 1) find the maximum value (FindMax); 2) delete the maximum value (DeleteMax); 3) add a new value x (Insert(x)). Obviously, the priority queue can be redefined by substituting operations 1) and 2) with FindMin and DeleteMin, respectively. Several structures, some implicitly stored in an array and some using more complex data structures, have been presented for implementing this data type, including max-heaps (or min-heaps) Conceptually, a max-heap is a binary tree having the following properties: a) heap-shape: all leaves lie on at most two adjacent levels, and the leaves on the last level occupy the leftmost positions; all other levels are complete.
26

b) max-ordering: the value stored at a node is greater than or equal to the values stored at its children. A max-heap of size n can be constructed in linear time and can be stored in an n-element array; hence it is referred to as an implicit data structure [g]. When a max-heap implements a priority queue, FindMax can be performed in constant time, while both DeleteMax and Insert(x) have logarithmic time. We shall consider a more powerful data type, the double-ended priority queue, which allows both FindMin and FindMax, as well as DeleteMin, DeleteMax, and Insert(x) operations. An important application of this data type is in external quicksort . A traditional heap does not allow efficient implementation of all the above operations; for example, FindMin requires linear (instead of constant) time in a maxheap. One approach to overcoming this intrinsic limitation of heaps, is to place a max-heap back-toback with a min-heap as suggested by Williams. This leads to constant time Find either extremum and logarithmic time to Insert an element or Delete one of the extrema, but is somewhat trickier to implement than the method following.

MIN-MAX HEAPS Given a set S of values, a min-max heap on S is a binary tree T with the following properties: 1) T has the heap-shape 2) T is min-max ordered: values stored at nodes on even (odd) levels are smaller (greater) than or equal to the values stored at their descendants (if any) where the root is at level zero. Thus, the smallest value of S is stored at the root of T, whereas the largest value is stored at one of the roots children; an example of a min-max heap is shown in Figure 1 A min-max heap on n elements can be stored in an array A[1 . . . n]. The ith location in the array will correspond to a node located on level L(log,i)l in the heap. A max-min heap is defined analogously; in such a heap, the maximum value is stored at the root, and the smallest value is stored at one of the roots children. It is interesting to observe that the Hasse diagram for a min-max heap (i.e., the diagram representing the order relationships implicit within the structure) is rather complex in contrast with the one for a traditional heap (in this case, the Hasse diagram is the heap itself); Figure 2 (p. 998) shows the Hasse
27

diagram for the example of Figure 1. Algorithms processing min-max heaps are very similar to those corresponding to conventional heaps. Creating a min-max heap is accomplished by an adaption of Floyds [4] linear-time heap construction algorithm. Floyds algorithm builds a heap in a bottom-up fashion. When the algorithm examines the subtree rooted at A[i] then both subtrees of A[i] are maxordered, whereas the subtree itself may not necessarily be max-ordered. The TrickleDown step of his algorithm exchanges the value at A[i] with the maximum of its children. This step is then applied recursively to this maximum child to maintain the max-ordering. In min-max heaps, the required ordering must be established between an element, its children, and its grandchildren. The procedure must differentiate between min- and max-levels. The resulting description of this procedure follows: procedure TrickleDown - - i is the position in the array if i is on a min level then TrickleDownMin(i) else TrickleDownMax(i) endif procedure TrickleDownMin(i) if A[i] has children then m := index of smallest of the children and grandchildren (if any) of A[i] if A[m] is a grandchild of A[i] then if A[m] < A[i] then swap A[i] and A[m] if A[m] > A[parent(m)] then swap A[m] and A[parent(m)] endif TrickleDownMin(m) endif else {A[m] is a child of A[i]] if A[m] < A[i] then
28

swap A[i] and A[m] endif endif

The procedure TrickleDownMax is the same except that the relational operators are reversed. The operations DeleteMin and DeleteMax are analogous to deletion in conventional heaps. Specifically, the required element is extracted and the vacant position is filled with the last element of the heap. The minmax ordering is maintained after applying the TrickleDown procedure. An element is inserted by placing it into the first available leaf position and then reestablishing the ordering on the path from this element to the root.An efficient algorithm to insert an element can be designed by examining the Hasse diagram (recall Figure 2). The leaf-positions of the heap correspond to the nodes lying on the middle row in the Hasse diagram. To reestablish the min-max ordering, the
new element is placed into the next available leaf position, and must then move up the diagram toward the top, or down toward the bottom, to ensure that all paths running from top to bottom remain sorted. Thus the algorithm must first determine whether the new element should proceed further down the Hasse diagram (i.e., up the heap on maxlevels) or up the Hasse diagram (i.e., up the heap on successive min-levels). Once this has been determined, only grandparents along the path to the root of the heap need be examined-either those lying on min-levels or those lying on max-levels. The algorithms are as follows:

procedure BubbleUp - - i is the position in the array if i is on a min-level then

29

if i has a parent cand A[i] > A[parent(i)] then swap A[i] and A[parent(i)] BubbleUpMax(parent(i)) else BubbleUpMin(i) endif else if i has a parent cand A[i] < A[parent(i)] then swap A[i] and A[parent(i)] BubbleUpMin(parent(i)) else BubbleUpMax(i) endif endif procedure BubbleUpMin(i) if A[i] has grandparent then if A[i] < A[grandparent(i)] then swap A[i] and A[grandparent(i)] BubbleUpMin(grandparent(i)) endif endif The cand (conditional and) operator in the above code evaluates its second operand only when the first operand is true. BubbleUpMax is the same as BubbleUpMin except that the relational operators are reversed. From the similarity with the traditional heap algorithms, it is evident that the min-max heap algorithms will exhibit the same order of complexity (in terms of comparisons and data movements). The only difference rests with the actual constants: for construction and deletion the constant is slightly higher, and for insertion the constant is slightly lower. The value of the constant for each operation is summarized in Table I: the reader is referred to [2] for a detailed derivation of these values. All logarithms are base 2.

30

Slight improvements in the constants can be obtained by employing a technique similar to the one used by Gonnet and Munro [S] for traditional heaps. The resulting new values are shown in Table II; again, details of the derivation can be found in [2]. In Table II the function g(x) is defined as follows: g(x) = 0 for x< 1 and g(a) = g(log(n)l) + 1. Double Ended Priority Trees There are 2 double ended priority structures called min-max heap and deaps which perform

Insertion, Deletion of the minimum element and Deletion of the maximum element in o(logn) time; Reporting of the maximum and minimum element in constant time. Min-Max heap: It has the following properties:

1. It is a complete binary tree. 2. A node at an odd (respectively, even) level is called a minimum (respectively, maximum) node. If a node x is a minimum node (respectively, maximum) node then all elements in its subtrees have values larger (respectively, smaller) than x. x.

DEAPS
The deap is a doubly ended queue. A deap is a complete binary tree data structure, which is either empty or satisfies the following properties: 1) The root contains no element.

2) The left subtree is a minimum heap data structure. 3) The right subtree is a maximum heap data structure.

4) If the right subtree is not empty, then i is any node in the left (minimum) subtree and j is the corresponding node in the right (maximum) subtree. If such a j does not exists, then j is the node in the right subtree that corresponds to the parent of i. The key value in node i is less than or equal to that in node j. The nodes i and j are called partners.
31

Example of a Deap

-5

30

12 15 20

8 Node i

11

19

18 10 Node j

Corresponding nodes

Array Representation
1 -5 2 30 3 7 4 12 5 15 6 20 7 8 8 11 9 19 10 18 11 10

Properties of Deap Data Structure We see that the maximum and the minimum element can be determined in constant time . The element j corresponding to element i (described in property 4) is computed as follows: if element j exists, parent).
32

otherwise j is set to (j-1)/2. (which is its

For the value 8 (at index i=7) the j partner is at index 11 and contains the value 10 . For the value 11 at index i=8 the j partner is evaluated to be 12 , and since index 12 does not exist, the j partner is at index 5 ( its parent ), containing the value 15 . InsertionIntoDeap (Deap, maxElements, x) if Deap is not full maxElements = maxElements + 1; if maxElements belongs to maximum Heap of the Deap i = maxElements if x < Deap[i] Deap[maxElements] = Deap[i]; BubbleUpMin (Deap, maxElements, x) else BubbleUpMax (Deap, maxElements, x) else log maxElements-1 j = (maxElements + 2 )/2 if j > maxElements j = j / 2; if x > Deap[j] Deap[maxElements] = Deap[j] BubbleUpMax (Deap, maxElements, x) else BubbleUpMin (Deap, maxElements, x) DeleteMin (Deap) Output Deap[1] Deap[1] <-- Deap[maxElements] t <-- Deap[1] maxElements <-- maxElements 1 HeapifyMin (Deap, 1) i <-- index of value t in Deap j <-- maximum partner of i if (Deap[i] > Deap[j]) swap Deap[i] and Deap[j] bubbleUpMax (Deap, j, Deap[j] LEFTIST HEAPS Definition 1: The distance of a node m in a tree, denoted dist[m], is the length of the shortest path from m down to a descendant node that has at most one child.
33

Definition 2: A leftist heap is a binary tree such that for every node m, (a) key[m] key[lchild[m]] and key(m) key[rchild[m]], and (b) dist[lchild[m]] dist[rchild[m]]. In the above definition, key[m] is the key stored at node m. We assume that there is a total ordering among the keys. We also assume that key[nil] = and dist[nil] = 1. Definition 3: The right path of a tree is the path m1, m2, . . . , mk where m1 is the root of the tree, mi+1 = rchild[mi] for 1 i < k, and rchild[mk] = nil. Figure 1 below shows two examples of leftist heaps. Here are a few simple results about leftist heaps that you should be able to prove easily: Fact 1: The left and right subtrees of a leftist heap are leftist heaps. Fact 2: The distance of a leftist heaps root is equal to the length of the trees right path. Fact 3: For any node m of a leftist heap, dist[m] = dist[rchild[m]] +1 (where, as usual, we take dist[nil] = 1).

Figure 1. In each node we record its key at the top half and its distance at the bottom half. The right path of T1 is 1, 5, 8 while the right path of T2 is 1.
34

In the examples above, T2 illustrates the fact that leftist heaps can be unbalanced. However, in INSERT, DELETEMIN and UNION, all activity takes place along the right path which, the following theorem shows, is short. Theorem: If the length of the right path in a leftist tree T is k then T has at least 2k+1 1 nodes. Proof: By induction on the height h of T. Basis (h=0): Then T consists of a single node and its right path has length k=0. Indeed, T has 1 21 1 nodes, as wanted. Inductive Step (h>0): Suppose the theorem holds for all leftist heaps that have height < h and let T be a leftist heap of height h. Further, let k be the length of Ts right path and n be the number of nodes in T. Consider two cases: Case 1: k=0 (i.e. T has no right subtree). But then clearly n 1=21 1, as wanted. Case 2: k>0. Let TL, TR be the left and right subtrees of T; nL, nR be the number of nodes in TL and TR; and kL, kR be the lengths of the right paths in TL and TR respectively. By Fact 1, TR and TL are both leftist heaps. By Facts 2 and 3, kR = k 1, and by definition of leftist tree kL kR. Since TL, TR have height < h we get, by induction hypothesis, nR 2k 1 and nL 2k 1. But n = nL + nR +1 and thus, n 2k 1 + 2k 1 +1=2k+1 1. Therefore n 2k+1 1, as wanted. From this we immediately get Corollary: The right path of a leftist heap with n nodes has length log(n +1) 1. Now lets examine the algorithm for joining two leftist heaps. The idea is simple: if one of the two trees is empty were done; otherwise we want to join two non-empty trees T1 and T2 and we can assume, without loss of generality, that the key in the root of T1 is the key in the root of T2. Recursively we join T2 with the right subtree of T1 and we make the resulting leftist heap into the right subtree of T1. If this has made the distance of the right subtrees root longer than the distance of the left subtrees root, we simply interchange the left and right children of T1s root (thereby making what used to be the right subtree of T1 into its left subtree and vice-versa). Finally, we update the distance of T1s root. The following pseudo-code gives more details. We assume that each node of the leftist heap is represented as a record with the following format

where the fields have the obvious meanings. A leftist heap is specified by giving a pointer
35

to its root. /* The following algorithm joins two leftist heaps whose roots are pointed at by r1 and r2, and returns a pointer to the root of the resulting leftist heap. */ function UNION (r1 , r2) if r1 = nil then return r2 else if r2 = nil then return r1 else if key[r1] > key[r2] then r1 r2 rchild[r1] UNION ( rchild[r1] , r2 ) if d(rchild[r1]) > d(lchild[r1]) then rchild[r1] lchild[r1] dist[r1]d(rchild[r1]) +1 return r1 end {UNION} function d(x) /* returns dist(x) */ if x = nil then return 1 else return dist[x] end {d} What is the complexity of this algorithm? First, observe that there is a constant number of steps that must be executed before and after each recursive call to UNION. Thus the complexity of the algorithm is proportional to the number of recursive calls to UNION. It is easy to see that, in the worst case, this will be equal to p1 + p2 where p1 (respectively p2) is 1 plus the length of the right path of the leftist heap whose root is pointed at by r1 (respectively r2). Let the number of nodes in these trees be n1, n2. By the above Corollary we have p1 log (n1 +1) , p2 log (n2 +1) . Thus p1 + p2 log n1 + log n2 + 2. Let n=max(n1, n2). Then p1 + p2 2 log n + 2. Therefore, UNION is called at most 2 log n +2 times and the complexity of the algorithm is O(log (max(n1, n2))) in the worst case. Figure 2 below shows an example of the UNION operation. Armed with the UNION algorithm we can easily write algorithms for INSERT and DELETEMIN:

36

Figure 2. The UNION operation. INSERT(e, r) {e is an element, r is pointer to root of tree} 1. Let r be a pointer to the leftist heap containing only e 2. return UNION (r , r). DELETEMIN(r) 1. min element stored at r (root of leftist heap) 2. r UNION(lchild[r] , rchild[r]) 3. return min. By our analysis of the worst case time complexity of UNION it follows immediately that the complexity of both these algorithms is O(log n) in the worst case, where n is the number of nodes in the leftist heap. In closing, we note that INSERT can be written as in the heap representation of priority queues, by adding the new node at the end of the right path, percolating its value up (if necessary), and switching right and left children of some nodes (if necessary) to maintain the properties of the leftist heap after the insertion. As an exercise, write the INSERT algorithm for leftist heaps in this fashion. On the
37

other hand, we cannot use the idea of percolating values down to implement DELETEMIN in leftist heaps the way we did in standard heaps: doing so would result in an algorithm with O(n) worst case complexity. As an exercise, construct a leftist heap where this worst case behaviour would occur.

Binomial Heaps
The binomial tree is the building block for the binomial heap. A binomial tree is an ordered tree - that is, a tree where the children of each node are ordered. Binomial trees are dened recursively, building up from single nodes. A single tree of degree k is constructed from two trees of degree k 1 by making the root of one tree the leftmost child of the root of the other tree. This process is shown in Figure 1. A binomial heap H consists of a set of binomial trees. Such a set is a binomial heap if it satisfies the following properties: 1. For each binomial tree T in H, the key of every node in T is greater than or equal to the key of its parent. 2. For any integer k >= 0, there is no more than one tree in H whose root has degree k. The algorithms presented later work on a particular representation of a binomial heap. Within the heap, each node stores a pointer to its leftmost child (if any) and its rightmost sibling (if any). The heap itself is a linked list of the roots of its constituent trees, sorted by ascending number of children. The following data is maintained for each non-root node x: 1. key[x] - the criterion by which nodes are ordered, 2. parent[x] - a pointer to the node's parent, or NIL if the node is a root node. 3. child[x] - a pointer to the node's leftmost child, or NIL if the node is childless, 4. sibling[x] - a pointer to the sibling immediately to the right of the node, or NIL if the node has no siblings, 5. degree[x] - the number of children of x.

Figure 1: The production of a binomial tree. A shows two binomial trees of degree 0. B Shows the two trees combined into a degree-1 tree. C shows two degree-1 trees combined into a degree-2 tree. Binomial Heap Algorithms
38

Creation A binomial heap is created with the Make-Binomial-Heap function, shown below. The Allocate-Heap procedure is used to obtain memory for the new heap. Make-Binomial-Heap() 1 H <--- Allocate-Heap() 2 head[H] <--- NIL 3 return H Finding the Minimum To find the minimum of a binomial heap, it is helpful to refer back to the binomial heap properties. Property one implies that the minimum node must be a root node. Thus, all that is needed is a loop over the list of roots. Binomial-Heap-Minimum(Heap) 1 best node <--- head[Heap] 2 current <--- sibling[best node] 3 while current != NIL 4 do if key[current] < key[best node] 5 best node <--- current 6 return best node Unifying two heaps The rest of the binomial heap algorithms are written in terms of heap unification. In this section, the unification algorithm is developed. Conceptually, the algorithm consists of two parts. The heaps are first joined together into one data structure, and then this structure is is manipulated into satisfying the binomial heap properties. To address the second phase first, consider two binomial heaps H1 and H2, which are to be merged into H = H1 U H2. Both H1 and H2 obey the binomial heap properties, so in each of them, there is at most one tree whose root has degree k, for k >= 0. In H, however, there may be up to two such trees. To recover the second binomial heap property, such duplicates must be merged. This merging process may result in additional work: when merging a root of degree m with one of degree n, the operation involves adding one as a child of the other - this creates a root with degree p, where p = n + 1 or p = m + 1. However, it is perfectly possible for there to already be a node with degree p, and so another merge is needed. This second merge has the same problem. If root nodes are considered in some arbitrary order, then after every merge, the entire list must be rechecked in case a new conflict has arisen. However, by requiring the list of roots to be in a monotonically increasing order, it is possible to scan through it in a linear fashion. This restriction is enforced by the auxiliary routine Binomial-Heap-Merge: Binomial-Heap-Merge(H1;H2) 1 H <--- Make-Binomial-Heap() 2 if key[head[H2]] < key[head[H1]]
39

3 then head[H] <--- head[H2] 4 current2 <--- sibling[head[H2]] 5 current1 <--- head[H1] 6 else head[H] <--- head[H1] 7 current1 <--- sibling[head[H1]] 8 current2 <--- head[H2] 9 current <--- head[H] 10 while current1 != NIL and current2 != NIL 11 do if key[current1] > key[current2] 12 then sibling[current] <--- current2 13 current <--- sibling[current] 14 current2 <--- sibling[current2] 15 else 16 sibling[current] <--- current1 17 current <--- sibling[current] 18 current1 <--- sibling[current1] 19 if current1 = NIL 20 then tail <--- current2 21 else tail <--- current1 22 while tail != NIL 23 do sibling[current] <--- tail 24 current <--- sibling[current] 25 tail <--- sibling[tail] 26 return head[H]

This routine starts by creating and initialising a new heap, on lines 1 through 9. The code maintains three pointers. The pointer current, stores the root of the tree most recently added to the heap. For the two input heaps, current1 and current2 record their next unprocessed root nodes. In the while loop, these pointers are used to add trees to the new heap, while maintaining the desired monotonic ordering within the resulting list. Finally, the case where the two heaps have diering numbers of trees must be handled - this is done on lines 19 through 25. Before the whole algorithm is given, one more helper routine is needed. The Binomial-Link routine joins two trees of equal degree: Binomial-Link(Root;Branch) 1 parent[Branch] = Root 2 sibling[Branch] = child[Root]
40

3 child[Root] = Branch 4 degree[Root] = degree[Root] + 1 And now, the full algorithm: Binomial-Heap-Unify(Heap1,Heap2) 1 head[Final Heap] <--- Binomial-Heap-Merge(Heap1,Heap2) 2 if head[Final Heap] = NIL 3 then return Final Heap 4 previous <--- NIL 5 current <--- head[Final_Heap] 6 next <--- sibling[current] 7 while next != NIL 8 do need merge <--- TRUE 9 if (degree[current] != degree[next]) 10 then need merge <--- FALSE 11 if (sibling[next] !=NIL and 12 degree[sibling[next]] = degree[next]) 13 then need merge <--- FALSE 14 if (need merge) 15 then if (key[current] key[next] 16 then sibling[current] <--- sibling[next] 17 Binomial-Link(current, next) 18 else if (previous !=NIL) 19 then sibling[previous] <--- next 20 Binomial-Link(next,current) 21 else head[Final Heap] <--- next 22 Binomial-Link(next,current) 23 else previous <--- current 24 current <--- next 25 next <--- sibling[current] 26 return Final_Heap The first line creates a new heap, and populates it with the contents of the old heaps. At this point, all the data are in place, but the heap properties (which are relied on by other heap algorithms) may not hold. The remaining lines restore these properties. The first property applies to individual trees, and so is preserved by the merging operation. As long as Binomial-Link is called with the arguments in the correct order, the first property will never be violated. The second property is restored by repeatedly merging trees whose roots have the same degree.
41

Insertion

To insert an element x into a heap H, simply create a new heap containing x and unify it with H: Binomial-Heap-Insert(Heap,Element) 1 New <--- Make-Binomial-Heap() 2 head[New] <--- Element 3 parent[New] <--- Element 4 sibling[New] <--- NIL 5 child[New] <--- NIL 6 degree[New] <--- 1 7 Binomial-Heap-Unify(Heap,New) Extracting the Minimum Extracting the smallest element from a binomial heap is fairly simple, due to the recursive manner in which binomial trees are constructed. Binomial-Heap-Extract-Min(Heap) 1 min <--- Binomial-Heap-Minimum(Heap) 2 rootlist <--- null ; 3 current <--- child[min] 4 while current != NIL 5 parent[current] <--- NIL 6 rootlist <--- current + rootlist 7 new <--- Make-Binomial-Heap() 8 head[new] <--- rootlist[0] 9 Heap <--- Binomial-Heap-Unify(Heap;new) 10 return min The only subtlety in the above pseudocode is on line six, where the next element is added to the front of the list. This is because, within a heap, the list of roots is ordered by increasing degree. (This assumption is behind, for example, the implementation of the Binomial-Heap-Merge algorithm.) However, when a binomial tree is built, the children will be ordered by decreasing degree. Thus, it is necessary to reverse the list of children when said children are promoted to roots. Decreasing a key Decreasing the key of a node in a binomial heap is also simple. The required node has its key adjusted, and is then moved up through the tree until it is no less than its parent, thus ensuring the resulting structure is still a binomial heap. Binomial-Heap-Decrease-Key(Heap; item; key) 1 key[item] <--- key
42

2 current <--- item 3 while parent[current] != NIL and key[parent[current]] > key[current] 4 tmp <--- data[current] 5 data[current] <--- data[parent[current]] 6 data[parent[current]] <--- tmp 7 current <--- parent[current] Deletion Deletion is simple, given the routines already discussed: Binomial-Heap-Delete(Heap, item) 1 min <--- Binomial-Heap-Minimum(Heap) 2 Binomial-Heap-Decrease-Key(Heap,item,min - 1) 3 Binomial-Heap-Extract-Min(Heap)

FIBONACCI HEAPS
Introduction Priority queues are a classic topic in theoretical computer science. The search for a fast priority queue implementation is motivated primarily by two net- work optimization algorithms: Shortest Path (SP) and Minimum Spanning Tree (MST), i.e., the connector problem. As we shall see, Fibonacci Heaps provide a fast and elegant solution. The following 3-step procedure shows that both Dijkstras SP-algorithm or Prims MST-algorithm can be implemented using a priority queue: 1. Maintain a priority queue on the vertices V (G). 2. Put s in the queue, where s is the start vertex (Shortest Path) or any vertex (MST). Give s a key of 0. Add all other vertices and set their key to infinity. 3. Repeatedly delete the minimum-key vertex v from the queue and mark it scanned. For each neighbor w of v do: If w is not scanned (so far), decrease its key to the minimum of the value calculated below and ws current key: SP: key(v) + length(vw), MST: weight(vw). The classical answer to the problem of maintaining a priority queue on the vertices is to use a binary heap, often just called a heap. Heaps are commonly used because they have good bounds on the time required for the following operations: insert O(log n), delete-min O(log n), and decrease-key O(log n), where n reflects the number of elements in the heap. If a graph has n vertices and e edges, then running either Prims or Dijkstras algorithms will require O(n log n) time for inserts and deletes. However, in the worst case, we will also perform e decrease-keys, because we may have to perform a
43

key update every time we come across a new edge. This will take O(e log n) time. Since the graph is connected, e n, and the overall time bound is given by O(e log n). As we shall see, Fibonacci heaps allow us to do much better. Definition and Elementary Operations The Fibonacci heap data structure invented by Fredman and Tarjan in 1984 gives a very efficient implementation of the priority queues. Since the goal is to find a way to minimize the number of operations needed to compute the MST or SP, the kind of operations that we are interested in are insert, decrease-key, link, and delete-min (we have not covered why link is a useful operation yet, but this will become clear later on). The method to achieve this minimization goal is laziness - do work only when you must, and then use it to simplify the structure as much as possible so that your future work is easy. This way, the user is forced to do many cheap operations in order to make the data structure complicated. Fibonacci heaps make use of heap-ordered trees. A heap-ordered tree is one that maintains the heap property, that is, where key(parent) key(child) for all nodes in the tree. Definition: A Fibonacci heap H is a collection of heap-ordered trees that have the following properties: 1. The roots of these trees are kept in a doubly-linked list (the root list of H), 2. The root of each tree contains the minimum element in that tree (this follows from being a heapordered tree), 3. We access the heap by a pointer to the tree root with the overall minimum key, 4. For each node x, we keep track of the degree (also known as the order or rank) of x, which is just the number of children x has; we also keep track of the mark of x, which is a Boolean value whose role will be explained later.

Fig.: A detailed view of a Fibonacci Heap. Null pointers are omitted for clarity.

For each node, we have at most four pointers that respectively point to the nodes parent, to one of its children, and to two of its siblings. The sibling pointers are arranged in a doubly-linked list (the child list of the parent node). We have not described how the operations on Fibonacci heaps are implemented, and their implementation will add some additional properties to H. The following are some elementary operations used in maintaining Fibonacci heaps:
44

Inserting a node x: We create a new tree containing only x and insert it into the root list of H; this is clearly an O(1) operation. Linking two trees x and y: Let x and y be the roots of the two trees we want to link; then if key(x) key(y), we make x the child of y; otherwise, we make y the child of x. We update the appropriate nodes degrees and the appropriate child list; this takes O(1) operations. Cutting a node x: If x is a root in H, we are done. If x is not a root in H, we remove x from the child list of its parent, and insert it into the root list of H, updating the appropriate variables (the degree of the parent of x is decremented, etc.). Again, this takes O(1) operations. We assume that when we want to cut/find a node, we have a pointer hanging around that accesses it directly, so actually finding the node takes O(1) time.

Fig. The Cleanup algorithm executed after performing a delete-min Marking a node x: We say that x is marked if its mark is set to true, and that it is unmarked if its mark is set to false. A root is always unmarked. We mark x if it is not a root and it loses a child (i.e., one of its children is cut and put into the root-list). We unmark x whenever it becomes a root. We shall see later on that no marked node will lose a second child before it is cut itself. The delete-min Operation Deleting the minimum key node is a little more complicated. First, we remove the minimum key from the root list and splice its children into the root list. Except for updating the parent pointers, this takes O(1) time. Then we scan through the root list to find the new smallest key and update the parent pointers of the new roots. This scan could take O(n) time in the worst case. To bring down the amortized deletion time (see further on), we apply a Cleanup algorithm, which links trees of equal degree until there is only one root node of any particular degree. Let us describe the Cleanup algorithm in more detail. This algorithm maintains a global array B[1 . . . log n], where B[i] is a pointer to some previously-visited root node of degree i, or Null if there is no such previously- visited root node. Notice, the Cleanup algorithm simultaneously resets the parent pointers of all the new roots and updates the pointer to the minimum key. The part of the algorithm that links possible nodes of equal degree is given in a separate subroutine LinkDupes, see Figure The subroutine

45

Fig.: The Promote algorithm ensures that no earlier root node has the same degree as the current. By the possible swapping of the nodes v and w, we maintain the heap property. We shall analyze the efficiency of the delete-min operation further on. The fact that the array B needs at most log n entries is proven in Section , where we prove that the degree of any (root) node in an n-node Fibonacci heap is bounded by log n. The decrease-key Operation If we also need the ability to delete an arbitrary node. The usual way to do this is to decrease the nodes key to and then use delete-min. We start by describing how to decrease the key of a node in a Fibonacci heap; the algorithm will take O(log n) time in the worst case, but the amortized time will be only O(1). Our algorithm for decreasing the key at a node v follows two simple rules:
1. If newkey(v) < key(parent(v)), promote v up to the root list (this moves the whole subtree

rooted at v). 2. As soon as two children of any node w have been promoted, immediately promote w. In order to enforce the second rule, we now mark certain nodes in the Fibonacci heap. Specifically, a node is marked if exactly one of its children has been promoted. If some child of a marked node is promoted, we promote (and unmark) that node as well. Whenever we promote a marked node, we unmark it; this is the only way to unmark a node (if splicing nodes into the root list during a deletemin is not considered a promotion). A more formal description of the Promote algorithm is given in Figure. This algorithm is executed if the new key of the node v is smaller than its parents key. SKEW HEAP Skew heaps are one of the possible implementations of priority queues. A skew heap is a selfadjusting form of a leftist heap, which may grow arbitrarily unbalanced because they do not maintain balancing information. Definition Skew Heaps are a self-adjusting form of Leftist Heap. By unconditionally swapping all nodes in the merge path Skew Heaps attempt to maintain a short right path from the root. The following diagram is a graphical representation of a Skew Heap.

46

From the above diagram it can be seen that a Skew Heap can be heavy at times on the right side of the tree. Depending on the order of operations Skew Heaps can have long or short, right hand side path lengths. As the following diagram shows by inserting the element 45 into the above heap a short right hand side is achieved:

Algorithm Skew Heaps are of interest as they do not maintain any balancing information but still can achieve amortized log n time in the Union Operation. The following operations can be executed on Skew Heaps: 1. MakeHeap (Element e) 2. FindMin (Heap h) 3. DeleteMin (Heap h) 4. Insert (Heap h, Element e) 5. Union (Heap h1, Heap h2) The only difference between a skew heap and a leftist heap is the union operation is treated differently in skew heaps. The swapping of the children of a visited node on the right path is performed unconditionally; the dist value is not maintained.
47

The purpose of the swapping is to keep the length of the right path bounded, even though the length of the right path can grow to Omega (n), it is quite effective. The reasoning behind this is that insertions are made on the right side and therefore creating a heavy right side. Then by swapping everything unconditionally a relatively light right side is created. So, the good behavior of skew heaps is due to always inserting to the right and unconditionally swapping all nodes. Refer to Bibliography for reference to Proof. Note that the operation Union is used for Inserts and DeleteMin as specified below. MakeHeap (Element e)
return new Heap(e);

FindMin(Heap h)
if (h == null) return null; else return h.key

DeleteMin(Heap h)
Element e = h.key; h = Union (h.left, h.right); return e;

Insert (Heap h, Element e)

z = MakeHeap(e); h = Union (h, z);

Union (Heap h1, heap h2)

Heap dummy; if (h1 == null) return h2;

else if (h2 == null) return h1; else { // Assure that the key of h1 is smallest if (h1.key > h2.key){ Node dummy = h1; h1 = h2; h2 = dummy; }} if (h1.right == null) // Hook h2 directly to h1
48

h1.right = h2; else // Union recursively h1.right = Union (h1.right, h2); // Swap children of h1 dummy = h1.right; h1.right = h1.left; h1.left = dummy; return h1; Applications Because of the self-adjusting nature of this Heap the Union Operation has sufficiently enhanced and runs in O (log n) amortized time. In some applications, pairs of heaps are repeatedly unioned; spending linear time per union operation would be out of the question. Skew Heap supports union in O (log n) amortized time and therefore are a good candidate for applications that will repeatedly call the Union operation. The self-adjustment in skew heaps has important advantages over the balancing in leftist heaps: Reduction of memory consumption (no balancing information needs to be stored) Reduction of time consumption (saving the updates of the dist information) Unconditional execution (saving one test for every processed node Binomial heaps with Lazy Meld Definition A lazy meld binomial heap is a collection of heap-ordered binomial trees B(i)s such that when n items are stored in the heap, the largest tree is B((log n)); the reason is that, as we have seen before, each B(i) has 2i nodes. Thus since a B(i) has no empty node, and no value is duplicated, in order to store n values, we need n nodes. Thus the biggest tree with n nodes is a B(log n). This kind of heap is implemented as a doubly linked circular list of binomial trees. It is then possible to have multiple B(i)s for the same i in a single heap. This increases the cost of the previously defined delete-min operation, since it has to find the new minimum value among a bigger number of trees than before. We thus have to redefine delete-min so as to make sure that after a delete-min operation, there is at most one tree per rank again in the heap. Operations on binomial heaps with lazy meld Definition . Insert(h, i) = meld(h, make-heap(i)). As you can see, this kind of heap is very lazy at insertion. Algorithm Meld Require: h, h Ensure: h = meld(h, h)
49

concatenate list of h and h update min pointer This algorithm runs in O(1) time. Algorithm Delete-min Require: h Ensure: h = delete-min(h) delete the min from its binomial tree and break this binomial tree into a heap h create array of ceil((log n)) empty pointers for all trees in h and h do insert the tree B(i) into the array at position i, linking and carrying if necessary so as to have at most one tree per rank compare root of inserted tree to min pointer and update min pointer suitably. end for Example Sequence of insert and one delete-min If we consider the following sequence of insert : insert(1), insert(3), insert(5), this will create the following heap h, made of doubly linked list with three B(0), with the min pointer on the first B(0) tree :

Then a delete-min will first remove the B(0) which has the min pointer on it. Since this tree is a B(0), then no new heap is made out of its other nodes. Then an array of empty pointers of size n is created. Now the first tree of h is inserted in the array, at the position 0, since it is a B(0). Then the same is done for the second tree of h. We want to place it at the position 0 of the array, but since it already contains a tree, then a link is performed, creating a B(1). Thus the resulting heap is

Amortized analysis The amortized analysis with bankers view is identical as before, except that we place now 2 c coins at each new

50

trees root, with c defined below. The amortized analysis with physicist view is as follows : the potential function is

= 2c trees in doubly linked list, where c is a suitable constant chosen below. Now we have the following amortized costs : Meld : amortized cost = O(1) + 0 = 0(1) Insert : amortized cost = O(1) + 2c = 0(1) Delete : let m be the number of trees of h and h right after the step 1 of the algorithm; let k be the number of link operations in the step 3. Then at the end of the algorithm, there are log n trees, since there is at most one tree per rank, and the biggest one has rank (log n). Since each link operation removes one tree, then m k log n. let c be a constant such that the acutal cost is c(m + k). Since m log n + k, then acutal cost c (log n + 2k) = c log n + 2ck. This is the c we use in the potential function.

the change in is as follows : the heap h containging at most log n trees, it adds at most log n trees. As each link removes one tree, then the change is 2c(log n k) = 2c log n 2ck. thus the amortized cost is c log n + 2ck + 2c log n 2ck = 3c log n. Summary

51

UNIT III

SEARCH STRUCTURES

BinarySearchTree: An Unbalanced Binary Search Tree A BinarySearchTree is a special kind of binary tree in which each node, u, also stores a data value, u:x, from some total order. The data values in a binary search tree obey the binary search tree property: For a node, u, every data value stored in the subtree rooted at u:left is less than u:x and every data value stored in the subtree rooted at u:right is greater than u:x. An example of a BinarySearchTree is shown in Figure Searching The binary search tree property is extremely useful because it allows us to quickly locate a value, x, in a binary search tree. To do this we start searching for x at the root, r. When examining a node, u, there are three cases:

52

Figure : A binary search tree 1. If x < u:x then the search proceeds to u:left; 2. If x > u:x then the search proceeds to u:right; 3. If x = u:x then we have found the node u containing x. The search terminates when Case 3 occurs or when u = nil. In the former case, we found x. In the latter case, we conclude that x is not in the binary search tree. BinarySearchTree T findEQ(T x) { Node *w = r; while (w != nil) { int comp = compare(x, w->x); if (comp < 0) { w = w->left; } else if (comp > 0) { w = w->right; } else { return w->x; } } return null; } Two examples of searches in a binary search tree are shown in Figure . As the second example shows, even if we dont find x in the tree, we still gain some valuable information. If we look at the last node, u, at which Case 1 occurred, we see that u:x is the smallest value in the tree that is greater than x. Similarly, the last node at which Case 2

53

Figure : An example of (a) a successful search (for 6) and (b) an unsuccessful search (for 10) in a binary search tree. occurred contains the largest value in the tree that is less than x. Therefore, by keeping track of the last node, z, at which Case 1 occurs, a BinarySearchTree can implement the find(x) operation that returns the smallest value stored in the tree that is greater than or equal to x BinarySearchTree T find(T x) { Node *w = r, *z = nil; while (w != nil) { int comp = compare(x, w->x); if (comp < 0) { z = w; w = w->left; } else if (comp > 0) { w = w->right; } else { return w->x; } } return z == nil ? null : z->x; } Addition To add a new value, x, to a BinarySearchTree, we first search for x. If we find it, then there is no need to insert it. Otherwise, we store x at a leaf child of the last node, p, encountered during the search for x. Whether the new node is the left or right child of p depends on the result of comparing x and p:x. BinarySearchTree bool add(T x) { Node *p = findLast(x); Node *u = new Node; u->x = x; return addChild(p, u); } BinarySearchTree
54

Node* findLast(T x) {

Node *w = r, *prev = nil; while (w != nil) { prev = w; int comp = compare(x, w->x); if (comp < 0) { w = w->left; } else if (comp > 0) { w = w->right; } else { return w; } } return prev; } BinarySearchTree bool addChild(Node *p, Node *u) { if (p == nil) { r = u; // inserting into empty tree } else { int comp = compare(u->x, p->x); if (comp < 0) { p->left = u; } else if (comp > 0) { p->right = u; } else { return false; // u.x is already in the tree } u->parent = p; } n++; return true; }

Figure : Inserting the value 8:5 into a binary search tree An example is shown in Figure . The most time-consuming part of this process is the initial search for x, which takes time proportional to the height of the newly added node u. In the worst case, this is equal to the height of the BinarySearchTree.
55

Removal Deleting a value stored in a node, u, of a BinarySearchTree is a little more difficult. If u is a leaf, then we can just detach u from its parent. Even better: If u has only one child, then we can splice u from the tree by having u:parent adopt us child (see Figure ): BinarySearchTree void splice(Node *u) { Node *s, *p; if (u->left != nil) { s = u->left; } else { s = u->right; } if (u == r) { r = s; p = nil; } else { p = u->parent; if (p->left == u) { p->left = s; } else { p->right = s; } } if (s != nil) { s->parent = p; } n--; } Things get tricky, though, when u has two children. In this case, the simplest thing to do is to find a node, w, that has less than two children such that we can replace u:x with w:x. To maintain the binary search tree property, the value w:x should be close to the value of u:x. For example, picking w such that w:x is the smallest value greater than u:x will do. Finding the node w is easy; it is the smallest value in the subtree rooted at u:right. This node can be easily removed because it has no left child. (See Figure ) BinarySearchTree void remove(Node *u) { if (u->left == nil || u->right == nil) { splice(u); delete u; } else { Node *w = u->right; while (w->left != nil) w = w->left; u->x = w->x;
56

splice(w); delete w; } } Summary The find(x), add(x), and remove(x) operations in a BinarySearchTree each involve following a path from the root of the tree to some node in the tree. Without knowing more about the shape of the tree it is difficult to say much about the length of this path, except that

Figure : Deleting a value (11) from a node, u, with two children is done by replacing us value with the smallest value in the right subtree of u. it is less than n, the number of nodes in the tree. The following (unimpressive) theorem summarizes the performance of the BinarySearchTree data structure: Theorem . A BinarySearchTree implements the SSet interface. A BinarySearchTree supports the operations add(x), remove(x), and find(x) in O(n) time per operation. Theorem compares poorly with Theorem 4.1, which shows that the SkiplistSSet structure can implement the SSet interface with O(logn) expected time per operation. The problem with the BinarySearchTree structure is that it can become unbalanced. Instead of looking like the tree in Figure it can look like a long chain of n nodes, all but the last having exactly one child. There are a number of ways of avoiding unbalanced binary search trees, all of which lead to data structures that have O(logn) time operations.

AVL trees
We now define the special balance property we maintain for an AVL tree. Definition . (1) A vertex of a tree is balanced if the heights of its children differ by one at most.
(2) An AVL tree is a binary search tree in which all vertices are balanced. The binary search tree

in Figure is not an AVL tree, because the vertex with key 16 is not balanced. The tree in Figure is an AVL tree.

57

Figure . An AVL tree. We now come to our key result. Theorem. The height of an AVL tree storing n items is O(lg(n)). PROOF For h N, let n(h) be the minimum number of items stored in an AVL-tree of height h. Observe that n(1) = 1, n(2) = 2, and n(3) = 4. Our first step is to prove, by induction on h, that for all h 1 we have n(h) > 2h/2 1. As the induction bases, we note that (5.1) holds for h = 1 and h = 2 because n(1) = 1 > 2 1 and n(2) = 2 > 21 1. For the induction step, we suppose that h 3 and that (5.1) holds for h 1 and h 2. We observe that by Defn 5.4, we have n(h) 1 + n(h 1) + n(h 2), since one of the two subtrees must have height at least h1 and the other height at least h 2. (We are also using the fact that if h1 h2 then n(h1) n(h2), this is very easy to prove and you should do this.) By the inductive hypothesis, this yields

The last line follows since 21/2+ 21 > 1. This can be seen without explicit calculations as it is equivalent to 2-1/2 > 1/2 which is equivalent to 2 1/2< 2 and this now follows by squaring. This completes the proof of .
58

Therefore for every tree of height h storing n items we have n n(h) > 2h/2 1. Thus h/2 < lg(n + 1) and so h < 2 lg(n + 1) = O(lg(n)). This completes the proof of the Theorem. It follows from our discussion of findElement that we can find a key in an AVL tree storing n items in time O(lg(n)) in the worst case. However, we cannot just insert items into (or remove items from) an AVL tree in a naive fashion, as the resulting tree may not be an AVL tree. So we need to devise appropriateinsertion and removal methods that will maintain the balance properties set out in Definition. Insertions Suppose we want to insert an item (k, e) (a key-element pair) into an AVL tree. We start with the usual insertion method for binary search trees. This method is to search for the key k in the existing tree using the procedure in lines 17 of findElement(k), and use the result of this search to find the right leaf location for an item with key k. If we find k in an internal vertex, we must walk down to the largest near-leaf in key value which is no greater than k, and then use the appropriate neighbour leaf. In effect we ignore the fact that an occurrence of k has been found and carry on with the search till we get to a vertex v after which the search takes us to a leaf l (see Algorithm ). The vertex v is the largest near-leaf. We make a new internal vertex u to replace the leaf l and store the item there (setting u.key = k, and u.elt = e). The updating of u from a leaf vertex to an internal vertex (which will have two empty child vertices, as usual) will sometimes cause the tree to become unbalanced (no longer satisfying Definition ), and in this case we will have to repair it. Clearly, any newly unbalanced vertex that has arisen as a result of inserting into u must be on the path from the new vertex to the root of the tree. Let z be the unbalanced vertex of minimum height. Then the heights of the two children of z must differ by 2. Let y be the child of z of greater height and let x be the child of y of greater height. If both children of y had the same height, then z would already have been unbalanced before the insertion, which is impossible, because before the insertion the tree was an AVL tree. Note that the newly added vertex might be x itself, or it might be located in the subtree rooted at x. Let V and W be the two subtrees rooted at the children of x. Let X be the subtree rooted at the sibling of x and let Y be the subtree rooted at the sibling of y. Thus we are in one of the situations displayed in Figures . Now we apply the operation that leads to part (b) of the respective figure. These operations are called rotations; consideration of the figures shows why. By applying the rotation we balance vertex z. Its descendants (i.e., the vertices below z in the tree before rotation) remain balanced. To see this, we make the following observations about the heights of the subtrees V,W,X, Y :
height(V ) 1 height(W) height(V ) + 1 (because x is balanced). In the Figures, we have always

assumed that W is the higher tree, but it does not make a difference. max{height(V ), height(W)} = height(X) (as y is balanced and height(x) >height(X)).

59

Figure . A clockwise single rotation

Figure . An anti-clockwise single rotation max{height(V ), height(W)} = height(Y ) (because height(Y ) = height(y) 2). The rotation reduces the height of z by 1, which means that it is the same as it was before the insertion. Thus all other vertices of the tree are also balanced. Algorithm overleaf summarises insertItem(k, e) for an AVL-tree. In order to implement this algorithm efficiently, each vertex must not only store references to its children, but also to its parent. In addition, each vertex stores the height of its left subtree minus the height of its right subtree (this may be 1, 0, or 1). We now discuss TinsertItem(n), the worst-case running time of insertItem on a AVL tree of size n. Let h denote the height of the tree. Line 1 of Algorithm requires time O(h), and line 2 just O(1) time. Line 3 also requires time O(h), because in the worst case one has to traverse a path from a leaf to the root. Lines 4.-6. only require constant time, because all that needs to be done is redirect a few references to subtrees. By Theorem , we have h O(lg(n)). Thus the overall asymptotic running time of Algorithm is O(lg(n)).
60

Figure. An anti-clockwise clockwise double rotation

Figure . A clockwise anti-clockwise double rotation Removals Removals are handled similarly to insertions. Suppose we want to remove an item with key k from an AVL-tree. We start by using the basic removeItem method for binary search trees. This means that we start by performing steps 1 7 of findElement(k) hoping to arrive at some vertex t such that t.key = k. If we achieve this, then we will delete t (and return t.elt), and replace t with the closest (in key-value) vertex u (u is a near-leaf vertex as beforenote that this does not imply both us children are leaves, but one will be). We can find u by walking down from t to the near-leaf with largest key value no greater than k. Then t gets us item and us height drops by 1. After this deletion and re-arrangement, any newly unbalanced vertex must be on the path from u to the root of the tree. In fact, there can be at most one unbalanced vertex (why?). Let z be this unbalanced vertex. Then the heights of the two children of z must differ by 2. Let y be the child of z of greater height and x the child of y of greater height. If both children of y have the same height, let x be the left child of y if y is the left child of z, and let x be the right child of y if y is the right child of
61

z. Now we apply the appropriate rotation and obtain a tree where z is balanced and where x, y remain balanced. However, we may have Algorithm insertItem(k, e) 1. Perform lines 1.-7. of findElement()(k, e) on the tree to find the right place for an item with key k (if it finds k high in the tree, walk down to the near-leaf with largest key no greater than k). 2. Neighbouring leaf vertex u becomes internal vertex, u.key k, u.elt e. 3. Find the lowest unbalanced vertex z on the path from u to the root. 4. if no such vertex exists, return (tree is still balanced). 5. else 6. Let y and x be child and grandchild of z on z u path. 7. Apply the appropriate rotation to x, y, z. reduced the height of the subtree originally located at z by one, and this may cause the parent of z to become unbalanced. If this happens, we have to apply a rotation to the parent and, if necessary, rotate our way up to the root. Algorithm on the following page gives pseudocode for removeItem(k) operating on an AVL-tree. Algorithm removeItem(k) 1. Perform lines 17 of findElement(k) on the tree to get to vertex t. 2. if we find t with t.key = k, 3. then remove the item at t, set e = t.elt. 4. Let u be near-leaf closest to k. Move us item to t. 5. while u is not the root do 6. let z be the parent of u 7. if z is unbalanced then 8. do the appropriate rotation at z 9. Reset u to be the (possibly new) parent of u 10. return e 11. else return NO SUCH KEY We now discuss the worst-case running-time TremoveItem(n) of our AVL implementation of removeItem. Again letting h denote the height of the tree, we recall that line 1 requires time O(h) = O(lg(n)). Lines 2. and 3. will take O(1) time, while line 4. will take O(h) time again. The loop in lines 59 is iterated at most h times. Each iteration requires time O(1). Thus the execution of the whole loop requires time O(h). Altogether, the asymptotic worst-case running time of removeItem is O(h) = O(lg(n)).

2-3 Trees
Balanced Search Trees
62

Many data structures use binary search trees or generalizations thereof. Operations on such search trees are often proportional to the height of the tree. To guarantee that such operations are ecient, it is necessary to ensure that the height of the tree is logarithmic in the number of nodes. This is usually achieved by some sort of balancing mechanism that guarantees that subtrees of a node never differ too much" in their heights (by either an additive or multiplicative factor). There are many kinds of balanced search trees. Here we study a particularly elegant form of balanced search tree known as a 2-3 tree. There are many other kinds of balanced search trees (e.g., red-black trees, AVL trees, 2-3-4 trees, and B-trees), some of which you will encounter in CS231. 2-3 Trees A 2-3 tree has three dierent kinds of nodes: A leaf, written as . 2. A 2-node, written as

X is called the value of the 2-node; l is its left subtree; and r is its right subtree. Every 2-node must satisfy the following invariants: (a) Every value v appearing in subtree l must be X. (b) Every value v appearing in subtree r must be X. (c) The length of the path from the 2-node to every leaf in its subtrees must be the same. 3. A 3-node, written as

X is called the left value of the 3-node; Y is called the right value of the 3-node; l is its left subtree; m is its middle subtree; and r is its right subtree. Every 3-node must satisfy the following invariants: (a) Every value v appearing in subtree l must be X. (b) Every value v appearing in subtree m must be X and Y . (c) Every value v appearing in subtree r must be Y . (d) The length of a path from the 3-node to every leaf in its subtrees must be the same.
63

The last invariant for 2-nodes and 3-nodes is the path-length invariant. The balance of 2-3 trees is a consequence of this invariant. The height of a 2-3 tree with n nodes cannot exceed log2(n + 1). Together, the tree balance and the ordered nature of the nodes means that testing membership in, inserting an element into, and deleting an element from a 2-3 tree takes logarithmic time. 2-3 Tree Examples Given a collection of three or more values, there are several 2-3 trees containing those values. For instance, below are all four distinct 2-3 trees containing first 7 positive integers.

We shall use the term terminal node to refer to a node that has leaves as its subtrees. To save space, we often will not explicitly show the leaves that are the childred of a terminal node. For instance, here is another depiction of the tree t2 above without the explicit leaves:

64

2-3 Tree Insertion: Downward Phase When inserting an element v into a 2-3 tree, care is required to maintain the invariants of 2-nodes and 3-nodes. As shown in the rules below, the order invariants are maintained much as in a binary search tree by comparing v to the node values encountered when descending the tree and moving in a direction that satisfies the order invariants. In the following rules, the result of inserting an element v into a 2-3 tree is depicted as a circled v with an arrow pointing down toward the tree in which it is to be inserted. X and Y are variables that stand for any elements, while triangles labeled l, m, and r stand for whole subtrees.

The rules state that elements equal to a node value are always inserted to the left of the node. This is completely arbitrary; they could be inserted to the right as well. Note that the tree that results from inserting v into a tree T had better not have a different height from T. Otherwise, the path-length invariant would be violated. We will see how this plays out below.

65

66

2-3 Tree Insertion: Upward Phase If there is a 2-node upstairs, the kicked-up value w can be absorbed by the 2-node:

By our assumptions about height, the resulting tree is a valid 2-3 tree. If there is a 3-node upstairs, w cannot simply be absorbed. Instead, the 3-node is split into two 2-nodes that become the subtrees of a new kicked-up node one level higher. The value w and the two 3-node values X and Y are appropriately redistributed so that the middle of the three values is kicked upstairs at the higher level:

67

The kicking-up process continues until either the kicked-up value is absorbed or the root of the tree is reached. In the latter case, the kicked-up value becomes the value of a new 2-node that increases the height of the tree by one. This is the only way that the height of a 2-3 tree can increase.Convince yourself that heights and element order are unchanged by the downward or upward phases of the insertion algorithm. This means that the tree result from insertion is a valid 2-3 tree. 2-3 Tree Insertion: Special Cases for Terminal Nodes The aforementioned rules are all the rules needed for insertion. However, insertion into terminal nodes is tedious because the inserted value will be pushed down to a leaf and then reflected up right away. To reduce the number of steps performed in examples, we can pretend that insertion into terminal nodes is handled by the following rules:

68

2-3 Tree Deletion: Upward Phase The goal of the upward phase of 2-3 tree deletion is to propagate the hole up the tree until it can be eliminated. It is eliminated either (1) by being \absorbed" into the tree (as in the cases 2, 3, and 4 below) or (2) by being propagated all the way to the root of the 2-3 tree by repeated applications of the case 1. If a hole node propagates all the way to the top of a tree, it is simply removed, decreasing the height of the 2-3 tree by one. This is the only way that the height of a 2-3 node can decrease. There are four cases for hole propagation/removal, which are detailed below. You should convince yourself that each rule preserves both the element-order and path-length invariants. 1. The hole has a 2-node as a parent and a 2-node as a sibling.

69

In this case, the heights of the subtrees l, m, and r are the same. 2. The hole has a 2-node as a parent and a 3-node as a sibling.

In this case, the heights of the subtrees a, b, c, and d are the same. 3. The hole has a 3-node as a parent and a 2-node as a sibling. There are two subcases: (a) The first subcase involves subtrees a, b, and c whose heights are one less than that of subtree d.

(b) The second subcase involves subtrees b, c, and d whose heights are one less than that of subtree a. When the hole is in the middle, there may be ambiguity in terms of whether to apply the right-hand rule of the rst subcase or the left-hand rule of the second subcase. Either application is OK.

70

2-3-4 Trees
2-3-4 trees are a kind of prefectly balanced search tree. They are so named because every node has 2, 3, or 4 children, except leaf nodes, which are all at the bottom level of the tree. Each node stores 1, 2, or 3 entries, which determine how other entries are distributed among its childrens subtrees. Each non-leaf node has one more child than keys. For example, a node with keys [20, 40, 50] has four children. Eack key k in the subtree rooted at the first child satisfies k<= 20; at the second child, 20 <= k<= 40; at the third child, 40<= k<= 50; and at the fourth child, k>= 50. B-trees: the general case of a 2-3-4 tree 2-3-4 trees are a type of B-tree. A B-tree is a generalized version of this kind of tree where the number of children that each node can have varies. Because a range of child nodes is permitted, Btrees do not need re-balancing as frequently as other self-balancing binary search trees, but may waste some space, since nodes are not entirely full. The lower and upper bounds on the number of child nodes are typically fixed for a particular implementation. For example, in a 2-3-4 tree, each non-leaf node may have only 2,3, or 4 child nodes. The number of elements in a node is one less than the number of children. A B-tree is kept balanced by requiring that all leaf nodes are at the same depth. This depth will increase slowly as elements are added to the tree, but an increase in the overall depth is infrequent, and results in all leaf nodes being one more hop further removed from the root. B-trees
71

have advantages over alternative implementations when node access times far exceed access times within nodes. This usually occurs when most nodes are in secondary storage, such as on hard drives. By maximizing the number of child nodes within each internal node, the height of the tree decreases, balancing occurs less often, and efficiency increases. Usually this value is set such that each node takes up a full disk block or some other size convenient to the storage unit being used. So in practice, B-trees with larger internal node sizes are more commonly used, but we will be discussing 2-3-4 trees since it is useful to be able to work out examples with a managable node size. 2-3-4 tree operations find(Key k) Finding an entry is straightforward. Start at the root. At each node, check for the key k; if its not present, move down to the appropriate child chosen by comparing k against the keys. Continue until k is found, or k is not found at a leaf node. For example, find(74) traverses the double-lined boxes in the diagram below.

Insert(KeyValPair p) insert(), like find(), walks down the tree in search of the key k. If it finds an entry with key k, it proceeds to that entrys left child and continues. Unlike find(), insert() sometimes modifies nodes of the tree as it walks down. Specifically, whenever insert() encounters a 3-key node, the middle key is ejected, and is placed in the parent node instead. Since the parent was previously treated the same way, the parent has at most two keys, and always has room for a third. The other two keys in the 3key node are split into two separate 1-key nodes, which are divided underneath the old middle key (as the figure illustrates).

72

For example, suppose we insert 60 into the tree depicted earlier. The first node traversed is the root, which has three children; so we kick the middle child (40) upstairs. Since the root node has no parent, a new node is created to hold 40 and becomes the root. Similarly, 62 is kicked upstairs when insert() finds the node containing it. This ensures us that when we arrive at the leaf node (labeled 57 in this case), theres room to add the new key 60.

Observe that along the way, we created a new 3-key node 62 70 79. We do not kick its middle key upstairs until the next time it is traversed. Again, the reasons why we split every 3-key node we encounter (and move its middle key up one level) are (1) to make sure theres room for the new key in the leaf node, and (2) to make sure that above the leaf nodes, theres room for any key that gets kicked upstairs. Sometimes, an insertion operation increases the depth of the tree by one by creating a new root. Remove(Key k)
73

2-3-4 tree remove() is similar to remove() on binary trees: you find the entry you want to remove (having key k). If its in a leaf node, you remove it. If its in an interior node, you replace it with the entry with the next higher key. That entry must be in a leaf node. In either case, you remove an entry from a leaf node in the end. Like insert(), remove() changes nodes of the tree as it walks down. Whereas insert() eliminates 3-key nodes (moving nodes up the tree) to make room for new keys, remove() eliminates 1-key nodes (sometimes pulling keys down the tree) so that a key can be removed from a leaf without leaving it empty. There are three ways 1-key nodes (except the root) are eliminated. 1. When remove() encounters a 1-key node (except the root), it tries to steal a key from an adjacent sibling. But we cant just steal the siblings key without violating the search tree invariant. This figure shows removes rotation action, when it reaches 30. We move a key from the sibling to the parent, and we move a key from the parent to the 1-key node. We also move a subtree S from the sibling to the 1-key node (now a 2-key node). Note that we cant steal a key from a non-adjacent sibling.

2. If no adjacent sibling has more than one key, a rotation cant be used. In this case, the 1-key node steals a key from its parent. Since the parent was previously treated the same way (unless its the root), it has at least two keys, and can spare one. The sibling is also absorbed, and the 1-key node becomes a 3-key node. The figure illustrates removes action when it reaches 10. This is called a fusion operation.

3. If the parent is the root and contains only one key, and the sibling contains only one key, then the current 1-key node, its 1-key sibling, and the 1-key root are merged into one 3-key node that serves as the new root. The depth of the tree decreases by one. Eventually
74

we reach a leaf. After processing the leaf, it has at least two keys (if there are at least two keys in the tree), so we can delete the key and still have one key in the leaf. For example, suppose we remove 40 from the large tree depicted earlier. The root node contains 40, which we mark xx here to remind us that we plan to replace it with the smallest key in the root nodes right subtree. To find that key, we move on to the 1-key node labeled 50. Following our rules for 1-key nodes, we merge 50 with its sibling and parent to create a new 3-key root labeled 20 xx 50.

Next, we visit the node labeled 43. Again following our rules for 1-key nodes, we move 62 from a sibling to the root, and move 50 from the root to the node containing 43.

Finally, we move down to the node labeled 42. A different rule for 1-key nodes requires us to merge the nodes labeled 42 and 47 into a 3-key node, stealing 43 from the parent node.

75

The last step is to remove 42 from the leaf node and replace xx with 42.

Running times A 2-3-4 tree with depth d has between 2d and 4d leaf nodes. If n is the total number of nodes in the tree, then n >=2(d+1)1. By taking the logarithm of both sides, we find that d is in O(log n). The time spent visiting a 2-3-4 node is typically longer than in a binary search tree (because the nodes and the rotation and fusion operations are complicated), but the time per node is still in O(1). The number of nodes visited is proportional to the depth of the tree. Hence, the running times of the find(), insert(), and remove() operations are in O(d) and hence in O(log n), even in the worst case.

Red-black Trees
Properties
A binary search tree in which The root is colored black All the paths from the root to the leaves agree on the number of black nodes No path from the root to a leaf may contain two consecutive nodes colored red Empty subtrees of a node are treated as subtrees with roots of black color. The relation n > 2h/2 - 1 implies the bound h < 2 log 2(n + 1).
76

Insertions

Insert the new node the way it is done in binary search trees Color the node red If a discrepancy arises for the red-black tree, fix the tree according to the type of discrepancy.

A discrepancy can result from a parent and a child both having a red color. The type of discrepancy is determined by the location of the node with respect to its grand parent, and the color of the sibling of the parent. Discrepancies in which the sibling is red, are fixed by changes in color. Discrepancies in which the siblings are black, are fixed through AVL-like rotations. Changes in color may propagate the problem up toward the root. On the other hand, at most one rotation is sufficient for fixing a discrepancy.

LLr

if A is the root, then it should be repainted to black

LRr

if A is the root, then it should be repainted to black

77

LLb

LRb

Discrepancies of type RRr, RLr, RRb, and RLb are handled in a similar manner.
insert 1

insert 2

insert 3 RRb discrepancy

78

insert 4 RRr discrepancy

insert 5

RRb discrepancy

Deletions
Delete a key, and a node, the way it is done in binary search trees. A node to be deleted will have at most one child. If the deleted node is red, the tree is still a red-black tree. If the deleted node has a red child, repaint the child to black. If a discrepancy arises for the red-black tree, fix the tree according to the type of discrepancy. A discrepancy can result only from a loss of a black node.

Let A denote the lowest node with unbalanced subtrees. The type of discrepancy is determined by the location of the deleted node (Right or Left), the color of the sibling (black or red), the number of red children in the case of the black siblings, and and the number of grand-children in the case of red siblings. In the case of discrepancies which result from the addition of nodes, the correction mechanism may propagate the color problem (i.e., parent and child painted red) up toward the root, and stopped on the way by a single rotation. Here, in the case of discrepancies which result from the deletion of nodes, the discrepancy of a missing black node may propagate toward the root, and stopped on the way by an application of an appropriate rotation.

Rb0

change of color, sending the deficiency up to the root of the subtree

79

Rb1

Rb2

Rr0

might result in LLb discrepancy of parent and child having both the red color

80

Rr1

Rr2

Similar transformations apply to Lb0, Lb1, Lb2, Lr0, Lr1, and Lr2.
81

B-trees

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

Preceding-Child(x) Returns the left child of key x. Move-Key(k, n1, n2) Moves key k from node n1 to node n2.
99

Merge-Nodes(n1, n2) Merges the keys of nodes n1 and n2 into a new node. Find-Predecessor-Key(n, k) Returns the key preceding key k in the child of node n. Remove-Key(k, n) Deletes key k from node n. n must be a leaf node.

Splay trees
A splay tree is another type of balanced binary search tree. All splay tree operations run in O(log n) time on average, where n is the number of entries in the tree, assuming you start with an empty tree. Any single operation can take (n) time in the worst case, but operations slower than O(log n) time happen rarely enough that they dont affect the average. Splay trees really excel in applications where a small fraction of the entries are the targets of most of the find operations, because theyre designed to give especially fast access to entries that have been accessed recently. Splay trees have become the most widely used data structure invented in the last 20 years, because theyre the fastest type of balanced search tree for many applications, since it is quite common to want to access a small number of entries very frequently, which is where splay trees excel. Splay trees, like AVL trees, are kept balanced by means of rotations. Unlike AVL trees, splay trees are not kept perfectly balanced, but they tend to stay reasonably well-balanced most of the time, thereby averaging O(logn) time per operation in the worst case (and sometimes achieving O(1) average running time in special cases). Well analyze this phenomenon more precisely when we discuss amortized analysis. Splay tree operations
find(Key k)

The find operation in a splay tree begins just like the find operation in an ordinary binary search tree: we walk down the tree until we find the entry with key k, or reach a dead end. However, a splay tree isnt finished its job. Let X be the node where the search ended, whether it contains the key k or not. We splay X up the tree through a sequence of rotations, so that X becomes the root of the tree. Why? One reason is so that recently accessed entries are near the root of the tree, and if we access the same few entries repeatedly, accesses will be quite fast. Another reason is because if X lies deeply down an unbalanced branch of the tree, the splay operation will improve the balance along that branch. When we splay a node to the root of the tree, there are three cases that determine the rotations we use. 1. X is the right child of a left child (or the left child of a right child): let P be the parent of X, and let G be the grandparent of X. We first rotate X and P left, and then rotate X and G right, as illustrated below.

100

The mirror image of this case where X is a left child and P is a right childuses the same rotations in mirror image: rotate X and P right, then X and G left. Both the case illustrated above and its mirror image are called the zig-zag case. 2. X is the left child of a left child (or the right child of a right child): the ORDER of the rotations is REVERSED from case 1. We start with the grandparent, and rotate G and P right. Then, we rotate P and X right. The mirror image of this case where X and P are both right childrenuses the same rotations in mirror image: rotate G and P left, then P and X left. Both the case illustrated below and its mirror image are called the zig-zig case.

We repeatedly apply zig-zag and zig-zig rotations to X; each pair of rotations raises X two levels higher in the tree. Eventually, either X will reach the root (and were done), or X will become the child of the root. One more case handles the latter circumstance. 3. Xs parent P is the root: we rotate X and P so that X becomes the root. This is called the zig case.

Heres an example of find(7). Note how the trees balance improves.

101

By inspecting each of the three cases (zig-zig, zig-zag, and zig), you can observe a few interesting facts. First, in none of these three cases does the depth of a subtree increase by more than two. Second, every time X takes two steps toward the root (zig-zig or zig-zag), every node in the subtree rooted at X moves at least one step closer to the root. As more and more nodes enter Xs subtree, more of them get pulled closer to the root. A node that initially lies at depth d on the access path from the root to X moves to a final depth no greater than 3 + d/2. In other words, all the nodes deep down the search path have their depths roughly halved. This tendency of nodes on the access path to move toward the root prevents a splay tree from staying unbalanced for long (as the example below illustrates).

first(), last() 102

These methods begin by finding the entry with minimum or maximum key, just like in an ordinary binary search tree. Then, the node containing the minimum or maximum key is splayed to the root. insert(KeyValPair p) insert begins by inserting the new entry p, just like in an ordinary binary search tree. Then, it splays the new node to the root. remove(Key k) An entry having key k is removed from the tree, just as with ordinary binary search trees. Let X be the node removed from the tree. After X is removed, splay its parent to the root. Heres a sequence illustrating the operation remove(2).

In this example, the key 4 moved up to replace the key 2 at the root. After the node containing 4 was

removed, its parent (containing 5) splayed to the root. If the key k is not in the tree, splay the node where the search ended to the root, just like in a find operation.

Tries
What Is A Trie? Let us, for a moment, step back and reflect on the many sort methods developed in the text. We see that the majority of these methods (e.g., insertion sort, bubble sort, selection sort, heap sort, merge sort, and quick sort) accomplish the sort by comparing pairs of elements. Radix sort (and bin sort, which is a special case of radix sort), on the other hand, does not perform a single comparison between elements. Rather, in a radix sort, we decompose keys into digits using some radix; and the elements are sorted digit by digit using a bin sort. Now, let us reflect on the dictionary methods developed in the text. The hashing methods use a hash function to determine a home bucket, and then use element (or key) comparisons to search either the home bucket chain (in the case of a chained hash table) or a contiguous collection of full buckets beginning with the home bucket (in the case of linear open addressing). The search tree data structures direct the search based on the result of comparisons performed between the search key and the element(s) in the root of the current subtree. We have not, as yet, seen a dictionary data structure that is based on the digits of the keys! The trie (pronounced ``try'' and derived from the word retrieval) is a data structure that uses the digits in the keys to organize and search the dictionary. Although, in practice, we can use any radix to decompose the keys into digits, in our examples, we shall choose our radixes so that the digits are natural entities such as decimal digits (0, 1, 2, 3, 4, 5, 6, 7, 8, 9) and letters of the English
103

alphabet (a-z, A-Z). Suppose that the elements in our dictionary are student records that contain fields such as student name, major, date of birth, and social security number (SS#). The key field is the social security number, which is a nine digit decimal number. To keep the example manageable, assume that the dictionary has only five elements. The name and SS# fields for each of the five elements in our dictionary are shown below. Name Jack Jill Bill | Social Security Number (SS#) | 951-94-1654 | 562-44-2169 | 271-16-3624

Kathy | 278-49-1515 April | 951-23-7625 Figure 1 Five elements (student records) in a dictionary To obtain a trie representation for these five elements, we first select a radix that will be used to decompose each key into digits. If we use the radix 10, the decomposed digits are just the decimal digits shown in Figure 1. We shall examine the digits of the key field (i.e., SS#) from left to right. Using the first digit of the SS#, we partition the elements into three groups--elements whose SS# begins with 2 (i.e., Bill and Kathy), those that begin with 5 (i.e., Jill), and those that begin with 9 (i.e., April and Jack). Groups with more than one element are partitioned using the next digit in the key. This partitioning process is continued until every group has exactly one element in it. The partitioning process described above naturally results in a tree structure that has 10-way branching as is shown in Figure 2. The tree employs two types of nodes--branch nodes and element nodes. Each branch node has 10 children (or pointer/reference) fields. These fields, child[0:9], have been labeled 0, 1, ..., 9 for the root node of Figure 2. root.child[i] points to the root of a subtrie that contains all elements whose first digit is i. In Figure 2, nodes A, B, D, E, F, and I are branch nodes. The remaining nodes, nodes C, G, H, J, and K are element nodes. Each element node contains exactly one element of the dictionary. In Figure 2, only the key field of each element is shown in the element nodes.

104

Figure 2 Trie for the elements of Figure 1 Searching a Trie To search a trie for an element with a given key, we start at the root and follow a path down the trie until we either fall off the trie (i.e., we follow a null pointer in a branch node) or we reach an element node. The path we follow is determined by the digits of the search key. Consider the trie of Figure 2. Suppose we are to search for an element with key 951-23-7625. We use the first digit, 9, in the key to move from the root node A to the node A.child[9] = D. Since D is a branch node, we use the next digit, 5, of the key to move further down the trie. The node we reach is D.child[5] = F. To move to the next level of the trie, we use the next digit, 1, of the key. This move gets us to the node F.child[1] = I. Once again, we are at a branch node and must move further down the trie. For this move, we use the next digit, 2, of the key, and we reach the element node I.child[2] = J. When an element node is reached, we compare the search key and the key of the element in the reached element node. Performing this comparison at node J, we get a match. The element in node J, is to be returned as the result of the search. When searching the trie of Figure 2 for an element with key 951-23-1669, we follow the same path as followed for the key 951-23-7625. The key comparison made at node J tells us that the trie has no element with key 951-23-1669, and the search returns the value null. To search for the element with key 562-44-2169, we begin at the root A and use the first digit, 5, of the search key to reach the element node A.child[5] = C. The key of the element in node C is compared with the search key. Since the two keys agree, the element in node C is returned. When searching for an element with key 273-11-1341, we follow the path A, A.child[2] = B, B.child[7] = E, E.child[3] = null. Since we fall off the trie, we know that the trie contains no element whose key is 273-11-1341. When analyzing the complexity of trie operations, we make the assumption that we can obtain the next digit of a key in O(1) time. Under this assumption, we can search a trie for an element with a d digit key in O(d) time. Keys With Different Length In the example of Figure 2, all keys have the same number of digits (i.e., 9). In applications in which different keys may have a different number of digits, we normally add a special digit (say #) at the end of each key so that no key is a prefix of another. To see why this is done, consider the example of Figure 2. Suppose we are to search for an element with the key 27. Using the search strategy just described, we reach the branch node E. What do we do now? There is no next digit in the search key that can be used to reach the terminating condition (i.e., you either fall off the trie or
105

reach an element node) for downward moves. To resolve this problem, we add the special digit # at the end of each key and also increase the number of children fields in an element node by one. The additional child field is used when the next digit equals #. Height of a Trie In the worst case, a root-node to element-node path has a branch node for every digit in a key. Therefore, the height of a trie is at most number of digits + 1. A trie for social security numbers has a height that is at most 10. If we assume that it takes the same time to move down one level of a trie as it does to move down one level of a binary search tree, then with at most 10 moves we can search a social-security trie. With this many moves, we can search a binary search tree that has at most 210 - 1 = 1023 elements. This means that, we expect searches in the social security trie to be faster than searches in a binary search tree (for student records) whenever the number of student records is more than 1023. The breakeven point will actually be less than 1023 because we will normally not be able to construct full or complete binary search trees for our element collection. Since a SS# is nine digits, a social security trie can have up to 109 elements in it. An AVL tree with 109 elements can have a height that is as much as (approximately) 1.44 log2(109+2) = 44. Therefore, it could take us four times as much time to search for elements when we organize our student record dictionary as an AVL tree than when this dictionary is organized as a trie! Space Required and Alternative Node Structures The use of branch nodes that have as many child fields as the radix of the digits (or one more than this radix when different keys may have different length) results in a fast search algorithm. However, this node structure is often wasteful of space because many of the child fields are null. A radix r trie for d digit keys requires O(rdn) child fields, where n is the number of elements in the trie. To see this, notice that in a d digit trie with n information nodes, each information node may have at most d ancestors, each of which is a branch node. Therefore, the number of branch nodes is at most dn. (Actually, we cannot have this many branch nodes, because the information nodes have common ancestors like the root node.) We can reduce the space requirements, at the expense of increased search time, by changing the node structure. For example, each branch node of a trie could be replaced by any of the following:

A chain of nodes, each node having the three fields digitValue, child, next. Node A of Figure 2, for example, would be replaced by the chain shown in Figure 3.

Figure 3 Chain for node A of Figure 2 The space required by a branch node changes from that required for r children/pointer/reference fields to that required for 2p pointer fields and p digit value fields, where p is the number of children fields in the branch node that are not null. Under the assumption that pointer fields and digit value fields are of the same size, a reduction in space is realized when more than two-thirds of the children fields in branch nodes are null. In the worst case, almost all the branch nodes have only 1 field that is not null and the space savings become almost (1 - 3/r) * 100%.

A (balanced) binary search tree in which each node has a digit value and a pointer to the subtrie for that digit value. Figure 4 shows the binary search tree for node A of Figure 2.

106

Figure 4 Binary search tree for node A of Figure 2 Under the assumption that digit values and pointers take the same amount of space, the binary search tree representation requires space for 4p fields per branch node, because each search tree node has fields for a digit value, a subtrie pointer, a left child pointer, and a right child pointer. The binary search tree representation of a branch node saves us space when more than three-fourths of the children fields in branch nodes are null. Note that for large r, the binary serach tree is faster to search than the chain described above.

A binary trie (i.e., a trie with radix 2). Figure 5 shows the binary trie for node A of Figure 2. The space required by a branch node represented as a binary trie is at most (2 * ceil(log2r) + 1)p.

Figure 5 Binary trie for node A of Figure 2

A hash table. When a hash table with a sufficiently small loading density is used, the expected time performance is about the same as when the node structure of Figure 1 is used. Since we expect the fraction of null child fields in a branch node to vary from node to node and also to increase as we go down the trie, maximum space efficiency is obtained by consolidating all of the branch nodes into a single hash table. To accomplish this, each node in the trie is assigned a number, and each parent to child pointer is replaced by a triple of the form (currentNode, digitValue, childNode). The numbering scheme for nodes is chosen so as to easily distinguish between branch and information nodes. For example, if we expect to have at most 100 elements in the trie at any time, the numbers 0 through 99 are reserved for information nodes and the numbers 100 on up are used for branch nodes. The information nodes are themselves represented as an array information[100]. (An alternative scheme is to represent pointers as tuples of the form (currentNode, digitValue, childNode, childNodeIsBranchNode), where childNodeIsBranchNode = true iff the child node is a branch node.) Suppose that the nodes of the trie of Figure 2 are assigned numbers as given below. This number assignment assumes that the trie will have no more than 10 elements.

Node

A B C D E F G H I J K

Number 10 11 0 12 13 14 1 2 15 3 4 The pointers in node A are represented by the tuples (10,2,11), (10,5,0), and (10,9,12). The pointers in node E are represented by the tuples (13,1,1) and (13,8,2). The pointer triples are stored in a hash table using the first two fields (i.e., the currentNode and digitValue) as the key. For this purpose, we may transform the two field key into an
107

integer using the formula currentNode * r + digitValue, where r is the trie radix, and use the division method to hash the transformed key into a home bucket. The data presently in information node i is stored in information[i]. To see how all this works, suppose we have set up the trie of Figure 2 using the hash table scheme just described. Consider searching for an element with key 278-49-1515. We begin with the knowledge that the root node is assigned the number 10. Since the first digit of the search key is 2, we query our hash table for a pointer triple with key (10,2). The hash table search is successful and the triple (10,2,11) is retrieved. The childNode component of this triple is 11, and since all information nodes have a number 9 or less, the child node is determined to be a branch node. We make a move to the branch node 11. To move to the next level of the trie, we use the second digit 7 of the search key. For the move, we query the hash table for a pointer with key (11,7). Once again, the search is successful and the triple (11,7,13) is retrieved. The next query to the hash table is for a triple with key (13,8). This time, we obtain the triple (13,8,2). Since, childNode = 2 < 10, we know that the pointer gets us to an information node. So, we compare the search key with the key of the element information[2]. The keys match, and we have found the element we were looking for. When searching for an element with key 322-167-8976, the first query is for a triple with key (10,3). The hash table has no triple with this key, and we conclude that the trie has no element whose key equals the search key. The space needed for each pointer triple is about the same as that needed for each node in the chain of nodes representation of a trie node. Therefore, if we use a linear open addressed hash table with a loading density of alpha, the hash table scheme will take approximately (1/alpha 1) * 100% more space than required by the chain of nodes scheme. However, when the hash table scheme is used, we can retrieve a pointer in O(1) expected time, whereas the time to retrieve a pointer using the chain of nodes scheme is O(r). When the (balanced) binary search tree or binary trie schemes are used, it takes O(log r) time to retrieve a pointer. For large radixes, the hash table scheme provides significant space saving over the scheme of Figure 2 and results in a small constant factor degradation in the expected time required to perform a search. The hash table scheme actually reduces the expected time to insert elements into a trie, because when the node structure of Figure 2 is used, we must spend O(r) time to initialize each new branch node (see the description of the insert operation below). However, when a hash table is used, the insertion time is independent of the trie radix. To support the removal of elements from a trie represented as a hash table, we must be able to reuse information nodes. This reuse is accomplished by setting up an available space list of information nodes that are currently not in use (see Section 3.5 (Simulating Pointers) of the text). Inserting into a Trie To insert an element theElement whose key is theKey, we first search the trie for an existing element with this key. If the trie contains such an element, then we replace the existing element with theElement. When the trie contains no element whose key equals theKey, theElement is inserted into the trie using the following procedure. Case 1 For Insert Procedure If the search for theKey ended at an element node X, then the key of the element in X and theKey are used to construct a subtrie to replace X.
108

Suppose we are to insert an element with key 271-10-2529 into the trie of Figure 2. The search for the key 271-10-2529 terminates at node G and we determine that the key, 271-16-3624, of the element in node G is not equal to the key of the element to be inserted. Since the first three digits of the keys are used to get as far as node E of the trie, we set up branch nodes for the fourth digit (from the left) onwards until we reach the first digit at which the two keys differ. This results in branch nodes for the fourth and fifth digits followed by element nodes for each of the two elements. Figure 6 shows the resulting trie.

Figure 6 Trie of Figure 2 with 271-10-2529 inserted Case 2 For Insert Procedure If the search for theKey ends by falling off the trie from the branch node X, then we simply add a child (which is an element node) to the node X. The added element node contains theElement. Suppose we are to insert an element with key 987-33-1122 to the trie of Figure 2. The search for an element with key equal to 987-33-1122 ends when we fall off the trie while following the pointer D.child[8]. We replace the null pointer D.child[8] with a pointer to a new element node that contains theElement, as is shown in Figure 7.

109

Figure 7 Trie of Figure 2 with 987-33-1122 inserted The time required to insert an element with a d digit key into a radix r trie is O(dr) because the insertion may require us to create O(d) branch nodes and it takes O(r) time to intilize the children pointers in a branch node. Removing an Element To remove the element whose key is theKey, we first search for the element with this key. If there is no matching element in the trie, nothing is to be done. So, assume that the trie contains an element theElement whose key is theKey. The element node X that contains theElement is discarded, and we retrace the path from X to the root discarding branch nodes that are roots of subtries that have only 1 element in them. This path retracing stops when we either reach a branch node that is not discarded or we discard the root. Consider the trie of Figure 7. When the element with key 951-23-7625 is removed, the element node J is discarded and we follow the path from node J to the root node A. The branch node I is discarded because the subtrie with root I contains the single element node K. We next reach the branch node F. This node is also discarded, and we proceed to the branch node D. Since the subtrie rooted at D has 2 element nodes (K and L), this branch node is not discarded. Instead, node K is made a child of this branch node, as is shown in Figure 8.

Figure 8 Trie of Figure 7 with 951-23-7635 removed

110

To remove the element with key 562-44-2169 from the trie of Figure 8, we discard the element node C. Since its parent node remains the root of a subtrie that has more than one element, the parent node is not discarded and the removal operation is complete. Figure 9 show the resulting trie.

Figure 9 Trie of Figure 8 with 562-44-2169 removed

The time required to remove an element with a d digit key from a radix r trie is O(dr) because the removal may require us to discard O(d) branch nodes and it takes O(r) time to determine whether a branch node is to be discarded. The complexity of the remove operation can be reduced to O(r+d) by adding a numberOfElementsInSubtrie field to each branch node.

111

UNIT IV GREEDY , DIVIDE AND CONQUER Quicksort

Quicksort is a fast sorting algorithm, which is used not only for educational purposes, but widely applied in practice. On the average, it has O(n log n) complexity, making quicksort suitable for sorting big data volumes. The idea of the algorithm is quite simple and once you realize it, you can write quicksort as fast as bubble sort. Algorithm The divide-and-conquer strategy is used in quicksort. Below the recursion step is described: Choose a pivot value. We take the value of the middle element as pivot value, but it can be any value, which is in range of sorted values, even if it doesn't present in the array. Partition. Rearrange elements in such a way, that all elements which are lesser than the pivot go to the left part of the array and all elements greater than the pivot, go to the right part of the array. Values equal to the pivot can stay in any part of the array. Notice, that array may be divided in non-equal parts. Sort both parts. Apply quicksort algorithm recursively to the left and the right parts. Partition algorithm in detail There are two indices i and j and at the very beginning of the partition algorithm i points to the first element in the array and j points to the last one. Then algorithm moves i forward, until an element with value greater or equal to the pivot is found. Index j is moved backward, until an element with value lesser or equal to the pivot is found. If i jthen they are swapped and i steps to the next position (i + 1), j steps to the previous one (j - 1). Algorithm stops, when i becomes greater than j. After partition, all values before i-th element are less or equal than the pivot and all values after jth element are greater or equal to the pivot.
112

Why does it work? On the partition step algorithm divides the array into two parts and every element a from the left part is less or equal than every element b from the right part. Also a and b satisfy a pivot b inequality. After completion of the recursion calls both of the parts become sorted and, taking into account arguments stated above, the whole array is sorted. Complexity analysis On the average quicksort has O(n log n) complexity, but strong proof of this fact is not trivial and not presented here. Still, you can find the proof in [1]. In worst case, quicksort runs O(n2) time, but on the most "practical" data it works just fine and outperforms other O(n log n) sorting algorithms. Implementation void quickSort(int numbers[], int array_size) { q_sort(numbers, 0, array_size - 1); } void q_sort(int numbers[], int left, int right) { int pivot, l_hold, r_hold; l_hold = left; r_hold = right; pivot = numbers[left]; while (left < right) { while ((numbers[right] >= pivot) && (left < right)) right--; if (left != right) { numbers[left] = numbers[right]; left++; } while ((numbers[left] <= pivot) && (left < right)) left++; if (left != right) { numbers[right] = numbers[left];
113

right--; } } numbers[left] = pivot; pivot = left; left = l_hold; right = r_hold; if (left < pivot) q_sort(numbers, left, pivot-1); if (right > pivot) q_sort(numbers, pivot+1, right); }

Strassen's Matrix Multiplication Algorithm

Matrix multiplication Given two matrices AM*P and BP*N, the product of the two is a matrix CM*N which is computed as follows: void seqMatMult(int m, int n, int p, double** A, double** B, double** C) { for (int i = 0; i < m; i++) for (int j = 0; j < n; j++) { C[i][j] = 0.0; for (int k = 0; k < p; k++) C[i][j] += A[i][k] * B[k][j]; } } Strassen's algorithm To calculate the matrix product C = AB, Strassen's algorithm partitions the data to reduce the number of multiplications performed. This algorithm requires M, N and P to be powers of 2. The algorithm is described below. 1. Partition A, B and and C into 4 equal parts: A= B= A11 A12 A21 A22 B11 B12 B21 B22
114

C=

C11 C12 C21 C22

2. Evaluate the intermediate matrices: M1 = (A11 + A22) (B11 + B22) M2 = (A21 + A22) B11 M3 = A11 (B12 B22) M4 = A22 (B21 B11) M5 = (A11 + A12) B22 M6 = (A21 A11) (B11 + B12) M7 = (A12 A22) (B21 + B22) 3. Construct C using the intermediate matrices: C11 = M1 + M4 M5 + M7 C12 = M3 + M5 C21 = M2 + M4 C22 = M1 M2 + M3 + M6 Serial Algorithm 1. Partition A and B into quarter matrices as described above. 2. Compute the intermediate matrices: 1. If the sizes of the matrices are greater than a threshold value, multiply them recursively using Strassen's algorithm. 2. Else use the traditional matrix multiplication algorithm. 3. Construct C using the intermediate matrices. Parallelization The evaluations of intermediate matrices M1, M2 ... M7 are independent and hence, can be computed in parallel. On a machine with Q processors, Q jobs can be run at a time. Initial Approach The initial approach to a parallel solution used a task pool model to compute M1, M2 ... M7. As shown in the diagram below, the second level of the algorithm creates 49 (7 * 7) independent multiplication tasks which can all be executed in parallel, Q jobs at a time.

115

A Realization Since the number of jobs is 49, ideally, on a machine with Q cores (where Q = 2q, Q <= 16), 48 of these would run concurrently while 1 would end up being executed later, thus, giving poor processor utilization. It would be a better idea to split the last task further. Final Parallel Algorithm The final solution uses thread pooling with a pool of Q threads (where Q is the number of processors), in conjunction with the Strategy pattern to implement Strassen's algorithm. The algorithm is described below: 1 If the sizes of A and B are less than the threshold 1.1 Compute C = AB using the traditional matrix multiplication algorithm. 2 Else use Strassen's algorithm 2.1 Split matrices A and B 2.2 For each of Mi i = 1 to 7 2.2.1 Create a new thread to compute Mi = A'i B'i 2.2.2 If the sizes of the matrices are less than the threshold 2.2.2.1 Compute C using the traditional matrix multiplication algorithm. 2.2.3 Else use Strassen's algorithm 2.2.3.1 Split matrices A'i and B'i 2.2.3.2 For each of Mij j = 1 to 7 2.2.3.2.1 If i=7 and j=7 go to step 1 with A = A'77 and B = B'77 2.2.3.2.2 Get a thread from the thread pool to compute Mij = A'ij B'ij 2.2.3.2.3 Execute the recursive version of Strassen's algorithm in this thread 2.2.3.3 Wait for the Mij threads to complete execution
116

2.2.3.4 Compute Mi 2.3 Wait for the Mi threads to complete execution 2.4Compute C

The above algorithm defines 3 distinct strategies to be used with Strassen's algorithm: 1. Execute each child multiplication operation in a new thread (M1, M2, ..., M7) 2. Execute each child multiplication operation in a thread (Mij) from the thread pool 3. Execute each child multiplication operation using recursion Conclusion Strassen's algorithm definitely performs better than the traditional matrix multiplication algorithm due to the reduced number of multiplications and better memory separation. However, it requires a large amount of memory to run. The performance gain is sub-linear which could be due to the fact that there are threads waiting for other threads to complete execution.

Convex Hulls
A polygon is convex if any line segment joining two points on the boundary stays within the polygon. Equivalently, if you walk around the boundary of the polygon in counterclockwise direction you always take left turns. The convex hull of a set of points in the plane is the smallest convex polygon for which each point is either on the boundary or in the interior of the polygon. One might think of the points as being nails sticking out of a wooden board: then the convex hull is the shape formed by a tight rubber band that surrounds all the nails. A vertex is a corner of a polygon. For example, the highest, lowest, leftmost and rightmost points are all vertices of the convex hull. Some other characterizations are given in the exercises.

117

We discuss three algorithms: Graham Scan, Jarvis March and Divide & Conquer. We present the algorithms under the assumption that: no 3 points are collinear (on a straight line) Graham Scan The idea is to identify one vertex of the convex hull and sort the other points as viewed from that vertex. Then the points are scanned in order. Let x0 be the leftmost point (which is guaranteed to be in the convex hull) and number the remaining points by angle from x0 going counterclockwise: x1; x2; : : : ; xn-1. Let xn = x0, the chosen point. Assume that no two points have the same angle from x0. The algorithm is simple to state with a single stack:

To prove that the algorithm works, it sucess to argue that: A discarded point is not in the convex hull. If xj is discarded, then for some i < j < k the points xi --> xj ---> xk form a right turn. So, xj is inside the triangle x0, xi, xk and hence is not on the convex hull.

118

What remains is convex. This is immediate as every turn is a left turn. The running time: Each time the while loop is executed, a point is either stacked or discarded. Since a point is looked at only once, the loop is executed at most 2n times. There is a constant-time subroutine for checking, given three points in order, whether the angle is a left or a right turn (Exercise). This gives an O(n) time algorithm, apart from the initial sort which takes time O(n log n). (Recall that the notation O(f(n)), pronounced order f(n)", means asymptotically at most a constant times f(n)".) Jarvis March This is also called the wrapping algorithm. This algorithm finds the points on the convex hull in the order in which they appear. It is quick if there are only a few points on the convex hull, but slow if there are many. Let x0 be the leftmost point. Let x1 be the first point counterclockwise when viewed from x0. Then x2 is the first point counterclockwise when viewed from x1, and so on.

Finding xi+1 takes linear time. The while loop is executed at most n times. More specifically, the while loop is executed h times where h is the number of vertices on the convex hull. So Jarvis March takes time O(nh). The best case is h = 3. The worst case is h = n, when the points are, for example, arranged on the circumference of a circle. Divide and Conquer Divide and Conquer is a popular technique for algorithm design. We use it here to find the convex hull. The first step is a Divide step, the second step is a Conquer step, and the third step is a Combine step. The idea is to:

119

Part 2 is simply two recursive calls. The first point to notice is that, if a point is in the overall convex hull, then it is in the convex hull of any subset of points that contain it. (Use characterization in exercise.) So the task is: given two convex hulls find the convex hull of their union. Combining two hulls It helps to work with convex hulls that do not overlap. To ensure this, all the points are presorted from left to right. So we have a left and right half and hence a left and right convex hull. Define a bridge as any line segment joining a vertex on the left and a vertex on the right that does not cross the side of either polygon. What we need are the upper and lower bridges. The following produces the upper bridge. 1. Start with any bridge. For example, a bridge is guaranteed if you join the rightmost vertex on the left to the leftmost vertex on the right. 2. Keeping the left end of the bridge fixed, see if the right end can be raised. That is, look at the next vertex on the right polygon going clockwise, and see whether that would be a (better) bridge. Otherwise, see if the left end can be raised while the right end remains fixed. 3. If made no progress in (2) (cannot raise either side), then stop else repeat (2).

We need to be sure that one will eventually stop. Is this obvious? Now, we need to determine the running time of the algorithm. The key is to perform step (2) in constant time. For this it is sucffiient that each vertex has a pointer to the next vertex going
120

clockwise and going counterclockwise. Hence the choice of data structure: we store each hull using a doubly linked circular linked list . It follows that the total work done in a merge is proportional to the number of vertices. And as we shall see in the next chapter, this means that the overall algorithm takes time O(n log n).

Tree vertex splitting

Directed and weighted binary tree Consider a network of power line transmission The transmission of power from one node to the other results in some loss, such as drop in voltage Each edge is labeled with the loss that occurs (edge weight) Network may not be able to tolerate losses beyond a certain level You can place boosters in the nodes to account for the losses Definition 1 Given a network and a loss tolerance level, the tree vertex splitting problem is to determine the optimal placement of boosters. You can place boosters only in the vertices and nowhere else More definitions

121

122

The above algorithm computes the delay by visiting each node using post-order traversal int tvs ( tree T, int delta ) { if ( T == NULL ) return ( 0 ); // Leaf node d_l = tvs ( T.left(), delta ); // Delay in left subtree d_r = tvs ( T.right(), delta ); // Delay in right subtree current_delay = max ( w_l + d_l,// Weight of left edge w_r + d_r ); // Weight of right edge if ( current_delay > delta ) { if ( w_l + d_l > delta ) { write ( T.left().info() ); d_l = 0; } if ( r_l + d_r > delta ) { write ( T.right().info() ); d_r = 0; }
123

} current_delay = max ( w_l + d_l, w_r + d_r ); return ( current_delay ); } Algorithm tvs runs in (n) time tvs is called only once on each node in the tree On each node, only a constant number of operations are performed, excluding the time for recursive calls Theorem 2 Algorithm tvs outputs a minimum cardinality set U such that d(T/U)<= on any tree T, provided no edge of T has weight > . Proof by induction: Base case. If the tree has only one node, the theorem is true. Induction hypothesis. Assume that the theorem is true for all trees of size <= n.

124

JOB SEQUENCING WITH DEADLINES ALGORITHM An SIMULATION Consider a scheduling problem where the 6 jobs have a profit of (10,34,67,45,23,99) and corresponding deadlines (2,3,1,4,5,3). Obtain the optimum schedule. What is the time complexity of your algorithm? Can you improve it? Ordering the jobs be nonincreasing order of profit: Jobs = (99, 67, 45, 34, 23, 10) Job No. =(6, 3, 4, 2, 5, 1) Deadlines =(2, 3, 1, 4, 5, 3) New job no. =(I, II, III. IV, V, VI) Job I is allotted slot [0,1]

125

126

Job VI has a deadline of 3 but we cannot shift the array to the left, so we reject job VI. The above is a the schedule. FAST Job Sequencing with deadlines. Consider a scheduling problem where the 6 jobs have a profit of (10,34,67,45,23,99) and corresponding deadlines (2,3,1,4,5,3). Obtain the optimum schedule. What is the time complexity of your algorithm? Can you improve it? Sort jobs in nondecreasing order of profit: Profits = (99, 67, 45, 34, 23, 10) Job no. = ( 6, 3, 4, 2, 5, 1) New no = ( I, II, III, IV, V, VI) Deadline = (2, 3, 1, 4, 5, 3) Start with all slots free.

127

128

129

130

131

132

133

134

135

136

UNIT V

DYNAMIC PROGRAMMING AND BACKTRACKING

MULTISTAGE GRAPHS ALL PAIR SHORTESET PATH - FLOYDS ALGORITHM Given a weighted connected graph (undirected or directed), the all-pair shortest paths problem asks to find the distances (the lengths of the shortest paths) from each vertex to all other vertices. It is convenient to record the lengths of shortest paths in an n-by-n matrix D called the distance matrix: the element dij in the ith row and the jth column of this matrix indicates the length of the shortest path from the ith vertex to the jth vertex (1 i,j n). We can generate the distance matrix with an algorithm called Floyds algorithm. It is applicable to both undirected and directed weighted graphs provided that they do not contain a cycle of a negative length.

column of matrix D (k=0,1,. . . ,n) is equal to the length of the shortest path among all paths from the ith vertex to the jth vertex with each intermediate vertex, if any, numbered not higher than k. In particular, the series starts with D(0), which does not allow any intermediate vertices in its paths; hence, D(0) is nothing but the weight matrix of the graph. The last matrix in the
137

series, D(n) contains the lengths of the shortest paths among all pathsthat can use all n vertices as intermediate and hence is nothing but the distance matrix being sought.

138

0/1 KNAPSACK PROBLEM (dynamic programming)

Let i be the highest-numbered item in an optimal solution S for W pounds. Then S` = S - {i} is an optimal solution for W - wi pounds and the value to the solution S is Vi plus the value of the subproblem. We can express this fact in the following formula: define c[i, w] to be the solution for items 1,2, . . . , i and maximum weight w. Then

139

This says that the value of the solution to i items either include ith item, in which case it is vi plus a subproblem solution for (i - 1) items and the weight excluding wi, or does not include ith item, in which case it is a subproblem's solution for (i - 1) items and the same weight. That is, if the thief picks item i, thief takes vi value, and thief can choose from items w - wi, and get c[i - 1, w - wi] additional value. On other hand, if thief decides not to take item i, thief can choose from item 1,2, . . . , i- 1 upto the weight limit w, and get c[i - 1, w] value. The better of these two choices should be made. Although the 0-1 knapsack problem, the above formula for c is similar to LCS formula: boundary values are 0, and other values are computed from the input and "earlier" values of c. So the 0-1 knapsack algorithm is like the LCS-length algorithm given in CLR for finding a longest common subsequence of two sequences. The algorithm takes as input the maximum weight W, the number of items n, and the two sequences v = <v1, v2, . . . , vn> and w = <w1, w2, . . . , wn>. It stores the c[i, j] values in the table, that is, a two dimensional array, c[0 . . n, 0 . . w] whose entries are computed in a row-major order. That is, the first row of c is filled in from left to right, then the second row, and so on. At the end of the computation, c[n, w] contains the maximum value that can be picked into the knapsack.

140

The set of items to take can be deduced from the table, starting at c[n. w] and tracing backwards where the optimal values came from. If c[i, w] = c[i-1, w] item i is not part of the solution, and we are continue tracing with c[i-1, w]. Otherwise item i is part of the solution, and we continue tracing with c[i-1, w-W]. Analysis This dynamic-0-1-kanpsack algorithm takes (nw) times, broken up as follows: (nw) times to fill the c-table, which has (n +1).(w +1) entries, each requiring (1) time to compute. O(n) time to trace the solution, because the tracing process starts in row n of the table and moves up 1 row at each step. N-QUEENS PROBLEM The problem is to place it queens on an n-by-n chessboard so that no two queens attack each other by being in the same row or in the same column or on the same diagonal. For n = 1, the problem has a trivial solution, and it is easy to see that there is no solution for n = 2 and n =3. So let us consider the four-queens problem and solve it by the backtracking technique. Since each of the four queens has to be placed in its own row, all we need to do is to assign a column for each queen on the board presented in the following figure.

Steps to be followed We start with the empty board and then place queen 1 in the first possible position of its row, which is in column 1 of row 1. Then we place queen 2, after trying unsuccessfully columns 1 and 2, in the first acceptable position for it, which is square (2,3), the square in row 2 and column 3. This proves to be a dead end because there i no acceptable position for queen 3. So, the algorithm backtracks and puts queen 2 in the next possible position at (2,4). Then queen 3 is placed at (3,2), which proves to be another dead end. The algorithm then backtracks all the way to queen 1 and moves it to (1,2). Queen 2 then goes to (2,4), queen 3 to (3,1), and queen 4 to (4,3), which is a solution to the problem.
141

The state-space tree of this search is given in the following figure.

Fig: State-space tree of solving the four-queens problem by back tracking. (x denotes an unsuccessful attempt to place a queen in the indicated column. The numbers above the nodes indicate the order in which the nodes are generated) If other solutions need to be found, the algorithm can simply resume its operations at the leaf at which it stopped. Alternatively, we can use the boards symmetry for this purpose. GRAPH COLORING A coloring of a graph is an assignment of a color to each vertex of the graph so that no two vertices connected by an edge have the same color. It is not hard to see that our problem is one of coloring the graph of incompatible turns using as few colors as possible. The problem of coloring graphs has been studied for many decades, and the theory of algorithms tells us a lot about this problem. Unfortunately, coloring an arbitrary graph with as few colors as possible is one of a large class of problems called "NP-complete problems," for which all known solutions are essentially of the type "try all possibilities." A k-coloring of an undirected graph G = (V, E) is a function c : V {1, 2,..., k} such that c(u) c(v) for every edge (u, v) E. In other words, the numbers 1, 2,..., k represent the k colors, and adjacent vertices must have different colors. The graph-coloring problem is to determine the minimum number of colors needed to color a given graph. a. Give an efficient algorithm to determine a 2-coloring of a graph if one exists.
142

b. Cast the graph-coloring problem as a decision problem. Show that your decision problem is solvable in polynomial time if and only if the graph-coloring problem is solvable in polynomial time. c. Let the language 3-COLOR be the set of graphs that can be 3-colored. Show that if 3COLOR is NP-complete, then your decision problem from part (b) is NP-complete. To prove that 3-COLOR is NP-complete, we use a reduction from 3-CNF-SAT. Given a formula of m clauses on n variables x, x,..., x, we construct a graph G = (V, E) as follows. The set V consists of a vertex for each variable, a vertex for the negation of each variable, 5 vertices for each clause, and 3 special vertices: TRUE, FALSE, and RED. The edges of the graph are of two types: "literal" edges that are independent of the clauses and "clause" edges that depend on the clauses. The literal edges form a triangle on the special vertices and also form a triangle on x, x, and RED for i = 1, 2,..., n. d. Argue that in any 3-coloring c of a graph containing the literal edges, exactly one of a variable and its negation is colored c(TRUE) and the other is colored c(FALSE). Argue that for any truth assignment for , there is a 3-coloring of the graph containing just the literal edges. The widget is used to enforce the condition corresponding to a clause (x y z). Each clause requires a unique copy of the 5 vertices that are heavily shaded; they connect as shown to the literals of the clause and the special vertex TRUE. e. Argue that if each of x, y, and z is colored c(TRUE) or c(FALSE), then the widget is 3colorable if and only if at least one of x, y, or z is colored c(TRUE). f. Complete the proof that 3-COLOR is NP-complete.

Fig: The widget corresponding to a clause (x y z), used in Problem KNAPSACK PROBLEM ( backtracking) The knapsack problem or rucksack problem is a problem in combinatorial optimization: Given a set of items, each with a weight and a value, determine the number of each item to include in a collection so that the total weight is less than a given limit and the total value is as large as possible. It derives its name from the problem faced by someone who is constrained by a fixed-size knapsack and must fill it with the most useful items.
143

The problem often arises in resource allocation with financial constraints. A similar problem also appears in combinatorics, complexity theory, cryptography and applied mathematics. The decision problem form of the knapsack problem is the question "can a value of at least V be achieved without exceeding the weight W?" E.g. A thief enters a store and sees the following items:

His Knapsack holds 4 pounds. What should he steal to maximize profit? Fractional Knapsack Problem Thief can take a fraction of an item.

144

145

If knapsack holds k=5 pds, solution is: 1 pds A 3 pds B 1 pds C General Algorithm-O(n): Given: weight cost w1 c1 w2 c2 wn cn

Knapsack weight limit K 1. Calculate vi = ci / wi for i = 1, 2, , n 2. Sort the item by decreasing vi 3. Find j, s.t. w1 + w2 ++ wj <= k < w1 + w2 ++ wj+1 Answer is { wi pds item i,
146

for i <= j K-i<=j wi pds item j+1

Flow Shop Scheduling Problem:

1. INTRODUCTION The purpose of this paper is twofold: (1) to provide a simulation program able to find the optimum / near optimum sequence for general flow shop scheduling problem with make-span minimization as main criteria; (2) to compare different dispatching rules on minimizing multiple criteria. Numerous combinatorial optimization procedures have been proposed for solving the general flowshop problem with the maximum flow time criterion. Many researches have been successful in developing efficient solution algorithms for flowshop scheduling and sequencing [1, 2, 3, 4, 5, 6, 7 and 8] using up to 10 machines. Dannenbring [2] found that for small size shop problems his heuristic outperformed others in minimizing the make-span for 1280 flowshop scheduling problems. Ezat and El Baradie carried a simulation study for pure flowshop scheduling with makespan minimization as a major criterion for n y90 on m y90 [9]. In This paper study general flow shop scheduling problem with make-span minimization as main criteria for n y 250 and m y 250 with different ranges of random numbers generated (0-99) for processing times matrix. 2. THE FLOWSHOP SCHEDULING PROBLEM The flowshop problem has interested researchers for nearly half a century. The flowshop problem consists of two major elements: (1) a production system of m machines; and (2) a set of n jobs to be processed on these machines. All n jobs are so similar that they have essential the same order of processing on the M machines, Fig. 1. The focus of this problem is to sequence or order the n jobs through the m machine(s) production system so that some measure of production cost is minimized [10]. Indeed, flowshop scheduling problem has been shown to be NP-complete for nonpreemptive schedules [11].

147

The assumptions of the flowshop problem are well documented in the production research literature [3,4,5,18]. In summary: 1) All n jobs are available for processing, beginning on machine1, at time zero. 2) Once started into the process, one job may not pass another, but must remain in the same sequence position for its entire processing through the m machines. 3) Each job may be processed on only a single machine at one time, so that job splitting is not permitted. 4) There is only one of each type of machine available. 5) At most, only one job at a time can be processed on an individual machine. 6) The processing times of all n jobs on each of the m machines are predetermined. 7) The set-up times for the jobs are sequence independent so that set-up times can be considered a part of the processing times. 8) In-process inventory is allowed between consecutive machines in the production system. 9) Non-preemption; whereas operations can not be interrupted and each machine can handle only one job at a time. 10) Skipping is allowed in this model. 3. THE PERFORMANCE CRITERIA The performance criteria are those most commonly used as proposed by Stafford [15], for optimizing the general flowshop model. 1. Makespan Throughout the half century of flowshop scheduling research, the predominant objective
148

function has been to minimize make-span. [10] The expression used is as follows: Minimize: Cmax 2. Mean Completion Time Conway et al. (1967), Panwalker and Khan (1975), Bensal (1977), and Scwarc (1983) have all discussed mean job completion time or mean flow time as an appropriate measure of the quality of a flowshop scheduling problem solution. Mean job completion time may be expressed as follows:

c=

n i=1

Job completion times / n

Job completion times / n 3. Total Waiting Time Minimizing total job idle time, while the jobs wait for the next machine in the processing sequence to be ready to process them, may be expressed as follows:

W( nxm ) =
4. Total Idle Time

i=1

j=1

wij

Overall all machine idle time will be considered in this model (the time that machines 2,. , M spend waiting for the first job in the sequence to arrive will be counted). Overall machine idle time may be minimized according to the following expression:

Minimize:Mi=1

j=1

wij

4. DISPATCHING RULES A dispatching rule is used to select the next job to be processed from a set of jobs awaiting service at a facility that becomes free. The difficulty of the choice of a dispatching rule arises from the fact that there are n! ways of sequencing n jobs waiting in the queue at a particular facility and the shop floor conditions elsewhere in the shop may influence the optimal sequence of jobs at the present facility [12]. Five basic dispatching rules have been selected to be investigated in this research. A brief description about each rule will be presented:
149

Rule (1) FCFS (First Come First Served): This rule dispatches jobs based on their arrival times or release dates. The job that has been waiting in queue the longest is selected. The FCFS rule is simple to implement and has a number of noteworthy properties. For example, if the processing times of the jobs are random variables from the same distribution, then the FCFS rule minimizes the variance of the average waiting time. This rule tends to construct schedules that exhibit a low variance in the average total time spent by the jobs in this shop. Rule (2) SPT (Shortest Processing Time): The SPT first rule is a widely used dispatching rule. The SPT rule minimizes the sum of the completion times SCj (usually referred as the flow time), the number of jobs in the system at any point in time, and the average number of jobs in the system over time for the following machine environments: set of unique machines in series, the bank of identical machines in parallel, and the proportionate flow shop. Rule (3) LPT (Longest Processing Time): The LPT rule is particularly useful in the case of a bank of parallel machines where the make-span has to be minimized. This rule selects the job with the longest processing (from the queue of jobs) to go next when a machine becomes available. Inherently, the LPT rule has a load balancing property, as it tends to avoid the situation where one long job is in process while all other machines are free. Therefore, after using the LPT rule to partition the jobs among the machines, it is possible to resequence the jobs for the individual machines to optimize another objective besides make-span. This rule is more effective when preemption is allowed. Rule (4) SRPT (Shortest Remaining Processing Time): The SRPT is a variation of SPT that is applicable when the jobs have different release dates. SRPT rule selects operations that belong to the job with the smallest total processing time remaining. It can be effective in minimizing the make-span when preemption is allowed. Rule (5) LRPT (longest Remaining Processing Time): The LRPT is a variation of LPT that selects the operations that belong to the job with the largest total processing time remaining. LRPT rule is of importance when preemption is allowed and especially in parallel identical machines. LRPT rule always minimizes the idle time of machines.

QUESTION BANK
UNIT I PART A 1. What is a program? 2. What is an algorithm? 3. Define the term data structure?
150

4. What is a data? 5. What are the applications of data structure? 6. What is linear data structure? 7. What is non linear data structure? 8. What is space complexity? 9. What is time complexity? 10. What is asymptotic notation? 11. What is Big oh notation (O)? 12. What is BIG THETA Notation? 13. What is big omega notation ()? 14. What is little oh notation? 15. What is worst case efficiency? 16. What is best case efficiency? 17. What is average case efficiency? 18. Define Amortized Cost. 19. What is Np complete? 20. What is Np hard? 21. Define recurrence relation. 22. Define Linked list ADT. 23. Define doubly linked list 24. Define circular linked list PART B 1. Explain in detail about asymptotic notations with examples. 2. List the properties of big oh notation and prove it. 3. Explain conditional asymptotic notation with example. 4. Explain in detail about amortized analysis. 5. Write notes on NP completeness and NP hard. 6. Explain recurrence equation with examples. 7. How linked list can be implemented using pointers and arrays 8. Explain implementation of doubly linked list using pointers. 9. How circular linked list can be implemented using pointers. 10. Explain tree traversals with an application. 11. How will you implement binary trees.

151

UNIT II PART A 1. 2. 3. 4. 5. 6. 7. 8. Define heap. What is heap property? What is min heap property? What is max heap property? Define Lefitist tree. Define binomial heap. What is fibonocci heap? What is skew heap?

PART B 1. What is heap data structure? What are the operations performed on heap? 2. Explain min-max heap operations with example. 3. Explain Deaps with example. 4. Write the driving and actual routine for merging leftlist heaps. 5. Write Insertion and delete routine for leftlist heaps. 6. Explain skew heap with an example. 7. Explain the structure for Binomial heap. 8. Explain the operations on for Binomial heap with an example. 9. Explain fibanocci heap with example. 10. Explain Lazy binomial heap.

UNIT III PART A 1. Define binary search tree. 2. Define AVL tree . Give example. 3. Define balance factor of AVL tree. 4. What is meant by rotation? When it is performed? 5. Define 2-3 tree. Give example. 6. Define 2-3-4 tree. 7. What is red black tree. 8. Define B-tree. 9. Define splay tree. 10. What trie.

PART B
152

1. Explain binary search tree in detail. 2. Write short notes on AVL trees. 3. Discuss AVL trees and its rotations with example. 4. Construct AVL tree for the given list of numbers (5,4,3,2,1,6,8). 5. How splaying is performed in trees. Explain with an example. 6. Write short notes on B-trees. 7. Explain 2-3 trees with example. 8. Explain 2-3-4 trees with example. 9. Explain Red black tree with example. 10. Explain tries with example.

UNIT IV PART A 1. 2. 3. 4. 5. 6. 7. 8. 9. Define greedy algorithm. What is divide and conquer method. Differentiate greedy technique with dynamic programming. Define quick sort. What is pivot element? State strassens algorithm. Define convex hull problem. What is tree vertex splitting. List the basic concepts of greedy method. PART B 1. Sort the elements using quick sort algorithm 10,2,7,23,34,8,1,9,11,5 2. How will you implement strassens algorithm for matrix multiplication. 3. Explain Convex hull problem with example. 4. Explain Tree-vertex splitting with example 5. Explain Job sequencing with deadlines with example 6. Explain Optimal storage on tapes with example

153

UNIT V PART A 1. 2. 3. 4. 5. 6. 7. 8. What is meant by backtracking. Give examples for backtracking techniques. What is meant by dynamic programming? Define multigraph. Define knapsack problem. What is flow shop scheduling. Define 8-queens problem. What is graph coloring? PART B 1. 2. 3. 4. 5. 6. Explain multistage graphs in detail. Discuss the solution of 0/1 knapsack problem using dynamic programming. Explain Flow shop scheduling in detail. What is back tracking? Explain in detail with 8-queens problem. Explain graph coloring with example. Discuss the solution of knapsack problem using backtracking.

154