The Growth of Functions

This lecture will introduce some tools and notations necessary to study algorithm efficiency. The lecture will cover
Asymptotic behaviour Big-O notation Simplifying big-O expressions Big-O of sums and products Big-Omega and Big-Theta notation

Asymptotic Behaviour
Definition
An algorithm is a finite set of precise instructions for performing a computation or for solving a problem.

Generally, the problem to be solved will have certain parameters

the algorithm needs to work for all valid parameters. e.g. an algorithm for multiplying two matrices should work on any two matrices.

An important question when comparing two algorithms for the same problem: which one is faster?
this will have many possible answers, and may depend on
the language in which it is written. the machine used to run the algorithm. the size of the problem instance, e.g. is this a 4x4 matrix or a 1024x1024 matrix?

11/2/2006

Lecture7

gac1

11/2/2006

Lecture7

gac1

Asymptotic Behaviour
Asymptotic analysis is a method of abstracting away from these details. This is achieved by
examining the run-time as the problem instance increases in size. ignoring constants of proportionality
if run-time doubles when going from a 2x2 matrix to a 3x3 matrix, thats more interesting than it going from 0.01 sec to 0.02 sec on a particular machine.

Big-O Notation
Definition
Let f and g be functions from the set of integers or the set of reals to the set of reals. The function f(x) is big-O of g(x) iff
cR+ kR+ x ( (x > k) ( |f(x)| c|g(x)| ) )

Notation
We will write f(x) is O(g(x)), when f(x) is big-O of g(x).

Example
You have implemented two division algorithms for n-bit numbers. One takes 100n2 + 17n + 4 microseconds. The other takes n3 microseconds. For some values of n, n3 < 100n2 + 17n + 4 (e.g. n = 5). But for large enough n, n3 will be bigger. We will concentrate on this big picture.
11/2/2006 Lecture7 gac1 3

Intuition
The constant k formalizes our concept of big enough problem instances. The constant c formalizes our concern to avoid worrying about constants of proportionality.
11/2/2006 Lecture7 gac1 4

Examples
cx f (x)

Simplifying big-O Expressions

Theorem
Let f(x) = anxn + an-1xn-1 + + a1x + a0, where i (aiR). Then f(x) is O(xn).

Example 1 f(x) is O(x)

k x

Proof
For this proof, we will assume the triangle inequality xy ( |x+y| |x| + |y| ). Using this property, |f(x)| |anxn| + |an-1xn-1| + + |a1x| + |a0| = xn(|an| + |an-1|/x + + |a1|/xn-1 + |a0|/xn) xn(|an| + |an-1| + + |a1| + |a0|) for x > 1 So using c = |an| + |an-1| + + |a1| + |a0| and k = 1 gives x ( (x > k) ( |f(x)| c|g(x)| ) ), completing the proof.
11/2/2006 Lecture7 gac1 6

ff Example 2 We will show that f(x) = x2 + 5x is O(x2). x2 > 5x whenever x > 5. So for x > 5, f(x) 2x2. So using k = 5, c = 2, we have f(x) is O(x2).
11/2/2006 Lecture7 gac1 5

Big-O of Sums
It is often necessary to characterise the growth of a sum of functions.
Let f(x) denote the run-time of a Delphi function fun1, and let g(x) denote the run-time of another Delphi function fun2. What is known about the run-time of the main program that first calls fun1 and then calls fun2?

Big-O of Sums
By the triangle inequality,
|f1(x)+f2(x)| |f1(x)| + |f2(x)| c1|g1(x)| + c2|g2(x)| for all x > max(k1, k2) c1 max( |g1(x)|, |g2(x)| ) + c1 max( |g1(x)|, |g2(x)| ) (c1 + c2) | max( |g1(x)|, |g2(x)| ) |

Theorem
Let f1(x) be O(g1(x)) and f2(x) be O(g2(x)). Then f1(x)+f2(x) is O( max(|g1(x)|,|g2(x)|) ).

Using c = c1 + c2, and k = max(k1, k2), we have that f1(x)+f2(x) is O( max( |g1(x)|, |g2(x)| ) ).

Example
Given that n! is O(nn), provide a big-O expression for n! + 34 n10. You may assume n is non-negative. n! is O(nn) and 34 n10 is O(n10). So n! + 34 n10 is O( max( |nn|, |n10| ) ) = O( max(nn, n10) ).

Proof
There exist k1,k2,c1,c2, such that
|f1(x)| c1|g1(x)| for all x > k1, and |f2(x)| c2|g2(x)| for all x > k2
11/2/2006 Lecture7 gac1 7

11/2/2006

Lecture7

gac1

Big-O of Products
A similar theorem exists for the big-O of a product of functions. Theorem
Let f1(x) be O(g1(x)) and f2(x) be O(g2(x)). Then f1(x)f2(x) is O( g1(x)g2(x) ).

Big-Omega Notation
Big-O notation provides a useful abstraction for upperbounds on function growth, but has some problems.
x2 is O(x2), but also x2 is O(x3) or even O(2x). Providing a lower-bound on function growth would be helpful, but is often much harder. Let f and g be functions from the set of integers or the set of reals to the set of reals. The function f(x) is big-Omega of g(x) iff
cR+ kR+ x ( (x > k) ( |f(x)| c|g(x)| ) ).

Proof
There exist k1,k2,c1,c2, such that
|f1(x)| c1|g1(x)| for all x > k1, and |f2(x)| c2|g2(x)| for all x > k2

Definition

So for x > max(k1, k2),

|f1(x)f2(x)| c1 c2 |g1(x)| |g2(x)| = c1 c2 |g1(x)g2(x)|.

Notation
We will write f(x) is (g(x)), when f(x) is big-Omega of g(x).

Using c = c1 c2 and k = max(k1, k2), we have that f1(x)f2(x) is O(g1(x)g2(x)).

11/2/2006 Lecture7 gac1 9 11/2/2006 Lecture7 gac1 10

Big-Omega Notation
Example
Show that x2 x2 6x is (x2).

Big-Omega Notation
Theorem
f(x) is (g(x)) iff g(x) is O(f(x)).

|x2

6x = 6/x). For x 7, (1 6/x) is positive. 2|(1 6/x) for x 7. 6x| = |x |x2|(1 6/7) for x 7. So choosing k = 7 and c = 1/7, we have that x2 6x is (x2). x2(1

Proof
If f(x) is (g(x)), then we have
|f(x)| c1|g(x)| for x > k1. Since c1 is positive, we have |g(x)| (1/c1)|f(x)|. Using c = 1/c1 and k = k1 gives g(x) is O(f(x)).

If g(x) is O(f(x)), then we have

|g(x)| c2|f(x)| for x > k2. Since c2 is positive, we have |f(x)| (1/c2)|g(x)|. Using c = 1/c2 and k = k2 gives f(x) is (g(x)).

11/2/2006

Lecture7

gac1

11

11/2/2006

Lecture7

gac1

12

Big-Theta Notation
Definition
Let f and g be functions from the set of integers or the set of reals to the set of reals. The function f(x) is big-Theta of g(x) iff f(x) is O(g(x)) and f(x) is (g(x)).

Big-Theta in Action
x2 x2 6x

Notation
We will write f(x) is (g(x)), or f(x) is order g(x), when f(x) is bigTheta g(x).

x2/7

Example
We recently proved that x2 6x is (x2). Also, from our theorem on big-O of polynomials, x2 6x is O(x2). Thus x2 6x is (x2). Note that while x2 6x is O(x3), it is not (x3) Note that while x2 6x is (x), it is not (x).

x Big-Theta is a tight bound on the function

it sandwiches the function f(x) between two scaled versions of the same function g(x). note that in the above example, we could not sandwich the function between scaled versions of g(x) = x, for example.
11/2/2006 Lecture7 gac1 14

11/2/2006

Lecture7

gac1

13

Test Your Knowledge

Which of these functions is (i) O(x), (ii) (x), (iii) (x) ?
(a) f(x) = 10 (b) f(x) = x2 + x + 1 (c) f(x) = 3x + 7

Summary
This lecture has covered
Asymptotic behaviour Big-O notation Simplifying big-O expressions Big-O of sums and products Big-Omega and Big-Theta notation

The next lecture will use these concepts to examine the complexity of algorithms.

11/2/2006

Lecture7

gac1

15

11/2/2006

Lecture7

gac1

16

The Complexity of Algorithms

This lecture will apply our analysis of the growth of functions to help us analyse algorithms. The lecture will cover
Time complexity and space-complexity Worst-case and average-case complexity Common complexity classes The time-complexity of some common algorithms
Linear search Binary search Bubble sort

Worst-Case Time Complexity

Algorithms may be compared by
how much time they take to execute (time complexity) how much memory they require (space complexity)

We will concentrate on time complexity in this lecture Often, an algorithm will take much longer on one input than on another, even when both inputs are the same size. Example
Consider the following algorithm, Linear Search, which looks for a particular element value in an array.

11/2/2006

Lecture8

gac1

11/2/2006

Lecture8

gac1

Linear Search
1. 2. 3. procedure linear_search( integer x, integer a[n] ) begin i := 1 while( i n and x a[ i ] ) i := i + 1 end

Worst/Average-Case Complexity
Given the array a[1] = 1, a[2] = 5, a[3] = 2, a call linear_search( 1, a ) will clearly take less time than a call linear_search( 2, a ).

Often, we are most interested in the worst-case complexity, e.g.

what is the worst possible time taken by linear_search, as a function of n, over all the possible values of x and a[ ]?

This algorithm terminates with i equal to the number of the element, so a[i] = x.
(assuming that there is at least one such element)

Sometimes, we are also interested in the average-case complexity, e.g.

what is the average time taken by linear_search, as a function of n, over all the possible values of x and a[ ]? typically, average case complexity is very difficult to analyse, and also requires information on the joint probability distribution of the inputs. we will restrict ourselves to worst-case complexity analysis.
3 11/2/2006 Lecture8 gac1 4

The algorithm works by comparing x to each element in a[ ] in turn. The first one to match is taken as the result.
11/2/2006 Lecture8 gac1

Linear Search
We will now find an upper bound for the worst-case time complexity of linear_search.
Line 1 executes once. Line 2 (the comparisons) executes at most n times. Line 3 executes at most n 1 times.

Binary Search
Binary search is another searching algorithm, which can be used on an array which is known to be sorted
we will assume it is sorted in ascending order
procedure binary_search( integer x, integer a[n] ) begin i := 1 { we know that x is somewhere between } j := n { the ith and the jth element of a, inclusive } while( i < j ) begin m := floor( (i+j)/2 ) if x > a[ m ] then i := m + 1 else j := m end end
Lecture8 gac1 6

Thus we have an execution time of

O(1) + O(n) + O(n-1) = O(n).

Big-O notation has helped us here, by allowing us to ignore that

we dont know the absolute amount of time taken for an assignment, an array indexing (i.e. finding a[i] from a and i), or an addition; we dont even know the ratios of how long these operations take.

1. 2. 3. 4. 5.

All we needed to assume was that each of these times is a constant (independent of n).
11/2/2006 Lecture8 gac1 5 11/2/2006

Binary Search
How this algorithm works:
Variables i and j record the lower and upper array indices which could contain x, respectively. In each iteration, the index half-way between i and j is found (m). If half-way is not an integer, it is rounded down (floor function). If x is bigger than element a[m], then x can only be above m in the array, since the array is sorted
in this case, we dont need to search from 1 to m, and so i := m+1

Binary Search
We will now find an upper bound for the worst-case time complexity of binary_search.
For simplicity, we will analyse the algorithm for the case where n is a power of 2. Originally i = 1 and j = 2k, so m = 2k-1. After the first iteration, we have either i = 2k-1 + 1 and j = 2k, or i = 1 and j = 2k-1. In either case, we have a total of 2k-1 elements to search. After the second iteration, we will have 2k-2 elements, and so on until we have one element left. So line 3 executes exactly k+1 times. Line 4 and 5 therefore execute k times, and lines 1 and 2 execute once. The total complexity is O(1) + O(k+1) + O(k) = O(k) = O(log2 n).
11/2/2006 Lecture8 gac1 8

If not, x could only be between 1 and m in the array

in this case, we dont need to search from m+1 to n, and so j := m

The algorithm terminates when i = j, i.e. there is only one possible element left to search.
11/2/2006 Lecture8 gac1 7

Bubble Sort
Bubble sort is an algorithm to sort an array in ascending order. procedure bubble_sort( real a[n] )
begin for i := 1 to n 1 for j := 1 to n i if a[ j ] > a[ j + 1 ] then swap a[ j ] and a[ j + 1 ] end

Bubble Sort
An example execution
a[1] a[2] a[3] a[4]
= = = = 1 5 3 0

pair-wise swap
a[1] a[2] a[3] a[4]
= = = = 1 0 3 5

The procedure
makes repeated passes over the array a[ ]. during each iteration, it repeatedly swaps neighbouring elements that are not in the right order. the first iteration ensures that the largest element in the array ends up in position a[n]. the second iteration ensures that the second largest element ends up in position a[n-1], and so on. after n-1 iterations, the array will be sorted.
11/2/2006 Lecture8 gac1 9

a[1] a[2] a[3] a[4]

= = = =

1 3 0 5

a[1] a[2] a[3] a[4]

= = = =

0 1 3 5

1st iteration

2nd iteration

3rd iteration

11/2/2006

Lecture8

gac1

10

A Quick Lemma
To analyse the complexity of bubble sort, we need the following lemma. Lemma
S ( n) = i =
i =1 n 1

Bubble Sort
To analyse the worst-case complexity of bubble sort
notice that there will be n-1 iterations of the outer loop for each iteration of the outer loop, there will be n-i iterations of the inner loop each iteration of the inner loop is O(1) the total number of iterations of the inner loop is thus

n(n 1) 2

for n > 1.

Proof
We will prove by induction. It is clearly true for n = 2. Assume true for n, and we will demonstrate it is true for n+1.

S (n + 1) = i = S (n) + n =
i =1
11/2/2006 Lecture8

(n + 1)n n(n 1) . +n= 2 2

gac1 11

(n i) = n(n 1) i = n(n 1)
i =1 i =1
11/2/2006 Lecture8 gac1

n 1

n 1

n(n 1) n(n 1) = 2 2

Each of these is O(1), so the total complexity is O(n2).

12

Common Complexity Classes

Some big-O complexity cases arise commonly, and so have been given special names.
Big-O Notation O(1) O(log n) O(n) O(nb) O(bn) (b > 1) Name Constant complexity Logarithmic complexity Linear complexity Polynomial complexity Exponential complexity
Line A: Line B:

Test Your Knowledge

An algorithm for n by n matrix multiplication is shown below. Give a big-O estimate of:
(i) The number of times line A executes (ii) The number of times line B executes (iii) The time complexity of the algorithm
procedure matrix_mult( real a[n][n], b[n][n], c[n][n] ) begin for i := 1 to n for j := 1 to n c[ i ][ j ] := 0 for k := 1 to n c[ i ][ j ] := c[ i ][ j ] + a[ i ][ k ]*b[ k ][ j ] end
Lecture8 gac1 14

11/2/2006

Lecture8

gac1

13

11/2/2006

Summary
This lecture has covered
Time complexity and space-complexity Worst-case and average-case complexity Common complexity classes The time-complexity of some common algorithms
Linear search Binary search Bubble sort

However, we are not yet able to analyse recursive algorithms. The next lecture will introduce recurrence relations, which will enable us to do so.
11/2/2006 Lecture8 gac1 15

Recurrence Relations
This lecture will cover
What is a recurrence relation? The substitution method. Solution of some linear homogeneous recurrence relations: degree 1 and degree 2. Solution of some linear non-homogeneous recurrence relations: degree 1 and degree 2.

What is a Recurrence Relation?

Definition
A sequence is a function from a subset of the set of integers (usually N or Z+) to a set S. We use the notation an for the image of n.

Definition
A recurrence relation for the sequence {an} is an equation that expresses an in terms of one or more of the previous terms of the sequence (i.e. a0, a1,, and an-1), for all n n0, where n0 N.

Example
an = an-1 + an-2 for n 2 is a recurrence relation.
11/2/2006 Lecture9 gac1 1 11/2/2006 Lecture9 gac1 2

Solving Recurrence Relations

Often, we are interested in finding a general formula for the value of an that does not depend on the previous values. Example
Let a0 = 500 denote my initial financial investment, in . After n years in the bank at 10% interest, the total value of my investment will be given by the recurrence relation an = 1.1an-1, for n 1. We can solve this recurrence relation to obtain an = 500(1.1)n, which does not depend on previous values of the sequence.
11/2/2006 Lecture9 gac1 3

The Substitution Method

The substitution method for recurrence solution is essentially a guessing method.
Guess the form of a solution, and try to prove it holds using induction.

Example
an = 2an/2 + n, for n > 1. (note the notation x for floor(x) ) We may have seen similar recurrences before, and be able to guess a solution an = O(n log2 n). Assume this is true for n/2. Then an 2c n/2 log n/2 + n for large enough n.
11/2/2006 Lecture9 gac1 4

The Substitution Method

so an cn log2(n/2) + n = cn log2 n - cn + n cn log2 n for c 1. Thus an = O(n log2 n).
(A detail: note that we required c 1 to make this proof work, but this is not a problem, as c was the constant from the big-O notation. Therefore the big-O approximation will still hold for any c > c. We can instead choose a c large enough to ensure c 1. It would have caused us a problem if the requirement was c 1).

Linear Homogeneous Recurrences

Definition
A linear homogeneous recurrence relation of degree k with constant coefficients is a recurrence relation of the form an = c1an-1 + c2an-2 + + ckan-k, where c1, c2, , ck are real numbers, and ck 0.

Linear homogeneous recurrence relations with constant coefficients are important

there are general methods to find their solution.

Of course substitution has a major disadvantage

you need to guess the answer! (how many of you would have guessed an = O(n log2 n)?)

Example
Our bank interest example an = 1.1an-1 is a linear homogeneous recurrence relation of degree 1 with constant coefficients.
11/2/2006 Lecture9 gac1 6

We need to turn our attention to methods that dont require guesswork.

11/2/2006 Lecture9 gac1 5

Degree 1 Linear Homogeneous

Indeed, our bank interest example suggests a simple general theorem for the degree 1 case. Theorem
an = cn is a solution to the recurrence relation an = can-1. The value of can can be found from the initial condition a0.

Degree 2 Linear Homogeneous

We will also prove a theorem dealing with the degree 2 case. Theorem
Let c1 and c2 be real numbers. Suppose r2 c1r c2 = 0 has two distinct roots r1 and r2. Then an = 1r1n + 2r2n is a solution to the recurrence relation an = c1an-1 + c2an-2. The values of 1 and 2 can be found from the initial conditions a0 and a1.

Proof
We can prove this with the substitution method. Assume true for n 1. We will show it holds for n. can-1 = ccn-1 = cn = an. It only remains to determine . Substituting n = 0, gives = a0.
11/2/2006 Lecture9 gac1 7

Proof
Assume true for n-1 and n-2. We will show it to hold for n. c1an-1 + c2an-2 = c1(1r1n-1 + 2r2n-1) + c2 (1r1n-2 + 2r2n-2) = 1r1n-2 (c1r1 + c2) + 2r2n-2(c1r2 + c2). But c1r1 + c2 = r12 and c1r2 + c2 = r22. So c1an-1 + c2an-2 = 1r1n-2r12 + 2r2n-2r22 = 1r1n + 2r2n = an.
11/2/2006 Lecture9 gac1 8

Degree 2 Linear Homogeneous

It remains to determine 1 and 2. Substituting n = 0 and n = 1 into an = 1r1n + 2r2n gives 1 + 2 = a0 and 1r1 + 2r2 = a1. So a1 = 1r1 + (a0 - 1)r2. So for r1 r2, 1 = (a1 a0r2)/(r1 r2). And also 2 = a0 - (a1 a0r2)/(r1 r2) = (a0r1 a1)/(r1 r2).

Degree 2 Linear Homogeneous

First, we note c1 = 1, c2 = 1, so we form the equation r2 r 1 = 0. This has solutions r = (1 5)/2. These are two distinct solutions, r1 = (1 + 5)/2 and r2 = (1 - 5)/2, so we can apply the previous theorem. Following through the arithmetic leads to

1 =

5+ 5 5 5 , 2 = 10 10
n n

Example
Obtain a solution to the recurrence an = an-1 + an-2, with a0 = 1, a1 = 1?
note that this is the famous Fibonacci sequence.
11/2/2006 Lecture9 gac1 9

Thus the solution is given by

5 + 5 1+ 5 5 5 1 5 + an = 10 2 10 2
11/2/2006 Lecture9 gac1

10

Linear Non-homogeneous Recurrences

Definition
A linear non-homogeneous recurrence relation of degree k with constant coefficients is a recurrence relation of the form an = c1an-1 + c2an-2 + + ckan-k + F(n), where c1, c2, , ck are real numbers, and ck 0.

Non-Homogeneous Recurrences
Proof
Assume true for n-1. We will show it holds for n. can-1 + k = c(cn-1 + k/(1 c)) + k = cn + k( 1 + c/(1 c) ) = cn + k/(1 c) = an. It remains to determine the value of from initial condition a0. Substituting n = 0, we obtain a0 = + k/(1 c), so = a0 - k/(1 c).

There are many results on these equations, but we will only look at the particularly important case when F(n) is a constant. Firstly, lets consider the degree 1 case. Theorem
Let c be a real number with c 1. Then an = cn + k/(1 c) is a solution to the recurrence relation an = can-1 + k. The value of can can be found from the initial condition a0.
11/2/2006 Lecture9 gac1 11

We will also consider the special case when c = 1

this case is just an arithmetic progression

Theorem
Then an = + nk is a solution to the recurrence relation an = an-1 + k. The value of can can be found from the initial condition a0.
11/2/2006 Lecture9 gac1 12

Non-Homogeneous Recurrences
Proof
Assume true for n-1. We will show it holds for n. an-1 + k = + (n 1)k + k = + nk = an. It remains to determine the value of from initial condition a0. Substituting n = 0, we obtain = a0.

Linear Non-homogeneous Recurrences

Proof
Again, we will assume this to be true for n-1 and n-2, and demonstrate it to be true for n. c1an-1 + c2an-2 + k = c1(1r1n-1 + 2r2n-1 k/(c1 + c2 1)) + c2 (1r1n-2 + 2r2n-2 k/(c1 + c2 1)) + k = 1r1n-2(c1r1 + c2) + 2r2n-2(c1r2 + c2) + k k(c1 + c2)/(c1 + c2 1) = 1r1n + 2r2n k/(c1 + c2 1) = an. It remains only to determine the values of 1 and 2 from the initial conditions a0 and a1.
11/2/2006 Lecture9 gac1 14

We can now proceed to look at the degree 2 case. Theorem

Let c1 and c2 be real numbers with c1 + c2 1. Suppose r2 c1r c2 = 0 has two distinct roots r1 and r2. Then an = 1r1n + 2r2n k/(c1 + c2 1) is a solution to the recurrence relation an = c1an-1 + c2an-2 + k. The values of 1 and 2 can be found from the initial conditions a0 and a1.
11/2/2006 Lecture9 gac1 13

Linear Non-Homogeneous Recurrences

For simplicity, let us define p = k/(c1 + c2 1). Substituting n = 0 and n = 1 gives
a0 = 1 + 2 + p and a1 = 1r1 + 2r2 + p

Non-Homogeneous Example
We will try to find a solution for the recurrence
an = an-1 + 2an-2 + 1, with a0 = 1, a1 = 1.

First, we note c1 = 1, c2 = 2, so we form the equation r2 r 2 = 0.

This has solutions r1 = 2 and r2 = -1.

So we have
a1 = 1r1 + (a0 1 p)r2 + p 1(r1 r2) = a1 a0 r2 + p(r2 1).

The solutions are distinct, and c1 + c2 1, so the theorem will help us find a solution. We obtain
an = 12n + 2(-1)n .

Thus
1 = (a1 a0 r2 + p(r2 1))/(r1 r2), since r1 r2, and 2 = a0 1 p.

The initial conditions give

1 = (a1 a0 r2 - (r2 1))/(r1 r2) = 1 2 = a0 1 + = .

The complete solution is then

11/2/2006 Lecture9 gac1 15 11/2/2006 Lecture9

1 1 an = 2 n + (1) n 2 2
gac1 16

Test Your Knowledge

Which of these are linear homogeneous recurrence relations with constant coefficients?
(a) (b) (c) (d) an = 3an-2. an = an-12. an = an-1 + 2an-3. an = an/2 + 1.

Summary
This lecture has covered
What is a recurrence relation? The substitution method. Solution of some linear homogeneous recurrence relations: degree 1 and degree 2. Solution of some linear non-homogeneous recurrence relations: degree 1 and degree 2.

Which of these linear recurrence relations with constant coefficients can be solved using the techniques in this lecture?
(a) (b) (c) (d) an = 2an-1 2an-2. an = 4an-1 + 4an-2. an = 3an-1 + 2an-2 + 1. an = 2an-1 2an-2 + 1.
Lecture9 gac1 17

We have ploughed through some [fairly unpleasant?] algebra, but it will be worth it for the next lecture
we can use these relations to analyse recursive algorithms.
11/2/2006 Lecture9 gac1 18

11/2/2006

Analysing Recursive Algorithms I

In this lecture, we will apply recurrence relations to the analysis of execution time for recursive algorithms. The lecture will cover
How to analyse recursive execution time Recursive and iterative factorial algorithms Recursive and iterative Fibonacci algorithms

Recursive Algorithms
Definition
An algorithm is called recursive if it solves a problem by reducing it to one or more instances of the same problem with smaller input.

We will start by looking at a recursive algorithm to calculate the factorial.

function factorial(n : non-negative integer) begin if n = 0 then result := 1 else result := n * factorial( n 1 ) end

11/2/2006

Lecture10

gac1

11/2/2006

Lecture10

gac1

Execution Time Analysis

To analyse the run-time of this algorithm, we note that
all inputs require a comparison (n = 0). if n = 0, no calculation is required just a store operation. if n 0, a multiplication is required, together with a call to factorial(n-1).

Execution Time Analysis

We may apply the theorems of the last lecture to solve this recurrence.
it is a degree 1 linear non-homogeneous recurrence relation with constant coefficients

The recurrence is of the form an = an-1 + k, so the solution has the form an = + nk.
k = 1, a0 = 0 So an = n. n multiplications are performed by a call to factorial(n). The run time will be O(n).

Let us denote by an the total number of multiplication operations performed by factorial(n). We have a0 = 0 and an = 1 + an-1, for n > 0.
A recurrence relation!
11/2/2006 Lecture10 gac1 3

11/2/2006

Lecture10

gac1

Recursive Fibonacci
For a different type of example, consider the following algorithm for calculating the Fibonacci sequence recall that fib(1) = 1, fib(2) = 1, and fib(n) = fib(n-1) + fib(n-2) for n > 2. function fib(n : positive integer) begin if n <= 2 then result := 1 else result := fib( n 1 ) + fib( n 2 ) end
11/2/2006 Lecture10 gac1 5

Recursive Fibonacci
We may wish to determine the number of addition operations required by this algorithm.
Let us denote this an.

An addition is only performed when n > 2. In this case

fib(n-1) is called, so we must include any additions performed by this. fib(n-2) is called, so we must include any additions performed by this. an extra addition is performed to add fib(n-1) and fib(n-2).

We obtain an = 1 + an-1 + an-2, for n > 2, with initial conditions a1 = 0 a2 = 0.

11/2/2006 Lecture10 gac1 6

Recursive Fibonacci
This is a linear non-homogeneous recurrence relation of degree 2 with constant coefficients.
Compare with an = c1an-1 + c2an-2 + k. We have c1 = 1, c2 = 1, k = 1.

Recursive Fibonacci
Substituting n = 1 and n = 2 gives
1r1 + 2r2 1 = 0 (*) 1r12 + 2r22 1 = 0 (**)

We can use the result of the previous lecture if

c1 + c2 1 r2 c1r c2 = 0 has two distinct roots (O.K.) (O.K.)

Multiplying (*) by r1 and subtracting from (**) gives:

2r22 1 2r2r1 + r1 = 0, i.e. 2 = (1 r1)/(r22 r2r1) = -1/5 and 1 = 1/5 (after some work!)

The two roots of this quadratic are r1 = (1 + 5)/2 and r2 = (1 - 5)/2, and the solution to the recurrence has the form an = 1r1n + 2r2n k/(c1 + c2 1) = 1r1n + 2r2n 1.
11/2/2006 Lecture10 gac1 7

The total number of additions performed by a call to fib(n) is therefore:

n n 1 1 + 5 1 5 1 an = 5 2 2

The total execution time is exponential in n.

11/2/2006 Lecture10 gac1 8

Iterative Factorial
We can compare the recursive versions of these algorithms to their iterative equivalents. An iterative factorial is shown below.
function factorial(n : non-negative integer) begin result := 1 for i = 1 to n result := result * i end

Iterative Factorial
Note that this doesnt mean the two version have the same run time.
The constant of proportionality in the O(n) hides the details. In practice, function calls (stack manipulation, etc.) often have a large overhead. But the result is still valuable: for large enough n, the execution time of the two ways of calculating factorial are out by at most a constant multiplicative factor.

This algorithm performs n multiplications, and has an execution time of O(n).

the same as the recursive version.
11/2/2006 Lecture10 gac1 9 11/2/2006 Lecture10 gac1 10

Iterative Fibonacci
An iterative version of the Fibonacci generator is shown below.
function fib(n : positive integer) begin x := 0 result := 1 for i = 1 to (n 1) begin z := x + result x := result result := z end end
Lecture10 gac1

Iterative Fibonacci
Addition is only performed once in the loop, so a total of (n-1) additions are performed. Apart from the initialization of x, result, and i (which are O(1)), the remaining operations take O(n) time.
the execution time is linear in n.

This compares very favourably with the recursive version: linear vs. exponential run-time! We can draw a valuable lesson from this example
recursion often makes the algorithm neater and easier to read, but it must be used with caution, as exponential run-times can result.

11/2/2006

11

11/2/2006

Lecture10

gac1

12

Test Your Knowledge

The algorithm below operates on a portion a[low to high] of an array a. When called, low <= high. Write a recurrence relation, expressing the number of addition operations in the algorithm below in terms of the number of elements n = high low + 1.
function atest(low: integer, high: integer) begin if low = high result := a[low]; else begin temp1 := atest( a, low+1, high ); temp2 := atest( a, low, high-1 ); result := temp1 + temp2; end end
11/2/2006 Lecture10 gac1 13

Summary
This lecture has covered
How to analyse recursive execution time Recursive and iterative factorial algorithms Recursive and iterative Fibonacci algorithms

We can now analyse some recursive algorithms. There is an important class of recursive algorithms, known as divide and conquer algorithms, that we cant yet analyse. For this, we need more recurrence relations.
11/2/2006 Lecture10 gac1 14

Divide-and-Conquer Recurrences
This lecture will introduce divide-and-conquer recurrence relations
used to analyse divide-and-conquer recursive algorithms

Divide-and-Conquer Recurrences
A divide-and-conquer algorithm breaks up a large problem into several proportionally smaller versions of the same problem.
this approach is often used for sorting, for example detailed examples in the next lecture.

The lecture will cover

divide-and-conquer recurrences a theorem for a common form of divide-and-conquer recurrence the Master Theorem

Analysing divide-and-conquer algorithms results in divide-and-conquer recurrences. Definition

A divide-and-conquer recurrence relation is an equation of the form f(n) = a f(n/b) + g(n).

11/2/2006

Lecture11

gac1

11/2/2006

Lecture11

gac1

The Growth of D&C Recurrences

For simplicity, let us consider how this recurrence grows when n = bk for some k Z+. In this case, we can write
f (n) = af (n / b) + g (n) = a 2 f (n / b 2 ) + ag (n / b) + g (n) = a 3 f (n / b 3 ) + a 2 g (n / b 2 ) + ag (n / b) + g (n) = a k f (1) + a j g (n / b j )
j =0 k 1

A Common Special Case

A common special case arises when g(n) is constant. Theorem
Let a 1 be a real number, b > 1 be an integer, and c > 0 be a real number. Let f be an increasing function that satisfies the recurrence relation f(n) = a f(n/b) + c whenever n is divisible by b. Then if a > 1, f(n) is O (n logb a ) Alternatively, if a = 1, f(n) is O(log n)

We will use this expansion to analyse special cases of the general D&C recurrence relation.
11/2/2006 Lecture11 gac1 3

Quick test:
Why do I need to write the base of the log in the first case, but not in the second?

11/2/2006

Lecture11

gac1

Special Case: Proof

To prove this, we will consider two cases separately
where n = bk for some k Z+, and otherwise.

Special Case: Proof

For the case where a = 1, aj = 1. So in this case, we have
f (n) = f (1) + ck = f (1) + c log b n

First, let us consider our expansion of the recurrence, when g(n) is constant.
f (n) = a k f (1) + a j g (n / b j )
j =0 k 1

this is the sum of an O(1) function and an O(log n) function, and is therefore an O(log n) function.

Let us now consider what happens if n is not an integer power of b.

In this case, we can sandwich n between two integer powers of b, i.e. bk < n < bk+1. f(n) is an increasing function, so f(n) f(bk+1) = f(1) + c(k+1) = (f(1) + c) + ck.

= a k f (1) + c a j
j =0

k 1

11/2/2006

Lecture11

gac1

11/2/2006

Lecture11

gac1

Special Case: Proof

But f(n) (f(1) + c) + ck (f(1) + c) + c logb n The RHS is still the sum of an O(1) and an O(log n) function, and so the LHS is also O(log n).

Special Case: Proof

Using the standard formula for the sum of k terms of a G.P. [can you remember it?], we obtain

f (n) = a k f (1) + c(a k 1) /(a 1) = a k [ f (1) + c /(a 1)] + c /(1 a )

but a k = a logb n = a log a n / log a b = n1/ log a b = n logb a so f (n) = n logb a [ f (1) + c /( a 1)] + c /(1 a ) this is the sum of an O(1) function and an O(n log a ) function, and is therefore an O(n log a ) function.
b b

Summary so far:
We have shown that for a = 1, f(n) is O(log n). We now need to examine the case a > 1.

Let us now consider the general expansion

f (n) = a f (1) + c a
k j =0 k 1 j

the second term is a geometric progression

11/2/2006 Lecture11 gac1 7

We finally have to consider the case where a > 1 and n is not an integer power of b.
11/2/2006 Lecture11 gac1 8

Special Case: Proof

Again, we will use the fact that we can sandwich n between two such powers: bk < n < bk+1. Since f(n) is increasing, we obtain

Examples
Example 1
Find a big-O expression for an increasing function satisfying f(n) = 5f(n/2) + 3. Compare with f(n) = a f(n/b) + g(n). This is a divide-andconquer recurrence relation with:
constant g(n) = c = 3. a = 5 1. b = 2 > 1.

f (n) f (b k +1 ) = a k +1 [ f (1) + c /( a 1)] + c /(1 a ) = a k [ f (1)a + ac /( a 1)] + c /(1 a )

n logb a [ f (1)a + ac /(a 1)] + c /(1 a )

Once again, this is the sum of an O(1) function and an O(n log a ) function, and is therefore an O(n log a ) function.
b b

We can therefore apply the theorem. log a log 5 2.33 f(n) is O(n ) = O(n ) or O(n ). Note that log2 5 = 2.32 to 2 decimal places. But for a big-O expression, we have used O(n2.33).
b 2

remember: big-O is an upper bound!

11/2/2006 Lecture11 gac1 9 11/2/2006 Lecture11 gac1 10

Examples
A plot of the function f(n) is shown below (*).
1800 1600

Examples
Example 2
Find a big-O expression for an increasing function satisfying f(n) = f(n/2) + 16. Compare with f(n) = a f(n/b) + g(n). This is a divide-andconquer recurrence relation with:
constant g(n) = c = 16. a = 1 1. b = 2 > 1.

1.5n2.33 f(n) 2.5n2

1400

1200

1000

800

600

400

200

10

12

14

16

18

20

n
(*) Initial conditions: f(1) = 0.24, f(3) = 12.93, f(5) = 42.52, f(7) = 93.13, f(9) = 167.26, f(11) = 266.96, f(13) = 393.99, f(15) = 549.91, f(17) = 736.11, f(19) = 954.88
11/2/2006 Lecture11 gac1 11

We can therefore apply the theorem. f(n) is O(log n).

11/2/2006

Lecture11

gac1

12

Examples
A plot of the function f(n) is shown below (*).
80

The Master Theorem

We can now turn our attention to cases when g(n) may be non-constant. There is a famous generalization of the previous theorem for cases where g(n) = cnd, for c > 0 and d 0.
This general case includes the previous theorem (d = 0).

16.5 log2 n f(n)

70

60

50

40

This generalization is known as The Master Theorem

It has this title because it is very useful in algorithm analysis. The proof of the Master theorem is long; we will not prove it. For those interested, Cormen, et al., have a proof [not examinable].
0 2 4 6 8 10 12 14 16 18 20

30

20

10

n
(*) Initial conditions: f(1) = 0, f(3) = 25.44, f(5) = 37.12, f(7) = 44.96 , f(9) = 50.72, f(11) = 55.36, f(13) = 59.2, f(15) = 62.56, f(17) = 65.44, f(19) = 68.
11/2/2006 Lecture11 gac1 13 11/2/2006 Lecture11 gac1 14

The Master Theorem

Theorem
Let a 1 be a real number, b > 1 be an integer, c > 0 be a real number, and d 0 be a real number. Let f be an increasing function that satisfies the recurrence relation f(n) = a f(n/b) + cnd, whenever n = bk, where k is a positive integer. If a < bd, then f(n) is O(n d ) . Alternatively, if a = bd, then f(n) is O(n d log n). log a Finally, if a > bd, then f(n) is O(n ).
b

Example
Example
Find a big-O expression for an increasing function f(n) satisfying f(n) = 8 f(n/2) + n2. Compare with f(n) = a f(n/b) + g(n). This is a divide-andconquer recurrence relation with:
g(n) = n2. This has the form g(n) = cnd, with c = 1 > 0 and d = 2 0. a = 8 1. b = 2 > 1.

We can therefore apply the Master Theorem. We have a > bd (8 > 4), so f(n) is O(n log 8 ) = O(n 3 ) .
2

11/2/2006

Lecture11

gac1

15

11/2/2006

Lecture11

gac1

16

Example
A plot of the function f(n) is shown below (*).
800

Test Your Knowledge

Provide a big-O estimate for f(n) satisfying each of the following recurrence relations, given that f(n) is an increasing function.
f(n) = f(n/4) + 1 f(n) = 5f(n/4) + 1 f(n) = f(n/4) + 3n f(n) = 5f(n/4) + 3n f(n) = 4f(n/4) + 3n

0.1n3 f(n)

700

600

500

400

300

200

100

10

12

14

16

18

20

n
(*) Initial conditions: f(1) = 1, f(3) = 10, f(5) = 25, f(7) = 60 , f(9) = 100, f(11) = 150, f(13) = 220, f(15) = 280, f(17) = 380, f(19) = 460.
11/2/2006 Lecture11 gac1 17 11/2/2006 Lecture11 gac1 18

Summary
This lecture has covered
divide-and-conquer recurrences a theorem for a common form of divide-and-conquer recurrence the Master Theorem

In the next lecture, we will put these tools to work analysing divide-and-conquer algorithms.

11/2/2006

Lecture11

gac1

19

Analysing Recursive Algorithms II

In this lecture, we will analyse the worst-case execution time of divide-and-conquer algorithms. The lecture will cover
obtaining divide-and-conquer recurrences example algorithms
binary search merge sort

Obtaining Recurrences
Divide-and-conquer recurrence relations occur when analysing the time-complexity of divide-andconquer algorithms. A divide-and-conquer recurrence relation f(n) = a f(n/b) + g(n) results when the algorithm, operating on a problem of size n (multiple of b),
1. decides how to split up the input; 2. splits the input into smaller segments, each of size n/b, recursively operating on a of those segments; 3. combines the resulting outputs to make to overall output. steps 1+3 combined take time g(n).

11/2/2006

Lecture12

gac1

11/2/2006

Lecture12

gac1

Binary Search
Binary Search is a classic example
we looked at an iterative version in Lecture 8 a slightly different recursive version is shown below: here the aim is to report success iff the value x is found in the (sorted) array a[ ]. procedure binary_search( integer x, integer a[n] )
1. begin if n = 1 then if a[1] = x then report success else report failure else if x < a[ n/2 ] then binary_search( x, a[1 to n/2]) else binary_search( x, a[n/2 + 1 to n]) end
Lecture12 gac1

Binary Search
Algorithm explanation
if the array is of length one, then we only need to look in one place to find x, i.e. a[1]. otherwise, we find the approximate mid-point, n/2. the array is sorted, so if x is less than the value at the mid-point, look in the first-half; otherwise, look in the second-half.

When n is a multiple of 2, the algorithm

splits the input into smaller segments, each of size no more than n/2; recursively operates on one of these
which one is decided by line 2

2.

sounds like a divide-and-conquer recurrence!

3 11/2/2006 Lecture12 gac1 4

11/2/2006

Binary Search
We can find a big-O estimate for the number of comparison operations. The function g(n)?
g(n) here models the overhead in the execution of line 1 and line 2. each one performs a comparison, so g(n) = 2.

Binary Search
Solution of this recurrence comes from the Master Theorem (or our special case).
Compare with f(n) = a f(n/b) + cnd. We have a = 1, b = 2, c = 2, d = 0. Since a = bd, f(n) is O(nd log n) = O(log n). Binary search is logarithmic-time.

The overall recurrence relation

Since the input is split into segments of size n/2, b = 2. Since only one of these is operated on, a = 1. We have f(n) = f(n/2) + 2.
11/2/2006 Lecture12 gac1 5 11/2/2006 Lecture12 gac1 6

Merge Sort
Merge sort is a famous algorithm for sorting arrays.
it has better asymptotic performance than bubble sort (we will prove this).

Merge Sort
procedure merge_sort( real a[n] ) begin if n = 1 then result := a else L1 = merge_sort( a[ 1 to n/2 ] ) L2 = merge_sort( a[ n/2+1 to n ] ) result := merge( L1, L2 ) end

The basic idea

start with an (unsorted) array of n numbers. divide the array into half: sort each half separately (recursively). combine the two sorted halves into a sorted whole.
[4,10,9,1]

split

[4,10]

split

[4]

merge

[4, 10]

merge

[1,4,9,10]

[10] [9,1]
11/2/2006

The algorithm is shown in pseudo-code above. For this algorithm to be efficient, we need an efficient implementation of merge().

[9] [1]
Lecture12

[1,9]
gac1 7 11/2/2006 Lecture12 gac1 8

Merge Procedure
The merge procedure can take advantage of the fact that its inputs are sorted arrays.
procedure merge( real L1[p], L2[q] ) begin count1 := 1 count2 := 1 countM := 1 do if count2 > q or L1[count1] < L2[count2] then begin result[countM] = L1[count1] count1 := count1 + 1 end else begin result[countM] = L2[count2] count2 := count2 + 1 end countM := countM + 1 while countM <= p + q end
Lecture12 gac1

Merge Procedure
Algorithm explanation
at each iteration of the while loop, it fills a single element of result[ ]. the source of this element depends on whether the current element of L1 or L2 is smallest the smallest is taken. the counters keep track of the current element being processed on each of the three lists.

1.

count1
[4, 10]

count1
[4, 10]

count1
[4, 10]

count1
[4, 10]

count1
[4, 10]

count2
[1,9]

count2
[1,9]

count2
[1,9]

count2
[1,9]

count2
[1,9]

countM
[?,?,?,?]
9 11/2/2006

countM
[1,?,?,?]

countM
[1,4,?,?]
Lecture12

countM
[1,4,9,?]
gac1

countM
[1,4,9,10]

2.

11/2/2006

iteration

10

Merge Procedure
Algorithm analysis
before we can estimate the execution time of merge_sort, we need to find the execution time of merge we will count the number of comparisons
the loop iterates p+q times there are at most 2(p+q) comparisons on line 1 the comparison on line 2 is performed p+q times

Merge Sort
We can now return to the analysis of merge_sort
when n is even, merge_sort
1. performs a comparison (n = 1). 2. performs two other merge_sorts on problems of size n/2. 3. calls merge, which performs at most 3n comparisons.

writing a recurrence for the number of comparisons gives f(n) = 2f(n/2) + 3n + 1.

this doesnt fit the form solved by the Master Theorem.

the total number of comparisons is therefore at most 3(p+q). merge is called with p = n/2, q = n - n/2. So the total number of comparisons is at most 3n.
11/2/2006 Lecture12 gac1 11

But we can combine steps 1 and 3, and say that merge_sort

1. performs two other merge_sorts on problems of size n/2. 2. performs some additional housekeeping which takes at most 4n comparisons.
11/2/2006 Lecture12 gac1 12

Merge Sort
Writing this as a recurrence relation, f(n) = 2f(n/2) + 3n
compare with f(n) = a f(n/b) + cnd, to get a = 2, b = 2, c = 4, d = 1. we have a = bd, so f(n) is O(nd log n) = O(n log n), from the Master Theorem.

Test Your Knowledge

For the code below, give a big-O estimate of
the number of additions the number of comparisons (including the for-loop comparisons) the execution time
procedure atest( real a[n] ) begin if n = 1 result := a[1] else begin temp := 0 stride := n/3 for i := 0 to 2 temp := temp + atest( a[ i*stride+1 to (i+1)*stride ] ) end end
11/2/2006 Lecture12 gac1 14

Note that the simplification we made (3n + 1 4n) has not affected the tightness of the bound in this case
f(n) = 2f(n/2) + 3n gives exactly the same bound.

Conclusion: merge_sort performs O(n log n) comparisons, but bubble_sort performs n(n-1)/2.
merge_sort is asymptotically faster.
11/2/2006 Lecture12 gac1 13

Summary
This lecture has covered
obtaining divide-and-conquer recurrences example algorithms
binary search merge sort

This was the last lecture on recurrence relations. In the next lecture we will look at the ideas of computability theory.
11/2/2006 Lecture12 gac1 15