Analysis of Algorithms

Issues:

Correctness Time efficiency Space efficiency Optimality

Approaches:

Theoretical analysis Empirical analysis

Design and Analysis of Algorithms - Chapter 2 1

Theoretical analysis of time efficiency

Time efficiency is analyzed by determining the number of repetitions of the basic operation as a function of input size Basic operation: the operation that contributes most towards the running time of the algorithm.
input size

T(n) copC(n)
running time execution time for basic operation Number of times basic operation is executed
2

Design and Analysis of Algorithms - Chapter 2

Input size and basic operation examples

Problem Input size measure Basic operation
Key comparison Floating point multiplication Floating point multiplication Visiting a vertex or traversing an edge
3

Search for key in list Number of items in of n items list n Multiply two matrices of floating point numbers Compute an Dimensions of matrices n #vertices and/or edges
Design and Analysis of Algorithms - Chapter 2

Graph problem

Empirical analysis of time efficiency

Select

a specific (typical) sample of inputs Use physical unit of time (e.g., milliseconds) OR Count actual number of basic operations Analyze the empirical data

Design and Analysis of Algorithms - Chapter 2

Best-case, average-case, worst-case

For some algorithms efficiency depends on type of input: Worst case: W(n) maximum over inputs of size n Best case: B(n) minimum over inputs of size n Average case: A(n) average over inputs of size n
Number of times the basic operation will be executed on typical input NOT the average of worst and best case Expected number of basic operations repetitions considered as a random variable under some assumption about the probability distribution of all possible inputs of size n
Design and Analysis of Algorithms - Chapter 2 5

Example: Sequential search

Problem:

Given a list of n elements and a search key K, find an element equal to K, if any. Algorithm: Scan the list and compare its successive elements with K until either a matching element is found (successful search) or the list is exhausted (unsuccessful search) Worst case Best case Average case
Design and Analysis of Algorithms - Chapter 2 6

Types of formulas for basic operation count

Exact

formula e.g., C(n) = n(n-1)/2 Formula indicating order of growth with specific multiplicative constant e.g., C(n) 0.5 n2 Formula indicating order of growth with unknown multiplicative constant e.g., C(n) cn2
Design and Analysis of Algorithms - Chapter 2 7

Order of growth
Most

important: Order of growth within a constant multiple as n Example:

How much faster will algorithm run on computer that is twice as fast? How much longer does it take to solve problem of double input size?
See

table 2.1
Design and Analysis of Algorithms - Chapter 2 8

Table 2.1

Design and Analysis of Algorithms - Chapter 2

Asymptotic growth rate

A way of comparing functions that ignores constant factors and small input sizes O(g(n)): class of functions f(n) that grow no faster than g(n) (g(n)): class of functions f(n) that grow at same rate as g(n) (g(n)): class of functions f(n) that grow at least as fast as g(n)

see figures 2.1, 2.2, 2.3

Design and Analysis of Algorithms - Chapter 2 10

Big-oh

Design and Analysis of Algorithms - Chapter 2

11

Big-omega

Design and Analysis of Algorithms - Chapter 2

12

Big-theta

Design and Analysis of Algorithms - Chapter 2

13

Establishing rate of growth: Method 1 using limits

0 order of growth of T(n) ___ order of growth of g(n) order of growth of T(n) ___ order of growth of g(n) order of growth of T(n) ___ order of growth of g(n)

limn T(n)/g(n) =

c>0

Examples: 10n

vs.

2 n2

n(n+1)/2
logb n

vs.
vs.

n2
logc n
14

Design and Analysis of Algorithms - Chapter 2

LHpitals rule
If

limn f(n) = limn g(n) = The derivatives f, g exist,

Then lim
n

f(n) g(n) =

lim f (n) n g (n)

Example: logn vs. n

Design and Analysis of Algorithms - Chapter 2 15

Establishing rate of growth: Method 2 using definition

f(n) is O(g(n)) if order of growth of f(n) order of growth of g(n) (within constant multiple) There exist positive constant c and non-negative integer n0 such that f(n) c g(n) for every n n0 Examples: 10n is O(2n2) 5n+20 is O(10n)

Design and Analysis of Algorithms - Chapter 2

16

Basic Asymptotic Efficiency classes

1 log n n n log n n2 n3 2n n! constant logarithmic linear n log n quadratic cubic exponential factorial
17

Design and Analysis of Algorithms - Chapter 2

Time efficiency of nonrecursive algorithms

Steps in mathematical analysis of nonrecursive algorithms: Decide on parameter n indicating input size Identify algorithms basic operation Determine worst, average, and best case for input of size n Set up summation for C(n) reflecting algorithms loop structure Simplify summation using standard formulas (see Appendix A)
Design and Analysis of Algorithms - Chapter 2 18

A simple example
Algorithm MaxElement(A[0..n-1])
// Determines the value of the largest element in a given array // Input: An array A[0..n-1] of real numbers // Output: The value of the largest element in A

maxval A[0] for i1 to n-1 do if A[i]>maxval maxval A[i] return maxval

Design and Analysis of Algorithms - Chapter 2 19

Another simple example

Algorithm UniqueElement(A[0..n-1])
// Check whether all the elements in a array are distinct // Input: An array A[0..n-1] of real numbers // Output: Returns true if all the elements are distinct

for i0 to n-2 do for ji+1 to n-1 do if A[i]=A[j] return false return true
Design and Analysis of Algorithms - Chapter 2 20

Other examples
Matrix

multiplication Selection sort Insertion sort Mystery Algorithm

Design and Analysis of Algorithms - Chapter 2

21

Matrix multipliacation

Design and Analysis of Algorithms - Chapter 2

22

A third simple example

Algorithm Binary(n)
// Input: A positive decimal integer n // Output: The number of binary digits in ns binary representation

count1 while n>1 do count count+1 n n/2 return count

Design and Analysis of Algorithms - Chapter 2 23

Selection sort

Design and Analysis of Algorithms - Chapter 2

24

Insertion sort

Design and Analysis of Algorithms - Chapter 2

25

Mystery algorithm
for i := 1 to n-1 do max := i ; for j := i+1 to n do if |A[j,i]| > |A[max,i ]| then max := j ; for k := i to n+1 do swap A[i,k] with A[max,k]; for j := i+1 to n do for k := n+1 downto i do A[j,k] := A[j,k] - A[i,k]*A[j,i]/A[i,i] ;
Design and Analysis of Algorithms - Chapter 2 26

Time efficiency of recursive algorithms

Steps in mathematical analysis of recursive algorithms: Decide on parameter n indicating input size Identify algorithms basic operation Determine worst, average, and best case for input of size n Set up a recurrence relation and initial condition(s) for C(n)-the number of times the basic operation will be executed for an input of size n (alternatively count recursive calls). Solve the recurrence to obtain a closed form or estimate the order of magnitude of the solution (see Appendix B) Design and Analysis of Algorithms - Chapter 2 27

Recursive evaluation of n !
Definition: n ! = 1*2**(n-1)*n Recursive definition of n!: n ! = (n-1)!*n Algorithm: if n=0 then F(n) := 1 else F(n) := F(n-1) * n return F(n) Recurrence for number of multiplications: M(n) = M(n-1) + 1 for n>0 M(0) = 0 Method: backward substitution

Design and Analysis of Algorithms - Chapter 2 28

Towers of Hanoi

Problem: to count the number of moves Recurrence for number of moves: M(n) = M(n-1) + 1 + M(n-1) M(1) = 1 for n>1

Alternatively: construct a tree for the number of recursive calls

Comment: inherent inefficiency

Design and Analysis of Algorithms - Chapter 2

29

Revisiting the Binary algorithm

Algorithm BinRec(n)
// Input: A positive decimal integer n // Output: The number of binary digits in ns binary representation if n=1 return 1 else return BinRec(n/2)+1

Recurrence for number of additions:

A(n) = A(n/2) + 1 for n>1

A(1) = 0

Apply the smoothness rule: n=2k

Design and Analysis of Algorithms - Chapter 2 30

Computing Fibonacci numbers (1)

1. 2. 3.

Definition based recursive algorithm Nonrecursive brute-force algorithm Explicit formula algorithm

4.

Algorithm based on matrix multiplications

Design and Analysis of Algorithms - Chapter 2

31

Computing Fibonacci numbers (2)

Leonardo Fibonacci, 2002. The Fibonacci sequence: 0, 1, 1, 2, 3, 5, 8, 13, 21, Fibonacci recurrence:

F(n) = F(n-1) + F(n-2) F(0) = 0 F(1) = 1

Algorithm F(n):
// Computes the n-th Fibonacci number recursively // Input: A nonnegative integer n // Output: The n-th Fibonacci number if n <= 1 return n else return F(n-1)+F(n-2)
Design and Analysis of Algorithms - Chapter 2

32

Computing Fibonacci numbers (3)

2nd order linear homogeneous recurrence relation with constant coefficients Characteristic equation Solution F(n) = n/5 n/5 = (1+5)/2 (golden ratio) = (1-5)/2

F(n) n/5

Construct a tree for the number of recursive calls Recursive algorithms solutions, not a panacea
Design and Analysis of Algorithms - Chapter 2 33

Computing Fibonacci numbers (4)

Algorithm F(n): // Computes the n-th Fibonacci number iteratively // Input: A nonnegative integer n // Output: The n-th Fibonacci number F[0] 0; F[1] 1; for i 2 to n do F[i] F[i-1] + F[i-2] return F(n) Complexity ?

Design and Analysis of Algorithms - Chapter 2 34

Computing Fibonacci numbers (5)

Alternatively algorithmic use of equation F(n) n/5 with smart rounding Relies on the exponentiation algorithm Solutions: (n), (logn)

Design and Analysis of Algorithms - Chapter 2

35

Computing Fibonacci numbers (6)

It holds that

F(n-1)

F(n)

0 1
1 1

F(n) F(n+1)

for n1,

Assuming an efficient way of computing matrix powers, solution in (logn)

Design and Analysis of Algorithms - Chapter 2

36

Important recurrence types:

One (constant) operation reduces problem size by one.

T(n) = T(n-1) + c T(1) = d Solution: T(n) = (n-1)c + d
linear

A pass through input reduces problem size by one.

T(n) = T(n-1) + cn T(1) = d Solution: T(n) = [n(n+1)/2 1] c + d

quadratic

One (constant) operation reduces problem size by half.

T(n) = T(n/2) + c T(1) = d Solution: T(n) = c lg n + d logarithmic

A pass through input reduces problem size by half.

T(n) = 2T(n/2) + cn T(1) = d Solution: T(n) = cn lg n + d n
Design and Analysis of Algorithms - Chapter 2

n log n
37

A general divide-and-conquer recurrence

T(n) = aT(n/b) + f (n)
1. 2.

where f (n) (nk)

3.

a < bk a = bk a > bk

T(n) (nk) T(n) (nk lg n ) T(n) (nlog b a)

Note: the same results hold with O instead of .

Design and Analysis of Algorithms - Chapter 2

38

Solving linear homogeneous recurrence relations with constant coefficients

Easy first: 1st order LHRRCCs:

C(n) = a C(n -1) C(0) = t Solution: C(n) = t an
A solution?: L(n) = r n

Extrapolate to 2nd order

L(n) = a L(n-1) + b L(n-2)

Characteristic equation (quadratic) Solve to obtain roots r1 and r2 e.g.: A(n) = 3A(n-1) - 2(n-2) General solution to RR: linear combination of r1n and r2n Particular solution: use initial conditions e.g.:A(0) = 1 A(1) = 3
Design and Analysis of Algorithms - Chapter 2 39