Analysis of Algorithms

Dr. Abu Sayed Md. Mostafizur Rahaman

Associate Professor
Department of Computer Science and Engineering
Jahangirnagar University

Theoretical analysis of time efficiency

Time efficiency is analyzed by determining the number of
repetitions of the basic operation as a function of input size

Basic operation: the operation that contributes most

towards the running time of the algorithm
input size

T(n) copC(n)
running time

execution time
for basic operation

Number of times
basic operation is
executed

Input size and basic operation examples

Problem

Input size measure

Basic operation

Searching for key in a

list of n items

Number of lists items,

i.e. n

Key comparison

Multiplication of two
matrices

Matrix dimensions or
total number of elements

Multiplication of two
numbers

Checking primality of
a given integer n

nsize = number of digits

Division
(in binary representation)

Typical graph problem #vertices and/or edges

Visiting a vertex or
traversing an edge
3

Best-case, average-case, worst-case

For some algorithms efficiency depends on form of input:
Worst case:

Cworst(n) maximum over inputs of size n

Best case:

Cbest(n) minimum over inputs of size n

Average case: Cavg(n) average over inputs of size n

Number of times the basic operation will be executed on typical input

NOT the average of worst and best case
Expected number of basic operations considered as a random variable
under some assumption about the probability distribution of all possible
inputs
4

Example: Sequential search

Worst case
n key comparisons
Best case
Average case

1 comparisons
(n+1)/2, assuming K is in A
5

Example: Sequential search (Average case)

Let the probability of successful search is equal to p
(0p1)
Let the probability of unsuccessful search is equal to 1-p
Cavg (n) =[1. p/n +2.p/n+ +i.p/n + + n.p/n] + n (1-p)
= p/n [1+2+3+ + i+ n] + n[1-p]
= p/n[{n(n+1)}/2] + n(1-p)
= p(n+1)/2 + n(1-p)
1. If p= 1 then successful search
2. If p=0 then unsuccessful search

Order of growth
Most important: Order of growth within a constant multiple
as n
Example:
How much faster will algorithm run on computer that is twice
as fast?

Example: C(n)= n(n-1)

T(2n)/T(n)={Cop C(2n)}/{Cop C(2n)}

Asymptotic order of growth

Order of growth:

Order of growth of an algorithms basic operation count as the

principal indicator of the algorithms efficiency.

Notations:

O (big oh)
(big omega), and
(big theta).

Assume

t (n) and g(n) can be any nonnegative functions defined on the set of
natural numbers.
t(n) will be an algorithms running time (usually indicated by its
basic operation count C(n)), and g(n) will be some simple function
to compare the count with.

Big-oh
Afunctiont(n)issaidtobeinO(g(n)),denotedt(n)O(g(n)),
ift(n)isboundedabovebysomeconstantmultipleofg(n)forall
largen,i.e.,ifthereexistsomepositiveconstantcandsome
nonnegativeintegern0suchthatt(n)cg(n)forallnn0.

Big-omega
Afunctiont(n)issaidtobein(g(n)),denotedt(n)(g(n)),if
t(n)isboundedbelowbysomepositiveconstantmultipleofg(n)
for all large n, i.e., if there exist some positive constant c and
somenonnegativeintegern0suchthatt(n)cg(n)forallnn0.

10

Big-theta
Afunctiont(n)issaidtobein(g(n)),denotedt(n) (g(n)),if
t(n)isboundedbothaboveandbelowbysomepositiveconstant
multiplesofg(n)foralllargen,i.e.,ifthereexistsomepositive
constants c1 and c2 and some nonnegative integer n0 such that
c2g(n)t(n)c1g(n)forallnn0.

11

Establishing order of growth using limits

0orderofgrowthofT(n)<orderofgrowthofg(n)
c>0orderofgrowthofT(n)=orderofgrowthof
g(n)
n)
orderofgrowthofT(n)>orderofgrowthofg(n)
Firsttwocases:
meanthatt(n)O(g(n)),
Lasttwocases:
meanthatt(n)(g(n)),
Secondcase:
meansthatt(n)(g(n)).
The limitbased approach is oftenmore convenient than theonebased onthe
definitions because it can take advantage of the powerful calculus techniques
namedLHospitalsrule
12

Examples
Compare the orders of growth of 1/2 n(n 1) and n2.

Compare the orders of growth of log2n and n.

13

Basic asymptotic efficiency classes

1

constant

log n

logarithmic

linear

n log n

n-log-n or linearithmic

n2

quadratic

n3

cubic

2n

exponential

n!

factorial
14

Time efficiency of nonrecursive algorithms

General Plan for Analysis
Decide on parameter n indicating input size
Identify algorithms basic operation
Determine worst, average, and best cases for input of size n
Set up a sum for the number of times the basic operation is
executed

Simplify the sum using standard formulas and rules (see

Appendix A of Anany Levitin book and previous lectures)

15

Example 1: Maximum element

BasicOperation:
There are two operations in the loops
body:
thecomparisonA[i]>maxvaland
theassignmentmaxvalA[i].
Sincethecomparisonisexecutedoneach
repetitionoftheloopandtheassignment
is not, we should consider the
comparison to be the algorithms basic
operation.

16

Example 2: Element uniqueness problem

BasicOperation:
Sincetheinnermostloopcontainsa
singleoperation,weshouldconsider
itasthealgorithmsbasicoperation

17

Example 3: Matrix multiplication

BasicOperation:
multiplicationasthebasicoperation

18

Plan for Analysis of Recursive Algorithms

Decide on a parameter indicating an inputs size.
Identify the algorithms basic operation.
Check whether the number of times the basic op. is executed may vary on
different inputs of the same size. (If it may, the worst, average, and best
cases must be investigated separately.)
Setup a recurrence relation with an appropriate initial condition
expressing the number of times the basic op. is executed.
Solve the recurrence (or, at the very least, establish its solutions order of
growth) by backward substitutions or forward substitution method.

19

Example 1: Recursive evaluation of n!

Definition: n ! = 1 2 (n-1) n for n 1 and 0! = 1
Recursive definition of n!: F(n) = F(n-1) n for n 1 and
F(0) = 1

20

Analysis

21

Solve the recurrence

M(n) = M(n-1) + 1, M(0) = 0
Backward Substitution
M(n) = M(n 1) + 1 substitute M(n 1) = M(n 2) + 1
= [M(n 2) + 1]+ 1= M(n 2) + 2 substitute M(n 2) = M(n 3) + 1
= [M(n 3) + 1]+ 2
= M(n 3) + 3.
... ...
...
...
= M(n i) + i.
...
= M(n-n)+n
= M(0)+n
=n
=(n)

22

Example 2: The Tower of Hanoi Puzzle

M(n1)moves

M(n1)moves

M(1)move
Recurrenceformoves: M(n)=M(n1)+1+M(n1)=2M(n1)+1
M(1)=1
23

Solve M(n) = 2M(n-1) + 1, M(1) = 1

Backward Substitution
M(n) = 2M(n 1) + 1 sub. M(n 1) = 2M(n 2) + 1
= 2[2M(n 2) + 1]+ 1= 22M(n 2) + 2 + 1 sub. M(n 2) = 2M(n 3) + 1
= 22[2M(n 3) + 1]+ 2 + 1= 23M(n 3) + 22 + 2 + 1.

24M(n 4) + 23 + 22 + 2 + 1, and after i substitutions, we get

M(n) = 2iM(n i) + 2i1 + 2i2 + . . . + 2 + 1= 2iM(n i) + 2i 1.
Since the initial condition is specified for n = 1, which is achieved for i
= n 1, we
get the following formula for the solution to recurrence:
M(n) = 2n1M(n (n 1)) + 2n1 1
= 2n1M(1) + 2n1 1= 2n1 + 2n1 1= 2n 1.

24

Tower of Hanoi

25

Fibonacci numbers
The Fibonacci numbers:
0, 1, 1, 2, 3, 5, 8, 13, 21,
The Fibonacci recurrence:
F(n) = F(n-1) + F(n-2)
F(0) = 0
F(1) = 1
General 2nd order linear homogeneous recurrence with
constant coefficients:
aX(n) + bX(n-1) + cX(n-2) = 0

26

Solving aX(n) + bX(n-1) + cX(n-2) = 0

Set up the characteristic equation (quadratic)
ar2
+
br
+
c

Solve to obtain roots r1 and r2

General solution to the recurrence
if r1 and r2 are two distinct real roots: X(n) = r1n + r2n
if r1 = r2 = r are two equal real roots:

X(n) = rn + nr n

Particular solution can be found by using initial conditions

27

Solve Recurrence for Fibonacci numbers

28

Solve Recurrence for Fibonacci numbers

= 1/5 and =1/5

29