Você está na página 1de 96

Algorithms

Elements of Algorithm Analysis


Partha Pratim Das
Department of Computer Science and Engineering
Indian Institute of Technology, Kharagpur
ppd@cse.iitkgp.ernet.in

T10KT Main Workshop


May 25, 2015

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

1 / 44

Getting Started
Why?
Set the motivation for algorithm analysis:
Why analyse?

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

2 / 44

Getting Started
Why?
Set the motivation for algorithm analysis:
Why analyse?

What?
Identify what all need to be analysed:
What to analyse?

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

2 / 44

Getting Started
Why?
Set the motivation for algorithm analysis:
Why analyse?

What?
Identify what all need to be analysed:
What to analyse?

How?
Learn the techniques for analysis:
How to analyse?

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

2 / 44

Getting Started
Why?
Set the motivation for algorithm analysis:
Why analyse?

What?
Identify what all need to be analysed:
What to analyse?

How?
Learn the techniques for analysis:
How to analyse?

Where?
Understand the scenarios for application:
Where to analyse?

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

2 / 44

Getting Started
Why?
Set the motivation for algorithm analysis:
Why analyse?

What?
Identify what all need to be analysed:
What to analyse?

How?
Learn the techniques for analysis:
How to analyse?

Where?
Understand the scenarios for application:
Where to analyse?

When?
Realize your position for seeking the analysis:
When to analyse?
Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

2 / 44

Why analyse?
Practical reasons:
Resources are scarce
Greed to do more with less
Avoid performance bugs
Core Issues:
Predict performance
How much time does binary search take?

Compare algorithms
How quick is Quicksort?

Provide guarantees
Size notwithstanding, Red-Black tree inserts in O(log n)

Understand theoretical basis


Sorting by comparison cannot do better than (n log n)

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

3 / 44

What to analyse?
Core Issue: Cannot control what we cannot measure
Time
Story starts here with Analytical Engine
Most common analysis factor
Representative of various related analysis factors like Power,
Bandwidth, Processors
Supported by Complexity Classes

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

4 / 44

What to analyse?
Core Issue: Cannot control what we cannot measure
Time
Story starts here with Analytical Engine
Most common analysis factor
Representative of various related analysis factors like Power,
Bandwidth, Processors
Supported by Complexity Classes

Space
Widely explored
Important for hand-held devices
Supported by Complexity Classes

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

4 / 44

What to analyse?
Examples:
Sum of Natural Numbers
Minimum of a Sequence of Numbers

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

5 / 44

What to analyse?
Sum of Natural Numbers
int sum(int n) {
int s = 0;
for(; n > 0; --n)
s = s + n;
return s;
}

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

6 / 44

What to analyse?
Sum of Natural Numbers
int sum(int n) {
int s = 0;
for(; n > 0; --n)
s = s + n;
return s;
}
Time T (n) = n (additions)

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

6 / 44

What to analyse?
Sum of Natural Numbers
int sum(int n) {
int s = 0;
for(; n > 0; --n)
s = s + n;
return s;
}
Time T (n) = n (additions)
Space S(n) = 2 (n, s)

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

6 / 44

What to analyse?
Minimum of a Sequence of Numbers
int min(int a[], int n) {
for(int i = 0; i < n; ++i)
cin >> a[i];
int t = a[--n];
for(; n > 0; --n)
if (t < a[--n])
t = a[n];
return t;
}

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

7 / 44

What to analyse?
Minimum of a Sequence of Numbers
int min(int a[], int n) {
for(int i = 0; i < n; ++i)
cin >> a[i];
int t = a[--n];
for(; n > 0; --n)
if (t < a[--n])
t = a[n];
return t;
}
Time T (n) = n 1 (comparison of value)

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

7 / 44

What to analyse?
Minimum of a Sequence of Numbers
int min(int a[], int n) {
for(int i = 0; i < n; ++i)
cin >> a[i];
int t = a[--n];
for(; n > 0; --n)
if (t < a[--n])
t = a[n];
return t;
}
Time T (n) = n 1 (comparison of value)
Space S(n) = n + 3 (a[]s, n, i, t)

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

7 / 44

What to analyse?
Minimum of a Sequence of Numbers
int min(int n) {
int x;
cin >> x;
int t = x;
for(; n > 1; --n) {
cin >> x;
if (t < x)
t = x;
}
return t;
}

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

8 / 44

What to analyse?
Minimum of a Sequence of Numbers
int min(int n) {
int x;
cin >> x;
int t = x;
for(; n > 1; --n) {
cin >> x;
if (t < x)
t = x;
}
return t;
}
Time T (n) = n 1 (comparison of value)

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

8 / 44

What to analyse?
Minimum of a Sequence of Numbers
int min(int n) {
int x;
cin >> x;
int t = x;
for(; n > 1; --n) {
cin >> x;
if (t < x)
t = x;
}
return t;
}
Time T (n) = n 1 (comparison of value)
Space S(n) = 3 (n, x, t)
Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

8 / 44

How to analyse?
Mathematical / Counting Models

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

9 / 44

How to analyse?
Mathematical / Counting Models
Asymptotic Analysis

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

9 / 44

How to analyse?
Mathematical / Counting Models
Asymptotic Analysis
Master Theorem

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

9 / 44

How to analyse? Counting Models


Mathematical / Counting Models
Core Idea: Total running time = Sum of cost frequency for all
operations
Need to analyse program to determine set of operations
Cost depends on machine, compiler
Frequency depends on algorithm, input data
Machine Model: Random Access Machine (RAM) Computing Model
Input data & size
Operations
Intermediate Stages
Output data & size

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

10 / 44

How to analyse? Counting Models


Query:
Can we assume that two large numbers of arbitrary size can be added up
in constant time? Why?

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

11 / 44

How to analyse? Counting Models


Examples:
Factorial of a Number

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

12 / 44

How to analyse? Counting Models


Factorial (Recursive)
int fact(int n) {
if (0 != n) return n*fact(n-1);
return 1;
}
Solution by Counting:

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

13 / 44

How to analyse? Counting Models


Factorial (Recursive)
int fact(int n) {
if (0 != n) return n*fact(n-1);
return 1;
}
Solution by Counting:
Time T (n) = n 1 (multiplication)

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

13 / 44

How to analyse? Counting Models


Factorial (Recursive)
int fact(int n) {
if (0 != n) return n*fact(n-1);
return 1;
}
Solution by Counting:
Time T (n) = n 1 (multiplication)
Space S(n) = n + 1 (ns in recursive calls)

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

13 / 44

How to analyse? Counting Models


Factorial (Iterative)
int fact(int n) {
int t = 1;
for(; n > 0; --n)
t = t * n;
return t;
}

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

14 / 44

How to analyse? Counting Models


Factorial (Iterative)
int fact(int n) {
int t = 1;
for(; n > 0; --n)
t = t * n;
return t;
}
Time T (n) = n (multiplication)

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

14 / 44

How to analyse? Counting Models


Factorial (Iterative)
int fact(int n) {
int t = 1;
for(; n > 0; --n)
t = t * n;
return t;
}
Time T (n) = n (multiplication)
Space S(n) = 2 (n, t)

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

14 / 44

How to analyse? Counting Models


Mathematical / Counting Models
Frequency Analysis Tool: Recurrence of count of operations
Define recurrence
Identify base condition
Solve recurrence in closed form

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

15 / 44

How to analyse? Counting Models


Examples:
Sum of Natural Numbers
Factorial of a Number
nth Fibonacci Number

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

16 / 44

How to analyse? Counting Models


Sum of n Natural Numbers
int sum(int n) {
int s = 0;
for(; n > 0; --n)
s = s + n;
return s;
}

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

17 / 44

How to analyse? Counting Models


Sum of n Natural Numbers
int sum(int n) {
int s = 0;
for(; n > 0; --n)
s = s + n;
return s;
}
T (n)

= T (n 1) + 1,
= 1,
Solve Recurrence in Long hand:
T (n) = T (n 1) + 1
= T (n 2) + (1 + 1)
Time T (n) (additions)
=
= T (1) + (n 1)
= 1 + (n 1)
= n

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

n>1
n=1

May 25, 2015

17 / 44

How to analyse? Counting Models


Sum of n Natural Numbers
int sum(int n) {
int s = 0;
for(; n > 0; --n)
s = s + n;
return s;
}
T (n)

= T (n 1) + 1,
= 1,
Solve Recurrence in Long hand:
T (n) = T (n 1) + 1
= T (n 2) + (1 + 1)
Time T (n) (additions)
=
= T (1) + (n 1)
= 1 + (n 1)
= n
Space S(n) = 2
Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

n>1
n=1

May 25, 2015

17 / 44

How to analyse? Counting Models


Factorial (Recursive)
int fact(int n) {
if (0 != n) return n*fact(n-1);
return 1;
}

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

18 / 44

How to analyse? Counting Models


Factorial (Recursive)
int fact(int n) {
if (0 != n) return n*fact(n-1);
return 1;
}
Time T (n) (multiplication)
T (n) = T (n 1) + 1, n > 0
= 0,
n=0
T (n)

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

18 / 44

How to analyse? Counting Models


Factorial (Recursive)
int fact(int n) {
if (0 != n) return n*fact(n-1);
return 1;
}
Time T (n) (multiplication)
T (n) = T (n 1) + 1, n > 0
= 0,
n=0
T (n) =
Space S(n)
S(n) =
=
S(n)

n
(ns in recursive calls)
S(n 1) + 1, n > 0
1,
n=0
n+1

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

18 / 44

How to analyse? Counting Models


Factorial (Iterative)
int fact(int n) {
int t = 1;
for(; n > 0; --n)
t = t * n;
return t;
}

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

19 / 44

How to analyse? Counting Models


Factorial (Iterative)
int fact(int n) {
int t = 1;
for(; n > 0; --n)
t = t * n;
return t;
}
T (n)

=
=

T (n 1) + 1,
0,

T (n)

Time T (n) (multiplication)

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

n>0
n=0

May 25, 2015

19 / 44

How to analyse? Counting Models


Factorial (Iterative)
int fact(int n) {
int t = 1;
for(; n > 0; --n)
t = t * n;
return t;
}
T (n)

=
=

T (n 1) + 1,
0,

T (n)

Time T (n) (multiplication)

S(n)

=
=

S(n 1),
2,

S(n)

Space S(n) (n, t)

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

n>0
n=0

n>0
n=0

May 25, 2015

19 / 44

How to analyse? Counting Models


Fibonacci (Recursive)
int fibo(int n) {
if (n > 1) return fibo(n-1)+fibo(n-2);
return n;
}

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

20 / 44

How to analyse? Counting Models


Fibonacci (Recursive)
int fibo(int n) {
if (n > 1) return fibo(n-1)+fibo(n-2);
return n;
}
Time T (n) (additions)
T (n) = T (n 1) + T (n 2) + 1, n > 1
= 0,
n = 0, 1
Solution by Generating Function:
T (n) = n (approx)
where =

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

1+ 5
2

May 25, 2015

20 / 44

How to analyse? Counting Models


Fibonacci (Recursive)
int fibo(int n) {
if (n > 1) return fibo(n-1)+fibo(n-2);
return n;
}
Time T (n) (additions)
T (n) = T (n 1) + T (n 2) + 1, n > 1
= 0,
n = 0, 1
Solution by Generating Function:
T (n) = n (approx)
where =
Space S(n) (ns in recursive calls)
S(n) = S(n 1) + 1, n > 1
= 1,
n = 0, 1
S(n)

1+ 5
2

n+1

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

20 / 44

How to analyse? Counting Models


Fibonacci (Iterative)
int fibo(int n) {
int FIBO[n+1];
if (n > 1) {
FIBO[0] = 0; FIBO[1] = 1;
for(int i = 2; i <= n; ++i)
FIBO[i] = FIBO[i-1] + FIBO[i-2];
return FIBO[n];
}
else return n;
}

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

21 / 44

How to analyse? Counting Models


Fibonacci (Iterative)
int fibo(int n) {
int FIBO[n+1];
if (n > 1) {
FIBO[0] = 0; FIBO[1] = 1;
for(int i = 2; i <= n; ++i)
FIBO[i] = FIBO[i-1] + FIBO[i-2];
return FIBO[n];
}
else return n;
}
Time T (n) (additions)
T (n) = T (n 1) + 1, n > 1
= 0,
n = 0, 1
T (n)

n1

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

21 / 44

How to analyse? Counting Models


Fibonacci (Iterative)
int fibo(int n) {
int FIBO[n+1];
if (n > 1) {
FIBO[0] = 0; FIBO[1] = 1;
for(int i = 2; i <= n; ++i)
FIBO[i] = FIBO[i-1] + FIBO[i-2];
return FIBO[n];
}
else return n;
}
Time T (n) (additions)
T (n) = T (n 1) + 1, n > 1
= 0,
n = 0, 1
T (n) = n 1
Space S(n) = n + 3 (FIBO[]s, n, i)
Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

21 / 44

How to analyse? Counting Models


Fibonacci (Iterative)
int fibo(int n) {
int x = 0, y = 1, t = n;
if (n > 1) {
for(int i = 2; i <= n; ++i) {
int t = x + y;
x = y;
y = t;
}
}
return t;
}

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

22 / 44

How to analyse? Counting Models


Fibonacci (Iterative)
int fibo(int n) {
int x = 0, y = 1, t = n;
if (n > 1) {
for(int i = 2; i <= n; ++i) {
int t = x + y;
x = y;
y = t;
}
}
return t;
}
Time T (n) (additions)
T (n) = T (n 1) + 1, n > 1
= 0,
n = 0, 1
T (n)

n1

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

22 / 44

How to analyse? Counting Models


Fibonacci (Iterative)
int fibo(int n) {
int x = 0, y = 1, t = n;
if (n > 1) {
for(int i = 2; i <= n; ++i) {
int t = x + y;
x = y;
y = t;
}
}
return t;
}
Time T (n) (additions)
T (n) = T (n 1) + 1, n > 1
= 0,
n = 0, 1
T (n) = n 1
Space S(n) = 4 (n, x, y, t)
Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

22 / 44

How to analyse? Asymptotic Analysis


Asymptotic Analysis
Core Idea: Cannot compare actual times; hence compare Growth or how
time increases with input size
Function Approximation (tilde () notation)
Common Growth Functions
Big-Oh (O(.)), Big-Omega ((.)), and Big-Theta ((.)) Notations
Solve recurrence with Growth Functions

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

23 / 44

How to analyse? Asymptotic Analysis


Function Approximation (tilde () notation)
Consider:
int count = 0;
for (int i = 0; i < N; i++)
for (int j = i+1; j < N; j++)
if (a[i] + a[j] == 0)
count++;
Operation
variable declaration
assignment statement
less than compare
equal to compare
array access
increment
Partha Pratim Das (IIT, Kharagpur)

Frequency
N +2
N +2
1
2 (N + 1)(N + 2)
1
2 N(N 1)
N(N 1)
1
2 N(N 1) to N(N 1)

Elements of Algorithm Analysis

May 25, 2015

24 / 44

How to analyse? Asymptotic Analysis


Estimate running time (or memory) as a function of input size N.
Ignore lower order terms.
when N is large, terms are negligible
when N is small, we dont care

Operation
variable declaration
assignment statement
less than compare
equal to compare
array access
increment

Frequency
N +2
N +2
1
2 (N + 1)(N + 2)
1
2 N(N 1)
N(N 1)
1
2 N(N 1) to N(N 1)

Approximation
N
N
12 N 2
12 N 2
N2
12 N 2 to N 2

f (n) g (n) means


f (n)
=1
N g (n)
lim

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

25 / 44

Common order-of-growth classifications


Definition. If f (N) ~ c g(N) for some constant c > 0, then the order of growth
of f (N) is g(N).

Ignores leading coefficient.


Ignores lower-order terms.
Ex. The order of growth of the running time of this code is N 3.
int count = 0;
for (int i = 0; i < N; i++)
for (int j = i+1; j < N; j++)
for (int k = j+1; k < N; k++)
if (a[i] + a[j] + a[k] == 0)
count++;

Typical usage. With running times.


where leading coefficient
depends on machine, compiler, JVM, ...

Courtesy: Algorithms by Robert Sedgewick & Kevin Wayne

42

time

Common order-of-growth classifications


200T

Good news. The set of functions


100T

1, log N, N, N log N, N 2, N 3, and 2N

logarithmic
constant

suffices to describe the order of growth of most common algorithms.


100K

200K

500K

size

hm
rit

ea

ea

lin

lin

ic

t ic

cubic

qua
dra

512T

exponential

log-log plot

time

64T

8T
4T
2T

logarithmic

constant
1K

2K

4K

8K

size

512K

Typical orders of growth


43

Courtesy: Algorithms by Robert Sedgewick & Kevin Wayne

Common order-of-growth classifications


order of
growth

name

typical code framework

description

example

T(2N) / T(N)

constant

a = b + c;

statement

add two
numbers

log N

logarithmic

divide in half

binary search

linear

for (int i = 0; i < N; i++)


{ ...
}

loop

find the
maximum

N log N

linearithmic

[see mergesort lecture]

divide
and conquer

mergesort

~2

N2

quadratic

double loop

check all
pairs

N3

cubic

for (int i = 0; i < N; i++)


for (int j = 0; j < N; j++)
for (int k = 0; k < N; k++)
{ ...
}

triple loop

check all
triples

2N

exponential

[see combinatorial search lecture]

exhaustive
search

check all
subsets

T(N)

while (N > 1)
N = N / 2; ...

for (int i = 0; i < N; i++)


for (int j = 0; j < N; j++)
{ ...
}

~1

44

Courtesy: Algorithms by Robert Sedgewick & Kevin Wayne

Commonly-used notations in the theory of algorithms

notation

provides

Big Theta

asymptotic
order of growth

example

shorthand for

used to

N2

(N2)

10 N 2
5 N 2 + 22 N log N + 3N

classify
algorithms

10 N 2
Big Oh

(N2) and smaller

O(N2)

100 N
22 N log N + 3 N

develop
upper bounds

N2
Big Omega

(N2) and larger

(N2)

N5
N 3 + 22 N log N + 3 N

develop
lower bounds

53

Courtesy: Algorithms by Robert Sedgewick & Kevin Wayne

How to analyse? Asymptotic Analysis


Formal Definitions of Asymptotic Notation
f (n)
O(g (n))
(g (n))
(g (n))
o(g (n))
(g (n))

:
:
:
:
:

Condition
c > 0 n0 > 0 n > n0 f (n) c.g (n)
c > 0 n0 > 0 n > n0 f (n) c.g (n)
c1 > 0 c2 > 0 n0 > 0 n > n0 c1 .g (n) f (n) c2 .g (n)
c > 0 n0 > 0 n > n0 |f (n)| c. |g (n)|
c > 0 n0 > 0 n > n0 |f (n)| c. |g (n)|

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

30 / 44

How to analyse? Master Theorem


Example:
Find the k th smallest element in A where A.length = n, 1 k n.
Algorithm: SELECT(A[1, n], k)
1 Split A into n/5 groups of 5 elements each.
2 Let b be the median of the i th group.
i
3 Let B = [b , b , , b
1 2
n/5 ].
4 medianB = SELECT(B, B.length/2).
5 Rearrange A so that all elements smaller than medianB come before
medianB, all elements larger than medianB come after medianB, and
elements equal to medianB are next to medianB.
6 j = position of medianB in rearranged A (if more medianBs, then
take the closest position to n/2).
7 If k < j return SELECT(A[1, j 1], k).
8 If k = j return medianB.
9 If k > j return SELECT(A[j + 1, n], k j).
Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

31 / 44

How to analyse? Master Theorem


Find the k th smallest element in A where A.length = n, 1 k n.
Algorithm: SELECT(A[1, n], k) T (n)
1 Split A into n/5 groups of 5 elements each. O(n)
2 Let b be the median of the i th group. O(1) per group O(n) total
i
3 Let B = [b , b , , b
1 2
n/5 ]. O(n)
4 medianB = SELECT(B, B.length/2). T (n/5)
5 Rearrange A so that all elements smaller than medianB come before
medianB, all elements larger than medianB come after medianB, and
elements equal to medianB are next to medianB. O(n)
6 j = position of medianB in rearranged A (if more medianBs, then
take the closest position to n/2). O(n)
7 If k < j return SELECT(A[1, j 1], k). T (j)
8 If k = j return medianB. O(1)
9 If k > j return SELECT(A[j + 1, n], k j). T (n j)
Solve: T (n) = T ( n5 ) + T ( 3n
4 ) + cn
Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

32 / 44

How to analyse? Master Theorem


Solving Recurrence by Hypothesize and Test
T ( n5 ) + T ( 3n
if n > 5
4 ) + cn,
c,
if n 5
Hypothesis: There exists a constant d such that T (n) < dn.
Base Case: n 5: T (n) c, if n 5. Since we want to show that
T (n) < dn for some d, we need d c.
Inductive Case: n > 5:
d n5
By Hypothesis
T ( n5 )
3n
3n
T( 4 )
d 4
By Hypothesis
T (n)
T ( n5 ) + T ( 3n
)
+
cn,
if
n>5
4
n
3n
d 5 + d 4 + cn
19
( 20
d + c)n
dn
We need this
T (n)

19
( 20
d + c)
d

d
20c

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

33 / 44

How to analyse? Master Theorem


Master Theorem
Core Idea: Asymptotic recurrence forms tend to recur across problems
Use a parametrized formulation
Solve for the general form for a set of cases
Given a situation
extract parameters
identify the case
adopt solution

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

34 / 44

How to analyse? Master Theorem


Master Theorem
Let T (n) be a monotonically increasing function that satisfies
T (n) = aT (n/b) + f (n)
T (1) = c
where a 1, b 2, c > 0. If f (n) (nd ) where d 0, then

T (n)

(nd )
(nd log n)
(nlogb a )

if a < b d
if a = b d
if a > b d

Corollary
If f (n) (nlogb a logk n) then
T (n) (nlogb a logk+1 n)
Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

35 / 44

How to analyse? Master Theorem


Master Theorem cannot be used if
T (n) is not monotone, ex: T (n) = sin n
f (n) is not a polynomial, ex: T (n) = 2T (n/2) + 2n

b cannot be expressed as a constant, ex: T (n) = T ( n)

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

36 / 44

How to analyse? Master Theorem


Examples
T (n) =
T ( n2 ) + 12 n2 + n

2T ( n4 ) + n + 42
3T ( n2 ) + 34 n + 1

a
1
2
3

Partha Pratim Das (IIT, Kharagpur)

b
2
4
2

d
2
1
2

Condition
2

1<2
1
2 = 42
3 > 21

Case

T (n) =

1
2
3

(nd ) = (n2 )

(nd log n) = ( n log n)


(nlogb a ) = (nlog2 3 )

Elements of Algorithm Analysis

May 25, 2015

37 / 44

Where to analyse?
Algorithmic Situation
Core Idea: Identify data configurations or scenarios for analysis
Best Case
Worst Case
Average Case
Probabilistic Case
Amortized Case

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

38 / 44

Types of analyses
Worst case. Running time guarantee for any input of size n.
Ex. Heapsort requires at most 2 n log2 n compares to sort n elements.
Probabilistic. Expected running time of a randomized algorithm.
Ex. The expected number of compares to quicksort n elements is ~ 2n ln n.
Amortized. Worst-case running time for any sequence of n operations.
Ex. Starting from an empty stack, any sequence of n push and pop
operations takes O(n) operations using a resizing array.
Average-case. Expected running time for a random input of size n.
Ex. The expected number of character compares performed by 3-way
radix quicksort on n uniformly random strings is ~ 2n ln n.

Also. Smoothed analysis, competitive analysis, ...


8

Tardos
Courtesy: Algorithm Design by Jon Kleinberg & Eva

When to analyse?
Determination of Quality of an Algorithm
Core Idea: Are we doing well, and can we do better?
Let A be an algorithm for solving a problem P of size n.
Is A the best algorithm to solve P?
Upper Bound Finding better Algorithm for the Problem

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

40 / 44

When to analyse?
Determination of Quality of an Algorithm
Core Idea: Are we doing well, and can we do better?
Let A be an algorithm for solving a problem P of size n.
Is A the best algorithm to solve P?
Upper Bound Finding better Algorithm for the Problem
If A is O(f (n)), then A is an O(f (n)) upper bound for P because we
already know an algorithm (that is, A) that can solve P in O(f (n)).

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

40 / 44

When to analyse?
Determination of Quality of an Algorithm
Core Idea: Are we doing well, and can we do better?
Let A be an algorithm for solving a problem P of size n.
Is A the best algorithm to solve P?
Upper Bound Finding better Algorithm for the Problem
If A is O(f (n)), then A is an O(f (n)) upper bound for P because we
already know an algorithm (that is, A) that can solve P in O(f (n)).
Let A1 be another algorithm that solves P in O(f1 (n)). Naturally, A1
is a better upper bound for P, if O(f1 (n)) O(f (n)).

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

40 / 44

When to analyse?
Determination of Quality of an Algorithm
Core Idea: Are we doing well, and can we do better?
Let A be an algorithm for solving a problem P of size n.
Is A the best algorithm to solve P?
Upper Bound Finding better Algorithm for the Problem
If A is O(f (n)), then A is an O(f (n)) upper bound for P because we
already know an algorithm (that is, A) that can solve P in O(f (n)).
Let A1 be another algorithm that solves P in O(f1 (n)). Naturally, A1
is a better upper bound for P, if O(f1 (n)) O(f (n)).
Let A2 be yet another algorithm that solves P in O(f2 (n)). Naturally,
A2 is a better upper bound for P, if O(f2 (n)) O(f1 (n)).

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

40 / 44

When to analyse?
Determination of Quality of an Algorithm
Core Idea: Are we doing well, and can we do better?
Let A be an algorithm for solving a problem P of size n.
Is A the best algorithm to solve P?
Upper Bound Finding better Algorithm for the Problem
If A is O(f (n)), then A is an O(f (n)) upper bound for P because we
already know an algorithm (that is, A) that can solve P in O(f (n)).
Let A1 be another algorithm that solves P in O(f1 (n)). Naturally, A1
is a better upper bound for P, if O(f1 (n)) O(f (n)).
Let A2 be yet another algorithm that solves P in O(f2 (n)). Naturally,
A2 is a better upper bound for P, if O(f2 (n)) O(f1 (n)).
Let A3 be . When do we stop?
Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

40 / 44

When to analyse?
Lower Bound Finding better Analysis of the Problem

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

41 / 44

When to analyse?
Lower Bound Finding better Analysis of the Problem
We must look at the input for P (really?). If that takes g (n) effort, P
has a lower bound of (g (n)). We cannot do better than (g (n)).

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

41 / 44

When to analyse?
Lower Bound Finding better Analysis of the Problem
We must look at the input for P (really?). If that takes g (n) effort, P
has a lower bound of (g (n)). We cannot do better than (g (n)).
Lower bound is established by Adversary Argument. Let P be the
problem of sorting n numbers. We claim that P is (n), that is, one
must look at all inputs (known as Input Complexity). If not, suppose it
is enough to consider only n 1 numbers. Then we can always design
the nth number as larger than the largest and make the sorting fail.

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

41 / 44

When to analyse?
Lower Bound Finding better Analysis of the Problem
We must look at the input for P (really?). If that takes g (n) effort, P
has a lower bound of (g (n)). We cannot do better than (g (n)).
Lower bound is established by Adversary Argument. Let P be the
problem of sorting n numbers. We claim that P is (n), that is, one
must look at all inputs (known as Input Complexity). If not, suppose it
is enough to consider only n 1 numbers. Then we can always design
the nth number as larger than the largest and make the sorting fail.
Input complexity may be lower than (n). Let P be a problem to find
a number that is not the largest in a list of n numbers. P is (2).

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

41 / 44

When to analyse?
Lower Bound Finding better Analysis of the Problem
We must look at the input for P (really?). If that takes g (n) effort, P
has a lower bound of (g (n)). We cannot do better than (g (n)).
Lower bound is established by Adversary Argument. Let P be the
problem of sorting n numbers. We claim that P is (n), that is, one
must look at all inputs (known as Input Complexity). If not, suppose it
is enough to consider only n 1 numbers. Then we can always design
the nth number as larger than the largest and make the sorting fail.
Input complexity may be lower than (n). Let P be a problem to find
a number that is not the largest in a list of n numbers. P is (2).
Similarly Output Complexity can be defined.

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

41 / 44

When to analyse?
Lower Bound Finding better Analysis of the Problem
We must look at the input for P (really?). If that takes g (n) effort, P
has a lower bound of (g (n)). We cannot do better than (g (n)).
Lower bound is established by Adversary Argument. Let P be the
problem of sorting n numbers. We claim that P is (n), that is, one
must look at all inputs (known as Input Complexity). If not, suppose it
is enough to consider only n 1 numbers. Then we can always design
the nth number as larger than the largest and make the sorting fail.
Input complexity may be lower than (n). Let P be a problem to find
a number that is not the largest in a list of n numbers. P is (2).
Similarly Output Complexity can be defined.

Suppose we find another adversary argument to show that P is


(g1 (n)). It is better for P, if (g (n)) (g1 (n)).

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

41 / 44

When to analyse?
Lower Bound Finding better Analysis of the Problem
We must look at the input for P (really?). If that takes g (n) effort, P
has a lower bound of (g (n)). We cannot do better than (g (n)).
Lower bound is established by Adversary Argument. Let P be the
problem of sorting n numbers. We claim that P is (n), that is, one
must look at all inputs (known as Input Complexity). If not, suppose it
is enough to consider only n 1 numbers. Then we can always design
the nth number as larger than the largest and make the sorting fail.
Input complexity may be lower than (n). Let P be a problem to find
a number that is not the largest in a list of n numbers. P is (2).
Similarly Output Complexity can be defined.

Suppose we find another adversary argument to show that P is


(g1 (n)). It is better for P, if (g (n)) (g1 (n)).
Suppose we find yet another adversary argument to show that P is
(g2 (n)). It is better for P, if (g1 (n)) (g2 (n)).
Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

41 / 44

When to analyse?
Lower Bound Finding better Analysis of the Problem
We must look at the input for P (really?). If that takes g (n) effort, P
has a lower bound of (g (n)). We cannot do better than (g (n)).
Lower bound is established by Adversary Argument. Let P be the
problem of sorting n numbers. We claim that P is (n), that is, one
must look at all inputs (known as Input Complexity). If not, suppose it
is enough to consider only n 1 numbers. Then we can always design
the nth number as larger than the largest and make the sorting fail.
Input complexity may be lower than (n). Let P be a problem to find
a number that is not the largest in a list of n numbers. P is (2).
Similarly Output Complexity can be defined.

Suppose we find another adversary argument to show that P is


(g1 (n)). It is better for P, if (g (n)) (g1 (n)).
Suppose we find yet another adversary argument to show that P is
(g2 (n)). It is better for P, if (g1 (n)) (g2 (n)).
And so on . When do we stop?
Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

41 / 44

When to analyse?
Determination of Quality of an Algorithm
Core Idea: Are we doing well, and can we do better?
Let A be an algorithm for solving a problem P of size n.
Is A the best algorithm to solve P?
Optimal Algorithm

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

42 / 44

When to analyse?
Determination of Quality of an Algorithm
Core Idea: Are we doing well, and can we do better?
Let A be an algorithm for solving a problem P of size n.
Is A the best algorithm to solve P?
Optimal Algorithm
A is Optimal for P, if O(f (n)) = (f (n)), that is, the lower bound
matches the upper bound.

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

42 / 44

When to analyse?
Determination of Quality of an Algorithm
Core Idea: Are we doing well, and can we do better?
Let A be an algorithm for solving a problem P of size n.
Is A the best algorithm to solve P?
Optimal Algorithm
A is Optimal for P, if O(f (n)) = (f (n)), that is, the lower bound
matches the upper bound.
Complexity of A is (f (n)).

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

42 / 44

When to analyse?
Determination of Quality of an Algorithm
Core Idea: Are we doing well, and can we do better?
Let A be an algorithm for solving a problem P of size n.
Is A the best algorithm to solve P?
Optimal Algorithm
A is Optimal for P, if O(f (n)) = (f (n)), that is, the lower bound
matches the upper bound.
Complexity of A is (f (n)).
Till an optimal algorithm is found we continuously work on the
algorithm to lower the upper bound and devise adversary arguments
to upper the lower bound.

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

42 / 44

When to analyse?
Analysis of Sorting
Upper Bound

Lower Bound

Optimal Algorithm

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

43 / 44

When to analyse?
Analysis of Sorting
Upper Bound
Brute-force: Generate all n! permutations and select the sorted
sequence: O(nn ), O(nn )

Lower Bound

Optimal Algorithm

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

43 / 44

When to analyse?
Analysis of Sorting
Upper Bound
Brute-force: Generate all n! permutations and select the sorted
sequence: O(nn ), O(nn )
Bubble Sort: O(n2 ), O(n2 )

Lower Bound

Optimal Algorithm

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

43 / 44

When to analyse?
Analysis of Sorting
Upper Bound
Brute-force: Generate all n! permutations and select the sorted
sequence: O(nn ), O(nn )
Bubble Sort: O(n2 ), O(n2 )
Insertion Sort: O(n2 ), O(n2 )

Lower Bound

Optimal Algorithm

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

43 / 44

When to analyse?
Analysis of Sorting
Upper Bound
Brute-force: Generate all n! permutations and select the sorted
sequence: O(nn ), O(nn )
Bubble Sort: O(n2 ), O(n2 )
Insertion Sort: O(n2 ), O(n2 )
Quick Sort: O(n2 ), O(n log n)

Lower Bound

Optimal Algorithm

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

43 / 44

When to analyse?
Analysis of Sorting
Upper Bound
Brute-force: Generate all n! permutations and select the sorted
sequence: O(nn ), O(nn )
Bubble Sort: O(n2 ), O(n2 )
Insertion Sort: O(n2 ), O(n2 )
Quick Sort: O(n2 ), O(n log n)
Merge Sort: O(n log n), O(n log n)

Lower Bound

Optimal Algorithm

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

43 / 44

When to analyse?
Analysis of Sorting
Upper Bound
Brute-force: Generate all n! permutations and select the sorted
sequence: O(nn ), O(nn )
Bubble Sort: O(n2 ), O(n2 )
Insertion Sort: O(n2 ), O(n2 )
Quick Sort: O(n2 ), O(n log n)
Merge Sort: O(n log n), O(n log n)
Heap Sort: O(n log n), O(n log n)

Lower Bound

Optimal Algorithm

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

43 / 44

When to analyse?
Analysis of Sorting
Upper Bound
Brute-force: Generate all n! permutations and select the sorted
sequence: O(nn ), O(nn )
Bubble Sort: O(n2 ), O(n2 )
Insertion Sort: O(n2 ), O(n2 )
Quick Sort: O(n2 ), O(n log n)
Merge Sort: O(n log n), O(n log n)
Heap Sort: O(n log n), O(n log n)

Lower Bound
Input Complexity: (n) must read all numbers.

Optimal Algorithm

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

43 / 44

When to analyse?
Analysis of Sorting
Upper Bound
Brute-force: Generate all n! permutations and select the sorted
sequence: O(nn ), O(nn )
Bubble Sort: O(n2 ), O(n2 )
Insertion Sort: O(n2 ), O(n2 )
Quick Sort: O(n2 ), O(n log n)
Merge Sort: O(n log n), O(n log n)
Heap Sort: O(n log n), O(n log n)

Lower Bound
Input Complexity: (n) must read all numbers.
Comparison Tree: (n log n) choose from one of n! or O(nn )
permutations with binary choice at every stage. Hence, O(log nn ) =
O(n log n).

Optimal Algorithm

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

43 / 44

When to analyse?
Analysis of Sorting
Upper Bound
Brute-force: Generate all n! permutations and select the sorted
sequence: O(nn ), O(nn )
Bubble Sort: O(n2 ), O(n2 )
Insertion Sort: O(n2 ), O(n2 )
Quick Sort: O(n2 ), O(n log n)
Merge Sort: O(n log n), O(n log n)
Heap Sort: O(n log n), O(n log n)

Lower Bound
Input Complexity: (n) must read all numbers.
Comparison Tree: (n log n) choose from one of n! or O(nn )
permutations with binary choice at every stage. Hence, O(log nn ) =
O(n log n).

Optimal Algorithm
Merge Sort / Heap Sort
Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

43 / 44

Summary
Need for analysing the running-time and space requirements of a
program.
Asymptotic growth rate or order of the complexity of different
algorithms.
Worst-case, average-case and best-case analysis.
Developing recurrence equations.
Solving recurrences using Inductive Derivation Procedure and the
Master Theorem.

Partha Pratim Das (IIT, Kharagpur)

Elements of Algorithm Analysis

May 25, 2015

44 / 44

Você também pode gostar