Escolar Documentos
Profissional Documentos
Cultura Documentos
1 / 44
Getting Started
Why?
Set the motivation for algorithm analysis:
Why analyse?
2 / 44
Getting Started
Why?
Set the motivation for algorithm analysis:
Why analyse?
What?
Identify what all need to be analysed:
What to analyse?
2 / 44
Getting Started
Why?
Set the motivation for algorithm analysis:
Why analyse?
What?
Identify what all need to be analysed:
What to analyse?
How?
Learn the techniques for analysis:
How to analyse?
2 / 44
Getting Started
Why?
Set the motivation for algorithm analysis:
Why analyse?
What?
Identify what all need to be analysed:
What to analyse?
How?
Learn the techniques for analysis:
How to analyse?
Where?
Understand the scenarios for application:
Where to analyse?
2 / 44
Getting Started
Why?
Set the motivation for algorithm analysis:
Why analyse?
What?
Identify what all need to be analysed:
What to analyse?
How?
Learn the techniques for analysis:
How to analyse?
Where?
Understand the scenarios for application:
Where to analyse?
When?
Realize your position for seeking the analysis:
When to analyse?
Partha Pratim Das (IIT, Kharagpur)
2 / 44
Why analyse?
Practical reasons:
Resources are scarce
Greed to do more with less
Avoid performance bugs
Core Issues:
Predict performance
How much time does binary search take?
Compare algorithms
How quick is Quicksort?
Provide guarantees
Size notwithstanding, Red-Black tree inserts in O(log n)
3 / 44
What to analyse?
Core Issue: Cannot control what we cannot measure
Time
Story starts here with Analytical Engine
Most common analysis factor
Representative of various related analysis factors like Power,
Bandwidth, Processors
Supported by Complexity Classes
4 / 44
What to analyse?
Core Issue: Cannot control what we cannot measure
Time
Story starts here with Analytical Engine
Most common analysis factor
Representative of various related analysis factors like Power,
Bandwidth, Processors
Supported by Complexity Classes
Space
Widely explored
Important for hand-held devices
Supported by Complexity Classes
4 / 44
What to analyse?
Examples:
Sum of Natural Numbers
Minimum of a Sequence of Numbers
5 / 44
What to analyse?
Sum of Natural Numbers
int sum(int n) {
int s = 0;
for(; n > 0; --n)
s = s + n;
return s;
}
6 / 44
What to analyse?
Sum of Natural Numbers
int sum(int n) {
int s = 0;
for(; n > 0; --n)
s = s + n;
return s;
}
Time T (n) = n (additions)
6 / 44
What to analyse?
Sum of Natural Numbers
int sum(int n) {
int s = 0;
for(; n > 0; --n)
s = s + n;
return s;
}
Time T (n) = n (additions)
Space S(n) = 2 (n, s)
6 / 44
What to analyse?
Minimum of a Sequence of Numbers
int min(int a[], int n) {
for(int i = 0; i < n; ++i)
cin >> a[i];
int t = a[--n];
for(; n > 0; --n)
if (t < a[--n])
t = a[n];
return t;
}
7 / 44
What to analyse?
Minimum of a Sequence of Numbers
int min(int a[], int n) {
for(int i = 0; i < n; ++i)
cin >> a[i];
int t = a[--n];
for(; n > 0; --n)
if (t < a[--n])
t = a[n];
return t;
}
Time T (n) = n 1 (comparison of value)
7 / 44
What to analyse?
Minimum of a Sequence of Numbers
int min(int a[], int n) {
for(int i = 0; i < n; ++i)
cin >> a[i];
int t = a[--n];
for(; n > 0; --n)
if (t < a[--n])
t = a[n];
return t;
}
Time T (n) = n 1 (comparison of value)
Space S(n) = n + 3 (a[]s, n, i, t)
7 / 44
What to analyse?
Minimum of a Sequence of Numbers
int min(int n) {
int x;
cin >> x;
int t = x;
for(; n > 1; --n) {
cin >> x;
if (t < x)
t = x;
}
return t;
}
8 / 44
What to analyse?
Minimum of a Sequence of Numbers
int min(int n) {
int x;
cin >> x;
int t = x;
for(; n > 1; --n) {
cin >> x;
if (t < x)
t = x;
}
return t;
}
Time T (n) = n 1 (comparison of value)
8 / 44
What to analyse?
Minimum of a Sequence of Numbers
int min(int n) {
int x;
cin >> x;
int t = x;
for(; n > 1; --n) {
cin >> x;
if (t < x)
t = x;
}
return t;
}
Time T (n) = n 1 (comparison of value)
Space S(n) = 3 (n, x, t)
Partha Pratim Das (IIT, Kharagpur)
8 / 44
How to analyse?
Mathematical / Counting Models
9 / 44
How to analyse?
Mathematical / Counting Models
Asymptotic Analysis
9 / 44
How to analyse?
Mathematical / Counting Models
Asymptotic Analysis
Master Theorem
9 / 44
10 / 44
11 / 44
12 / 44
13 / 44
13 / 44
13 / 44
14 / 44
14 / 44
14 / 44
15 / 44
16 / 44
17 / 44
= T (n 1) + 1,
= 1,
Solve Recurrence in Long hand:
T (n) = T (n 1) + 1
= T (n 2) + (1 + 1)
Time T (n) (additions)
=
= T (1) + (n 1)
= 1 + (n 1)
= n
n>1
n=1
17 / 44
= T (n 1) + 1,
= 1,
Solve Recurrence in Long hand:
T (n) = T (n 1) + 1
= T (n 2) + (1 + 1)
Time T (n) (additions)
=
= T (1) + (n 1)
= 1 + (n 1)
= n
Space S(n) = 2
Partha Pratim Das (IIT, Kharagpur)
n>1
n=1
17 / 44
18 / 44
18 / 44
n
(ns in recursive calls)
S(n 1) + 1, n > 0
1,
n=0
n+1
18 / 44
19 / 44
=
=
T (n 1) + 1,
0,
T (n)
n>0
n=0
19 / 44
=
=
T (n 1) + 1,
0,
T (n)
S(n)
=
=
S(n 1),
2,
S(n)
n>0
n=0
n>0
n=0
19 / 44
20 / 44
1+ 5
2
20 / 44
1+ 5
2
n+1
20 / 44
21 / 44
n1
21 / 44
21 / 44
22 / 44
n1
22 / 44
22 / 44
23 / 44
Frequency
N +2
N +2
1
2 (N + 1)(N + 2)
1
2 N(N 1)
N(N 1)
1
2 N(N 1) to N(N 1)
24 / 44
Operation
variable declaration
assignment statement
less than compare
equal to compare
array access
increment
Frequency
N +2
N +2
1
2 (N + 1)(N + 2)
1
2 N(N 1)
N(N 1)
1
2 N(N 1) to N(N 1)
Approximation
N
N
12 N 2
12 N 2
N2
12 N 2 to N 2
25 / 44
42
time
logarithmic
constant
200K
500K
size
hm
rit
ea
ea
lin
lin
ic
t ic
cubic
qua
dra
512T
exponential
log-log plot
time
64T
8T
4T
2T
logarithmic
constant
1K
2K
4K
8K
size
512K
name
description
example
T(2N) / T(N)
constant
a = b + c;
statement
add two
numbers
log N
logarithmic
divide in half
binary search
linear
loop
find the
maximum
N log N
linearithmic
divide
and conquer
mergesort
~2
N2
quadratic
double loop
check all
pairs
N3
cubic
triple loop
check all
triples
2N
exponential
exhaustive
search
check all
subsets
T(N)
while (N > 1)
N = N / 2; ...
~1
44
notation
provides
Big Theta
asymptotic
order of growth
example
shorthand for
used to
N2
(N2)
10 N 2
5 N 2 + 22 N log N + 3N
classify
algorithms
10 N 2
Big Oh
O(N2)
100 N
22 N log N + 3 N
develop
upper bounds
N2
Big Omega
(N2)
N5
N 3 + 22 N log N + 3 N
develop
lower bounds
53
:
:
:
:
:
Condition
c > 0 n0 > 0 n > n0 f (n) c.g (n)
c > 0 n0 > 0 n > n0 f (n) c.g (n)
c1 > 0 c2 > 0 n0 > 0 n > n0 c1 .g (n) f (n) c2 .g (n)
c > 0 n0 > 0 n > n0 |f (n)| c. |g (n)|
c > 0 n0 > 0 n > n0 |f (n)| c. |g (n)|
30 / 44
31 / 44
32 / 44
19
( 20
d + c)
d
d
20c
33 / 44
34 / 44
T (n)
(nd )
(nd log n)
(nlogb a )
if a < b d
if a = b d
if a > b d
Corollary
If f (n) (nlogb a logk n) then
T (n) (nlogb a logk+1 n)
Partha Pratim Das (IIT, Kharagpur)
35 / 44
36 / 44
2T ( n4 ) + n + 42
3T ( n2 ) + 34 n + 1
a
1
2
3
b
2
4
2
d
2
1
2
Condition
2
1<2
1
2 = 42
3 > 21
Case
T (n) =
1
2
3
(nd ) = (n2 )
37 / 44
Where to analyse?
Algorithmic Situation
Core Idea: Identify data configurations or scenarios for analysis
Best Case
Worst Case
Average Case
Probabilistic Case
Amortized Case
38 / 44
Types of analyses
Worst case. Running time guarantee for any input of size n.
Ex. Heapsort requires at most 2 n log2 n compares to sort n elements.
Probabilistic. Expected running time of a randomized algorithm.
Ex. The expected number of compares to quicksort n elements is ~ 2n ln n.
Amortized. Worst-case running time for any sequence of n operations.
Ex. Starting from an empty stack, any sequence of n push and pop
operations takes O(n) operations using a resizing array.
Average-case. Expected running time for a random input of size n.
Ex. The expected number of character compares performed by 3-way
radix quicksort on n uniformly random strings is ~ 2n ln n.
Tardos
Courtesy: Algorithm Design by Jon Kleinberg & Eva
When to analyse?
Determination of Quality of an Algorithm
Core Idea: Are we doing well, and can we do better?
Let A be an algorithm for solving a problem P of size n.
Is A the best algorithm to solve P?
Upper Bound Finding better Algorithm for the Problem
40 / 44
When to analyse?
Determination of Quality of an Algorithm
Core Idea: Are we doing well, and can we do better?
Let A be an algorithm for solving a problem P of size n.
Is A the best algorithm to solve P?
Upper Bound Finding better Algorithm for the Problem
If A is O(f (n)), then A is an O(f (n)) upper bound for P because we
already know an algorithm (that is, A) that can solve P in O(f (n)).
40 / 44
When to analyse?
Determination of Quality of an Algorithm
Core Idea: Are we doing well, and can we do better?
Let A be an algorithm for solving a problem P of size n.
Is A the best algorithm to solve P?
Upper Bound Finding better Algorithm for the Problem
If A is O(f (n)), then A is an O(f (n)) upper bound for P because we
already know an algorithm (that is, A) that can solve P in O(f (n)).
Let A1 be another algorithm that solves P in O(f1 (n)). Naturally, A1
is a better upper bound for P, if O(f1 (n)) O(f (n)).
40 / 44
When to analyse?
Determination of Quality of an Algorithm
Core Idea: Are we doing well, and can we do better?
Let A be an algorithm for solving a problem P of size n.
Is A the best algorithm to solve P?
Upper Bound Finding better Algorithm for the Problem
If A is O(f (n)), then A is an O(f (n)) upper bound for P because we
already know an algorithm (that is, A) that can solve P in O(f (n)).
Let A1 be another algorithm that solves P in O(f1 (n)). Naturally, A1
is a better upper bound for P, if O(f1 (n)) O(f (n)).
Let A2 be yet another algorithm that solves P in O(f2 (n)). Naturally,
A2 is a better upper bound for P, if O(f2 (n)) O(f1 (n)).
40 / 44
When to analyse?
Determination of Quality of an Algorithm
Core Idea: Are we doing well, and can we do better?
Let A be an algorithm for solving a problem P of size n.
Is A the best algorithm to solve P?
Upper Bound Finding better Algorithm for the Problem
If A is O(f (n)), then A is an O(f (n)) upper bound for P because we
already know an algorithm (that is, A) that can solve P in O(f (n)).
Let A1 be another algorithm that solves P in O(f1 (n)). Naturally, A1
is a better upper bound for P, if O(f1 (n)) O(f (n)).
Let A2 be yet another algorithm that solves P in O(f2 (n)). Naturally,
A2 is a better upper bound for P, if O(f2 (n)) O(f1 (n)).
Let A3 be . When do we stop?
Partha Pratim Das (IIT, Kharagpur)
40 / 44
When to analyse?
Lower Bound Finding better Analysis of the Problem
41 / 44
When to analyse?
Lower Bound Finding better Analysis of the Problem
We must look at the input for P (really?). If that takes g (n) effort, P
has a lower bound of (g (n)). We cannot do better than (g (n)).
41 / 44
When to analyse?
Lower Bound Finding better Analysis of the Problem
We must look at the input for P (really?). If that takes g (n) effort, P
has a lower bound of (g (n)). We cannot do better than (g (n)).
Lower bound is established by Adversary Argument. Let P be the
problem of sorting n numbers. We claim that P is (n), that is, one
must look at all inputs (known as Input Complexity). If not, suppose it
is enough to consider only n 1 numbers. Then we can always design
the nth number as larger than the largest and make the sorting fail.
41 / 44
When to analyse?
Lower Bound Finding better Analysis of the Problem
We must look at the input for P (really?). If that takes g (n) effort, P
has a lower bound of (g (n)). We cannot do better than (g (n)).
Lower bound is established by Adversary Argument. Let P be the
problem of sorting n numbers. We claim that P is (n), that is, one
must look at all inputs (known as Input Complexity). If not, suppose it
is enough to consider only n 1 numbers. Then we can always design
the nth number as larger than the largest and make the sorting fail.
Input complexity may be lower than (n). Let P be a problem to find
a number that is not the largest in a list of n numbers. P is (2).
41 / 44
When to analyse?
Lower Bound Finding better Analysis of the Problem
We must look at the input for P (really?). If that takes g (n) effort, P
has a lower bound of (g (n)). We cannot do better than (g (n)).
Lower bound is established by Adversary Argument. Let P be the
problem of sorting n numbers. We claim that P is (n), that is, one
must look at all inputs (known as Input Complexity). If not, suppose it
is enough to consider only n 1 numbers. Then we can always design
the nth number as larger than the largest and make the sorting fail.
Input complexity may be lower than (n). Let P be a problem to find
a number that is not the largest in a list of n numbers. P is (2).
Similarly Output Complexity can be defined.
41 / 44
When to analyse?
Lower Bound Finding better Analysis of the Problem
We must look at the input for P (really?). If that takes g (n) effort, P
has a lower bound of (g (n)). We cannot do better than (g (n)).
Lower bound is established by Adversary Argument. Let P be the
problem of sorting n numbers. We claim that P is (n), that is, one
must look at all inputs (known as Input Complexity). If not, suppose it
is enough to consider only n 1 numbers. Then we can always design
the nth number as larger than the largest and make the sorting fail.
Input complexity may be lower than (n). Let P be a problem to find
a number that is not the largest in a list of n numbers. P is (2).
Similarly Output Complexity can be defined.
41 / 44
When to analyse?
Lower Bound Finding better Analysis of the Problem
We must look at the input for P (really?). If that takes g (n) effort, P
has a lower bound of (g (n)). We cannot do better than (g (n)).
Lower bound is established by Adversary Argument. Let P be the
problem of sorting n numbers. We claim that P is (n), that is, one
must look at all inputs (known as Input Complexity). If not, suppose it
is enough to consider only n 1 numbers. Then we can always design
the nth number as larger than the largest and make the sorting fail.
Input complexity may be lower than (n). Let P be a problem to find
a number that is not the largest in a list of n numbers. P is (2).
Similarly Output Complexity can be defined.
41 / 44
When to analyse?
Lower Bound Finding better Analysis of the Problem
We must look at the input for P (really?). If that takes g (n) effort, P
has a lower bound of (g (n)). We cannot do better than (g (n)).
Lower bound is established by Adversary Argument. Let P be the
problem of sorting n numbers. We claim that P is (n), that is, one
must look at all inputs (known as Input Complexity). If not, suppose it
is enough to consider only n 1 numbers. Then we can always design
the nth number as larger than the largest and make the sorting fail.
Input complexity may be lower than (n). Let P be a problem to find
a number that is not the largest in a list of n numbers. P is (2).
Similarly Output Complexity can be defined.
41 / 44
When to analyse?
Determination of Quality of an Algorithm
Core Idea: Are we doing well, and can we do better?
Let A be an algorithm for solving a problem P of size n.
Is A the best algorithm to solve P?
Optimal Algorithm
42 / 44
When to analyse?
Determination of Quality of an Algorithm
Core Idea: Are we doing well, and can we do better?
Let A be an algorithm for solving a problem P of size n.
Is A the best algorithm to solve P?
Optimal Algorithm
A is Optimal for P, if O(f (n)) = (f (n)), that is, the lower bound
matches the upper bound.
42 / 44
When to analyse?
Determination of Quality of an Algorithm
Core Idea: Are we doing well, and can we do better?
Let A be an algorithm for solving a problem P of size n.
Is A the best algorithm to solve P?
Optimal Algorithm
A is Optimal for P, if O(f (n)) = (f (n)), that is, the lower bound
matches the upper bound.
Complexity of A is (f (n)).
42 / 44
When to analyse?
Determination of Quality of an Algorithm
Core Idea: Are we doing well, and can we do better?
Let A be an algorithm for solving a problem P of size n.
Is A the best algorithm to solve P?
Optimal Algorithm
A is Optimal for P, if O(f (n)) = (f (n)), that is, the lower bound
matches the upper bound.
Complexity of A is (f (n)).
Till an optimal algorithm is found we continuously work on the
algorithm to lower the upper bound and devise adversary arguments
to upper the lower bound.
42 / 44
When to analyse?
Analysis of Sorting
Upper Bound
Lower Bound
Optimal Algorithm
43 / 44
When to analyse?
Analysis of Sorting
Upper Bound
Brute-force: Generate all n! permutations and select the sorted
sequence: O(nn ), O(nn )
Lower Bound
Optimal Algorithm
43 / 44
When to analyse?
Analysis of Sorting
Upper Bound
Brute-force: Generate all n! permutations and select the sorted
sequence: O(nn ), O(nn )
Bubble Sort: O(n2 ), O(n2 )
Lower Bound
Optimal Algorithm
43 / 44
When to analyse?
Analysis of Sorting
Upper Bound
Brute-force: Generate all n! permutations and select the sorted
sequence: O(nn ), O(nn )
Bubble Sort: O(n2 ), O(n2 )
Insertion Sort: O(n2 ), O(n2 )
Lower Bound
Optimal Algorithm
43 / 44
When to analyse?
Analysis of Sorting
Upper Bound
Brute-force: Generate all n! permutations and select the sorted
sequence: O(nn ), O(nn )
Bubble Sort: O(n2 ), O(n2 )
Insertion Sort: O(n2 ), O(n2 )
Quick Sort: O(n2 ), O(n log n)
Lower Bound
Optimal Algorithm
43 / 44
When to analyse?
Analysis of Sorting
Upper Bound
Brute-force: Generate all n! permutations and select the sorted
sequence: O(nn ), O(nn )
Bubble Sort: O(n2 ), O(n2 )
Insertion Sort: O(n2 ), O(n2 )
Quick Sort: O(n2 ), O(n log n)
Merge Sort: O(n log n), O(n log n)
Lower Bound
Optimal Algorithm
43 / 44
When to analyse?
Analysis of Sorting
Upper Bound
Brute-force: Generate all n! permutations and select the sorted
sequence: O(nn ), O(nn )
Bubble Sort: O(n2 ), O(n2 )
Insertion Sort: O(n2 ), O(n2 )
Quick Sort: O(n2 ), O(n log n)
Merge Sort: O(n log n), O(n log n)
Heap Sort: O(n log n), O(n log n)
Lower Bound
Optimal Algorithm
43 / 44
When to analyse?
Analysis of Sorting
Upper Bound
Brute-force: Generate all n! permutations and select the sorted
sequence: O(nn ), O(nn )
Bubble Sort: O(n2 ), O(n2 )
Insertion Sort: O(n2 ), O(n2 )
Quick Sort: O(n2 ), O(n log n)
Merge Sort: O(n log n), O(n log n)
Heap Sort: O(n log n), O(n log n)
Lower Bound
Input Complexity: (n) must read all numbers.
Optimal Algorithm
43 / 44
When to analyse?
Analysis of Sorting
Upper Bound
Brute-force: Generate all n! permutations and select the sorted
sequence: O(nn ), O(nn )
Bubble Sort: O(n2 ), O(n2 )
Insertion Sort: O(n2 ), O(n2 )
Quick Sort: O(n2 ), O(n log n)
Merge Sort: O(n log n), O(n log n)
Heap Sort: O(n log n), O(n log n)
Lower Bound
Input Complexity: (n) must read all numbers.
Comparison Tree: (n log n) choose from one of n! or O(nn )
permutations with binary choice at every stage. Hence, O(log nn ) =
O(n log n).
Optimal Algorithm
43 / 44
When to analyse?
Analysis of Sorting
Upper Bound
Brute-force: Generate all n! permutations and select the sorted
sequence: O(nn ), O(nn )
Bubble Sort: O(n2 ), O(n2 )
Insertion Sort: O(n2 ), O(n2 )
Quick Sort: O(n2 ), O(n log n)
Merge Sort: O(n log n), O(n log n)
Heap Sort: O(n log n), O(n log n)
Lower Bound
Input Complexity: (n) must read all numbers.
Comparison Tree: (n log n) choose from one of n! or O(nn )
permutations with binary choice at every stage. Hence, O(log nn ) =
O(n log n).
Optimal Algorithm
Merge Sort / Heap Sort
Partha Pratim Das (IIT, Kharagpur)
43 / 44
Summary
Need for analysing the running-time and space requirements of a
program.
Asymptotic growth rate or order of the complexity of different
algorithms.
Worst-case, average-case and best-case analysis.
Developing recurrence equations.
Solving recurrences using Inductive Derivation Procedure and the
Master Theorem.
44 / 44