Escolar Documentos
Profissional Documentos
Cultura Documentos
1. Algorithm Analysis
1. Algorithm Analysis
1.1 Mathematics Review 1.2 Introduction to Algorithm Analysis 1.3 Asymptotic notation and Growth of functions 1.4 Case Study 1.5 Data Structures and Algorithm Analysis
Exponents XA XB XA XB (X A )B XN + XN 2N + 2N X A+ B X A B X AB 2N +1
= = = =
= 2 X N = X 2N
Logarithms By default, logarithms used in this course are to the base 2. X A = B logX B = A logA B =
logC B logC A ,
A, B, C > 0, A = 1
log AB = log A + log B ; A, B > 0 log A/B = log A log B log (AB ) = B log A log X < X X > 0 log 1 = 0, log 2 = 1, log 1, 024 =
Malek Mouhoub, CS340 Fall 2010
Summations
N i=1
ai = a1 + a2 + + aN
N i=1
limN Linearity
N i=1 N i=1
ai =
i=1
ai = a1 + a2 + (innite sum)
N i=1 N i=1
ai + d
bi
i=1
f (i))
f (N ) = N f (N ) f (i) =
N i=1
f (i)
n0 1 i=1
f (i)
Summations
Geometric series :
if 0
N i=0
Ai =
AN +1 1 A 1 N i=0
Ai
1 1 A
Arithmetic series :
N i=1 N i=1 N i=1
i=
N (N +1) 2
N 2 +N 2
N2 2
i2 = ik
N3 3
k = 1
N 1 i=1 i
if k = 1 then HN =
loge N
Products
N i=1
ai = a1 a2 aN
N i=1
log(
ai ) =
N i=1
log ai
Proving statements
Proof by induction
1. Proving a base case : establishing that a theorem is true for some small values. 2. Inductive hypothesis : the theorem is assumed to be true for all cases up to some limit
k.
3. Given this assumption, show that the theorem is true for k
+1
Proof by Counter example : nd an example showing that the theorem is not true. Proof by Contradiction : Assuming that the theorem is false and showing that this
assumption implies that some known property is false, and hence the original assumption was erroneous.
Boss gives the following problem to Mr Dupont, a fresh hired BSc in computer science (to test him . . . or may be just for fun) :
T (1) T (2)
What is T (100) ?
= 3 = 10
T (n) = 2T (n 1) T (n 2)
Mr Dupont decides to directly code a recursive function tfn, in Java, to solve the problem : if (n==1) { return 3; } else if (n==2) {return 10;} else { return 2 tfn(n-1) - tfn(n-2); } 1. First mistake : no analysis of the problem
risk to be red !
2. Second mistake : bad choice of the programming language
10
3 10 17
it takes 4.19 seconds waits . . . and then kills the program !
n = 35 n = 50
it takes only 1.25 seconds waits and then kills the program !
11
Finally, Mr Dupont decides to (experimentally) analyze the problem : he times both programs and plots the results.
100
seconds
10
Java C
0.1 25 30 35 40
It seems that each time n increases by 1 the time increases by 1.62. At n the C program took 13.79 seconds, so for n
= 40
= 100 he estimates :
12
13
This solution provides a much better complexity in time but despite of the space complexity :
n = 100 takes only a fraction of a second, but for n = 10, 000, 000 (a test that may make the boss happy . . . if it succeeds) a segmentation fault occurs. Too much
memory required.
14
15
How to solve such problems ? 1. Analyze the problem on paper in order to nd an efcient algorithm in terms of time and memory space complexity. (a) First look at the problem :
= = = = =
3 10 17 24 31
T (n) = 7n 4
return 7n-4
16
An algorithm analyst might ask : 1. What makes the rst program so slow ? 2. How fast are the 3 programs asymptotically ? 3. Is the last version really the ultimate solution ?
17
Let us look at the recursion tree for the rst program at n=4.
4 3 2 1 2
Here each circle represents one call to the routine tfn. So, for n=4 there are 5 such calls. In general, a call to tfn(n) requires a recursive call to tfn(n-1)(represented by the shaded region on the left) and a call to tfn(n-2)(shaded region on the right).
18
f (n) f (1)
= =
f (n 1) + f (n 2) + 1 f (2) = 1
1.618n .
This agrees very well with the times we presented earlier where each increase of n by 1 increases the time by a factor of a little under 1.62. We say such growth is exponential with asymptotic growth rate O (1.618 This answers question (1).
n
).
19
for (i=3;i<=n;i++)
This loop contained two or three assignments, a multiplication and a subtraction. We say such a loop takes O (n) time. This means that running time is proportional to n. Recall that increasing n from 100 million to 300 million increased the time from approximately 3 to approximately 9 seconds. The last program has one multiplication and one subtraction and takes O (1) or constant time. This answers question (2).
20
The answer to the last question is also NO. If the boss asked for
T (123456789879876543215566340014733134213)
we would get integer overow on most computers.
21
The addition and subtraction require a linear-time algorithm. A simple algorithm for multiplication requires a quadratic-time
cost.
22
The third question, is the last program the ultimate solution, is more of a computer science question. A Computer Scientist might ask : 1. How do you justify counting function calls in the rst case, counting array assignments in the second case, counting variable assignments in the third, and counting arithmetic operations in the last ? 2. Is it really true that you can multiply two arbitrary large numbers together in constant time ? 3. Is the last program really the ultimate one ?
23
What data structures are best, and in what situations ? Which models should we use to analyze algorithms in practice ? When trying to improve the efciency of a given program, which
aspects should we focus on rst ?
24
T (N ) = f (N ) where N is in general a natural number. The exact value of the function depends on :
the speed of the machine; the quality of the compiler and optimizer; the quality of the program that implements the algorithm; the basic fundamentals of the algorithm
Worst-case running time is a bound over all inputs of a certain size N . (Guarantee) Average-case running time is an average over all inputs of a certain size N . (Prediction)
26
-notation
For a given function g (n), we denote by (g (n)) the set of functions
(g (n)) = {f (n) : c1 , c2 , and n0 such that 0 c1 g (n) f (n) c2 g (n) for all n n0 }
We say that g (n) is an asymptotically tight bound for f (n). Example : The running time of insertion sort is T (n)
= (n2 ).
27
-notation
For a given function g (n), we denote by (g (n)) the set of functions :
28
Big-Oh notation For a given function g (n), we denote by O (g (n)) the set of functions :
O(g (n)) = {f (n) : c and n0 such that 0 f (n) cg (n) for all n n0 }
We say that g (n) is an asymptotic upper bound for f (n). Note that O -notation is, in general, used informally to describe asymptotically tight bounds (-notation).
29
Big-Oh notation
Exponential : dominant term is some constant times 2N . Cubic : dominant term is some constant times N 3 . We say O(N 3 ). Quadratic : dominant term is some constant times N 2 . We say O(N 2 ). O(N log N ) : dominant term is some constant times N log N . Linear : dominant term is some constant times N . We say O(N ). Logarithmic : dominant term is some constant times log N. Constant : c.
30
Obvious algorithm is sequential scan. Running time is O(N ) (linear) because we repeat a xed
amount of work for each element in the array.
32
Fundamental problem in graphics. Solution : Calculate the distance between each pair of points,
and retain the minimum distance.
N (N 1)/2 pairs of points, so the algorithm is quadratic. Better algorithms that use more subtle observations are known.
33
34
35
The problem
+ Ai+1 + + Aj ).
Examples :
-2, 11, -4, 13, -4, 2 1, -3, 4, -2, -1, 6
36
37
Analysis
Loop of size N inside of loop of size N inside of loop of size N means O (N 3 ), or cubic algorithm. Slight over-estimate that results from some loops being of size less than N is not important.
38
Can use this to estimate time for larger inputs : T (N ) = cN 3 T (10N ) = c(10N )3 = 1000cN 3 = 1000T (N ) Inputs size increases by a factor of 10 means that running time
increases by a factor of 1,000.
How to improve
Remove a loop; not always possible. Here it is : innermost loop is unnecessary because it throws
away information.
+ Ai+1 + + Aj 1 + Aj + Ai+1 + + Aj 1
Just computed Ai
40
41
Analysis
Same logic as before : now the running time is quadratic, or O(N 2 ). As we will see, this algorithm is still usable for inputs in the tens
of thousands.
Recall that the cubic algorithm was not practical for this amount
of input.
42
Can use this to estimate time for larger inputs : T (N ) = cN 2 T (10N ) = c(10N )2 = 100cN 2 = 100T (N ) Inputs size increases by a factor of 10 means that running time
increases by a factor of 100.
Linear Algorithms
Linear algorithm would be best. Running time is proportional to amount of input. Hard to do
better for an algorithm.
44
Recursive algorithm
Compute all three possibilities, and use the maximum. First two possibilities easily computed recursively.
Malek Mouhoub, CS340 Fall 2010 45
Implementation :
Easily done with two loops. For maximum sum that starts in the rst half and extends to the last element in the rst half, use a right-to-left scan starting at the last element in the rst half. For the other maximum sum, do a left-to-right scan, starting at the rst half.
46
Analysis
Let T (N ) = the time for an algorithm to solve a problem of size N . Then T (1) = 1 (1 will be the quantum time unit; constants
dont matter).
T (N ) = 2T (N/2) + N
Two recursive calls, each of size N/2. The time to solve each recursive call is T (N/2) by the above denition. Case three takes O (N ) time; we use N , because we will throw out the constants eventually.
47
Bottom Line
1=11 2 T (1) + 2 = 4 = 2 2 = 21 2 2 T (2) + 4 = 12 = 4 3 = 22 3 2 T (4) + 8 = 32 = 8 4 = 23 4 2 T (8) + 16 = 80 = 16 5 = 24 5 2 T (16) + 32 = 192 = 32 6 = 25 6 2k (k + 1) = N (1 + log N ) = O(N log N )
48
N log N Any recursive algorithm that solves two half-sized problems and
does linear non-recursive work to combine/split these solutions will always take O (N will always hold.
This is a very signicant improvement over quadratic. It is still not as good as O(N ), but is not that far away either.
There is a linear-time algorithm for this problem. The running time is clear, but the correctness is non-trivial.
49
/** Linear-time maximum contiguous subsequence. sum algorithm */ int maxSubSum4( const vector<int> & a ) { /* 1*/ int maxSum = 0, thisSum = 0; /* 2*/ for( int j = 0; j < a.size( ); j++ ) { /* 3*/ thisSum += a[ j ]; /* 4*/ if( thisSum > maxSum ) /* 5*/ maxSum = thisSum; /* 6*/ else if( thisSum < 0 ) /* 7*/ thisSum = 0;} /* 8*/ return maxSum;}
50
The Logarithm
Formal Denition
For any B,
N > 0, logB N = K if B K = N
Examples :
log 32 = 5 (because 25 = 32) log 1024 = 10 log 1048576 = 20 log 1billion = about 30
The logarithm grows much more slowly than N , and slower than N .
Malek Mouhoub, CS340 Fall 2010 51
Examples of the Logarithm Bits in a binary number : how many bits are required to represent
N consecutive integers ?
Repeated doubling : starting from X
= 1, how many times should X be doubled before it is at least as large as N ? = N , if N is repeatedly halved, how many iterations must be applied to make N smaller than or equal to 1 ? (Halving rounds up).
52
Why log N
B bits represents 2B integers. Thus 2B is at least as big as N , so B is at least log N . Since B must be an integer, round up if
needed.
53
54
Static Searching
Given an integer X and an array A, return the position of X in A or an indicator that it is not present. If X occurs more than
once, return any occurrence. The array A is not altered.
Worst case : O(N ); every item is examined. Average case :O(N/2); half the items are examined. Can we do better if we know the array is sorted ?
Malek Mouhoub, CS340 Fall 2010 55
Binary Search
O(N ) time per operation, because we must insert O(N ) time per operation, because we must slide
elements that are to the right of the deleted element over one spot to maintain contiguity. Find :
57
A scalar item
A sequential vector
A linked list
A n-dimentional space
Malek Mouhoub, CS340 Fall 2010
A hierarchical tree
58
The most important property to express of any entity in a system is its type. In this course we used entities that are structured objects (e.g., an object that is a collection
of other objects).
59
60
waste of memory space. O(N ) for find, constant time for findKth But O(N ) is required for insertion and deletion in the worst
case.
findKth(3)=52 List 1 2 3 4 5 34 12 52 16 22 1 2 3 4 5
find(52)=3 34 12 52 16 22 1 2 3 4 5
remove(34) 12 52 16 22 1 2 3 4 5
insert(1,34) 34 12 52 16 12
n findKth=List[Kth] O(C)
n find(X): O(n)
n insert(kth,X) O(n)
62
Linked Lists
Non contiguous allocation of memory. O(N ) for find O(N ) for findKth (but better time in practice if the calls to findKth are in sorted order by the argument). Constant time for insertion and deletion.
63
Head 1 2 3 4 5 6 7 8
Data 34 52 22 12 16
Link 5 Head 7 \ 3 4 34 12 52 16 22
64