Asymptotic Analysis

Review essential math Big O, Big Omega, Big Theta Analyzing algorithms

Based on: Data Structures and Algorithms by Clifford A. Shaffer (Virgina Tech Univ.)
Wednesday, June 20, 2012 1

Goals of this Course

Reinforce the concept that costs and benets exist for every data structure. Learn the commonly used data structures.
These form a programmer's basic data structure toolkit.

Understand how to measure the cost of a data structure or program.

These techniques also allow you to judge the merits of new data structures that you or others might invent.

Wednesday, June 20, 2012

The Need for Data Structures

Data structures organize data
Such organization leads to more efcient programs

Computers keep getting more powerful

Allowing more complex applications.

More complex applications demand more calculations. Complex computing tasks are unlike our everyday experience.

Wednesday, June 20, 2012

Organizing Data

Any organization for a collection of objects can be searched, processed in any order, or modied. The choice of data structure and algorithm can make the difference between a program running in a few seconds or many days.

Wednesday, June 20, 2012

Efciency

A solution is said to be efcient if it solves the problem within its resource constraints of:
Space (amount of memory, RAM, used) Time

The cost of a solution is the amount of resources that the solution consumes.

Wednesday, June 20, 2012

Selecting a Data Structure

Select a data structure as follows:

Analyze the problem to determine the basic operations that must be supported. Quantify the resource constraints for each operation. Select the data structure that best meets these requirements.

Wednesday, June 20, 2012

Some Questions to Ask

Are all data inserted into the data structure at the beginning, or are insertions interspersed with other operations? Can data be deleted? Are all data processed in some well-dened order, or is random access allowed?

Wednesday, June 20, 2012

Costs and Benets

Each data structure has costs and benets. Rarely is one data structure better than another in all situations. Any data structure requires:
space for each data item it stores, time to perform each basic operation, programming effort.

Wednesday, June 20, 2012

Costs and Benets (cont)

Each problem has constraints on available space and time. Only after a careful analysis of problem characteristics can we know the best data structure for the task. Bank (real) example:
Start account: a few minutes Transactions: a few seconds Close account: overnight

Wednesday, June 20, 2012

Mathematical Background
Set concepts and notation Logarithms Recursion Induction Proofs Summations Recurrence Relations

Wednesday, June 20, 2012

10

Set

Wednesday, June 20, 2012

11

Logarithms
Denition: The logarithm is the inverse function of the function that raises a base to an exponent
Example: The inverse function of y=2x is x=log2 y

Wednesday, June 20, 2012

12

Summation

Wednesday, June 20, 2012

13

Mathematical Proof

Three ways of doing a mathematical proof

Direct Proof Proof by Contradiction Proof by mathematical induction

Wednesday, June 20, 2012

14

Algorithm Efciency
There are often many approaches (algorithms) to solve a problem. How do we choose between them? At the heart of computer program design are two (sometimes conicting) goals.
1. To design an algorithm that is easy to understand, code, debug. 2. To design an algorithm that makes efcient use of the computers resources.

Wednesday, June 20, 2012

15

Algorithm Efciency (cont)

Goal (1) is a desirable goal of every developer and in large projects it is the concern of Software Engineering.
CECS 277, 343

Goal (2) is the concern of data structures and algorithm analysis.

CECS 274, 277, 328

When goal (2) is important, how do we measure an algorithms cost?

Wednesday, June 20, 2012

16

How to Measure Efciency?

1. Empirical comparison

Run program many times, on different inputs, measure the amount of time taken

2.Asymptotic Algorithm Analysis

Analyze the performance of the algorithm as the size of the problem becomes larger

Critical resources: memory and CPU Factors affecting running time:

For most algorithms, running time depends on size of the input. Running time is expressed as T(n) for some function T on input size n.
Wednesday, June 20, 2012 17

Examples of Growth Rate

Example 1.
// Return position of largest value in "A" static int largest(int[] A) { int indexOfLarge = 0; // Position of largest

for (int i=1; i<A.length; i++) if (A[indexOfLarge] < A[i]) indexOfLarge = i; // Remember pos return indexOfLarge; } // Return largest pos

Wednesday, June 20, 2012

18

Examples (cont)
Example 2: The assignment statement. Example 3: sum = 0; for (i=1; i<=n; i++) for (j=1; j<n; j++) sum++; }

Wednesday, June 20, 2012

19

Growth Rate Graph

Wednesday, June 20, 2012 20

Best, Worst, Average Cases

Not all inputs of a given size will take an algorithm the same time to run. Linear search for K in an array of n integers:
Begin at rst element in array and look at each element in turn until K is found

Best case? Worst case? Average case?

Wednesday, June 20, 2012

21

Which Analysis to Use?

While average time appears to be the fairest measure, it may be difcult to determine. When is the worst case time important?

Wednesday, June 20, 2012

22

Faster Computer or Algorithm?

Suppose we buy a computer 10 times faster. n: size of input that can be processed in one second on old computer (in 1000 computational units) n: size of input that can be processed in one second on new computer (in 10,000 computational units) T(n) 10n 10n2 10n n 100 10 3 n 1,000 31.6 4 Change n = 10n n= 10n n = n + 1 n/n 10 3.16 1 + 1/n

Wednesday, June 20, 2012

23

Asymptotic Analysis: Big-oh

Denition: For T(n) a non-negatively valued function, T(n) is in the set O(f(n)) if there exist two positive constants c and n0 such that T(n) <= cf(n) for all n > n0. Use: The algorithm is in O(n2) in [best, average, worst] case. Meaning: For all data sets big enough (i.e., n> n0), the algorithm always executes in less than cf(n) steps in [best, average, worst] case.
Wednesday, June 20, 2012 24

Big-oh Notation (cont)

Big-oh notation indicates an upper bound. Example: If T(n) = 3n2 then T(n) is in O(n2). Look for the tightest upper bound:
While T(n) = 3n2 is in O(n3), we prefer O(n2).

Wednesday, June 20, 2012

25

Big-Oh Examples
Example 1: Finding value X in an array (average cost). Use Linear Search algorithm Asymptotic Analysis:
Then T(n) = csn/2. For all values of n > 1, csn/2 <= csn. Therefore, the denition is satised for f(n)=n, n0 = 1, and c = cs. Hence, T(n) is in O(n).

Wednesday, June 20, 2012

26

Big-Oh Examples
Example 2:
Suppose T(n) = c1n2 + c2n, where c1 and c2 are positive. c1n2 + c2n <= c1n2 + c2n2 <= (c1 + c2)n2 for all n > 1. Then T(n) <= cn2 whenever n > n0, for c = c1 + c2 and n0=1.

Therefore, T(n) is in O(n2) by denition. Example 3: T(n) = c. Then T(n) is in O(1).

Wednesday, June 20, 2012

27

A Common Misunderstanding

The best case for my algorithm is n=1 because that is the fastest. WRONG! Big-oh refers to a growth rate as n grows to . Best case is dened for the input of size n that is cheapest among all inputs of size n.

Wednesday, June 20, 2012

28

Big-Omega
Denition: For T(n) a non-negatively valued function, T(n) is in the set (g(n)) if there exist two positive constants c and n0 such that T(n) >= cg(n) for all n > n0. Meaning: For all data sets big enough (i.e., n > n0), the algorithm always requires more than cg(n) steps.

Lower bound.
Wednesday, June 20, 2012 29

Big-Omega Example

T(n) = c1n2 + c2n. c1n2 + c1n >= c1n2 for all n > 1. T(n) >= cn2 for c = c1 and n0 = 1. Therefore, T(n) is in (n2) by the denition. We want the greatest lower bound.

Wednesday, June 20, 2012

30

Theta Notation

When big-Oh and coincide, we indicate this by using (big-Theta) notation. Denition: An algorithm is said to be in (h(n)) if it is in O(h(n)) and it is in (h(n)).

Wednesday, June 20, 2012

31

A Common Misunderstanding
Confusing worst case with upper bound. Upper bound refers to a growth rate. Worst case refers to the worst input from among the choices for possible inputs of a given size.

Wednesday, June 20, 2012

32

Time Complexity Examples (1)

Example: The assignment statement
a = b; This assignment takes constant time, so it is (1).

Example: Simple loop

sum = 0; for (i=1; i<=n; i++) sum += n;

Wednesday, June 20, 2012

33

Time Complexity Examples (2)

Example, nested loops: sum = 0; for (j=1; j<=n; j++) for (i=1; i<=j; i++) sum++; for (k=0; k<n; k++) A[k] = k;

Wednesday, June 20, 2012

34

Time Complexity Examples (3)

Example, continued: sum1 = 0; for (i=1; i<=n; i++) for (j=1; j<=n; j++) sum1++; sum2 = 0; for (i=1; i<=n; i++) for (j=1; j<=i; j++) sum2++;
Wednesday, June 20, 2012 35

Time Complexity Examples (4)

Example -- More nested loops:
sum1 = 0; for (k=1; k<=n; k*=2) for (j=1; j<=n; j++) sum1++; sum2 = 0; for (k=1; k<=n; k*=2) for (j=1; j<=k; j++) sum2++;

Wednesday, June 20, 2012

36

Other Control Statements

while loop
Analyze like a for loop.

if statement
Take greater complexity of then/else clauses.

switch statement
Take complexity of most expensive case.

Subroutine (method) call

Complexity of the subroutine.

Wednesday, June 20, 2012

37

Binary Search

Search for an element with value k in sorted array How many elements are examined...
in the best case? worst case? average case?

Wednesday, June 20, 2012

38