Lec 02 Intro Algorithms

Advanced Data Structures and
Algorithms

CS 361 - Fall 2014
Tamer Nadeem
Dept. of Computer Science
Lec. #02: Introduction to Algorithm Analysis

Page 2 CS 361 - Advanced Data Structures and Algorithms Fall 2014
Class Objective/Overview
Main Objective: Start thinking about designing and analyzing
algorithms.
Doing a Timing Analysis
Understand running-time complexity/analysis of an algorithm
using Big-O notation and Big-Oh Operations
Apply Big-O analysis on simple sorting/searching algorithms
Worst-Case and Average Case Analysis
Understand simple sorting and searching techniques that will be
used as examples.
Understand and use of recursion.

Introduction
Algorithm
The basic method; it determines the data-items computed.
Also, the order in which those data-items are computed (and hence the order
of read/write data-access operations).
An algorithm can be analyzed in terms of time efficiency and/
or space utilization.
An algorithms running time is influenced by several factors:
Speed of the machine running the program
Language in which the program was written. Programs in assembly language
run faster than those in C or C++, which in turn run faster than those in Java.
Efficiency of the compiler that created the program
The size of the input: processing 1000 records will take more time than
processing 10 records.
Organization of the input: item at the top of the list will take less time to find

Program = Algorithm + Data Structure

Algorithm Complexity
A code of an algorithm is judged by its correctness, its ease
of use, and its efficiency.
This course focus on the computational complexity (time
efficiency) of algorithms that apply to container objects (data
structures) that hold a large collection of data.
We will learn how to develop measures of efficiency (criteria)
that depends on n, the number of data items in the container.
The criteria of computational complexity often include the
number of comparison tests and the number of assignment
statements used by the algorithm.

Running Time
The term running time is often used to refer to the
computational complexity/time efficiency (how fast the
algorithm).
Approaches:
Theoretical analysis Empirical Analysis
Empirical analysis:
Count the number of times specic operations are performed by
executing an instrumented version of the program.
Measure directly the actual program-execution time in a run.
Example:
Instrumented code: countComparisons++; //initialized elsewhere
if (x < y) small = x;
else small = y;

Running Time
The running time depends on the input (e.g., an already
sorted sequence is easier to sort).
Parameterize the running time by the size of the input n
Seek upper bounds on the running time T(n) for the input
size n, because everybody likes a guarantee.
T(n) function counts the frequency of the basic operations
in terms of n.
Number of times the instructions in the algorithm are
executed.

Theoretical analysis of running time
Running time T(n) is analyzed by determining the number of
repetitions of the basic operation as a function of input size (n)
Basic operation: the operation that contributes most towards the
running time of the algorithm
Parameterize the running time by the size of the input n
Seek upper bounds on the running time T(n) for the input size n,
because everybody likes a guarantee.

running time
execution time
for basic operation
Number of times
basic operation is
executed
input size
T(n) ! c
op
C(n)

Example on Analyzing Running Time
1. n = read input from user
2. sum = 0
3. i = 0
4. while i < n
5. number = read input from user
6. sum = sum + number
7. i = i + 1
8. mean = sum / n
Pseudo code algorithm
illustrates the calculation of
the mean (average) of a set
of n numbers:
The running time/computing
time/time complexity/time
efficiency of this algorithm in
terms of input size n is:

T(n) = 4n + 5.
Statement Number of times executed
1 1
2 1
3 1
4 n+1
5 n
6 n
7 n
8 1

Another Example of Runtime Analysis
The total running time, T(N), for this algorithm is:
T(N) = (2N
2
+ 3N + 1) * t
asst
+ (N
2
+ 2N + 1) * t
comp
+ (4N
2
+ 2N) * t
add
+ (3N
2
+ N) * t
mult

= c
1
N
2
+ c
2
N + c
3
1: for ( i = 0; i < N; ++ i ) {
2: a [ i ] = 0;
3: for ( j = 0; j < N; ++ j )
4: a [ i ] = a [ i ] + i * j ;
5: }
c
1
= 2t
asst
+ t
comp
+ 4t
add
+ 3t
mult
c
2
= 3t
asst
+ 2t
comp
+ 2t
add
+ t
mult
c
3
= t
asst
+ t
comp

Example: Sequential Search
Statement Number of times executed
1 1
2 ?
3 ?
4 1
1.
2.
3.
4.

Best-case, Average-case, Worst-case
For some algorithms efficiency depends on form of input:
Worst case: T
worst
(n) maximum running time over inputs of size n
An upper bound on the running time for any input
guarantee that the algorithm will never take longer
The worst case can occur fairly often
Best case: T
best
(n) minimum over inputs of size n
Average case: T
avg
(n) average over inputs of size n
Number of times the basic operation will be executed on typical input
NOT the average of worst and best case
Expected number of basic operations considered as a random variable
under some assumption about the probability distribution of all possible
inputs
May be difficult to calculate/define what average means

Back to Sequential Search Example
Worst case: T
worst
(n) =
Best case: T
best
(n) =
Average case: T
avg
(n) =

Order of growth
Example: T(n) = cn
2

How much faster will algorithm run on computer that is twice as fast?
How much longer does it take to solve problem of double input size?
Main Idea:
Ignore machine-dependent constants.
Look at growth of T(n) as n !

Asymptotic Analysis
"

Asymptotic Notation: Big-Oh
The basic idea of big-Oh notation is:
Suppose f and g are both real-valued functions
of a real variable x.
If, for large values of n > n
0
, the graph of f lies
below the graph of some multiple (c) of g such
that: f(n) ! c g(n) when n " n
0

! f is of order g, i.e., f(n) = O(g(n)).
! The growth rate of f(n) is less than or equal to
the growth rate of g(n)
! g(n) represents an upper bound on f(n).
Also called asymptotic upper bound.

Big Oh notation is used to capture the most dominant term in
a function, and to represent the growth rate.
Ignores constant factors and small input sizes

Ex1: f(n) = 5n, g(n) = n, is f = O(g)?
we have to show the existence of a constant c as given in Definition 1.
Clearly 5 is such a constant so f(n) = 5 * g(n).
We could choose a larger C such as 6, because the definition states that
f(n) must be less than or equal to C * g(n), but we usually try and find the
smallest one.
Ex2: f(n) = 4n+5, g(n) = n, is f = O(g)?
we need to produce a constant c such that: f(n) <= c * n for large n .
If we try c = 4, this doesn't work because 4n + 5 is not less than 4n. We
can set c to at least 9 to for all values of n. If n = 1, c has to be 9, but c can
be smaller for greater values of n (if n = 100, C can be 5).
Ex3: f(n) = n
2
, g(n) = n, is f = O(g)?
Big Oh Examples

Ex4: f(n) = 4n
2
, g(n) = n
2
, is f = O(g)?
Similar to previous example, clearly we can choose c to be 5 so f(n) = 5 *
g(n).
Ex5: f(n) = n
2
+ 3n - 1, g(n) = n
2
, is f = O(g)?
f(n) = n
2
+ 3n 1
< n
2
+ 3n (subtraction makes things smaller so drop it)
<= n
2
+ 3n
2
(since n <= n
2
for all integers n)
<= 4n
2

Therefore, if C = 4, we have shown that f(n) = O(n
2
).
O(n
2
): reads order n-squared or Big-Oh n-squared
Notice that all we are doing is finding a simple function that is an upper
bound on the original function. Because of this, we could also say that:
f(n) = O(n
3
) since (n
3
) is an upper bound on n
2
Big Oh Examples

Ex6: f(n) = 2n
2
. Then
f(n) = O(n
4
)
f(n) = O(n
3
)
f(n) = O(n
2
) (best answer, asymptotically tight)
Rule of thump: Drop low-order terms; ignore leading
constants.
Examples:
100n
3
+ 30000n " O(n
3
)
3n
3
+ 90n
2
5n + 6046 " O(n
3
)
100n
3
+ 2n
5
+ 30000n " O(n
5
)
Big Oh Examples

Omega (!):
We say that f(n)= #(g(n)) iff there exists
positive constants c, and n such that c g(n) $
f(n) for all n % n
0
Example: 3n
3
+ 90n
2
5n + 6046 = #(n
3
)
!-notation & "-notation
Theta ("):
We say that f(n)= &(g(n)) iff there exist
positive constants c
1
, c
2
, and n
0
such that
c
1
g(n) $ f (n) $ c
2
g(n) for all n % n
0
In other words f(n)= &(g(n)) iff
f(n) = O(g(n)) and f(n) = #(g(n))
Example: 3n
3
+ 90n
2
5n + 6046 = &(n
3
)

Functions in order of increasing growth
rate
Function Name
C Constant
LogN Logarithmic
Log
2
N Log-squared
N Linear
NlogN NlogN
N
2
Quaratic
N
3
Cubic
2
n
Exponential

Functions in order of increasing growth
rate

Big Oh Examples
N
2
/ 2 3N = O(N
2
)
1 + 4N = O(N)
7N
2
+ 10N + 3 = O(N
2
) = O(N
3
)
log
10
N = log
2
N / log
2
10 = O(log
2
N) = O(log N)
sin N = O(1); 10 = O(1), 10
10
= O(1)

log N + N = O(N)
log
k
N = O(N) for any constant k
N = O(2
N
), but 2
N
is not O(N)
2
10N
is not O(2
N
)
) (
2
1
N O N N i
N
i
= ! "
#
=
) (
3 2
1
2
N O N N i
N
i
= ! "
#
=

Constant Time Algorithms: An algorithm is O(1) when its
running time is independent of the number of data items. The
algorithm runs in constant time.
e.g., direct insert at rear of array involves
a simple assignment statement and thus
has efficiency O(1)
Linear Time Algorithms: An algorithm is O(n) when its running
time is proportional to the size of the list.
e.g., sequential search. When
the number of elements
doubles, the number of
operations doubles.
Special Cases
front rear
Direct Insert at Rear
Sequential Search for the Minimum Element in an Array
32 46 8 12 3
minimum element found
in the list after n comparisons
n = 5
1 2 3 4 5


n
n
2
log
2
n
Polynomial Algorithms:
Algorithms with running time O(n
2
) are quadratic.
practical only for relatively small values of n.
Whenever n doubles, the running time of the algorithm increases by
a factor of 4.
Algorithms with running time O(n
3
) are cubic.
efficiency is generally poor; doubling the size of n increases the
running time eight-fold.
Logarithmic Time Algorithms: The logarithm of n, base 2, is
commonly used when analyzing computer algorithms
Ex. log
2
(2) = 1, log
2
(75) = 6.2288
When compared to the functions
n and n
2
, the function log
2
n
grows very slowly
Special Cases

Comparison of different O()
n log
2
n n log
2
n
n
2
n
3
2
n
2 1 2 4 8 4
4 2 8 16 64 16
8 3 24 64 512 256
16 4 64 256 4096 65536
32 5 160 1024 32768 4294967296
128 7 896 16384 2097152
3.4 x 10
38
1024 10 10240 1048576 1073741824
1.8 x 10
308
65536 16 1048576 4294967296
2.8 x 10
14
Forget it!

Algebraic Rule 1: f(n) + g(n) O(f(n) + g(n))
Algebraic Rule 2: f(n) * g(n) O(f(n) * g(n))
Algebraic Rule 3: O(c * f(n)) = O(f(n))

The Algebra of Big-O

Proof of Algebraic Rule 3: O(c * f(n)) = O(f(n))
1. Given running time T(n) = O(c * f(n))
2. By O() definition, c
1
,n
0
|n>n
0
! T(n) <= c
1
(c * f(n))
3. ! c
1
,n
0
|n>n
0
! T(n) <= (c
1
* c) f(n))
4. Let c
2
= c
1
* c ! c
1
,n
0
|n>n
0
! T(n) <= c
2
* f(n))
5. ! T(n) = O(f(n))
6. ! O(c * f(n)) = O(f(n))

Similary, O(c
1
* f(n) + c
2
* g(n)) = O(f(n) + g(n))

Algebraic Rule 4: Larger Terms Dominate a Sum
ifn
0
|n>n
0
, f(n)>=g(n),
then O(f(n) + g(n)) = O(f(n))


Proof of Algebraic Rule 4: n>n
0
, f(n)>=g(n),
then O(f(n) + g(n)) = O(f(n))
1. Given running time T(n) = O(f(n) + g(n))
2. By O() definition, c,n
1
|n>n
1
! T(n) <= c(f(n) + g(n))
3. Assume n>n
0
, f(n)>=g(n) then
n > max(n
0
,n
1
) ! T(n) <= c (f(n) + g(n)) ' (1)
n > max(n
0
,n
1
) ! f(n) > g(n) ' (2)
4. Using (2) to replacing g(n) by f(n) in (1) then
n > max(n
0
,n
1
) ! T(n) <= c (f(n) + f(n)) ' (3)
n > max(n
0
,n
1
) ! T(n) <= 2c * f(n) ' (4)
5. ! T(n) = O(f(n))

Algebraic Rule 5: Logarithms are Fast
k % 0, O(log
k
(n)) O(n)

Analysis Rules for Program

Statements

1. Expressions and Assignments
The complexity of an expression is the sum of the complexity of all
of the operations within it.
Assignment statements have a complexity equal to the sum of the
complexities of the expressions to either side of the = operator,
plus the complexity of the actual copy.
2. Function Calls
Function/Procedure calls are counted as the complexity of their
bodies.
3. Loops
Compute the run time by adding up the time required for all the
iterations.
Analysis Rules for Program Statements
Individual Statements

3. Loops
Compute the run time by adding up the time required for all the
iterations.
The running time of a loop is at most:
where t
init
is the time required to do the loop initialization, t
condition
is the
time to evaluate the loop condition, with t
final
condition being the time to
evaluate the loop condition the final time (when we exit the loop), and t
body

is the time required to do the loop body.
Example: A simple loop

for ( int j = 0; j < n; ++ j )
a [ i +10* j ] = i + j ;
t
loop
= O(c
init
+ ( (c
condition
+ c
increment
+ c
body
) + c
condition
)

= O(c
0
+ n * c
1
)
= O(n)

3. Conditional Statement
The worst case time for the if is the slower of the two possibilities.
The running time of an if-then-else is at most:
where t
condition
is the time to evaluate the if condition, with t
then
is the time to
do the then body, and t
else
is the time required to do the else body.
A missing else clause (or, for that matter, any empty statement
list) is O (1).

1. Sequences of Statements
The time for consecutive (straight-line) statements is the sum of the
individual statements.
2. Nested Statements
In general, nested statements are evaluated by applying the other
rules from the inside out.

Example: A simple nested loop

for ( int i = 0; i < n; ++ i )
{
for ( int j = i ; j < n; ++ j )
{
a [ i ] [ j ] = a [ j ] [ i ] ;
}
}

Combining Statements

2. Nested Statements (contd)
In general, nested statements are evaluated by applying the other
rules from the inside out.

Example: A simple nested loop

for ( int i = 0; i < n; ++ i )
{
for ( int j = i ; j < n; ++ j )
{
a [ i ] [ j ] = a [ j ] [ i ] ;
}
}

Combining Statements
Analysis of Sorting/Searching

Algorithms

Another Example: Insertion Sort
Consider A[j]=9. Not in the correct place.
Need to make room for 9.
We shift all elements right, starting from 10.

Example of Insertion Sort











Algorithm of Insertion Sort

outer loop
inner loop

Code of Insertion Sort
void insertionSort (int *a , int size)
{
int i , j , n = size;
int key;
// place a[i] into the sublist
// a[0] . . . a[i-1], 1 <= i < n,
// so it is in the correct position
for (int j=1; j<n; ++j) // outer loop
{
key = a[j];
i=j-1;
while ((i > 0) && (key < a[i] )) // inner loop
{
a[i+1] = a[i];
i = i - 1;
}
a[i+1] = key;
}
}

Comments on Insertion Sort
If the key value is greater than all the sorted elements in the
sublist, the algorithm does 0 iterations of the inner loop.
If the initial array is sorted:
Then each new key will be greater than all the ones already
inserted into the array.
For each outer loop iteration, the algorithm does 0 iterations of
the inner loop.
This special case is very common. Many practical problems
require the construction of a sorted sequence of elements
from a set of data that is already sorted or nearly so, with
only a few items out of place.
Note the "work from the back is very efficient for such inputs.

Algorithm of Insertion Sort

Insertion sort analysis
T(N) counts number of comparisons
T(N) = 1 + 2 + 3 + ' + (n-1)
= (n-1)n/2
= n
2
/2 n/2
T(N) = O(n
2
)
Worst-case Analysis
Best Case Analysis
T(N) = 1 + 1 + 1 + ' + 1
= n
T(N) = O(n)
Average Case Analysis
T(N) = O(n
2
)

Sequential Search
int seqSearch (const int arr[ ], int first, int last, int target)
// Look for target within a sorted array arr [first . . last -1].
// Return the position where found, or last if not found .
{
int i = first;
while (i < last && arr[i] != target )
i++;
return i ;
}
6 4 2 9 5 10
index = seqSearch(arr, 0, 8, 3);
7
Index 0 1 2 3 4 5 6 7
3
target = 3
8
match at index = 5
return index 5
Search algorithm start with a target value
and employ sequential visit to the
elements looking for a match.
If target is found, the index of the matching
element becomes the return value.
0 1 2 6 7 8
first last
Array arr
5 4 3

Sequential Search for a Sorted Array
int seqSearch (const int arr[ ], int first, int last, int target)
// Return the position where found, or last if not found .
{
int i = first;
while (i < last && arr[i] < target )
i++;
return i ;
}
Previous code for any general array (i.e., non sorted).
When arrays have been sorted, they could be searched using a
slightly faster variant of the sequential sort.
If array is sorted, then encountering a value greater than the target
would imply that the target value is not in the array.
Replacing the != operator by a < causes the algorithm to exit the loop
as soon as we see a value that is equal to or greater than the target .

Binary Search
An alternative method of searching sorted arrays is the
binary search
Example: Search for number 85 in the
63-entry list
Six probes are needed with a 63-entry list,
in the worst case
1
2
5
6
8
12
16
17
18
21
23
24
30
32
33
35
38
40
44
45
47
49
54
57
58
59
64
66
67
69
70
71
74
75
77
80
81
83
85
87
88
90
91
93
99
100
101
102
105
107
110
111
116
117
118
120
122
125
126
128
130
131
133
1
2
3
4
5
First Last Middle Entry Outcome
1 63 32 71 >
33 63 48 102 <
33 47 40 87 <
33 39 36 80 >
37 39 38 83 >
37 37 37 85 =

Binary Search Tree
1
2
5
6
8
12
16
17
18
21
23
24
30
32
33
35
38
40
44
45
47
49
54
57
58
59
64
66
67
69
70
71
74
75
77
80
81
83
85
87
88
90
91
93
99
100
101
102
105
107
110
111
116
117
118
120
122
125
126
128
130
131
133
1
5
8
16
18
23
30
33
38
44
47
54
58
64
67
70
74
77
81
85
88
91
99
101
105
110
116
118
122
126
130
133
2 12 21 32 40 49 59 69 75 83 90 100 107 117 125 131
6 24 45 66 80 93 111 128
17 57 87 120
35
71
102
Example 1:
Find 47
< 71
> 35
< 57
> 45
< 49
Found
63-item list
Example 2:
Find 112
> 71
> 102
< 120
> 111
< 117
< 116 Not found

Binary Search analysis
T(N) counts number of comparisons
T(N) = O(log n)
Worst-case Analysis
Best Case Analysis
T(N) = O(1)
Average Case Analysis
T(N) = O(log n)

Code of Binary Search
int binSearch (const int arr[ ], int first, int last, int target)
// Return the position where found, or , or last if not found
{
int mid; // index of the midpoint
int midValue ; // object that is assigned arr[mid]
int origLast = last ; // save original value of last
// repeatedly reduce the area of search
// until it is just one element
while (first < last) { // test for nonempty sublist
mid = (first + last) / 2;
midValue = arr[mid] ;
if (target == midValue)
return mid;
else if (target < midValue)
last = mid; // search lower sublist , reset last
else
first = mid+1; // search upper sublist , reset first
}
return origLast;
}
Recursion

The Concept of Recursion
Lets consider the problem of
evaluating the power x
n
where x
is real number and n is a
nonnegative integer.
Double power(double x, int n){
double product = 1;
int i;

for (i = 1; i <= n; i++)
product *= x; //x
n
= x**x (n times)

return product;
}
Iterative Approach
Another approach, since x
n
= x
m

* x
(n-m)
, we can split the problem
to smaller problems. For
example, 2
15
could be computed
as 2
5
* 2
10
= 32 * 1024.
Recursion, in simple, is solving a problem by solving smaller problems
of same form and using their results to compute the final result.
Recursion is the process a function goes through when one of the
steps of the function involves invoking the function itself. A procedure
that goes through recursion is said to be 'recursive'.

The Concept of Recursion
Double power(double x, int n){
if (n == 0)
return 1;
else
return x * power(x, n-1);
}
Recursive Approach
A function exhibits recursive behavior when it can be defined
by two properties:
A simple base case (or cases) (stopping condition)
A set of rules that reduce all other cases toward the base
case

Example: Computing x
n
Base case: x
0
= 1
Rule: x
n
= x
n-1
* x
Double factorial(int x){
if (x == 0)
return 1;
else
return x * factorial(x-1);
}
Recursive Approach
Example: Computing x!
Base case: 0! = 1
Rule: x! = x

* (x-1)!

Fibonacci Numbers
int fib(int n){
if (n <= 1)
return n;
else
return fib(n-1) + fib(n-2);
}
Recursive Approach
Fibonacci numbers are the sequence of integers
0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, '
Example: Computing fib(n) //Fibonacci element with index n

Base case: fib(0) = 0
fib(1) = 1
Rule: fib(n) = fib(n-1)
+
fib(n-2)
int fibiter(int n){
int oneback = 1, twoback = 0, current, i;

if (n <= 1)
return n;
else
for (i=2; i <= n; i++){
current = oneback + twoback;
twoback = oneback;
oneback = current;
}
return current;
}
Iterative Approach

Evaluating Fibonacci Numbers
#include <iostream>
#include d_timer.h

int fibiter(int n){ //from previous slide
. . . . .
}

int fib(int n){ //from previous slide
. . . . .
}

int main(){
timer t1, t2;
t1.start();
fibiter(45);
t1.stop();
t2.start();
fib(45);
t2.stop();

cout << Time required by Iterative version is << t1.time() << sec << endl;
cout << Time required by Recursive version is << t2.time() << sec << endl;

return 0;
}

Tower of Hanoi
Needle A
. . . . . . . .
Needle C
Needle B
Needle C
. . . . . . . .
Needle B
Needle A

Tower of Hanoi
Needle A Needle B Needle C
1
Needle B Needle C Needle A
2
3
Needle A Needle B Needle C Needle A
Needle B Needle C
4
Needle A
Needle B Needle C
5
6
Needle A Needle B Needle C Needle A Needle B Needle C
7
Questions?

Page 69 Fall 2014 CS 361 - Advanced Data Structures and Algorithms
Due Sun Sep 14, 11:59pm
Written Assignment
Weiss, Chapter #2:
Assignment #2: Algorithm Analysis
Question # Page #
1 71
7 71
8.(a, b, & e) 72
12 73
25 74
26.(a-d) 75
Programming Assignment
Weiss, Chapter #2:
You program should read inarray.txt file that contains the array elements in
format identical to one shown in the question. Then, it outputs the majority
element to the standard output.

Question # Page #
26.e 75

Page 70 Fall 2014 CS 361 - Advanced Data Structures and Algorithms
Due Sun Sep 14, 11:59pm
Assignment #2: Algorithm Analysis
Submission Format:
Written Assignment
Create single PDF file with name: cs361_assignment_2
Have a cover page with your name and your email
Submit through Blackboard.
Programming Assignment
Make sure your program compiles and executes using g++ on Depts Linux
machines.
Creat a Readme.txt file that list how to compile and execute your program.
Include your name and your email.
Your main file should be named main.cpp.
You should have a Makefile.
Zip all files (.h, .c, Makefile, etc.) and name it: progam1.zip
Submit through Blackboard.

Lec 02 Intro Algorithms

Enviado por

Dados do documento

Título original

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

Lec 02 Intro Algorithms

Enviado por

Direitos autorais:

Formatos disponíveis

Advanced Data Structures and

Você também pode gostar