Relevant Mathematics

COSC 3101 Design and Analysis of Algorithms
Lecture 2. Relevant Mathematics:

–The time complexity of an algorithm
–Adding made easy
–Recurrence relations
COSC 3101 Winter, 2004

Question from last class: recurrence relations
•T(1) = k for some constant k
•T(n) = 4 T(n/2) + k’ n + k’’ for some
constants k’ and k’’
MULT(X,Y):
If |X| = |Y| = 1 then RETURN XY
Break X into a;b and Y into c;d
RETURN
MULT(a,c) 2n + (MULT(a,d) + MULT(b,c)) 2n/2 + MULT(b,d)

Question from last class: how does Mult know the length of X and Y (i.e. n)?
Answer: data type of X, Y, a, b, c, d must include length field.

The Time Complexity of an
Algorithm
Specifies how the running time
depends on the size of the input.

Purpose?

Purpose
• To estimate how long a program will run.
• To estimate the largest input that can reasonably be given to the

program.
• To compare the efficiency of different algorithms.
• To help focus on the parts of code that are executed the largest
number of times.
• To choose an algorithm for an application.

Time Complexity Is a Function
• Specifies how the running time depends on
the size of the input.
• A function mapping “size” of input 

“time” T(n) executed .

Definition of Time?

Definition of Time
• # of seconds (machine dependent).

• # lines of code executed.
• # of times a specific operation is performed (e.g.,
addition).
Which?

Definition of Time
• # of seconds (machine dependent).

• # lines of code executed.
• # of times a specific operation is performed (e.g.,
addition).
• These are all reasonable definitions of time, because

they are within a constant factor of each other.

Size of Input Instance?
83920
Suppose input to the algorithm is the integer 83920.

Size of Input Instance
5
83920 1’’
2’’
• Size of paper: n = 2 in2
• # of bits: n = 17 bits
• Which are reasonable?
# of digits: n = 5 digits
• Value: n = 83920

5
83920 1’’
2’’
• Size of paper: n = 2 in2 • Intuitive
• # of bits: n = 17 bits
• # of digits: n = 5 digits
• Value: n = 83920

5
83920 1’’
2’’
• # of bits: n = 17 bits • Formal
• # of digits: n = 5 digits
• Value: n = 83920

5
83920 1’’
2’’
• # of bits: n = 17 bits • Formal
• # of digits: n = 5 digits • Reasonable
• Value: n = 83920 # of bits =
3.32 * # of digits

5
83920 1’’
2’’
• Size of paper - n = 2 in2 • Intuitive
• # of bits - n = 17 bits • Formal
• # of digits - n = 5 digits • Reasonable
• Value - n = 83920 • Unreasonable
# of bits = log2(Value)
Value = 2#ofbits
Two Example Algorithms
• Sum N array entries:
A(1) + A(2) + A(3) + …
• Factor value N:
Input N=5917 & Output N=97*61.
Algorithm: N/2, N/3, N/4, …. ?
Time?

A(1) + A(2) + A(3) + … Time = N
• Factor value N:
Algorithm N/2, N/3, N/4, …. ?
Time = N

A(1) + A(2) + A(3) + … Time = N
• Factor value N:
Time = N
Is this reasonable?

A(1) + A(2) + A(3) + … Time = N
• Factor value N:
Time = N
No!
One is considered fast and the other slow!
A(1) + A(2) + A(3) + …
• Factor value N:

A(1) + A(2) + A(3) + … size n = N
• Factor value N:
size n = log N

A(1) + A(2) + A(3) + … size n = N
• Factor value N:
size n = log N
Time as function of input size?

A(1) + A(2) + A(3) + … size n = N
• Factor value N: Time = N = n
size n = log N
Time = N = 2n

A(1) + A(2) + A(3) + … size n = N
• Factor value N: Time = N = n
size n = log N
Time = N = 2n
Linear vs Exponential Time!

14,23,25,30,31,52,62,79,88,98

10
14,23,25,30,31,52,62,79,88,98
• # of elements: n = 10 elements

10
14,23,25,30,31,52,62,79,88,98
• # of elements: n = 10 elements
Is this reasonable?

10
14,23,25,30,31,52,62,79,88,98
• # of elements: n = 10 elements ~ Reasonable
If each element has size c

# of bits = c * # of elements

10
6, 3, 8, 1, 2, 10, 4, 9, 7, 5
• # of elements: n = 10 elements Reasonable
If each element is in [1..n]

each element has size log n
# of bits = n log n ≈ n

Time Complexity Is a Function
• Specifies how the running time depends on the size of the input.
• A function mapping:
– “size” of input  “time” T(n) executed .
• Or more precisely:
– # of bits n needed to represent the input  # of operations T(n) executed .

Which Input of size n?
There are 2n inputs of size n.
Which do we consider
for the time T(n)?

Which Input of size n?
Typical Input But what is typical?
Average Case For what distribution?
Worst Case • Provides an upper bound

for all possible inputs.
• Easiest to compute

Time Complexity of an Algorithm
The time complexity of an algorithm is
the largest time required on any input
of size n.
• O(n2): Prove that for every input of size n,
the algorithm takes no more than cn2 time.
• Ω(n2): Find one input of size n, for which
the algorithm takes at least this much time.
• θ (n2): Do both.
What is the height of tallest
person in the class?
Bigger than this? Smaller than this?
Need to look at Need to look at

only one person every person
Time Complexity of a Problem
The time complexity of a problem is
the time complexity of the fastest algorithm
that solves the problem.
• O(n2): Provide an algorithm that solves the problem in

no more than this time.
• Ω(n2): Prove that no algorithm can solve it faster.
• θ (n2): Do both.

Adding:
The Classic Techniques
Evaluating ∑i=1 f(i).
n

Arithmetic Sum
∑i=1..n i=1+2+3+...+n
= ?

1 + 2 + 3 + ... + n-1 + n = S
n + n-1 + n-2 + ... + 2 + 1 = S
(n+1) + (n+1) + (n+1) + ... + (n+1) + (n+1) = 2S
n (n+1) = 2S

1 + 2 + 3 + ... + n-1 + n = S
n + n-1 + n-2 + ... + 2 + 1 = S
(n+1) + (n+1) + (n+1) + ... + (n+1) + (n+1) = 2S
n (n+1) = 2S

1 + 2 + 3 + ... + n-1 + n = S
n + n-1 + n-2 + ... + 2 + 1 = S
(n+1) + (n+1) + (n+1) + ... + (n+1) + (n+1) = 2S
n (n+1) = 2S

1 + 2 + 3 + ... + n-1 + n = S
n + n-1 + n-2 + ... + 2 + 1 = S
(n+1) + (n+1) + (n+1) + ... + (n+1) + (n+1) = 2S
n (n+1) = 2S
n (n + 1)
S=
2
1 + 2 + 3 + ... + n-1 + n = S
n + n-1 + n-2 + ... + 2 + 1 = S
(n+1) + (n+1) + (n+1) + ... + (n+1) + (n+1) = 2S

Algebraic argument
n (n+1) = 2S
Let’s restate this

argument using a
geometric
representation

1 + 2 + 3 + ... + n-1 + n = S
= number of white dots
n + n-1 + n-2 + ... + 2 + 1 = S
(n+1) + (n+1) + (n+1) + ... + (n+1) + (n+1) = 2S
n (n+1) = 2S
1 2........n
1 + 2 + 3 + ... + n-1 + n = S
= number of white dot
n + n-1 + n-2 + ... + 2 + 1 = S
= number of yellow dot

(n+1) + (n+1) + (n+1) + ... + (n+1) + (n+1) = 2S
n (n+1) = 2S
n ....... 2 1 1 2........n
1 + 2 + 3 + ... + n-1 + n = S
n + n-1 + n-2 + ... + 2 + 1 = S

(n+1) + (n+1) + (n+1) + ... + (n+1) + (n+1) = 2S
n (n+1) = 2S
There are n(n+1) n

dots in the grid n
n+1 n+1 n+1 n+1 n+1

1 + 2 + 3 + ... + n-1 + n = S
n + n-1 + n-2 + ... + 2 + 1 = S

(n+1) + (n+1) + (n+1) + ... + (n+1) + (n+1) = 2S
n (n+1) = 2S
n (n + 1) n
S= n
2
n
n
n+1 n+1 n+1 n+1 n+1
COSC 3101
Note = Θ (# of terms · last term)) Winter, 2004
Arithmetic Sum
∑i=1..n i=1+2+3+...+n
= Θ (# of terms · last term)
True whenever terms increase slowly

Geometric Sum
∑i= 0..n 2i = 1 + 2 + 4 + 8 +. . . + 2n
= ?

Geometric Sum

Geometric Sum
∑i= 0..n 2i = 1 + 2 + 4 + 8 +. . . + 2n
= 2 · last term - 1

Geometric Sum
∑i= 0..n ri = r0 + r1 + r2 +. . . + rn
= ?

Geometric Sum
S = 1 + r + r + r +... + r
2 3 n
n +1
Sr = r + r + r +... + r + r
2 3 n
n +1
S − Sr = 1 −r
n +1
−1
r
S=
r −1
Geometric Sum
n +1
When r>1?
r −1
∑i= 0..n ri =
r −1

Geometric Sum
n +1 When r>1
r −1
∑i= 0..n ri = = θ(r )
n Biggest Term
r −1

Geometric Increasing
∑i= 0..n ri = r0 + r1 + r2 +. . . + rn
= Θ (biggest term)
True whenever terms increase quickly

Geometric Sum
n +1
When r<1?
1 − r
∑i= 0..n r =
i
1 −r

Geometric Sum
n +1
When r<1
1 − r
∑i= 0..n r =
i
= θ(1) Biggest Term
1 −r

Bounded Tail
∑i= 0..n ri = r0 + r1 + r2 +. . . + rn
= Θ (1)
True whenever terms decrease quickly

Harmonic Sum
∑i=1..n 1/i
= 1/1 + 1/2 + 1/3 + 1/4 + 1/5 + …+ 1/n
= ?

Sum of Shrinking Function
n
f(i) = 1
∑i=1..n f(i) = n
n
f(i) = ?
∑i=1..n f(i) = n1/2
∞
f(i) = 1/2i
∑i=1..n f(i) = 2
n
f(i) = 1/i
∑i=1..n f(i) = ?
Harmonic Sum
From this it follows that

1 n
1 n
1
log2 n  ≤ ∑ ≤ log2 n  + 1 ⇒ ∑ = θ (log2 n )
2 i =1 i i =1 i
NB: Error in Jeff’s notes, p.30, bottom:

Harmonic Sum
∑i=1..n 1/i
= 1/1 + 1/2 + 1/3 + 1/4 + 1/5 + …+ 1/n
= Θ (log(n))

Approximating Sum
by Integrating
∑i=1..n f(i) ≈ ∫x=1 .. n f(x) dx

The area under the curve
COSC 3101 approximates the sum Winter, 2004
Approximating Sums by Integrating:
Arithmetic Sums
n
n
1 c +1
∑
i =0
c
i ≅ ∫ x dx
c
=
c +1
n = θ (n c +1
) = θ (n ⋅ n c
)
0
= θ (number of terms × last term)

(True whenever terms increase slowly)

Approximating Sums by Integrating:
Geometric Sums
n
n
1
∑
i =0
b i
≅ ∫ b dx x
=
ln b
(b n
− 1) = θ (b n
)
0
= θ (last term)
(True whenever terms increase rapidly)

Approximating Sum
by Integrating
Harmonic Sum

Approximating Sum
by Integrating
Problem: Integrating may be hard too.

Adding Made Easy
We will now classify (most) functions f(i) into four classes:
–Geometric Like
–Arithmetic Like
–Harmonic
–Bounded Tail
For each class, we will give an easy rule for approximating

its sum θ( ∑i=1..n f(i) )
Classifying Animals
Vertebrates
Birds
Fish
Mammals Giraffe
Reptiles
Dogs

Classifying Functions
Functions
f(n) = 8·2n / n100 + n3
2θ(n) ⇒ ∑i=1..n f(i) = θ(f(n))
Poly.
Exp.
nθ(1) 2θ(n) θ(2n / n100 )

θ(2n / n100 ) θ(2 )
n ⇒ ∑ i=1..n f(i) = θ(2 n
/ n 100
8·2n / n100 + n3
Irrelevant
20.5n < 8· 2n / n100 + n3 < 2n
COSC 3101 Significant Less significant Winter, 2004
Adding Made Easy
Four Classes of Functions
3
4

Adding Made Easy

Geometric Like:
f(n) ≥ 2Ω(n) ⇒ ∑i=1..n f(i) = θ(f(n))
If the terms f(i) grow sufficiently quickly, then the

sum will be dominated by the largest term.
Classic example:
∑i=1..n 2i = 2n+1 -1 ≈ 2 f(n)

Geometric Like:
f(n) ≥ 2Ω(n) ⇒ ∑i=1..n f(i) = θ(f(n))
If the terms f(i) grow sufficiently quickly, then the

sum will be dominated by the largest term.
For which functions f(i) is this true?

How fast and how slow can it grow?

Geometric Like:
When f(n) = rn = 2θ(n)
∑i=1..n r i = 1 + r + r + r +. . . + r
2 3 n
n +1
r −1
= = θ(r
n
) Last Term
r −1
= θ(f(n)) when r>1.
Upper Extreme: ∑i=1..n (1000)i ≈ 1.001(1000)n

= 1.001 f(n)
Lower Extreme: ∑i=1..n (1.0001)i ≈ 10,000 (1.0001)n

COSC 3101
= 10,000 f(n) Winter, 2004
Geometric Like:
f(n) ≥ 2
Ω(n)
⇒ ∑i=1..n f(i) = θ(f(n))

= 1.001 f(n)
Because the constant is sooo big,

the statement is just barely true.

COSC 3101
= 10,000 f(n) Winter, 2004
Geometric Like:
f(n) ≥ 2
Ω(n)
⇒ ∑i=1..n f(i) = θ(f(n))

= 1.001 f(n)
Even bigger?

COSC 3101
= 10,000 f(n) Winter, 2004
Geometric Like:
f(n) ≥ 2
Ω(n)
⇒ ∑i=1..n f(i) = θ(f(n))
2 i
2 n
No Upper Extreme: ∑i=1..n 2 ≈ 2 = 1f(n)

2 2
Even bigger!

COSC 3101
= 10,000 f(n) Winter, 2004
Geometric Like:
f(n) ≥ 2
Ω(n)
⇒ ∑i=1..n f(i) = θ(f(n))
2 i
2 n
No Upper Extreme: ∑i=1..n 2 ≈ 2 = 1f(n)

2 2
Functions in between?

COSC 3101
= 10,000 f(n) Winter, 2004
Geometric Like:
f(n) ≥ 2Ω(n) ⇒ ∑i=1..n f(i) = θ(f(n))
n n
2 2
e.g. f (n ) = 8 100 + n = θ (f (n )) = θ ( 100 )
3
n n
(The strongest function determines the class.)

Geometric Like:
f(n) ≥ 2
Ω(n)
⇒ ∑i=1..n f(i) = θ(f(n))
Easy Hard
f(n) = 8· 2n f(n) ≤ ∑i=1..n f(i) ≤ c·f(n)
n100 f(i) ≤ ?·f(n)
f(i)≤ ?·f(i
f(i) .51·f(i ) )
+1+1
8·2(i+1)
i100
f(i+1) = (i+1)100 8·2i = 2
f(i) (i+1)100
i100
1
= 2 (1+1/ )100 ≥ 1.9 ≥ 1/.51
i

2n n
Example: f (n ) = 8 100 Claim: ∑ f (i ) ∈θ (f (n )) Proof:
n i =1
n n
1) ∑f (i ) ≥ f (n ) → ∑f (i ) ∈ Ω(f (n ))
i =1 i =1
n
2) Seek c > 0,n0 >0 such that ∑ f (i ) ≤ cf (n ) ∀n ≥ n0
i =1
f (i )
Suppose that (*) ∃n0 > 0 < 1∀i ≥ n0
f (i + 1)
Then f (i ) < αf (i + 1) for some α < 1, ∀i ≥ n 0 → f (i ) < α kf (i + k ) → f (i ) ≤ α n −if (n )
n n
1 n
Thus ∑f (i ) ≤ ∑α
i =1 i =1
n −i
f (n ) <
1 −α
f (n ) → ∑f (i ) ∈ O (f (n ))
i =1
f (i ) 2i / i 100 (i + 1)100 (1 + 1/ i )100

It remains to prove (*) = = =
f (i + 1) 2i +1 /(i + 1)100 2i 100 2
f (i ) (1 + 1/ i )100 1
Thus < 1 iff < 1 ↔ 1 + 1/ i < 2.01 ↔ i > .01 = 143.8
f (i + 1) 2 2 −1
n n
Thus (*) holds for n0 = 144 → ∑f (i ) ∈O (f (n )) → ∑f (i ) ∈θ (f (n ))
i =1 i =1

Geometric Like:
f(n) ≥ 2Ω(n) ⇒ ∑i=1..n f(i) = θ(f(n))
In general, if f(n) = c ban nd loge (n)
c∈?
>0
a∈?
b ∈> ?0
d ∈> ?1
e ∈ ?(-∞,∞)
f(n) = Ω,(-∞,∞)
θ?
Then ∑i=1..n f(i) = Ω
θ(f(n)).
≥ 2 (n)

Geometric Like:
f(n) ≥ 2
Ω(n)
⇒ ∑i=1..n f(i) = θ(f(n))
Examples:

Geometric Like:
f(n) ≥ 2
Ω(n)
⇒ ∑i=1..n f(i) = θ(f(n))
Do All functions in 2Ω(n) have this property?

Maybe not.

Geometric Like:
f(n) ≥ 2
Ω(n)
⇒ ∑i=1..n f(i) = θ(f(n))
Functions that oscillate with exponentially

increasing amplitude do not have this property.

Geometric Like:
f(n) ≥ 2
Ω(n)
⇒ ∑i=1..n f(i) = θ(f(n))
Functions expressed with +, -, ⋅ , ÷ , exp, log

do not oscillate continually.
They are well behaved for sufficiently large n.
These do have this property.

Adding Made Easy
Done

Adding Made Easy

Arithmetic Like:
f(n) = n ⇒ ∑i=1..n f(i) = θ(n·f(n))
θ(1)-1
If the terms f(i) are increasing or decreasing

relatively slowly, then the sum is roughly
the number of terms, n, times the final
value.

Arithmetic Like:
f(n) = n
θ(1)-1
⇒ ∑i=1..n f(i) = θ(n·f(n))

value.
Simple example:
∑i=1..n 1 = n · 1

Arithmetic Like:
f(n) = n ⇒ ∑i=1..n f(i) = θ(n·f(n))
θ(1)-1

value.
Is the statement true for this function?
∑i=1..n i=1+2+3+...+n

Arithmetic Like:
f(n) = n
θ(1)-1
⇒ ∑i=1..n f(i) = θ(n·f(n))
∑i=1..n i =
1 + . . . + n/2 + . . . + n
Half the terms are roughly the same

(within a multiplicative constant)

Arithmetic Like:
f(n) = n
θ(1)-1
⇒ ∑i=1..n f(i) = θ(n·f(n))
∑i=1..n i =
1 + . . . + n/2 + . . . + n
But half the terms are roughly the same

and the sum is roughly the number
terms, n, times this value
∑i=1..n i = θ(n · n)
Arithmetic Like:
f(n) = nθ(1)-1
⇒ ∑i=1..n f(i) = θ(n·f(n))
Is the statement true for this function?
∑i=1..n i2 = 1 2 + 2 2 + 3 2 + . . . + n 2

Arithmetic Like:
f(n) = n ⇒ ∑i=1..n f(i) = θ(n·f(n))
θ(1)-1
∑i=1..n i =
12 + . . . + (n/2)2 + . . . + n2
1
/4 n 2
Again half the terms
are roughly the same.

Arithmetic Like:
f(n) = n ⇒ ∑i=1..n f(i) = θ(n·f(n))
θ(1)-1
∑i=1..n i =
12 + . . . + (n/2)2 + . . . + n2
1
/4 n 2
Again half the terms
are roughly the same.
∑i=1..n i2 = θ(n · n2)
Arithmetic Like:
f(n) = n ⇒ ∑i=1..n f(i) = θ(n·f(n))
θ(1)-1
Consider f(i) = nθ(1)

(i.e. f(i) increasing)
 ∑i=1..n f(i)
≈ area under curve


Arithmetic Like:
f(n) = n ⇒ ∑i=1..n f(i) = θ(n·f(n))
θ(1)-1
area of small square

≤ ∑i=1..n f(i)
≤ area of big square

Arithmetic Like:
f(n) = n ⇒ ∑i=1..n f(i) = θ(n·f(n))
θ(1)-1
/2 · f(n/2)
n
= area of small square

≤ ∑i=1..n f(i)
= n · f(n)

Arithmetic Like:
f(n) = n ⇒ ∑i=1..n f(i) = θ(n·f(n))
θ(1)-1
θ(n · f(n))
= n/2 · f(n/2)
≤ ∑i=1..n f(i)
= n · f(n)

Arithmetic Like:
f(n) = n ⇒ ∑i=1..n f(i) = θ(n·f(n))
θ(1)-1
θ(n · f(n))
?
= /2 · f( /2)
n n

≤ ∑i=1..n f(i)
= n · f(n)

Arithmetic Like:
f(n) = nθ(1)-1
⇒ ∑i=1..n f(i) = θ(n·f(n))
f(n) = n2
∑i=1..n i2 = 1 2 + 2 2 + 3 2 + . . . + n 2
=?
The key property is

f(n/2) = θ(f(n))

Arithmetic Like:
f(n) = nθ(1)-1
⇒ ∑i=1..n f(i) = θ(n·f(n))
f(n) = n2
∑i=1..n i2 = 1 2 + 2 2 + 3 2 + . . . + n 2
⇒ = θ(n·f(n)) = θ(n · n2)
The key property is

f(n/2) = (n/2)2 = 1/4 n2 = θ(n2) = θ(f(n))

Arithmetic Like:
f(n) = n ⇒ ∑i=1..n f(i) = θ(n·f(n))
θ(1)-1
f(n) = nθ(1)
∑i=1..n f(i) = ?

Arithmetic Like:
f(n) = n θ(1)-1
⇒ ∑i=1..n f(i) = θ(n·f(n))
f(n) = nθ(1)
∑i=1..n ir = 1 r + 2 r + 3 r + . . . + n r
The key property is

f(n/2) = θ(f(n))

Arithmetic Like:
f(n) = n θ(1)-1
⇒ ∑i=1..n f(i) = θ(n·f(n))
f(n) = nθ(1)
∑i=1..n ir = 1 r + 2 r + 3 r + . . . + n r
⇒ = θ(n·f(n)) = θ(n · nr)
The key property is

f(n/2) = (n/2)r = 1/2r nr = θ(nr) = θ(f(n))

Arithmetic Like:
f(n) = n ⇒ ∑i=1..n f(i) = θ(n·f(n))
θ(1)-1
f(n) = nθ(1)
∑i=1..n 2i = 21 + 22 + 23 + . . . + 2n
The key property is

f(n/2) = θ(f(n))

Arithmetic Like:
f(n) = n ⇒ ∑i=1..n f(i) = θ(n·f(n))
θ(1)-1
f(n) = nθ(1)
∑i=1..n 2i = 21 + 22 + 23 + . . . + 2n
⇒ = θ(n·f(n)) = θ(n · 2n)
The key property is

f(n/2) = 2(n/2) = θ(2n) = θ(f(n))

Arithmetic Like:
f(n) = n
θ(1)-1
⇒ ∑i=1..n f(i) = θ(n·f(n))
n
1 1
Upper Extreme: ∑i
i =1
1000
≅
1001
n 1001
=
1001
n ⋅ f (n )
All functions
in between.

Adding Made Easy
Half done

n
f(i) = 1
∑i=1..n f(i) = n
n
f(i) = ?1/i1/2
∑i=1..n f(i) = n1/2
n
f(i) = 1/i
∑i=1..n f(i) = log n
∞
f(i) = 1/2i
∑i=1..n f(i) = 2
Arithmetic Like:
f(n) = n ⇒ ∑i=1..n f(i) = θ(n·f(n))
θ(1)-1
If most of the terms f(i) have roughly the

same value, then the sum is roughly the
number of terms, n, times this value.
Does the statement hold

for functions f(i) that shrink?

Arithmetic Like:
f(n) = n
θ(1)-1
⇒ ∑i=1..n f(i) = θ(n·f(n))
If most of the terms f(i) have roughly the

same value, then the sum is roughly the
number of terms, n, times this value.

for the Harmonic Sum?
∑i=1..n 1/i = 1/1 + 1/2 + 1/3 + 1/4 + 1/5 + … + 1/n

Arithmetic Like:
f(n) = n θ(1)-1
⇒ ∑i=1..n f(i) = θ(n·f(n))

∑i=1..n 1/i = 1/1 + 1/2 + 1/3 + 1/4 + 1/5 + … + 1/n
∑i=1..n f(i) = ?
θ(n · f(n))

Arithmetic Like:
f(n) = nθ(1)-1
⇒ ∑i=1..n f(i) = θ(n·f(n))

∑i=1..n 1/i = 1/1 + 1/2 + 1/3 + 1/4 + 1/5 + … + 1/n
∑i=1..n f(i) = ∑i=1..n 1/i = ?

θ(n · f(n))

Arithmetic Like:
f(n) = nθ(1)-1
⇒ ∑i=1..n f(i) = θ(n·f(n))

∑i=1..n 1/i = 1/1 + 1/2 + 1/3 + 1/4 + 1/5 + … + 1/n
∑i=1..n f(i) = ∑i=1..n 1/i = θ(log n)
θ(n · f(n))
1 n
1 n
1
⇒ log2 n  ≤ ∑ ≤ log2 n  + 1 ⇒ ∑ = θ (log2 n )
2 i =1 i i =1 i
Arithmetic Like:
f(n) = nθ(1)-1
⇒ ∑i=1..n f(i) = θ(n·f(n))

∑i=1..n 1/i = 1/1 + 1/2 + 1/3 + 1/4 + 1/5 + … + 1/n
∑i=1..n f(i) = ∑i=1..n 1/i = θ(log n)

? = θ(n · f(n))

Arithmetic Like:
f(n) = nθ(1)-1
⇒ ∑i=1..n f(i) = θ(n·f(n))

∑i=1..n 1/i = 1/1 + 1/2 + 1/3 + 1/4 + 1/5 + … + 1/n
∑i=1..n f(i) = ∑i=1..n 1/i = θ(log n)

θ(1) = θ(n · 1/n) = θ(n · f(n))

Arithmetic Like:
f(n) = nθ(1)-1
⇒ ∑i=1..n f(i) = θ(n·f(n))

∑i=1..n 1/i = 1/1 + 1/2 + 1/3 + 1/4 + 1/5 + … + 1/n
∑i=1..n f(i) = ∑i=1..n 1/i = θ(log n)

≠
θ(1) = θ(n · 1/n) = θ(n · f(n))
No the statement does not hold!

Adding Made Easy
not included

Arithmetic Like:
f(n) = n ⇒ ∑i=1..n f(i) = θ(n·f(n))
θ(1)-1

for the almost Harmonic Sum?
∑i=1..n 1/i0.9999
Shrinks slightly slower

than harmonic.

Arithmetic Like:
f(n) = n
θ(1)-1
⇒ ∑i=1..n f(i) = θ(n·f(n))

∑i=1..n 1/i0.9999
∑i=1..n f(i) = ∑i=1..n 1/i0.9999 = ?

= θ(n · f(n))

Arithmetic Like:
f(n) = nθ(1)-1
⇒ ∑i=1..n f(i) = θ(n·f(n))

∑i=1..n 1/i0.9999
∑i=1..n f(i) = ∑i=1..n 1/i0.9999 = θ(n0.0001 )

= θ(n · f(n))

Arithmetic Like:
f(n) = nθ(1)-1
⇒ ∑i=1..n f(i) = θ(n·f(n))

∑i=1..n 1/i0.9999
∑i=1..n f(i) = ∑i=1..n 1/i0.9999 = θ(n0.0001 )

? = θ(n · f(n))

Arithmetic Like:
f(n) = nθ(1)-1
⇒ ∑i=1..n f(i) = θ(n·f(n))

∑i=1..n 1/i0.9999
∑i=1..n f(i) = ∑i=1..n 1/i0.9999 = θ(n0.0001 )

θ(n0.0001 ) = θ(n · 1/n0.9999 ) = θ(n · f(n))

Arithmetic Like:
f(n) = n
θ(1)-1
⇒ ∑i=1..n f(i) = θ(n·f(n))

∑i=1..n 1/i0.9999
∑i=1..n f(i) = ∑i=1..n 1/i0.9999 = θ(n0.0001 )

=
θ(n0.0001 ) = θ(n · 1/n0.9999 ) = θ(n · f(n))
The statement does hold!

Arithmetic Like:
f(n) = n
θ(1)-1
⇒ ∑i=1..n f(i) = θ(n·f(n))
Intermediate Case: ∑i=1..n 1 = n · 1

= n · f(n)
All functions
in between.
Lower Extreme: ∑i=1..n 1/i0.9999 = θ(n0.0001 )

= θ(n · f(n))
Arithmetic Like:
f(n) = n
θ(1)-1
⇒ ∑i=1..n f(i) = θ(n·f(n))
Upper Extreme: ∑i=1..n i1000 = 1/1001 n1001

= 1/1001 n · f(n)
Intermediate Case: ∑i=1..n 1 = n · 1

= n · f(n)
Lower Extreme: ∑i=1..n 1/i0.9999 = θ(n0.0001 )

= θ(n · f(n))
Arithmetic Like
Conclusion
If f(n) = c ban nd loge (n)

c∈?
a∈?
>0
b ∈= ?0
or b d=∈ 1?
e∈ ?
> -1
f(n) = Ω, (-∞,∞)
θ?
then ∑i=1..n f(i) =-1θ(n
+θ(1) · f(n)).
n
COSC 3101
(For +, -, ⋅ , ÷ , exp, log functions f(n))
Winter, 2004
Adding Made Easy
Done

Adding Made Easy
Harmonic

Harmonic Sum
1 n
1 n
1
⇒ log2 n  ≤ ∑ ≤ log2 n  + 1 ⇒ ∑ = θ (log2 n )
2 i =1 i i =1 i
∑i=1..n 1/i
= 1/1 + 1/2 + 1/3 + 1/4 + 1/5 + …+ 1/n
= Θ (log(n))
Adding Made Easy

Bounded Tail:
f(n) ≤ n ⇒ ∑i=1..n f(i) = θ(1)
-1-Ω(1)
If the terms f(i) decrease towards zero

sufficiently quickly,
then the sum will be a constant.
The classic example

∑i=0..n 1/2i = 1 + 1/2 + 1/4 + 1/8 + … = 2.

Bounded Tail:
f(n) ≤ n ⇒ ∑i=1..n f(i) = θ(1)
-1-Ω(1)
Upper Extreme: ∑i=1..n /i

1 1.0001
= θ(1)

Bounded Tail:
f(n) ≤ n ⇒ ∑i=1..n f(i) = θ(1)
-1-Ω(1)
Upper Extreme: ∑i=1..n /i

1 1.0001
= θ(1)
All functions
in between.
1
2i
No Lower Extreme: ∑i=1..n 22 = θ(1).
Bounded Tail
Conclusion
Case 1: Case 2:
f (n ) = cb n log n
an d e
f (n ) = cn d loge n
c >0
c >0
0 <b <1 d < −1
a >0 e ∈ ( −∞, +∞)
d ∈ ( −∞, +∞)
e ∈ (−∞, +∞)
Adding Made Easy
Done
Adding Made Easy
Missing Functions
n
n
n l o ng
logn
/n
1
/nlogn
Adding Made Easy
• Geometric Like: If f(n) ≥ 2Ω(n), then ∑i=1..n f(i) = θ(f(n)).
• Arithmetic Like: If f(n) = nθ(1)-1, then ∑i=1..n f(i) = θ(n · f(n)).
• Harmonic: If f(n) = 1/n , then ∑i=1..n f(i) = logen + θ(1).
• Bounded Tail: If f(n) ≤ n-1-Ω(1), then ∑i=1..n f(i) = θ(1).
(For +, -, ⋅ , ÷ , exp, log functions f(n))
This may seem confusing, but it is really not.

It should help you compute most sums easily.
Recurrence Relations
T(1) = 1
T(n) = a T(n/b) + f(n)

≈ Time of Recursive Program
procedure Eg(int n) Recurrence relations
if(n≤ 1) then arise from the timing
put “Hi” of recursive programs.
else
loop i=1..f(n) Let T(n) be the # of “Hi”s
put “Hi” on an input of “size” n.
loop i=1..a
Eg(n/b)
procedure Eg(int n)
if(n≤ 1) then Given size 1,
put “Hi” the program outputs
else T(1)=1 Hi’s.
loop i=1..f(n)
put “Hi”
loop i=1..a
Eg(n/b)

procedure Eg(int n)
if(n≤ 1) then Given size n,
put “Hi” the stackframe outputs
else f(n) Hi’s.
loop i=1..f(n)
put “Hi”
loop i=1..a
Eg(n/b)

procedure Eg(int n)
if(n≤ 1) then
put “Hi”
else
loop i=1..f(n)
put “Hi” Recursing
loop i=1..a on a instances of size n/b
Eg(n/b) generates T(n/b) “Hi”s.

procedure Eg(int n)
if(n≤ 1) then
put “Hi”
else
loop i=1..f(n)
put “Hi” Recursing
loop i=1..a a times
Eg(n/b) generates a·T(n/b) “Hi”s.

procedure Eg(int n)
if(n≤ 1) then For a total of
put “Hi” T(1) = 1
else T(n) = a·T(n/b) + f(n)
loop i=1..f(n)
put “Hi” “Hi”s.
loop i=1..a
Eg(n/b)

Solving Technique 1
Guess and Verify
•Recurrence Relation:
T(1) = 1 & T(n) = 4T(n/2) + n
•Guess: G(n) = 2n2 – n
•Verify: Left Hand Side Right Hand Side
T(1) = 2(1)2 – 1 1
T(n) 4T(n/2) + n
= 2n2 – n = 4 [2(n/2)2 – (n/2)] + n
= 2n2 – n

Solving Technique 2
Guess Form and Calculate Coefficients
T(1) = 1 & T(n) = 4T(n/2) + n
•Guess: G(n) = an2 + bn + c
T(1) = a+b+c 1
T(n) 4T(n/2) + n
= an2 +bn+c = 4 [a (n/2)2 + b (n/2) +c] + n
= an2 +(2b+1)n + 4c

Solving Technique 2
T(1) = 1 & T(n) = 4T(n/2) + n
•Guess: G(n) = an2 + bn + 0
T(1) = a+b+c 1
T(n) 4T(n/2) + n
= an2 +bn+c = 4 [a (n/2)2 + b (n/2) +c] + n
c=4c = an2 +(2b+1)n + 4c
c=0

Solving Technique 2
T(1) = 1 & T(n) = 4T(n/2) + n
•Guess: G(n) = an2 - 1n + 0
T(1) = a+b+c 1
T(n) 4T(n/2) + n
= an2 +bn+c = 4 [a (n/2)2 + b (n/2) +c] + n
b=2b+1 = an2 +(2b+1)n + 4c
b=-1

Solving Technique 2
T(1) = 1 & T(n) = 4T(n/2) + n
•Guess: G(n) = an2 - 1n + 0
T(1) = a+b+c 1
T(n) 4T(n/2) + n
= an2 +bn+c = 4 [a (n/2)2 + b (n/2) +c] + n
= an2 +(2b+1)n + 4c
a=a

Solving Technique 2
T(1) = 1 & T(n) = 4T(n/2) + n
•Guess: G(n) = 2n2 - 1n + 0
T(1) = a+b+c 1
a+b+c=1
a-1+0=1 T(n) 4T(n/2) + n
a=2
= an2 +bn+c = 4 [a (n/2)2 + b (n/2) +c] + n
= an2 +(2b+1)n + 4c

Solving Technique 3
Approximate Form and
Calculate Exponent
T(1) = 1 & T(n) = aT(n/b) + f(n)
which is bigger?
Guess

Solving Technique 3
Calculate Exponent
T(1) = 1 & T(n) = aT(n/b) + f(n)
•Guess: aT(n/b) << f(n)
•Simplify: T(n) ≈ f(n)
In this case, the answer is easy.

T(n) = Θ (f(n))

Solving Technique 3
Calculate Exponent
T(1) = 1 & T(n) = aT(n/b) + f(n)
•Guess: aT(n/b) >> f(n)
•Simplify: T(n) ≈ aT(n/b)
In this case, the answer is harder.

Solving Technique 3
Calculate Exponent
T(1) = 1 & T(n) = aT(n/b)
( / )
loga
•Guess: G(n) = cn = cn
α logb

T(n) aT(n/b)
= cnα = a [c (n/b) α ]
= c a b−α nα
1 = a b-α
bα = a
α log b = log a
α = loga /logb
Solving Technique 3
Calculate Exponent
T(1) = 1 & T(n) = 4T(n/2)
( / )
loga
( / ) 2log4
•Guess: G(n) = cn = cn
α = cn logb
= cn log2

T(n) aT(n/b)
= cnα = c [a (n/b) α ]
= c a b−α nα
1 = a b-α
bα = a
α log b = log a
α = loga /logb
Solving Technique 3
Calculate Exponent
T(1) = 1 & T(n) = aT(n/b) + f(n)
If bigger then If bigger then

( loga
/logb )
T(n) = Θ (n ) T(n) = Θ (f(n))
And if aT(n/b) ≈ f(n)

what is T(n) then?
Technique 4: Decorate The Tree
•T(1) = 1
T(1) = 1
• T(n) = a T(n/b) + f(n)
T(n) = f(n)
a
T(n/b) T(n/b) T(n/b) T(n/b)

T(n) = f(n)
a
T(n/b) T(n/b) T(n/b) T(n/b)

T(n) = f(n)
a
f(n/b)
a
T(n/b) T(n/b) T(n/b)
T(n/b2)T(n/b2)T(n/ b2T(n/
) b2)

T(n) = f(n)
a
f(n/b) f(n/b) f(n/b) f(n/b)

a a a a
) b2) T(n/b2)T(n/b2)T(n/ b2T(n/
) b 2) T(n/b2)T(n/b2)T(n/ b2T(n/
) b2) T(n/b2)T(n/b2)T(n/ b2T(n/
)

T(n) = f(n)
a
f(n/b) f(n/b) f(n/b) f(n/b)

a a a a
) b2) T(n/b2)T(n/b2)T(n/ b2T(n/
) b 2) T(n/b2)T(n/b2)T(n/ b2T(n/
) b2) T(n/b2)T(n/b2)T(n/ b2T(n/
)
11111111111111111111111111111111 . . . . . . 111111111111111111111111111111111
Evaluating: T(n) = aT(n/b)+f(n)
Level
0
1
2

Level Instance
size
0
1
2

Level Instance
size
0 n
1
2

Level Instance
size
0 n
1 n/b
2

Level Instance
size
0 n
1 n/b
2 n/b2

Level Instance
size
0 n
1 n/b
2 n/b2
i n/bi
h n/bh

Level Instance
size
0 n
1 n/b
2 n/b2
i n/bi
h n/bh

Level Instance
size
0 n
1 n/b
2 n/b2
i n/bi
h n/bh = 1
base case
Level Instance
size
0 n
1 n/b
2 n/b2
i n/bi
h n/bh = 1

Level Instance
size
0 n
1 n/b
2 n/b2
i n/bi
h = logn /logb n/bh = 1

bh = n
h log b = log n
COSC 3101 h = logn /logb Winter, 2004
Level Instance Work
size in stack
frame
0 n
1 n/b
2 n/b2
i n/bi
h = logn /logb 1

Level Instance Work
size in stack
frame
0 n f(n)
1 n/b f(n/b)
2 n/b2 f(n/b2)
i n/bi f(n/bi)
h = logn /logb 1 T(1)

Level Instance Work # stack
size in stack frames
frame
0 n f(n)
1 n/b f(n/b)
2 n/b2 f(n/b2)
i n/bi f(n/bi)
h = logn /logb n/bh T(1)

frame
0 n f(n) 1
1 n/b f(n/b) a
2 n/b2 f(n/b2) a2
i n/bi f(n/bi) ai
h = logn /logb n/bh T(1) ah

frame
0 n f(n) 1
1 n/b f(n/b) a
2 n/b2 f(n/b2) a2
i n/bi f(n/bi) ai

frame
0 n f(n) 1
1 n/b f(n/b) a
2 n/b2 f(n/b2) a2
i n/bi f(n/bi) ai

logn
/logb
a =a
h
COSC 3101
loga
/logb Winter, 2004
Level Instance Work # stack Work in Level
frame
0 n f(n) 1
1 n/b f(n/b) a
2 n/b2 f(n/b2) a2
i n/bi f(n/bi) ai
h = logn /logb n/bh T(1)

n /logb
loga

frame
0 n f(n) 1 1 · f(n)
1 n/b f(n/b) a a · f(n/b)
2 n/b2 f(n/b2) a2 a2 · f(n/b2)
i n/bi f(n/bi) ai ai · f(n/bi)
h = logn /logb n/bh T(1)     

loga /
n /logb
loga
n logb · T(1)
Total Work T(n) = ∑i=0..h ai⋅ f(n/bi)

= ∑i=0..h ai⋅ f(n/bi)
If a Geometric Sum
∑i= 0..n xi = θ(max(first term, last term))

frame
0 n f(n) 1 1 · f(n)
1 n/b f(n/b) a a · f(n/b)
2 n/b2 f(n/b2) a2 a2 · f(n/b2)
i n/bi f(n/bi) ai ai · f(n/bi)
h = logn /logb n/bh T(1)     

loga /
n /logb
loga
n logb · T(1)
Dominated by Top Level or Base Cases

Master Theorem: Intuition
Suppose T (n ) = aT (n / b ) + f (n ), a ≥ 1, b > 1
Work at top level = f (n )

Work at bottom level = number of base cases = n logb a = n log a /log b
Running time = max(work at top, work at bottom) = max(f (n ), n logb a )
If they are equal, then all levels are important:
Running time = sum of work over all levels = n logb a log n

Theorem 4.1 (Master Theorem)
Suppose T (n ) = aT (n / b ) + f (n ), a ≥ 1, b > 1
1. IF ∃ε > 0 such that f (n ) ∈ O (n logb a −ε )

Dominated by base cases
THEN T (n ) ∈ θ (n logb a
)
2. IF f (n ) ∈ θ (n logb a ) Work at each level is comparable:

THEN T (n ) ∈ θ (n logb a log n ) Sum work over all levels
3. IF ∃ε > 0 such that f (n ) ∈ Ω(n logb a +ε ) Dominated by top level work
AND
∃c < 1,n0 >0 such that af (n / b ) ≤ cf (n ) ∀n ≥ n0
THEN T (n ) ∈ θ (f (n ))
Additional regularity condition

Example: T (n ) = 4T (n /2) + n 3 /log 5 n
a = 4  log a
 n b =n
2
b = 2
f (n ) = n 3 /log5 n
Thus f (n ) ∈ Ω(n logb a +ε ) (Case 3: dominated by top level)
Additional regularity condition:

∃c < 1, n0 >0 such that af (n / b ) ≤ cf (n ) ∀n ≥ n 0
Thus we require that 4(n/2) 3 /log 5(n /2) ≤ cn 3 /log 5 n

5
1  logn   log n 
↔c ≥   ↔   < 2 = 1.15
.2
2  log n /2   logn /2 
↔ n > 21.15 / 0.15 = 203.2.
5 5
1  log256  1  8
e . g . Let n0 = 256 → c ≥   =   = 0.975
2  log128  2 7 
Thus regularity condition holds for n0 = 256, c = 0.98
Thus T (n ) = θ (f (n )) = θ (n 3 /log5 n )

Example 2: T (n ) = 4T (n /2) + 2n
a = 4  log a
 n b =n
2
b = 2
f (n ) = 2n
Thus f (n ) ∈ Ω(n logb a +ε ) (Case 3: dominated by top level)
Additional regularity condition:

∃c < 1, n0 >0 such that af (n / b ) ≤ cf (n ) ∀n ≥ n 0
Thus we require that 4 ⋅ 2n/2 ≤ c 2n

↔ c ≥ 4 ⋅ 2−n / 2
1
Let n0 = 6 → c ≥
2
→ regularity condition holds for n0 = 6, c = 0.5
Thus T (n ) = θ (f (n )) = θ (2n )

Example 2: T (n ) = 4T (n /2) + n log5 n
a = 4
 n logb a = n 2
b = 2
f (n ) = n log5 n
Thus f (n ) ∈ O (n logb a −ε ) (Case 1: dominated by base cases)
Thus T(n)=θ (n logb a )

Example 2: T (n ) = 4T (n /2) + n 2
a = 4
 n logb a = n 2
b = 2
f (n ) = n 2
Thus f (n ) ∈ θ (n logb a ) (Case 2: all levels significant)
Thus T (n ) = θ (n logb a logn )

Evaluating: T2(n) = 4 T2(n/2+n1/2 )+ n3
Sufficiently close to:

T(n) = 4T(n/2)+ n3 = θ(n3).
T2(n) = ?θ(n3).

Evaluating: T(n) = aT(n-b)+f(n)
Work
Instance # stack Work in
Level in stack
size frames Level
frame
0 n f(n) 1 1 · f(n)
1 n-b f(n-b) a a · f(n-b)
2 n-2b f(n-2b) a2 a2 · f(n-2b)
i n-ib f(n-ib) ai ai · f(n-ib)
hh == ?/b
n
/b n
/b
n
n-hb T(0) a a · T(0)
|base case| = 0 = n-hb
Likely dominated by base cases Exponential
h = /b
n
Evaluating: T(n) = 1T(n-b)+f(n)
Work
Instance # stack Work in
Level in stack
size frames Level
frame
0 n f(n) 1 f(n)
1 n-b f(n-b) 1 f(n-b)
2 n-2b f(n-2b) 1 f(n-2b)
i n-ib f(n-ib) 1 f(n-ib)
hh ==n?/b n-hb T(0) 1 f(0)
Total Work T(n) = ∑i=0..h f(b·i) = θ(f(n)) or θ(n·f(n))

End

Relevant Mathematics

Enviado por

Dados do documento

Descrição original:

Título original

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

Relevant Mathematics

Enviado por

Direitos autorais:

Formatos disponíveis

COSC 3101 Design and Analysis of Algorithms

Lecture 2. Relevant Mathematics:

–Adding made easy

COSC 3101 Winter, 2004

MULT(a,c) 2n + (MULT(a,d) + MULT(b,c)) 2n/2 + MULT(b,d)

Answer: data type of X, Y, a, b, c, d must include length field.

COSC 3101 Winter, 2004

COSC 3101 Winter, 2004

COSC 3101 Winter, 2004

• To estimate the largest input that can reasonably be given to the

• To compare the efficiency of different algorithms.

• To choose an algorithm for an application.

COSC 3101 Winter, 2004

• A function mapping “size” of input 

COSC 3101 Winter, 2004

COSC 3101 Winter, 2004

• # of seconds (machine dependent).

COSC 3101 Winter, 2004

• # of seconds (machine dependent).

• These are all reasonable definitions of time, because

COSC 3101 Winter, 2004

Suppose input to the algorithm is the integer 83920.

COSC 3101 Winter, 2004

COSC 3101 Winter, 2004

COSC 3101 Winter, 2004

COSC 3101 Winter, 2004

COSC 3101 Winter, 2004

COSC 3101 Winter, 2004

COSC 3101 Winter, 2004

COSC 3101 Winter, 2004

Size of Input Instance?

COSC 3101 Winter, 2004

COSC 3101 Winter, 2004

Time as function of input size?

COSC 3101 Winter, 2004

COSC 3101 Winter, 2004

COSC 3101 Winter, 2004

COSC 3101 Winter, 2004

COSC 3101 Winter, 2004

COSC 3101 Winter, 2004

• # of elements: n = 10 elements ~ Reasonable

If each element has size c

COSC 3101 Winter, 2004

• # of elements: n = 10 elements Reasonable

If each element is in [1..n]

COSC 3101 Winter, 2004

COSC 3101 Winter, 2004

COSC 3101 Winter, 2004

Average Case For what distribution?

Worst Case • Provides an upper bound

COSC 3101 Winter, 2004

Bigger than this? Smaller than this?

Need to look at Need to look at

• O(n2): Provide an algorithm that solves the problem in

COSC 3101 Winter, 2004

COSC 3101 Winter, 2004

COSC 3101 Winter, 2004

n + n-1 + n-2 + ... + 2 + 1 = S

(n+1) + (n+1) + (n+1) + ... + (n+1) + (n+1) = 2S

COSC 3101 Winter, 2004

n + n-1 + n-2 + ... + 2 + 1 = S

(n+1) + (n+1) + (n+1) + ... + (n+1) + (n+1) = 2S