Growth of functions: asymptotic notation

To characterize the time cost of algorithms, we focus on functions that map input size to (typically, worst-case) running time. (Similarly for space costs.) We are interested in precise notation for characterizing running-time dierences that are likely to be signicant across dierent platforms and dierent implementations of the algorithms. This naturally leads to an interest in the asymptotic growth of functions. We focus on how the function behaves as its input grows large. Asymptotic notation is a standard means for describing families of functions that share similar asymptotic behavior. Asymptotic notation allows us to ignore small input sizes, constant factors, lower-order terms in polynomials, and so forth.

Big O notation (asymptotic upper bound)

Denition f (n) O (g (n)) if there are positive c , n0 s.t. |f (n)| c |g (n)| for all n n0 .

We assume that these functions map reals to reals. They may be partial functions, but, as well explain shortly, we will assume that they are asymptotically-dened on N . It is common to write f (n) = O (g (n)) instead of f (n) O (g (n)). Beware: this is not the same as writing O (g (n)) = f (n). What about the awkwardness of writing f (n) instead of f , and g (n) instead of g ? (That is, writing an expression for the value of function f at n, when we wish to name the function f itself.) It corresponds to the standard (and convenient) practice of writing, say, O (n2 ) to stand for the set of functions f : R R s.t. for some positive c , n0 , |f (n)| c |n2 | for all n n0 .

Is g (n) = O (g (n)) ?

Asymptotically dened on N
We say were interested in functions from R to R, but . . . . . . what about a function like log2 n ? Problem: not dened for n 0. An approach that couldnt talk about log-space and log-time would not be very useful. And what about a function like n! ? (Commonly dened with domain N .)

It is convenient to consider any (partial) function from R to R that is asymptotically dened on N that is, dened for all natural numbers n n0 , for some constant n0 . So lets assume that in asymptotic notation we consider (partial) functions from R to R that are asymptotically dened on N .

Ignoring constants
Asymptotic notation is supposed to allow us to ignore constants. Lets look at how this works. . . Is n + k = O (n)? f (n) O (g (n)) if there are positive c , n0 s.t. |f (n)| c |g (n)| for all n n0 .

If k = 0, the result is trivial. (Take c = 1, n0 = 1.) If k is positive: We can take c = 2 and n0 = k . Notice that |n + k | 2|n| What if k is negative? Then we can take c = 1 and n0 = k , and notice that |n + k | |n| So, for all k R, n + k = O (n ) . for all n k . for all n k .

If f (n) = O (g (n)) , is kf (n) = O (g (n)) ? f (n) O (g (n)) if there are positive c , n0 s.t. |f (n)| c |g (n)| for all n n0 .

If k = 0, the result is immediate, so assume otherwise. Assume f (n) = O (g (n)). So there are positive constants c , n0 s.t. |f ( n ) | c |g ( n ) | Take c = c |k | and n0 = n0 . Observe that, for all n n0 , |kf (n)| = = |k | |f (n)| | k | c |g ( n ) | c |g ( n ) | for all n n0 .

So kf (n) = O (g (n)) whenever f (n) = O (g (n)).

If f (n) = O (g (n)) , is f (n) + k = O (g (n)) ? f (n) O (g (n)) if there are positive c , n0 s.t. |f (n)| c |g (n)| for all n n0 .

Assume that f (n) = O (g (n)). We need positive c and n0 s.t. |f (n) + k | c |g (n)| for all n n0 .

In general, such c and n0 may fail to exist. But heres a simple condition that will guarantee their existence. . . Assume that |g (n)| has an asymptotic nonzero lower bound. That is, for some positive b and m0 , |g ( n ) | b Since f (n) = O (g (n)), there are c , n0 s.t. f (n) c g (n) for all n n0 . for all n m0 .

It suces to take c = |k /b | + c and n0 = m0 + n0 . Lets check that. . .

Given: For some positive b and m0 , |g ( n ) | b and, for some positive c , n0 , |f ( n ) | c |g ( n ) | NTS: There are positive c and n0 s.t. |f ( n ) + k | c |g ( n ) | Take c = |k /b | + c and n0 = m0 + n0 . Notice that, for all n n0 , |f ( n ) + k | = = = |f (n)| + |k | c |g (n)| + |k | n )| c |g (n)| + |k | |g ( b c |g (n)| + |k /b | |g (n)| (c + |k /b |) |g (n)| c | g (n )| for all n n0 . for all n n0 . for all n m0 ,

So, if |g (n)| has a nonzero asymptotic lower bound, then f (n) + k = O (g (n)) whenever f (n) = O (g (n)).

Polynomials and big O

Our results thus far can explain, for instance, the well-known fact that ank = O (nk ) and, similarly, that n k + b = O (n k ) But what about drawing the conclusion that ank + b = O (nk ) (k 0) (k 0)

It looks like this should be true, but wed need another argument. There is a much more general fact about polynomials and Big O that you should know: Any polynomial expression whose highest order term is ank belongs to O (n k ) . Lets prove it. . .

Polynomials of degree at most k belong to O (nk )

Consider an arbitrary expression of the form ak nk + ak 1 nk 1 + + a2 n2 + a1 n1 + a0 n0 with k N and ak , . . . , a0 R. (Notice that any of ak , . . . , a0 may equal zero. If all do, the result is trivial, so well assume that at least one does not.) For convenience, well write this expression in the form
k X i =0

ai n i .

Notice that, for all n 1, P k i =0 ai ni

= =

Pk

i i =0 |ai n | Pk i i =0 |ai |n P k |a i | n k Pi =0 k k i =0 |ai | |n |

which shows that

k X i =0

a i n i = O (n k ) .

(Take n0 = 1 and c =

Pk

i =0 |ai |.)

If f (n) = O (g (n)) and g (n) = O (h(n)), then f (n) = O (h(n))

Assume f (n) = O (g (n)) and g (n) = O (h(n)). It follows that there are positive c , n0 , c , n0 s.t. |f (n)| c |g (n)| and |g (n)| c |h(n)| Take c = c c and n0 = n0 + n0 . Notice that |f (n)| c |g (n)| c c |h(n)| = c |h(n)| for all n n0 . for all n n0 . for all n n0

If f (n) = O (h(n)) and g (n) = O (h(n)), then f (n) + g (n) = O (h(n))

Assume f (n) = O (h(n)) and g (n) = O (h(n)). It follows that there are positive c , n0 , c , n0 s.t. |f (n)| c |h(n)| and |g (n)| c |h(n)| Take c = c + c and n0 = n0 + n0 . Then |f (n) + g (n)| |f (n)| + |g (n)| c |h(n)| That is, f (n) + g (n) = O (h(n)). for all n n0 . for all n n0 . for all n n0

It follows that if fi (n) = O (g (n)) for all i N , then

k X i =0

fi (n) = O (g (n))

for any k N .

If f (n) + g (n) = O (h(n)) and f (n) and g (n) are asymptotically nonnegative, then f (n) = O (h(n))
Assume that f (n) + g (n) = O (h(n)) and that both f (n) and g (n) are asymptotically nonnegative. It follows that there are positive c , n0 , m0 , m0 s.t. |f (n) + g (n)| c |h(n)| and and 0 f (n ) 0 g (n) for all n n0

for all n m0 for all n m0

Take c = c and n0 = n0 + m0 + m0 . Then |f (n)| c |h(n)| for all n n0 .

What if one of f (n),g (n) is not asymptotically nonnegative? (Consider, for example, f (n) = n, g (n) = n, h(n) = 0.)

If f (n) g (n) = O (h(n)) and g (n) has a positive asymptotic lower bound, then f (n) = O (h(n))
Assume that f (n) g (n) = O (h(n)) and that g (n) has a positive asymptotic lower bound. So there are positive c , n0 , b , m0 s.t. |f ( n ) g ( n ) | c |h ( n ) | and b g (n ) for all n m0 . for all n n0

Take c = c /b and n0 = n0 + m0 . Notice that b |f (n)| = |bf (n)| |f (n) g (n)| c |h(n)| Therefore, |f ( n ) | c |h(n)| = c |h(n)| b for all n n0 for all n n0

Is it true that if f (n) g (n) = O (h(n)), then at least one of f (n), g (n) belongs to O (h(n))? Suppose n , if n is even n , if n is odd f (n ) = g (n ) = h (n ) = 0 0 , otherwise 0 , otherwise

If f (n) = O (g (n)) and f (n) = O (g (n)), then f (n) f (n) = O (g (n) g (n))
Assume f (n) = O (g (n)) and f (n) = O (g (n)). So there are positive c , n0 , c , n0 s.t. |f (n)| c |g (n)| and |f (n)| c |g (n)| for all n n0 . Take c = c c and n0 = n0 + n0 . Then, for all n n0 , |f ( n ) f ( n ) | = = = = |f (n)| |f (n)| c |g (n)| c |g (n)| (c c ) |g (n)| |g (n)| c (|g (n)| |g (n)|) c |g (n) g (n)| for all n n0

It follows that if fi (n) = O (gi (n)) for all i N , then ! k k Y Y f i (n ) = O gi (n)

i =0 i =0

for any k N .

nj = O (nk ) if j > k

Assume j > k . Suppose there are positive c , n0 s.t. |n j | c |n k | Then nj k c for all n n0 . for all n n0 .

But this is not possible, since j k is positive. Contradiction. We conclude that the supposition was false. Hence nj = O (nk ) when j > k .

Asymptotic lower bound, asymptotically tight bound

Denition (asymptotic lower bound) f (n) (g (n)) if there are positive c , n0 s.t. c |g (n)| |f (n)| for all n n0 .

Claim: f (n) = (g (n)) i g (n) = O (f (n)). Easy to show. (Try it.)

Denition (asymptotically tight bound) f (n) (g (n)) if there are positive c , d , n0 s.t. c |g (n)| |f (n)| d |g (n)| Theorem
? ?

for all n n0 .

f (n) = (g (n)) i f (n) = (g (n)) and f (n) = O (g (n)).

f (n) = (f (n)) f (n) = (f (n)) You might want to consider whether other properties of big O hold for , .

Other asymptotic notations

Denition (asymptotically strict upper bound) f (n) o (g (n)) if for every positive c there is a positive n0 s.t. |f (n)| < c |g (n)| for all n n0 .

Notice: If f (n) = o (g (n)) then f (n) = O (g (n)). Claim: If f (n) = o (g (n)) then f (n) = (g (n)).

Denition (asymptotically strict lower bound) f (n) (g (n)) if for every positive c there is a positive n0 s.t. c |g ( n ) | < |f ( n ) | for all n n0 .

Notice: If f (n) = (g (n)) then f (n) = (g (n)). Claim: If f (n) = (g (n)) then f (n) = O (g (n)). Claim: f (n) = o (g (n)) i g (n) = (f (n)).

A few more laws

8 o (h(n)) > > > > < O (h(n)) (h(n)) If f (n) = (g (n)) and g (n) = > > (h(n)) > > : (h(n)) 9 > > > > = > > > > ; 8 o (h(n)) > > > > < O (h(n)) (h(n)) then f (n) = > > (h(n)) > > : (h(n)) 9 > > > > = > > > > ;

If f (n) =

o (g (n)) (g (n))

and g (n) =

O (h(n)) (h(n))

then f (n) = o (h(n)) (h(n)) .

If f (n) =

O (g (n)) (g (n))

and g (n) =

o (h(n)) (h(n))

then f (n) = o (h(n)) (h(n)) .

Some facts about hierarchy of asymptotic classes

nn beats factorial: n! = o (nn ) Factorial beats exponential: k n = o (n!) Exponential beats polynomial: n k = o (b n ) if b > 1

Higher-degree polynomial beats lower-degree polynomial: n j = o (n k ) if 0 j < k

Polynomial beats (poly)logarithmic:

k loga b n = o (n )

if b > 1 and k > 0

a Note: loga b n is an abbreviation for (logb n) .

The base of a (poly)logarithm doesnt matter:

k logk a n = (logb n)

if a, b > 1

The base of an exponential matters: a n = o (b n ) if 1 a < b

Adding a constant to the exponent (of an exponential) doesnt matter: an = (an+k )

A constant factor in the exponent matters: an = o (abn ) if a, b > 1

A hierarchy of asymptotically 2strict bounds

2 nn n! 3n n 2n 2n n3 2 n log n n2 n log n n n log3 n log2 n log n p log n log log n log log log n 1 1/n 0
n