1

1.2.1 Complexity
2
nd
edition

When size matters
Knowledge Component 1: Theoretical Foundations
Ian F. C. Smith
EPFL, Switzerland
2
Module Information
Intended audience
Intermediate

Key words
Complexity
Program execution time
Big Oh notation

Author
Ian Smith, EPFL, Switzerland

Reviewer (1
st
Edition)
William Rasdorf, NCSU
3
What there is to learn
At the end of modules 1.2.1 and 1.2.2, there will be
answers to the following questions:
Are there certain tasks that computers cannot do
and if so, can faster computers help in these cases?
What are the cases when computational requirements
are nearly independent of task size?
Can small changes have a big effect?
Can one classify tasks in order to know whether or
not a program will perform well for full-scale tasks?
Why is engineering experience so valuable?




4/42
Complexity

Execution Time

Big Oh Notation

Classification in Big Oh Notation
Outline
5/42
Complexity
Complexity is a central theme in computer science
and is an important topic in CAE.

One of the most practical aspects involves the
classification of algorithms according to their
ability to cope with different levels of computational
complexity.

There are three types of complexity:
computational, descriptive and cognitive.
6/42
Three Types of Complexity
Computational complexity: This type is used to
classify well-structured tasks where the goal is to
find an efficient solution.

Although efficiency can be measured in terms of
use of memory and hardware resources,
algorithms are traditionally classified according to
factors that influence execution time.

7/42
Three Types of Complexity (contd.)
Descriptive complexity: A classification of the
level of difficulty involved with providing
descriptive structures to systems that have
varying degrees of intricacy.

Cognitive complexity: A classification of the
level of difficulty related to describing and
simulating human thought for various activities.

8/42

This course provides an introduction only to the
fundamental concepts of computational
complexity.

The two other types of complexities do not yet have
fixed classification schemas.

Nevertheless, they are both important areas in
computer science and their sub-domains are
gradually attracting the interest of engineers.
9/42
Complexity

Execution Time

Big Oh Notation

Classification in Big Oh Notation
Outline
10/42
Execution Time
Rather than absolute values, trends with respect to
particular factors are of most interest, viz. growth of
execution time in relation to the increase in the size
of the task.

Important factors:

Formulation of the problem (task complexity)
Algorithm complexity
Size of input
Presentation of the desired results
Hardware capacity (memory, peripherals, etc.)
Operating environment
11/42
Execution Time (contd.)
In this course, we will concentrate on the first two
aspects. It is assumed that the size of the input and
requirements related to the results are not critical
factors.
It will be shown that under certain conditions the first
two aspects can create extreme situations that are
independent of hardware capacity and operating
environments.
These aspects will remain important regardless of
advances in computing equipment.
12/42
Example 1: Part A
Estimation of trends in execution times of a program
upon modifying key parameters.

Part A

Task: Write a program that finds the minimum value
of the following function by sampling the solution
space at regular intervals.
13/42
Example 1: Part A (contd.)
0
1
2
3
-15 -5 5 15
F(x) = 1 + x
2
/4000 cos(x) for x [-10,10]

(Griewanks function)
14/42
What would be the best number of samples?

All other aspects being equal, if the number of
samples is doubled, the execution time is doubled. If
the number of samples is tripled, the execution time
is tripled.

Therefore, a sampling algorithm of this type is said to
be linear with respect to the number of samples
required.
Example 1: Part A (contd.)
15/42
Note: This is not the best algorithm for this task.

What is the best algorithm?

Example 1: Part A (contd.)
16/42
Answer
Take the differential of f(x), set it to zero and solve
for x. This algorithm requires no sampling and
execution time is the same regardless of the range of
values of x.

Discussion
Most tasks can be solved in several ways using
algorithms that have varying sensitivity to execution
time.
Example 1: Part A (contd.)
17/42
Example 1: Part B
Part B

Task: Write a program that finds the minimum value
of the following function by sampling along each axis
at regular intervals.





(General form of Griewanks function in N variables)

N
i
i
N
i
x
N i i
i x x f
i
1 1
4000 , 1
)) / (cos( 1 ) (
2
18/42
For a given number of variables N, the number of
samples increases by 2
N
if the number of sampling
points on each axis is doubled and by 3
N
if the number
of axis points is tripled.

For constant N, this algorithm is said to be
polynomial with respect to the number of sampling
points on each axis.

Example 1: Part B (contd.)
19/42
If the number of sampling points along each axis is
constant while N varies, the algorithm is said to be
exponential with respect to the number of
variables.
Example 1: Part B (contd.)
20/42
Example 1: Part B (contd.)
For 10 sampling points along each axis:
N = 2 requires 100 samples
N = 3 requires 1000 samples
N = 4 requires 10000 samples
N = 10 requires 10 billion samples
N Number of samples
10 20 30 40
1 10 20 30 40
2 100 400 900 1600
3 1000 8000 27000 64000
4 10000 160000 810000 2560000
Polynomial
Exponential

21/42

In practice, exponentially complex algorithms are
feasible only for small tasks.

The algorithm may be clear and concise and the
program may only comprise a few lines, but
execution may take hours, days or even longer.
Example 1: Part B (contd.)
22/42
What are three types of complexity?

What is a good measure of computational
complexity?

What is more desirable from a computational
point of view; polynomial or exponential
complexity? Why?
Review Quiz I
23/42
What are three types of complexity?

Computational complexity
Descriptive complexity
Cognitive complexity

Only computational complexity has fixed
classification schemas. This course includes one
of them.
Answers to Review Quiz I
24/42
What is a good measure of computational
complexity?
Trends with respect to growth of execution time in
relation to the increase in a parameter.

What is more desirable from a computational
point of view; polynomial or exponential
complexity?
Polynomial complexity because execution time
grows less rapidly with increases in task size.
Answers to Review Quiz I
25/42
Complexity

Execution Time

Big Oh Notation

Classification in Big Oh Notation
Outline
26/42
One of the popular notations expressing the
relationship between task size and amount of
computational resources required.

It provides a model of relative execution time and
task size. Therefore, Big Oh notation is very
useful for classifying levels of computational
complexity.

"Big Oh" Notation
27/42
If n = task size (integer, n>0)
f(n) = execution time (>0)
g(n) = relative execution time

Then there exists a positive constant c such that
f(n) c g(n)


Note the inequality!
"Big Oh" Notation (contd.)
28/42
We are interested in upper bound, or worst case,
estimates of execution time.

For small values of n, say n<n
0
, this may not be
true. An extra condition is nn
0
.

"Big Oh" Notation (contd.)
29/42
"Big Oh" Notation: O(g(n))
The constant, c, contains machine specific aspects of
execution (hardware, compiler, etc.). These aspects
are assumed to be independent of task size n (nn
0
).

The function g(n) represents the trend in execution
time for varying values of task size, n.

The notation O(g(n)) signifies "Order of g(n)".
30/42
Examples Revisited
Example 1, Part A
The complexity is O(n) with respect to the number
of samples taken in the interval [10,10]

Example 1, Part B
With three variables (N=3), the complexity is O(n
3
)
with respect to the number of samples, n, taken in
the interval [10, 10]

31/42
Examples Revisited (contd.)
Example 1, Part B
With 10 samples taken in the interval [10, 10] on
each axis, the complexity is O(10
N
) with respect to
the number of variables (axes), N.
32/42
Complexity

Execution Time

Big Oh Notation

Classification in Big Oh Notation
Outline
33/42
Classifications in Big Oh Notation
Logarithmic time
O(log(n)): f(n) c log(n)

Linear time
O(n): f(n) c n

Polynomial time
O(n
2
): f(n) c n
2

Exponential time
O(2
n
): f(n) c 2
n

34/42
Classifications in Big Oh Notation
(contd.)
Factorial time
O(n!): f(n) c n!

Double exponential time
O(n
n
): f(n) c n
n


Note: Big Oh notation is independent of the
number of commands a machine executes
in a second.
35/42
Simplification
What are the Big Oh notations for the following?

1. f(n) = 5n
3
+ 4n
2
+ 3n + 5

2. f(n) = 3n
3
+ 2
n

3. f(n) = 3n + log(n)

4. f(n) = 6n! + 2n
4
+ n
36/42
Simplification Answers

1. f(n) = 5n
3
+ 4n
2
+ 3n + 5 O(n
3
)

2. f(n) = 3n
3
+ 2
n
O(2
n
)


3. f(n) = 3n + log(n) O(n)

4. f(n) = 6n! + 2n
4
+ n O(n!)


Explanations follow.
37/42
Simplification Answers (contd.)

1. f(n) = 5n
3
+ 4n
2
+ 3n + 5

f(n) = 5n
3
+ 4n
2
+ 3n + 5

f(n) 5n
3
+ 4n
3
+ 3n
3
+ 5n
3

f(n) 17n
3


f(n) is O(n
3
)

38/42
Simplification Answers (contd.)

2. f(n) = 3n
3
+ 2
n

n 3n
3
2
n
3n
3
/2
n
3n
3
+ 2
n
13(2
n
)
1 3 2 1.5 5 26
2 24 4 6 28 52
3 81 8 ~10 89 104
4 192 16 12 208 208
5 375 32 11.7 407 416
6 648 64 10 712 832
Comparing the two right-hand-side columns,
f(n) 13(2
n
). Therefore, f(n) is O(2
n
)

39/42
Simplification Answers (contd.)

3. f(n) = 3n + log n

log n < n

f(n) 4n


Therefore, f(n) is O(n)

40/42
Simplification Answers (contd.)

4. f(n) = 6n! + 2n
4
+ n


n 6n!

2n
4
+ n 2n
4
+n / 6n!

6n!+2n
4
+n

6(6n!)

1 6 3 0.5 9 36
2 12 34 2.8 46 72
3 36 165 4.6 201 216
4 144 516 3.6 660 864
5 720 1255 1.7 1975 4320
Comparing the two right-hand-side columns,
f(n) 36(n!). Therefore, f(n) is O(n!)

41/42
Summary
Complexity is an important topic in engineering

Three types of complexity: computational,
descriptive, cognitive

The Big Oh notation is useful for characterizing
computational complexity

High complexity (e.g. exponential or factorial
complexity) => excessive execution times for large
problems
42/42
Remarks
When complexity is exponential, it is not possible to
find solutions in a reasonable amount of time for
large values of n

Increases in computer power does not help

Presence of well-defined algorithms is not the only
criterion for computability. For example, a well-
defined algorithm that has factorial complexity
would not be computable for high values of task
size.
Further reading
Computers and Intractability: A Guide to the Theory
of NP-Completeness, M. Garey and D. Johnson, W.H.
Freeman and Company, New York, 1979

Algorithmics, The Spirit of Computing, 3rd ed., D.
Harel, Haddison-Wesley, 2004

Engineering Informatics - Fundamentals of
Computer-Aided Engineering, B. Raphael and I.F.C.
Smith, Wiley, 2013