1

1. 1. Introduction Introduction
Objectives
What is a problem?
How do we state a problem?
What does it mean to solve it?
What is an algorithm?
How do we describe it?
What different kinds of algorithms are there?
What does it mean for an algorithm to be correct? What does it mean for an algorithm to be correct?
How can we prove correctness?
What does it mean for an algorithm to be efficient?
How can we calculate and compare efficiency of algorithms?
Can we be sure weve found the most efficient algorithm for
a problem?
What design techniques are available?
What is a Problem?
A Problem is something we want to solve, or a question we
want to answer.
A computational problem is a general question to be
answered.
General: has inputs (parameters), whose values are left
unspecified.
The answers are the outputs The answers are the outputs.
To state a problem:
1. Define the inputs, and what properties they must satisfy
2. Define a relation between inputs and outputs, giving the
properties a solution must satisfy.
Note that in real problems, inputs and outputs may be
interleaved over time: we make the simplifying assumption that
inputs come before outputs.
Examples of Computational Problems
Example 1: Searching a sequence of integers
1.(a) sequence of integers, s
(b) search value x (an integer)
2.(a) output index i such that s[i ] = x , if one exists,
otherwise i = 1.
Example 2: Searching an ordered sequence of
integers integers
1.(a) sequence of integers, s, in ascending order
(b) search value x (an integer), known to be in the
sequence s
2.(a) output index i such that s[i ] = x .
Example 3: Sorting integers
1. A sequence s of integers
2. A new sequence s with the same elements of s, but in
ascending order.
Examples of Computational Problems
Example 3:
Suppose you have one machine and a set of n jobs
a
1
, a
2
, , a
n
to process on that machine. Each job
a
j
has a processing time t
j
, a profit p
j
, and a deadline
d
j
. The machine can process one job at a time, and
a job a
j
must run uninterruptedly for t
j
consecutive
time units. If job a
j
is completed by its deadline d
j
,
you receive a profit p
j
, but if it is completed after its y p p
j
p
deadline, you receive a profit of 0. Give an
algorithm to find the schedule that obtains the
maximum amount of profit, assuming that all
processing times are integers between 1 and n.
What is the running time of your algorithm?
Problem instances
An instance is obtained by taking a problem and defining
particular values for the inputs.
Example: sorting integers
1. input s = {3, 1, 4, 1, 5, 9, 2, 7}
2. output s = {1, 1, 2, 3, 4, 5, 7, 9}
Sometimes the output for a problem instance is not uniquely p p q y
defined:
Example: searching integers
1.(a) input s = {3, 1, 4, 1, 5, 9, 2, 7}
(b) input x = 1
2. output i = 2 OR i = 4.
Underspecification is sometimes desirable.
2
What is an algorithm?
A set of instructions giving a step-by-step procedure
which, given some inputs, produces some outputs.
An algorithm is any well-defined computational
procedure that takes some value, or a set of values
as input and produces some value, or a set of values
as output.
May be defined
i E li h ( l t id ) in English (e.g. enrolment guide)
by diagrams (e.g. instructions for assembling kitset
furniture)
in a computer programming language
with pseudocode
in some other more-or-less formal language (e.g. knitting
pattern).
Data structure is a way of storing and organising
data in order to facilitate access and modification
Real-life problems
Examples of real-life problems that are solved by
algorithms:
The Human Genome Project
Internet Searches
E-Commerce
Finding Shortest ro tes Finding Shortest routes
Matrix Multiplication
Techniques for design and analysis of algorithms;
necessary to design own algorithms
Hard problems no efficient solution is known,
e.g. NP Complete
*
Algorithms as a Technology
Computers are fast but not infinitely fast
Memory is cheap but not free
We need efficient algorithms to manage computer
resources
Efficiency of algorithms will determine how long it will
take to solve problems; these differences are much more take to solve problems; these differences are much more
significant than differences in hardware and software
*
Algorithm M/C
Speed
Language No of
Instructions
Time to
sort
n=1Million
Insertion
Sort
1Billion
Ins/Sec
M/C 2n
2
2,000secs
Merge
Sort
10Million
Ins/Sec
HL 50nLog
2
n Approx
100secs

Algorithms and Other Techniques
Like other techniques (pipelining, OO, AOP
etc), algorithms are undergoing developments
Simple applications may not require algorithms
but they rely on algorithms for facilities such as
GUI, network routing, compilers etc that are , g, p
based on algorithms
Knowledge of algorithms differentiate
programmers from coders
Most programmers today arent programmers at all. They are
coders. There is a big difference. Joseph DeCarolis; IEEE
Spectrum, January 2006. www.spectrum.ieee.org
*
One by one, each as yet
unsorted array element is
inserted into its proper
l ith t t th
Insertion Sort
values [ 0 ]
[ 1 ]
36
24
11
place with respect to the
already sorted elements.
On each pass, this causes
the number of already
sorted elements to increase
by one.
[ 2 ]
[ 3 ]
[ 4 ]
24
10
6
12
Works like someone who
inserts one more card at
a time into a hand of cards
that are already sorted.
Insertion Sort
12
that are already sorted.
To insert 12, we need to
make room for it by moving
first 36 and then 24.
3
Works like someone who
inserts one more card at
a time into a hand of cards
that are already sorted.
Insertion Sort
13
that are already sorted.
To insert 12, we need to
make room for it by moving
first 36 and then 24.
Works like someone who
inserts one more card at
a time into a hand of cards
that are already sorted.
Insertion Sort
14
that are already sorted.
To insert 12, we need to
make room for it by moving
first 36 and then 24.
Works like someone who
inserts one more card at
a time into a hand of cards
that are already sorted.
Insertion Sort
15
that are already sorted.
To insert 12, we need to
make room for it by moving
first 36 and then 24.
Insertion Sort: Pass One
values [ 0 ]
[ 1 ]
36
24
SORTED
U
16
[ 2 ]
[ 3 ]
[ 4 ]
24
10
6
12
N
S
O
R
T
E
D
Insertion Sort: Pass Two
values [ 0 ]
[ 1 ]
24
36
SORTED
17
[ 2 ]
[ 3 ]
[ 4 ]
36
10
6
12
U
N
S
O
R
T
E
D
Insertion Sort: Pass Three
values [ 0 ]
[ 1 ]
10
24
S
O
R
18
[ 2 ]
[ 3 ]
[ 4 ]
24
36
6
12
UNSORTED
T
E
D
4
Insertion Sort: Pass Four
values [ 0 ]
[ 1 ]
6
10
S
O
R
19
[ 2 ]
[ 3 ]
[ 4 ]
10
24
36
12
R
T
E
D
UNSORTED
Insertion Sort: Pass Five
values [ 0 ]
[ 1 ]
6
10
S
20
[ 2 ]
[ 3 ]
[ 4 ]
10
12
24
36
O
R
T
E
D
Insertion Sort
Algorithms solve problems
An algorithm may solve:
a computational problem:
To search for x in sequence s, try each index i between 1
and | s | in turn; if you find s[i ] = x , output i , otherwise
output 1.
an instance of a problem: p
To sort the sequence {3, 1, 4, 1, 5, 9, 2, 7}, output the new
sequence {1, 1, 2, 3, 4, 5, 7, 9}.
a restricted set of instances of a problem:
To search for x in sequence s, do a binary search; if you
find s[i ] = x , output i , otherwise output 1.
If an algorithm is being proposed as a correct
solution to some computational problem, it is
important that it solves all instances.
Correctness
An algorithm is correct with respect to a problem
if
for every instance of the inputs (that is, the specified
properties of the inputs are satisfied)
the algorithm halts, and
the outputs produced satisfy the specified input/output
relation. relation.
Correctness must be proven.
One counter-example is enough to disprove
correctness, but (in general), no amount of
testing can prove correctness.
Note that correctness of an algorithm doesnt
necessarily guarantee correctness of a program
that claims to implement the algorithm.
Another Example: bubble sort
inputs: array A of integers, of length n
outputs: ordered array A, with same elements as
A0
1. for i 1 to n
2. for j n down to i + 1
3. if A[j ] < A[j 1] then
4. swap A[j ] and A[j 1]
We can assume the inputs satisfy the
specification, i.e. that A is an array of integers and
n is its length. Our array indexes count from 1, so
the values are A[1], . . . ,A[n].
5
Example: bubble sort
Does it halt?
Yes, it always does in
n
i=1
(n i ) iterations.
Does it produce correct output?
Each time round the outer loop, j will eventually point to the
smallest element in the range i + 1..n. This element will then
bubble to the position A[i ]. So the i
th
iteration of the outer
loop puts the i
th
smallest element into position A[i ] loop puts the i
th
smallest element into position A[i ].
We could formalise that argument using logic and
mathematics.
We will usually satisfy ourselves with informal arguments
such as this; it is nonetheless important that our arguments
are thorough and convincing.
Later, we will explicitly use loop invariants to make our
(informal) arguments more convincing.
Bubble Sort Illustration
Analysis
Suppose two algorithms are both correct
solutions to some computational problem. How do
we decide which one to use?
Availability in library
Easy to code (or prove correct)
Resources needed for typical problem
Resources needed for worst-case problem
Resources:
computation time
maximum memory use
memory turnover
number of logic gates, length of wire
Mostly, we consider time.
Analysing computation time
How long does it take to sort a list?
One way to analyse computation time is to write a program, execute it,
and measure how long it takes.
The answer will depend on
the algorithm used
the size of the input list p
the elements in the input list
the programming language chosen
how the algorithm is implemented
the CPU clock speed
the operating system
the amount of free memory
the quality of the compiler
the other tasks running on the computer
Only the first three have anything to do with the algorithm.
Ignoring implementation detail
We would like to give an indication of running time
for an algorithm that depends only on the
algorithm itself and its input.
Often, we can express running time as
proportional to some (fixed) function of the size of p p ( )
the input, and ignore the actual values of the input.
Time f (n)
or equivalently,
Time = k f (n) for some k that depends on
implementation.
Ignoring implementation detail
In formal algorithm analysis (this course), we
usually say
The algorithm is in the order of f (n), written (f (n)).
We can also say an algorithm is O(f (n)) if we can
show that the time is proportional to at most f (n).
Later, well define and O (and also , for lower
bounds) more precisely.