Lower Bounds

Lower bound: an estimate on a

minimum amount of work needed to
solve a given problem
Lower bound can be
an exact count
an efficiency class ()
Tight lower bound: there exists an
algorithm with the same efficiency as
the lower bound
1

Methods for Establishing

Lower Bounds
trivial lower bounds
information-theoretic arguments
(decision trees)
adversary arguments
problem reduction
2

Trivial Lower Bounds

Based on counting the number of
items that must be processed in
input and generated as output
Examples
finding max element
sorting
element uniqueness
Not always
useful
3

Decision Trees
A convenient model of algorithms
involving comparisons in which:

internal nodes represent

comparisons

leaves represent outcomes

4

Adversary Arguments
Adversary argument: a method of
proving a lower bound by playing role
of adversary that makes algorithm
work the hardest by adjusting input

Example: Guessing a number

between 1 and n with yes/no
questions

Adversary: Puts the number in a

larger of the two subsets generated
6

Lower Bounds by Problem

Reduction
Idea: If problem P is at least as
hard as problem Q, then a lower
bound for Q is also a lower bound
for P.
Hence, find problem Q with a
known lower bound that can be
reduced to problem P in question.
Then any algorithm that solves P
7

Class P
P: the class of decision problems that are solvable in
O(p(n)) time, where p(n) is a polynomial of problems
input size n
Examples:
searching
element uniqueness
graph connectivity
graph acyclicity

Class NP
NP (nondeterministic polynomial): class of decision
problems whose proposed solutions can be verified in
polynomial time = solvable by a nondeterministic
polynomial algorithm
A nondeterministic polynomial algorithm is an abstract
two-stage procedure that:
generates a solution of the problem (on some input)
by guessing
checks whether this solution is correct in polynomial
time
By definition, it solves the problem if its capable of
generating and verifying a solution on one of its tries

Example: CNF satisfiability

Problem: Is a boolean expression in its conjunctive normal
form (CNF) satisfiable, i.e., are there values of its variables
that make it true?
This problem is in NP. Nondeterministic algorithm:
Guess a truth assignment
Substitute the values into the CNF formula to see if it
evaluates to true
Example: (A | B | C) & (A | B) & (B | D | E) & (D | E)
Truth assignments:
ABCDE
0 0 0 0 0
. . .
1 1 1 1 1

Checking phase: O(n)

NP-Complete Problems
A decision problem D is NP-complete if it is as hard as
any
problem in NP, i.e.,
D is in NP
every problem in NP is polynomial-time reducible to D
NP problems

NP-complete
problem

Cooks theorem (1971): CNF-sat is NP-complete

Backtracking
Suppose you have to make a series of
decisions, among various choices, where
You dont have enough information to know
what to choose
Each decision leads to a new set of choices
Some sequence of choices (possibly more than
one) may be a solution to your problem

Backtracking is a methodical way of trying

out various sequences of decisions, until you
find one that works
13

Backtracking
Construct the state-space tree [State space is the set of all
paths from root node to other nodes ]
nodes: partial solutions
edges: choices in extending partial solutions
Explore the state space tree using depth-first search
Prune nonpromising nodes
DFS stops exploring subtrees rooted at nodes that cannot
lead to a solution and backtracks to such a nodes parent to
continue the search

Backtracking Algorithm
Based on depth-first recursive search
Approach
1. Tests whether solution has been found
2. If found solution, return it
3. Else for each choice that can be made
a) Make that choice
b) Recur
c) If recursion returns a solution, return it

4. If no choices remain, return failure

Some times called search tree

15

Example: n-Queens Problem

Place n queens on an n-by-n chess board so
that no two of them are in the same row,
column, or diagonal
1

queen 1

queen 2

queen 3

queen 4

State-Space Tree of the 4-Queens Problem

Hamiltonian Circuits Problem

A Hamiltonian circuit or tour of a graph is a
path that starts at a given vertex, visits each
vertex in the graph exactly once, and ends at
the starting vertex.
Some graphs do not
contain
circuits
. v2
v1 Hamiltonian
v2
v3
v1
v3
v4

v5

v6

v4

v5

v6

A state space tree for this problem is as follows. Put the starting vertex at level 0
in the tree, call this the zero'th vertex on the path. At level 1, consider each
vertex other than the starting vertex as the first vertex after the starting one. At
level 2, consider each of these vertices as the second vertex, and so on. You
may now backtrack in this state space tree.

State-space tree for finding a Hamiltonian circuit. The

numbers above the nodes of the tree indicate the order in
which the nodes are generated.

Subset-sum problem
Given positive numbers wi, 1in, and
m, find all subsets of the wi whose sum
is m
eg) (w1,w2,w3,w4) = (11,13,24,7) and
m=31
Solutions are (11,13,7) and (24,7)

Method
Sort the sets elements in increasing order
The root of the tree represents the starting
point, with no decisions about the given elements made as yet.

Its left
and
right
children
represent,
respectively, inclusion and exclusion of (first
element) in a set being sought.
Similarly, going to the left from a node of the first level
corresponds to inclusion of (second element) while going to the
right corresponds to its exclusion, and so on.
Thus, a path from the root to a node on the i th level of the tree
indicates which of the first I numbers have been included in the
subsets represented by that node.

 We record the value of s, the sum of these numbers,

in the node.
If s is equal to d, we have a solution to the problem.
We can either report this result and stop
or, if all the solutions need to be found, continue by
backtracking to the nodes parent.
If s is not equal to d, we can terminate
the node as nonpromising if either of the
following two inequalities holds:

0
w/o 3

with 3
3

with 5

w/o 5

14
X

with 6

8
with 7

w/o 7

14+7>15

9
X
9+7>15

15
solution

with 5
3

w/o 6

with 6

8
X
8<15

5
w/o 6

with 6

w/o 5
0

w/o 6

11

5
X

3+7<15

11+7>14

5+7<15

0+13<15

Branch and Bound

The branch-and-bound problem solving method is very

similar to backtracking in that a state space tree is used to
solve a problem. The differences are that B&B
(1)does not limit us to any particular way of traversing
the tree and
(2) is used only for optimization problems.

A B&B algorithm computes a number (bound) at a node to determine

whether the node is promising. The number is a bound on the value of
the solution that could be obtained by expanding the state space tree
beyond the current node.

The main idea

Set up a bounding function, which is used to compute a bound
(for the value of the objective function) at a node on a statespace tree and determine if it is promising
Promising (if the bound is better than the value of the
best soluton so far:expand beyond the node.
Nonpromising (if the bound is no better than the value of
the best solution so far--:not smaller for a minimization
problem and not larger for a maxmization problem ): not
expand beyond the node (pruning the state-space tree).

28

Terminate a search path at the current node in a state-space tree for

any one of the following three reasons:

The value of the nodes bound is not better than the value of
the best solution seen so far.
The node reprsents no feasible solutions because the
constraints of the problem are already violated.
The subset of feasible solutions reprsented by the node
consists of a single point(and hence no further choices can be
made)-in this case we compare the value of the objective
function for this feasible solution with that of the best
solution seen so far and update the latter with the former if
the new solution is better.

29

Assignment problem

Assigning n people to n jobs so that the total cost of the

assignment is as small as possible

30

0
Start
Ib=10

a1

Ib=17

Ib=10

Cost=13

Ib=13

Ib=14

Ib=18

7
b

Ib=17

Ib=13

10

Ib=20

b1

8
c

lb=25

complete state-space tree for the

instance of the assignment problem
solved with the best-first branch-andbound algorithm
31

Knapsack problem

given n items of known weights wi and values vi ,

It is convenient to order the items of a given instance in descending order
by their value-to-weight ratios. Then the first item gives the best payoff
per weight unit and the last one gives the worst payoff per weight unit,
with ties resolved arbitrarily

(1) v1/w1v2/w2 vn/wn

(2) bound function:
ub=v+(W-w)(vi+1/wi+1)
32

Example
Item
1
2
3
4

weight value value/weight

4
40
10
7
42
6
5
25
5
3
12
4

W=10

33

w=0,v=0
Ub=100
With 1
w=4,v=40
Ub=76
w=11

w=0,v=0
Ub=60
w=4,v=40
Ub=70

w=9,v=65
Ub=69

w=12

W/o 1

w=4,v=40
Ub=64

w=9,v=65
Ub=65
34

Traveling Salesman ProblemBounding

Function

Because every vertex must be entered and exited exactly once, a

lower bound on the length of a tour is the sum of the minimum
cost of entering and leaving every vertex.
For a given edge (u, v), think of half of its weight as the the
exiting cost of u, and half of its weight as the entering cost of
v.
The total length of a tour = the total cost of visiting( entering
and exiting) every vertex exactly once.
The lower bound of the length of a tour = the lower bound of
the total cost of visiting (entering and exiting ) every vertex
exactly once.
Calculation:
for each vertex, pick top two shortest adjacent edges (their sum
divided by 2 is the lower bound of the total cost of entering and
exiting the vertex);
add up these summations for all the vertices.

Assume that the tour starts with vertex a and that b is visited
before c.
36

Traveling salesman
0

a
Ib=14

a,b

3
a,c

Ib=14

a,b,e

a,b,d

Ib=16

Ib=19

Ib=16

a,b,c,d,e,a

a,d

a,e

Ib=16

Ib=19

a,b,c
8
l=24

b is not before c
10

11

a,b,c,d,e,a

a,b,d,c,e,a

a,b,d,e,c,a

l=19

I=24

I=16
37

Approximation Approach
Apply a fast (i.e., a polynomial-time) approximation
algorithm to get a solution that is not necessarily
optimal but hopefully close to it
Accuracy measures:
accuracy ratio of an approximate solution sa
r(sa) = f(sa) / f(s*) for minimization problems
r(sa) = f(s*) / f(sa) for maximization problems
where f(sa) and f(s*) are values of the objective
function f for the approximate solution sa and actual
optimal solution s*
performance ratio RA of the algorithm A
the lowest upper bound of r(sa) on all instances

 A polynomial-time approximation
algorithm is said to be a c
approximation algorithm, where c
1, if the accuracy ratio of the
approximation
it produces does not exceed c for any
instance of the problem in question:
r(sa) c

 A heuristic is a common-sense rule

drawn from experience rather than
from
a
mathematically
proved
assertion.
For example, going to the nearest
unvisited city in the traveling
salesman
problem
is
a
good
illustration of this notion.

Approximation Algorithms for the

Traveling
Salesman Problem
Given n cities with known distances between
each pair, find the shortest tour that passes
through all the cities exactly once before
returning to the starting city
Alternatively: Find shortest Hamiltonian
circuit in a weighted connected graph

 Example:
2

8
c

4
d

TSP by Exhaustive Search

Tour
abcda
abdca
acbda
acdba
adbca
adcba

Cost
2+3+7+5 = 17
2+4+7+8 = 21
8+3+4+5 = 20
8+7+4+2 = 21
5+4+3+8 = 20
5+7+3+2 = 17

The Nearest Neighbour

Algorithm
The following well-known greedy algorithm is based on the
nearest-neighbor
heuristic: always go next to the nearest unvisited city.
Step 1 Choose an arbitrary city as the start.
Step 2 Repeat the following operation until all the cities
have been visited:
go to the unvisited city nearest the one visited last (ties can be
broken arbitrarily).
Step 3 Return to the starting city.

Multifragment-Heuristic
Algorithm
Stage 1: Sort the edges in nondecreasing order of weights.
Initialize the set of tour edges to be an empty set
Stage 2: Add next edge on the sorted list to the tour, skipping
those whose addition would have created a vertex of
degree 3 or a cycle of length less than n. Repeat
this step until a tour of length n is obtained
Step 3
Return the set of tour edges.

 {(a, b), (c, d), (b, c), (a, d)}

Minimum-Spanning-Tree
Based Algorithms

Twice-Around-the-Tree
Algorithm
Stage 1: Construct a minimum spanning tree of the graph
(e.g., by Prims or Kruskals algorithm)
Stage 2: Starting at an arbitrary vertex, create a path that goes
twice around the tree and returns to the same vertex
Stage 3: Create a tour from the circuit constructed in Stage 2 by
making shortcuts to avoid visiting intermediate vertices
more than once

Local Search Heuristics for

TSP
Start with some initial tour (e.g., nearest neighbor). On
each iteration, explore the current tours neighborhood
by exchanging a few edges in it. If the new tour is
shorter, make it the current tour; otherwise consider
another edge change. If no change yields a shorter
tour, the current tour is returned as the output.
Example of a 2-change
C4

C1

C3

C2

C4

C1

C3

C2

2-opt
Take two edges
(v,w) and (x,y) and
replace them by
(v,x) and (w,y) OR
(v,y) and (w,x) to
get a tour again.
Costly: part of tour
should be turned
around
53

Knapsack problem:
Input: n items: (w1, v1), , (wn, vn) and a
number W
the i-th items is worth vi dollars with weight wi.

at most W pounds to be carried.

Output: Some items which are most

valuable and with total weight at most W.
Two versions:
0-1 Knapsack: Items cannot be divided into pieces
fractional knapsack: Items can be divided into
pieces

05/01/16

Page 55

Solve the fractional Knapsack

problem:

Greedy on the value per pound vi/wi.

Each time, take the item with maximum vi/wi .
If exceeds W, take fractions of the item.

Example: (1, 5$), (2, 9$), (3, 12$), (3, 11$) and w=4.

vi/wi :
5
4.5
4.0
3.667

First: (1, 5$), Second: (2, 9$), Third: 1/3 (3,

12$)

Total W: 1,
3,
4.
item

05/01/16

Can only take part of

Page 56