by Lale Yurttas, Texas

A&M University
Chapter 7 1
Copyright The McGraw-Hill Companies, Inc. Permission required for reproduction or display.
OPTIMIZATION
by Lale Yurttas, Texas
A&M University
2
OPTIMIZATION
Root finding and optimization are related, both
involve guessing and searching for a point on a
function.
Fundamental difference is:
Root finding is searching for zeros of a function or
functions
Optimization is finding the minimum or the
maximum of a function of several variables.
Chapter 7
by Lale Yurttas, Texas
A&M University
3
figure PT4.1
Chapter 7
by Lale Yurttas, Texas
A&M University
4
Mathematical Background
An optimization or mathematical programming
problem generally be stated as:
Find x, which minimizes or maximizes f(x) subject to

* , , 2 , 1 ) (
* , , 2 , 1 ) (
p i b x e
m i a x d
i i
i i

Where x is an n-dimensional design vector, f(x) is
the objective function, d
i
(x) are inequality
constraints, e
i
(x) are equality constraints, and a
i

and b
i
are constants
Chapter 7
by Lale Yurttas, Texas
A&M University
5
Optimization problems can be classified on the
basis of the form of f(x):
If f(x) and the constraints are linear, we have linear
programming.
If f(x) is quadratic and the constraints are linear,
we have quadratic programming.
If f(x) is not linear or quadratic and/or the
constraints are nonlinear, we have nonlinear
programming.
When equations(*) are included, we have a
constrained optimization problem; otherwise,
it is unconstrained optimization problem.

Chapter 7
by Lale Yurttas, Texas
A&M University
Chapter 7 6
ONE-DIMENSIONAL
UNCONSTRAINED OPTIMIZATION
In multimodal functions,
both local and global
optima can occur. In
almost all cases, we are
interested in finding the
absolute highest or
lowest value of a
function.

Figure 13.1
by Lale Yurttas, Texas
A&M University
Chapter 7 7
How do we distinguish global optimum
from local one?
By graphing to gain insight into the behavior of the
function.
Using randomly generated starting guesses and picking
the largest of the optima as global.
Perturbing the starting point to see if the routine
returns a better point or the same local minimum.
Bracketing methods:
a) Golden search method
b) Quadratic method
Open method:
a) Newtons method
by Lale Yurttas, Texas
A&M University
Chapter 7 8
by Lale Yurttas, Texas
A&M University
Chapter 7
9
1. Start with two guesses [x
l
and x
u
] that bracket the extremum
point of f(x).
2. Calculate the two interior points x1 and x2 according to the
golden ratios:





3. If f(x
1
)>f(x
2
), x
l
= x
2

f(x
1
)<f(x
2
), x
u
= x
1
4. Step 3 is repeated until the extremum is found (within the
specified tolerance)
d x x
d x x
x x d
u
l
l u

2
1
) (
2
1 5
Use the golden section search to find the
maximum of

Within the interval x

= 0 and x
u
= 4. True value
of this function is 1.7757.
by Lale Yurttas, Texas
A&M University
Chapter 7 10
10
) sin( 2 ) (
2
x
x x f
PROCEDURE OF
QUADRATIC METHOD
1. Start with three initial guesses (x
0
, x
1
and x
2
) that
jointly bracket the extremum point of f(x).
2. Calculate the next point x
3
:


3. If f(x
3
) > f(x
1
), x
1
=x
3
, x
0
=x
1
and discard x
0
if f(x
1
) > f(x
3
), x
2
=x
3
.
4. Step 2 is repeated until the extremum is found
(within the specified tolerance)

by Lale Yurttas, Texas
A&M University
Chapter 7 11
) )( ( 2 ) )( ( 2 ) )( ( 2
) )( ( ) )( ( ) )( (
1 0 2 0 2 1 2 1 0
2
1
2
0 2
2
0
2
2 1
2
2
2
1 0
3
x x x f x x x f x x x f
x x x f x x x f x x x f
x

Use quadratic interpolation to approximate the

maximum of

Within initial guesses x
0
= 0, x
1
= 1 and x
2
= 4.
True value of this function is 1.7757.
by Lale Yurttas, Texas
A&M University
Chapter 7 12
10
) sin( 2 ) (
2
x
x x f
PROCEDURE OF
NEWTONS METHOD
1. Start with one initial guesses (x
0
)
2. Calculate the next iteration point:



3. Step 2 is repeated until the extremum is found
(within the specified tolerance)

by Lale Yurttas, Texas
A&M University
Chapter 7 13
) (
) (
1
i
i
i i
x f
x f
x x

Use Newtons method to find the maximum of

With an initial guesses x
0
= 2.5. True value of this
function is 1.7757.
by Lale Yurttas, Texas
A&M University
Chapter 7 14
10
) sin( 2 ) (
2
x
x x f
by Lale Yurttas, Texas
A&M University
Chapter 7 15
CONSTRAINED OPTIMIZATION
LINEAR PROGRAMMING
An optimization approach that deals with
meeting a desired objective such as maximizing
profit or minimizing cost in presence of
constraints such as limited resources
Mathematical functions representing both the
objective and the constraints are linear.
by Lale Yurttas, Texas
A&M University
Chapter 7 16
STANDARD MAXIMIZATION PROBLEM
Basic linear programming problem consists of two
major parts:
The objective function
A set of constraints
For maximization problem, the objective function is
generally expressed as

n n
x c x c x c
2 2 1 1
Z Maximize
c
j
= payoff of each unit of the jth activity that is
undertaken
x
j
= magnitude of the jth activity
Z= total payoff due to the total number of activities
by Lale Yurttas, Texas
A&M University
Chapter 7 17
The constraints can be represented generally as


Where a
ij
=amount of the ith resource that is
consumed for each unit of the jth activity and;
b
i
=amount of the ith resource that is available
With additional constraints:
xi 0 and bi 0

i n in i i
b x a x a x a
2 2 1 1
Characteristics:
1. Objective function is to be maximized
2. All variables are non-negative (x
i
0)
3. Each linear constraint is in the form of to a
non-negative constants (b
i
0)
by Lale Yurttas, Texas
A&M University
Chapter 7 18
by Lale Yurttas, Texas
A&M University
Chapter 15 19
THE SIMPLEX METHOD
Assumes that the optimal solution will be an extreme
point.
The approach must discern whether during problem
solution an extreme point occurs.
To do this, the constraint equations are reformulated
as equalities by introducing slack variables.


by Lale Yurttas, Texas
A&M University
Chapter 15 20
A slack variable measures how much of a constrained resource
is available, e.g.,
7x
1
+ 11 x
2
77
If we define a slack variable S
1
as the amount of raw gas that
is not used for a particular production level (x
1
, x
2
) and add it
to the left side of the constraint, it makes the relationship
exact.
7x
1
+ 11 x
2
+ S
1
= 77
If slack variable is positive, it means that we have some slack
that is we have some surplus that is not being used.
If it is negative, it tells us that we have exceeded the
constraint.
If it is zero, we have exactly met the constraint. We have used
up all the allowable resource.
by Lale Yurttas, Texas
A&M University
Chapter 15 21
Maximize

0 , , , , ,
6
9
80 8 10
77 11 7
175 150
4 3 2 1 2 1
4 2
3 1
2 2 1
1 2 1
2 1

S S S S x x
S x
S x
S x x
S x x
x x Z
by Lale Yurttas, Texas
A&M University
Chapter 15 22
We now have a system of linear algebraic equations.
For even moderately sized problems, the approach
can involve solving a great number of equations. For
m equations and n unknowns, the number of
simultaneous equations to be solved are:

)! ( !
!
m n m
n
C
n
m

by Lale Yurttas, Texas

A&M University
Chapter 15 23
Figure 15.3