Escolar Documentos
Profissional Documentos
Cultura Documentos
OPTIMIZATION
Theory
Algorithms
Software
Applications
Nick Sahinidis
University of Illinois at
Urbana-Champaign
Department of
Chemical Engineering
OPTIMIZATION
in 30 seconds
f
x
optimum
HISTORICAL
DEVELOPMENTS in
OPTIMIZATION
300 B.C: Shortest distance from a
point to a line (Euclid)
1600s: Leibniz/Newton (Calculus)
1847: Gradient methods (Cauchy)
1875: Minimum free energy
principle (Gibbs)
Late 40s: Linear Programming
Dantzig, ...
Linear objective function and constraints
TODAYs
CHALLENGES
Stochastic Optimization
Hedging against uncertainty
Global Optimization
Problems with multiple-local minima
Computational Complexity
Approximation Schemes
Fast (Polynomial Time)
Near-Optimality Proofs
Applications
Supply Chain Management
Process and Product Design
Molecular Design and Analysis
Computational Chemistry
Bioinformatics
Software
General Purpose
Robust; Easy to use
THE MULTIPLE-MINIMA
DIFFICULTY
f
PROCESS
SUPERSTRUCTURE
Mixed-integer (non)linear
programming formulations
(MINLP)
HEAT EXCHANGER
NETWORK SYNTHESIS
H1 @ 300
f3
t1
f1
f7
t2
H1 @ 320
f5
C1 @ 100
F = 10
C1 @ 280
f6
H2 @ 300
f2
f4
t3
t4
f8
H2 @ 340
MOLECULAR
STRUCTURE PREDICTION
E
r ro
kb
ij
ij
ij
(ij)B
o
kijk ijk ijk
(ijk)
kijkl
(ijkl)
Aij
12
(ij)NB rij
k cos nijkl
ijkl
Bij
6
r
ij
r
qiq j
4 r
o ij
(ij)NB
MEDICAL DIAGNOSIS
& PROGNOSIS
Linear Separation of Sets:
A
B
A
B
A
A
B
A
UTILITY-BASED
PORTFOLIO OPTIMIZATION
UTILITY
RISK-AVERSE
RISK-NEUTRAL
RISK-SEEKING
WEALTH
UTILITY
GOAL
WEALTH
Problem Formulation
(P)
min f (x, y)
Objective Function
s.t. g(x, y) 0
Constraints
x Zp
Integrality Restrictions
y Rn
Continuous Variables
Challenges:
Multimodal Objective
111111111111111
000000000000000
000000000000000
111111111111111
000000000000000
111111111111111
000000000000000
111111111111111
000000000000000
111111111111111
000000000000000
111111111111111
000000000000000
111111111111111
000000000000000
111111111111111
000000000000000
111111111111111
000000000000000
111111111111111
Objective
Integrality
Discrete Set
Nonconvex Constraints
Objective
Nonconvexity in
Feasible Space
MOTIVATION FOR
AN ALGORITHM
Diverse Application Areas
Combinatorial Chemistry
Molecular Biology
Finance
Parameter Estimation
Design of Experiments
Engineering Design
Process Synthesis
CAD
Layout Design
Theoretical Challenges
Important Subclasses
Mixed-Integer Linear Programming
Continuous Nonlinear Programming
Nonlinear 0-1 Programming
NP-hard Problem
Murty and Kabadi (1987)
MINLP ALGORITHMS
Branch and Bound
Bound problem over successively refined
partitions
Falk and Soland, 1969
Gupta and Ravindran, 1985
Convexification
Outer-approximate with increasingly tighter
convex programs
Tuy, 1964
Sherali and Adams, 1994
Decomposition
Project out some variables by solving
subproblem
Duran and Grossmann, 1986
Visweswaran and Floudas, 1990
Our approach
Branch and Bound
Convexification
Range reduction
Constraint Propagation
Duality-based
Objective
P
U
R
L
Variable
a. Lower Bounding
Variable
b. Upper Bounding
Objective
P
R
U
L
R2
R1
R1
R2
Fathom
Variable
c. Domain Subdivision
Subdivide
d. Search Tree
Finite sequences
An infinite sequence
branch and bound methods applied to global
optimization problems generate infinite processes,
In contrast, in integer programming these are finite
procedures.
Horst & Tuy (1996)
Factorable Functions
(McCormick, 1976)
Denition: The factorable functions are recursive
compositions of sums and products of functions of
single variables.
Example: f (x, y, z, w) =
exp(xy + z ln w)z 3
x1 = xy
x2 = ln(w)
x5
x3
exp( xy + z ln w) z 3
x2
x1
x4
x7
x3 = zx2
x6
0.5
x4 = x1 + x3
x5 = exp(x4 )
x6 = z 3
x7 = x5 x6
f = x7
COMPUTATIONAL RESULTS
WITH STANDARD
BRANCH-AND-BOUND
Ex.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
*:
Ntot
Nopt
Nmem
3 1
2
0.8
1007 1
200 210
2122* 1 113* 1245*
17 1
5
6.7
1000* 1 1000* 417*
1 1
1
0.3 Ntot Total number of nodes
Nopt Node where optimum found
205 1
37 43
Nmem Max. no. nodes in memory
43 1
8
1
2192* 1 1000* 330* T
CPU sec (SPARC 2)
1 1
1
0.4
81 1
24 19
3 1
2
0.6
7 2
3
1.3
7 3
3
3.4
15 8
5
3.4
2323* 1 348* 1211*
1000* 1 1001* 166*
1 1
1
0.5
85 1
14 11.4
3162* 1 1001* 778*
7 1
4
1.2
9 1
4
1.2
75 6
13 11.7
7 3
2
1.5
17 9
9
2.9
Did not converge within limits of
T 1200 (=20 min), and N mem 1000 nodes.
CHALLENGES
Standard Global Optimization
Converges Slowly
Algorithmic Issues
Build tight relaxations
Reduce domain by automatically
identifying sub-optimal regions
Develop finite instead of merely
convergent algorithms
Implementation Issues
Existing NLP solves are often unreliable
Numerical instabilities are common with
nonlinear terms
Ease of use
Capability to sole problems of industrialscale magnitude
RANGE REDUCTION
Relaxed Value Function
x
xL
xU
PROBING
Q. What if a variable does not go to
a bound?
A. Use probing: temporarily fix
variable at a bound.
zU
U
x
x*
xU
IMPLIED RESULTS
Monotone Complementarity Bounds
for Convex Programs
Mangasarian and McLinden
Mathematical Programming, 1985
Linearity-Based Tigthening
Andersen and Andersen
Mathematical Programming, 1995
SEPARABLE
CONCAVE MINIMIZATION
min f (x) = f k (x k )
k
f k (xk ) concave, k.
Ax b
L
x xx
LOWER BOUNDING
f(x)
g(x)
xL
[xL,xU]
xU
Locating partition:
Typically, at the midpoint (exhaustiveness)
When possible, at the best currently known
solution
Makes underestimators exact at the
candidate solutions
x2
x*
x1L
x1U
x1
d
xA
PROOF OF FINITENESS
g(x)
f(x)
TIGHT RELAXATIONS
f (x )
Concave over-estimator
Convex under-estimator
x
f (x )
Concave envelope
Convex envelope
x
f (x )
x
Convex/concave envelopes
often finitely generated
g(x)
m
f(x)
n
p
o
(0,0)
3x
(2, 4)
x2
f (x) =
x
(0, 0)
(1, 1)
x
x
3x 2
x [0, 1]
x [1, 2]
i g(xi ) |
i
i = 1, i (0, 1), xi X
i xi = x;
i
for all x X
Equivalence
g(x) = xy
x
x2
xy
Convex Envelope
yU
x/y
xL
xU
yL
neighborhood of (x0 , y 0 )
Example II: X is
yU
x2 y 2
xL
xU
yL
s, x g(x) ,
x/y
xL
xL x xU
xU
yL
yL y yU
z x/y
cross-multiplying
U
yL y yU
xL /y U z xU /y L
x xx
L
y yy
L
zy x
xL x xU
Relaxing
Simplifying
zy (z xL /y U )(y y U ) x
zy (z xU /y L )(y y L ) x
yL y yU
yL y yU
xL x xU
xL x xU
0.5
2
2x4
2y4
x
4 2
Ratio: x/y
xU x
xU xL
yp (z zp ) y(z zp ) x
x xL
4 2
yp max
yp min
yU
x/y - Envelope
z zp , zp 0
xU x
xU xL
xU x
xU
xL
,y y
xU xL
L
U xx
, y yL
xU xL
x xL
xU xL
y
x
4
x/y - Factorable
2x4
2y4
x/y
xL
xU
yL
1
(xU
xL )(y U
yL)
yL
(x x)(y y) +
U
xL
yU
(xU x)(y y L ) +
xU
xU
L
U
(x x )(y y) + U (x xL )(y y L )
L
y
y
{xL , y L } {xL , y U }
{xU , y L } {xU , y U }
xy + xyL + xyU
yL yU
min y U x xL y + xL y L , y L x xU y + xU y U
Outer Approximation
Motivation:
Convex Programming Solvers are not robust
Linear programs can be solved eciently
Outer-Approximation:
Convex Functions are underestimated by tangent lines
(x)
xL
xU
Quadratic Convergence
with Projective Error Rule
Theorem:
Let (xj ) be a convex function over [xlj , xuj ] and
p the desired projective approximation error
Outer-approximate (xj ) at the end-points
At every iteration of the Sandwich Algorithm
construct an underestimator at the point that
maximizes the projective error of function with
current outer-approximation.
l
Let k = (xuj xlj )(xu
j xj )/
0
k4
N (k) =
k2 ,
k>4
iterations
USER MODULE
Range Reduction
Relaxation
Solver Links
Local Search
I/O Handler
Range Reduction
Debugging Facilities
Heuristics
f (x, y)
a gi (x, y) b
i = 1, . . . , m
U
xL
x
j
j
j
j = 1, . . . , n
yjL yj yjU
j = 1, . . . , p
xj real
yj integer
Form of fi (x, y) and gi (x, y):
f and g are recursive sums, products and ratios of
the following terms:
exponential, exp()
logarithmic, log()
monomial, ()a
Example: (Factorable Constraint)
1
x2 y 0.3 z
p2
+ exp
x2 p
y
xy 100
BARON ACCESS
US EDU
23%
Others
34%
US COM
17%
Europe
26%
COMPUTATIONAL RESULTS
BRANCH-AND-BOUND
BRANCH-AND-REDUCE
No probing
Ex.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
*:
Ntot
Nopt
Nmem
With Probing
3 1
2
0.8 1 1 1 0.5 1
1007 1
200 210
1 1 1 0.2 1
2122* 1 113* 1245* 31 1 7 20
9
17 1
5
6.7 3 1 2 0.4 1
1000* 1 1000* 417*
5 1 3 1.5 5
1 1
1
0.3 1 1 1 0.3 1
205 1
37 43
25 1 8 5.4 7
43 1
8 10
1 1 1 0.8 1
2192* 1 1000* 330* 19 1 8 5.4 13
1 1
1
0.4 1 1 1 0.4 1
81 1
24 19
3 1 2 0.6 1
3 1
2
0.6 1 1 1 0.2 1
7 2
3
1.3 3 1 2 0.7 1
7 3
3
3.4 7 3 3 2.7 3
15 8
5
3.4 1 1 1 0.3 1
2323* 1 348* 1211*
1 1 1 2.2 1
1000* 1 1001* 166*
1 1 1 3.7 1
1 1
1
0.5 1 1 1 0.5 1
85 1
14 11.4 9 1 4 1.8 1
3162* 1 1001* 778* 47 1 12 16.7 23
7 1
4
1.2 1 1 1 0.5 1
9 1
4
1.2 3 1 2 0.4 3
75 6
13 11.7 47 1 9 6.5 7
7 3
2
1.5 3 1 2 0.5 3
17 9
9
2.9 5 1 3 0.8 5
Did not converge within limits of
T 1200 (=20 min), and N mem 1000 nodes.
COMPUTER: SUN Sparc Station 2
1
1
1
1
1
1
1
1
1
1
1
1
1
3
1
1
1
1
1
1
1
1
1
1
1
1 0.7
1 0.3
5 48
1 0.3
3 2.4
1 0.3
2 5.8
1 0.8
4 7
1 0.4
1 0.7
1 0.2
1 0.7
2 3
1 0.3
1 2.4
1 4
1 0.6
1 1.4
5 15.4
1 0.5
2 0.5
4 5
2 0.6
3 1
m
20
20
20
20
20
20
20
20
20
20
40
40
40
40
40
50
50
50
50
50
n
25
25
25
25
25
50
50
50
50
50
25
25
25
25
25
100
100
100
100
100
k
0
50
100
200
400
0
50
100
200
400
0
50
100
200
400
0
50
100
200
400
GOP (1993)
E=1%
(relative)
P&R (1990)
E=0.1%
(relative)
HP 730
CRAY 2
(parallel)
min
avg max
1
2
4
1
1
1
1
2
3
2
7
17
7
14
32
3
6
13
1
2
3
2
5
14
4
9
28
20
32
45
avg
0.5
2
17
33
82
0.6
17
47
109
std dev
0.01
2
20
28
58
0.01
31
49
80
0.5
1
3
25
0.02
0.6
4
26
BARON (1996)
=.000001
(absolute)
IBM RS/6000
Power PC
min avg
max
0.3
0.4
0.5
1
1
1
1
2
3
2
4
6
4
10
16
1
1
1
2
2.5
3
2
4
7
4
8
19
11
20
48
0.3
0.4
0.4
1
1
1
1
2
3
2
4
5
6
15
22
6
7
14
8
12
18
9
17
27
14
65
160
131
345
663
COMPUTATIONAL
RESULTS for
WDBC DATA
Wisconsin Diagnostic Breast
Cancer data:
ftp://ftp.cs.wisc.edu/math-prog/cpodataset/machine-learn/cancer1
2 classes: benign & malignant
9 attributes: clump thickness, uniformity
of cell size, uniformity of cell shape,
marginal adhesion, single epithelial cell
size, bare nuclei, bland chromatin,
normal nucleoli, mitoses
353 instances: 188 benign, 165 malignant
T raining Set
Size
150
176
235
353
BLP Size
(m,n,t)
(300,152,66)
(352,220,100)
(470,276,128)
(706,396,165)
%-Error in
T raining Set
0
0
1.7
2.5 **
%-Error in
T esting Set*
4
3
2.5
2.5 **
f21
3% S
x11
Blend X
$6
1% S
x21
Pool
2.5% S
$9
X 100
$16
x12
f12
2% S
$10
x22
cost
min
s.t.
Blend Y
X-revenue
1.5% S
$15
Y 200
Y -revenue
3f11 + f21
x11 + x12
qx11 + 2x21
x11 + x21
qx12 + 2x22
x12 + x22
Mass balance
2.5
Quality Requirements
1.5
Demands
Foulds 92
Computer
Linpack
Ben-Tal 94
GOP 96
BARON 99
BARON 00
CDC 4340
HP9000/730
RS6000/43P
RS6000/43P
> 3.5
49
59.9
59.9
**
106
106
Tolerance
Problem
Ntot
Ttot
Ntot
0.7
Haverly 2
Haverly 3
Haverly 1
Ttot
Ntot
Ttot
Ntot
Ttot
Ntot
Ttot
12
0.22
0.09
0.09
12
0.21
0.09
0.13
14
0.26
0.13
0.07
Foulds 2
3.0
0.10
0.04
Foulds 3
10.5
2.33
1.70
Foulds 4
25
125.0
2.59
0.38
Foulds 5
125
163.6
0.86
0.10
Ben-Tal 4
25
0.95
0.11
0.13
Ben-Tal 5
283
41
5.80
1.12
1.22
Example 1
6174
425
15
12.30
Example 2
10743
1115
19
8.56
Example 3
79944
19314
13
18.30
Example 4
1980
182
5.12
CH3
Cl
CH2
N
Cl
CH2
CH
Cl
Cl
CH2
Combinatorial Choice
F
Freon (CCl2F2)
F
Cl
Cl
Property Prediction
Satises
Property
Requirements?
Compressor
Tevp
Evaporator
max
s.t.
Hve
Cpla
Hve 18.4
Cpla 32.2
Pvpe 1.4
Property Prediction Constraints
Structure Feasibility Constraints
Nonnegativity Constraints
Integrality Requirements
Cyclic Groups
Halogen Groups
Oxygen Groups
Nitrogen Groups
Sulfur Groups
CH3
CH2 r
OH
NH2
SH
CH2
r
r
> CH r
Cl
> NH
> CH
> CH r
Br
I
O r
r
r
> NH
>C<
r
r
> C < rr
= CH2
> C < rr
r
r
> C < rr
CHO
= CH r
COOH
CN
NO2
= CH
=C<
> CO
> N
> CO
= N
=C=
= C < rr
COO
CH
= C <r
=O
S r
= N r
= C < rr
Number of Groups = 44
Maximum Selection Size = 15
Candidates = 39, 895, 566, 894, 524
Tb = 198.2 +
ni Tbi
= 15.2518
i=1
Tb
Tc =
N
i=1
0.584 + 0.965
N
i=1
ni Tci (
ni Tci )2
Pc =
(0.113 + 0.0032
N
i=1
N
i=1
n i ai
ni Pci
1
4.1868
i=1
17.11 + 25.2
ni Cp0ci 3.91 10
2
Tavg
Hvb = 15.3 +
ni Cp0di + 2.06 10
3
Tavg
Tavgr
Tcndr
Tevpr
+ 0.25
1.742
1 Tavgr
ni Hvbi
Hve = Hvb
i=1
Tbr =
Tavgr
1 Tavgr
i=1
i=1
N
(1 Tavgr )1/3
0.45
13.4721ln(Tbr ) + 0.43577Tbr6
Cpla =
)2
Tbr
ni Cp0ai 37.93 +
Cp0a =
15.6875
Tb
h=
Tc
Tavg
=
Tc
Tcnd
=
Tc
Tevp
=
Tc
= 5.97214 ln
0.169347Tbr6
1 Tevp /Tc
0.38
1 Tb /Tc
Tbr ln(Pc /1.013)
1 Tbr
G = 0.4835 + 0.4605h
k=
Pc
1.013
6.09648
Tbr
+ 1.28862ln(Tbr )
h/G (1 + Tbr )
ni 2
ni 1 if
i=1
iDSR
YA
ni Nmax YA A
ni = 2ZB ;
iA
YC
iB
ni Nmax YC C
iDR
YA + YC 1 YM YA + YC
ni = 2ZT
iT
ni Nmax YR R
ni (2 bi ) = 2m
iR
N
ni bi 2(
i=1
N
i
N
ni 1);
i=1
N
ni bi
ni nj (bj 1) + 2
ni
i=1
iO,
ni 1 if
iSD
ni 1 and
iS/D
ni 1 if
iST
iSSR
ni 1 if
single-bonded
iD/S
iS/T
ni 1 if
ni 1
ni 1 and
j = 1, . . . , N
i=1
ni 1
ni
i=1
iSDR
ni = 2ZSR
iSR
ni = 2ZDR
iM
i=1
ni = 2ZS ;
iS
iD
ni Nmax YM M
3YR
ni 1
iSR/D
ni = 2ZD
iC
YM
ni 1 and
iD/SR
ni Sai if
iH
ni
ni 1
iO,
double-bonded
ni = 0
iH
ni Dai if
iH
ni = 0
iH
iT /S
ni 1 and
iS/SR
ni 1 and
iS/DR
ni 1
iSR/S
ni
iO,
triple-bonded
iH
ni = 0
iH
ni 1
iDR/S
ni Tai if
iH
ni 2(
iO
ni 1)
iH
Molecular Structures
Molecular Structure
Hve
Cpla
ClNO
(Cl)(N =)(= O)
1.5971
IF
(I)(F)
O3
(r O r )3
Molecular Structure
Hve
Cpla
CH2 FN
1.1177
1.5297
ClF
(Cl)(F)
1.0480
1.5219
CClFO
(O =)(= C <)(Cl)(F)
1.0179
1.4855
1.3464
FNO
(F)(N =)(= O)
0.9893
ClFO
(Cl)(O)(F)
1.2880
0.9822
0.9342
1.2767
C3 H4
0.9283
1.2594
C2 F2
( C)2 (F)2
0.9229
CHClO
(Cl)(CH =)(= O)
1.1804
C3 H6 O
(CH3 )2 (= C <)(= O)
0.8978
FSH
(F)(SH)
1.1697
C3 H3 FO
0.8868
1.1653
1.1574
C2 H6
1.1219
CHFO3
C3 H3 F3
CH3 Cl
(CH3 )(Cl)
1
2
hour
(CH3 )2
0.8632
0.5977
Reduce
Isolate
Convexify
x*
BRANCH-AND-REDUCE
Engineering
design
Supply chain
operations
Molecular
design