Robust Optimization &

Machine Learning
1. Optimization
Models
Short Course
Robust Optimization and Machine Learning What is Optimization?
Definition
Examples
Nomenclature
Other standard forms
Extensions

Convexity
Global vs. local optima

Lecture 1: Optimization Models Convex problems

Software
Non-convex problems
Non-convex problems

References
Laurent El Ghaoui

EECS and IEOR Departments

UC Berkeley

Spring seminar TRANSP-OR, Zinal, Jan. 16-19, 2012

 Robust Optimization &
Outline Machine Learning
1. Optimization
Models

What is Optimization?
Definition
Examples
What is Optimization? Nomenclature
Definition Other standard forms
Examples Extensions

Nomenclature Convexity
Other standard forms Global vs. local optima
Extensions Convex problems
Software
Non-convex problems
Non-convex problems
The Role of Convexity References
Global vs. local optima
Convex problems
Software
Non-convex problems
Non-convex problems

References
 Robust Optimization &
Outline Machine Learning
1. Optimization
Models

What is Optimization?
Definition
Examples
What is Optimization? Nomenclature
Definition Other standard forms
Examples Extensions

Nomenclature Convexity
Other standard forms Global vs. local optima
Extensions Convex problems
Software
Non-convex problems
Non-convex problems
The Role of Convexity References
Global vs. local optima
Convex problems
Software
Non-convex problems
Non-convex problems

References
 Robust Optimization &
Optimization problem Machine Learning
1. Optimization
A standard form Models

An optimization problem is a problem of the form

What is Optimization?

p∗ := min f0 (x) subject to fi (x) ≤ 0, i = 1, . . . , m, Definition

Examples
x
Nomenclature
Other standard forms
where Extensions

I x ∈ Rn is the decision variable ; Convexity

Global vs. local optima
I f0 : Rn → R is the objective (or, cost ) function; Convex problems
Software
n
I fi : R → R, i = 1, . . . , m represent the constraints ; Non-convex problems
Non-convex problems
∗
I p is the optimal value . References

Often the above is referred to as a “mathematical program” (for

historical reasons).
 Robust Optimization &
Example Machine Learning
1. Optimization
Least-squares regression Models

What is Optimization?
Definition
Examples
Nomenclature
Other standard forms
Extensions

Convexity
Global vs. local optima
Convex problems
Software
Non-convex problems
Non-convex problems

References

min kX T w − yk2
w

where
I X = [x1 , . . . , xm ] is a n × m matrix of data points (xi ∈ Rn );

I y is a response vector;
I k · k2 is the l2 (i.e., Euclidean) norm.
I Many variants (with e.g., constraints) exist (more on this later).
I Perhaps the most popular / useful optimization problem.
 Robust Optimization &
Example Machine Learning
1. Optimization
Linear classification Models

What is Optimization?
Definition
Examples
Nomenclature
Other standard forms
Extensions

Convexity
Global vs. local optima
Convex problems
Software
Non-convex problems
Non-convex problems

References

m
X
min max(0, 1 − yi (w T xi + b))
w,b
i=1

where
I X = [x1 , . . . , xm ] is a n × m matrix of data points (xi ∈ Rn );

I y ∈ {−1, 1} is a binary response vector;

I Many variants (with e.g., constraints) exist (more on this later).
I Very useful for classifying data (e.g., text documents).
 Robust Optimization &
Nomenclature Machine Learning
1. Optimization
A toy optimization problem Models

min 0.9x12 − 0.4x1 x2 − 0.6x22 − 6.4x1 − 0.8x2 What is Optimization?

x Definition

s.t. −1 ≤ x1 ≤ 2, 0 ≤ x2 ≤ 3. Examples
Nomenclature
Other standard forms
Extensions

Convexity
I Feasible set in light blue. Global vs. local optima
Convex problems
I 0.1- suboptimal set in darker Software

blue. Non-convex problems

Non-convex problems
I Unconstrained minimizer : x0 ; References
optimal point: x ∗ .
I Level sets of objective function in
red lines.
I A sub-level set in red fill.
 Robust Optimization &
Other standard forms Machine Learning
1. Optimization
Models
Equality constraints. We may single out equality constraints, if any:

min f0 (x) subject to hi (x) = 0, i = 1, . . . , p, What is Optimization?

x Definition
fi (x) ≤ 0, i = 1, . . . , m, Examples
Nomenclature
Other standard forms
where hi ’s are given. Of course, we may reduce the above problem to Extensions

the standard form above, representing each equality constraint by a Convexity

pair of inequalities. Global vs. local optima
Convex problems
Software
Non-convex problems
Non-convex problems

References

Abstract forms. Sometimes, the constraints are described abstractly

via a set condition, of the form x ∈ X for some subset X of Rn . The
corresponding notation is
min f0 (x).
x∈X
 Robust Optimization &
Minimization vs. maximization Machine Learning
1. Optimization
Models
Some problems come in the form of maximization problems. Such
problems are readily cast in standard form via the expression
What is Optimization?
Definition
max f0 (x) = − min : g0 (x), Examples
x∈X x∈X Nomenclature
Other standard forms

where g0 := −f0 . Extensions

Convexity
Global vs. local optima
Convex problems
Software
Non-convex problems
Non-convex problems

References

I Minimization problems correspond to loss, cost or risk

minimization.
I Maximization problems typically correspond to utility or return
(e.g., on investment) maximization.
 Robust Optimization &
Penalization Machine Learning
1. Optimization
Models
A trade-off between two objecgives is commonly accomplished via a
penalized problem:
What is Optimization?
max f (x) + λg(x), Definition
x Examples
Nomenclature
where f and g represent loss and risk functions, and λ > 0 is a Other standard forms
risk-aversion parameter. Extensions

Convexity
Global vs. local optima
Convex problems
Software
Non-convex problems
Non-convex problems

References

Example: penalized least-squares

min kX T w − y k22 + λkwk22

w

Here, the risk term kwk22 controls the variance associated with noise
in X .
 Robust Optimization &
Robust optimization Machine Learning
1. Optimization
Definition Models

In many instances the problem data is not known exactly. Assume that
What is Optimization?
the functions fi in the original problem also depend on an “uncertainty” Definition
vector u that is unknown, but bounded: u ∈ U , with the set U given. Examples
Nomenclature
Other standard forms
Extensions

Convexity
Global vs. local optima
Convex problems
Software
Non-convex problems

Robust counterpart: Non-convex problems

References

min max f0 (x, u)

x u∈U
subject to ∀ u ∈ U , fi (x, u) ≤ 0, i = 1, . . . , m.

I Robust counterparts are sometimes tractable.

I If not, systematic procedures exist to generate approximations.
 Robust Optimization &
Robust optimization Machine Learning
1. Optimization
Geometry Models

Given a ∈ Rn , b ∈ R, consider the constraint in x ∈ Rn

What is Optimization?

(a + u)T x ≤ b, Definition
Examples
Nomenclature

with u’s components are only known within a given set U. The robust Other standard forms
Extensions
counterpart is: Convexity
∀ u ∈ U : (a + u)T x ≤ b. Global vs. local optima
Convex problems
Software
Non-convex problems
Non-convex problems

References

Robust counterpart when A is a box (left panel) and a sphere (right panel).
 Robust Optimization &
Stochastic optimization Machine Learning
1. Optimization
Definition Models

In stochastic programming, the uncertainty is described by a random

What is Optimization?
variable, with known distribution. Definition
Examples
Nomenclature
Other standard forms
Extensions

Two-stage stochastic linear program with recourse: Convexity

Global vs. local optima

min aT x + f (x) : f (x) = E[ min c(w)T y ]. Convex problems

Software
x∈X w y ∈Y(x,w)
Non-convex problems
Non-convex problems

References
I x-variables correspond to decisions taken now.
I y -variables correspond to decisions taken when uncertainty w is
revealed.

I Stochastic problems are usually very hard.

I Most known approaches are very expensive to solve.
 Robust Optimization &
Outline Machine Learning
1. Optimization
Models

What is Optimization?
Definition
Examples
What is Optimization? Nomenclature
Definition Other standard forms
Examples Extensions

Nomenclature Convexity
Other standard forms Global vs. local optima
Extensions Convex problems
Software
Non-convex problems
Non-convex problems
The Role of Convexity References
Global vs. local optima
Convex problems
Software
Non-convex problems
Non-convex problems

References
 Robust Optimization &
Global vs. local minima Machine Learning
1. Optimization
The curse of optimization Models

I Point in red is globally optimal What is Optimization?

Definition
(optimal for short). Examples
Nomenclature
I Point in green is only locally Other standard forms

optimal. Extensions

Convexity
I In many applications, we are Global vs. local optima

interested in global minima. Convex problems

Software
Non-convex problems
Non-convex problems

References

Curse of optimization
Optimization algorithms for general problems can be trapped in local
minima.
 Robust Optimization &
Convex function Machine Learning
1. Optimization
Definition Models

A function f : Rn → R is convex if it satisfies the condition

What is Optimization?

∀ x, y ∈ Rn , λ ∈ [0, 1] : f (λx + (1 − λ)y ) ≤ λf (x) + (1 − λ)f (y ) Definition

Examples
Nomenclature

Geometrically, the graph of the function is “bowl-shaped”. Other standard forms

Extensions

Convexity
Global vs. local optima
Convex problems
Software
Non-convex problems
Non-convex problems

References

Convex function. Non-convex function.

 Robust Optimization &
Convexity and local minima Machine Learning
1. Optimization
Models
When trying to minimize convex functions, specialized algorithms will
always converge to a global minimum, irrespective of the starting
What is Optimization?
point, provided some (weak) assumptions on the function hold. Definition
Examples
Nomenclature
Other standard forms
Extensions

Convexity
Global vs. local optima
Convex problems
Software
Non-convex problems
Non-convex problems

References

The Newton algorithm.

 Robust Optimization &
Convex optimization Machine Learning
1. Optimization
Definition Models

The problem in standard form

What is Optimization?

p∗ := min f0 (x) subject to fi (x) ≤ 0, i = 1, . . . , m, Definition

Examples
x
Nomenclature
Other standard forms
is convex if the functions f0 , . . . , fm are all convex. Extensions

Convexity
Global vs. local optima
Convex problems
Software
Non-convex problems
Non-convex problems

References

Examples:
I Linear programming (f0 , . . . , fm affine).
I Quadratic programming (f0 convex quadratic, f1 , . . . , fm affine).
I Second-order cone programming (f0 linear, fi ’s of the form
kAi x + bi k2 + ciT x + di , for appropriate data Ai , bi , ci , di ).
 Robust Optimization &
Software for convex optimization Machine Learning
1. Optimization
Models
I Free (if you have matlab): CVX [3], Yalmip, Mosek’s student
version [1].
What is Optimization?
I Really free: [4] (in development). Definition
Examples
I Commercial: Mosek, CPLEX, etc. Nomenclature
Other standard forms
Extensions

Convexity
Global vs. local optima
Convex problems
Software
Non-convex problems
Non-convex problems

References
 Robust Optimization &
Non-convex problems Machine Learning
1. Optimization
Examples Models

I Boolean/integer optimization: some variables are constrained to

What is Optimization?
be Boolean or integers. Convex optimization can be used for Definition
getting (sometimes) good approximations. Examples
Nomenclature
I Cardinality-constrained problems: we seek to bound the number Other standard forms
Extensions
of non-zero elements in a vector variable. Convex optimization
Convexity
can be used for getting good approximations. Global vs. local optima
Convex problems
I Non-linear programming: usually non-convex problems with Software

differentiable objective and functions. Algorithms provide only Non-convex problems

Non-convex problems
local minima. References

Not all non-convex problems are hard! e.g., low-rank approximation

problem.
 Robust Optimization &
Outline Machine Learning
1. Optimization
Models

What is Optimization?
Definition
Examples
What is Optimization? Nomenclature
Definition Other standard forms
Examples Extensions

Nomenclature Convexity
Other standard forms Global vs. local optima
Extensions Convex problems
Software
Non-convex problems
Non-convex problems
The Role of Convexity References
Global vs. local optima
Convex problems
Software
Non-convex problems
Non-convex problems

References
 Robust Optimization &
References Machine Learning
1. Optimization
Models

E. D. Andersen and K. D. Andersen. What is Optimization?

Definition
The MOSEK optimization package. Examples
Nomenclature
Other standard forms
A. Bental, L. El Ghaoui, and A. Nemirovski. Extensions

Robust Optimization. Convexity

Princeton Series in Applied Mathematics. Princeton University Press, Global vs. local optima
Convex problems
October 2009. Software
Non-convex problems
S. Boyd and M. Grant. Non-convex problems

The CVX optimization package, 2010. References

S. Boyd and T. Tinoco.

The CVXPY optimization package, 2010.

S. Boyd and L. Vandenberghe.

Convex Optimization.
Cambridge University Press, 2004.

G. Cornuejols and R. Tütüncü.

Optimization Methods in Finance.
Cambridge University Press, 2007.