Global Optimization

Project 3: Mode-Pursuing Sampling Method (MPS)

Miguel Daz-Rodrguez
July 14, 2014

Objective

Thoroughly study the Mode-Pursuing Sampling Method.

Mode-Pursuing Sampling Method

Engineering design search for the global optimal of a function with respect to the design variables.
Global optimization algorithm such as simulated annealing (SA) or genetic algorithms (GA) can be
use for finding the optimal. However, engineering design problems often rely on computationally expensive objective functions. For instance, a design project involving a finite element solution (FEM).
SA and GA algorithms expend huge time in finding the global minimum for this kind of problems
because the algorithm are based on intensive evaluation of the objective function. Metamodeling
approaches have become popular for dealing with such problems. Sampling data from the objective
function allow to fit a surrogate model, which is computational less expensive than the original model;
then, the optimization is performed over the surrogate model. Thus, metamodeling approaches require an accurate surrogate model such that its global optimum matches with the optimal solution
of the original model. One such of method is the Mode-Persuing Sampling Methods (MPS) which is
the algorithm discussed in this report.
MPS is an algorithm relying on discriminative sampling that provides an intelligent mechanism
to use the information from past iterations in order to lead the search toward the global optima.
The method applies to continuous variables [7], and also to discrete variables [6]. The basic of the
algorithm is summarized in Figure 1.

Conceptual Illustration of the MPS

For understanding of the MPS, each step of the algorithm is illustrated by solving the function
presented in [5]. That is,
f (x) = 2x31 32x1 + 1

(1)

where the minimum is located at f (2.31) = 48.3.

Figure 2 shows f (x) within the interval [5 3]. The initial random sampling [(n + 1) (n + 2) /2 + 1 n] =
3, being n the dimension of x. The function f (x) is calculated for each sample points, which are
called expensive points. Figure 2 also plots the initial sample points (red dot).
The initial samples allow fitting a linear spline interpolating function. Then, n0 = 104 uniformly
distributed points are generated using the spline function, which are called cheap points. Figure 5
shows the objective function and the sampling points from the linear spline function.
In order to pursue the mode of the function, f (x) is sorted starting from its minimum value to its
maximum value. The function values can be arrange in K contours {K1 , K2 , . . . , Kn }, when the first

Start
Initial random sampling [(n+1)(n+2)/2+1-np] points

Mode-pursuing sampling of np points

Find [(n+1)(n+2)/2+1] points around

the current mode; obtain the sub-region

Perform quadratic fitting in the sub-region

No

| R12 ! 1 |< ! R

Update the speed

control factor r

Yes
Randomly generate [n/2] points;
Refit the function in the sub-region;
Add the points to the point set.
No
2
| R New
! 1 |< ! R

and Diff < ! d

Yes
Perform local optimization
No

The min in the sub-region?

Obtain its real function

value and add it to the
point set.

Yes
Exit

Figure
1: 4Flowchart
Mode-Pursuing
Sampling
Figure
Flowchart ofof
thethe
proposed
global optimization
method.Method [7]
100

80

60

40

20
f(x)

28
0

20

40

60

80
3

1
x

Figure 2: Objective function for the illustrative example, blue line f (x), red dots represent sampling
points (expensive points).

100

80

60

f(x), spline

40

20

20

40

60

80
3

1
x

Figure 3: Example case 1, blue line f (x), red dots represent cheap points (f (x) interpolation function).
contour contains lower values of the objective function and the contour Kn the maximum values. The
function f (x) is used to find the cumulative distribution function by computing the cumulative sum
of g (x) = c0 f (x), where co = max(f (x)) (G (x) = cumsum(g (x)). The probability of selecting
1/b
points near to the function mode can be increased by modifying G (x) using G (x) .
Figure 4 shows the curve representing f (x), g (x) (For the case of N/K=1, which means that only
one contour E1 is considered), G (x) the cumulative distribution function. The figure shows that,
due to the fact that the G curve is relatively flat between 7000 to 10000, this points have lower
chances to be selected for further sampling than other sampling points. However, points in that
area always have probabilities larger than zero. On the other hand, in order to better control the
sampling process [4] introduce a speed control factor. Figure 4 also shows the speed G function. It
can be noted that samples from 5000 to 10000 have lower chances to be selected for further sampling
than others. Thus, the probability for further sampling near to the minimum value increased with
the speed G function.
The MPS progress by generating more sample points that will be around the current minimum point
increasing the chances of finding a better minimum. In order to find the global minimum a quadratic
model is fitted to some of the expensive points that are in a sufficiently small neighbourhood around
the current minimum.
The quadratic model can be expressed as follows,
y = 0 +

n
X
i=1

i xi +

n
X

ii x2i +

n
XX

i=1

ij xi xj

(2)

i<j j=1

where i , ii , ij , stand for the regression coefficients, x is the vector of design variables, and y the
response. The above equation is the standard equation for response surface methodology (RSM) [5].
The models goodness fitness can be assessed by the R2 coefficient. That is,
n
P
i=1
R2 = P
n

(
y y)

(3)
2

(yi y)

i=1

where yi are the fitted function values, yi are the function values at the sample points and y is the
3

f(x)

60
40
20
0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

1000

2000

3000

4000

5000
6000
Sample points

7000

8000

9000

10000

g(x)

60
40
20
0

G(x)

1
0.5
0

speed G(x)

1
0.5
0

Figure 4: Example of MPS function F1.

mean of yi . In general, R2 gets values between [0 1]. The closer the R2 value to 1, the better the
modeling accuracy.
The quadratic model is fitted considering the expensive points of the sub-region ([(n + 1)(n + 2)/2 + 1]),
and R2 value is computed. If 1 R2 < R , where R is a user define threshold value, then generate
n/2 expensive points within the sub-region. After that the model is fitted again using all points in the
2
2
sub-region and compute a new the Rnew
. Finally, If 1 Rnew
< R , then perform local optimization

to find the optimum x .

The process ends when x lies in the sub-region, if not, this point is added to the original set of
expensive points, and the process continue to the next iteration.
Figure 5 shows the quadratic model fit around the neighborhood points for the first iteration, while
Figure 6 shows the second and third iteration. Figure8(c) presents a detail view of the fitted model
showing that the quadratic model matches with the objective function in the neighborhood of the
solution (R2 > 0.999).

Implementation of the MPS in Matlab

The MPS was implemented in Matlab. A briefly description of the method explaining relevant details
for its implementation is presented below.
Step 1: Generate m initial points x1 , x2 , . . . , xm that are randomly generated on S(f). This parameter is usually small. The objective function is computed over these m points, they are called
expensive points (in terms of computational burden).
1
2

%STEP 1
% initial sampling

100

80

60

40

f(x)

20

20

40

60

80
3

1
x

Figure 5: Example case 1, blue line f (x), red line represent second order polynomial, black dots
represent sampling expensive points.

y = sampling(nv,nInit,xlv,xuv);

4
5
6

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

% call the objective function and calculate the objective function's value
fy=objfun(y);

function [x] = sampling(nv,ns,lb,ub)

sample='y';
x=[];
while sample6='n'
y = rand(nv,ns);
for i=1:ns
y(:,i)=(y(:,i)'.*(ublb)+lb)';
x=[x,y(:,i)];
[m,n]=size(x);
if(n==ns)
sample='n';
break;
end
end
end

Step 2: Use the m function values f (x1 ) , f (x2 ) , . . . , f (xm ) to fit a linear spline function.
=
f (x)

m
X

i ||x xi ||

i=1

= f (xi ), i = 1, 2, . . . , m where |||| stands for the Euclidean norm.

such that f (x)
1
2
3
4

%STEP 2
A = zeros(nupdate);
for i = 1:nupdate
A(i,:) = sum(abs(repmat(y(:,i),1,nupdate)y),1); % ...
f(x)=a1 | xx1 | + a2 | xx2 | + ...+an | xxn| , ...
a(a1,a2,...an)*A=f(x)(f(x1),f(x2),...f(xn))

(4)

80

60

60

40

40

20

20
f(x)

100

80

f(x)

100

20

20

40

40

60

60

80
3

1
x

80
3

(a)

1
x

(b)

20

25

30

35

f(x)

40

45

50
x=2.319
f(x)=48.267

55

60

0.5

1.5

2.5

3.5

(c)

Figure 6: Evolution of the fitting model in the aplication of MPS for study case: (a) Second Iteration;
(b) Third Iteration; (c) Zoom close to the minimum for the Third Iteration

5
6

end
coef = A\fy'

Step 3: Define g (x) = c0 f (x), where c0 is any constant such that c0 f (x), for all x in S(f ).
Since g(x) is nonnegative on S(f ), it can be viewed as a PDF, up to a normalizing constant,
whose modes are located at those xi s where the function values are the lowest among f (xi ).
Then, the sampling algorithm provided by [4] can be applied to generate a random sample
xm+1 , xm+2 , . . . , x2m from S(f ) according to g (x). These sample points have the tendency to
concentrate about the minimum of f (x ).
1
2
3
4
5

% sampling n0 points
x=sampling(nv,n0,xlv,xuv);
fx = zeros(n0,1);
for i = 1:n0
fx(i) = sum(abs(repmat(x(:,i),1,nupdate)y),1)*coef;
points on the model
end

% make the n0 ...

The function must be positive, f (x) > 0 in order to build g(x) function.
1
2
3

if (min(fx) <0)
fx=fxmin(fx);
end

% so that function is positive

Then, the PDF g (x) is normalized.

1
2
3
4

[fx id] = sort(fx);

x = x(:,id);
gx=max(fx)fx;
gx=cumsum(gx)/sum(gx);

%PDF function

Generating a new set of points approaching towards the global minimum.

1
2
3
4
5

6
7

u = rand(nv,1);
nu = zeros(k,1);
for i = 1:(k1)
id = find(u < fx(i));
nu(i) = length(id); % nu is the number of points to be picked at ...
each contour
u(id) = [];
end

8
9
10
11
12
13
14
15
16

x1 = [];
for i = 1:k
id = (i1)*nk + ceil(nk*rand(nu(i),1));
x1 = [x1 x(:,id)];
end
% Find the real function values at those selected sample points
y = [x1 y]; % add selected samples x1 into y
fy=[objfun(x1) fy]; % call objective function and calculate new objective ...
function's values

Step 4: Combine the sample points obtained in Step 3 with the initial points in Step 1 to form the
set x1 , x2 , ..., x2m and repeat Steps 2 to 3 until satisfying a certain stopping criterion.
1
2
3
4
5
6
7
8

9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

if(singular=='n')
% calculate (n+1) r square
[rsquare b]=rsquarefunc(xx,fkk');
fid = fopen('b.dat','w'); % save coefficient matrix
fprintf(fid,'%6.4f\t', b);
fclose(fid);
if(rsquare 0.98)
% randomly produce round(nv/2) samples in space [ min(ykk) max(ykk)] to ...
check fitting a secondorder function
y11 = sampling(nv,round(nv/2),min(ykk,[],2)',max(ykk,[],2)');
f11=objfun2(y11);
ykk=[ykk y11];
fkk=[fkk f11];
y=[y y11];
% in order count the number of evaluation
fy=[fy f11]; % in order count the number of evaluation
xx=coefmatrix(ykk);
rsquare=rsquarefunc(xx,fkk');
%printing rsquare
fit=fittingfunc(ykk);
% obj=objfun(ykk);
% maxdiff=max(abs(objfit));
maxdiff=max(abs(fkkfit));
if(rsquare0.99 & (maxdiff diff))
% calculate minimum point and minimum function value
options=optimset('LargeScale','off');

[x,fval]=fmincon('fittingfunc',ykk(:,1),[],[],[],[],xlv,xuv,[],options);
f=objfun2(x);
y=[x y];
fy=[f fy];
% check if the minimun point exists in the fitting area
if (min(ykk,[],2)x & xmax(ykk,[],2))
dialog='n';
end
end
end

24
25
26
27
28
29
30
31
32
33
34

end

Experimental Results

5.1

Unconstrained optimization

The performance of the MPS is evaluated by solving the unconstrained optimization presented in
Table 1. The first eight test functions are taken from [2], test functions 9 and 11 from [5] and test
function 10 from [7]. Due to the limitation of the algorithm, for n > 10 the MPS loses performance
(n stand for the number of variables), functions 1-10 are optimized for 2 variables.
Table 1: Test Objective Function, unconstrained optimization problems, unimodal and multimodal
functions.
Name Function
Limits
2
P
F1
f1 =
x2i
5.12 xi 5.12
i=1


2
F2
f2 = 100 x21 x2 + (1 x1 )2
2.048 xi 2.048
2
P
F3
f3 =
int (xi )
5.12 xi 5.12
F4

f4 =

i=1
2
P
i=1

F5

ix4i + Gauss (0, 1)

f5 = 0.002 +

25
P

1


j=1

1.28 xi 1.28

j+

2
P

655.36 xi 655.35


(xi aij )6

i=1

F6
F7

f6 = 10V +

2
P

xi , sin

p


|xi | , V = 4189.829101

i=1
2
P

x2i 10cos (2xi ), A = 10

 Q

 
2 
P
2
xi 2
xi

f8 = 1 +

cos
i=1
4000
i
i=1
h

i
2
1 + (x1 + x2 + 1) 19 14x1 + 3x22 14x2 + 3x22
h
f8 = 30 + (2x1 3x2 )2 18 32x1 + 12x2 + 48x2
1


36x1 x2 + 27x22
21 4
f10 = 4x21 10
x1 + 13 x61 + x1 x2 4x22 + 4x42
3
f11 = 2x1 32x1 + 1

f7 = 20A +

500 xi 500
5.12 xi 5.12

i=1

F8

F9
F10
F11

500 xi 500

2 xi 2
2 xi 2
5 xi 3

Each function is optimized 30 times. The algorithm stops when a global minimum is found or when
200 iterations have been reached. The global minimum is considered a point that lies in the fitting
interval with a R2 = 0.999, and with maximal difference between the function values less than 0.01.
The performance is measured by the percentage of the number of the times the algorithm finds the
optimal solution. The solution is considered optimal when the difference between the MPS solution
and the actual optimal function values is less than 0.01 (normalized with respect to the boundaries
of the search space). Figure 7 shows the results for the unconstrained test functions. Overall, the
8

0.9

0.8

Performance Index

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0
1

6
7
Function Fi

10

11

Figure 7: Performance of the MPS when solving the unconstrained test function F1-F11, Results
correspond to a total of 30 runs.
MPS has found the optimal solution, but has failed for F3 which is a function that contains many
flat surface. In halve of the cases MPS presents a performance above 40, but most of the time it
fails when searching for the global minimum. The performance of the algorithm could improved if
the number of iteration is increased. Due to the fact that overtime the algorithm guaranties that
converge to the global minimum [7]. Also, the number of contours can be increased to study whether
this fact improves the performance or not.
Figure 8 and 9 show the sample points generated by the MPS method for the test function F1-F10.
The red dot (*) locates the minimum value provided by the MPS. Overall, the MPS generates points
close to the optimal point.

5.2
5.2.1

Constrained optimization
Case Study 1:

In order to evaluate the performance of the MPS for constrained optimization problem, two more
cases are presented. The first problem is taken from [3], and can be written as follows:
x1 x2

min

subjected to

x2 2x41 8x31 + 8x21 + 2

x2 4x41 32x31 + 88x21 96x1 + 36
0 x1 3
0 x2 5

(5)

The above problem was solved using the MPS algorithm. A total of 30 runs were performed. The
MPS was able to find the global minimun with 96% of success x1 = 2.3295, x2 = 3.1785, and
f (x ) = 5.508. In Figure 10 the blue dots represent the sampling points generated by the MPS, the
red line represents the first constraint function, while the black line the second constraint function.The
figure also shows how the sampling points are generated and how those point are concentrating
toward the optimal. The global minimum locates at the constraint boundaries. The number of
function evaluations performed by the MPS was 17 which is less than the 21 number of evaluation
obtained using fmicon function of Matlab.
9

2
7

4
1.5
6

1
2
0.5

x2

x2

0.5

2
1
3

1.5
4
4
5
5

0
x1

2
2

1.5

0.5

0
x1

(a)

0.5

1.5

(b)

0.5

x2

x2

0.5

5
5

0
x1

0.5

0
x1

(c)

0.5

(d)
500

600
400

400

300

200
200

x2

x2

100

100
200
200

300

400

400
600
600

400

200

0
x1

200

400

600

500
500

400

300

200

400

300

200

100

100

200

300

400

0
x1

(g)

100

200

300

400

500

5
5

0
x1

(f)
500

x2

x2

(e)

100

500
500

400

300

200

100

0
x1

100

200

300

400

500

(h)

Figure 8: Sampling point generated when optimizing functions F1-F8: (a) F1; (b) F2; (c) F3; (d)
10
F4; (e) F5; (f) F6; (g) F7; and, (h) F8.

1.5

0.5

0.5

x2

x2

1.5

0.5

0.5

1.5

1.5

2
2

1.5

0.5

0
x1

0.5

1.5

2
2

1.5

0.5

(a)

0
x1

0.5

1.5

(b)

Figure 9: Sampling point generated when optimizing functions F9-F10: (a) F9; (b) F10

3.5

x2

2.5

1.5

0.5

0
0

0.5

1.5
x1

2.5

Figure 10: Constrained optimization test problem 1: blue dot stand for objective function evaluation,
red line for constraint function 1, and black line for constraint function 2
5.2.2

Case Study 2:

The second constrained optimization problem, taken from [1], consist of the design of a two-frame
member subjected to the out-of-plane load, P, as is shown in Figure 11. Besides L = 100 inches,
there are three design variables: the frame width (d), height (h), and wall thickness (t), having the
following ranges of interest: 2.5 d 10, 2.5 h 10, and 0.1 t 1.0.
The objective is to minimize the volume of the frame subject to stress constraints and size limitations.
The optimization problem can be written as:

11

z
(1)

P
L
x

L
(2)

(3)

t
h
d
Figure 11: Constrained optimization test problem 2: A two-member frame.

Figure 7 The two-member frame.

min V = 2L 2dt + 2ht + 2t2

1/2
12 + 3 2
40000

1/2
22 + 3 2
40000
2.5 d 10
2.5 h 10
0.1 t 1.0

subjected to

(6)

In the above equations 1 , 2 , and are respectively the bending stresses at point (1) (also point(3))
and (2) and the torsion stress of each member. They are defined by the following equations:

M1 h
2I
M2 h
2I
T
2At

(7)
(8)
(9)

where,

M1

M2

I
J

2EI (3U1 + U2 L)
L2
2EI (3U1 + U2 L)
L2
GJU3

L h

(10)
(11)
(12)
3

(1/12) dh3 (d 2t) (h 2t)

h
i
2
2
= 2t (d t) (h t) / (d + h 2t)

A =

(d t) (h t)

(13)
(14)
(15)

Given the constants E = 3.0E7, G = 1.154E7 and the load P = 10000. The displacements U1
(vertical displacement at point (2)), U2 (rotation about line (3)-(2)), and U3 (rotation about line
(1)-(2)) are calculated by using the finite element method are given by:

31
12

24
EI
6L
L
6L

6L


2
4L2 + GJ
EI L
0

6L
U1
P
U2 = 0
0


GJ 2
2
U3
0
4L + EI L

(16)

The optimal solution for this problem is located at d = 7.798.h = 10.00, and t = 0.01 with
V = 703.916.
The problem was solved using the MPS . A total of 30 runs were conducted. The obtained optima
in each run was practically the same as the analytical optimum. The MPS performs 19 evaluation
of the objective function to achieve the global minimum. The number of evaluations was less than
that obtained using fmicon function of Matlab (38).

Conclusions

In this report the Mode-Pursuing Sampling optimization method was studied. The performance of
the method was evaluated by solving unconstrained and constrianed optimization problems. All the
simulation were performed in Matlab, and a description of the implementation was presented. The
implementation of the MPS requires the tunning of only one parameters, the difference coefficient,
which is recommended to set to 0.01. For most of the test function, the MPS was able to identify the
global optimum. The MPS method can deal with problem with computational expensive objective
function, like a FEM solution. This was shown when comparing the number of function evaluations
solving the constrained problem using MPS and fmincon function of Matlab. Since the number of
points generated when evaluating the surrogate model is high, ten thousand points, the MPS could
suffer of lack of computer memory. Thus, as the authors of the method have indicated, MPS suited
well for problem with less than 10 design variables.
Further research can be the implementation of the MPS in the design of Parallel Manipulator with
maximum workspace. One the one hand, finding the workspace can lead to a computer expensive
objective function that additionally has constrains. On the other hand, the design parameters in
such kind of problems is less than 10.

References
[1] Arora, J. S. (1989). Introduction to Optimum Design. McGraw-Hill Higher Education, New York,
NY, USA.
[2] Digalakis, J. and Margaritis, K. (2000). An experimental study of benchmarking functions for
genetic algorithms. Systems, Man, and Cybernetics, 2000 IEEE International Conference on,
5:38103815 vol.5.
[3] Floudas, C. A. and Pardalos, P. M. (1990). A Collection of Test Problems for Constrained Global
Optimization Algorithms. Springer-Verlag New York, Inc., New York, NY, USA.
[4] Fu, J. C. and Wang, L. (2002). A random-discretization based monte carlo sampling method and
its applications. Methodology And Computing In Applied Probability, 4(1):525.
[5] Gary, W., Dong, Z., and Aitchison, P. (2001). Adaptive response surface method - a global optimization scheme for approximation-based design problems. Engineering Optimization, 33(6):707
733.
[6] Sharif, B., Wang, G. G., and ElMekkawy, T. Y. (2008). Mode pursuing sampling method for
discrete variable optimization on expensive black-box functions. Journal of Mechanical Design,
130(2):021402102140211.
[7] Wang, L., Shan, S., and Wang, G. G. (2004). Mode-pursuing sampling method for global optimization on expensive black-box functions. Engineering Optimization, 36(4):419438.

13