Experimental

Design, Response
Surface Analysis,
and Optimization
 Outline

 Motivation and Terminology

 Difficulties in Solving the Basic
Problem
 Examples of Factors and Responses
 Types/Examples of Experimental
Design
 Full Factorial Designs
 Randomness of Effects
 Example: Full Factorial Design
 Situations with Many Factors
 Response Surfaces and Metamodels
 Regression Analysis
 Response Surface Methodology
 Motivation and
Terminology
 Useful when there are many alternatives to
consider (e.g., numerous capacity levels of
various types, numerous parameters for a
proposed inventory system)
 Two basic types of variables: factors and
responses
 Factors: input parameters:
-controllable or uncontrollable
-quantitative or qualitative
 Responses: outputs from the simulation
model:
-uncertain in nature
 Basic problem: find the best levels (or values
of the parameters) in terms of the responses
 Experimental Design can tell you which
alternatives to simulate so that you obtain the
desired information with the least amount of
simulation
 Difficulties in Solving
the Basic Problem
 Multiple Responses

 Uncertain Responses
 Examples of Factors

1. Mean interarrival time (uncontrollable,

quantitative)
2. Mean service time (controllable or
uncontrollable, quantitative)
3. Number of servers (controllable,
quantitative)
4. Queuing discipline (controllable,
qualitative)
5. Reorder point (controllable, quantitative)
6. Mean interdemand time (uncontrollable,
quantitative)
7. Distribution of interdemand time
(uncontrollable, qualitative)
 Examples of Responses

1. Mean daily production rate

2. Mean time in the system for patients
3. Mean inventory level
4. Number of customers who wait for more
than 5 minutes
 Types/Examples of
Experimental Designs
 Completely Randomized Designs

 Randomized Complete Block Designs

 Nested Factorial Designs

 Split Plot Type Designs

 Latin Square Type Designs

 Full Factorial Designs

 Fractional Factorial Designs

 2k Factorial Designs

 Suppose that we have k (k > 2)

factors. A 2k factorial design would
require that two levels be chosen for
each factor, and that n simulation
runs (replications) be made at each
of the 2k possible factor-level
combinations (design points). For 3
factors, this yields a Design Matrix:
 2k Factorial Designs

Design Matrix for 3 Factors

Factor 1 Factor 2 Factor 3

Design Point Level Level Level
Response
1 - - - O1
2 + - - O2
3 - + - O3
4 + + - O4
5 - - + O5
6 + - + O6
7 - + + O7
8 + + + O8

“+” refers to one level of a factor and “-” refers to

the other level. Normally, for quantitative factors,
the smallest and largest levels for each factor are
chosen.
 2k Factorial Designs:
Estimating Main Effects
 The main effect of factor 1 is the change in
the response variable as a result of the
change in the level of the factor, averaged
over all levels of all of the other factors

e1 
O2  O1    O4  O3    O6  O5    O8  O7 
4
 If the effect of some factor depends on the
level of another factor, these factors are
said to interact.
 The degree of interaction (two-factor
interaction effect) between two factors i
and j is defined as half the difference
between the average effect of factor i when
factor j is at its “+” level ( and all factors
other than i and j are held constant) and
the average effect of i when j is at its “-”
level; for example,
1  ( O  O3 )  (O8  O7 ) (O2  O1 )  ( O6  O5 ) 
e12   4  
2 2 2
 Example: Full Factorial (2k)
Design
 Consider a simulation model of reorder
point, reorder quantity inventory system.
The two decision variables, or factors, to
consider are the reorder point (P) and the
reorder quantity (Q) for the inventory
system. The maximum and minimum
allowable values for each are given below:

Factor Minimum Maximum

Value Value
P 20 40
Q 15 50

 Suppose that the response variable

output by the model is the long-run
average monthly cost (composed of three
components: holding cost, shortage costs,
and ordering costs) in thousands of
dollars.
 Example: Full Factorial (2k)
Design
 A 22 factorial design matrix with simulation
results (for 10 replications at each design
point) might be given by:

Design Point Factor Level for Response

P Q
1 - - 135.6
2 + - 128.2
3 - + 121.7
4 + + 131.5

where a factor level of “-” indicates the

minimum possible value for that factor, and
a factor level of “+” indicates the maximum
possible level; for example, design point 2
has P=40 and Q=15.
 Example: Full Factorial (2k)
Design
 The response given is the average cost over
the 10 replications. Now, the main effects are
given by:

(O3  O1 )  (O4  O2 ) (1217

.  1356
. )  (1315
.  128.2)
eQ    53
.
2 2

(O2  O1 )  (O4  O3 ) (128.2  1356

. )  (1315
.  1217
.)
eP    12
.
2 2
 The interaction effect (ePQ) is given by:

135.6  128.2  1217

.  1315
.
e PQ   8.6
2
 Therefore, the average effect of increasing P
from 20 to 40 is to increase monthly cost by
1.2, and the average effect of increasing Q
from 15 to 50 is to decrease monthly cost by
5.3. Hence, it would be advisable to set P as
low as possible and set Q as high as possible.
 Example: Full Factorial (2k)
Design
 Also since the interaction effect is
positive, it would seem advisable to set
P and Q at opposite levels. (Of course,
all of the above could be inferred from
a cursory analysis of the responses for
the various design points.) Note also
that the literal interpretation of main
effects assumes no interaction effects
(pages 669 and 670 of Law and Kelton,
1991).
 Randomness of Effects
 Note that the main and interaction
effects computed in the previous
examples are just random variables.
To determine if the effects are
“significant” or real, and not due to
random fluctuations, one could
compute values for the main and
interaction effects 10 times (once for
each replication) and form confidence
intervals for each of the main effects,
and the interactions effect. If the
confidence interval contains 0, then
the effect is not statistically
significant. (Note that statistical
significance does not necessarily
imply practical significance).
 Situations with Many
Factors
 When there are many factors to
consider, full factorial, or even
fractional factorial designs may not be
feasible from a computational
standpoint.
 Other types of design (e.g., Plackett-
Burman designs or “supersaturated
designs”) may be appropriate (Mauro,
1986).
 Another approach is to reduce the
number of factors to consider via
“factor screening” techniques,
involving, for example, group
screening in which a whole group is
treated as a factor.
 Response Surfaces and
Metamodels
 A response surface is a graph of a
response variable as a function of the
various factors.
 A metamodel (literally, model of the
simulation model), is an algebraic
representation of the simulation
model, with the factors as
independent variables and a response
as the dependent variable. It
represents an approximation of the
response surface.
 The typical metamodel used in a
simulation application is a regression
model.
 Response Surfaces and
Metamodels
 A metamodel through the use of
response surface methodology can be
used to find optimal values for a set of
factors. It can also be used to answer
“what if” questions. (Experimentation
with a metamodel is typically much
less expensive than using a simulation
model directly).
 An experimental design process
assumes a particular metamodel, e.g.,

E[ C ( P, Q )]  B0  B1 P  B2 Q  B12 PQ
 Basic Concepts of
Regression Analysis
 Regression is used to determine the
“best” functional relation among
variables.
 Suppose that the functional
relationship is represented as:

E(Y) = f (X1, ..., Xp / B1, ..., BE)

where E(Y) is the expected value of

the response variable Y; the
X1, ..., Xp are factors; and the
B1, ..., BE are function parameters;
e.g.,

E(Y) = B1 + B2 X1 + B3 X2 + B4 X1 X2
 Basic Concepts of
Regression Analysis

 The observed value for Y, for a

given set of X ’s, is assumed to be a
random variable, given by:

Y = f (X1, ..., Xp/B1, ..., BE) +

Where  is a random variable with

 2
mean equal to 0 and variance E .
The values for B1,...,BE are
obtained by minimizing the sum of
squares of the deviations.
 Response Surface
Methodology
Source: (Fu, 1994)
 Response surface methodology
(RSM) involves a combination of
metamodeling (i.e., regression) and
sequential procedures (iterative
optimization).
 Response Surface
Methodology
 RSM involves two phases:
 Fit a linear regression model to some initial
data points in the search space (through
replications of the simulation model).
Estimate a steepest descent direction from
the linear regressions model, and a step
size to find a new (and better) solution in
the search space. Repeat this process
until the linear regression model becomes
inadequate (indicated by when the slope of
the linear response surface is
approximately 0; i.e., when the interaction
effects become larger than than the main
effects).
 Fit a nonlinear quadratic regression
equation to this new area of the search
space. Then find the optimum of this
equation.
 Terminology in
Experimental Design
Source:(Ostle, 1963)
 Replication - the repetition of the basic
experiment
 Treatment - a specific combination of
several factor levels
 Experimental Unit - the unit to which a
single treatment is applied to one
replication of the basic experiment
 Experimental Error - the failure of two
identically treated experimental units to
yield identical results
 Confounding - the “mixing up” of two or
more factors so that it’s impossible to
separate the effects
 Terminology in
Experimental Design
 Randomization - randomly assigning
treatments to experimental units (assures
independent distribution of errors)
 Main Effect (of a factor) - a measure of the
change in a response variable to changes
in the level of the factor averaged over all
levels of all the other factors
 Interaction is an additional effect (on the
response) due to the combined influence
of two or more factors