Parallel and Distributed Simulation

SS 2015

Background and Applications

Adelinde Uhrmacher
Modeling and Simulation Research Group
University of Rostock

Introduction & Motivation

What is parallel / distributed simulation?
Why are we interested in this subject?
Virtual environments vs. analytic simulations
Historical Perspective
Literature:

Richard Fujimoto:
Parallel and distributed Simulation Systems,
John Wiley, New York, 2000.
Transparencies are partly taken from a lecture given by Richard
Fujimoto at the Georgia Institute of Technology and a tutorial at
the Parallel, distributed Simulation Conference, Bologna, Italy,
June 2000.

Adelinde M. Uhrmacher
University of Rostock

What is parallel / distributed simulation?

Simulation - a system that represents or emulates the
behavior of another system over time; a computer
simulation is one where the system doing the emulating is
a computer program
Parallel (distributed) simulation refers to the technology
concerned with executing computer simulations over
computing systems containing multiple processors
Tightly coupled multiprocessor systems
Workstations interconnected via a network
(e.g., the Internet)

Adelinde M. Uhrmacher
University of Rostock

Why are we interested in this subject?

Reduced model execution time

Up to n-fold reduction using n CPUs

Scalable performance

Maintaining the same execution speed for bigger

models/virtual environments by using more CPUs

Geographically distributed users and/or resources

(e.g., databases, specialized equipment)

Co-location is expensive! May be impractical

Integrate simulations running on different platforms

Network perspective rather than to export them to a

platform

Fault tolerance

Not as easy as it might seem!

Adelinde M. Uhrmacher
University of Rostock

Enable Simulation of Big Models

Cell level simulation of an ATM (packet) network
Simulate one hour of network operation
Network with 1000 links
155 Mbits/second links @ 20% utilization
53 byte packets (cells)
One simulator event per cell transmission (link)
500 k events / second simulator speed
150 hours for a single simulation run!
Larger, more complex networks?

Next generation Internet: Million nodes

Higher link bandwidths

Adelinde M. Uhrmacher
University of Rostock

Fast Execution: On-Line Decision Aids

interactive
simulation
environment
live
data
feeds

analysts and
decision
makers
situation
database

forecasting tool
(fast simulation)

Simulation tool is used for fast analysis of alternate courses of

action in time critical situations
Initialize simulation from situation database
Faster-than-real-time execution to evaluate effect of
decisions
Applications: air traffic control, battle management
Simulation results may be needed in only seconds
Adelinde M. Uhrmacher
University of Rostock

Introduction & Motivation

What is parallel / distributed simulation?

Why are we interested in this subject?
Virtual environments vs. analytical simulations
Historical Perspective

Adelinde M. Uhrmacher
University of Rostock

Virtual Environments
Uses: training (e.g., military, medicine, emergency
planning), entertainment, social interaction?
Simulations are often used in virtual environments to
create dynamic computer generated entities
Adversaries and helpers in video games
Defense: Computer generated forces (CGF)

Automated forces
Semi-automated forces

Physical phenomena

Trajectory of projectiles
Buildings "blowing up"
Environmental effects
(e.g., rain washing out terrain)

Adelinde M. Uhrmacher
University of Rostock

System Analysis
"Classical" application of simulation; here, focus on
"discrete event" simulation
Telecommunication networks
Transportation systems
Electronic systems
(e.g., microelectronics, computer systems)
Battlefield simulations (blue army vs. red army)
Ecological systems
Manufacturing systems
Logistics
Cell biological systems
Focus typically on planning, system design, and
understanding
Adelinde M. Uhrmacher
University of Rostock

Virtual Environments vs. Analysis

Typical Characteristics
Analysis

Virtual environment

typical
objective

quantitative analysis of create realistic or entertaining representation

complex systems

execution
pacing

as-fast-aspossible

real-time (paced, syn.

Simulation-wall clock time)

human
interaction

if included, often
external observer

integral to controlling
entities

accuracy

statistically correct
results

human perception
often plays a large role

A major
issue

Validity, reproducibility

plausibility, natural
touch and feel

Adelinde M. Uhrmacher
University of Rostock

10

Principal Application Domains

Parallel Discrete Event
Simulation (PDES)
Discrete event simulation to
analyze systems
Fast model execution
(as-fast-as-possible)
Produce same results as a
sequential execution
Typical applications
Telecommunication
networks
Computer systems
Transportation systems
Military strategy and
tactics
Adelinde M. Uhrmacher
University of Rostock

Distributed Virtual
Environments (DVEs)
Networked interactive,
immersive environments
Scalable, real-time
performance
Create virtual worlds that
appear realistic
Typical applications
Training
Entertainment
Social interaction

11

Historical Perspective
High Performance Computing Community
Chandy/Misra/Bryant
second generation algorithms
algorithm
Time Warp algorithm
making it fast and
early experimental
easy to use
data

1975

1980

1985

Cloud computing
Grid-Based approaches

1990

2000
2005
1995
SIMulator NETworking (SIMNET) High Level Architecture
(1996 - today)
(1983-1990)

Defense Community
Dungeons and Dragons
Board Games
Adventure
(Xerox PARC)

Approximative
Multi-resolution approaches
Service-oriented
approaches

2010

Distributed Interactive Simulation (DIS)

Aggregate Level Simulation Protocol (ALSP)
(1990 - 1997ish)
Multi-User Video Games

Multi-User Dungeon (MUD)

Games

Adelinde M. Uhrmacher
University of Rostock

Internet & Gaming Community

12

Parallel / Distributed Simulation Today

High Performance Computing Community
After a slow start, the technology is beginning to be
embraced because of efforts such as the High Level
Architecture
Defense Community
Technology has been fully embraced
Training
War gaming
Test & evaluation

Gaming Community
Technology becoming heavily used
Server-based systems
Internet gaming
Adelinde M. Uhrmacher
University of Rostock

13

Summary
Several reasons to execute simulations over multiple
computers
Performance
Geographical distribution
Easier integration of systems (interoperability), reuse
Virtual environment vs. system analysis
Have different requirements, sometimes resulting in
use of different approaches and techniques
Developed from largely disjoint communities
Research and development communities
High performance computing
Defense
Internet and gaming
Adelinde M. Uhrmacher
University of Rostock

14

Outline
Hardware platforms
Parallel computers
Multi-Core machines
Distributed computers
Simulation fundamentals
State
Time
Time flow mechanisms

Adelinde M. Uhrmacher
University of Rostock

Parallel and Distributed Simulation SS

15

Parallel and Distributed Computers

Parallel computers (tightly coupled processors)
Shared memory multiprocessors
Distributed memory multicomputers
Multi-core PCs
Distributed computers (loosely coupled processors)
Networked workstations
Parallel Computers

Distributed Computers

Processors

Machine room/office

Building, city, global

Physical extent

Homogeneous

Often heterogeneous

Comm.Network

Custom switch

Commercial LAN / WAN

A few to tens of
Comm. Latency
(small messages) microseconds

Adelinde M. Uhrmacher
University of Rostock

Hundreds of microseconds to seconds

Parallel and Distributed Simulation SS

16

Shared Memory Multiprocessors

Examples: SUN Enterprise, SGI Origin
CPU

CPU

cache

cache

CPU

. . .

cache

Interconnection network
memory

. . .

memory

I/O
devices

Programming model: shared variables; synchronization via locks

Processor 1

{ shared int i;L

Lock(L)
i = i +1;
Unlock(L)

Adelinde M. Uhrmacher
University of Rostock

Processor 2

{shared int i;L

Lock(L)
i = i +1;
Unlock(L)

}
17

Multi-CPU, Multi-Core PCs

While Moore's Law, which predicts exponentially increasing microelectronic
circuit density over time, is expected to continue into the next decade, it is
clear that further hardware performance increases will result primarily from
multi-core circuits that rely on parallel processing rather than improved clock
rates
While the need to simulate larger, more complex systems will continue into
the foreseeable future, our ability to simulate such systems with sequential
software will be diminishing in the years ahead. Long a nice to have
capability, parallel simulation will increasingly become a must have.
However, many major hurdles remain before parallel execution can become
widely exploited. Development of scalable parallel simulation software, easy
to use synchronization techniques, effective load balancing and resource
allocation approaches, automated fault tolerance, and time shared use of
computing platforms each by itself presents a formidable challenge, yet must
all become straightforward or automatic if widespread adoption of parallel
simulation methods is to be successful.
Richard Fujimoto, Modeling Simulation and Parallel Computation, the Future is now!
Springsim 2007
Adelinde M. Uhrmacher
University of Rostock

18

Distributed Memory Multicomputers

Examples: IBM SP, Intel Paragon
CPU
cache

CPU
memory

communications
controller

cache

. . .

memory

communications
controller

Interconnection network

Programming model: no shared variables; message passing

Processor 1

{int i;

Send (2, &i, sizeof(int))

Adelinde M. Uhrmacher
University of Rostock

Processor 2

{int j;

Receive (2, &j, sizeof(int))

}
19

Parallel and Distributed Simulation

simulation
model

Parallel simulation involves the execution

of a single simulation program on a
collection of tightly coupled processors
(e.g., a shared memory multiprocessor).
P

P
M

P
M

P
M

parallel
processor

Replicated trials involves the

execution of several, independent
simulations concurrently on
different processors
Adelinde M. Uhrmacher
University of Rostock

Distributed simulation involves the

execution of a single simulation
program on a collection of loosely
coupled processors
(e.g., PCs interconnected by a LAN
or WAN).
20

Outline
Hardware platforms
Parallel computers
Distributed computers
Simulation fundamentals
State
Time
Time flow mechanisms

Adelinde M. Uhrmacher
University of Rostock

21

Parallel jobs

Feedback (jobs )

Parallel analysis

Replication 1
(Job 1)

Analysis (Rep 1
, Job 1)

Cancellation
(rep 1, job 1)
.

Replication m
(Job 1)

Analysis (Rep
m, Job 1)

Replication 1
(Job n)

Analysis (Rep 1
, Job n)

Replication m
(Job n)

Analysis (Rep
m, Job n)

Distributed
model

Distributed /
sequential
simulation
algorithm

Iterative / Upon
completion of
replication

Job distribution

Partitioning / Load
balancing

Job 1

.
Experiment

Parallel
replications

.
.

Job n

Parallel job
analysis

Analysis Job
1

.
.
.

Analysis
Experiment

Analysis Job
n

Iterative / Upon
completion of all
replications

Iterative / Upon
completion of all
jobs

(Remote / Distrib.)
data sink w.
seq./par. access

Multi CPU, Multi Core, Cluster, Grid, Cloud, ...

SOA, Web Services, RMI, MPI, ...
Adelinde M. Uhrmacher
University of Rostock

HLA, Data bases, ...

22

Simulation Fundamentals
A simulation is an experiment with a model in this case
performed on a computer. Please note that in the parallel
distributed community often no clear separation between
formally described model and algorithm that executes this
model does exist.

Program variables (state variables) represent the current

state of the physical system
Simulation program modifies state variables to model the
evolution of the physical system over time

Adelinde M. Uhrmacher
University of Rostock

23

Time
physical system: the actual or imagined system being modeled
simulation: a system that emulates the behavior of a physical
system

main()
{ ...
double clock;
...
}

physical system

physical time:

11

12

1
2

10
9

4
7

simulation

time in the physical system

Noon, December 31, 1999 to noon January 1, 2000

simulation time: representation of physical time within the

simulation

Floating point values in interval [0.0, 24.0]

wallclock time: time during the execution of the simulation,

usually output from a hardware clock

9:00 to 9:15 AM on September 10, 1999

Adelinde M. Uhrmacher
University of Rostock

24

Simulation Time
Simulation time is defined as a totally ordered set of values
where each value represents an instant of time in the
physical system being modeled.
For any two values of simulation time T1 representing
instant P1, and T2 representing P2:
Correct ordering of time instants

If T1 < T2, then P1 occurs before P2

9.0 represents 9 PM, 10.5 represents 10:30 PM

Correct representation of time durations

T2 T1 = k (P2 P1) for some constant k

1.0 in simulation time represents 1 hour of physical
time

Adelinde M. Uhrmacher
University of Rostock

25

Paced vs. Unpaced Execution

Modes of execution
As-fast-as-possible execution (unpaced):
no fixed relationship necessarily exists between advanced in
simulation time and advances in wallclock time
Real-time execution (paced):
each advance in simulation time is paced to occur in synchrony
with an equivalent advance in wallclock time
Scaled real-time execution (paced):
each advance in simulation time is paced to occur in synchrony
with S* an equivalent advance in wallclock time (e.g., 2x
wallclock time)

Adelinde M. Uhrmacher
University of Rostock

26

Paced vs. Unpaced Execution

Modes of execution
With unpaced we refer to As-fast-as-possible
We rather use paced than Real-time execution for an execution
that advances in simulation time in synchrony with (K*)
an equivalent advance in wallclock time
As Real-time simulation often refers to certain real-time
constraints for executing the simulation (either in paced or
unpaced mode)
Faster than real-time typically implies a simulation used for
decision processes.

Adelinde M. Uhrmacher
University of Rostock

27

Simulation Taxonomy
computer simulation

discrete models
event driven

continuous models

time-stepped

Continuous simulation
State changes occur continuously across time
Typically, behavior described by differential equations

Discrete simulation
State changes only occur at discrete time instants
Times stepped: time advances by fixed time increments
Event driven: time advances occur with irregular increments
Adelinde M. Uhrmacher
University of Rostock

28

Time Stepped vs. Event Stepped

state variables

state variables

Goal: compute state of system over simulation time

simulation time

time stepped execution

Adelinde M. Uhrmacher
University of Rostock

simulation time

event driven execution

29

The Stepped Execution (Paced)

While (simulation not completed)
{
Wait until (W2S(wallclock time) current simulation time)
compute state of simulation at end of this time step
advance simulation time to next time step
}
Simulation Time = W2S(W) = T0 + S (W W0)
W = wallclock time;
S = scale factor
W0 (T0) = wallclock (simulation) time at start of simulation
(assume simulation and wallclock time use same time units)
Adelinde M. Uhrmacher
University of Rostock

30

Summary
Hardware Platforms
Tightly coupled multiprocessors: fast communication
Networked workstations: larger message latencies
Important to distinguish among simulation time, wallclock
time, and time in the physical system
Paced execution (e.g., immersive virtual environments)
vs. unpaced execution (e.g., simulations to analyze
systems)
Continuous and discrete simulation use different execution
paradigms, time advance mechanisms
Time stepped vs. event stepped
Here, focus on discrete event simulations
Adelinde M. Uhrmacher
University of Rostock

31

The next weeks

Basics of Parallel Discrete Event Simulation
DEVS Advanced Techniques
Conservative Techniques
Optimistic Techniques
Advanced Optimistic Techniques
Partitioning, Loadbalancing
GPU based executions
HLA (High Level Architecture)

Adelinde M. Uhrmacher
University of Rostock

32