Escolar Documentos
Profissional Documentos
Cultura Documentos
Richard M. Fujimoto
College of Computing
Georgia Institute of Technology
Atlanta, GA 30332-0280
fujimoto@cc.gatech.edu
References
HLA:
F. Kuhl, R. Weatherly, J. Dahmann, Creating Computer Simulation
Systems: An Introduction to the High Level Architecture for
Simulation, Prentice Hall, 1999.
(http://hla.dmso.mil)
Outline
Part I:
Introduction
Part II:
Time Management
Part III:
Distributed Virtual Environments
Parallel and Distributed Simulation
simulation Parallel simulation involves the
model execution of a single simulation
program on a collection of tightly
coupled processors (e.g., a shared
memory multiprocessor).
P P P P
parallel
processor
M M M
LAN interconnect
Stand-Alone vs. Federated Simulation Systems
Stand-Alone Federated Simulation
Simulation System Systems
Simulator 2
Process 1 Simulator 1 Simulator 3
Process 2
Process 3
Run Time Infrastructure-RTI
Process 4 (simulation backplane)
• federation set up & tear down
• synchronization, message ordering
• data distribution
Parallel simulation environment
Fundamental concepts:
• system state (state variables)
• state transitions (events)
• simulation time: totally ordered set of values representing time in the
system being modeled (physical system)
• simulator maintains a simulation time clock
A discrete event simulation computation can be viewed as a sequence of
event computations
H0 H1
Time Event Time Event Time Event Time Event Time Event
0 H0 Send Pkt
1 H1 Recv Pkt
1 H1 Send Ack
2 H0 Recv Ack
2 H0 Send Pkt
6 H0 Retrans 6 H0 Retrans 6 H0 Retrans
100 Done 100 Done 100 Done 100 Done 100 Done
Simulation LP (Subnet 1)
Recv Pkt
@15
LP LP
(Subnet 2) (Subnet 3)
all interactions between LPs must be via messages (no shared state)
The “Rub”
Golden rule for each logical process:
“Thou shalt process incoming messages in time
stamp order!!!” (local causality constraint)
Parallel Discrete Event Simulation Example
Physical system
Recv Pkt
@15
LP LP
(Subnet 2) (Subnet 3)
all interactions between LPs must be via messages (no shared state)
The Synchronization Problem
Local causality constraint: Events within each logical process must be
processed in time stamp order
Observation: Adherence to the local causality constraint is sufficient to
ensure that the parallel simulation will produce exactly the same
results as the corresponding sequential simulation*
Synchronization (Time Management) Algorithms
• Conservative synchronization: avoid violating the local causality
constraint (wait until it’s safe)
– 1st generation: null messages (Chandy/Misra/Bryant)
– 2nd generation: time stamp of next event
• Optimistic synchronization: allow violations of local causality to occur,
but detect them at runtime and recover using a rollback mechanism
– Time Warp (Jefferson)
– approaches limiting amount of optimistic execution
* provided events with the same time stamp are processed in the same order
as in the sequential execution
Part II:
Time Management
H1
9 8 2
H3
logical
process one FIFO
H2 H3 5 4
queue per
incoming link
H1 7
(waiting
on H2)
15 H3
10
(waiting
H2 on H1)
(waiting 9 8
on H3)
A cycle of LPs forms where each is waiting on the next LP in the cycle.
No LP can advance; the simulation is deadlocked.
Deadlock Avoidance Using Null Messages
Break deadlock by having each LP send “null” messages indicating a
lower bound on the time stamp of future messages it could send.
H1 7
(waiting
11 on H2)
15 H3
10 8
(waiting
H2 on H1)
(waiting 9 8
on H3)
H1 7
(waiting
6.0 on H2)
7.5
6.5
15 H3
10 5.5 7.0
(waiting
H2 on H1)
(waiting 9 8
on H3)
0.5
Assume minimum delay between hosts is 3 units of time
• H3 initially at time 5
• H3 sends null message, time stamp 5.5; H2 sends null message, time stamp 6.0
• H1 sends null message, time stamp 6.5; H3 send null message, time stamp 7.0
• H2 sends null message, time stamp 7.5; H1 can process time stamp 7 message
Five null messages to process a single event
H1 7
(waiting
on H2)
15 H3
10
(waiting
H2 on H1)
(waiting 9 8
on H3)
H2@6.1
(blocked)
H2 10 15
7 transient
H1
H1@5 LBTS = ?
min (6, 10, 7) (assume zero lookahead) Simulation time
5 6 7 8 9 10 11 12 13 14 15 16
LP1 Future
wallclock time
cut point: an instant dividing process’s computation into past and future
cut: set of cut points, one per process
cut message: a message that was sent in the past, and received in the future
consistent cut: cut + all cut messages
cut value: minimum among (1) local minimum at each cut point and (2) time
stamp of cut messages; non-cut messages can be ignored
It can be shown LBTS = cut value
A Simple LBTS Algorithm
LP4 cut
message
LP3
Past
LP2
LP1 Future
wallclock time
Initiator broadcasts start LBTS computation message to all LPs
Each LP sets cut point, reports local minimum back to initiator
Account for transient (cut) messages
• Identify transient messages, include time stamp in minimum computation
– Color each LP (color changes with each cut point); message color = color of sender
– An incoming message is transient if message color equals previous color of receiver
– Report time stamp of transient to initiator when one is received
• Detecting when all transients have been received
– For each color, LPi keeps counter of messages sent (Sendi) and received (Receivei)
– At cut point, send counters to initiator: # transients = (Sendi – Receivei)
– Initiator detects termination (all transients received), broadcasts global minimum
Another LBTS Algorithm
LP4 cut
message
LP3
Past
LP2
LP1 Future
wallclock time
- barrier -
wallclock
time wait - barrier -
- barrier -
wait wait
- barrier -
Synchronous algorithm
DO WHILE (unprocessed events remain)
barrier synchronization; flush all messages from the network
LBTS = min (Ni + LAi); Ni = time of next event in LPi; LAi = lookahead of LPi
all i
S = set of events with time stamp ≤ LBTS
process events in S
endDO
Variations proposed by Lubachevsky, Ayani, Chandy/Sherman, Nicol
Topology Information
ORD
4:00 2:00
6:00
LAX JFK
0:30 10:45
SAN
10:00
The distance from LPA to LPB is the minimum amount of simulated time
that must elapse for an event in LPA to affect LPB
Distance Matrix:
11 D [i,j] = minimum distance from LPi to LPj
3
LPA LPB LPA LPB LPC LPD
4 LPA 4 3 1 3
3 1 4 1 LPB 4 5 3 1
min (1+2, 3+1)
2 LPC 3 6 4 2
LPC LPD
LPD 5 4 2 4
2
13 15
11 Distance Matrix:
3
D [i,j] = minimum distance from LPi to LPj
LPA LPB
LPA LPB LPC LPD
4
1 1 LPA 4 3 1 3
3 4
2 LPB 4 5 3 1
min (1+2, 3+1)
LPC LPD LPC 3 6 4 2
2 LPD 5 4 2 4
13 15
ORD
4:00 2:00
6:00
LAX JFK
0:30 10:45
SAN
10:00
Each LP A using declares a lookahead value LA; the time stamp of any event
generated by the LP must be ≥ TA+ LA
• Used in virtually all conservative synchronization protocols
• Relies on model properties (e.g., minimum transmission delay)
4
Optimized
(deterministic service
3 time)
Optimized (exponential
Speedup
service time)
2
Classical (deterministic
service time)
1 Classical (exponential
service time)
0
1 2 4 8 16 32 64 128 256
Number of Jobs in Network
H1
9 8 2
H3
5 4
logical
process
H2 H3
Input Queue
(event list) processed event
12 21 35 41 unprocessed event
snapshot of LP state
State Queue
anti-message
Output Queue
12
12 42
(anti-messages) 19
18
Adding rollback:
• a message arriving in the LP’s past initiates rollback
• to roll back an event computation we must undo:
– changes to state variables performed by the event;
solution: checkpoint state or use incremental state saving (state queue)
– message sends
solution: anti-messages and message annihilation (output queue)
Anti-Messages
42
positive message
anti-message
42
Output Queue 12
12 42 2(b) send anti-message
19
(anti-messages)
18
Input Queue
(event list)
12 18 21 35 41
Output Queue
12
12
(anti-messages) 19
AFTER
Processing Incoming Anti-Messages
Case I: corresponding message has not yet been processed
– annihilate message/anti-message pair
may cause “cascaded” rollbacks; recursively applying eliminates all effects of error
Observation: A lower bound on the time stamp of any rollback that can
occur in the future is needed.
Global Virtual Time (GVT) is defined as the minimum time stamp of any
unprocessed (or partially processed) message or anti-message in the
system. GVT provides a lower bound on the time stamp of any future
rollback.
• storage for events and state vectors older than GVT (except one state
vector) can be reclaimed
• I/O operations with time stamp less than GVT can be performed.
• GVT algorithms are similar to LBTS algorithms in conservative
synchronization
Observation: The computation corresponding to GVT will not be rolled
back, guaranteeing forward progress.
Time Warp and Chandy/Misra Performance
8
Time Warp (64 logical
7 processes)
6 Time Warp (16 logical
processes)
Speedup
5 Deadlock Avoidance
(64 logical processes)
4
Deadlock Avoidance
3 (16 logical processes)
Deadlock Recovery (64
2 logical processes)
1 Deadlock Recovery (64
logical processes)
0
0 16 32 48 64
Message Density
(messages per logical process)
• eight processors
• closed queueing network, hypercube topology
• high priority jobs preempt service from low priority jobs (1% high priority)
• exponential service time (poor lookahead)
Other Optimistic Algorithms
Principal goal: avoid excessive optimistic execution
Interface Interface
Specification Specification
Runtime Infrastructure(RTI)
Services to create and manage the execution of the federation
• Federation setup / tear down
• Transmitting data among federates
• Synchronization (time management)
Interface Specification
Category Functionality
Create and delete federation executions
Federation Management join and resign federation executions
control checkpoint, pause, resume, restart
Establish intent to publish and subscribe
Declaration Management
to object attributes and interactions
Create and delete object instances
Control attribute and interaction
Object Management
publication
Create and delete object reflections
exchange information
• Update/Reflect Attribute Values (Object Mgt)
• Send/Receive Interaction (Object Mgt)
• Time Advance Request, Time Advance Grant (Time Mgt)
• Request Attribute Ownership Assumption (Ownership Mgt)
• Modify Region (Data Distribution Mgt)
terminate execution
• Resign Federation Execution (Federation Mgt)
• Destroy Federation Execution (Federation Mgt)
HLA Message Ordering Services
The HLA provides two types of message ordering:
• receive order (unordered): messages passed to federate in an arbitrary order
• time stamp order (TSO): sender assigns a time stamp to message; successive
messages passed to each federate have non-decreasing time stamps
Receive Time Stamp
Property
Order (RO) Order (TSO)
Latency low higher
reproduce before and after
no yes
relationships?
all federates see same
no yes
ordering of events?
execution repeatable? no yes
typical applications training, T&E analysis
• receive order minimizes latency, does not prevent temporal anomalies
• TSO prevents temporal anomalies, but has somewhat higher latency
Advancing Logical Time
HLA TM services define a protocol for federates to advance logical time; logical
time only advances when that federate explicitly requests an advance
• Time Advance Request: time stepped federates
• Next Event Request: event stepped federates
• Time Advance Grant: RTI invokes to acknowledge logical time advances
federate
Time Advance Request
or Time Advance Grant
Next Event Request
RTI
next
TSO
TSO message
messages
RTI next
T’ local
event
local logical
events federate T
current time
time
Federate: next local event has time stamp T
• If no TSO messages w/ time stamp < T, advance to T, process local event
• If there is a TSO message w/ time stamp T’ ≤ T, advance to T’ and process TSO
message
Next Event Request (NER)
• Federate invokes Next Event Request (T) to request its logical time be
advanced to time stamp of next TSO message, or T, which ever is smaller
• If next TSO message has time stamp T’ ≤ T
– RTI delivers next TSO message, and all others with time stamp T’
– RTI issues Time Advance Grant (T’)
• Else
– RTI advances federate’s time to T, invokes Time Advance Grant (T)
L- ∆T
∆T
∆L L’
network
TSO queue
13 Current Time =8 Current Time =9
LBTS0=10 Lookahead = 2 Lookahead = 8
11
8 MNET0=8
• LBTSi: Lower Bound on Time Stamp of TSO messages that could later
be placed into the TSO queue for federate i
– TSO messages w/ TS ≤ LBTSi eligible for delivery
– RTI ensures logical time of federate i never exceeds LBTSi
• MNETi: Minimum Next Event Time is a lower bound on the time stamp of
any message that could later be delivered to federate i.
– Minimum of LBTSi and minimum time stamp of messages in TSO
queue
Event Retraction
Previously sent events can be “unsent” via the Retract service
– Update Attribute Values and Send Interaction return a “handle” to
the scheduled event
– Handle can be used to Retract (unschedule) the event
– Can only retract event if its time stamp > current time + lookahead
– Retracted event never delivered to destination (unless Flush
Queue used)
Sample execution sequence: NER: Next Event Request
UAV: Update Attribute Values)
TAG: Time Advance Grant (callback)
1. NER (100) 2. TAG (90)
2. Receive Interaction (90)
Vehicle
1. Handle=UAV (100) 3. Retract(Handle)
Observer
Wallclock time
1. Vehicle schedules position update at time 100, ready to advance to time 100
2. receives interaction (break down event) invalidating position update at time 100
3. Vehicle retracts update scheduled for time 100
Optimistic Time Management
Mechanisms to ensure events are processed in time stamp order:
• conservative: block to avoid out of order event processing
• optimistic: detect out-of-order event processing, recover (e.g., Time Warp)
Primitives for optimistic time management
• Optimistic event processing
– Deliver (and process) events without time stamp order delivery guarantee
– HLA: Flush Queue Request
• Rollback
– Deliver message w/ time stamp T, other computations already performed at times >
T
– Must roll back (undo) computations at logical times > T
– HLA: (local) rollback mechanism must be implemented within the federate
• Anti-messages & secondary rollbacks
– Anti-message: message sent to cancel (undo) a previously sent message
– Causes rollback at destination if cancelled message already processed
– HLA: Retract service; deliver retract request if cancelled message already delivered
• Global Virtual Time
– Lower bound on future rollback to commit I/O operations, reclaim memory
– HLA: Query Next Event Time service gives current value of GVT
Optimistic Time Management in the HLA
Functionality:
• allows federates with different time management requirements (and local TM
mechanisms) to be combined within a single federation execution
– DIS-style training simulations
– simulations with hard real-time constraints
– event-driven simulations
– time-stepped simulations
– optimistic simulations
Introduction
Dead reckoning
Data distribution
Example: Distributed Interactive Simulation (DIS)
“The primary mission of DIS is to define an infrastructure for linking
simulations of various types at multiple locations to create realistic,
complex, virtual ‘worlds’ for the simulation of highly interactive
activities” [DIS Vision, 1994].
Image network
Generator Other Vehicle interface
State Table
terrain
database control/
own vehicle sound
display
dynamics generator
interface
controls
and panels
visual display
3 6 7 4 8
1. Detect trigger press
Image network
Generator interface 2. Audio “fire” sound
Other Vehicle
State Table 3. Display muzzel flash
terrain
database control/ 4. Send fire PDU
own vehicle sound
display
dynamics generator
interface 5. Display muzzel flash
1 2 6. Compute trajectory,
Controls/panels
display tracer
7. Display shell impact
visual display
5 9 4 8 11
8. Send detonation PDU
Image network 9. Display shell impact
Generator Other Vehicle interface
10. Compute damage
State Table
terrain
11. Send Entity state PDU
database control/
own vehicle sound indicating damage
display
dynamics generator
interface
10
Controls/panels
DIS Design Principles
• Autonomy of simulation nodes
– simulations broadcast events of interest to other simulations; need not determine
which others need information
– receivers determine if information is relevant to it, and model local effects of new
information
– simulations may join or leave exercises in progress
• Transmission of “ground truth” information
– each simulation transmits absolute truth about state of its objects
– receiver is responsible for appropriately “degrading” information (e.g., due to
environment, sensor characteristics)
• Transmission of state change information only
– if behavior “stays the same” (e.g., straight and level flight), state updates drop to a
predetermined rate (e.g., every five seconds)
• “Dead Reckoning” algorithms
– extrapolate current position of moving objects based on last reported position
• Simulation time constraints
– many simulations are human-in-the-loop
– humans cannot distinguish temporal difference < 100 milliseconds
– places constraints on communication latency of simulation platform
Part III:
Distributed Virtual Environments
Introduction
Dead reckoning
Data distribution
Distributed Simulation Example
1000 1050
High
Fidelity close DRM
Model enough? aircraft 1
timeout? simulator for
aircraft 1
aircraft 1
over threshold DRM
or timeout aircraft 2
entity state
update PDU
DRM
aircraft 1
simulator for
aircraft 2
display update
Potential problems:
• Discontinuity may occur when position update arrives;
may produce “jumps” in display
• Does not take into account message latency
Time Compensation
A B C
true position
state update t1
t2 update with time
message
compensation
DRM estimate of
D E
true position
display update
Smoothing
Reduce discontinuities after updates occur
• “phase in” position updates
• After update arrives
– Use DRM to project next k positions
– Interpolate position of next update
A B C interpolated
true position position
t1
state update
t2
message D
DRM estimate of
E
true position
display update
extrapolated position
used for smoothing
Introduction
Dead reckoning
Data distribution
Data Distribution Management
A federate sending a message (e.g., updating its current location) cannot
be expected to explicitly indicate which other federates should receive
the message
Basic Problem: which federates receive messages?
• broadcast update to all federates (SimNet, early versions of DIS)
– does not scale to large numbers of federates
• grid sectors
– OK for distribution based on spacial proximity
• routing spaces (STOW-RTI, HLA)
– generalization of grid sector idea
Content-based addressing: in general, federates must specify:
• Name space: vocabulary used to specify what data is of interest and to
describe the data contained in each message (HLA: routing space)
• Interest expressions: indicate what information a federate wishes to
receive (subset of name space; HLA: subscription region)
• Data description expression: characterizes data contained within
each message (subset of name space; HLA: publication region)
HLA Routing Spaces
1.0
• Federate 1 (sensor): subscribe to S1
• Federate 2 (sensor): subscribe to S2
S2
• Federate 3 (target): update region U
0.5 U
S1
update messages by target are sent to
federate 1, but not to federate 2
0.0
0.0 0.5 1.0
new region 33 34 35 36 37 38 39 40
25 26 27 28 29 30 31 32
existing subscription region
17 18 19 20 21 22 23 24
Leave group
Join group 9 10 11 12 13 14 15 16
no operations issued 1 2 3 4 5 6 7 8
1.0
• Multicast group associated with U
S2 • Membership of U: S1
0.5 U • Modify U to U’:
S1 – determine subscription regions
overlapping with U’
– Modify membership accordingly
0.0
0.0 0.5 1.0
• Associate a multicast group with each update region
• Membership: all subscription regions overlapping with update region
• Changing subscription (update) region
– Must match new subscription (update) region against all existing
update (subscription) regions to determine new membership
• No duplicate or extra messages
• Change subscription: typically requires interprocessor communication
• Can use grids (in addition to regions) to reduce matching cost
Distributed Virtual Environments: Summary
• Perhaps the most dominant application of distributed
simulation technology to date
– Human in the loop: training, interactive, multi-player video games
– Hardware in the loop
• Managing interprocessor communication is the key
– Dead reckoning techniques
– Data distribution management
• Real-time execution essential
• Many other issues
– Terrain databases, consistent, dynamic terrain
– Real-time modeling and display of physical phenomena
– Synchronization of hardware clocks
– Human factors
Future Research
Directions
The good news...
• Battle management
• Enterprise control
• Transportation Systems decision makers
Research Challenges
• Ultra fast model execution
– Much, much faster-than-real-time execution
– Instantaneous model execution: Spreadsheet-like performance
• Ultra fast execution of multiple runs
• Automated simulation analyses: smart, self-managed
devices and systems?
• Ubiquitous simulation?
Closing Remarks