A Modular System Architecture for Sensor Data

Processing of ADAS Applications

Michael Darms, Hermann Winner

Propulsion & Drivetrain

Abstract— In this article a modular system architecture for Secondary Safety Systems
fusion of data from environment sensors for advanced driver- Vehicle Stabilising (Primary Safety)
assistance systems (ADAS) is proposed. The architecture allows Vehicle Conducting Systems
different applications to have access to the fused sensor data by Information Systems
processing the data with respect to specific demands of different

Drive train management

Vehicle Stability Control

Transmission Control/
application groups. In the article the growing interconnection of

Engine Management
Pre-Crash Restraint
Suspension Control
functions is illustrated by two examples and the most relevant Applications

Navigation
consequences of this trend are given. The question, if an

ACC
application independent processing of sensor data is feasible is
discussed and the most important aspects for the design of the
system architecture are given. This provides the basis for the
explanation of the system architecture. Components
Surround Sensors
Long Range Radar X
Index Terms— ADAS, Adaptive Filter, Sensor Fusion, System GPS + Digital Map X (x)
Architecture Vehicle Dynamics Sensors
Wheel Speed X X X X X (x) X
Longitudinal Acceleration X (x) (x) X (x)
Lateral Acceleration X (x) (x) (x)
I. INTRODUCTION Yaw Rate X X X X X (x)
Steering (Wheel) Angle (x) (x) X X (x)

A DVANCED Driver Assistance Systems (ADAS) rely on

information about the vehicle environment provided by
different types of sensors, like Radar, Video or Lidar for
example. Up to now the information provided by a sensor is
usually connected to a specific application directly, like Radar
to Adaptive Cruise Control (ACC) or Video to Lane
Departure Warning (LDW). In the future costs will be reduced
by making the information delivered by (environment) sensors
available for use by different applications (see for example
[1], [2]). In the first part of the article the growing
interconnection of functions is illustrated and the main
consequences of this trend are pointed out.
One of the most important aspects is the structure
(architecture) of data processing in terms of software and
hardware. Theoretical models (see for example [3], [14]) have
been developed for different data fusion architectures, each of
which is associated with specific advantages and
Dipl.-Wirtsch.-Ing. Michael Darms is research associate at the Department
of Automotive Engineering at Darmstadt University of Technology,
Petersenstr. 30, D-64293 Darmstadt, Germany (e-mail: mdarms@fzd.tu-
darmstadt.de).
Prof. Dr. rer. nat. Hermann Winner is head of this department (email:
hwinner@fzd.tu-darmstadt.de).
This article was prepared within the framework of a joint research project
with Continental Teves AG & Co oHG, Guerickestr. 7, D-60488 Frankfurt,
Germany. Dipl.-Ing. Maxim Arbitman is the project manager.
Actuators
Brake Torque
Engine Torque
X
X
X
X X X
Restraint actuators X
Wheel Spring/Damper (x) X
Fig. 1. Allocation of Sensors and Actuators to Applications (Today).
disadvantages.
In the article the question, if an application independent
data processing is practicable is discussed. Then the main
aspects for the development of a system architecture due to
the multisensor approach are explained. The architecture itself
is then explained in section IV with reference to the results of
the preceding sections.
II. GROWING INTERCONNECTION OF FUNCTIONS
A. Brief look into History
Electronic linking of systems in the automobile started in
the Eighties with the development of the electronic
transmission control and the traction control system, which
both interact with the engine management. In the first
generation of these systems, it was only possible to use
dedicated point to point connections for signal transfer
(mostly PWM based). Specific functions were involved such
as, for example, ignition advance angle adjustment for rapid
engine torque reduction. For a longer lasting torque reduction
of the traction control system, even a stand-alone component
 Propulsion & Drivetrain
Secondary Safety Systems
Vehicle Stabilising (Primary Safety)
Vehicle Conducting Systems
Information Systems

Applications

Side Obstacle Detection/

Lane Departure Warning

Drive train management

Vehicle Stability Control
Lane Change Assistant

Lane Keeping Support

Adaptive Light Control
Park Distance Control

Transmission Control/
(Semi-)Active Parking
Congestion Assistant

Engine Management
Vision Enhancement

Pre-Crash Restraint
Suspension Control

Collision Mitigation
ACC + Stop&Go
Backing Aid
Navigation
Components
Surround Sensors
Short Range Sensor (front) X X X X X (x)
Short Range Sensor (side) (x) X X (x)
Short Range Sensor (rear) X X (x)
Long Range Radar (x) (x) X (x) (x) (x) X
Mono-Video (x) (x) X X (x) (x) (x) (x) (x)
Stereo-Video (x) (x) X X X X X (x) X
GPS + Digital Map X (x) (x) (x) (x) (x) (x) (x)
Vehicle Dynamics Sensors
Wheel Speed X (x) X (x) (x) X X X X X X X X X X (x) X
Longitudinal Acceleration (x) X X (x) (x) (x) (x) X (x)
Lateral Acceleration (x) (x) X (x) X X (x)
Yaw Rate X (x) (x) (x) (x) (x) X X (x) (x) X X X X (x)
Vehicle Travel X (x) X X X (x)
Steering (Wheel) Angle (x) (x) (x) (x) X X (x) X X X X X (x) X
Actuators
Brake Torque X X X X X
Engine Torque X X X (x) X X
Steering Angle/Torque X X (x) (x) X
Restraint actuators X (x)
Wheel Spring/Damper (x) X (x)

Fig. 2. Allocation of Sensors and Actuators to Applications (Future).

was developed with which the air feed was controlled in series expansion in networking.
with the throttle. If you analyze this matrix, you get 135 links. In relation to
Another reason for the networking of functions occurred as an application, about 8 components are used, which in turn
a result of the desire to save on redundant sensors (for must be available to roughly 7 applications. The savings
example: use of ABS rpm sensors for transmission control and gained by networking are significant (135-18=117); the same
tachometer). applies to the corresponding increase in interfaces.
As a result of the foreseeable electronic networking, several Even if this compilation of components and
approaches towards a vehicle compliant data bus system came sensors/actuators is more or less arbitrarily, it is possible to
about at virtually the same time from which CAN made its derive from this that approximately five times more functional
mark as a standard. interfaces are created as a result of the multiple use of
components.
B. Present Situation
If you examine current applications in a car, a matrix can be D. Consequences
created, which allocates sensors and actuators to applications The use of information from different sensors in multiple
(see Fig. 1; components, which only serve one core function, applications results in a high level of interconnection. The
are not listed). In this selection, on the one hand one of the most important consequences are:
eleven components on the average serves 3.5 of the eight • Bandwidth, network topology. Even today's data
applications. On the other hand every application uses about networks make full use of the entire bandwidth, which usually
5.5 commonly used components. In total, there are 39 links. means that several bus systems find their way into higher class
The number of components has been reduced by 39-11=28 cars. These bus systems are assigned to different clusters
due to networking. They are replaced by 28 system-wide which exchange very little data with one another by bridges or
interfaces. gateways (e.g. power train, body electronics, multimedia). A
higher degree of networking results, on the one hand, in a
blurring of the clearly defined limits of the clusters, and, on
C. Future Situation
the other, in an overall increase in data volume. Bus systems
If you look further into the future, a significantly higher such as FlexRay or TTP, which allow higher transmission
degree of networking will be attained in a higher value car rates than CAN, offer useful solutions.
(see Fig. 2; for further simplification, interconnections due to • Software redundancy. Sensor signals are further
X-by-wire systems, were ignored). Driver assistance and processed and combined with other signals using model
safety systems, which for their part are being introduced with assumptions. If each function processes the data for itself,
new environment sensors, are mainly responsible for the
 fx
fx
fx
fx
t
t4
t3
t2
t1
x
Fig. 3. Four distance measurements (x) at four different times (tn) with
corresponding probabilities (fx). One possible model explaining the data is an
object moving with constant velocity (solid line); another possible model is a
non moving object (dashed line).
significant overlapping occurs and similar software modules
are developed without being coordinated with one another.
• Reliability, failure behavior. Reducing the number of
components by interconnecting functions generally leads to an
increase in the reliability of the system as a whole. However,
the failure of a given single component that is essential to
several systems can result in the failure of all of these systems.
• Development process. Despite the advantages offered
by computer-aided development support, it becomes
significantly more difficult to maintain a complete overview
of systems characterized by extensive interconnection of
functions, even if comprehensive specifications are defined.
Furthermore, several different partners, some outside the
company, are involved in the development of the overall
system. Given a reasonable structure for the overall system
and corresponding interfaces, the development process can be
efficiently arranged and further development can be controlled
more effectively.
III. SENSOR DATA PROCESSING IN VIEW OF DIFFERENT
APPLICATIONS
A. Model Based Filtering
The use of the same sensor data for different applications as
described above raises the idea, to have an application
independent data processing. Using the raw data, one would
design an estimator that processes the data in such a way, that
one has the best estimate that all applications can work with.
There is still the question, if this is feasible (see also [10]
for example). An estimated value is defined using a model of
the dynamics of actual parameter and the knowledge of the
measurement error. A very common estimator for dynamic
systems is the (Extended) Kalman Filter (KF, see e.g. [11]).
The Problem is that there is no such thing as a "correct"
model, especially when you observe an object over a longer
period of time. An observed car may stand still, it may move
with a constant velocity or a constant acceleration, for
example.
Fig. 3 explains this problem with a constructed example. If
you look at the four distance measurements at four different
Precision
A
B
Delay Time
Fig. 4. Demands for Delay-Time and Precision from different Applications
(A, B)
times with the corresponding probabilities, at least two models
may be correct: you may have observed an object moving
with a constant velocity or an object which is standing still.
The measured data can be explained by both models, even if
the first one is more likely in this example.
What you can also see from this example is that the
model/assumption describing the dynamic behavior best can
only be determined after the gathered data has proved it to be
significant.
Thus depending on the application, assumptions regarding
the choice of a model must be made beforehand in order to
achieve the best estimation for the relevant situation “in time”.
The estimation may also be incorrect if the assumptions are
wrong. Thus a trade-off between precision and delay time has
to be made, based on the function of the application (see also
Fig. 4).
In the case of an ACC system, the assumption that a vehicle
moves as would be expected for a “normal driving” may be
appropriate in nearly all situations encountered on a highway.
The assumption is usually not applicable to safety systems,
however. When filtering the data based on the “normal
driving” assumption, the state estimation of an object may
well be too “conservative” for a collision-avoidance algorithm
(see also [8], [10]).
B. Adaptive Filters
As a way to cope with the problem of varying models, one
usually uses an adaptive filter. A very common approach is
the Interacting Multiple Model Estimator (IMM, see e.g.
[11]). This approach assumes that the observed object is in
one of a finite number of modes, which can be described by a
corresponding suitable model. The algorithm then does a soft
switching between these modes according to computed mode
probabilities [11, p. 466].
Nevertheless the models/modes are motivated by the
application. If you do not expect a car doing a full brake, it is
not necessary to have a model for this mode available. It may
even be a bad decision to have too many (and maybe similar)
models in the IMM estimator, as the algorithm then may not
be able to “distinguish” between them, causing a lack in filter
performance [11, p. 475].
Furthermore adding a new application requiring a new
model to the overall system will modify the behavior of this
“system wide” filter and you would have to make sure again
that the other applications can work with this modification
without a loss in performance and without errors.
C. Application based Data Processing
As a result of this discussion it seems to be more efficient to
have different data processing for different applications. As a
way to reduce complexity one can define application groups,
which rely on the same filter/estimator, even if this might not
be optimal in all circumstances. (Remark: the use of the IMM
estimator as such an application specific filter may still be a
good choice).
IV. SENSOR FUSION ASPECTS
A. General Aspects
Besides having the data from one sensor available for
different applications it is also possible to combine the data
from different sensors.
In general terms, the following four fundamental aspects
apply to a multisensor system (according to [6], [7]):
• Redundancy. Measurement precision can be enhanced
by using redundant information. In addition, the fault
tolerance of the overall system increases, as the failure of one
sensor does not necessarily result in the failure of the system
as a whole.
• Complementarity. Means that information is enhanced
by input from different sensor types. This can be visualized
both spatially (e.g. expanding the covered range) and in
relation to an observed object (e.g. independent position and
velocity measurements). Where applicable, ambiguities can
also be resolved using complementary sensors.
• Time precision. Acquisition speed can also be increased
with the aid of multiple sensors. This can be accomplished, for
example, through parallel processing of information or even
through an appropriate chronological configuration of sensor
information.
• Overall costs. Costs and benefits associated with the use
of multiple sensors must be compared with the costs and
benefits of a system with fewer sensors.
B. Data Association
If there is more than one object to be tracked (“multitarget
tracking”), measurements acquired by the sensor(s) must be
associated to the specified tracks (“data association”; see [3],
for example). This relationship is established by predicting
measurements on the basis of a model and calculating a
predicted range within which the measurements should fall
(“gating”). Afterwards, the measurements are assigned to the
track which matches best, whereby different optimization
strategies may be used (see e.g. [3], [13]).
In addition to information about the (dynamic) states of an
object used in an application, additional information
accumulated by the sensor may be used during the association
process (e.g. bitmap information about an object in a video
sensor). This will result in fewer errors in the association
process as compared to relying on state estimations only, as
more information is used in this step.
V. ARCHITECTURE ASPECTS
A. Centralized versus Decentralized Architecture
With regard to data fusion at object level, literature
distinguishes in general between three different configurations
(see e.g. [14]): centralized, autonomous (decentralized) and
hybrid data fusion architectures. The configurations are
distinguished in greater detail in [3]. This article discusses
only the two extreme configurations (centralized and
decentralized) in order to illustrate the most important
differences.
In the centralized version, all data processing is performed
in a central unit. All available information (raw sensor data) is
consolidated in a central unit, without pre processing. A
central database for all applications is created on the basis of
an all-encompassing model.
The opposite pole is the decentralized structure. In this
configuration, all data is processed by the sensor (model-
based filtering, object tracking). The tracks generated in this
way are fused at a higher level using a track-to-track
correlation based on the estimated states in the sensors.
Both configurations offer advantages and disadvantages. In
the central configuration, all data is transmitted to the central
unit without loss of information and can be processed with
maximum accuracy. Furthermore, there is no danger of
inconsistencies due to dissimilar model assumptions in the
different processing units.
On the other hand, a large bandwidth is required to transmit
data from the sensor to the central unit. All processing power
is concentrated at one point in the architecture and has to be
dimensioned accordingly. The most significant disadvantage
is that it will be difficult to modify or extent the system in any
way, as every change requires reoptimization of the central
unit.
In the second case, (decentralized architecture) only a small
bandwidth is needed to transmit the information (tracks in this
case). System modifications/expansions can also be realized
with relative ease, as all data is processed in the sensor.
One problem associated with the use of the decentralized
arrangement is that the consistency of model assumptions over
the whole system is not guaranteed. Different sensors may use
different assumptions to filter the data. Furthermore, errors in
individual sensors are statistically dependent on each other,
and this must be taken into consideration as well (see e.g.
[11]).
Table I summarizes the most important aspects of this
discussion. The results suggest that a hybrid architecture
incorporating the advantages of both approaches is desirable.
B. Feedback from the Applications to the Sensors
If the characteristics of sensors are variable (e.g. field of
view, detection rage), it is important to ensure defined
feedback from applications to sensors. Ultimately, one can
only proceed from the specific application in order to
determine which aspect of the environment is of interest at
any given time and thus to define the sensor configuration.
 TABLE I
CENTRALIZED VERSUS DISTRIBUTED DATA PROCESSING
Requirement centralized decentralized
No loss of information
+ -
High degree of accuracy + -
Consistent model assumptions + -
Low bandwidth - +
Homogenous processor load - +
Easily modifiable - +
However, different applications can pose conflicting
requirements, as in the case of a variable detection range of a
medium- to close-range sensor used in such applications as
Stop&Go, Parking Assistant or airbag preconditioning.
These conflicts have to be resolved. In this process, the
applications must be informed, whether a requirement has
been met or not, in order to be able to react accordingly.
C. Classification of Objects
In addition to the straight estimation of state variables for a
recorded object (e.g. velocity, etc.), classification of the
objects (i.e. identification of the type of an object.) is
important for both applications and state estimation
algorithms.
In addition to enabling applications to derive longer-term
strategies, this information also makes it possible to select
appropriate model assumptions for use in estimating state
variables (a traffic sign has different physical constraints in
comparison to a car, for example).
Moreover, a long-term prediction of the behavior of an
object that is based on more than physical constraints alone is
possible only if the type of object is known (the behavior of
an observed pedestrian may differ from that of a detected
cyclist).
VI. MODULAR SYSTEM ARCHITECTURE
The following section describes a system architecture
developed on the basis of the results discussed above (see also
[9]). It is shown in Fig. 5.
A. Sensor Level
The sensors represent the lowest level. Raw data signal
processing may be preconditioned with the aid of data traced
back from the fusion level. This is useful, for example, in
video image processing (definition of search areas) as a means
of reducing computational load (see for example [12]).
The measured data is then passed on to the association
function. Here, whenever possible, the measured data is
associated to the previously established tracks, which are
traced back from the fusion level. With the association step
established at the sensor level, additional information from the
sensor that is not reported to the higher-level data processing
function at the fusion level can be used for this task. In this
way, association is not based on the estimated states of an
ApplicationLevel
Application Level
Application Level
Application
Requirement/
L2 View Feedback
Fusion Level
Coordination of
Requirements
Fa F1 ... Fn
L1 Object-Class
Requirement
Association II, Classification, for Sensor
Verification
L1 View
Sensor Level
Translation/Compression Translation
Sensor-
Association I configuration
Sensor – Signal Processing
Fig. 5. Modular System Architecture for Fusing Environment Sensor Data
object (such as velocity, position, etc.) alone, but also on such
sensor-specific information as the reflectivity measured by a
laser scanner or image-data detected by a video sensor. The
association step may be skipped for sensors in cases in which
an association step at the sensor level is undesirable (due to
the sensor principle or to cost considerations). Data that
cannot be associated (e.g. objects which are detected for the
first time) will be sent to the fusion level without association.
The next step at the sensor level is the
translation/compression, through which data is converted into
a generalized view (so called “L1 view”). Wherever possible,
data in L1 view has not yet been subjected to model-based
filtering. This takes place at the fusion level only, thus
guaranteeing that the model remains uniform over the entire
fusion process.
Translation makes it possible to standardize data from
sensors based on different principles or from different
generations of sensors. However, as it is not possible to obtain
a standardized view of all sensor data conceivable (new
principles might still be developed), the L1 view must be
capable of expansion.
Separation between pre processing and filtering will
actually not be possible in every case. Thus a model is
required even for the interpretation of radar data readings, for
example. If separation is not possible, the minimum
requirement should be that potential falsification and
misinterpretation of the data based on model assumptions is
minimized and filtering is performed only to the extent
necessary for a particular identification.
B. Fusion Level
Data entering the fusion level is first verified to minimize
errors. Classification is performed using all available sensor
information and data already obtained from a track. Data
which is not associated to a track so far is associated at this
stage, based on state variables only. Data that cannot be
associated and which has passed the verification step is used
to establish a new track.
There are two major advantages to incorporating a second
association step at the fusion level, in addition to first step at
the sensor level. First of all, the association step may be left
out at the sensor level in order to reduce costs; secondly, there
is a chance that two sensors may detect a new object at the
same time and report it to the fusion level as a new track
(depending on the timing of the sensors). If an association step
is incorporated at the fusion level resulting multiple track
initializations will be minimized.
The associated and classified measurements are then passed
on to a filter bank, which consists of different model-based
filters customized for different types of applications and one
filter especially designed for the requirements of the
association. This filter works separately from the filters
adapted for the applications needs by using purely application
independent object properties.
Each application dependent filter can be (statically)
assigned either to one individual application or to a group of
applications. In this way, multiple implementations are
avoided and sufficient flexibility remains for the various
requirements of different applications.
Filter parameters can be influenced on the basis of
identified object type and dynamic application requirements
(which must be coordinated in the case of a multiple
allocation). The result of the estimation process is the so
called “L2 view”, which is passed on to the application level
and also traced back to fusion and sensor levels.
In contrast to the L1 view, the L2 view is independent of
sensor principles. Further development of this view will
depend solely on new application requirements.
C. Application Level
The applications receive data in the L2 view and can select
the filter properties (models), which best match their
requirements. From the standpoint of a given application,
there is only one “virtual sensor”, which delivers data in
accordance with the requirements of the specific application.
The process through which information is generated remains
transparent.
The properties of the virtual sensor (e.g. the dynamics of
the data filtering process, the field of view of the “virtual
sensor”) can be influenced dynamically by the applications.
The requirements are prioritized at the fusion level and
converted into generalized sensor requirements, which are
again formulated in a standardized language (as in the L1
view) and converted into a special sensor configuration at the
sensor level.
Because different requirements may conflict and thus
remain unfulfilled for some applications, feedback from the
fusion level to the application level enables the application to
react accordingly.
VII. CONCLUSIONS & SUBSEQUENT STEPS
The definition of a system architecture for sensor data
fusion of environment sensor data for Advanced Driver
Assistance Systems must account for the fact that processing
of sensor data is application dependent.
The system architecture presented in this article takes this
into account. Furthermore it allows the addition of new
applications without any modification of the underlying filters
of existing applications and so allows keeping already tested
modules in the ongoing development process.
The concept is open with regard to both sensors and
applications. The communication bandwidth between the
different layers is reduced significantly as compared to that
required for raw data transmission, while error propagation is
minimized and remains transparent.
The architecture itself is independent on the hardware
topology. It might be implemented either on a single processor
system in which the different layers correspond to different
software modules or on a multiprocessor system in which each
instance of the sensor level or the application level
corresponds to one piece of hardware and the L1 and L2
views are transmitted via a bus system.
Defining a standardized L1 and L2 view acceptable to
multiple manufactures would then make it possible to use
hardware (and software) modules from different suppliers
interchangeably.
In the next step, the architecture will be implemented in a
test vehicle during the project PRORETA [14]. The two main
hypotheses (“Different application groups need different
model assumptions in order to work optimally” and “Data
association at the sensor level is more reliable than data
association at the fusion level”) of the system architecture will
be analyzed in a real environment.
REFERENCES
[1] Becker, J.-C., “Fusion der Daten der objekterkennenden Sensoren eines
autonomen Straßenfahrzeugs.” Dissertation, TU-Braunschweig,
Fortschr.-Ber. VDI Reihe 8, Nr. 948, VDI Verlag, 2002.
[2] Klein, L. A., “Sensor and Data Fusion Concepts and Applications,” 2nd
edition, SPIE, 1999.
[3] Bar-Shalom, Y., Xiao-Rong, L. “Multitarget-Multisensor Tracking:
Principles and Techniques,” 3rd printing, YBS, 1995.
[4] Sparbert, J., Dietmayer, K., Streller, D., “Lane Detection and Street Type
Classification using Laser Range Images,” in Proc. IEEE Intelligent
Transportation Systems Conference, 2001.
[5] Vukotich, A., Kirchner, A., “Sensorfusion für Fahrerassistenzsysteme,”
in VDI-Berichte 1646, pp. 857-875, VDI Verlag, 2001.
[6] Lou, R.C., Kay, M.K., “Multisensor Integration and Fusion in Intelligent
Systems”, in: Autonomous Mobile Robots Volume 1, IEEE Computer
Society Press, Los Alamitos, California, 1991.
[7] Joerg, K.-W., “Echtzeitfähige Multisensorintegration für autonome
Mobile Roboter,” Mannheim, etc., BI-Wiss.-Verl., 1994.
[8] Darms, M., Winner, H., „Fusion von Umfelddaten für
Fahrerassistenzsysteme“, in Stiller, C., Maurer, M., Workshop
Fahrerassistenzsysteme FAS2003, pages 13-16, 2003.
[9] Darms, M., Winner, H., “A General System Architecture for Fusion of
Data from Environment Sensors for ADAS” ITS 2005, Hannover,
03.06.2005.
[10] Randler, M, Wilhelm, U., Lucas, B., “Anforderungen von Komfort- und
Sicherheitsfunktionen an die Umwelthypothese der Umfeldsensorik,” in
Stiller, C., Maurer, M., Workshop Fahrerassistenzsysteme FAS2003, pp.
42-44, 2003.
[11] Bar-Shalom, Y., Xiao-Rong, L. “Estimation with applications to tracking
and navigation,” Wiley, 2001.
[12] Franke A.G., Levi, P. “Robust Vehicle Tracking Fusing Radar and
Vision,” International Conference on Multisensor Fusion and Integration
for Intelligent Systems, 2001.
[13] Hall, D.L., McMullen, S. “Mathematical Techniques in Multisensor Data
Fusion,” 2nd edition, Artech House, 2004.
[14] Project Homepage PRORETA, www.iat.tu-darmstadt.de/proreta,
22.04.2005.