Accessible and Assistive ICT: Veritas

Accessible and Assistive ICT
VERITAS
Virtual and Augmented Environments and Realistic User
Interactions To achieve Embedded Accessibility DesignS
247765
Integration of Multimodal Interfaces into the

VERITAS Simulation and Testing Framework
Deliverable No. D2.8.2

SubProject No. SP2 SubProject Title Innovative VR models, tools
and simulation environments
Workpackage WP2.8 Workpackage Multimodal interfaces
No. Title
Activity No. A2.8.4 Activity Title Integration of the multimodal
interaction tools into the
VERITAS VR Testing and
Evaluation Environment
Authors Panagiotis Moschonas, Athanasios Tsakiris,
Georgios Stavropoulos, Sofia Segouli, Ioannis
Paliokas, Dimitrios Tzovaras (CERTH/ITI)
Status F (Final)
Dissemination Level: Pu (Public)
File Name: VERITAS_D2.8.2_final
Project start date and 01 January 2010, 48 Months

duration
VERITAS D2.8.2 PU Grant Agreement # 247765
Version History table

Version Dates and comments
no.
1 First version created on February 2012, analysing the methodological

framework.
2 Second version created on April 2012 including the components details.
3 Draft version created and sent for peer review on May 2012.
June 2012 iii CERTH/ITI

Table of Contents
Executive Summary....................................................................... 10
1 Introduction ............................................................................. 12
2 Architecture of the Multimodal Interfaces ................................. 13
2.1 Positioning in the VERITAS Global Architecture ................................. 13
2.2 Local Architecture Analysis ................................................................. 16
3 Methodology ............................................................................ 20
3.1 Adaptation Stage ................................................................................. 21
3.1.1 Assignment of Multimodal Interfaces to the adapted environment ......... 22
3.1.2 Analysis of the Simulation Model based on the Multimodal Interfaces
Models 22
3.1.3 Definition of the alternative tasks ........................................................... 22
3.2 Runtime Stage .................................................................................... 23
3.2.1 Analysis of the Virtual User Models disability model.............................. 23
3.2.2 Modality Compensation and Replacement process ............................... 27
4 Integration Status of the Multimodal Interfaces Architecture
Components .................................................................................. 31
4.1 Multimodal Interfaces Manager ........................................................... 31
4.1.1 Functional Description ........................................................................... 31
4.1.2 Data Flow .............................................................................................. 32
4.1.3 Integration Status .................................................................................. 33
4.1.4 Development Status .............................................................................. 35
4.2 Speech Recognition Module ............................................................... 35
4.2.2 Data Flow .............................................................................................. 35
4.3 Speech Synthesis Module ................................................................... 37
4.3.2 Data Flow .............................................................................................. 38
4.4 Haptics Module ................................................................................... 39
4.4.2 Data Flow .............................................................................................. 40
June 2012 iv CERTH/ITI

4.5 Sign Language Synthesis Module ....................................................... 41
4.5.2 Data Flow .............................................................................................. 41
4.6 Symbolic Module ................................................................................. 43
4.6.2 Data Flow .............................................................................................. 43
4.7 Screen Reader Module ....................................................................... 45
4.7.2 Data Flow .............................................................................................. 45
4.8 Screen Magnifier Module .................................................................... 47
4.8.2 Data Flow .............................................................................................. 48
5 Application Layer Interface ...................................................... 50
5.1 Multimodal Interfaces Models Management API ................................. 50
5.2 Interfacing Modules Management API ................................................ 54
6 Conclusions and Future Plans ................................................. 59
References .................................................................................... 60
June 2012 v CERTH/ITI

List of Figures
Figure 1: The global architecture of Veritas. The box with the red dashed line
depicts the WP2.8 products, i.e. the Multimodal Interfaces Models and the
Multimodal Interfaces Manager (VerMIM). ....................................................... 14
Figure 2: The local architecture of the Multimodal Interfaces. .......................... 17
Figure 3: Portion of a VUM file containing the <disabilityModel> section;
The depicted disability model describes a user with two disabilities:
Gonarthrosis and Rheumatoid Arthritis. ........................................................... 18
Figure 4: In this figure the Virtual User Model information that is valuable to the
VerMIM component is depicted; the disabilitys ICF and its affected task
information are used in the modality compensation step. ................................. 27
Figure 5: Data flow of the modality compensation process for the automatic
simulations, i.e. for simulations that a real user is not present. ........................ 29
Figure 6: Data flow of the Modality compensation process in the immersive and
interactive simulations. ..................................................................................... 30
Figure 7: Data flow and implementation diagram of the Multimodal Interfaces
Manager. .......................................................................................................... 33
Figure 8: Data flow of the Speech Recognition module. ................................... 36
Figure 9: Data flow of the Speech Synthesis module. ...................................... 38
Figure 10: Data flow of the Speech Synthesis module. ................................... 40
Figure 11: Data flow of the Sign Language Synthesis module. ........................ 42
Figure 12: Data flow of the Symbolic Module. .................................................. 43
Figure 13: Data flow of the Screen Reader module. ........................................ 46
Figure 14: Data flow of the Screen Magnifier module. ...................................... 48
List of Tables
Table 1: Table that represents which tools are connected with the VerMIM and
what features are provided to the tools after the integration process. .............. 16
Table 2: The two stages of adding and using the Multimodal Interfaces in the
VERITAS applications. ..................................................................................... 21
Table 3: The Multimodal Interfaces adaptation process steps.......................... 22
Table 4: ICF codes, their descriptions and the affected user modalities. ......... 26
Table 5: The Multimodal Interfaces Components. ............................................ 31
Table 6: Integration status of the Multimodal Interfaces Manager. ................... 34
Table 7: Development status of the Multimodal Interfaces Manager. ............... 35
Table 8: Integration status of the Speech Recognition Module. ....................... 37
Table 9: Development status of the Speech Recognition Module. ................... 37
Table 10: Integration status of the Speech Synthesis Module. ......................... 39
June 2012 vi CERTH/ITI

Table 11: Development status of the Speech Synthesis Module...................... 39

Table 12: Integration status of the Haptics Module. ......................................... 41
Table 13: Development status of the Haptics Module. ..................................... 41
Table 14: Integration status of the Sign Language Synthesis Module. ............. 43
Table 15: Development status of the Sign Language Synthesis Module. ......... 43
Table 16: Integration status of the Symbolic Module. ....................................... 44
Table 17: Development status of the Symbolic Module. ................................... 45
Table 18: Integration status of the Screen Reader Module. ............................. 47
Table 19: Development status of the Screen Reader Module. ......................... 47
Table 20: Integration status of the Screen Magnifier Module. .......................... 49
Table 21: Development status of the Screen Magnifier Module. ...................... 49
June 2012 vii CERTH/ITI

List of Abbreviations
Abbreviation Explanation
Ax.y.z VERITAS Activity
CAD Computer-aided design

CAL3D, Cal3D 3D Character Animation Library
DLL Dynamic Link Library
Dx.y.z VERITAS Deliverable
DoF Degree of Freedom
GUI Graphical User Interface
GVUM Generic Virtual User Model

IK Inverse Kinematics
LGPL GNU Lesser General Public License
ODE Open Dynamics Engine
OGRE Open Source 3D Graphics Engine
OS Operating System
OSG OpenSceneGraph
PoI Point of Interest
RoM Range of Motion
RRT Rapidly-exploring Random Tree
SP Sub-project
WP Work-package
VR Virtual Reality
VUM Virtual User Model
June 2012 viii CERTH/ITI

List of Abbreviations for VERITAS tools

Abbreviation Explanation
MOUSI Veritas User Model Ontology to UsiXML Converter
VerGen Veritas User Model Generator
VerMP Veritas Model Platform
VerSEd-3D Veritas 3D Simulation Editor
VerSEd-GUI Veritas GUI Simulation Editor
VerAE Veritas Avatar Editor
VerSim-3D Veritas 3D Core Simulation Viewer
VerSim-GUI Veritas GUI Core Simulation Viewer
IVerSim-3D Veritas 3D Immersive Simulation Viewer
VerIM Veritas Interaction Manager
June 2012 ix CERTH/ITI

Executive Summary
The D2.8.2 manuscript describes the integration process of the VERITAS
WP2.8 Multimodal Interfaces tools with the Simulation Framework and more
precisely, with the tools that have been implemented as part of the conducted
work of WP2.1.
The main component developed as part of WP2.8 is the VERITAS Multimodal
Interfaces Manager, or shortly VerMIM. Its primary objective is to provide a set
of Multimodal Interfaces tools and methods for testing, during all the
development stages, the accessibility of multimodal interface solutions
addressed to users with special needs, such as older adults and people with
different types of impairments. VerMIM exports to the simulation framework the
functionality of managing the Mulimodal Interfaces Models (A2.8.2). Moreover, it
contains the modality compensation methods (A2.8.3) that have been created
to improve the interaction efficiency and accessibility for the users with
disabilities.
More specifically, the inclusion of the Multimodal Interfaces toolset into the
VERITAS simulation framework tries to achieve three core objectives: a) to
support the interaction process related to user interfaces of different modalities,
b) to formalize and manage properly the user interaction process via the
multimodal interface models, and c) to provide the functionality of an advanced
modality compensation process based on prioritisation rules to the rest
VERITAS accessibility assessment tools.
The conducted work of VERITAS WP2.8 has been devoted to the development
of tools which will allow the engineers to design multimodal interfaces
specifically devoted to the demands of persons with special needs (older people
and people with disabilities). The generated Multimodal Interface Models are
able to simulate the various steps of the interaction process of mono- and
multimodal interfaces and link them to the sensorial capabilities of the users.
These models have been integrated into the VERITAS Framework and are
used in the Modality Compensation process.
Additionally, various modality interfaces have been developed and connected to
the VerMIM component. The interfaces include speech synthesisers and
recognisers, haptic devices manipulators, sign language synthesisers, alerting
modules, screen readers and screen magnifiers. The inclusion of these
interfaces into the VerMIM has taken into account not only the integration of
these tools to the VERITAS platform, but also to existing simulation platforms.
Other interfaces have been taken from 3rd party open source tools and others
have been developed from scratch. The variety of the different interface
characteristics has been homogenized under one common Multimodal
Interfaces application layer interface. The supported unimodal interfaces may
run simultaneously to produce interfaces of multimodal nature.
The deliverable starts with an introductory section. After that it describes the
Multimodal Interfaces Manager placement inside the global VERITAS
architecture is given. The methodology that has been followed for performing
the Modality Compensation process is then depicted and details concerning the
data flow, the integration status and the development status are provided for
each of the Multimodal Interfaces components. The document continues with
June 2012 10 CERTH/ITI
the descriptions of the basic classes of the provided Application Layer Interface;
classes which concern the management of the interfaces and the models are
shortly described with their most important functions. Such descriptions are
valuable to the developers of 3rd party tools that will intent to be connected with
the VerMIM.
Concluding, in their current phase most of the tools are at their final stage and
the integration process in all of them has been almost completed. Extra
functionality may be added to the tools as the integration and optimization
process continues as part of A2.8.5. Additionally, minor changes may have to
be applied to the tools after their validation of their evaluation performance as
part of WP3.6/7.

1 Introduction
The main objective of the D2.8.2 deliverable is to describe the methodology that
has been followed in order to integrate the Mutlimodal Interfaces components
with the VERITAS simulation framework. A bibliographic review of multimodal
interfaces and assistive devices, as well as their addressed user groups, is
omitted from this manuscript, as it is included in the deliverable D2.8.1 [1], the
prequel of this document.
This document contains information about the integration process of the WP2.8
products with the products of the WP2.1 [2]. The integration process resulted
into one core component in which seven interface modules have been pre-
integrated. The core component is the Multimodal Interfaces Manager, or
VerMIM, which is responsible for managing the Multimodal Interfaces Models,
for performing the modality compensation process and for providing access to a
list of seven assistive modules that are implementation of interfaces tools.
The seven interfaces modules are mostly 3rd party tools that when combined
provide to the VERITAS framework the multimodal interfacing layer. The
modules include interfaces based on speech recognition, speech synthesis,
alerting methods, sign language synthesis, haptics support, screen magnifiers
and screen readers. The integration of these interfaces and their management
is crucial for the improvement of the user interactions with the tested product,
especially when they have vision, hearing, motor or cognitive impairments.
The present manuscript is split into six sections, including this one.
In Section 2, the architecture of the Multimodal Interfaces is presented. Initially
the positioning of the VerMIM inside the VERITAS global architecture is given.
Then, an analysis of the local VerMIM and its components is explained.
Section 3 describes the methodology of that is hidden under the modality
compensation process and how the various WP2.8 data are used in the
process. Specifically, this section describes the process in two stages: the
adaptation and the runtime. The first is irrelevant of the Virtual User model and
basically is performed as an adaptation step of the Multimodal Interfaces in the
3D Scene or the GUI interface that is going to be tested. The second step is
automatic and initiates every time a simulation session takes place.
The Multimodal Interfaces developed components are described in Section 4.
For each of the eight components a functional description, data flow, integration
and implementation status is given. The architecture diagrams are also
presented and the data input / output are depicted.
In Section 5, the Multimodal Interface Application Layer Interfaces is described.
The integration of the VerMIM not only targets to the rest VERITAS tools, but
also targets to provide its functionality to 3rd party simulation applications. This
section contains information of the provided classes and their functions.
Finally, in Section 6 conclusions regarding the work performed and the
capabilities of the developed tools are reported and future plans are given.

2 Architecture of the Multimodal Interfaces

This section positions the Multimodal Interfaces Manager (VerMIM) into the
global architecture of VERITAS and describes its internal architectural
components.
2.1 Positioning in the VERITAS Global Architecture

The VERITAS global architecture and the various tools that have been
produced so far are depicted in Figure 1. The two main products of the activities
of the WP2.8 are:
a) The repository of the Multimodal Interfaces Models: which is, basically, a
collection of UsiXML files stored locally.
b) The Multimodal Interfaces Manager (VerMIM) which provides to the rest
VERITAS tools the needed functionality for using the Models and
managing the multimodal interfaces libraries.
As it is depicted, the Multimodal Interfaces Models have been constructed as
part of the work of A2.8.2 and they are stored in a repository. Each model is
stored inside an UsiXML file and is used by the Multimodal Interface Manager
(VerMIM) component to perform the Modality Compensation and Replacement
process.
In this point, it must be clarified that the VerMIM is not a stand alone executable
as it has been implemented as C++ dynamic link library. This means that has to
be integrated into other VERITAS executable in order to be used. More about
the implementation and the integration status of the VerMIM can be read in
Section 4. For now, it must be said that the the VerMIM is integrated with the
VerSEd tools and the VerSim tools.
As it is depicted in the architecture diagram, the VerMIM component exchanges
information with the following tools:
3D Simulation Editor (VerSEd-3D): this tool is used for adapting the 3D
scene file and a simulation model. The VerSEd-3D is used for adapting
the virtual representations of non-ICT products, e.g. moveable and static
objects with mass. Here, the VerMIMs purpose is two-fold: a) for
assigning multimodal interfaces to the virtual scene objects and b) for
analysing the simulation model into alternative task sequences using the
Multimodal Interfaces Models.
3D Core Simulation Viewer (VerSim-3D): this tool is used for previewing
the simulation sessions. It reads the information provided by the VerSEd-
3D tool and performs the simulation of the avatar interactions with the
adapted 3D scene. The whole simulation session runs automatically
without the interference of a real user. The VerMIMs purpose here is to
perform the modality compensation process and if necessary, it replaces
an avatar action with another that suits better to the scenario
circumstances; this process is called the modality compensation and
replacement process.

GUI Simulation Editor (VerSEd-GUI): this tool is used for adapting a

simulation model that is destined to be used for evaluation of an ICT
product. The ICT products are tested by running simulations on their
graphical user interfaces (GUIs). The GUI Simulation Editor adapts
various applications interfaces, such as typical executable programs,
websites or mobile phones interfaces (by using the appropriate mobile
phone emulators). In such cases, the VerMIM is used to assign
alternative interfaces to the GUI, e.g. assignment of vocal commands
which replace the interactions, e.g. mouse clicks, with the GUI.
Figure 1: The global architecture of Veritas. The box with the red dashed line depicts the
WP2.8 products, i.e. the Multimodal Interfaces Models and the Multimodal Interfaces
Manager (VerMIM).

GUI Simulation Viewer (VerSim-GUI): this tool is used for evaluating the
accessibility of a graphical user interface. It supports two modes:
automatic and interactive.
o In the automatic mode the program drives automatically the
mouse pointer based on the parameters of the Virtual User Model
and applies any motor, vision, hearing and cognitive impairments
to the interface. The whole process is driven by a pre-recorded
scenario file captured by the VerSEd-GUI tool. The scenario file is
created by an ideal GUI user, i.e. a user that performs the whole
sequence without any task deviations or mistakes. In this case,
the VerMIM is used to provide alternative task sequences in cases
of scenario failure, based on the interfaces that have been
assigned to the GUI by the VerSEd-GUI tool.
o In the immersive mode the VerSim-GUI takes as input the actual
mouse movements of a real user and applies motor, visual,
hearing and cognitive filtering to the users actions. The VerMIM
can be used in such cases to provide alternative interfacing
capabilities to the user, such as GUI navigation via speech
recognition.
Interaction Manager (VerIM): this VERITAS component is used to drive
the VR Interaction Tools from WP2.7. It is used in combination with the
3D Immersive Simulation Viewer (IVerSim-3D) for simulation sessions in
which an immersed user interacts with the VR environment. In this case,
VerMIM can provide multimodal interfacing tools, such as speech
synthesis, recognition, haptics functionality, etc, to the VR environment.
After the integration process of the VerMIM with the above-mentioned tools
various feature will be added to them. Table 1 summarises the added features.
More about the description and nature of the added features can be read in
Section 3.
VERITAS Features provided by the

Tool / Simulation Type Multimodal Interfaces Manager
Component (VerMIM) after the integration
Assignment of alternative
interfaces to the 3D scene objects.
Simulation adaptation
VerSEd-3D of a 3D scene/product. Alternative task sequences
definition (based on the Multimodal
Interfaces Models).
Modality compensation and
Automatic simulation in
VerSim-3D Usage of alternative interfaces or
3D environments
alternative task-sequences when
the avatar fails in a task.
VerSEd-GUI Simulation adaptation Assignment of alternative

VERITAS Features provided by the

Tool / Simulation Type Multimodal Interfaces Manager
Component (VerMIM) after the integration
of a GUI. interfaces to the GUI components.

Alternative task sequences
definition (based on the Multimodal
Interfaces Models).
Automatic or Interactive
VerSim-GUI Usage of alternative interfaces or
simulation of a GUI.
the avatar fails in a task.
Usage of alternative interfaces or
Simulation of real
VerIM / the avatar fails in a task.
immersed users in VR
IVerSim-3D
environments. Augmentation of the
communication with the VR
environment using the multimodal
interfaces tools (speech
recognition, speech synthesis, etc.)
Table 1: Table that represents which tools are connected with the VerMIM and what
features are provided to the tools after the integration process.
2.2 Local Architecture Analysis

The local Multimodal Interfaces local architecture is depicted in Figure 2. The
basic architectural component is the Multimodal Interfaces Manager (or
VerMIM). This component receives the Virtual User and Simulation Models
information and selects the appropriate Multimodal Interfaces Models when
needed. It is also responsible for providing alternative interfaces to the rest
Veritas Architecture Components.
The Multimodal Interfaces Manager receives three kinds of input data, all of
which are described using UsiXML formatted files. These are:
The Virtual User Model (VUM) files: these files contain information that
describes the disabilities and capabilities of the virtual user. More
specifically, the VerMIM reads from this file the <disabilityModel>
section, e.g. the part of the file that contains the disabilities of the virtual
user (Figure 3). The disability models data are needed to find which
users modalities are affected in order to perform the modality
compensation and replacement process or to select the appropriate
multimodal interfaces.

Veritas Architecture Component
Immersive
Core Simulation VerSEd-3D/GUI IVerSim-3D
Simulation
Platform VerSim-3D/GUI VerIM
Platform
Speech Recognition
Module
Virtual
User Haptics
Model Module
(UsiXML)
Speech Synthesis
Module
Simulation Multimodal Interfaces
Model Manager (VerMIM)
(UsiXML) Sign Language
Synthesis Module
Symbolic Module
Multimodal
Interfaces
Models
Repository Screen Reader
Module
Screen Magnifier
Module
Figure 2: The local architecture of the Multimodal Interfaces.
The Simulation Model files: these kinds of files describe in an abstract

manner the sequence of the tasks of an application scenario. The
simulation model scenario results into one task sequence that is specific
per application area. The VerMIM uses this file to perform the analysis of
the modalities required in each task and then, using the Multimodal
Interfaces Models, produces alternative task sequences that depend on
alternative users modalities.
The Multimodal Interface Models: these UsiXML files are used to
produce alternative task sequences of the default sequence described in
the Simulation Model. Every Multimodal Interface Model contains
alternative task paths of a simple task. Moreover, it contains information
about the required modalities of each task.
The Multimodal Interfaces Manager uses the above files to perform the modality
compensation and replacement process, as well as to select appropriate
interface modules. The available interface modules are the following:
Speech recognition module: used for converting the users speech into
text. This module is appropriate in cases where a user has its vision
modality affected due to visual impairments or with users with reduced
motor capabilities.

Haptics module: used for converting the users motor sensory functions
into tactile feedback. Used in cases of sever vision impaired users.
Speech synthesis module: produces speech from text. Useful for users
with vision impairments.
Sign language synthesis module: generates animation sequences of a
humanoid moving its hands. Useful for the hearing impaired users.
Symbolic module: generates multimodal alerts, i.e. sounds, visual
signals. Useful for users with cognitive, perceptual problems.
Screen reader module: reads the text of various windows applications
using speech synthesis.
Screen magnifier module: magnifies a portion of the screens desktop.
Figure 3: Portion of a VUM file containing the <disabilityModel> section; The depicted
disability model describes a user with two disabilities: Gonarthrosis and Rheumatoid
Arthritis.

Finally, depending on the simulation type (Table 1), VerMIM is connected with a
VERITAS tool and adds extra functionality to it.

3 Methodology
This section describes the methodology of adding the Multimodal Interfaces
functionality to the rest VERITAS tools. The methodology steps are split into two
main stages:
Adaptation stage: this stage contains the users assignment of
alternative interfaces to the simulated environment objects and
components, either 3D scene or GUI, e.g. the assignment of a speech
recognition interface to a virtual telephone device. Moreover this stage
requires the definition of the alternative tasks contained in the Multimodal
Interfaces Models.
Runtime stage: this stage contains the analysis of the Virtual User Model
file and identification of the virtual users affected modalities. It also
contains the prioritization process on the selection of the alternative task
sequences. The modality compensation and replacement process also
takes place in this stage.
The two stages along with the tools involved and the actions that are performed
are presented in Table 2.
Stage Tools Involved Sequence of Actions
Adaptation VerSEd-3D, a. Manually assign the multimodal and

alternative interfaces to the:
VerSEd-GUI
3D virtual objects, if it is about 3D
environment simulation (i.e. non-ICT
products).
GUI components, i.e. buttons, text
input boxes, alert message boxes, etc.
b. Perform an automatic analysis of the
loaded Simulation Model and
automatically produce alternative task
sequences that use a variety of
modalities, using the Multimodal
Interfaces Models.
c. Manually define each alternative task of
each produced task sequence.

Stage Tools Involved Sequence of Actions
Runtime VerSim-3D, a. Automatically, perform an analysis of the

Virtual User Models disability model and
VerSim-GUI,
find the affected modalities.
IVerSim-3D &
b. Perform the modality compensation and
VerIM replacement process. This process is
defined by the presence or not of a real
user in the simulation session:
If automatic simulation, i.e. if not a real
user is present, perform an analysis of
the loaded Simulation Model and
automatically produce alternative task
sequences using the Multimodal
Interfaces Models. Then sort the
alternative sequences list, putting first
the sequences that contain the most
tasks which remain unaffected and
run the first sequence. This process
defines that the modality
compensation is based on
prioritisation rules and involves the
tools: VerSim-3D and VerSim-GUI
(the latter running in automatic mode).
If immersive or interactive simulation,
i.e. if a real user is involved, enable
the interfaces that have been already
defined in the adaptation process and
run the default task sequence defined
in the simulation model. If a task fails,
repeat the process using now a task
with another modality based on the
multimodal interface models. This
process defines that the modality
compensation is based on modality
replacement and involves the VerSim-
GUI (running in interactive mode) and
IVerSim-3D tools.
Table 2: The two stages of adding and using the Multimodal Interfaces in the VERITAS
applications.
3.1 Adaptation Stage

The adaptation stage is performed once per simulation scenario and its
configuration steps are independent of the Virtual User Models. It requires a
Simulation Model file and the adaptation of either a 3D scene or a GUI. The
adaptation stage involves the VerSEd-3D and VerSEd-GUI tools integrated with
the VerMIM component.
The Multimodal Interfaces adaptation process is consisted of three steps,

mentioned in Table 3 and analysed in the following paragraphs.
Multimodal Interfaces Adaptation Performed automatically / manually

Step and other information.
1. Assignment of the Multimodal

This step is performed by the products
Interfaces to the adapted
developer / designer.
environment
This step is performed automatically by
2. Simulation Model Analysis into
the VerMIM component using the
alternative task sequences
Multimodal Interfaces Models repository.
This step is performed by the products
3. Definition of each alternative task
developer / designer.
Table 3: The Multimodal Interfaces adaptation process steps.
3.1.1 Assignment of Multimodal Interfaces to the adapted environment

The first step in the adaptation stage is to assign the interfaces and the
available commands to the 3D environment objects or GUI components.
Initially, product designer or developer must define which of the interfaces will
be available for the adapted scenario. After that the parameters of each
interface have to be specified. For example, if a speech synthesis interface is
defined to be used, voice parameters, such as male/female, language, etc.
must be defined. If the adaptation regards a 3D scene, the interfaces are
assigned to the scenes virtual objects. Otherwise, if the adaptation regards a
GUI, the interfaces are assigned to the GUI components, i.e. buttons, message
boxes, etc. Special other characteristics can be defined, e.g. the requirement of
a specific phrase spoken by the user in the case of the speech recognition
interface.
This interface adaptation information is stored inside the simulation scenario file
by the VerSEd-3D/GUI tools and is readable by the viewer tools.
3.1.2 Analysis of the Simulation Model based on the Multimodal

Interfaces Models
The 2nd step of the multimodal interfaces adaptation is performed automatically
by the Multimodal Interfaces Manager. VerMIM reads the simulation model and
analyses it into a sequence of tasks. For each task the VerMIM searches inside
the Multimodal Interfaces Models repository to find alternative tasks (or mini-
sequences of tasks) that replace the modalities of the default task sequence.
Then it generates a list of alternative task sequences based on all task
combinations.
3.1.3 Definition of the alternative tasks

The final step of the adaptation process requires the definition of each task
contained in the sequence list of the previous step. The tasks are defined by the
designer/developer. This step is the most time consuming one of the three and

requires the assignment of rules (based on a conditions and result system) to

each task [3].
Having completed this task, the Multimodal Interfaces adaptation information is
stored inside the simulation scenario file which is readable by the VerSim tools.
3.2 Runtime Stage

The runtime stage uses the information from the adaptation process and
augments the simulation sessions with Multimodal Interfaces. This stage is
performed by the VerSim-3D, VerSim-GUI, IVerSim-3D tools and is parted by
two main steps: a) the analysis of the Virtual User Model file and b) the Modality
Compensation and Replacement process. The runtime stage requires both the
Simulation and Virtual User Model.
3.2.1 Analysis of the Virtual User Models disability model

In this step, VerMIM reads the disability model section of the Virtual User Model
file. More specifically, it identifies the affected modality of each users disability
element. This is done by using the ICF entry of the disability xml element. The
ICF entry is the code of the International Classification of Functioning, Disability
and Health [4]. The ICF codes are mapped to their affected modalities internally
in the VerMIM (Table 4). Besides the ICF definitions the Multimodal Interface
Manager reads the affected tasks of each disability (whereever they are
present) and makes use of the later in the modality compensation process.
ICF Code Description Affected modality
b110 Consciousness functions cognitive
b114 Orientation functions cognitive
b117 Intellectual functions cognitive
b122 Global psychosocial functions cognitive
b126 Temperament and personality functions cognitive
b130 Energy and drive functions -
b134 Sleep functions -
b139 Global mental functions, other specified and cognitive

unspecified
b140 Attention functions cognitive
b144 Memory functions cognitive
b147 Phychomotor functions cognitive

b152 Emotional functions cognitive
b156 Perceptual functions cognitive
b160 Thought functions cognitive
b164 Higher-level cognitive functions cognitive
b167 Mental functions of language cognitive
b172 Calculation functions cognitive
b176 Mental function of sequencing complex cognitive

movements
b180 Experience of self and time functions cognitive
b189 Specific mental functions, other specified cognitive

and unspecified
b198 Mental functions, other specified cognitive
b199 Mental functions, unspecified cognitive
b210 Seeing functions vision
b215 Functions of structures adjoining the eye vision
b220 Sensations associated with the eye and vision

adjoining structures
b229 Seeing and related functions, other specified vision

and unspecified
b230 Hearing functions hearing
b235 Vestibular functions hearing
b240 Sensations associated with hearing and hearing

vestibular function
b249 Hearing and vestibular functions, other hearing

specified and unspecified
b250 Taste function tasting
b255 Smell function smelling
b260 Proprioceptive function cognitive

b265 Touch function haptic
b270 Sensory functions related to temperature and cognitive

other stimuli
b279 Additional sensory functions, other specified -

and unspecified
b280 Sensation of pain -
b289 Sensation of pain, other specified and -

unspecified
b298 Sensory functions and pain, other specified -
b299 Sensory functions and pain, unspecified -
b310 Voice functions speaking
b320 Articulation functions speaking
b330 Fluency and rhythm of speech functions speaking
b340 Althernative vocalization functions speaking
b398 Voice and speech functions, other specified speaking
b399 Voice and speech functions, unspecified speaking
b710 Mobility of joint functions motor
b715 Stability of joint functions motor
b720 Mobility of bone functions motor
b729 Functions of the joints and bones, other motor

specified and unspecified
b730 Muscle power functions motor
b735 Muscle tone functions motor
b740 Muscle endurance functions motor
b749 Muscle functions, other specified and motor

unspecified
b750 Motor reflex functions motor

b755 Involuntary movement reaction functions motor
b760 Control of voluntary movement functions motor
b765 Involuntary movement functions motor
b770 Gait pattern functions motor
b780 Sensations related to muscles and motor

movement functions
b789 Movement functions, other specified and motor

unspecified
b798 Neuromusculoskeletal and movement- motor

related functions, other specified
b799 Neuromusculoskeletal and movement- motor

related functions, unspecified
Table 4: ICF codes, their descriptions and the affected user modalities.
A typical example of a disability model and the information that is parsed from it
is depicted in Figure 4.

Figure 4: In this figure the Virtual User Model information that is valuable to the VerMIM
component is depicted; the disabilitys ICF and its affected task information are used in
the modality compensation step.
3.2.2 Modality Compensation and Replacement process

The Modality Compensation and Replacement process is different to automatic
and immersive/interactive simulations. The following two paragraphs describe
the process.
3.2.2.1 Modality Compensation in Automatic Simulations

An overview of the data flow during the Modality Compensation process of the
Multimodal Interfaces Manager is depicted in Figure 5.
Initially, the disability model of the Virtual User Model is parsed by the Core
Simulation platform and its data are transferred to VerMIM. As already
described the data contain the ICF codes and any affected by the disabilities
tasks. Then the ICF codes are mapped to their respective modalities by the
VerMIM component. Meanwhile, from the adaptation process any interfaces
that are contained in the 3D scene/GUI are made known to the VerMIM. So, by
this point the Multimodal Interfaces Manager has available the following
information:
the affected user modalities,
a list of possible affected tasks and
a list of available interfaces assigned to the objects/components of the
virtual design under test.

Then the VerMIM reads the Simulation model and via the usage of the
Multimodal Interfaces Models creates a list of task sequences. Multiple task
sequences are created based on the derived combinations of the available
Multimodal Interfaces Models. Each task sequence has its tasks annotated with
modalities that are necessary for it to be performed successfully.
Finally, the VerMIM performs the Modality Compensation process by sorting the
generated task sequences. The prioritisation that takes places takes into
account the fact that the sequences containing the lesser of the users affected
modalities, as well as affected tasks, are sorted first. Then the list of the
generated sequences is passed to the VerSim-GUI or VerSim-3D tools in order
to perform the simulation sessions.
3.2.2.2 Modality Compensation in Immersive (3D) and Interactive (GUI) Simulations

The modality compensation process in Immersive and Interactive simulations is
impossible by nature to be the same as the one described in Paragraph 3.2.2.1.
This is due to the fact that in the immersive/interactive simulation process a real
user interacts with environment following a specific task sequence, because the
generation of a list of alternative task sequences and their prioritisation would
take too much time to be performed.
So in order to cope with this problem the modality compensation process is
performed in an online mode by the VerMIM. The VerMIM constantly
supervises the simulation process and if the real user fails a task, it searches
the respective Multimodal Interfaces Models and produces an alternative task.
This alternative task is made known to the Core Simulation Platform in order to
adjust its task sequence. The whole process is depicted in Figure 6.

Modality Compensation Virtual

User 3D or GUI
In Automatic Simulations Model Design
(WP1.6)
Parse Disability Model

(Core Simulation Platform)
Assign the Multimodal Interfaces
to the 3D scene or GUI
(VerSEd-3D, VerSEd-GUI)
ICF Affected
Codes Tasks
Available
Interfaces
Map ICF to modalies
(Multimodal Intefaces
Manager)
Multimodal
Interfaces Affected
Models Modalities
(A2.8.2)
Simulation Creation of alternative

Model Task sequences
(WP2.2~ (Multimodal Intefaces
WP2.6) Manager)
TaskSequenceA
TaskSequenceB
TaskSequenceC
...
Modality Compensation
Manager)
Legend Prioritisation
TaskSequenceA 1st alternative
Input
Runtime TaskSequenceB 2nd alternative
File
Action Data TaskSequenceC 3rd alternative
...
Figure 5: Data flow of the modality compensation process for the automatic simulations,
i.e. for simulations that a real user is not present.

Modality Compensation Virtual

User 3D or GUI
In Immersive & Model Design
Interactive Simulations (WP1.6)
Parse Disability Model

(Core Simulation Platform)
Assign the Multimodal Interfaces
to the 3D scene or GUI
(VerSEd-3D, VerSEd-GUI)
ICF Affected
Codes Tasks
Available
Simulation Interfaces
Map ICF to modalies
Model (Multimodal Intefaces
(WP2.2~ Manager)
WP2.6)
Affected
Modalities
Creation of the default Multimodal
task sequence Interfaces
(Core Simulation Platform) Models
(A2.8.2)
Default TaskSequence
Modality Compensation:
No Find an alternative task
Run Task Success?
Manager)
Yes
Move to No All tasks

next Task Succeeded? Alternative Task
Yes
Legend
Exit Input
File Runtime
Action Data
Figure 6: Data flow of the Modality compensation process in the immersive and
interactive simulations.

4 Integration Status of the Multimodal Interfaces

Architecture Components
This section contains functional descriptions, data flows, integration and
implementation statuses of the Multimodal Interfaces components. The
components that are going to be described in this section are presented in
Table 5. The core functionality of the Multimodal Interfaces is provided by the
VerMIM component. The rest components are implementations of 3rd party
interface libraries. These components have been integrated with the VerMIM.
Multimodal Interfaces Component Integration
The VerMIM has been integrated with

the VERITAS Framework tools, all of
which are executables:
VerSEd-3D (A2.1.5/6),
Mutlimodal Interfaces Manager
(VerMIM) VerSEd-GUI (A2.1.5/6),
VerSim-3D (A2.1.1),
VerSim-GUI (A2.1.1),
IVerSim-3D (A2.1.2) + VerIM (A2.1.3).
Speech Recognition Module
Speech Synthesis Module
Haptics Module The Modules have been integrated into
the VerMIM component, either as
Sign Language Synthesis Module statically or dynamically linked libraries.
Symbolic Module
Screen Reader Module
Screen Magnifier Module
Table 5: The Multimodal Interfaces Components.
4.1 Multimodal Interfaces Manager

The Multimodal Interfaces Manager (or VerMIM) is the main product of the work
carried out through the WP2.8.
4.1.1 Functional Description

The VerMIM component provides to the VERITAS simulation adaptor (VerSEd)
and viewing (VerSim) tools the necessary functionality to use the Multimodal
Interfaces. VerMIM contains parsers of Multimodal Interfaces models, manages
the communication with 3rd party interfaces and performs the modality
compensation process.
VerMIM is not connected directly with the Core Simulation Platform or the
Immersive Simulation Platform libraries. Instead it has been integrated to the

tools that integrate the simulation platforms and connects to them indirectly. The
VerMIM has been implemented as Dynamic Link Library and thus it has to be
integrated into other executable tools to run. The DLL implementation allows the
Multimodal Interfaces Manager to easily be integrated into 3 rd party simulators.
4.1.2 Data Flow

The Multimodal Interfaces Manager accepts as input the Virtual User Model and
the Simulation Model data (Figure 7). VerMIM exchanges information with the
following VERITAS tools:
VerSEd-3D / VerSEd-GUI (A2.1.5/6): the VerMIM provides to the VerSEd
the list of the supported interfaces so that the designer can assign them
to the scene objects (or GUI components, if it is about the GUI
adaptation). Moreover, the VerMIM analyses the simulation model and
provides to the editors the alternative task sequences, so that the
designer may define them properly. These data exchanges take place in
the adaptation process of the 3D scene.
VerSim-3D / VerSim-GUI (A2.1.1): the VerMIM exchanges information
with these tools in order to perform the Modality Compensation Process.
The information is exchanged in code level. VerMIM checks the current
state of the simulation and if a task has failed, it receives a signal to find
a Multimodal Interfaces Model that will replace the task with another
modality. The data exchange takes place while the simulation is running.
IVerSim-3D (A2.1.2) and VerIM (A2.1.3): the Immersive Simulation
Viewer uses the Multimodal Interface modules offered by the VerMIM.
The request of module activation is sent to the VerMIM and the latter
initialises it and starts its management.
Interfacing Modules: The VerMIM is integrated with seven interfacing
modules, covering a wide variety of interfaces tools. Most of the
interfaces are 3rd party open source tools and integrated into the VerMIM
DLL. The VerMIM manages can manage these tools and can combine
their functionality.

Core Simulation Immersive Simulation

Platform (DLL) Platform (DLL)
Speech Recognition
Module
Veritas
Tool (EXE):
VerSEd-3D/GUI
VerSim-3D/GUI Haptics
IVerSim-3D Module
Modality
Virtual Compensation Speech Synthesis
User Process Module
Model
(UsiXML)
Multimodal
Sign Language
Interfaces
Synthesis Module
Manager
(VerMIM DLL)
Simulation
Model Symbolic Module
(UsiXML) Module
Screen Reader
Module
Selection of
Multimodal Management
Multimodal
Interfaces of Multimodal
Interfaces
Models Interfaces Screen Magnifier
Models
Repository Module
Figure 7: Data flow and implementation diagram of the Multimodal Interfaces Manager.
4.1.3 Integration Status

The integration status of the Multimodal Interfaces Manager is presented in
Table 6.
Programming C++
language
Integrated Microsoft Visual Studio 2005 (v8.0.50727.762)

development
environment
(IDE)
Integration Dynamic Link Library (DLL)

format
Integration Exports functionality for parsing Multimodal Interface Model

short files, managing them and using them to generate alternative
description task sequences that are based onto other modalities. It also

contains functionality of mapping the disability ICF codes to

affected modalities. Finally it contains managers capable of
connecting to and manipulating interfaces modules, such as
speech synthesis tools, screen readers, etc.
Filename MultimodalInterfacingTools.dll
Integration The Multimodal Interfaces Manager is fully integrated into the

status with following tools:
other
VERITAS 3D Simulation Viewer (VerSim-3D, part of
VERITAS
A2.1.1).
tools and
components VERITAS GUI Simulation Viewer (VerSim-GUI, part of
A2.1.1).
VERITAS 3D Simulation Editor (VerSEd-3D, part of
A2.1.5/6).
VERITAS 3D Immersive Simulation Viewer (IVerSim-3D
, part of A2.1.2) in combination with the VERITAS
Interaction Manager (VerIM, part of A2.1.3).
The VerMIM is not integrated as a DLL into the VERITAS GUI
Simulation Editor (VerSEd-GUI, part of A2.1.5/6), but it gets the
needed information by importing its xml files.
Other None
adjustments
Dependencies Dependencies are configurable via an xml file in which the

available interfaces modules are defined. If all the modules are
activated, the VerMIM with have the following dependencies:
CMU Sphinx libraries [5] (for speech recognition).
Festival Speech Synthesis libraries [6] (for quality
speech synthesis).
eSpeak libraries [7] (for compact speech synthesis).
Atlas libraries [8] (for sign language synthesis).
OpenCV libraries [9] (for visual alerts of the symbolic
module and for the screen magnifying).
NVDA libraries [10] (for screen reading).
Pending None
integration
actions
Table 6: Integration status of the Multimodal Interfaces Manager.

4.1.4 Development Status

The development status of the Multimodal Interfaces Manager is presented in
Table 7.
Development Final
status
Under development
Progress up to The Multimodal Interface Manager has been integrated

date in to the WP2.1 tools.
A satisfactory list of Multimodal Interfaces Models has
been created.
The modality compensation process has been defined
for both automatic and immersive simulation types.
Pending Regarding WP2.8:

actions
Iterative testing and optimization of the Multimodal
Interface Manager is a pending action (A2.8.5, M30).
Regarding SP3:
Pilot testing and evaluation of the Multimodal Interfaces.
Table 7: Development status of the Multimodal Interfaces Manager.
4.2 Speech Recognition Module

The Speech Recognition module is used for helping users with visual and/or
motor impairments that cannot interact properly with the virtual environment.

The Speech Recognition Module receives speech audio data and converts it to
text strings. The Module can work in two modes:
online which is useful for immersive and interactive simulations. Here
the user is a real person and the speech-audio is captured in real time by
a microphone. The module is able to convert the spoken phrases to text
in real time and provide the output text to the simulation engine.
offline in which a pre-recorder speech-audio file can be used. This
method is suitable for simulating virtual users, i.e. avatars.
The speech recognition implementation is based on the CMU Sphinx toolkit [5].
The Sphinx architecture is based on two models: a) the language model, which
contains the dictionary and grammar, b) the acoustic model, which contains the
respective audio information.
The Speech Recognition module supports English, French and Russian.
However, the Sphinx framework supports training of new languages and
acoustic models.
4.2.2 Data Flow

The VerMIM component provides the audio signal to the Speech Recognition
module and the latter converts into text strings. The text is then transferred to
the simulation platform via the VerMIM API (Figure 8). As it is depicted in the
Figure, the module is communicating directly only with the VerMIM component.
Immersed Real time

User speech
Audio
VerMIM signal
Prerecorded
Avatar speech file
Immersive
SimulImmeation
Platform
Speech Recognition Text VerMIM

Module
Core
Simulation
Platform
Figure 8: Data flow of the Speech Recognition module.

The integration status of the Speech Recognition Module is presented in Table
8.
Programming C++
language

development
environment
(IDE)

format
Integration Exports functionality for converting speech audio data to text

short strings. Supports input from microphone and from a variety of
description files (war, raw). Currently three languages are supported
(English, French and Russian), but the tool contains functions
that can be used for training new acoustic models.
Filenames sphinxbase.dll, pocketsphinx.dll

Integration The Module is fully integrated and managed by the Multimodal

status with Interfaces Manager (VerMIM).
other
VERITAS
tools and
components
Other None
adjustments
Dependencies The Module requires the CMU Sphinx libraries to run as well
the language model, the acoustic model and a dictionary file of
the language that is interpreted.
Pending None
integration
actions
Table 8: Integration status of the Speech Recognition Module.

The development status of the Speech Recognition Module is presented in
Table 9.
Development Final
status
Under development
Progress up to The Speech Recognition Module has been fully integrated with
date the Multimodal Interfaces Manager.
Pending For small dictionaries the tool achieves perfect results.

actions However, for large dictionaries the user has to speak with
clarity. So, iterative testing and optimization of the Speech
Recognition module is a pending action (A2.8.5, M30).
Table 9: Development status of the Speech Recognition Module.
4.3 Speech Synthesis Module

The Speech Synthesis module is used for assisting users with vision
impairments.

The Speech Synthesis module is used to convert text string into speech audio.
The Speech Synthesis Module uses two speech synthesis engines: eSpeak [7]
and Festival [6]. The eSpeak engine is compact and provides faster the speech
synthesis result, while the Festival engine is used to produce a higher level of
speech quality. A variety of voices and languages are supported [1].

4.3.2 Data Flow

As it is depicted in Figure 9, the Speech Synthesis Module receives the text
from the VerMIM component and then generates the speech audio. The Module
plays directly the speech from the speakers/headphones.
Speech
Simulation Text Text
VerMIM Synthesis
Platform
Module
Speech Immersed
Audio User
Figure 9: Data flow of the Speech Synthesis module.

The integration status of the Speech Synthesis Module is presented in Table
10.
Engine Festival v2.1 eSpeak v1.45
Programming C++, Scheme C++

language

development
environment
(IDE)
Integration Statically-linked library Dynamic Link Library (DLL)

format (LIB)
Integration Exports functionality for converting text to speech audio.

short Various languages and voices are supported.
description
Filenames libFestival.lib espeak_lib.dll

(integrated into the
MultimodalInter-
facingTools.dll
library)

other

VERITAS
tools and
components
Other None
adjustments
Dependencies Requires the Festival Requires the eSpeak libraries to

engine configuration files run.
(Scheme files). The
Festival libraries are not
required, because they
have been already
included in the VerMIM
DLL.
Pending None
integration
actions
Table 10: Integration status of the Speech Synthesis Module.

The development status of the Speech Synthesis Module is presented in Table
11.
Development Final
status
Under development
Progress up to The Speech Recognition Module has been fully integrated with
date the Multimodal Interfaces Manager.
Pending Testing and optimization of the module is a pending action as

actions part of A2.8.5 (M30).
Table 11: Development status of the Speech Synthesis Module.
4.4 Haptics Module

The Haptics Module allows the VERITAS framework to communicate with
various haptic devices and it is useful for users having vision impairments.

The Haptics Module has been implemented in order to allow the Multimodal
Interfaces Manager to support the interaction with various haptic devices, such
as the Sensables PHANTOM Desktop device [11]. The Haptics Module makes
use of the CHAI3D API open-source library [12].

4.4.2 Data Flow

The user interacts with the haptic device, and information, such as the position
and orientation of the device stylus, is passed through the Haptics Module to
the VerMIM component and further to the simulation engine. After the
information has been processed, the simulation platform can send various force
feedback to the user's hand. The whole process is depicted in Figure 10.
position,
orientation
Immersed Haptics
User Module
force
feedback
Haptic device
Simulation
VerMIM
Platform
Figure 10: Data flow of the Speech Synthesis module.

The integration status of the Haptics Module is presented in Table 12.
Programming C++
language

development
environment
(IDE)
Integration Statically-linked library (LIB)

format
Integration Provides the VerMIM an interface to haptic devices

short manipulation functions. The following devices are tested and
description fully supported:
Sensable's PHANTOM Desktop [11]
Sensable's PHANTOM Omni [13]
Novint Falcon device [14]
Filename chai3d.lib (integrated into the MultimodalInter-

facingTools.dll library)


other
VERITAS
tools and
components
Other None
adjustments
Dependencies The Chai3D libraries are not required, because they have been
already included in the VerMIM DLL.
Pending None
integration
actions
Table 12: Integration status of the Haptics Module.

The development status of the Haptics Module is presented in Table 13.
Development Final
status
Under development
Progress up to The Haptics Module has been fully integrated with the
date Multimodal Interfaces Manager.

Table 13: Development status of the Haptics Module.
4.5 Sign Language Synthesis Module

The Sign Language Synthesis Module is used to convert text into sign language
gestures and is targeted to usage by deaf users.

VerMIM provides the Sign Language Synthesis with annotated text and the
module uses it to generate sign language gestures via 3D animation. The
processing is based on the ATLAS tool (Automatic TransLAtion into Sign
language [8]) and is currently under a demo usage license. In its current phase,
only the Italian language is supported.
4.5.2 Data Flow

The text must have special annotation (xml language is used). The output is a
3D rendered virtual actor who performs the sign language gestures.

Annotated
Text VerMIM text
Sign Language Immersed

Module User
Figure 11: Data flow of the Sign Language Synthesis module.

The integration status of the Sign Language Synthesis Module is presented in
Table 14.
Programming C++
language

development
environment
(IDE)

format
Integration Provides the VerMIM with the functionality of converting text

short into 3D sign language gestures an avatar is used. The
description module currently supports the Italian sign language.
Filename ATLASPlayer.dll

other
VERITAS
tools and
components
Other None
adjustments
Dependencies OGRE [15] runtime libraries are needed for rendering the 3D
Avatar.

Pending None
integration
actions
Table 14: Integration status of the Sign Language Synthesis Module.

The development status of the Sign Language Synthesis Module is presented in
Table 15.
Development Final
status
Under development
Progress up to The Sign Language Module has been fully integrated with the

Table 15: Development status of the Sign Language Synthesis Module.
4.6 Symbolic Module

The symbolic Module has been constructed in order to provide the developer
with various alerting tools.

The Symbolic Module can be used to solve computer-to-human communication
issues that appear on vision, hearing and cognitive impaired users. Text, audio,
graphics (basic shapes, lines, boxes, circles and ellipses) alerts can been
combined in order to output multimodal information to the immersed user.
4.6.2 Data Flow

The data flow of the Symbolic Module is depicted in Figure 12. Initially, the
VerMIM sends to the Symbolic module the request of generating an alert. Then
the Symbolic Module generates the respective alert of message.
Text alert
Symbolic Graphics alert Immersed

VerMIM
Module User
Sound alert
Figure 12: Data flow of the Symbolic Module.


The integration status of the Symbolic Module is presented in Table 16.
Programming C++
language

development
environment
(IDE)
Integration Embedded in the Multimodal Interfaces Manager DLL.

format
Integration Generates alert using text, graphics and sounds.

short
description
Filename None as the Symbolic Module has been created as part of the
VerMIM and is fully integrated with the MultimodalInter-
facingTools.dll library.

other
VERITAS
tools and
components
Other None
adjustments
Dependencies OpenCV libraries are needed for generating the visual and text
messages.
Pending None
integration
actions
Table 16: Integration status of the Symbolic Module.

The development status of the Symbolic Module is presented in Table 17.
Development Final
status
Under development

Progress up to The Symbolic Module has been fully integrated with the

Table 17: Development status of the Symbolic Module.
4.7 Screen Reader Module

The Screen Reader Module is used for helping users with visual impairments
that cannot interact properly with the virtual environment.

The Screen Reader Module attempts to identify and interpret what is being
displayed on the screen (or, more accurately, sent to standard output, whether
a video monitor is present or not). This interpretation is then represented to the
user with text-to-speech or a Braille output device.
The Screen Reader Module implementation is based on the NVDA [10] free and
open source screen reader, which provides feedback via synthetic speech and
Braille. NVDA allows blind and vision impaired people to access and interact
with the Windows operating system and many third party applications. It is
bundled with eSpeak, a free, open-source, multi-lingual speech synthesizer.
The major highlights of the Screen Reader Module include:
Built in speech synthesizer supporting over 43 languages.
Reporting of textual formatting where available such as font name and
size, style and spelling errors.
Automatic announcement of text under the mouse and optional audible
indication of the mouse position.
Support for common accessibility interfaces such as Microsoft Active
Accessibility, Java Access Bridge, IAccessible2 and UI Automation.
4.7.2 Data Flow

The Screen Reader Module identifies and interprets the content sent to
standard output (e.g., screen) and represents this interpretation to the user
using synthetic speech and/or a Braille device (Figure 13).

Content, mainly
Textual, sent to
standard output
Request
for screen Screen Reader
VerMIM
reading Module
Braille
Data
Synthetic
Speech
Braille
Device
Immersed
User
Figure 13: Data flow of the Screen Reader module.

The integration status of the Screen Reader Module is presented in Table 18.
Programming C++
language

development
environment
(IDE)

format
Integration The NVDA screen reader is bundled with the Screen Reader
short Module and it is called as an external application by the

description VerMIM.
Filename ScreenReader.dll
Integration The Screen Reader Module is fully integrated and managed by

status with the Multimodal Interfaces Manager (VerMIM).
other
VERITAS
tools and
components
Other None
adjustments
Dependencies The Screen Reader Module depends on the NDVA open

source screen reader, which is bundled with the module.
Pending None
integration
actions
Table 18: Integration status of the Screen Reader Module.

Development Final
status
Under development
Progress up to The Screen Reader Module has been fully integrated with the
Pending Iterative testing and optimization of the Screen Reader Module

actions is a pending action (A2.8.5, M30).
Table 19: Development status of the Screen Reader Module.
4.8 Screen Magnifier Module

The Screen Magnifier module is used for helping users with mild visual
impairments that have problems in perceiving optical information properly.

The Screen Magnifier module takes as input portions of the screen, magnifies
them and provides as output a magnified version of these screen portions. Its
functionality is similar to the ones that are commonly used as assistive devices
for impaired PC users.

The Module works in online mode, being useful for immersive and interactive
simulations. In both cases, an image depicting a portion of the screens desktop
is captured in real time, which is centred to the position of the mouse pointer
over the screens desktop. The Module takes this screen portion as input,
together with a parameter indicating the desired magnification level (e.g. 2x,
3x). The given portion of the screen is magnified according to the given
magnification level. The result, an image depicting the magnified screen portion
is provided as output to the immersed user.
The screen magnification implementation is based on the OpenCV framework
[9].
4.8.2 Data Flow

The VerMIM component sends a request to the Screen Magnifier Module to
capture an image depicting the portion of the screen to-be magnified, along with
the desired magnification level. The Module magnifies the original image and
overlays the result to the screen. This process is repeated iteratively at each
simulation step.
Screen
Portion
Image
Magnified
Screen Magnifier Screen Immersed
VerMIM Request
Module Portion User
Image
Figure 14: Data flow of the Screen Magnifier module.

The integration status of the Symbolic Module is presented in Table 20.
Programming C++
language

development
environment
(IDE)

format
Integration Exports functionality for magnifying portions of the screen.

short Takes as input an image depicting a portion of the screen
description desktop, along with the desired magnification level, magnifies
the image and provides the magnified image as output.

Filename ScreenMagnifier.dll

status with Interfaces Manager (VerMIM)
other
VERITAS
tools and
components
Other None
adjustments
Dependencies The module requires the OpenCV library.
Pending None
integration
actions
Table 20: Integration status of the Screen Magnifier Module.

Development Final
status
Under development
Progress up to The Screen Magnifier Module has been fully integrated with the
Pending The Module currently supports the magnification of the input at

actions 2x and 3x the size of the input image. However, it will be
investigated whether the magnification of the input at further
size modes will provide added value. Iterative testing and
optimization of the Screen Magnifier Module is a pending
action.
Table 21: Development status of the Screen Magnifier Module.

5 Application Layer Interface

This Section provides information regarding the Application Layer Interface of
the Multimodal Interfaces Manager and its Modules. The information presented
here is valuable for the 3rd party software developer that needs to integrate the
the Multimodal Interfaces Manager DLL to his/her tools. The classes and
functions are split into two main categories:
the Multimodal Interfaces Models management classes and
the Interfacing Modules management classes.
5.1 Multimodal Interfaces Models Management API

These classes of this subsection contain the VerMIMs functionality for parsing,
selecting and organising the Multimodal Interfaces Models files. Moreover, they
are used for performing the task sequence analysis, as well as finding
alternative tasks for it.
Class Function
MInteractionModel MInteractionModel()
Stores information of a The class constructor. Each constructor creates an
Multimodal Interaction empty task tree and stores it in the object.
Model. Functions for
parsing the UsiXML ParseFromUsiFile()
files and analysing their Parses the Multimodal Interfaces Model UsiXML file
task trees are provided. and fills the tree of the object.
GetStrName()
Returns the name of the Model.
GetStrUsi()
Returns the name of the UsiXML file.
GetTaskById()
Returns a task stored in the model using its Id.

Class Function
SetApplicationArea(),
GetApplicationArea()
Sets and gets the application area type. The
application area type can be any of the following:
ApplicationArea_Generic,
ApplicationArea_SmartLivingSpaces,
ApplicationArea_Automotive,
ApplicationArea_Workplace,
ApplicationArea_Infotainment,
ApplicationArea_PersonalHealthcare.
GetRootTask()
Returns the task that resides on the task trees root.
GetNextPossibleNextTasks()
Gets as input a task and returns a list of the next task
candidates.
MInteraction- MInteractionModelManager()
ModelManager
Constructor of the manager. The VerMIM uses only
Class that manages the one manager. The manager is capable of managing all
Multimodal Interfaces the models contained in the repository.
Models. Contains
functions for parsing ParseXmlFile()
and retrieving the Parses an xml index file that contains the list of the
models from the repository models. The index categorises the models
repository. per their application area. After parsing the index each
model is parsed and stored in the manager.
SearchModel()
Searches a model by its name and its application area
and returns a list of matched models.
GetModels()
Returns all the models of a specific application area.
MTask MTask()
Represents the task Constructor of a Multimodal Interfaces Models Task.
element of a Multimodal
Interfaces Model. Each GetId()
model contains a task- Returns the id of the task. The id of a task is unique.

Class Function
tree and the nodes of GetName()

the tree are MTask Returns the name of the task.
elements.
GetModality()
Returns the modality that is necessary for performing
the task. The modality can be any of the following:
Modality_Null (used for initialisation)
Modality_Motor,
Modality_Vision,
Modality_Speech,
Modality_Hearing,
Modality_Cognition
GetMeans()
Returns the means that are needed for performing the
task action. Most of the times, the means parameter is
a body part, such hand, finger, etc.
GetObject()
Returns the object that is required for performing the
task action. An example of an object could be the
speech recogniser device.
GetType()
Returns the type of the task. The type can be any of
the following:
Type_Abstraction: used for complex tasks
that do not contain any interactions, but can be
decomposed into primitives tasks. The actions
of these tasks are contained in their children
tasks.
Type_Interaction: used for describing the
task which contain the interfaction.
AddChild(), GetChild(), GetNumChildren()

The Multimodal Interfaces Model contains a set of
tasks that are organised in a tree structure. So a task is
actually a tree node. These functions are used to insert
descendant nodes to the tree and retrieve them.

Class Function
GetParent()
Retrieves the parent task of the task.
SearchTaskById()
Performs recursive search of the task in the tree by
using its id.
GetTaskEnabling()
Returns the next task that is enabled after the current
task has been completed.
GetFirstEnablingChild()
Returns the first child task that is enabled after the
current task has been completed.
AddTaskChoiceBelow(),
GetTaskChoiceBelow(),
GetNumTaskChoicesBelow()
Functions that create (and get) task choices. A task
can lead to several other task choices (alternative
tasks).
MTaskSequence MTaskSequence()
Class that produces The constructors of task sequence.
objects of task
sequences. A task AddTask()
sequence is series of Adds a task to the end of the sequence.
task objects. The
VerMIM analyses a GetTask()
simulation model and
creates several task Get a task a specific task location.
sequence, which GetNumTasks()
actually are all the
possible task Gets the number of tasks contained in the sequence.
combinations.
ICFElement ICFElement()
Used to represent a Constructor of an ICF element.
mapping of an ICF code
to its affected SetCode(), GetCode()
modalities. Each Sets and returns the ICF code of an element.
element contains three
parts: a) the ICF code, SetName(), GetName()
b) the name of the body
function type, and c) the Sets and returns the name of an element.

Class Function
affected modality. SetModality(), GetModality()

Sets and returns the affected modality of the ICF code.
ICFParser ICFParser()
This class provides the Constructor of the class. Accepts as input the filename
functionality of reading of the xml mapping file. The constructor performs the
an xml file which maps parsing of the file in its initialisation.
the ICF codes to their
respective modalities. SearchICFElementByCode()
Searches an ICF element by its code.
5.2 Interfacing Modules Management API

This subsection describes the classes that are used for the management of the
provide interfacing modules.
Class Function
Module Module()
Class Module is an The constructor of the Module Class. The constructor is
abstract class that is called only by the derived classes objects constructors.
used for basic
management of an AsHapticsModule()
interface. AsSignLanguageSynthesisModule()
Contains initialisation AsSpeechSynthesisModule()
and termination
routines for the AsSpeechRecognitionModule()
interfaces. AsSymbolicModule()
AsScreenReaderModule()
AsScreenMagnifierModule()
Casts the module object into its derived class. Used
when it is needed to provide the special functionality of
the derived class objects.
GetName()
Returns the name of the Module.

Class Function
GetType()
Returns the type of the Module. It can be one of the
following:
Type_SpeechSynthesis,
Type_SpeechRecognition,
Type_Haptics,
Type_SignLanguageSynthesis,
Type_Symbolic,
Type_ScreenReader,
Type_Symbolic.
SetDataDir()
Set the directory where the Modules specific data reside.
Initialize()
Initialises the module this function must be called
before any other request to the module.
Terminate()
Terminates the module this function must be called
after the usage of a module in order to free any resources
that are occupied by it.
ModuleHaptics SelectDevice()
Class that is used for Selects a haptic device to manipulate.
manipulation of a
haptic device. The GetNumDevices()
class objects that Returns the number of the connected haptic devices.
need to be created
have to be equal to GetPosition()
the number of the
haptic devices Returns the 3D position of the haptic devices end
connected to the effector.
system.
GetRotation()
Return the 3D orientation of the haptic devices end
effector.
ApplyForce()
Sends a request to the device to generate a 3D force.

Class Function
GetForce()
Receives the current force that is generated from the end
effector.
GetLinearVelocity()
Receives the current linear velocity of the end effector.
GetAngularVelocity()
Receives the ends effectors angular velocity.
ModuleSpeechSyn- ModuleSpeechSynthesis()
thesis
Class constructor: receives as input the speech synthesis
Class for managing engine type, which can be one of the following:
the Speech Synthesis
SubType_Festival_v2_1
module. Two speech
synthesis engines are SubType_eSpeak_v1_45_04
supported: Festival
(high quality, but SetVoice()
slow) and eSpeak
Sets the voice of the speech synthesizer.
(fast, but lower
quality). If necessary, SayText()
two objects of this
class, each one Converts the input text into speech.
assigned to each
engine, can be used
simultaneously.
ModuleSpeechRec- ModuleSpeechRecognition()
ognition
Class Constructor.
Class that is used for
manipulating the SetDictionary()
speech recognition Sets the dictionary file. The dictionary contains all the
engine. Supports two words that are recognisable. Decreasing the dictionary
modes of speech size, increases the recognition accuracy.
recognition: a) offline,
i.e. from reading an SetLanguageModel()
audio file a file and b)
via receiving the Sets the language model. The language model contains
users speech from the rules that form the languages syntax.
the microphone.
SetAcousticModel()
Sets the acoustic model, i.e. the transcription of a word to
its phonemes.

Class Function
RecognizeRawDataFromFile()
Takes as input the filename of a raw audio data file and
converts it to text.
RecognizeFromMicrophone()
Receives speech audio from the microphone and
converts into text.
SetSamplesPerSecond()
Sets the sampling frequency of the input audio data.
SetMaximumRecordingDuration()
Sets the sizes of the audio buffers (in sec).
ModuleSignLangu- ModuleSignLanguageSynthesis()
ageSynthesis
Class constructor
Class that is used for
the Sign Language LoadSingleSentence()
Module. Currently Loads an xml file that contains the annotated text to be
implemented as a animated as sign language gestures.
demo application.
Play()
Plays the animated sign language sequence of the
loaded text.
ModuleSymbolic ModuleSymbolic()
Class that is used for Class constructor
managing the
Symbolic module. Wait()
Pauses the module; useful for playing a sequence of
events.
PlaySoundFromFile()
Loads a plays an audio file.
StopAllSounds()
Stop all audio playback.
ShowWindow(), HideWindow(),
SetWindowSize(), SetWindowColor()
Window manipulation functions, for visual alerts.

Class Function
DisplayText()
Displays text on screen.
ModuleScreenRead Start()
er
Initiates the screen reading module. The module parses
Class for the screens text that is under the mouse pointer and
manipulating the converts it into speech.
screen reader.
Stop()
Stops the screen reading.
ModuleScreenMagn Start()
ifier
Initiates the screen magnification process.
Class for
manipulating the SetDimensions()
Screen Magnifier Sets the dimensions, i.e. width and height, of the screen
Module. portion that will be magnified.
SetZoom()
Sets the magnification zoom level.
Stop()
Stops the screen magnification process.

6 Conclusions and Future Plans

In this document a thorough and technical description of the integration process
of the Multimodal Interfaces with the VERITAS Simulation Framework has been
given. The components-products of WP2.8 have been analysed and their
functionalities have been described.
The main component, the VerMIM, provides all the needed functionality that the
WP2.8 description demands: management of the Multimodal Interfaces models,
analysis of the simulation models to alternative task sequences, and modality
compensation techniques for both real and virtual users. Moreover, the VerMIM
is able to manage a variety of interfaces, as well as combine them to produce
an adapting multimodal interface suite.
Seven interfacing modules managed have been described through
architectural, implementation and integration aspects. It is worthy to mention
that most of the interfacing modules are based on 3 rd party open source
libraries. The integration of such an extensive functionality under one unified
and consisted component proved a difficult task, due to the libraries different
programming languages, the need of configuration data for each tool and
problems with memory and resources management (as some of the interfaces
conflicted because they used the same system devices). Thus, special care has
been given to preserve the consistency on the initialization and manipulation of
these modules.
Users with mild or severe vision, hearing, and motor impairments are addressed
through the variety of the interfaces provided by the VerMIM component. Tools
that use the speech and motor as primary modalities (synthesizers, screen
readers, haptics) are useful for users that have severe vision impairments, while
tools based on vision (sign language synthesizers and the symbolic module)
address to the hearing impaired. Solutions are also provided for the motor-
impaired users via the provided speech recognition module.
Future plans are concentrated towards the iterative testing and optimization of
the Multimodal Interfaces toolset as part of activity A2.8.5. Through this
process, several changes may be applied to the current Multimodal Interfaces
setup. Furthermore, possible additions to the toolset will be also investigated.
The most probable candidate is to add support for Braille devices.
Concluding, it can be written that the integration process of the Multimodal
Interfaces toolset with the VERITAS Simulation Framework has reached its final
implementation state. As both the integration (A2.8.4) and the testing (A2.8.5)
processes have not yet finished (they end in Month 36), several things can be
corrected and new features may need to be added. However, even in its current
state, the integration process has shown its great potential either as part of the
VERITAS framework or as a library of tools to be used with 3 rd party simulation
software.

References
[1] Panagiotis Moschonas, Dimitrios Tzovaras, George Ghinea, VERITAS
Deliverable D2.8.1 First prototypes of the multimodal interface tool set,
December 2011.
[2] Panagiotis Moschonas, Athanasios Tsakiris, Ioannis Paliokas,

Georgios Stavropoulos, Dimitrios Tzovaras, VERITAS Deliverable
D2.1.4 Integrated Core Simulation Platform and Exportable Toolbox,
June 2012.
[3] PERCRO, CERTH, VRMMP, VERITAS Deliverable D2.1.3 Interaction,

Virtual User and Simulation Adaptor, December 2011.
[4] World Health Organisation, International Classification of Functioning,

Disability and Health, ICF, http://www.who.int/classifications/icf/en/
[5] Carnegie Mellon University (CMU) Sphinx, Open Source Toolkit For
Speech Recognition, http://cmusphinx.sourceforge.net/
[6] Alan W. Black, Centre for Speech Technology Research, University of

Edinburgh, Festival Speech Synthesis System,
http://www.cstr.ed.ac.uk/projects/festival/
[7] Jonathan Duddington, eSpeak text to speech,

http://espeak.sourceforge.net/
[8] ATLAS, Automatic Translation into Sign Language, Bando Converging

Technologies 2007, Regione Piemonte, http://www.atlas.polito.it/
[9] OpenCV, Open Source Computer Vision, 2011

http://opencv.willowgarage.com/wiki/
[10] NonVisual Desktop Access, NVDA, http://www.nvda-project.org/
[11] Sensable Technologies, PHANTOM Desktop haptic device,

http://www.sensable.com/haptic-phantom-desktop.htm
[12] Conti F, Barbagli F, Morris D, Sewell C. "CHAI 3D: An Open-Source

Library for the Rapid Development of Haptic Scenes", IEEE World
Haptics, Pisa, Italy, March 2005.
[13] Sensable Technologies, PHANTOM Omni haptic device,

http://www.sensable.com/haptic-phantom-omni.htm
[14] Novint Technologies, Inc. (NVNT), Novint Falcon,

http://www.novint.com/index.php/products/novintfalcon
[15] OGRE, Open Source 3D Graphics Engine, http://www.ogre3d.org/

Accessible and Assistive ICT: Veritas

Enviado por

Dados do documento

Título original

Direitos autorais

Formatos disponíveis

Compartilhar este documento

Compartilhar ou incorporar documento

Opções de compartilhamento

Você considera este documento útil?

Este conteúdo é inapropriado?

Direitos autorais:

Formatos disponíveis

Accessible and Assistive ICT: Veritas

Enviado por

Direitos autorais:

Formatos disponíveis

Accessible and Assistive ICT

Integration of Multimodal Interfaces into the

Deliverable No. D2.8.2

Dissemination Level: Pu (Public)

File Name: VERITAS_D2.8.2_final

Project start date and 01 January 2010, 48 Months

Version History table

1 First version created on February 2012, analysing the methodological

2 Second version created on April 2012 including the components details.

June 2012 iii CERTH/ITI

4.4.3 Integration Status .................................................................................. 40

June 2012 v CERTH/ITI

June 2012 vi CERTH/ITI

Table 11: Development status of the Speech Synthesis Module...................... 39

June 2012 vii CERTH/ITI

Ax.y.z VERITAS Activity

CAD Computer-aided design

GVUM Generic Virtual User Model

VUM Virtual User Model

June 2012 viii CERTH/ITI

List of Abbreviations for VERITAS tools

MOUSI Veritas User Model Ontology to UsiXML Converter

VerGen Veritas User Model Generator

VerMP Veritas Model Platform

VerSEd-3D Veritas 3D Simulation Editor

VerSEd-GUI Veritas GUI Simulation Editor

VerAE Veritas Avatar Editor

VerSim-3D Veritas 3D Core Simulation Viewer

VerSim-GUI Veritas GUI Core Simulation Viewer

IVerSim-3D Veritas 3D Immersive Simulation Viewer

VerIM Veritas Interaction Manager

June 2012 ix CERTH/ITI

June 2012 11 CERTH/ITI

June 2012 12 CERTH/ITI

2 Architecture of the Multimodal Interfaces

2.1 Positioning in the VERITAS Global Architecture

June 2012 13 CERTH/ITI

GUI Simulation Editor (VerSEd-GUI): this tool is used for adapting a

June 2012 14 CERTH/ITI

VERITAS Features provided by the

June 2012 15 CERTH/ITI

VERITAS Features provided by the

of a GUI. interfaces to the GUI components.

2.2 Local Architecture Analysis

June 2012 16 CERTH/ITI

Veritas Architecture Component

Figure 2: The local architecture of the Multimodal Interfaces.

The Simulation Model files: these kinds of files describe in an abstract

June 2012 17 CERTH/ITI

June 2012 18 CERTH/ITI

June 2012 19 CERTH/ITI

Stage Tools Involved Sequence of Actions

Adaptation VerSEd-3D, a. Manually assign the multimodal and

June 2012 20 CERTH/ITI

Stage Tools Involved Sequence of Actions

Runtime VerSim-3D, a. Automatically, perform an analysis of the

3.1 Adaptation Stage

The Multimodal Interfaces adaptation process is consisted of three steps,

Multimodal Interfaces Adaptation Performed automatically / manually

1. Assignment of the Multimodal

3.1.1 Assignment of Multimodal Interfaces to the adapted environment

3.1.2 Analysis of the Simulation Model based on the Multimodal

3.1.3 Definition of the alternative tasks