AUGMENTED REALITY: FUSING THE REAL AND SYNTHETIC WORLDS
Mauro Dalla Mura, Michele Zanin, Claudio Andreatta, Paul Chippendale
Technologies of Vision, Fondazione Bruno Kessler, Trento, Italy
{dallamura, mizanin, andreatta, chippendale}@fbk.eu
ABSTRACT
Augmented Reality (AR) offers a means to inject virtual in-
formation into real scenes. In the past few years, AR has
been receiving greater attention thanks to considerable ad-
vancements in the hardware of consumer-level portable de-
vices. In this paper, we illustrate how mobile AR can be ex-
ploited to intuitively visualize, and moreover generate new,
geo-data. We explore these two concepts through i) the vi-
sualization and interaction modality of geo-data as AR layers
and ii) the exploitation of mobile devices as opportunistic sen-
sors for generating information relating to a users immediate
surroundings.
Index Terms Augmented reality, image registration,
visualization, opportunistic sensors, data collection.
1. INTRODUCTION
Augmented Reality is a technology which enables the blend-
ing of virtual and real scenes in both time and space [1, Ch. 1],
opening up the possibility to inject information into a scene
when viewed through a portable device. AR can be useful in
many applicative domains [1], such as in touristic guidance
System (GIS) domains [1, Ch. 33]. In [3], a GIS-based photo-
realistic visualisation method for viewing natural landscapes
through video-based off-line AR was proposed. ARVino, a
demonstrator for the augmented visualisation of geolocated
wine-growing data was presented in [4]. In [5] it was shown
how geo-data could be exploited for labeling mountains that
are really visible inside photos, adding their heights above
sea level and their distances from the camera. Nevertheless,
a stable and uncluttered integration of AR technologies for
geoscience applications is still to be achieved.
In this paper, we present two reciprocal aspects relating to
geoscience applications, in which AR technology can be ex-
ploited to link the real world to geo-data archives. In the rst
case, AR can be seen as a tool to enable a user to experience
an immersive new modality for accessing and interacting with
geo-data through the visual overlaying of synthetic informa-
tion onto the real scene. In the second case, since convincing
AR applications require a very precise registration of the de-
vice with the viewed scene, information can be gleaned from
reality through the devices sensors and can thus be exploited
for augmenting geo-databases.
 
(e.g., providing detailed descriptions of monuments), enter-



tainment (e.g., visualizing information about the players in a







football match, or, overlaying interactive gaming characters
into a real scene), urban planning (e.g., rendering virtual con-
struction models into a site), medical applications (e.g., pro- 


 








 

viding assistance during surgical operations), etc..
The latest developments in AR technology (i.e., in terms
 
of software and hardware) are enabling applications to run
entirely on consumer level mobile devices (e.g., smartphones Fig. 1. AR concept applied to geoscience and remote sensing.
and tablets) in real-time [1, 2]. Obviously, this increases the
potential impact that such technologies can have in our real Figure 1 illustrates the way in which AR can be exploited
life due to their profusion. in the elds of geoscience and remote sensing, highlighting
Thanks to these recent advances and the availability of the proposed symbiotic relationship.
massive geographic referenced archives of data (e.g., remote
sensing, geo-referenced social photo websites and geoscience
data) an effective and meaningful integration of AR and geo- 2. REGISTERING THE REAL TO
data is becoming feasible. Recently, we have started to wit- THE SYNTHETIC WORLD
ness a convergence of AR and the Geographic Information
This research has been funded by the European 7th Framework Program, In order to provide an immersive and convincing experience,
under grant VENTURI (FP7-288238). AR relies on a precise, consistent, and solid registration

978-1-4673-1159-5/12/$31.00 2012 IEEE 170 IGARSS 2012

between the real scene and a synthetic representation of the
world [1]. This is essential for a precise and non-jittery in-
sertion of markers or points of interest into the environment
(e.g., accurate placement of toponyms or overlays).
The registration of real scenes to synthetic models can
be performed in different ways. Latest generation smart de-
vices are equipped with on-board hardware, such as Global
Positioning System (GPS), magnetometer, gyroscope, and
accelerometer, that can provide a rough estimate of pose
and position. When geo-positional aware landmarks are in
view (e.g., mountain peaks, distinctive architecture, anthropic
structures, etc.,) photo-to-world registration can be driven by
vision-based features.
Vision-based registration is often obtained through the
cross-correlation of geo-referenceable synthetic features to
visually detectable naturally occurring ones [6]. In this ap-
proach, 3D-models are built around a users location (increas-
ingly appearing inside the EXIF metadata of photos) [5, 6]
from geo-data such as Digital Elevation Models (DEMs),
geo-referenced maps, and vectorial GIS layers.
As a consequence of registration, a direct mapping of
land-based photo pixels to geographic coordinates can be
obtained.


process is presented. In this scheme, alignment is obtained
by nding matches between fragments of an edge-map (gen-
erated using image processing), to similar proles in a syn-
thetically rendered panorama created by ray-tracing [5]. In
this example, the correlation of features is achieved through
spherical cross-correlation [6].
Upon registration, rendered virtual content can be over-
laid into a real scene, being subsequently adapted in response
to camera pose changes (e.g., a users movements). These
pose changes can be detected and tracked by exploiting on-
board hardware sensors (e.g., accelerometers and gyros)
and/or through the temporal monitoring of visual feature
displacements.
Using such a terrain-aware approach avoids the overlay-
ing of occluded content or the representation of details that
are not visible due to their distance from the viewing point.
Thus, in this way, a convincing blend between the real scene
and superimposed virtual information can be provided as nat-
urally as possible.
3. AUGMENTING THE USERS VIEW
Once a real scene is geo-registered, it is possible to offer the
user intuitive access to many forms of geo-data.
Public and private geo-data (e.g., geographical toponyms,
GIS layers, 3D buildings models) represent potentially rich
sources of augmentation material that can easily be sup-
plied through web services. Since the superimposition of

  augmented overlays is subject to their actual visibility from
the users vantage point, this becomes a natural modality to

 

 query, lter and show only the information pertaining to the

 
users surroundings. Touchscreen and gesture-based interac-
tions, now commonly available in consumer-level devices,
can provide an intuitive interface to browse and manipulate
augmented content.
This means of information delivery can directly benet
the general public since it provides intuitive visualizations and

     enables simple geo-data operations; moreover, it can also pro-
vide powerful tools for geo-scientists.
It is well known that eld work campaigns are of fun-
 damental importance in remote sensing, however, they are
usually very time-consuming and often tedious [7]. In this
respect, AR scene augmentation provides a valuable tool to
aid operations such as the in-situ validation of remote sens-
ing products (e.g., land cover thematic maps). In the same
way, it can also provide real-time guidance for eld surveys
gathering reference data (e.g., ground truth collection). Fur-




thermore, a more accurate registration of the device perform-
ing the augmented overlays (e.g., using hardware and vision-
based approaches) can help to reduce location errors residing
Fig. 2. Flowchart of a vision-based photo-to-world registra- in geo-data, which is one of the issues that affects eld mea-
tion [5]. surements [7].
The interpretation of remote sensing data can also be aided by
In Figure 2 a typical pipeline of a vision-based registration scene augmentation. For example, by re-projecting and over-

171
laying geo-referenced satellite or aerial images on the real
scene can help users to associate features in an image to cor-
responding objects in the real world. This can be particularly
useful when dealing with active sensor data (e.g., SAR and
LIDAR), or optical data acquired at spectral wavelengths be-
yond the visible spectrum (e.g., hyperspectral images). This
in-situ visualization of geo-data, whether static or time-series,
enables a user to easily detect and track abrupt changes in the
landscape and facilitates the intuitive observation of land-use
evolution (e.g., culture transitions in agricultural elds). In
this scenario, archives of geo-tagged photos residing in large
web databases (e.g., Flickr and Panoramio), can also be ex-
ploited (once geo-referenced) to extend the spatio-temporal
coverage of a landscape.
In Figure 3 we show how an augmented layer can provide
the user with additional information about their surroundings,
illustrating the names of visible peaks, the path of a highway
and the course of a river. As this gure exemplies, it is a
straightforward procedure to intuitively verify the semantic
correctness and spatial precision of geo-information pertain-
ing to a region.

Fig. 3. View from Fai della Paganella (Trento, Italy) aug-

4. AUGMENTING THE GEO-DATA ARCHIVES
Thanks to accurate real-worldsynthetic registration, the
spatial coverage of each pixel from an aligned photo is
known. Thus, pixels from a ground-level image can eas-
ily be re-projected into a nadiral view (i.e., a viewshed),
for generating new GIS layers or for comparisons to other
satellite images.
In this way, we can consider AR-enabled portable devices
as ground-based opportunistic mobile sensors which are ex-
ploitable to acquire high spatial resolution information from
the scene. This can provide geoscientists with the ability to
create new, or to update existing geo-data archives. From
this perspective, through the collection and accurate interpre-
tation of geo-specic information registered by many users,
it is possible to augment, model and document an evolving
landscape. For example, the use of up-to-date and ground-
level visual evidence could be exploited in tasks such as the
detection of changes in land usage with respect to previously
available data, the monitoring of agricultural, forest and urban
developments, the estimation of snow coverage, the extent of
ooding and the generation of damage assessment maps.
Furthermore, since the acquisition is performed at the
ground level, the users immediate surroundings, as sensed
by a devices camera, can provide information at a very high
level of precision, although limited geographically. This in-
formation can be useful for describing the land cover type.
If multiple geo-registered views of the same area are avail-
able, it is also possible to generate new 3D terrain datasets or
building models using photogrammetry techniques.
Figure 4 illustrates the landscape in Figure 3 re-projected
onto a top-down view as if it had been acquired from a nadi-
mented with the toponyms of visible mountains peak, a high-
way (black line) and the Adige river (blue line).
All of the geo-data used for the registration and augmentation
is publicly available on the internet: toponyms were obtained
from GeoNames (http://www.geonames.org/), the
DEM used for the registration is the SRTM 90m Digital Ele-
vation Data distributed by CGIAR Consortium for Spatial In-
formation (http://srtm.csi.cgiar.org/), and the
track of the highway and the river are taken from Open-
StreetMap (http://www.openstreetmap.org/).
Source http://tev.fbk.eu/marmota.
ral aerial view. Once re-projected, the new viewpoint can be
rapidly compared to previously acquired remote sensed im-
ages, providing hints on the evolution of a landscape. Need-
less to say, there are precision issues due to the spatial distor-
tion introduced by the uneven sampling of the landscape onto
the ground (i.e., the spatial resolution varies according to the
distance from the camera, as can be noticed in Figure 4), or by
the limited spatial coverage, or the presence of foreground ob-
jects masking the landscape. Nevertheless, such social data
sources might represent a valuable new resource of observa-
tional data since it comes at no cost. Furthermore, ground-
based views of the environment can capture vertical surfaces,
produce usually cloud-free images, and could represents the
only near-live acquisition available for a disaster area (e.g., in
a scenario of damage assessment).

172

Fig. 4. Viewshed of photograph from Figure 3, re-projected
onto the ground surface. An equally spaced grid (in original
photo pixels) is overlaid onto the viewshed to illustrate un-
even spatial resolution due to re-projection.
5. DISCUSSION AND CONCLUSION
When augmenting real scenes and populating geo-data archives,
some aspects are essential in order to provide the user with a
realistic experience and to generate accurate information.
The re-projection of geo-data onto the users viewpoint
for scene augmentation is typically a demanding op-
eration and must be performed in a fast and accurate
manner. Efcient rendering techniques for the fusion of
real and synthetic information must be considered. The
use of Graphics Processing Units (GPUs), which have
already proved to be effective for performing fast ren-
dering and the processing of large amounts of remote
sensing data [8], are promising to provide a solution to
this problem as GPUs are now available in many smart
devices.
Precise localization and device pose are essential com-
ponent to deliver accurate AR overlays. Joint vision-
and sensor-assisted approaches must be considered to
increase the robustness of the registration chain.
High-speed wireless data-transfer speeds between
portable devices and web servers are essential since
large amounts of geo-data need to be transferred back
and forth. Network transfers can be optimized by con-
sidering compression algorithms and ofine storage.
173
The potential of hybrid systems that can harmonize infor-
mation from aerial and ground-based social sensors is only
just beginning to be explored, but this eld of research is
promising to accelerate the access and creation of shared and
comprehensive data records. In this scenario, Augmented
Reality has the potential of becoming a novel and inexpen-
sive tool for accessing and interacting with geo-data through
consumer-level portable devices. Furthermore, thanks to AR
technology that is able to precisely align a real scene with
geo-referenced 3D models, it is possible to think of mobile
imaging devices as remote sensors acquiring ground-level in-
formation. From this perspective, users devices can be ex-
ploited for the updating of geo-data archives in a collaborative
and participative way.
6. REFERENCES
[1] B. Furht, Handbook of Augmented Reality, Springer,
2011.
[2] M. Billinghurst and A. Henrysson, Mobile architectural
augmented reality, in Mixed Reality In Architecture, De-
sign And Construction, Xiangyu Wang and Marc Aurel
Schnabel, Eds., pp. 93104. Springer Netherlands, 2009.
[3] P. Ghadirian and I.D. Bishop, Integration of augmented
reality and GIS: a new approach to realistic landscape vi-
sualisation, Landscape and Urban Planning, vol. 86, no.
3-4, pp. 226232, 2008.
[4] G.R. King, W. Piekarski, and B.H. Thomas, ARVino-
outdoor augmented reality visualisation of viticulture
GIS data, in Mixed and Augmented Reality, 2005. Pro-
ceedings. Fourth IEEE and ACM International Sympo-
sium on, 2005, pp. 5255.
[5] P. Chippendale, M. Zanin, and C. Andreatta, Spatial and
temporal attractiveness analysis through geo-referenced
photo alignment, in IEEE International Geoscience and
Remote Sensing Symposium, 2008. IGARSS 2008, July
2008, vol. 2, pp. II1116 II1119.
[6] L. Baboud, M. Cadik, E. Eisemann, and H. P. Seidel,
Automatic photo-to-terrain alignment for the annotation
of mountain pictures, in IEEE Conference on Com-
puter Vision and Pattern Recognition (CVPR), 2011, June
2011, pp. 4148.
[7] R.M. McCoy, Field Methods In Remote Sensing, Guil-
ford Press, 2005.
[8] M. Lambers and A. Kolb, Gpu-based framework for dis-
tributed interactive 3d visualization of multimodal remote
sensing data, in Proc. Int. IEEE Geoscience and Remote
Sensing Symposium (IGARSS), 2009, vol. 4, pp. IV57
IV60.