Title: Bringing Meaning to Tourism: The Use of Augmented Reality to Navigate Foreign

Language Environments
Topic Area: Knowledge Management; Innovation and Technology
1. Introduction - The Problem of Foreign Languages for the Tourist Industry
The tremendous expansion in tourism in the two past decades, in line with declining
costs for travel, has allowed more people to travel who lack the advanced linguistic
skills necessary to optimally navigate foreign environments. This is particularly true in
regard to less well educated, nouveau riche travelers from newly industrializing
countries with a desire for foreign travel and experiences in exotic locations such as
Europe. These travelers, though perhaps capable of reading signs using what many
perceive to be difficult Chinese ideograms or some other script, could be entirely baffled
by signs using common Western languages such as French or English.
At the same time, more highly educated and affluent travelers from places such as
Europe and America are visiting tourist destinations where their skills are severely
challenged by languages that are quite different in form from the languages they are
familiar with. Although English has become a lingua franca for much of the tourist
industry, English and other Western European languages are still not yet a part of the
everyday environment for more than perhaps a quarter of the worlds population.
Consequently, attempts by more affluent travelers to achieve an authentic experience in
a radically foreign environment will inevitably be constrained by the limited linguistic
access afforded to the traveler. This difficulty is especially felt when alphabets are nonLatin in their form, or in environments where pictographs are used in place of alphabets.
The linguistic cues found on signs and other textual information sources in foreign
environments are hidden from the traveler as a result of linguistic differences, and
threaten to make the tourism experience difficult or even potentially dangerous.
Technology has stepped in to help with these problems, offering translation software,
electronic dictionaries, and the like, but problems still persist. Electronic dictionaries, for
example, are of little use when there is no alphabet to facilitate navigation, as is the
case for logographic/pictographic languages such as Chinese or Japanese, or in the
case where alphabets are non-Latin in form, such as for Thai or Korean. What is
needed is a tool to magically transform the environment into one that is linguistically
familiar to the traveler, without the need to actually alter the environment from the
perspective of the native. In this presentation we will look at how augmented reality (AR)
is making the whole world accessible to any traveler, regardless of native language or
foreign language experience.
This paper will, for practical reasons, focus only on the problems faced by foreign
travelers dealing with text or pictographic-based information sources. It is recognized
that a great deal of information is available only through verbal communication, and this
information will remain unavailable to the tourist as long as we lack real-time verbal

language translation software. Nonetheless, AR applications focusing only on text or

pictographs will be of great utility to travelers. Since these AR applications are either
available today or will be available in the near future, they will have a more immediate
impact on the travel experience than real-time verbal language translation, which yet
faces formidable technological challenges.
2. Introduction to Augmented Reality
Augmented reality is perhaps less well known than virtual reality (VR), but in the near
term it promises to transform our world in more and more useful ways than the artificial
digital and virtual reality environments that have become known to people in movies
such as Avatar. Augmented reality, a term first used in the early 1990s by two
engineers at Boeing named Caudell and Mizell (van Krevelen and Poelman, p.2), does
not create a strictly artificial reality, but merely enhances the reality we experience in our
everyday lives, overlaying descriptions, instructions, or explanations on top of the
environment we perceive through our normal senses. In the most technologically
advanced forms of AR, the actual environment itself is not changed. Only our perception
of that environment is altered, in a way to make it more understandable or richer in
context.
In order to better understand AR technology, I would ask you to consider a bottle of
laxative sitting on a shelf in a Japanese pharmacy. A traveler visiting Japan on a ten day
vacation has had some problems with the food and developed a problem with
constipation while in a remote area of the country. While not life threatening or
necessitating a visit to a doctor, the visitor would certainly enjoy the rural sights more if
some regularity were restored to the the daily toilet routine. There is only one pharmacy
in the small town where the traveler is staying, and the traveler speaks no Japanese,
while no English is spoken by the sales clerk in the pharmacy. The traveler has noticed
some packages in the pharmacy with pictures of what appear to be a human stomach,
but there are no products with familiar names to choose from, and the available
Japanese products have only Japanese characters printed on the packages.
Fortunately, the traveler is carrying a smartphone and has an Internet connection
through the telephone network.
The traveler points a smartphone camera at the products on the pharmacy shelves and
AR technology begins to work. The AR technology first establishes that there is a
barcode on the packages. It has found a marker to work with. The AR technology then
also establishes the location of the pharmacy, as provided by a gravimetric device or
GPS signal and the smartphones ability to make use of this signal. Finally, the
smartphone camera notices there are objects, the Japanese text, on the package, and
using OCR software is able to read the information. As a result, due to the marker, the
visitor knows the name of the product and any other information encoded in the
barcode. (With smarter barcodes, even information such as the way in which the
product is intended to be used could be provided.) The visitor also learns, as a result of
information provided by querying a databases based on the location of the pharmacy,
that a sales clerk named Tomoko, who works from 9 a.m. to noon everyday, is able to

speak some English and thus might be able to assist our traveler. Finally, the OCR
software within the camera has translated the objects, or Japanese text, on the
package into Italian, the native language of the traveler, and our traveler now has found
the Japanese version of the familiar laxative product.
All of this AR magic is made possible as a result of something known as an augmented
reality browser, similar to the browsers used on personal computers, to navigate the
world through the lens of the smartphone camera. The video feed provided by the
smartphone, combined with a network Internet connection to a central database of
information, transforms the environment seen by the traveler through the browser into a
world filled with relevant information, in the native language of the traveler. Using the
markers, location, and/or objects we have just mentioned the AR browser is able to
generate the information they overlay on our environment.
3. AR Technology with Markers, Location, and Objects
Markers represent the least technologically advanced means for implementing AR. A
marker is something such as a barcode that is consciously added to the environment in
order to provide information. In most cases it requires that the environment be altered in
order to provide information that can be used within the environment, since barcodes
and other common forms of markers are not usually embedded on anything other than
products. One might imagine, for example, a sign with a barcode pasted to the bottom
of the sign. The barcode or marker can be scanned by the video feed from a
smartphone, and that code will be deciphered by the central database to overlay the
appropriate textual information on the browser window to the augmented reality world.
The barcode or marker might, in our sign example, indicate we are in a tsunami zone
and need to exercise caution. Perhaps we also noted the wave on the sign, but were
not certain if this indicated this was an ideal location for swimming or surfing, or if
danger was indicated. The amount of information embedded in a barcode can be large,
so we might also learn the sign was put in place last year, and therefore the information
is fairly current and probably worth considering.
Augmented reality applications based on location, as provided by gravimetric or GPS
technology, do not require the environment be changed in order to overlay information
on the environment. These applications are more flexible than marker-based AR, but
require greater technological sophistication from the AR device than marker-based
systems. They of course require that the AR device, such as the smartphone, be
equipped with a means to compute relative location on the basis of GPS signals. At
present this is one of the limiting factors in the adoption of AR, as commercial GPS
devices do not yet have the accuracy to locate objects as small as a sign with pinpoint
accuracy. Although the military is able to blow up ships, buildings, and other objects with
missiles coordinated by GPS signals, commercial users can presently expect nothing
more than to be able to identify buildings on the basis of location, and even in these
cases errors of up to hundreds of meters is not uncommon. Therefore, the type of
information that can be overlaid on an environment using location-based AR is still
limited by errors in GPS signal accuracy. Nonetheless, once a location is established,

AR technology can begin to provide the user with a wide variety of information about the
area. That information can come from databases used by Twitter, Facebook, or other
online sources of social information.
The final form of AR we are looking at involves object recognition, and the subset of
OCR-enabled translation. Although we do not need to know the location of the object to
provide this type of information, the processing requirements for translating information
using object recognition place a substantial strain on todays smartphones, and robust
network connections are also a necessity in most cases. In our sign example, no
barcode or marker would be required if we could rely on our smartphone and its OCR
software to translate the Japanese characters on the sign. There would be no need to
physically alter the environment in order for our visitor to have the same information,
through AR technology, native Japanese visitors to the area would have. Fortunately, a
great deal of effort has gone into developing OCR and object recognition algorithms,
and a number of smartphone applications have already been introduced.
4. Smartphone Translation Applications and Limited Augmented Reality
The potential benefits of augmented reality are so great because browsers will be able
to combine marker, location, and object recognition technologies to construct an entire
world of information unavailable to tourists who cannot read the language used in the
country where they are traveling. However, even today tourists have access to more
limited applications that use just a small spectrum of AR technology. Instead of viewing
the world through the augmented reality browser, for example, we are able to use our
smartphone cameras and software such as Word Lens, Camera Translator,
CamDictionary, Photo Translator, or similar applications to provide limited OCR-based
translation of signs. Even these more limited applications work remarkably well, with a
great number of language choices, and can be of tremendous assistance to travelers. It
is now possible to travel in Japan, China, or other exotic environments without worrying
that signs will be unintelligible without a working knowledge of the foreign language.
Lesser used languages, such as Lao or Cambodian, at present lack the population base
to make the development of OCR translation software commercially viable, but it is only
a matter of time before we will also have applications to make reading signs in Vientiane
or Phnom Penh a trivial matter.
Some countries, of course, already translate some of their signs into English,
particularly in large cities, and this translation already greatly facilitates ease of travel.
Travel to more remote and less densely populated locations, however, still presents
difficulties, as governments lack the economic resources to provide translations for each
and every sign that is put up. Moreover, while English translations are certainly
welcome, they are not a solution to the navigation problems of every foreign visitor. As
we pointed out at the beginning of this paper, due to declining transportation costs,
greater ease of travel, and higher incomes all over the world, more people who lack
basic foreign language skills are now traveling abroad. English translations might be
useful to more educated tourists, but of limited use in its raw form to these travelers.

Nonetheless, there is still a case to be made for providing English or some other
Western language translations on signs whenever possible, and particularly in countries
where text is not alphabet-based, or not Roman in its form. English itself is an object
that can easily be decoded by OCR software available today. The Italian visitor to
Cambodia, for example, might not be able to read a sign in the native Cambodian/
Khmer alphabet, but if that sign included an English translation an educated traveler
would probably be able to understand the information. With the assistance of an OCR
translation application, the English itself could even be translated into its Italian
equivalent to make the information available to anyone able to read Italian. Eventually,
when Khmer-Italian translation software is available the English language translations
would become unnecessary, but until that time they provide a method to make the
information accessible to the widest possible group of travelers.
5. Practical Augmented RealityApplications for Language Assistance in Tourism
The least technologically sophisticated augmented reality applications have the
potential to be most easily achieved, but they will not necessarily be implemented.
Attaching markers or barcodes to existing signs, for example, would allow for reliable
transmission of information in any language, providing tourists with translations for signs
such as high voltage or keep out. While some signs already provide pictographic
images to indicate no smoking or some similar concept, a bar code could provide
more sophisticated information in any language needed by the person using the
smartphone as a decoding device. It is quite easy, for example, to provide instructions
such as do not enter with a short barcode. One need not worry about the language of
the person scanning the bar code, since the smartphone would provide the correct
translation of the barcode information. Providing the same information in English would
also be useful, but to a person unable to understand English the barcode would be just
as useful.
Barcodes and markers can be decoded locally, with software residing on the
smartphone itself, if the set of phrases or languages required is limited. In this case no
Internet connection would be required. For true AR implementations, however, a
broadband Internet connection to enable access to a central database where a larger
set of information is required. For more sophisticated AR implementations, a variety of
databases will be queried, to provide even more information. In fact, the amount of
information available through AR browsers, even today, has the potential to overwhelm
the user with essentially irrelevant or misleading information. (Julier, Simon et.al., 2000)
What AR technology will eventually provide is a type of filter, or filtering algorithms, to
discard information we do not really require while gathering as much relevant
information as possible. We will also require a filter to authenticate information,
guaranteeing it is accurate, unbiased, and current, as shall be mentioned in the next
section of this paper.
The information problem presents itself because AR information will certainly be
primarily user-generated, as most Internet-based information is today. The most capable
augmented reality browsers in use today are constructed using something known as

ARML, or augmented reality markup language. As the name suggests, these browsers
use the same type of HTML/XML commands to construct their version of the real world
that web browsers use to construct our view of the World Wide Web. Developers of the
software and information that will be needed to make the augmented reality universal
language translation dream a reality will need to become proficient in ARML, and learn
how to work with at least one of the major AR platforms, such as Layar, Junaio, or
Wikitude. I believe the travel industry can expedite the adoption of AR by producing the
framework content creators will need to seed the AR world with information. It can also
provide some system to authenticate information in order to guarantee that users
receive accurate, unbiased, and current information. It would be in the best interests of
the travel industry to be proactive in its embrace of AR technology, working with major
AR players in order to help establish standards.
6. Future Challenges for Augmented Reality - Hardware Issues
Although it is most likely that augmented reality will initially become a part of the
travelers world through a device such as the smartphone, Arth and Schmalstieg (2011)
believe that smartphones are not yet truly ready to deliver full augmented reality to the
traveling masses. One basic problem smartphones have is they are not meant to
provide continuous camera operations for long periods of times due to limitations in
battery technology. I do not believe this is a serious problem, however, as I expect
augmented reality will be used only when needed, rather than on a continuous basis. I
believe that in the immediate future we will view the world through our own eyes for
most of the waking day, and only access an augmented reality version of the world
when we need specific information.
Other challenges for augmented reality relate to the slow processor speeds of
smartphones. OCR translation will certainly present the traveler with short time lags, as
a result of these slow processing speeds, making it impossible for road signs to be
translated instantaneously, for example, from a speeding car or train. Standing in front
of a sign, though, and waiting for a translation, will not be a great inconvenience for
most travelers.
A more serious issue relates to network speeds, and network availability. Network
connections will certainly be required for sophisticated AR applications to access all
information relevant to the traveler in the foreign environment, and these connections
should be as fast as possible to enable transfers of the large volume of information the
traveler will need to construct a complete augmented reality environment. Unfortunately,
due to the fact that augmented reality developments for smartphones are at the mercy
of hardware and network service providers, who make their decisions based on factors
that may not take the needs of augmented reality developers in account, this will be a
constraint on the use of augmented reality. We will certainly eventually begin to see
independent augmented reality devices brought to the market, perhaps in the form of
goggles, for example, that are developed specifically with the power and network needs
of augmented reality applications in mind.

7. AR Information Issues - Free Market and Regulatory Considerations

The most difficult issues with AR will present themselves as software issues, and social
issues. As we have noted, most of the information needed to construct a useful
augmented reality environment will be produced by common individuals. When I post a
remark on a social information network, in my native English language, about the the
sign I have seen on some store near my home in Japan, that remark becomes part of
the database of information available to all subsequent travelers to the area. If this
remark was posted to Twitter, for example, that remark now becomes part of the Twitter
database. If I have noted that this particular store is open for business from 9 a.m. to 5
p.m., that information could now also become available to any augmented reality
environment that is created using the Twitter database.
In this example I have not relied on OCR translation software to interpret the sign, but
merely on my own linguistic ability to produce a translation of the information posted on
the sign in Japanese. However, my linguistic ability may be incomplete, or even so
deficient as to produce errors. Perhaps the sign actually indicated that the store was
open during these times, except on Monday, when it is closed for business. As a result
of my own incomplete knowledge, or misunderstanding, I have introduced a piece of
data that is only partially correct. Others will now be interacting with this environment,
with their understanding of the store, on the basis of information that is not entirely true.
This error may have been introduced as a result of carelessness on my part. Perhaps I
failed to read the entire sign due to a lack of time. Perhaps this is the first and only error
I have made while attempting to add data to the Twitter database. Although my
translations are normally reliable, in this particular instance the translation was deficient.
However, there is also the possibility that I consistently post incorrect information.
Perhaps my linguistic skills are so poor that I should not be subjecting others to what
amounts to little more than slightly educated guesses about my environment. Who,
though, knows that the information I provide should not be trusted?
It is also possible, though, that I am purposefully posting incorrect information. Perhaps
I have a bias against this store, for some reason, and am consciously seeking to
destroy its business. Again, who is to know what my motives are when I post the
information, and whether or not that information can be trusted?
These issues are so important because augmented reality is a constructed reality,
constructed by information that is subject to concerns about credibility, timeliness, and
relevance. To the extent that any of these are compromised, the environment itself
becomes compromised, and perhaps seriously so. These same issues are present on
the World Wide Web, of course, and have not been resolved there. It is as possible to
find incorrect, biased information as it is to find accurate, relevant information on web
pages. Some of the most glaring biases or inaccuracies can be eliminated merely by
large numbers. If there are 99 positive reviews for a hotel, for example, from a source
such as Tripadvisor, and 1 negative review, I can probably assume the 1 negative

response was an aberration. What I still will not know, however, is if the person posting
this review consistently produces these types of reviews. A good filter would allow me to
eliminate all information this person posted if I discovered this person had no credibility.
Augmented reality browsers will certainly filter the information they provides to the
traveler and, as is the case for Google and other search engines, the way in which the
filtering is performed will determine how useful the information will be. Filtering, though,
also represents a form of control over expression, and we need to consider the
implications of this form of censorship.
There are also important privacy concerns that are related to augmented reality
environments, and these need to be considered by anyone involved with this new
technology. In one sense we are all now consumers as well as products. This idea goes
beyond the term prosumer, used by Alvin and Heidi Toffler, which was coined to
express our dual function as producer and consumer. (Toffler, 2007) But as human
social products, our very location, our being, becomes the source for data consumed
by others. To what degree do I have the right to deny others knowledge about my
location, about who I am, and when I surrender this right by checking in with a service
such as FourSquare, do I lose all control over those rights? These are some of the
concerns I hope the tourist industry will address when it works with AR developers to
help develop standards and establish methods by which user-created information can
be evaluated for credibility, timeliness, and relevance.
8. Conclusions
A successful tourism experience presents the exotic in a form that is understandable
and in some way familiar to the traveler. It does not overwhelm the cognitive or sensory
inputs of the traveler. The widespread adoption of augmented reality browsers, and
development of the software and information needed to enable them in any foreign
environment, as demonstrated in this presentation, will make the linguistic difficulties
and associated cognitive overload of past travel a forgotten inconvenience. Merely
viewing the world through one of these augmented reality browsers, most likely through
a device such as a smartphone, will transform the world and the tourist experience for
the traveler.
References
Arth, Clemens and Dieter Schmalstieg. (2011) Challenges of Large-Scale Augmented
Reality on Smartphones, ISMAR 2011 Workshop: Enabling Large-Scale Outdoor
Mixed Reality and Augmented Reality.
Azuma, Ronald T. (1997) A Survey of Augmented Reality, Presence: Teleoperators
and Virtual Environments, 6: 4 (August), 355-385.
Azuma, Ronald, et. al. (2001) Recent Advances in Augmented Reality, IEEE Computer
Graphics and Applications, (November/December), 34-47.

Bimber, Oliver and Ramesh Raskar. (2005) Spatial Augmented Reality: Merging Real
and Virtual Worlds. Wellesley, MA: A. K. Peters, Ltd.
Cawood, Stephen and Mark Fiala. (2008) Augmented Reality: A Practical Guide.
Pragmatic Bookshelf.
Feiner, Steven, Blair MacIntyre, Tobias Hollerer and Anthony Webster. (1997) A Touring
Machine: Prototyping 3D Mobile Augmented Reality Systems for Exploring the Urban
Environment, Personal Technologies 1 (4), 208-217.
Furht, Borko, ed. (2011) Handbook of Augmented Reality. New York: Springer Science
and Media, LLC.
Hollerer, Tobias and Steven Feiner. (2004) Mobile Augmented Reality in
Telegeoinformatics: Location-Based Computing and Services. Hassan Karimi and Amin
Hammad, eds. Boca Raton, Florida: CRC Press, LLC.
Julier, Simon, Marco Lanzagorta, Yohan Baillot, Lawrence Rosenblum, Steven Feiner,
Tobias Hollerer, and Sabrina Sestito. (2000) Information Filtering for Mobile Augmented
Reality in Proceedings of the IEEE 2000 International Symposium on Augmented
Reality.
Kent, James. (2011) The Augmented Reality Handbook - Everything you need to know
about Augmented Reality.
Madden, Lester. (2011) Professional Augmented Reality Browsers for Smartphones:
Programming for juniao, Layar and Wikitude. West Sussex, UK: John Wiley & Sons.
Toffler, Alvin and Heidi. (2007) Revolutionary Wealth: How It Will Be Created and How It
Will Change Our Lives. New York: Doubleday.
van Krevelen, D.W.F. and Ronald Poelman. (2010) A Survey of Augmented Reality
Technologies, Applications and Limitations, International Journal of Virtual Reality 9 (2),
1-20.