03

may 2013

AUGMENTED REALITY, ART AND TECHNOLOGY

Re+Public: Re+Imagining Public Space
BC “Heavy” Biermann

Pre-Digital Augmented Reality
Maarten H. Lamers Vestibular Stimulation

as an Augmented Reality Layer?
Antal Ruhl

AR[t]
Magazine about Augmented Reality, art and technology

MAY 2013
2

HOW & NOSm MURAL AUGmEnT, ThE HEAVY PROJECTS, ARTICLE On pAGE 8

3

Colophon
ISSN
2213-2481

COnTACT
The Augmented Reality Lab (AR Lab) Royal Academy of Art, The Hague (Koninklijke Academie van Beeldende Kunsten) Prinsessegracht 4 2514 AN The Hague The Netherlands +31 (0)70 3154795 www.arlab.nl info@arlab.nl

EDITORIAL TEAm
Hanna Schraffenberger, Mariana Kniveton, Yolande Kolstee, Jouke Verlinden

COnTRIBUTORS
AR LAB & PARTnERS: Wim van Eck, Edwin van der Heide, Pieter Jonker, Maarten Lamers, Maaike Roozenburg GUEST COnTRIBUTORS: Alejandro Veliz Reyes, Antal Ruhl, Lotte de Reus, Matt Ramirez, Oliver Percivall, Robin de Lange, BC “Heavy” Biermann

GRAphIC DESIGn
Esmé Vahrmeijer

pRInTInG
Klomp Reproka, Amersfoort

COVER
Our AR[t]y cover is a work by Royal Academy of Art student Donna van West who participated in the Smart Replica’s project, see: www.donnavanwest.nl

www.arlab.nl

4

Lamers AUGmEnTED EDUCATIOn Robin de Lange VESTIBULAR STImULATIOn AS An AR LAYER? Antal Ruhl HOW DID WE DO IT Wim van Eck AR[t] PICK: ImmATERIALS Editors' Choice BELIEVABILITY Edwin van der Heide NOnLInEAR STORIES TOLD BY CUpS AnD SAUCERS Lotte de Reus 5 .Table of contents 30 WELCOmE to AR[t] 50 WhO OWnS ThE SpACE 2 Yolande Kolstee 06 08 16 18 24 26 30 34 40 44 50 56 62 64 72 76 RE+PUBLIC: RE+ImAGInInG PUBLIC SpACE BC “Heavy” Biermann — PhD ThE MISADVEnTURES In AR Oliver Percivall AUGmEnTED pEDAGOGIES Alejandro Veliz Reyes MARTY. ThE nEW AFFORDABLE hEADSET — PART 2 Pieter Jonker and Yolande Kolstee A STUDY In SCARLET Matt Ramirez SUBJECT: InTERVIEW From: Hanna Schraffenberger To: Lev Manovich InTRODUCInG AR TO FIRST YEAR GRAphIC DESIGn STUDEnTS Yolande Kolstee PRE-DIGITAL AR Maarten H.

FINE ART STUDENTS SHOW THEIR WORK ON THE POP-UP GALLERY TWO GRAPHIC DESIGN STUDENTS (DAVE POPPING AND GABOR KEREKES) MADE A MOIRÉ-PATTERN AS A DECODING KEY 6 .

scientists. the magazine about Augmented Reality.arlab.. events and developments. look out for our call for contributions.. special attention is given to education: Experiences of bringing AR into the classroom are presented in separate articles by Alejandro Veliz Reyes. These drawings & 3D model will be downloadable from our website. in the part News Picks we post short news items on AR artists. Robin de Lange considers the long-term ramifications of extending both the mind and cognition itself with AR. Lotte de Reus discusses a spatial audio intervention to enhance museum exhibits. In Re+Public’s article we can read about blurring private properties’ boundaries. On this website. Maarten Lamers takes us back to pre-digital AR with his story on Pepper’s Ghost and at the other end of the spectrum Antal Ruhl explores the potential of using vestibular stimulation in order to create new AR experiences. which will be posted on our website soon! .nl. describing some AR software programmes which are widely available. In our new section AR[t] Pick. Yolande Kolstee. we feature a short science fiction sequel by Oliver Percivall. Wim van Eck continues his AR tutorials in the series ‘How did we do it’. In this issue. we have chosen 'Immaterials' — the result of a collaboration between the onformative design studio and Christopher Warnow. We invite you to visit our new website. Crucial to AR experiences is the concept of believability. and information about our experiments in the cultural domain. to print out at your local 3D print facility! Hanna Schraffenberger sets about interviewing Lev Manovich.WELCOME. A recurring topic is set out by Yolande Kolstee: the legal ramifications of AR initiated in AR[t] #2. to the third edition of AR[t]. which might lead to an item there. Furthermore. art and technology! In this issue we present articles by contributors from all over the world who are involved in stretching the borders of augmented reality. The results are a wide variety of implementations and reflections in different creative contexts. we share artworks that caught our eye. and in ways they were previously physically unable to do. Please feel free to contact us to tell us what caught your eye. We are very pleased to introduce to you the improved version of Marty: the video see-through AR head-up display. Should you like to contribute to issue #4. via the same URL www. Head of AR Lab 7 In the second part of this issue. on the edge of art and technology. We feature both articles with a philosophical perspective and articles with a more technical point of view. well-known among many people since the publication of his book ‘The Language of New Media’ in 2001. We are confidant you will enjoy this issue. in the attempt to leverage AR to allow artists to make incursions into public spaces. Matt Ramirez and Yolande Kolstee. which will provide you with all the information on our artistic and technical research — but also that of fellow researchers. explored by Edwin van der Heide.

2011). RON ENGLISH.OUTDOOR ADVERTISING: AR | AD TAKEOVER (NYC. photo by WILL SHERMAN BC “HEAVY” BIERMANN — PHD RE+PUBLIC: RE+IMAGINING PUBLIC SPACE 8 .

Re+Public is dedicated to using emerging media technology. Re+Public seeks to leverage AR in an effort to allow artists to make incursions into public spaces in ways they were previously physically unable to do. Re+Public has developed an experimental mobile device application that digitally resurfaces three specific areas of public space: outdoor advertising. or a willingness to break the law. OUTDOOR ADVERTISING: AR | AD TAKEOVER (NYC.RE+PUBLIC A creative collaboration between The Heavy Projects (Los Angeles) and the Public Ad Campaign (New York City). 2011) AR presents unique opportunities to creatively problematize the political and economic systems that shape the awareness of both individuals and the public writ large. With this goal in mind. the ability to pay for its usage. Blurring private property boundaries. to alter current expectations of our public media environment generally dictated by property ownership. murals. Seeking to understand a profit-driven governmentality and its affect on 9 . this article focuses on these domains and demonstrates how Re+Public has used AR to transcend current private property boundaries. which lies at the heart of our endeavor to re+imagine public space as a more open visual commons. and buildings. As such. and augmented reality (AR) in particular.

In New York City. To this end. During Art Basel 2012. we envision AR as the first step in the evolution of better tools of expression that can democratize public media production. in the space. users could trigger web-based information related to the showcased artists whose work has historically addressed commercial advertising in public space such as: Ron English. as Drawing from an international pool of talent. Viewing previous murals online. 2012) and Wynwood Walls (Miami. PosterBoy. In 2012. the mural series commenced with a recreation of Keith Haring’s famous 1982 mural followed by work by such recognized artists as Os Gemeos. digital world. Re+Public was commissioned by the Wynwood Walls to create an AR experience. Specifically. Kenny Scharf. we augmented ads in Times Square with artistic content. and Shepard Fairey. Re+Public investigated how civic authorities allow certain private parties to profit while preventing or discouraging other forms of public media production. Beginning with the 25th and 26th Street complex of six separate buildings. whereas other technologies tend to disconnect the viewer from their immediate physical surroundings. they might start demanding a better version of public messaging than the billboard default. Nunca. In other words. real estate developer and arts supporter Tony Goldman started the Bowery Mural with Jeffery Deitch and Deitch Projects. It is precisely this kind of spatial “aura” that distinguishes AR from other types of emerging media technology. artists who have contributed to the Wynwood Walls include Os Gemeos. 10 . at both the Bowery and Wynwood Walls sites. In other words. Dr. In 2008. Owned by Goldman Properties since 1984. OUtdoor MUrals: Bowery Wall (NYC. AR permits users to see these murals as if they were actually back on the wall. removes the viewer from the space by placing them squarely in the absent. Barry McGee. The Bowery Mural Wall is an outdoor mural exhibition space in Manhattan. Retna. Re+Public used AR to resurrect murals that once existed on the wall.public space. While users could certainly view the previous murals online. John Fekner. and Swoon among many others. in situ. we used AR to rupture public space with a new kind of artistic interactivity. Faile. D. AR connects the digital with the physical in an intimately “present” way. The AR | AD Takeover used street level ads and billboards to trigger a curated digital art installation that displayed on mobile devices. In addiIn 2009. by pointing a mobile device at the present mural. Saner & Sego. Once individuals experience this AR version of reality. 2012) In contradistinction to the use of AR to problematize the consumptive monologue of outdoor advertising. they move in perspectival relation to the viewer. and OX. that the current artist has painted over. for example. Our digital infiltration into public space and takeover of commercial ads created a place of dialogic interaction rather than a monologic con- sumptive message. We foresee AR mobile device technology as a first step in the transformation of public space into an arena shaped by user created content. Goldman endeavored to create a center that developed the area’s pedestrian potential. which commercial ads do not necessarily have to dominate. Aiko. AR is an incremental step towards showing the public an alternate view of their landscape. Goldman Properties and Tony Goldman who was looking to transform the industrial warehouse district of Miami also conceived the Wynwood Walls. users are able to see the former murals. Invader.

OUTDOOR MURALS: BOWERY WALL (NYC. With the How and Nosm. In considering our deployment at the Wynwood Walls in particular. over and above those embodied in the 2D mural. Aiko. In the case of each traditional mural. we made it appear as if the paint colors were draining out of the mural. we wrestled with this new type of work and wondered if it constituted a new mode of art. which commercial ads do not necessarily have to dominate. In addition to an immersive garden with a bridge. and flowing waterfall. and Retna. 11 . it is arguable that the AR assets represent original works in that they contain a sufficiently new visual expression of ideas. mural where all of the elements were separated in Z-space. Re+Public created 3D. Without pretending to discern any immediate resolution. we created a 3D Kabuki theater that allowed users to walk into a digitized version of the AR is an incremental step towards showing the public an alternate view of their landscape. with the McGinness. RESURRECTED KEITH HARING tion to resurrecting a Shepard Fairey mural that he recently painted over with a new mural in tribute to the recently deceased Tony Goldman. the AR overlay used the 2D paintings as feature tracking markers and source material to produce an original expres- sion of creative content. With the Retna. Finally. rather abstract 3D environment and permitted users to both pull apart and reconstruct the mural elements. Ryan McGinness. stream. we worked directly with MOMO and collaborated on an AR version of his indoor mural at the Nicelook Gallery on site. we built the mural shapes in 3D and animated them to extend out of the wall and placed them both on the ground and above the wall. interactive environments for four other murals by How and Nosm. 2012). we created another immersive. Additionally.

fUll view Retna MUral AUgment.Retna MUral AUgment. screenshot 12 .

in our attempt to use AR to re+imagine public space. we created the first example of what we refer to as “city visions. and their unique architecture much in the same way. in the visual urban messaging systems that surround them. we used AR to provide an artistic rendering of the re+imagined building by projecting it into a Blade Runner style future.” Specifically. we converted the Bradbury Building. This city vision type of deployment potentially provides more practical architectural and urban planning uses and maintains our notion that the AR experience should be spatially relevant in order to maintain the physi- cal aura that may have drawn the viewer to the building in the first place. made it possible for MOMO to put his art on buildings that he could not have accessed in his traditional 2D format. instead of placing converted art on the structure. the AR assets should maintain some logical connection to the building or space upon which we have attached them. Working with muralist MOMO.AUGMENTED ARCHITECTURE: PEARL PAINT. or entire buildings. in some meaningful way. site of many interior shots in the film Blade Runner (1982) into a futuristic version of itself. In this way. or “skinned” physical buildings in urban centers by overlaying 3D content onto the physical environment. To this end. the digital overlay will become a much more seamless and natural part of our daily existence. WILLIAMSBURG ART & HISTORICAL CENTER BUILDING AUGMENTED WITH MOMO URBAN ART 13 . It is our hope that these early entrants will help create experiences that consider art and design as an important part of the way the public adopts this technology. Both structures have a long “art history” in the city and AR allowed us to blur the lines between private and public space. we digitally resurfaced. It is vital to the health of any city that its inhabitants are able to participate. using AR. rather than the current commercial hegemony. we really see the city as a canvas that allows for a multiplicity of voices to enter into our visual landscape. In Los Angeles. we placed his art on both the Pearl Paint and the Williamsburg Art & Historical Center buildings in New York City. we have also experimented with using buildings. 2012) In addition to using both outdoor ads and murals as the markers that trigger our AR deployments. murals. FINAL REMARKS Whether it is outdoor ads. With the coming advancements in wearable computing. WILLIAMSBURG ART & DESIGN BUILDINGS (NYC. Converting MOMO’s 2D designs into digital 3D models optimized for mobile. Ultimately. Re+Public seeks to continue to deploy AR in an effort to democratize access to our shared visual environment and alter the current expectations of urban media in accomplishing our core mission of re+imagining public space. we chose three buildings that had a particular cultural significance and. 2012) BRADBURY BUILDING (LA. In other words.

design. which explores augmented reality. As a kind of synthesis between scholarly inquiry and emerging media. Heavy creates innovative interfaces between digital design and physical worlds in ways that that provoke the imagination and challenge existing styles of art. Building upon existing technological and theoretical frameworks. Since 2007. academia. BC has worked as both a university professor and a tech developer in Anaheim. 2012). art and semiotics in public space. Prague. and Saint Louis. OUTDOOR MURALS: WYNWOOD WALLS (MIAMI. and interaction. BC “Heavy” Biermann possesses an interdisciplinary background that comprises technology. BC has internationally presented his academic work. and the arts. photo by JORDAN SEILER 14 .BC “Heavy” Biermann Deriving his pseudonym from his penchant for philosophical discussion. RESURRECTED SHEPARD FAIREY. After finally giving up his painfully amateur skateboarding career due to a bum right knee… BC plans to use his extra time continuing to examine meaningful ways to fuse tech + creativity. With a PhD in Humanities [Intermedia Analysis] from the Universiteit van Amsterdam. BC founded The Heavy Projects [and its collaborative spin-off Re+Public] to investigate how the fusion of creativity and technology can uncover new modes of relaying ideas.

15 .

MARTY. 2011 16 .0 that is based on the HMZ T2. Mien S. Frido E. If they are valuable we will publish about it in our next AR[t] magazine. however. take a while. And finally: Yes we were already decades ago inspired by fighter pilots with heads-up display and of course Steve Mann. meaning that no large scale industrial production can be based on this design without the consent of the copyright owner. We are aiming at adapting Marty to version 2. R. ThE nEW AFFORDABLE hEADSET — PART 2 Pieter Jonker AND Yolande Kolstee In AR[t] 2 we announced the new MARTY videosee-through headset as a design based on the Sony HMZ T1. In markerless systems. Anna P. Jouke C. or anything at any time be my display?” DISCLAImER The industrial design of Marty is meant to do experiments by researchers. Pieter P. still many companies do not understand that you need two camera’s to see virtual objects in 3D and two camera’s to track (salient) keypoints in 3D in order to track your head pose while walking around with your headset in any unknown environment.nl. artists and designers in the area of AR and stimulate industry to come up with affordable AR equipment. Hetty van Emmerik. Anchorage. however. When the new AR Lab website will be in the air at about the same time as this AR[t] 3 magazine is published. which has another face mask. AR-toolkit like markers can do with one camera as the size of the marker is known and REFEREnCES ¢¢ Bridging the multiple reality gap: Application of Augmented Reality in new product development. One of the aims of the AR Lab is to stimulate AR in the world by putting our developments in public domain. But also lower backpain. Segers. It is. In the mean time be our guest and contribute to AR to come up with your own solutions. We are well aware that the Sony HMTZ T2 is on the market. the geek that walked around with an AR display. so anyone with access to rapid prototyping facilities can reproduce Marty for his/her own scientific or artistic research. Piet Van den Bossche. the air. Wim Gijselaers. we will post the design files as well as a photo series how to assemble the Marty. cramped shoulders from long computer days and the sighing. Jonker. natural features like dominant corners or lines in the scene are tracked. For more information: info@arlab. Alaska. with the aim to do research on co-operative AR. no assumptions on scale can be made and two cameras that see that same feature are necessary. Due to a lack of manpower and funds this might. hence there is extra knowledge on scale. subject to copyright. It was designed by Niels Mulder from studio RAAR on assignment of the TU-Delft. We are also aware of Google glasses and other equipment and we welcome those developments! However. partner of the AR Lab. Man and Cybernetics. Chatzimichali. “why can laying on bed or in an armchair not the ceiling or the white wall. October 9-12. Keep an eye on our website as we will also link to the software to do 3D pose tracking based on natural feature tracking. USA. Verlinden In proceedings of the IEEE International Conference on Systems. Smulders.

ARLAB. THE NEW AFFORDABLE HEADSET.MARTY. 17 .NL) WE WILL SOON POST A PHOTO SERIES ON HOW TO ASSEMBLE THE MARTY HEADSET. Image by FERENC MOLNAR ON OUR NEW WEBSITE (WWW.

nL> DATE: 2013/4/3 15:53 SUBJECT: InTERVIEW AR[t] mAGAZInE TO: LEV MAnOVICh Maybe you remember me from Facebook.WORDpRESS. 2001. I’d like to interview you even more. P. So I’ll challenge myself to challenge you. LEV. I finally also read The Language of New Media.---------.S.S. my supervisor (Edwin van der Heide) told me that I could/should be more critical towards my interview partners.FORWARDED mESSAGE ---------FROm: HAnnA SChRAFFEnBERGER <HAnnA@ARLAB. I realized that I would like to interview you about Augmented Reality for the AR[t] magazine. As a consequence. ImAGE SOURCE: WWW.JEWIShphILOSOphYpLACE. So I hope you’ll agree to an interview for the magazine? Best regards. .COm Dear Lev.P. ThE LAnGUAGE OF nEW mEDIA. ThE MIT pRESS. I work at the Augmented Reality Lab in The Hague and I am one of the editors of the AR[t] magazine. Maybe we can print my questions in issue 3 and your answers in issue 4? 18 MAnOVICh. After my last few interviews. A short time ago. When I read your article The Poetics of Augmented Space. Hanna P.

the user is presented with auditory information that relates to the immediate surrounding space. because I think that this point of view unnecessarily limits AR to the visual sense. Firstly. Wouldn’t you? Augmented Space What is special about AR compared to other forms of Augmented Space? In your article The Poetics of Augmented Space you discuss the concept of Augmented Space. one gets to listen to a mix of edited sounds that blend in with the sounds of the surroundings. monitoring. Augmented Space refers to all those physical spaces that are overlaid with dynamic information such as shopping malls and entertainment centers that are filled with electronic screens and all those places where one can access information wirelessly on phones. While navigating the environment. ubiquitous computing. 19 . In The Poetics of Augmented Space. you mention Janet Cardiff’s audio walks as great examples of laying information over physical space.Augmented Reality What is Augmented Reality? To begin with. for example. In The Poetics of Augmented Space you describe AR as ‘the laying of dynamic and context-specific information over the visual field of a user’. In contrast to ‘typical’ visual AR. as well as spoken narrative elements and instructions such as where to go and what to look at. Personally. And secondly. Besides AR. I would call this Augmented Reality. you mention several other technological developments in the context of Augmented Space. I would like to ask you what you consider Augmented Reality (AR) to be. among which. These walks are designed for specific walking routes. It would be great if you’d address the topic once more. tablets or laptops. because our readers might not have read your article.

20 . Edwin (my supervisor) and I have recently given this topic a lot of thought and we were fascinated by the questions: “What is actually augmented in Augmented Reality? What else can (we imagine to) be augmented?” We came up with the answer. nor does it relate to my environment. Listening to a radio play. but something else. something virtual augments something real. I’d even say that often it’s not the space that is augmented. However.tangible interfaces and smart objects. I don’t think augmentation is limited to a space or an environment. More specifically. the sound of the radio just exists as a separate. you mention software that performs tasks according to the mood. I might consider it an augmentation of the activity (not moving. most often. In AR. you have only discussed the augmentation of space. In our view. a song is just a song. just to find. For example. Here. I was wondering whether you have considered other manifestations of augmentation as well. For example. pattern of work. Hence. This information Information and space — one coherent gestalt? In The Poetics of Augmented Space you raise a question that intrigues me a lot. space is one of the possibilities. and has nothing to do with the surrounding space. we have considered things like augmented objects. Most of the time. that I’d turn around. information and the surrounding space can be perceived as one single mixed thing as well as independently. but likewise. sitting still) or an augmentation of the user (me). the ticking of a red streetlight might perfectly mix in with the rhythm of the song that is currently playing. I am doubtful whether our experience of a space is affected by this kind of information. there are more possibilities. to the activity the person is performing (sitting still). the virtual augments that to which it relates. Let’s imagine that my phone registered that I have been sitting still for a long time and reminds me to take a short break to stretch my legs. augmented content and augmented activities. something virtual augments something real. Do you think that all forms of augmentation bring along an augmentation of space or influence our experience of the immediate surrounding space? What else can be augmented? Something I really like about your article is that you see augmentation as an idea and a practice rather than a collection of technologies. The voice of the newsreader doesn’t mix with the voice of my colleague. the information and my spatial surroundings aren’t perceived as a single gestalt. augmented perception. So judging from my experience of listening to the radio. that nothing is happening there. Is AR just one of many related recent phenomena that play a role in overlaying the physical space with information? What’s special about AR compared to other forms of Augmented Space? that in AR. but nevertheless. relates to one individual in the space (me). Do the real space and the dynamically presented information add up to one single coherent phenomenological gestalt or are they processed as separate layers? I am a bit of a sound-person and it has always fascinated me that sometimes the sounds of a radio seem to mix in with environmental sounds. But of course. What is augmented depends on what the additional content relates to. I am curious whether you’d agree. so far. For example. augmented humans. but I don’t think it has anything to do with the surrounding space. focus of attention or interests of their user. there is a relationship between both. the newsreader might tell me about a traffic jam and thereby inform me about my immediate physical space. but I don’t consider it an augmentation of space. an event could sound so real and so nearby. However. But besides these two options. independent layer of content.

disappear or morph into each other in an otherwise real. information and space might be related. we find examples that date back centuries.g. a virtual bird is sitting on a real tree) there is no second space. Of course. smell like them. On the other hand. Personally. glass and special lighting in order to let objects seem to appear. Virtual Reality. The 3D models that are usually integrated in real space don't come close to the level of photorealism we know from cinema. But even if the concept isn’t new. which contains both virtual and real elements. Will photorealism be traded in for a form of realism that encompasses all senses? Do you think new media will develop towards a more multimodal form? AR & the second space From The Language of New Media. I was wondering: How does AR fit in the widespread aspiration towards realism? On the one hand. I understood that throughout media history. Often. When we imagine a typical AR scenario in which virtual objects are integrated into a real scene (e. For example. the virtual leaves the realm of virtual space and enters our real physical environments — with respect to that the images might be experienced as more realistic than ever… Will AR take the quest for realism to a new level? I can imagine. visual AR could be considered a huge step back. I don’t think of AR as a recent phenomenon. An example of ancient AR is the Pepper’s Ghost trick (which is discussed by Maarten Lamers on page 24). when striving for realism. even when they don’t add up to one phenomenological gestalt. there are more and more so-called AR applications. You note that new technological developments illustrate how unrealistic the previous existing images were. However. I’d like to ask it again with respect to the history of new media. taste like them and behave like them. If we look at contemporary AR and compare that with other forms of new media. Is this a fundamental change in visual culture? AR & the quest for realism The quest for realism in computer graphics New Media One of the main questions I want to ask you is: What makes Augmented Reality special? I have posed that question with respect to other forms of augmented space. what’s special about it and what isn’t? is something that has always bored me. this function of the screen applies AR & cinema In The Language of New Media. websites and CD-ROMs — to cinema. the screen was used to separate two absolutely different spaces.I think the same is true for Augmented Space. AR technologies and new media works that work with AR. At the same time they remind us that current images will also be superseded. current manifestations of AR might still bring something new and special to the table. you relate different forms of new media — e. So some questions I’d like you to answer with respect to Augmented Space are: When are information and space perceived independently from each other — would you still call these occurrences Augmented Space? When are information and space perceived as separate but related layers? And when and why do they add up to one single gestalt? equally to renaissance paintings and to modern computer displays. when we consider the concept of AR. It uses a second room. the virtual things that appear to exist in our physical space should not only look like real things — ideally they also feel like them. It’s the same physical space. physical environment. How about the relation between AR and cinema? 21 .g.

list on a page). (Technically speaking. all new media works are databases. Can we understand this as a pure database? What are the consequences of working with spatialized elements? What are the inherent limitations and possibilities when working with this form? (I can imagine it has consequences. The same could easily be done in AR. they will be inevitably ordered. So the only way to create a pure database is to spatialise it. In The Language of New Media you write about the elements of a database: “If the elements exist in one dimension (time of a film. L. The poetics of augmented space. semantic. (2001). we cannot assume that elements will form a narrative when they are accessed in an arbitrary order. Could we say that when working with AR. L. you did not only One of the main claims in The Language of New Media is that at their basis. a virtual pen and a virtual painting.” In AR. where an image of a dream appears over a man’s sleeping head. is constructing an interface to such a database. when creating a new media work. Do we have montage in AR as well? (You give the example of montage within a shot. These virtual objects are displayed as part of a real room when a user views the augmented environment with a smartphone. Visual Communication. you also pointed your readers towards aspects of new media that were still relatively unexplored at that time and you suggested directions for practical experimentation.) Does visual AR use similar concepts as cinema? Does cinema use other techniques to create fictional realities that are not (yet) used in AR? Does AR use techniques that might be adapted by cinema in the future? lationship to something real. this can be seen as an analogy to compositing in films: an attempt to blend the virtual and the real into a seamless whole ‘augmented’ reality. 5(2). AR research is very concerned with registering virtual objects in real space. provide a theory of new media. Are there certain aspects of Augmented Reality you consider especially interesting for future experiments and explorations? AR & future research References 22 . 219-240. The language of new media. As far as I understand it.I’m certainly not a cinema expert. virtual elements are distributed in real space. You argue that what artists or designers do. distributing the elements in space.) AR as spatialized databases With The Language of New Media. we could say the real world serves as a database index for those virtual elements. (2006).) What is the interface to access the database? Is it my phone? What does the artist create? I think it is usually the virtual content and its re¢¢ ¢¢ Manovich. Do you agree? You oppose compositing to montage: while compositing aims to blend different elements into a single gestalt. montage aims to create visual.. stylistic. The MIT press. for storytelling? As you point out. Let’s apply this database theory to a typical AR scenario in which virtual objects (seem to) appear in a real environment. artists and designers create a database for an existing interface? I have one more question about databases. but I guess most of what we see in visual AR has been present in cinema for a long time. Manovich. For example. e. and emotional dissonance between them. The database might hold a virtual chair. We can see this as a database filled with virtual objects.g..

“[.” Lev Manovich. distributing the elements in space..] the only way to create a pure database is to spatialise it.. The Language of New Media 23 Image COURTESY OF LEV MAnOVICh .

I was most impressed by the augmented reality of Pepper’s Ghost. I lied. Besides meeting Mickey Mouse. 1978 n Valentij rten a a M PRE-DIGITAL AUGMENTED REALITY MAARTEN H. LAMERS In the first issue of this magazine.. I mentioned how the “Mozzies” mobile game of 2004 was my earliest encounter with augmented reality.. 24 ...Disneyla nd. With no computer required. by 26 years! In 1978 my parents took me and my brother to Disneyland.

paU PaUl BoU . if you know other pre-digital augmented reality techniques. . I asked my dad how the illusion worked. digital technology lets us define.Image coU rtesy of D avid http Llewellyn ://ravenmanor . Actually. In effect. and on the seat between us appeared the scariest moving ghost ever! Instantly our heads turned. in current head-up see-through displays Pepper’s Ghost technology is less apparent but nonetheless used in the same fashion. Pepper’s Ghost technique was first described in the 16th century and later refined by John Pepper around 1860. most realistically.net lboUrke rke.com/projects/holodesk/) project. its use is apparent. 25 Image of coUrtesy . Disney augmented the reality that we hold our mirrored image to be: pre-digital augmented reality. facing each other. Naturally. darkness and reflecting light.nl. Thank God. The mirror showed us some weird illusion. render and interact with virtual content. However. To me it is interesting that we still rely on John Pepper’s idea to add digital content to our optical reality. But good old-fashioned Pepper’s Ghost projection is what augments our reality with that content.microsoft. rotated 90 degrees to the right. send a short description to lamers@liacs. of which I can remember only one thing: the car stopped. a mechanical funhouse car drove us through Disney’s “Haunted Mansion” ride. Who would have guessed that such basic illusionary tricks are crucial to what we now consider cutting-edge technology? In fact. Scary stuff. but also as part of modern optical see-through AR technology. the ghost wasn’t really between us. Combining this with a large mirror.com Sitting beside my brother. and help me put AR in perspective. It is still used in amuse- ment parks and museums today. In Microsoft Research’s HoloDesk (see research. he explained what is known as Pepper’s Ghost: a simple but clever technique that creates holographic scenes in 2 or 3 dimensions. He explained something about glass. facing a large mirror.

I used my body orientation to alter my balance. 26 . Applying a specific current to those mastoids will result in an altered balance perception related to both to the intensity and the direction of the current. Galvanic Vestibular Stimulation (GVS) stimulates the human vestibular system (balance organ) by means of electric stimulation. In order to use GVS as a form of augmented reality. To find out whether GVS can be used as a new form of AR. This technique has been around for a long time and many studies have been done in order to learn more about the human balance system. Since I also wanted to investigate what the effect would be to our balance system. I constructed a device and developed a series of experiments. but never has it been used as an extra sensory input. usually an anode and a cathode. When I became interested in galvanic vestibular stimulation I realized that this could become a whole new form of AR. I needed a connection to the physical world. Two electrodes.ImAGE BY KRISTIn SmITh VESTIBULAR STImULATIOn AS An AUGmEnTED REALITY LAYER? Antal RUhl Over the past few years the field of Augmented Reality has started to include non-visual forms of augmentation to their scope. are placed on the mastoids (the bones behind the ear).

In modus B. it would push me back to the center. own orientation. while modus B could amplify it. when put in modus A. Without even using external data to augment an extra layer to reality I created an altered world based on data from the physical world. There were two possible modi for the control unit. based on my 27 . This provided me with the setup as shown in Figure 1. I realized that. The data from the accelerometer is sent to the control unit. and to make sure this wouldn’t influence the results. I mounted an accelerometer on top of my head-mounted GVS device in order to measure its orientation. it had the opposite effect. The control unit calculates the appropriate intensity and direction of the current in real-time and provides feedback to the electrodes. In other words.GALVANIC VESTIBULAR STIMULATION CAN BE USED TO ALTER BALANCE PERCEPTION Setup To do so. Modus A could counteract my balance. it felt like my resistance was really low since the device backed up every movement I made. it felt like I was moving through a liquid or a thick syrup-like medium. in modus B. so it took more effort to move around. Is this a form of augmented reality or is it an alter- Conditions The first step of my research project was measuring the effect of an altered balance. The middle part is left uncovered to retain the full use of users’ auditive orientation. While testing. The headphones are in fact merely rings around the ears. if I was tilted to the left in modus A. The GVS device counteracted all my movements. it would push me further to the left. The electrodes are incorporated in a pair of headphones to make sure they are pressed against the mastoids properly. Testing this simple setup immediately revealed some potential AR applications of vestibular stimulation.

The tests that I have done focused on enhanced performances in everyday life situations. modified and used as an input to alter the actuators (in this case the electrodes which in turn alter my own balance). Other issues were the bright light flashes I experienced while testing. which could be used to train divers or astronauts who work in other environments. Using only these two simple orientationbased modi. Tests involved bus rides. perhaps more Figure 1: Galvanic VestibUlar StimUlation device troublesome. but for what use? The examples point towards simulations of the physical world. However. This might be of use for people with an impaired balance system or in those situations when accurate balance is crucial. I have used this device for over two weeks to see if there was any progress. Another. we can already create an enriched environment. which is the broader field of manipulating users’ sensory perception through a wearable device. This system could enhance a simulated environment by distorting the balance organ based on the simulators’ virtual orientation. I experienced some burn marks of the skin around the mastoids.) is measured. Although we have to be very careful about drawing conclusions from these results (given the self-experimentation and singlesubject constraints). and nausea. before we can use this. experience I had was the adapta- 28 . The skin’s resistance is an important factor in getting the electric pulses down to the desired area. walking to targets on the street. I found that in certain cirnate reality? Be that as it may. Or simulate the G-forces in a racing game’s sharp corners. But these issues might all be overcome with the right execution. Using electrode gel and covering the electrodes with sponges soaked in a salt-water solution did the trick. walking over a balance beam. For example. Can this be used as an interface? I demonstrated a system in which the real world environment. I was wondering whether I can diminish or amplify motion sickness. But you can also think about fighter pilots who fly in simulators that don’t alter pilots’ balance when flying upside down. But can we also use it in a more traditional AR environment? In other words. This creates a broad range of possibilities. (in this case body orientation. it does fall within the definition of mediated reality. During the testing phase. cumstances my balance actually improved while using this device. and letting them experience their surroundings in a modified fashion. there are some practical considerations to keep in mind. But a more practical solution must be found. both are not very userfriendly. playing Wii Balance Board games and many other examples. improve my balancing skills etc. could we use this system to add computer generated data to an extra sensory layer of our environment? The easy answer is: Yes! It’s not hard to imagine that you can enrich game play with wave motions for example. Using my self-constructed device. for now.

Your body is extremely good at adapting to new situations and environments. Media Technology at Leiden University and Design and Media Arts at UCLA. can be found here: www. And it did so in two ways. Apparently I was much more sensitive to a change in stimulation than to the actual stimulation itself. While using GVS for augmented reality purposes is still in a research stage. The same effect occurred with my balance organ. I saw that. Arnhem. Antal worked at distinguished design companies in Amsterdam and Barcelona.com/media_tech/paper. it gave me the experience as if I was moving through a liquid. when I was wearing the device while it was set in modus A. giving off the impression of moving through a low friction environment. Antal studied Industrial Design at the The Hague University of Applied Sciences. it did so quite convincingly. Experiments with Galvanic Vestibular Stimulation in Daily Activities: An Informal Look into its Suitability for HCI. the effect was constant. Antal has also started a company to develop interactive installation for brands. So when I turned it off. and only changed the pulse-width to affect my balance. You might remember that study from the 1960’s where a test subject was wearing reversal goggles. the subject saw the world upside down because the body had adapted to this new situation. Secondly. it felt like I had switched it to modus B. Los Angeles. science and art. The presented paper. the possibilities are endless if you use your imagination. has had several (national and international) performances and exhibition.tion effect. which are very important design-considerations if we want to use a GVS device as an interface. But when I used an alternating pulse. travelled the world and currently works a freelance artist/ designer. my body had to adapt to this ‘new’ non-stimulated situation. This article is based on research presented at Chi Sparks 2011 (Chi Nederland 15th conference).antalruhl.pdf Antal Ruhl Antal Ruhl is a media artist with a background in design. After removing the GVS device I experienced the exact opposite. Interactive and playful objects are much better in holding someone´s attention while they convey their message”. The Netherlands. “Using the technological possibilities at hand we can enrich our work and create an engaging object.antalruhl.creativeinmotion. Creative in Motion is a creative brand activation studio: www. These objects vary from kinetic sculptures to interactive installations. After removing the goggles. but they always share the goal to intrigue people.com 29 . Sometimes they are purely conceptual or formal and sometimes they serve a more commercial purpose. While my body needed some time to get used to my ‘new and improved’ balance. When I used a constant pulse on the electrodes (a constant current in the same direction) the effect diminished quite soon. and removed it after that time. festivals and event. published a scientific paper.nl For an overview of his work or to contact Antal please visit: www. when I was wearing the device for about fifteen minutes. thus. After a few days the brain adapted to the newly displayed environment and reversed the image back to normal. but at a much quicker pace. His work can be described as visually and technically attractive with a focus on natural and physical phenomena. 2011. June 23. Antal creates objects that let us rethink our environment.

but is in fact fake. or in other words. The e-mail is believable. while not real. We’ve learned to get suspicious and need to verify the credibility of the story. we enter a situation in which the content appears to be believable. What we also learn from 30 . The examples from the bank’s e-mail as well as the novel show us that believability is actually independent from something being real or fake. When we read a novel.BELIEVABILITY Edwin van der Heide Believability is something we deal with on a continuous basis. When we receive an e-mail that claims to be from our bank with the request to enter our account information on ‘their’ website in order to upgrade its security. and we enjoy it. appears to be authentic. An important factor for a story to be believable is that we can relate ourselves to it (or the story relates to us). the story can be entirely fictional and it nevertheless draws us in without questioning its realness or truthfulness. The story is believable. independent from something being real or ‘virtual’. it pretends to be real.

When I was writing my paper about the interactive art installation Spatial Sounds (100dB at 100km/h) for the Third International conference on HumanRobot Personal Relationships (HRPR) I was introduced to a topic addressed by Kerstin Dautenhahn (2007) that fascinated me: The believability of a Robot.Image coUrtesy StUdio Edwin van der Heide the example of the novel is that things that happen in the story don’t have to be possible in reality. PittsbUrgh. time-travel. The reason that it fascinated me is that it made me not only think about believability in the context of robots. I realized that the believability (of the behavior) of an artwork was not (yet) seen as a fundamental topic and might deserve its own study and experimentation. 31 . USA. more specifically. We can ‘simply’ imagine these things and believe them. but it also triggered me to think about the believability of an artwork and. 2009 .Spatial SoUnds (100dB at 100km/h) at Wood Street Galeries. or never go to bed. the believability of the behavior of an interactive artwork (as opposed to the believability of a robot’s behavior). In a novel we can meet creatures from Mars.

How do the two differ from each other? With the robot we can imagine that he read our mind and therefore made a coffee for us. It’s by imagining that we can change what we believe and thereby turn a machine into a believable affective robot. There is another possible difference between the coffee machine and the robot making coffee.e. Spatial Sounds (100dB at 100km/h) is an example of a believable robot in the sense that the visitors believe they 32 . it has to be useful and perform tasks around the house. for example. however. imagine that the coffee machine made that coffee especially for us things start to change. welcome and/ or thank the user for using it. It is a machine-like object but does not resemble existing machines. a robot (or interactive installation) doesn’t have to represent something (else).Kerstin Dautenhahn has an interest in socially intelligent robots: “A robot companion in a home environment needs to ‘do the right things’.” I tried to imagine an example and came up with the idea of a robot that makes and brings you a cup of coffee. If we. i. if it lasts or wears off. but it also has to ‘do the things right’. We imagine that the robot has a certain amount of intelligence and. Soon after I had that idea I had to think about a full automatic coffee machine that grinds the beans and steams fresh milk for each individual cup of coffee that it makes. in a manner that is believable and acceptable to humans. It is interesting to question whether these machines indeed make us believe they show affection. it allows us to somehow identify ourselves with it. This might be the reason that in Japan a lot of machines talk or include animations to. and if so. Does this then mean that the representation is a requirement for believability? No. It can be an abstract work that is believable on its own. Nevertheless. In the case of the coffee machine. we don’t imagine any intelligence and we simply think of it as a machine without affection for us. In the HRPR paper about Spatial Sounds (100dB at 100km/h) I put it like this: “The installation can be seen as a non-verbal abstract robot and does not imitate an animal or human-like look or behavior. i.e. Robots are often made to look and behave like humans (the humanoid) and the coffee machine isn’t. perhaps. even ‘feels’ affection for us.

while we keep on believing other things that we know are fake. 27-33. Japan. it can be abstract and nevertheless be believable. Spatial Sounds (100dB at 100km/h) in the Context of Human Robot Personal Relationships. We have also learned that something doesn't have to be possible in order to be believable. the real and the virtual in augmented reality believable? We have learned that something doesn’t have to be true to be believable. 2011). 59. (2007). we’ve learned that something doesn’t have to represent something. LNICST Vol.. K. J. Philosophical Transactions of the Royal Society B: Biological Sciences. completely lose their believability. E. 362(1480).edu/ research/theses/abstract-affective-robotics Spatial SoUnds (100dB at 100km/h) at DAF Tokyo. Interesting to read in this context is Alwin de Rooij’s (2010) graduation research project for the Media Technology Master program on ‘Abstract Affective Robotics’. 2006 — Image coUrtesy StUdio Edwin van der Heide 33 . A.leiden. 679-704. http://mediatechnology. What aspects make combinations of. ¢¢ van der Heide. (2010). ¢¢ de Rooij.. In: Lamers M. The former makes me believe that believability forms an interesting perspective to think about what we can imagine in augmented realities! There are certain things we believe in that.understand the behavior of the installation and find it worthwhile to interact with. and interactions between. The aspect of believability is so strong that people accept the installation as a real being and want to interact with it over and over” (van der Heide. (2011). References ¢¢ Dautenhahn. Verbeek F. once we discover they are fake. H. Besides that. Human-Robot Personal Relationships (HRPR 2010). Abstract Affective Robotics. Socially intelligent robots: dimensions of human–robot interaction. I’m curious how the abovementioned thoughts apply to augmented realities.

Maaike RoozenbUrg Introduction In museum exhibitions historical objects are usually shown by visual display. the so-called ‘smart replica’ was proposed in the previous issue of Ar[t] (Roozenburg. Depicted in figure 1. in the shape of a 3D print.NOnLInEAR STORIES TOLD BY CUpS AnD SAUCERS Smart Replic as with responsive 3D aUdio Lotte de ReUs. To provide a way around this situation. that stretches the boundaries of the replica’s concept as an autonomous object based on a historical artefact. New methods of access and new digitization strategies based on the study of the relationship between the ‘bits’ and ‘atoms’ are being developed. something that needs reconsideration in our present time where the ‘experience’ is essential. in a showcase with extra textual information added to it. Starting with the first import of porcelain from China in the seventeenth century by the Dutch East Indies Company. As a result. The objects that are central in this project are seven teacups and saucers that are part of the collection of Museum Boijmans Van Beuningen. These are currently on display in a new exhibition on design and pre-industrial design. but to analyse. let alone use them. In other words. communicate and enhance those qualities of the historical artefact that are the most meaningful to us. Completed as a graduation project. 2012): a new kind of reproduction. Here we present the design project of Lotte de Reus in connection with this paradigm shift. Museum visitors can never touch the objects. JoUke Verlinden. visitors ‘scan’ the displayed objects from a distance. it presents an auditory environment to augment the artefact in an unobtrusive and non-linear way. each of these teacup and saucer sets represents a milestone in the Dutch history of porcelain. 34 . and ending with the small scale production of porcelain in the Netherlands. followed by the invention of Delft’s blue as an attempt to copy Chinese porcelain. the goal is not to make the most realistic copy of the original. now. We hypothesize that by using these 3D imaging techniques the value of our cultural heritage can be increased.

Our hypothesis is that more active means will lead to a more comprehensive museum experience. including museum shops. These types of encounters could correspond with respectively a museum or a theme park. consisting of stories. An important part of the collection are the nonlinear narratives. the exhibitions of Boijmans Van Beuningen can be characterised as enriching experiences: an encounter in which the visitor is conscious of the artefact and the (hi)story it represents. materials. paying a visit to an exhibition enables the visitor to have an intimate encounter with the artefacts on display (van Dongen. 2012). The artefacts shown and experienced in an exhibition construct a powerful medium which instils visitors with thoughts they perhaps have not yet fostered. –Visitors’ overall interaction with Boijmans’ exhibitions can mainly be described as scanning. The engagement of visitors is limited as they are not experts on the particular subject of the exhibition. They draw audiences in through exhibitions and related facilities. and tate the active assemblage of old and new stories – and how do these stories refer back to the replica’s original? How can digital databases be employed in linking smart replicas to their collections? On a philosophical level: does the original still attract interest? In the case of pre-industrial utensils such as the teacups and saucers this question is very relevant. while the information passively waits for the visitor act upon it. and integrity. The participants walked into a room. understanding. satisfaction. Augmented matter — the mixture between bits and atoms — allows novel interaction techniques to embody these networks of information. In the vision of the curator. thereby increasing the opportunity to reflect and learn even after the visit. This means that a museum requires the visitor to reach out for information. Concept of tangible interaction with responsive audio For the seven teacups and saucers that have been chosen to represent the history of the Dutch quest for porcelain. Because of the passive character of the objects and their corresponding information. 2012). networks of information associated with the object. leaving the exhibition behind. context and reference. as shown in Table 1. This storytelling model categorises the information in three layers: engagement. 35 . In the project discussed here. Knowledge and culture become more accessible because of the actual presence of the objects. locations. These qualities feed the resulting research questions: How can museums anticipate and facili- Figure 1. THE SEVEN CUPS AND SAUCERS picture by HUGO BES In the spectrum of recreational activities. For this design project our aim was to convey the following qualities of interaction: intrigue. the ECR-model is used to classify the valuable information (Wong. full of untold stories. looked at an object briefly. went on to the next. rituals. the collection’s past and the like. it takes effort on the part of the visitors to maintain concentration.Active enrichment but not entertainment? Museums and their collections house a specific group of objects with their corresponding historical data. a focus group that visited the museum exactly proved this problem case in hand. restaurants and so forth. started at a random showcase.

Context: Draw the visitors’ attention. to browse the additional information for example. available after the exhibition. Interaction characteristics Spatialised audio. The app contains more background information on porcelain and the objects. the spatial audio of a narrator starts.ECR_Phase Engagement: Draw the visitors’ attention. Enabling spatial audio As discussed in Table 1. The audio clips are played when the visitor shows interest through their actions. the moment when a visitor has some immediate sensory. Information plaque.  Narration 2 — information on specifics that can directly be related to the object (audio). These audio clips give the impression that they are experiencing more: it is as if their auditory attention has doubled their visual attention (Erens. the narra- tor focuses on the features of the teacup or saucer (“audio specifics”). Triggered by handling an object. she can consult the app at her convenience. Background information (text and images). Proposed dialogUe between visitor and exhibition. connecting the narrative with what is visualised about the object. Exhibition app. every teacup and saucer holds a series of three audio clips. The narrator tells about the role the specific objects (“audio story”). Arriving in close proximity of the teacup and saucer. Narration 1. Table 1. it attracts the visitor to go to other objects accompanied by a spatial soundscape. Triggered by proximity to object. Identification data (text. Close proximity to object. c. When the visitor picks up a replica. Ambient soundscape (audio). e. Together with an app that is made available. When the visitor returns home. emotional or intellectual response to the artefact. b. proximity to an object and the han- 36 . When the visitor is in the proximity of an object. In this stage the visitor is encouraged to turn and explore the object. visitors are attracted to participate in the experience by other visitors who are listening to the 3D audio clips. 2012). In the first phase. It provides more detailed and interpretive information about the work. emotional or intellectual response to the artefact. the moment when a visitor has some immediate sensory. Reference: Gives the visitor the opportunity to draw conclusions and connect to related resources. the visitor is intrigued and triggered to participate in the exhibition. d. depending on user location / viewing angle. The visitor can put down an object and pick up another object. a 3D ambient soundscape that fits in with the history of the specific teacup and saucer will appear — once the first soundscape has been heard. visual). Information a. while retaining the soundscape. information on the role (audio).

Presenting the information via audio. The technology to spatialise audio was developed two decades ago. does not clutter the visual sense. position and orientation tracking of the human head as well as the objects at hand is required. it works more associatively. they were asked to choose between the two displays and support their reasoning. The soundscape is the auditory equivalent of an ambient image. 7-10 February 2013). as they are fit for indoor use. Furthermore.dling of an object. cf. The character of 3D audio relates to the idea that the sounds seem to come from sound sources placed anywhere in a space (a surround sound effect). Figure 2. Here optical or magnetic tracking principles also make sense. scanning behaviour can be transformed into an immersive encounter. without visual clutter and conserves traditional values on how art should be experienced. Presenting the information via responsive audio facilitated the visitors to consciously turn the object to find the image that the narrator referred to. Preliminary evaluation The core of the concept was tested during the Object design fair (Rotterdam. Ambient soundscapes were on or off. Participants appeared to be pleased with the responsive auditory system. non-visual immersive content. and speaks directly to the visitor (Erens. the brain places the sources of the audio clip in your head. see Burgess and Verlinden (1993) for example. Information about the teacups and saucers was presented via text or audio clips. even those visitors who preferred text to audio clips. 37 . 2012). typically with so-called earcons which represent specific objects or events. SPATIAL AUDIO INTERFACE OF AUDIOSTAGE. Auditory feedback enriches the visitors’ experience actively. whilst the visitor is holding the object. To connect this aspect to the exhibition visitor. yielding four permutations. Figure 2. The essential observations include: Participants seem to enjoy the ambient soundscape: it triggers the imagination and the recognition that the objects used to be utensils and not art object as they are now. When listening through headphones. With the use of tangible replicas and 3D audio clips. 140 people interacted with a selection of the configurations presented in pairs. We used AudioStage to produce our audio clips with a visual interface. An example of a high quality 3D audio clip is “In your head” by Big Orange. audio clips allow temporal cues.

triggers an immersive encounter in which this sensibility is restored: the passive. understanding is nurtured because the encounter is intuitive and the information presented by the artefact responds to the visitor’s body language. it rather extends the existing strengths with emerging technologies such as indoor tracking and spatialised soundscapes. tangible interaction with physical replicas and the connection between information and corresponding visuals.Conclusion Once the teacups and saucers were objects of dailylife and their form. and can be connected to various forms of social networking websites and location-based services. the experience is saved.  Implementation of responsive audio tours in the current exhibition context. texture. In essence this proposal presents a new type of relationship between visitor and object that has interaction qualities equal to a human conversation. the experience fits in with the integrity values of the museum Boijmans Van Beuningen – the 3D scanned and printed. Thirdly. moulded physical replicas afford what Dutch historian Johan Huizinga has called a “historical sensation”. decoration make sense primarily in the context of their functions and relations to other objects. by the layered structure of the narratives. 38 . Lastly. 2005). The proposed system is by no means the first auditory guide for exhibits. a satisfactorily quality is propelled. Because the visitor has a direct intellectual sensory dialogue with the object. weight. one-sided encounter with the objects now becomes an active two-sided encounter. this poses an intriguing quality that pulls the visitor in by using 3D audio clips and ambient soundscapes. it is more or less compatible with existing gear already employed by many museums. the feeling as though you are somehow “in touch” with the past (Ankersmit. this 1. colour. will lead to a more meaningful experience. Firstly. Combining ambient soundscapes. as well as the people who used them. 2. Secondly. creating curiosity and making visitors want to engage with the artefacts.  Experimentation with augmenting untouchable artwork with ambient soundscapes. which can be browsed in a non-linear mode by the visitor. Through the app. Future work includes experimenting and researching the effect of the design in the environment of a museum: Figure 3. substance. Impression of the preliminary evalUation. Furthermore.

fr/web/en/prods/audiostage ¢¢ Interactive installations regarding preindustrial utensils: http://new. Lotte de Reus Lotte de Reus recently received her Master’s degree in Design for Interaction at the Faculty of Industrial Design Engineering at Delft University of Technology. ¢¢ Halbertsma. Amsterdam. Many thanks to DOEN foundation for funding the Smart Replicas project (including the graduation project of Lotte de Reus) and many thanks to Mareco prototyping for their contribution in the form of 3D micro prints. M. D. Sublime historical experience. Wim Verwaal. something she had not yet encountered in her studies. (1999). C. Wolf Song and Ir. Verlinden. New technologies such as augmented audio. In July 2012 she started her graduation project for the Smart Replicas project.php ¢¢ Van Dongen. ¢¢ The Futures Channel. ¢¢ Ministerie van Onderwijs. 14 December. Bewaren om teweeg te brengen. Interview on the museum context.nl Acknowledgements We would like to thank Alexandra van Dongen.C. Rotterdam: Erasmus Centrum voor Kunst.  Creating guidelines on how the concept could be implemented to suit different kinds of objects in the museum (or maybe even utensils in our everyday context).com/conversations_ archive/wong_conversation. Conversation with Curtis Wong.R. References ¢¢ ¢¢ Ankersmit. During the project she was pleasantly surprised by the world of audio. More information ¢¢ Video presentation of this project (headphones required): http://www.nl. Proceedings of Virtual Reality Systems Conference (Fall ‘93).  Prototyping indoor tracking and interaction sensing possibilities. (1993). special interest for her. 39 . (2005). 4. Stanford University Press. we would like to acknowledge the valuable advice and support by Cilia Erens. Her portfolio can be found at www. Furthermore..longcat. Lotte would again like to work in the domains that combine art and technology.big-orange. Professor Joris Dik. immersive soundscapes and 3D audio now hold a new.youtube. (2012). A. (1995). J. [online] Available at: http:// www. Dr. Burgess. 21 November 2012. Interviewed by Lotte de Reus.blogspot. Interview on soundscapes. with special attention to smartphone infrastructure. F.nl Official Smart Replica blog: http://smartreplicas.thefutureschannel. An architecture for spatial audio servers. Museale strategie. com/2008/05/new-work-detroit-institute-of-1 ¢¢ ¢¢ In your head. dierentuinen en musea.lottedereus. Interviewed by Lotte de Reus. ¢¢ Erens. (2012). Cultuur en Wetenschap (2005). she was driven by a fascination for porcelain and the wish to create effective storytelling experiences. a high quality 3D audio example: http://www. Themaparken. Rotterdam.com/ watch?v=enR1Ggbuf_8 ¢¢ Software to render binaural output by visually placing audioclips in 3D: http://www. curator at the Museum Boijmans Van Beuningen for her constructive collaboration. In the future.pentagram. A. Woerden: Drukkerij Zuidam..en Cultuurwetenschappen.3.

GUIDO LAGERWEIJ 40 40 .Image: REVEALInG IDEnTITY.

I contended that we as passers-by might not be aware of any virtual content at all and. unsure whether or not they are they are talking about you… Sander Veenhof.463.185. we don’t have the means with which to recognize the virtual content floating in the air around us. In AR[t] 2. when we don’t have the right app (software) and/or mobile (hardware). seeing his art work Biggar and playing with it for the first time. in theory. the blog Wassom. made a lot of people aware that AR could be everywhere.678 virtual cubes whose colours can be changed by simply tapping on a mobile phone. amongst other sources. helps them to realize that this AR art piece encapsulates the entire earth with its 7. this is not the case. The only thing you need to do 41 . everybody has the same access to virtual content. For some this might be a bit frustrating. and actually is everywhere and that this is a special characteristic of AR for as a new medium for artists. de facto.com. Dutch AR artist.WhO OWnS ThE SpACE 2 discUssing legal issUes of AR Yolande Kolstee WHAT IS PRESENT IN AR SPACE? As mentioned in ‘Who owns the space’ in AR[t] 2. giving off the impression as if people are gossiping about them in a different language or at a very low tone. we keep a close look on the developments concerning the invasion of our physical space with virtual information. using information from.and artistic purposes. Although. For many people. and discuss some legal aspects thereof.

This adult content will certainly appeal to a lot of people and this will definitely result in another large group of people coming into contact with AR. 42 42 PREVALEnCE OF AR John Moe (the host of Marketplace Tech Report and handles web content for the program). Natural feature tracking systems are in rapid development. detect the salient points.” The group is composed of three young Londoners who decided to make a public case against Google Glass and other similar devices. and negligence. On March 11. what happens when this information about our eye-movements is stored in databases.to actually see Biggar. ranging from public resistance to ‘adult’ AR (AR porn that is) via licensing. let our camera survey the space. wrote in 2011 that “Augmented Reality has been the Next Big Thing for a while now. entering information about the physical world (‘burglary information’ as we called this in the last issue of AR[t]) into the virtual world. TWO WAY TRAFFIC For 2013. This virtual sculpture has true omnipresence.2013. Using the eye-movements of the person wearing AR glasses gives extra accuracy. Real immersive AR depends on knowing exactly what our eyes are looking at.5 out of 5. we see a concern regarding the possibility to record without the public knowing this. although it never manages to become the Actual Current Big Thing”. inserting a lot of information from the physical world. predicted five issues related to AR Law for 2012. What is at stake is. EYE-TRACKING Another type of warning from Mr Wassom is related to the information derived from tracking our eye-movements to detect were we are looking at. Since most of the public are not familiar with AR marketing activities like this. it floats above holy places in Rome or in the Himalayas.000 billboards with Augmented Reality markers throughout many cities in the Netherlands. and the girls will come to life allowing you to watch them in a video right there on the street. maybe voluntarily via players of AR games. “Stop the Cyborgs!” Mr Wassom recognized he was not right in all his predictions. The “No Glass” logo that the café published on its website http://the5pointcafe. into the virtual world. on the other hand. WhAT IS RECORDED ? With the announced introduction of Google glass. that there’s no obvious way to know when the device is on or what it’s actually doing (recording or not). However. the app distributed via Layar works by pointing your smartphone at one of these posters. some predictions were made and trends were spotted. to AR patents. it will have a significant informative impact. is to download this app via Layar. blogger and specialist in AR Law. and. above warzones and above peaceful places… Brian Wassom. the highly anticipated augmented reality device set to be released later this year. On the Stop the Cyborgs site. and make a virtual grid on which we position our virtual objects or scenes. analysed and even sold to marketing companies. the group raises a significant concern: namely. Seattle’s 5 Point Cafe became the  first known establishment to publicly ban Google Glass. we can enter a room. Concerns raised range from on the one hand. this information is used to position the virtual content there where our eyes are looking at. but for him it was more a matter of time than a matter of substantial incorrect prediction. scoring 1. without any prior screening or approval from an examination committee. When downloaded. in February 2013 an adult magazine has distributed around 1. again.com was developed and released (under a Creative Commons license) by a new London-based group called “Stop the Cyborgs. as well as above industrial places. .

snapsapp. For the time being. As an example. com “and suddenly everyone is wearing it and this becomes as prevalent as smartphones—you can see it becomes very intrusive very quickly. and still for some. To add correct or incorrect virtual information to a virtual object or a space. is possible with our smartphones and is widely spread among youtube videos. is as simple as assigning correct or incorrect information to any subject. we have privacy laws to handle this. 43 43 . we have to change those laws. FURThER READInG  Critical blog on law and social media: http://wassom. our current laws might be sufficient. and will be considered as such. our actual privacy laws are sufficient enough to deal with privacy concerns in augmented reality. see www. “But if the Google glass will become prevalent” as one the people from ‘Stop the Cyborg’ argues in a conversation with a journalist of arstechnica.com and www. we don’t need special AR criminal legislation. Facebook or in a space around us.com  Critical site on wearable technology: http://stopthecyborgs. it’s about the social culture around it”.Images coUrtesy of GoldRUn.org FALSE STAYS FALSE Recording without the recorded person knowing this. which has been working in the augmented reality field since 2006.and if those laws are not adequate enough. An incorrect accusation or incorrect information remains incorrect. even closely related people within our Academy. I can take our AR Lab. The intertwinement of AR in our day-to-day lives is actually quite slow — especially compared to the revolutionary predictions. However. regardless whether it is posted on Twitter.com This quote gives us a direction for discussing the abovementioned AR related legal issues. object or space in our physical world. the concept of AR is completely new. via an AR app.goldrUnner2013. It’s not about the tech. We are getting used to it at a nice pace. We might come to the preliminary insight that as for now.

BOOK COVER. ILLUSTRATOR: PAUL HADCOCK 44 . ThE AUGmEnTED STAR.OLIVER PERCIVALL.

POINTING PEOPLE FOR EXAMPLE TO PLACES OF INTEREST OR NEAREST TUBE STATIONS. 45 . PART ONE FOLLOWS OUR ACCIDENTAL EXPLORER LYNDEN AS HE POSTS HIS ADVENTURES TO AN ONLINE FORUM. WHAT IF THERE WAS SOME AUGMENTED REALITY TEXT MOST OF US WERE NOT SUPPOSED TO SEE AND YOU HAD A DEVICE THAT WAS ABLE TO READ IT… WOULD YOU FOLLOW ITS INSTRUCTIONS? THIS IDEA FASCINATED ME INTO WRITING A SERIES OF STORIES TITLED THE MISADVENTURES IN AUGMENTED REALITY. I DISCOVERED ITS MAIN USES WERE AS A NAVIGATION TOOL USED IN CITY CENTERS. THE THOUGHT OF CONCEALED TEXTUAL MARKERS STRUCK A STORY IDEA IN MY MIND.The Misadventures in Augmented Reality OLIVER PERCIVALL I FIRST LEARNED ABOUT AUGMENTED REALITY AROUND TWO YEARS AGO.

the shy inventor type. And because of this I have a story to tell.com and www. It’s not easy to write long paragraphs on this VAARR. My uncle made it possible to post directly online from this device.com Life in this newly discovered domain is a far cry from home. everything is entirely unfamiliar when you first arrive. But once you start exploring all the hidden signs and directions it’s impossible to stop. But be warned: Augmented Reality text as we know it was not developed within The Augmented Star so it makes it pretty hard to find your way out. A few years ago I wasn’t exactly a technology fanatic although admittedly I had more than a passing interest in new gadgets. So my blog entries below are concise. the temperature. He created two actually. But nearly nine years ago now he disappeared from his hometown and has never been seen since.Part One My uncle was one clever soul. see www. shows unseen Augmented Reality text none of which is detected by any other device. I don’t think he’s lost.snapsapp. You’re immediately plunged into its rich diverse landscapes and to find your way around or locate anything at all you must be in possession of the special device — The VAARR or Very Advanced Augmented Reality Reader. the noise levels. Images coUrtesy of GoldRUn. Now there is probably nothing I don’t know about the integrated software algorithms used in Augmented Reality. One was entrusted to me. the beings. 46 . There’s no one thing that makes it uniquely different. He could probably be best described as a recluse. But that was nearly three years ago now before I set foot within the Augmented Star.goldrUnner2013. Using his ingenious engineering skills he somehow created a VAARR (Very Advanced Augmented Reality Reader) It works just the same as other regular Augmented Reality apps I am led to believe but with one unique difference… It has a setting that once activated. not wanting to be found is my guess and I think I know why. It’s everything. The smell.

The first post I made earlier was date stamped Apr 12 2011 but the post I made just a couple of hours after that is stamped July 12 2011. To find my uncle. I’ve come here with a strict agenda. Blog Entry #1 Lynden Apr 12 2011 ‘It’s not about gazing into the future. Blog Entry #3 Lynden Oct 12 2011 Just noticed a glitch with this online forum that I would like to point out to the moderators. From learning a new food recipe to breathtaking advertising campaigns.The Ultimate Frequency Forum This forum was created for anyone to post to that feels lost. those that do not wish to be found or find themselves beyond the confines of earthly boundaries. the Augmented Reality here really is something else. There’s just so much to get distracted by. Three months later? I think not! 47 . It puts things in perspective when vehicles move from 0 to 600 MPH in less time than it takes to chalk a pool cue. Every time. Blog Entry #2 Lynden Jul 12 2011 OK it took me a couple of hours to get acquainted here this time. And you never arrive at the same place twice. it’s about seeing the present through a unique lens. On the downside crossing the street can be something of a perilous activity.’ That’s the first message you see when you arrive in The Augmented Star.

After leaving school I took a basic Computer studies diploma.Blog Entry #4 Lynden Jan 12 2012 It’s funny because since I’ve been wandering the streets here the last few hours I’ve regretted never going to Cambridge University. I feel I’m ready for that science degree now! O. So equipped with a reasonable understanding of Augmented Reality and enjoying a challenge. Percivall I have always had an interest in technology and gadgets. I set myself the complex task of plotting my science fiction novel and created an alternative fictional world that became The Augmented Star. Although I feel like I have been. It appears there is nothing you can’t learn here by waving the VAARR around. The Augmented Star is now available on The http://tiny. Must be something in the air here as my clothes are becoming dirty and ragged really quickly. The mysterious side of technology especially intrigued me including the possibilities of where it could take us in ‘What If’ scenarios. which eventually led to a career in various I.cc/augmented-star Amazon Kindle Store. The whole place is an encyclopedia of knowledge. L. They promote learning a Science degree in Artificial Intelligence can be completed in three months here. Knowledge is fluidly brought to astonishing life everywhere by Augmented Reality. I just learned that a thimbleful of a neutron star would weigh over 100 million tons.T support roles leading up to Project Management. jsut follow the QR code on the left! 48 .

I wonder what the odds are for walking into an Augmented Universe like this one. Wow that’s a curious fact. I’ve come to realize now that I may never be able to leave the Augmented Star. The AR campaigns here really resonate with customers in a way that most other ad platforms fall miserably short. End of part one Images COURTESY OF GOLDRUN. After an exhausting and bloody battle earlier today. It feels like I’m on the run myself. I saw an advert on a billboard for an automobile just now.GOLDRUNNER2013. It can read over 150 types of Augmented Reality text and other types too that alternative devices can’t. I have learned some creatures here would even kill to own one. Blog Entry #6 Lynden Jul 12 2012 Blog Entry #7 Lynden Oct 12 2012 Just received some news that was pretty hard to comprehend. Someone is coming. I’ve met a lot of people here within The Augmented Star and most of them know my uncle but apparently he’s on the run from some bad people.COM Turns out my VAARR is a pretty valuable commodity here. 49 . Using my VAARR triggered a full Augmented Reality breakdown of its features from performance figures to finance options and then it invited you to take a virtual test drive in a car simulation game.COM AND WWW. I’ve gotta move quickly.SNAPSAPP.Blog Entry #5 Lynden Apr 12 2012 The risk of being struck by a falling meteorite for a human is one occurrence every 9300 years. SEE WWW.

Augmented models are those which blend a physical and a digital counterpart in a resulting synchronic manner. it also Digital physicality. modifiability. a complex orchestration of users. As a result. SIGGRAPH or the ISMAR community. augmented models. This work claims that beyond physical or digital modelling techniques. tools and techniques merge into tailored modelling flow works according to each project’s requirements. scanning and reverse engineering methods allow a fluent interaction between models which embeds information and knowledge as the design process proceeds. This recent conference had. manipulability) as well as digital (accuracy. This design-technology synchronic co-evolution has been historically aligned with the zeitgeist of the architectural practice. Likewise. such as ubiquitous computing systems or environmentally responsive technologies. hence taking advantage of the benefits of physical modelling methods (dexterity skills. this section is titled with the topic of the latest conference of the Association of Education and Research on Computer Aided Architectural Design in Europe (eCAADe) held at the Czech Technical University in Prague (September 2012): 50 . . Indeed. One of the main grips in which this topic is grounded is that the growing academic concern on the dialogs between physical and digital realms in architecture is becoming blurry. Physical digitality. Even if the semantics of augmentation suggests an ‘increment on size or amount’. and further evidence of this trend is the growing participation of architects and designers in professional events such as Ars Electronica. augmented models. The construction of a physical model by computercontrolled machinery requires. since the distinction between both cannot be fully depicted in a world where digital information is continuously embedded into ‘real world’ situations. augmented reality now gives us the chance to build hybrid. among others).augmented reality now gives us the chance to build hybrid.AUGmEnTED mODELLInG In ThE DESIGn STUDIO: AUGmEnTED pEDAGOGIES Alejandro Veliz Reyes University of Salford Digital physicality / Physical digitality From very early stages of the design process (sketches and diagrams) up to detailing and construction. the concept of augmentation has a major role in this work. so far. unavoidably. in which new technological developments shape the way we express ourselves. in addition to the technical definition of augmented reality.. developers.. the biggest amount of published articles in the history of the eCAADe Association Conference. Also. models are built by using several different — yet interrelated techniques. and define new design forms and organizational/social schemes. a digital counterpart.

In the same design crit session. synthesis) are mainly developed in architectural education programs around the world. This scenario might not be unknown for any architecture student. says an architectural design instructor while chopping a piece of cardboard off a building model. analysis. representing) and high-order cognitive skills (critical thinking.TRAINING SESSION ON AUGMENTED REALITY AUTHORING AND AUGMENTED MODELLING. UNIVERSITY OF SALFORD. suggests some enhancement. instructors and observers engage in a dialog based on the use of plans and sections. this approach to the augmentation of reality fits with the major aim of educational research which is to enhance and improve educational processes and methods. As stated by Allen Cunnigham in 2005. after a centennial adaptation and evolution the studio teaching scheme and “project-based education around architecture employing the studio system is the most advanced method of teaching complex problem solving that exists”. diagrams and sketches. Actually. thus naming this work as ‘Augmented Pedagogies’. SCHOOL OF THE BUILT ENVIRONMENT. since the studio teaching scheme has been largely acknowledged as the core practice-based module in which both design (composition. the architecture studio and its interactions as a subject matter is a quite complex challenge. physical mock-ups. MODELS IN THE DESIGN STUDIO “It looks better like this”. As a result. planning. rendered views of a project. 51 . students.

The usefulness of models within the design studio is clear. Beyond the fact that its construction itself entails the development of technical skills, models also embed design information and knowledge, affecting organizational dynamics (the design critique or peer to peer collaborative work), the creation of students’ ‘toolkits’ or the final presentation of the design solutions, among other benefits. During this research being conducted at the University of Salford (UK), the extent of the impact of augmented models in this complex studio-system is yet to be depicted. Augmented models will be used, therefore, as a way to understand how new technologies impact design education and how can we describe that impact

from a scientific research perspective, that is following the guiding principles of generalizability, communicability and transferability of that resulting knowledge. The deep impact of new digital tools in design pedagogy has been explored recently by design theorists, such as Dr Rivka Oxman. The particularity of the design studio as a research setting is spiced by theoretical underpins that can potentially lead the path to depict this impact. For example, it has been stated that the studio teaching is usually an unstructured process, in which perceptions and interpretations of information and models play a major role in the students’

DRAFT VERSION OF AN AUGMENTED MODEL FOR INTERIOR DESIGN. FURNITURE CAN BE RE-ARRANGED IN A PHYSICAL MODEL OF A HOUSE ROOM (STUDENTS’ WORK).

52

progression in the courses, mostly based on ‘design dialogs’ between students, and students and instructors. Also, digital tools have the potential to not only re-shape the toolkits being used for design, but also mediate in the way design methods are structured, offers new ‘digital materials’ to work with or changes the very nature of the design problems to be faced in different courses. It is not clear, however, how this impacts occurs. The interactions within the studio that make use of representations and models to design are well established ‘rituals’ such as peer-to-peer collaborative activities or the design critique, but the nature of each studio differs from each other. Variables such as the experience of the instructors, the background of the students, the nature of the design problems to be faced or the institutional standpoint turn the studio into highly context-dependant modules. As a result those variables are usually highly controlled and the study of the impact of different technologies is commonly constrained to the description of technical challenges to be solved, the development of new systems/software or metrics of student satisfaction, rather than on the provision of a theoretical account of their impact into this complex teaching/learning process. The lack of a theory that describes how technology re-shapes the studio results in very limited knowledge re-usability and in turn, into very caged and localized pedagogical frameworks that do not allow cross-institutional or cross-disciplinary collaboration, to evaluate the constant infusion and evolution of new digital tools for educational purposes or to re-use a pedagogical approach and its associated knowledge.
IMAGES OF THE TRAINING SESSIONS ON AUGMENTED REALITY AUTHORING AND AUGMENTED MODELLING. STUDENTS OF THE PROGRAMS MSC DIGITAL ARCHITECTURAL DESIGN AND MSC BUILDING INFORMATION MODELLING AND INTEGRATED DESIGN, SCHOOL OF THE BUILT ENVIRONMENT, UNIVERSITY OF SALFORD.

A THEORETICAL APPROACH TOWARDS AUGMENTED PEDAGOGIES
Indeed, there is not fixed methodology to study the impact of digital technologies in design education. Rather, each study requires an own

53

standpoint in terms of validity and fitness to the research problem and the subject matter. As Wanda Orlikowski and Suzanne Iacono (2001) state on their work on information systems theory research, this corresponds to the fact that the use of technologies depend on the context and hence, “there is no single, one-size-fits-all conceptualization of technology that will work for all studies. As a result, IS researchers need to develop the theoretical apparatus that is appropriate for their particular types of investigations, given their questions, focus, methodology, and units of analysis.” In order to overcome this challenge, this ongoing research proposes a theoretical approach to depict the impact of augmented models in design education. By following a grounded theory methodology, observations and recordings are being collected in diverse settings on an attempt to de-

scribe the resulting studio dynamics by using augmented models. Several trainings on augmented reality and augmented modelling have been made at the University of Salford (MSc Digital Architectural Design, MSc in Building Information Modelling and Integrated Design), and two more experimental settings are now being arranged in different European countries. These multiple settings are not only intended to provide a wide view of the subject being studied, but also fits with the current recommendations for theory construction methodologies, since the manipulation and observation of data in many divergent ways and the juxtaposition of different conflicting realities and sources counteracts the tendency of reaching false or incomplete results, or informationprocessing biases of the investigator. This work is expected to be finished by end-2014.

VIEW OF THE STUDENTS’ TOOLKIT DURING THE FIRST AR WORKSHOP AT THE UNIVERSITY OF SALFORD.

54

Related literatUre and soUrces
¢¢ Achten, H., Pavlicek, J., Hulin, J., Matejdan, D. (Eds.). (2012). Digital Physicality / Physical Digitality - Proceedings of the 30th eCAADe Conference - Volume 1 / 2. Czech Technical University in Prague, Faculty of Architecture (Czech Republic) 12-14 September 2012. ¢¢ Buechner, J. (2011). Fictional Entities and Augmented Reality: A Metaphysical Impossibility Result. Journal of Evolution & Technology, 22(Minds and Machines Special Issue), 53-72. ¢¢ Cunningham, A. (2005). Notes on education and research around architecture. The Journal of Architecture, 10(4), 415-441. ¢¢ Eisenhardt, K. M. (1989). Building theories from case study research. Academy of management review, 14(4), 532-550. ¢¢ Orlikowski, W. J., & Iacono, C. S. (2001). Research commentary: Desperately seeking the "IT" in IT research—a call to theorizing the it artifact. Information systems research, 12(2), 121-134. ¢¢ Oxman, R. (2008). Digital architecture as a challenge for design pedagogy: theory, knowledge, models and medium. Design Studies, 29(2), 99-120. I finished my Architecture degree in 2007, in Chile. Today I am a teaching assistant and PhD student on digital architectural design at the University of Salford (United Kingdom) and my current research interests are design pedagogy, augmented reality, and collaborative technologies for design. I have been involved in research activities since 2007, but my background is quite mixed and has involved the domains of sustainable design, buildings in extreme/cold environments, mass customization and digital design. My published work has led me to present on conferences and invited lectures in Chile, United Kingdom, Italy, Belgium, Slovenia, Czech Republic, Canada, among others.

Alejandro Veliz Reyes

55

56 SCARLET PROJECT © UnIVERSITY OF MAnChESTER .

educators are understandably cautious electing to concentrate. particularly where the emphasis is on the teaching material. After all.A STUDY IN SCARLET MATT RAMIREZ Laura Skilton Learning and Teaching Co-ordinator Mimas. Visual Arts Data Service University for the Creative Arts Introduction Augmented Reality (AR) was identified in the 2011 Horizon Report1 as a key technology trend with potential impact on education. perhaps rightly. Crafts Study Centre University for the Creative Arts Marie-Therese Gramstadt Educational Technologist. The report provided the catalyst for the SCARLET (Special Collections using Augmented Reality to Enhance Learning and Teaching) project.UNIVERSITY OF MANCHESTER Jean Vacher Curator. Whenever the words “technical innovation” is spoken in education circles. The benefits to student learning should always be central to the introduction of any new technology and AR is no different. Some of the most persuasive arguments are described below: established learning methods and styles. this can inevitably lead to dissatisfaction and resistance. in education technology should be transparent and not an overpowering driver. supporting independent learning. The University of Manchester Rose Lock Special Collections Supervisor University of Sussex MIMAS . as a way of leveraging innovative technology with pedagogical processes. new technology should integrate seamlessly into Students can view and touch real manuscripts/ editions in conjunction with guided support from trusted sources. to be armed with a long list of tangible gains for adopting the technology. on deep rooted pedagogical benefits rather than short-term gimmickry. If the focus for the student is the technology. Crafts Study Centre/ Research Associate. the learning experience can be diluted. Many observers have already buried AR as a fleeting fad in education due to its lack of use cases and documented impact studies. The benefits of using Augmented Reality in Education 57 . It is always useful when dealing with new methods of delivery. In addition. users do not want to spend time adapting to a new way of learning.

Students are used to having access to electronic information on demand.  Abstract concepts or ideas that might otherwise be difficult for students to comprehend can be presented through an enhanced learning environment offering access to source historical artefacts and online research in situ. videos and information to images in the form of “GLUE” based channels. maximizing  the opportunity for interaction. images. Junaio was the only AR browser to harness optical tracking functionality. linking 3D models. so this experience can be foreign and a barrier to their use of special collections. and a slightly larger percentage of what is shown to them.g. rare books within the controlled conditions of reading rooms. 58 .  AR can harness both asynchronous (emailing tutor questions) and synchronous (discussion with peers) e-learning methods. not the technology. By implementing an object based AR experience. students could simultaneously experience SCARLET (June 2011. whilst enhancing the learning experience by surrounding the book with digitised content.  The learning curve for new users engaging  with mobile AR through browsers is relatively shallow enabling the learning/pedagogy to be the driver. such as an early printed book in the library. isolated from much of the secondary supporting materials and a growing mass of related digital assets. augmented 3D models that overlay the physical image and require user touch gestures to proceed) and spark enthusiasm. but when we become actively involved in an experience. Layering AR on texts/images can encourage  interaction (e. Sources for primary content were ten key editions of ‘The Divine Comedy’ by Dante (between 1472 and 1555) particularly important in terms of publishing and intellectual history and the world-renowned oldest fragment of the Gospel of John. encouraging critical response and the adoption of new perspectives and positions. learners will remember and retain the majority of the information presented to them. academics. learning technologists and students. translations. A mixed team was pulled together dedicated to enhancing the student experience through the application of technology including librarians. preparing them for solo research. The SCARLET project. This is in opposition to traditional didactic methods that are predominantly teacher led. manuscripts and the magic of original primary materials. At the start of the project in 2011. This coupled with an open API and compatibility on Android. while embracing the potential of AR.May 2012) The main challenge with the use of special collections in learning and teaching is that students are required to consult archives. iOS and Nokia devices would prove decisive in the decision to use Junaio. and information on related objects. a national centre of expertise at The University of Manchester. concentrated on delivering the benefits to student learning without being a flag bearer for the technology.  AR promotes ‘active’ teaching. SCARLET was led by the Learning and Teaching team at Mimas2. 3D models. Users retain a very small amount of the information that is delivered.

Emphasising the need to align technology to teaching and learning objectives was paramount from the outset to maximise student benefit and impact. Throughout the project lifespan. UNIVERSITY OF MANCHESTER.g. planning for essay or presentation). providing a foundation for further investigation and research. AR was most successful layered over the printed marker instead of signposting to other web based resources already accessible using traditional teaching scaffolds (e. The team went on to win the 2nd prize for the ALT learning technologist of the year team award3 and won the Innovation in HE award at Education Innovation 20134. UNIVERSITY OF MANCHESTER. DANTE AR CONTENT © MIMAS. student evaluation was critical. In addition. Students who had little prior subject knowledge found the app most useful.g.Evaluation A dominant theme that became evident in the evaluation was that the two academics found differing responses dependent on student user groups. Further funding was made available through the SCARLET+ project whose primary focus was to apply the process and framework to other institutions special collections embedding the methodology using a toolkit5 (University of Sussex and the Craft Study Centre at the University for the Creative Arts). further funding was made available to work with two other UK institutions to facilitate and guide them in the development of AR content. SCARLET+ (June 2012 – April 2013) Following the success of the SCARLET project. 59 . student feedback noted that AR experiences should be best used as part of a learning activity (either independent or group based) acting as enabler to achieving a key course objective (e. directly feeding into the application using an iterative development approach. making them unique and packaged in digestible chunks produces more positive feedback and value. The learning experience was most enhanced by AR when information delivered was FRAGMENT OF THE GOSPEL OF JOHN AR CONTENT © MIMAS. CMS). It would also liaise with staff at these universities to transfer the skills learnt and hone the toolkit that was used as a high level framework for replicating similar AR practices in education. Simply adding existing web assets to an object is insufficient. contextual and less generic.

This ensured that the content developed made an impact on learning and teaching as well as enabling AR skills to spread across the institutions. multi-layered information accurately locating building types. Mimas has the Crafts Study Centre at the University for the Creative Arts7. Sharing was a key element at the heart 60 . Conclusion To conclude. UNIVERSITY OF MANCHESTER. This project has both developed Mimas’ understanding of implementing AR in education and embedding best practice and methodology to other institutions. cannula application). building heights and ground usage to name a few. This provides a wealth of rich. Both succeeded in providing a showcase for the Special Collections held at their respective institutions using AR. the two projects have presented important findings in the impact of AR in education. as with SCARLET. a crucial part in decision making processes. Crucial to the success was ensuring that. In conjunction with this. a multi-disciplinary team approach was adopted. bringing their static objects to life. There is vast potential in this field enabling students to examine x-rays/biopsies.g. collaborated with colleagues from the Landmap9 service to create an AR experience around the UKMap dataset. early development has started using 3D object recognition in the clinical skills laboratories.AR VISUALISATION OF UKMAP DATA © MIMAS. It demonstrates a visual representation of raw materials that are often extremely large in size and difficult to comprehend. delivering a suite of rich materials especially given the small amount of funding that was available. focusing on content from 1980s’ mass observation and 20th century crafts. using AR to identify abnormalities. Working with other subject areas In parallel with SCARLET and SCARLET+. Mimas have worked with the Manchester Medical School8 helping students to understand pattern recognition. In the area of geo-spatial mapping. The challenge was to incorporate this tabular data in a visual 3D model that a handheld device could render efficiently. layering anatomical information and reinforcing learning with instructional demonstrations (e. The partner universities involved were the Special Collections at the University of Sussex and 6 which will allow students to access surrounding resources.

Further information: http://teamscarlet.ac. While it may not be suitable for all students and situations. Blender.csc. It is hoped that the legacy from these small projects will be to inspire others to undertake similar work and display the student led benefits AR can offer.com/2012/09/13/scarletteam-are-joint-runner-up-in-learning-technology-awards 4.ac.ucreative. Matt's role is also concerned with the research and development of new technologies (e.wordpress. 3. 5.ac.uk/library/ specialcollections 7. References 1.uk Landmap: http://landmap. allowing active based learners to better absorb the transfer of knowledge.uk SCARLET team are joint second in learning and teaching awards.g. when employed well it can capture the attention like few other technological mediums. wordpress. section on augmented reality: http://wp.ac. science. It is particularly effective in explaining abstract concepts visually.ac.uk 61 . This aims to improve the student experience by embracing innovative learning methods and providing cutting edge support materials.of SCARLET and SCARLET+. ac.mms. this follows on from his involvement in the award winning SCARLET project.nmc. wordpress.uk/mediawiki/index. 9.co. Education Innovation 2013: http://educationinnovation. Mimas: http://mimas. AR opens up huge possibilities for creating immersive learning activities. 8. 13th September 2012: http://teamscarlet. 6.com Matt Ramirez Matt Ramirez is currently working on the technical development and support of the JISC funded SCARLET+ Augmented Reality project.com and @team_scarlet.sussex. He has over 15 years experience in web design and e-learning content development for a variety of subject areas including medicine.uk/cms SCARLET Toolkit: http://scarlet. Flash. iBook/mobile/multimedia development projects and haptics) with the Manchester Medical and Dental Schools. this coupled with a strong team ethic enabling stakeholders to buy into the long-term vision. special collections and business. These have used multimedia content authoring tools such as Flex. Crafts Study Centre at the University for the Creative Arts: http://www.mimas.uk Manchester Medical School: http://www. Read more about Matt’s work at http://teamscarlet.php/Main_Page Special Collections at the University of Sussex: http://www.manchester.org/ horizon2011/sections/augmentedreality 2. 2011 HORIZON report. IT. Unity to name a few.

as AR Lab are thoroughly convinced that it is important to let first-year students. in cooperation with the Studium Generale programme. aspects are added to the world of graphics regarding the Internet and social media. because it will change aspects of their discipline as well as their ‘metier’. We. At the beginning of the Royal Academy of Arts’ current study year. Most important. started to work with their classes of 27 students each (!) together with some third year students as well. a programme covering lectures throughout the year. At various other art academies we see similar projects. Whether a concept is devised with colour pencils.INTRODUCING AR TO FIRST YEAR GRAPHIC DESIGN STUDENTS YOLANDE KOLSTEE Graphic Design students learn how to work with the latest Adobe suit. The search to get to know what AR is and how it could be a 62 . in effect. felt pens or a collage of other material is not the most important issue at that moment. Photoshop and website development tools. while. freshmen and -women. they often have a (underlying) preference for pen and ink based handwork. both young and digitally savvy. With Augmented Reality new. meaning a short. while also taking co-authorship and performance into account. one-day exhibition of the results. Two Graphic Design teachers. However. however. are the courses in which concept development forms the main part. at the University for the Creative Arts. The idea was to work towards a special PopUp Gallery. the majority of mainstream teachers place analogue and digital techniques at the same level when developing concepts. experience augmented reality. one of which is the Scarlet (Special Collections using Augmented Reality to Enhance Learning and Teaching) project. very new. in the UK. and the department involved. When working with Augmented Reality — a very suitable medium for graphic designers. we made a head start with the introduction of AR via projects differing in length and with different student groups. by the way — new elements should be considered in the concept and design phase. What is important for the Academy. is that they want the students to become acquainted with creative or artistic research.

Words emerged by moving strips of black paper over the seemingly incoherent rows of letters. the idea was to provide an enriching experience for the students and to give them the tools with which they could incorporate AR in their work. a publication of all the projects was made with a spread for each student in which the description and intention of their project were incorporated. Ultimately. is that they appear to invite spectators to look at the world from another perspective. no addition to reality and certainly no interaction would be possible. Without digital technology. We asked them to think about the kind of information they wanted to disclose in specific situations. 63 . Jorick won with his project 'Get your shit together' where animated QR-codes formed text. Finally. These are some tools one associates with Augmented Reality (AR). What all the students’ projects have in common.” In completing the first step of the two step project. We were lucky to have two lecturers who gave their talk on AR as part of the Studium Generale programme preceding the Pop-Up Gallery: AR artist Sander Veenhof and data-visualisation expert Niels Schrader. In the words of the students’ teachers : “Spectacles. the students gave us a huge input of examples they encountered: they found all sorts of material via social media or as search results that they consequently discussed. without the aid digital techniques and devices? Following that. Aurasma — a simple and elegant tool for beginners in the AR app world. at first. The first was a very active YouTube-Google search. phones and tablets. What type of problem would they want to solve using AR? With what classification should it work – analogue and/or digital? With this exercise we hoped to widen the concept. so to say.g. The end-results would be displayed during our first Pop-Up Gallery on AR. This is obvious because the assumption in AR is that reality must be supplemented with digital information. The concept developing part. headsets. The rationale underlying the research project therefore lies especially in the use of digital techniques to make this extra layer onto reality visible. Not one. Jente won with her project 'Had je dat gedicht?' (roughly translated as 'How does that rhyme with you?'). while also seeking to come up with new applications and reasons for the use of Augmented Reality. It is exactly this assumption that we want to investigate. Is Augmented Reality only achievable by the use of digital means? Can hidden information be augmented. We provided technical help when students were really eager to get their ideas to work as a fully functioning app. These types of projects show the broad range of ways in which an extra layer on top of the physical world poses questions about what we see. A special recommendation was given for the dyslexia project of Janne van Hooff who visualised how people suffering from dyslexia see letters and numbers. Or it would be just reality. They were part of the jury that awarded the two ‘best’ projects at the end of the first Pop-Up Gallery. for the students themselves as well as for us. We showed them our AR projects and introduced them to e. step two. Error messages we are confronted with on a daily basis are transformed into poems in order to take some of the zing out of stressful situations. were not completely convinced about using AR and how they could incorporate this technique in their projects. The same goes for those students who. regardless whether they are analogue or digital. The variety of final projects was staggering and impressive. but two winners were awarded: Jente Hoogeveen en Jorick de Quaatsteniet.meaningful technique for graphic designers began comprised two steps. and the second step was to learn to play with the idea of adding extra information — both analogue and/or digital — a concept development trajectory of sorts. turned out to be very exciting. we invited freshmen and -women from the Graphic Design department to get started with Augmented Reality.

64 .

or switch on the layer showing the nervous system for example. This way the user is asked to involve their environment in the process of learning words. with which developers are trying to create new. just as in many other fields. interesting.CREATIVASTOCK. Another interesting project in development is the Sesame Street app Big Bird’s Words (Qualcomm.COM ample is the 4D Anatomy project by daqri (2012). where you can explore the physiology of a human being by moving the display device along a piece of paper with markers. On the screen you see a 3D model of the human body. Many of these experiments draw on of the possibilities to explore virtual information spatially. These examples show some of the new forms of interaction and presenting information. WWW. 65 . researchers and developers are experimenting with potential applications of Augmented Reality technologies. 2013). it gives points to the user. With sliders and buttons you can set the transparency of the skin.AUGMENTED EDUCATION HOW AR TECHNOLOGIES EXTEND OUR MINDS ROBIN DE LANGE In the field of education. which uses the latest text recognition algorithms. The (young) users of this upcoming app are asked to look for certain words in their home and aim their device at it. When the device recognizes the word. An interesting exImage by MARCELO GERPE. and memorable learning experiences.

the latter having some speed advantage. which Clark and Chalmers employ to introduce the Parity Principle: “if a part of the world functions as a process EXTEnDED MInD AnD ThE PARITY PRInCIpLE Andy Clark and David Chalmers start their renowned paper ‘The Extended Mind’ with a thought experiment. Kirsh and Maglio found that the physi- which. Furthermore. (Kirsh and Maglio. Clark and Chalmers show that true mental events can extend in the environment as well. but now the person can choose either to mentally rotate the shape or physically rotate it by pressing a button. the notebook can be seen as an extension of Otto’s memory. namely playing the video game Tetris. In this experiment the reader is asked to consider three cases and to think about how much cognition is present in each case: (a) A person sits in front of a computer screen which displays geometrical shapes and is asked to fit the shapes in the displayed ‘sockets’. the human mind is not bound by the borders of skin and skull. also trigger debates about the very goals of education itself. we would have no hesitation in recognizing as part of the cognitive process. or using ‘old-fashioned’ mental rotation. Clark and Chalmers present a thought experiment involving the fictional characters Otto and Inga. In the same manner. Clark and Chalmers argue. then that part of the world is (so we claim) part of the cognitive process. cal rotation in (b) is actually much faster than the mental rotation. this information may even be seen as part of our cognitive process. and in my opinion should. 1994) It is this example of the human capacity to manipulate the environment to solve problems.In this article I will argue that the developments in AR technologies will make digital information sources much more transparently available to us. 1998) According to the Parity Principle. AR technologies could not only lead to new learning methods. Otto can be said to have a belief about the location of the museum before he actually consults his notebook. but were also trying to determine whether the shape fits in the slot. thereby simplifying the task. (c) The same situation in a possible future. while Inga’s biological memory is functioning properly. before she recalls this from her internal memory. I will first introduce the concept of the Extended Mind. Because of this change of perspective regarding external information sources. players were not only physically rotating the shapes to fit the slot. (b) The same situation as in (a). All the examples of cognitive extension that Clark1 gives in his books and papers are not the typical futuristic technologies that come to mind when thinking about humans merging with tech- 66 . but could. In their paper about cognitive performances while playing Tetris. under the right circumstances. were it done in the head. Now. Otto has Alzheimer’s disease and uses a notebook to serve the function of his memory. To backup this claim. where the person can choose between using his neural implant that does the rotation operation as fast as the computer does in (b). by rotating the shapes in his mind. these kinds of cases are actually not as abstract as they might seem: they describe a very well-known real-life situation. Inga is thought to have a belief about the location of the museum. To make this claim plausible.” (Clark and Chalmers. In certain cases. who are remembering how to get to the museum. Thereby. By showing how beliefs are not bound by the borders of the body.

The technological aid should always be available when needed. Could digital information sources. One example he mentions is the use of pen and paper when doing long multiplications. then wait for the downloading of the page 67 . should be as low as with an equivalent function of the brain (Smart et al. and we are wearing our watch. By writing down figures in certain locations. In fact. actually part of this process. we use the pen to manipulate the external memory source. according to Clark this is not simply loosely formulated informal language. The writing utensils play a crucial role in this cognitive process and are therefore. it is quite clear that the Web with which we interact on a daily basis cannot be seen as part of our cognition. actually become parts of our minds? Could digital information sources actually become parts of our minds? To begin to answer this question and decide whether a part of the world should (temporarily) be seen as part of the cognitive process. Although the possibilities of Brain-Machine Interfaces and neural implants such as in case (c) offer very exciting new ways of communicating with technology. 2003). we can refer to the three criteria suggested by Clark: 1. the paper. From our critical stance we do not immediately believe most information we encounter on the Web something which is. 2003). 2008). During this time these technologies have become ubiquitous. a measure of the combination of time and physical and cognitive effort. ‘you’ is only “the hybrid biotechnological system that now includes the wristwatch as a proper part” (Clark. for example parts of the Web. this direct interaction with brains is by no means necessary to become part of the cognitive process (nor are they sufficient for cognitive extension: communicating with technology through a Brain Machine Interface usually still takes too much cognitive effort. From this. You actually do know what time it is.) Now. They have become socially accepted and actually shaped culture itself. (This proven transparency of the wrist watch is what makes the development of smart watches interesting. wrist watches have been around for many decades. according to the Parity Principle. the technologies which Clark considers as cognitive extensions of our cyborg minds are hardly identified as technology anymore. The external resource should be directly available without difficulty. we use an algorithm that divides the process of multiplying arbitrary large numbers into very simple steps. Now. 2. we often answer “yes”. Furthermore. active role in cognitive processes. The user has to put physical and cognitive effort into navigating the browser to the right page. due to the open character of the Web. writing utensils even for centuries. An interesting question is whether more modern external information sources could obtain the same status as these age-old technologies and play a similar. that Clark considers us to be “natural born cyborgs” (Clark. look at our watch and see what time it is. Another example shows that it has become common to talk about the information that is in some of our technologies as if part of our own knowledge.) Our brains incorporate the world and some of the technologies therein in their cognitive processes in such an intimate way. Subsequently we raise our arm. More precisely: the information access costs.nology. the information access costs when retrieving information from the Web is way too high. The information retrieved from the external source should be directly endorsed and trusted. When somebody asks us on the street whether we know what time it is. To calculate the product of two numbers. 3. probably a wise attitude.

tion knows when you’re busy driving for example and doesn’t bother you then. user interfaces are becoming more intuitive. telecommunication service providers keep improving their networks to decrease download times. etc. would you say that you know the meaning of this word? I can imagine that you — after you get more and more used to the device and have experienced this situation a few times already — might say ‘yes’. quicker and more intuitive ways. But what is it with AR technologies. Despite of what many critics expected. I think you might start to feel like you really do know it.and scan through the text to find the information he needs. technology is ever-evolving and many of the present advancements will make information available to us in more reliable.2 More so. the applica- The main goal of education should be to train the technologically extended cognitive system. clear description of the word pops up immediately in the corner of your field of view. Now. In the widespread current way of interacting. information can be presented to the user at all LOWERInG InFORmATIOn ACCESS COSTS However. know the meaning of a certain word that is not in your biological memory. in a very real sense. that they could lower information access cost so significantly? Of course Head-Up Displays (HUDs) play a great part in this. You’ve been using this application for a few months now and it’s starting to have quite a good sense of when to present you with the meaning of a word you encounter. Processors in smartphones are becoming faster. when a friend (who is not really into new technology and rather asks a friend to help him) asks you whether you 68 . Let us consider the following scenario to get a better idea: For a few years now you have your own AR glasses. An application running in the background does speech and text recognition and keeps track of words you (the biological ‘you’) don’t know the meaning of. it has become socially quite acceptable to be wearing a Head-Up Display at all times. the information access costs of retrieving information from the Web are way too high to be considered as part of the cognitive process. and a short. batteries are getting better. When it does indeed become ordinary to wear HUDs. by eliminating the physical effort of getting your smartphone from your pocket and having to hold it in within your view. Of course. The technological developments that are associated with Augmented Reality especially have the potential to lower the information access costs for digital sources considerably. similar to the situation with the wrist watch.

you can interact with digital information in similar ways to how you interact with the physical world. well designed.times. I suggest that under certain conditions. One could argue from this perspective that the main goal of education should be to train the technologically extended cognitive system. Furthermore. personalized information sources are able to compete with mental resources in terms of costs of information access. possibilities are created to present information in context-sensitive ways responding both to the environment and the user. By doing so. EXTEnDED COGnITIOn AnD EDUCATIOn In the previous I have introduced the concept of extended cognition and explored the possibilities of digital technology for cognitive extension. intuitive interface. which is of course the main idea of AR3. these digital information sources could then be seen as proper parts of our hybrid minds. SEE hTTp://DAqRI.COm 69 . it could be decided to store certain information that we want to remember in an easily accessible. Adopting this philosophical perspective can have huge consequences for a field like education. Instead of trying to store all information in biological memory by endless repetition. These are the characteristics of AR that create the potential of making digital information much more transparently available to us. personalized cloud of knowledge. digital information can be placed over the world. at the exact moment when it is needed. According to the Parity Principle. Image COURTESY OF GAIA DEmpSEY AnD DAqRI. creating a very natural. By accepting digital external information sources as a proper part of memory. Another important aspect of AR is the use of information from different sensors and smart algorithms doing image and speech recognition. By combining these.

J.uk. An extended view of the mind. 114-126). D. Rorty. O. (2012). (2013). System Upgrade . Natural-Born Cyborgs. Oxford: Oxford Univ. Education as Socialization and as Individualization. R. (2008). REFEREnCES ¢¢ Blomberg. Cognitive science . 4D ANATOMY. Furthermore.com/ media/videos/big-birds-words-demo-reelvideo ¢¢ ¢¢ Clark. S. and the future of human intelligence. Network-Enabled Cognition: The Contribution of Social and Technological Networks to Human Cognition . there should be an active discussion on the goals of education and what technologies students can use to reach these goals. To come to these decisions. (2008)... ¢¢ Qualcomm. Het Semantic Web en netwerktechnologische cognitieve uitbreidingen. P. On distinguishing epistemic from pragmatic action. Press. 513-549. Hendler. Digital technology is developing very rapidly. & Maglio.. robindelange. & Chalmers. Retrieved from http://www. (1994). (2009). in which external resources have an active role in the cognitive process. careful decisions have to be made about how to use it in education. New York USA: Basic Books. For the most part of their education. Braines. In R. Offloading Cognition onto Cognitive Technology. The Extended Mind. Minds. students still only get to use some basic technologies: a pen. E. Supersizing the Mind. ¢¢ Rorty. Oxford: Oxford Univ. The 70 . Smart. & Harnad. Do socio-technical systems cognise? 2nd Symposium on Computing and Philosophy . Embodiment. Big Birds words. A. I.. (2012).this task of storing information could be off-loaded to an external source which is constantly available to us at low information access costs. (1999). (1996). ¢¢ ¢¢ Clark. P.com/wikisofie ¢¢ Dennett. D. Retrieved from http://site. London: Penguin Books. a lot of research on the use of technology in the learning process is needed. ¢¢ Clark. can offer a valuable perspective in this discussion. A. Kinds of Minds: Toward an Understanding of Consciousness. Philosophy and Social Hope (pp. Harnad. cal) calculator5.. 10-23. ¢¢ daqri. (1998). Dror.ac.qualcomm. Engelbrecht. Cognition and the Web. a piece of paper and maybe a dictionary or an outdated (graphi4 de Lange. Action and Cognitive Extension. P... (2003). A. Analysis . E. D. Technologies. In I. Press. ESRC/EPSRC Technology Enhanced Learning Programme. N. (2008). ¢¢ ¢¢ ¢¢ Dror. Smart. Kirsh. Cognition Distributed: How Cognitive Technology Extends Our Minds (pp. Retrieved from http://www.Realising the Vision for UK Education. R. (2010 (In Press)). 1-23). (2011). P. This critical attitude towards the use of technology is very understandable.com/labs/4d-anatomy ¢¢ The general view on the use of technology in education is quite different from the view expressed in this article though. London: tel. D.daqri. & S. & Shadbolt.

School of Electronics and Computer Science: http://eprints. 5. However. it does use certain AR technologies intensively to provide context-sensitive information to the user who interacts with the world. Retrieved June 24. Besides his studies. even remains the same price! Robin de Lange Robin de Lange has a bachelor’s degree in Physics and Philosophy and has followed courses on Artificial Intelligence. you would probably not understand the meaning directly and need to look up more information. Moore’s law seems to be failing here. I will refer to Clark further on. (2011. EnDnOTES 1. M. 2011. For more information: www. Because Clark has written many other papers and books on this subject. For this reason. from University of Southampton .ac. Computational Culture .uk/16649/1/Network-Enabled_Cognitionv17. He is now a student at the Media Technology MSc program at Leiden University and is particularly interested in technologically extended cognition. If the word would be jargon of a field you are not familiar with for example. thereby showing educational challenges and possibilities. he is looking for funding to do a PhD within his field of interest.ecs. Thinking Beyond the Brain: Educating and Building. Most notably. The hardware in these devices stays roughly the same. He is a freelance video producer and science communicator. from the Standpoint of Extended Cognition. November). one might argue that the example does not really show AR. For his graduation research project he is developing an Augmented Reality application that supports the graphical solving of mathematical equations. At the moment. 3.com 71 . Robin has taken part in several entrepreneurial projects.Extended Mind and Network-Enabled Cognition. This characteristic of AR of overlay ing the physical world with virtual objects is not really present in this scenario. 4. 2. The information access costs of looking up a word in the dictionary go through the roof. thereby increasing the costs of information access. he was the co-owner of a company that specialized in homework guidance and tutoring. soton.pdf ¢¢ Wheeler.robindelange. The initial paper ‘The Extended Mind’ was written by Andy Clark and David Chalmers.

HOW DID WE DO IT: WhICh AUGmEnTED REALITY SOFTWARE DOES ThE AR LAB USE? Wim van Eck 72 .

or we supervise 73 . there are so many programs available it is difficult to actually have tried them all. ‘augmented reality software for interactions’ and ‘head-mounted augmented reality’. Easy-to-use for mobile devices Since the AR Lab is based at the Royal Academy of Art The Hague most of our projects are developed together with our students. This doesn’t mean we claim that this is the best software and that there are no alternatives. ‘easy-to-use for desktop augmented reality’. THE APPLICATION WILL PREMIER AT MUSEUM CATHARIJNECOVENT (UTRECHT. we merely aim to give an insight into our daily workflow.THIS APPLICATION. First of all. IS DEVELOPED BY THE AR LAB USING VUFORIA AND UNITY 3D. I will give an overview of the software we use and why we use it. At the AR Lab we use a variety of augmented reality programs to realize our projects. COMMISSIONED BY THE RCE (CULTURAL HERITAGE AGENCY OF THE NETHERLANDS). THE PAINTING 'ISAAC BLESSING JACOB' BY THE DUTCH PAINTER GOVERT FLINCK (16151660) HAS BEEN AUGMENTED TO GIVE INSIGHT INTO THE PAINTERS PRACTICE AND OFFERS SCIENTIFIC DATA IN A PLAYFUL WAY. NETHERLANDS) ON MAY 17TH. and updates and new programs pop up all the time. Often we are asked “What is the best augmented reality software?” which is a difficult question to answer. choosing which software to use really depends on what you want to achieve in the end. Secondly. For this overview we created four categories: ‘easy-to-use for mobile devices’.

however. removing any technical barriers (Figure 1).The app provides clear information for every step you have to take.com) quickly became our favourite choice for this group of students. still prefer to present their creations on Fig. The ‘BuildAR Pro 2 Trial Download’ is the newest version for Windows and features both marker tracking and image based tracking. Some of our students. it’s more difficult to import 3D models from. When downloading BuildAR. You can use this version for free. Furthermore. adds an easy-to-use graphical interface. but there is a ‘demo’ watermark on your video-background and you won’t be able to save your scene (Figure 2).buildar. Aurasma (www. Cinema 4d. and also to import images and videos which are not on your mobile device. fast camera movements will result in loss of tracking. and photographs and videos which are stored on the device can be used to augment the chosen scene. so easy-to-use software is preferable. Easy-to-use for desktop augmented reality Nowadays most software companies seem to focus on mobile devices. though the OSX version lacks some options and seems to be less stable. they often have their own smartphone or tablet and are very keen to develop for it. By downloading the free Aurasma app from the iOS App Store or Google Play store students can immediately begin creating their first project. 2 a large screen. a not very common option which can give great results. BuildAR is based on the world-renowned ARToolKit.aurasma. but students who want to develop their own application.nz) is the software we use most often. you only need to register to be able to access it. 1 not the best around. When more possibilities are needed there is also a free online application (www.aurasma. for example. desktop augmented reality software is getting scarcer. ‘BuildAR Pro Mac 74 . The tracking quality of Aurasma is Fig. as well as 3D models. Markers can be generated on the spot by taking pictures with the device. and BuildAR (http://www. It is very easy though to use video with transparency. all they need is their mobile device.com/partners). but it is definitely good enough for most projects. It runs on both Windows and OSX. there are quite some versions which can be a bit confusing. This well documented application makes it easier to precisely position your virtual objects. It is a pity that Aurasma only seems to give good support for 3ds Max and Maya. Many of these students don’t have a technical background. co.

Lastly. Unity makes Head-mounted augmented reality For head mounted based augmented reality we have solely used software and hardware developed by TU Delft. which is only available for Windows and is free of charge. audio and 3D models. AR software for interactions A program we often use at the AR Lab ourselves is Vuforia (https://www. BuildAR is easy-to-use.Trial Download’ is the OSX version and only features marker tracking. Also. you will need to buy the Unity iOS or Android add-on to be able to export to a mobile device.com) allowing you to easily export to iOS and Android. the new version can augment your scene with images. Unity (Figure 3) is a popular game engine offering excellent visual quality on mobile devices. Semiconductor company Qualcomm offers Vuforia for free and Unity has a free version as well. 3 75 . this issue of AR[t] features more information about this system. Fig.unity3d. Vuforia’s image based tracking is extremely stable. there are many tutorials around and the Unity forum is a great source for information. The only limitation is a ‘HITLabNZ’ logo. In issue 2 of AR[t] we introduced our latest augmented reality headset named Marty and its tracking software. but many of our students don’t mind this logo so much though. video. it very easy to import 3d-models including animations from almost all 3d-packages. However. the demo-limitations are the same as in the Windows version. the tracking quality stays good even if the tracking image is partly occluded or when there is little light available.com). Android (Eclipse) or use it as a plugin for Unity (www.vuforia. You can natively develop for iOS (Xcode). Besides image markers there are also frame markers available. Adding interactions through scripting is relatively easy. there is the ‘BuildAR Free version Download (2008 version)’. which use a pattern of black and white cubes positioned around the image.

AR[t] PICK ImmATERIALS BY DESIGn STUDIO OnFORmATIVE AnD ChRISTOphER WARnOW. 2011 76 .

COm) AnD ThE COmpUTATIOnAL DESIGnER ChRISTOphER WARnOW (ChRISTOphERWARnOW.com/work/immaterials/ 77 .COm). ThE pROJECT EXpLORES hOW InFORmATIOn CAn BE InTEGRATED InTO phYSICAL SpACE.IMMATERiALS IS ThE RESULT OF A COLLABORATIOn BETWEEn ThE DESIGn STUDIO OnFORmATIVE (WWW. ThE RESULTInG FORmS DEpICT pOSSIBLE DATA SETS AnD EXAmInE ThE DESIGn pOSSIBILITIES BETWEEn TEChnOID hOLOGRAmS AnD pERSOnAL nOTES. DATA IS pLACED In A ROOm.com/portfolio/?p=244 http://www. USInG ThE LIGhT pAInTInG TEChnIqUE.onformative. Further information: ¢¢ ¢¢ http://christopherwarnow.OnFORmATIVE.

and models for perceptualization. 78 . Mariana took up editing duties for this current issue. she is a part time student at the Royal Academy of Art (KABK) and runs her own graphic design studio Ooxo. Maritime and Materials Engineering (3ME). web design. PIETER JOnKER Delft University of Technology P. Her interests are in graphic design. EDWIn VAn DER HEIDE Leiden University evdheide@liacs. She holds the post of Lector (Dutch for researcher in professional universities) in the ­ field of ‘Innovative Visualisation Techniques in higher Art Education’ for the Royal Academy of Art. game design and creative research. Beside’s running his own studio he’s part-time assistant professor at Leiden University (LIACS / Media Technology MSc programme) and heading the Spatial Interaction Lab at the ArtScience Interfaculty of the Royal Conservatoire and Arts Academy in The Hague. His main tasks are developing Augmented Reality projects.nl ESmÉ VAhRmEIJER Royal Academy of Art (KABK) e.vahrmeijer@kabk.ContribUtors  WIm VAn ECK Royal Academy of Art (KABK) w. The Hague. MAARTEn LAmERS Leiden University lamers@liacs.nl Yolande Kolstee is head of the AR Lab since 2006. YOLAnDE KOLSTEE  Royal Academy of Art (KABK) Y.nl MARIAnA KnIVETOn Royal Academy of Art (KABK) m. parallel image processing architectures. robot vision. Faculty Mechanical. typography. scientific creativity. space and interaction. real-time 3d animation. bio-hybrid computer games. His main interests and fields of research are: real-time embedded­ image processing.Kolstee@kabk.nl Wim van Eck is the 3D animation specialist of the AR Lab. Her research interests include interaction in interactive art and (non-visual) Augmented Reality.kniveton@kabk.nl HAnnA SChRAFFEnBERGER Leiden University hkschraf@liacs.nl Mariana Kniveton is currently a master student at Utrecht Universtity. After a brief stint as a cover model for AR[t] #2. among others. Specializations include social robotics.nl Edwin van der Heide is an artist and researcher in the field of sound. photography and education. Pieter Jonker is Professor at  Delft University of Technology.Jonker@tudelft.nl Hanna Schraffenberger works as a researcher and PhD student at the Leiden Institute of Advanced Computer Science (LIACS) and at the AR Lab in The Hague.vaneck@kabk. supporting and supervising students and creating 3d content. Maarten Lamers is assistant professor at the Leiden Institute of Advanced Computer Science (LIACS) and board member of the Media Technology MSc program.P. Esmé Vahrmeijer is the graphic designer and webmaster of the AR Lab. Since september 2012 she has worked as an intern at the Research Department IVT and the AR Lab. His interests are. studying New Media and Digital Culture. robot learning and Augmented Reality. Besides her work at the AR Lab.

augmented reality.com Lotte de Reus recently received her Master’s degree in Design for Interaction at the Faculty of Industrial Design Engineering at Delft University of Technology. Heavy creates innovative interfaces between digital design and physical worlds in ways that that provoke the imagination and challenge existing styles of art.nl MATT RAMIREZ teamscarlet. With a background in virtual reality and interaction design.verlinden@tudelft. he leads the “Augmented Matter in Context” lab that focuses on blend between bits and atoms for design and creativity. and interaction. Lotte would again like to work in the domains that combine art and technology. science.JOUKE VERLINDEN Delft University of Technology j. This interest has resulted in the science fiction novel "The Augmented Star". special collections and business.T Project Management.robindelange. He is interested in where technology could take us.lottedereus. He is looking for funding to do a PhD on technologically extended cognition. He creates objects that let us rethink our environment. ALEJANDRO VELIZ REYES University of Salford ROBIN DE LANGE www. NEXT ISSUE The next issue of AR[t] will be out in the fourth quarter of 2013. Oliver Percivall works in I.theheavyprojects.com OLIVER PERCIVALL Antal Ruhl is a media artist with a background in design. LOTTE DE REUS www. 79 .com Robin de Lange is a student at the Media Technology MSc program at Leiden University and part-time entrepreneur.wordpress. and collaborative technologies for design ANTAL RUHL www.c. BC "Heavy" is founder of The Heavy Projects [and its collaborative spin-off Re+Public]. These objects vary from kinetic sculptures to interactive installations. Matt Ramirez has over 15 years experience in web design and e-learning content development for a variety of subject areas including medicine. design. IT. His research interests are design pedagogy.nl GUEST CONTRIBUTORS BC “HEAVY” BIERMANN The Heavy Projects www.antalruhl. Alejandro Veliz Reyes is a teaching assistant and PhD student on digital architectural design at the University of Salford (United Kingdom). science and art.com Jouke Verlinden is assistant professor at the section of computer aided design engineering at the Faculty of Industrial Design Engineering. In the future.

Sign up to vote on this title
UsefulNot useful