Final Report HART Demonstrator Project Lisa Wymore Assistant Professor, UC Berkeley Department of Theater, Dance, and Performance

e Studies The project is entitled: Portable Durable Tele-Immersion Technology for Teaching, Artistic Performance, Global Networking, and Experiments in Creativity and Collaboration The project was initially conceived in Summer 2008. At that time I was creating a largescale performance project in conjunction with the Merce Cunningham Dance Company residency at UC Berkeley for a November 2008 performance. Cal Performances, the prestigious presenting organization at UC Berkeley, produced the event on November 14th. The perfor mance was entitled: Panorama-Multi-Media and involved Happening numerous art and technology units across campus. For Panoram I continued my creative a explorations with Tele-Immersion, a technology that I have been working with since 2005 with the labs here at UC Berkeley and at the Univer sity of Illinois predominately. For this project it became clear that we needed to create a portable Tele-Immersion system for the November 14 t h performance so that Tele-Immersion could be experienced by the audience members attending the event. So, working with Ruzena Bajcsys TeleImmersion lab within the Electrical Engineer ing and Computer Science Department at UC Berkeley, we cr eated two portable systems that existed 100 feet apart from each other. User s could engage with the system and see one another across the performance space and also on large monitors sharing the virtual meeting environment. For those unfamiliar with Tele-Immersion technology, it can best be described as a distributed collaboration tool that utilizes 3D image based rendering of users in real time. Remote users can meet in virtual envir onments and share virtual worlds together, not as traditional avatars which use animation graphics, but as new kinds of avatars that use real time motion and video based images of users. Further more, any 3D data can be uploaded to the virtual environment to be investigated within collaborative virtual settings. Examples of the kind of work we do in the UC Berkeley Tele- Immersion lab can be seen at the following website: http://tele-immersion.citris-uc.org/video This HART Demonstr ation Project funded a graduate student, Ram Vasudevan, to create a more robust ver sion of the portable Tele-Immersion system to be used within a dedicated dance studio space within my Department of Theater, Dance, and Performance Studies. The larger goal of this project is to create what I am calling Integrated Distance Labs ( ID-labs) within other art/dance based departments across the country and hopefully throughout the world, that allows a new kind of choreography to emerge. We are also working on the integration of body based sensors and sound within the system. We have tests with UC Merced and UC Davis planned this coming spring 2010, as well as tests on the UC campus between the dance studio I D-lab and the Tele-Immer sion lab in Electrical Engineering and Computer Science. Furthermore, in spr ing 2011 an entire evening of perfor mance will be dedicated to questions around dance and technology and

the portable Tele-I mmersion lab, that HART has funded, will be at the heart of this perfor mance. I mention these things because they address the major issues that we are tackling with Tele-I mmersion at this time: 1. The robustness and transferability of Tele-I mmersion technology to a larger public, and 2. The integration of sound and body based sensors into Tele-Immersion technology, in real time, to enhance remote user experience and collaboration within the system. The hope is that the expansion of Tele-Immersion labs, using the same vision and rendering protocols will allow for a network of distributed labs to communicate and collaborate around a variety of 3D data and information. The results that we have produced from our Demonstration Project funding: 1. A highly functional lab has been built in Zellerbach Room 170 dance studio. This space is now a dedicated dance lab and will soon become an Integrated Dance Lab (ID- lab) with body based sensors, surround sound speakers and dedicated Tele-I mmersion technology installed into the space permanently 2. All equipment needed for the Portable Tele-Immersion system has been purchased and tested. a. Projection equipment is installed b. Special stage lights have installed c. Internet has been installed (3 IP addresses) d. Adequate electricity has been installed e. Proper projection sur faces and flooring has been installed 3. The renderer necessary for bi-located Tele-Immersion has been updated in collaboration with UC Daviss CAVE labs. 4. Intense software updates and code updates have been completed by Ram Vasudevan (funded by the HART Demonstration project) and the numer ous other computer scientists and engineers working in the UC Tele-Immersion lab. 5. A test between the two UC Berkeley Tele- Immersion labs (dance studio and EECS lab) has occurred using one of the 3D cameras. (see images below) Later this f all we will test with four 3D cameras. 6. Future tests are scheduled between UC Merced and UC Davis with regard to Tele-I mmersion technology in general. See website listed above for videos showing the kinds of collaboration that we are working on. 7. Other Dance Programs and Departments are showing interest in developing TeleImmersion (ID-labs). Namely UC Santa Cruz and UC Irvine will be working in partnership with the UC Berkeley labs over the next two years.

Regarding my academic efforts: I t is hard for me to repor t on this in the tr aditional sense. I am a chor eographer working with electrical engineers and computer scientists on a highly complex and multi-modal project within a research based institution. The only way that I can answer this is to say that the Demonstration Project is inspiring on many levels to me and in a sense inspiration and creativity are my resear ch. Dance involves bodies in space and time and Tele-Immersion technology prioritizes bodies in space and time. It also involves people collaborating together using visual feedback, which is another means by which dance choreography is made. So in essence, by work with TeleImmersion in general has been extremely helpful to my research as a dance choreographer. With regard to the HART demonstration project, bringing a portable Tele-I mmersion lab into a dance studio is unprecedented. It is my hope that by creating this lab it will become a beacon for other departments to do the same so that a new kind of body based interactive mechanism for collaboration and communication can be developed. This tactic has been working. As mentioned above, other UC dance departments ar e getting involved and a renowned dance and technology artist from the University of Utah, Ellen Bromberg, will be on campus in spring 2011 to work directly with the dance studio Portable Tele-Immersion lab, funded by HART. So this pr oject has had incredible impact on my visibility as a dance ar tist who engages with and collaborates with technology. This has had a positive effect f or the public perception of UC Berkeleys engagement with new technologies in relationship to the arts in particular. With regard to impact within the larger Project Bamboo community, it is difficult to say. Project Bamboo prioritizes easily shared programs and established digital systems rather than development of physical labs. It is my hope that Tele- Immersion technology remains one of the future concepts for Pr oject Bamboo to invest in. One can imagine that a collaborative 3D environment in real time, based on real body movements and real body images will one day be highly usef ul and gr een technology. It goes beyond traditional webcasts or tele-conferences because information can be viewed from infinite angles, the user can engage with the system as either first person or viewer, there is infinite amount of digital data that can be used within the system for analysis, collaboration, and archiving purposes. In some ways it is a kind of futuristic library or public space/commons a place to come and find information, meet in virtual rooms or environments, make things in collaboration, explore through multi-modal approaches (vision, sound, body based sensors, robotic interaction with data, etc.)

Conclusions that I can draw from my demonstrator project relating to research, technology and/or collaboration: 1. The hardest thing for a lab is to maintain engineers who are able to take the technology into new dir ections and at the same time refine and codify code and protocols so that when new engineers begin working on the project all is not lost. 2. There never seems to be enough support for projects that use physical spaces and labs, and/or technology that is based in real time. These spaces require a dedicated manager, upkeep, and the like, and r eal time requires incredible bandwidth technology. But for a dance artist/r esearcher wor king with technology it is essential to have physical sites and streaming real time data as tools. 3. If we seek to involve dance depar tments, or other interested parties, from throughout the world in this project what are the best practices for distr ibuting protocols, code, equipment needs, etc. so that others can build labs of their own. 4. How can we keep abreast of the latest, less expensive, technologies to make portable Tele-Immersion available to more people?

5. How do we tackle issues of internet bandwidth for real time interactivity. 6. Is off-line data useful to other humanists? And should more of f-line engagement with Tele-Immersion be investigated? Regarding lessons learned from this project: As I write this section I am realizing that I am continually learning from this project. The major concepts and ideas that I take away from this project are: 1. It is essential to have engaged and talented graduate students working on multidisciplinary projects. I cannot speak highly enough of Ram Vasudevan, who was the graduate student funded thr ough this Demonstration Project. 2. Collaboration on a pr oject such as this requires weekly meetings and involvement from all who work in the lab; even if other projects seem unrelated the exchange of infor mation and ideas that comes f rom meeting is essential. 3. Technology labs are filled with extremely creative people who often dont always speak the same language. And being an artist entering this world it requires patience to become an active participant. Interaction over a long period of time, sharing of presentations and regular meetings are essential for success. 4. It takes much longer to accomplish a specific goal within a collaborative setting. Often the engineer s that I work with ar e engaged with other research projects and it takes time and perseverance to create a successful collaboration between artists and technologists. From the technologist perspective Ram Vasudevan, has learned much from his work on this Demonstration Project. He has established an entirely novel way of dealing with images and texture within the Tele-I mmersion system. All together there are 48 cameras, which can simultaneously capture images with 15 20 frames per second. The clusters are arranged around the user to capture a full-body 360-degree model in real time. Each of the clusters performs stereo calculation on pairs of cameras by matching reliable features. The features are selected using a novel approach by first performing triangulation of the image, which divides the image into triangular segments of various sizes depending on the texture and color consistency of the scene. The homogenous (i.e. textureless) regions result in larger triangles while the textured regions have many small triangles. The stereo matching is performed on the acquired triangle nodes. Using this tr iangulation has several benefits, (1) it helps find reliable features, (2) it reduces the number of matches that need to be performed, (3) it keeps a global structure of the 3D representation, and (4) it allows for efficient compr ession of the data. This adaptive triangulation allows for fast (under 25 ms) and robust real-time 3D reconstruction. The reconstruction rate is, however, limited by the image acquisition rate of the cameras and the networ k bandwidth to the frame rate of about 20 FPS. Once the stereo calculation is performed, the triangulation is encoded along with the color and dispar ity information for each node. The data is packaged and sent to the local renderer or remote gateway. The gateway controls the network traffic by routing the packets over long distances to minimize the delays. Once the renderer receives the data, 3D mesh is extracted and transformed into a common coordinate system. The renderer

needs to have exact information of the camer a clusters, which is obtained by prior geometric camera calibration. The render ing algorithm combines the information from diff erent views by weighted contribution from different clusters, (overlapping clusters may reconstruct the same surface) using real-time ray-tracing technique to generate appropriate images in the visual buffer. Future Directions: As one can see from the above information, Tele-Immersion technology is a complex system involving endless calibration, software and hardware updates, and synchronization of data from distributed labs. The Demonstrator Project f unding has been essential in establishing a kind of codified protocol for futur e work in this area. Rams mathematical calculations, attention to calibration and his cleaning up of the code have made for a more robust and realistic experience. Gregorij Kurillo, the Head Engineer in the lab, has also added to the system incredibly by establishing renderer protocols, expanding the visual perspective within the system, adding new 3D data, establishing collaboration with UC Merced and UC Davis, and much mor e than I can list here. Also, post-docs, grad students and visiting computer scientist from all over the world have contributed to the lab in areas r elated to visual texture, human motion detection, skeletonization, sensors, robotics, and the like. It is a VERY active lab, led by Ruzena Bajscy. I have ever y intention of continuing my work as collaborator in Tele-Immersion technology adding my expertise with regard to body movement, collaboration protocols, creativity enhancement, live performance engagement, and improvement of computer human interaction. Another aspect of the pr oject that I would like to seek support for relates more to project Bamboo and that is the creation of an inter active website (or shared tool) where protocols, programs, lab designs, best practices, equipment lists, data sets, results from experiments, publications, ongoing discourses, etc. are made public. The desire is to foster this kind of open engagement with the public so that TeleImmersion technology is adopted within other university/research settings and infor mation is shared. As I mentioned earlier in this report, my future work will involve the creation of an Integrated Distance Lab (I D-lab). I am currently working with Po Yan, and engineer within the UC Berkeley Tele-Immersion lab, to work with body based sensors and sound. Imagine a moment when a dancer enters a 3D Tele-Immersive environment surrounded by multiple 3D digital cameras and displays, where internal and external cues for creative movements come not only from physical objects located within the 3D Tele-Immersive virtual space but also from remote participants who are geographically distributed, yet visually present in the virtual space. The dancers collaborate with one another utilizing visual cues to solve choreographic problems or to investigate new movement potential within this virtual dance studio. Suddenly the choreographer has an entir ely new paradigm within which to make creative choices. Now imagine adding a somatically derived oper ation to this model the addition of body-based sensors whose outputs can be synchronized to sound, and this sound can be streamed, in near real time, and with

some synchronized accuracy, to the visual data being received by the user . The 3D TeleImmersion system will now allow user s to not only see one another but also hear each other. In a way it is like they can sense each others bodily responses through the digital divide and make movement decisions based on this information. The sensors can be synchronized to realistic body sounds ie: the heartbeat, or they can be linked to musical digital interfaces that allow for infinite sonic possibilities - using a visual programming language f or multi-media called Max/MSP1. The dancer s will then be able to perceive each other through both visual an audio feedback within the virtual dance studio. Offline, and in the "real! dance studio,dthe main way dancers sense each other is through listening to breath patterning and through the sensing of other dancers levels of exertion and effort. The ID-lab seeks to replicate, albeit through a different mode of awareness, the feelings that dancers experience within "real dance studios. The ID-lab will do this by attaching small, very sensitive, microphones near the users mouth and also by adding two sensitive body sensors: a physiological biosensor that captures heart rate and br eath patterns; and an accelerometer that captures velocity of gesture and spatial intent of gesture. Using MaxMSP the output from these sensors will be converted to sound files locally and streamed to the participating Integrated Distance Labs. The user at each site will hear the streamed sound files from the participating Integrated Distance Lab via a surround sound speaker system. Each lab will be equipped with the same sound system for calibration, and synchronization purposes. With the sound being experienced "around! the user and the sounds being heard constructed from the users bodies themselves, a r ich audio wor ld will be created. A computer scientist and signal-processing expert, Gerald Friedland, is advising the pr oject on the audio and sensor data streaming parts. The Demonstrator Process: Should demonstrators be included as part of future Bamboo activity? I f so, how would you improve the demonstrator process? I have just a few suggestions: 1. I felt that the day the UC Berkeley HART funded projects wer e able to share displays with one another was really useful. It was a kind of mini-conference where ideas were shared, people got to mingle and look at displays/tables, and talk about their projects with on another. I wonder how that collegially experience could continue? Per haps another mini- conference or open studio to visit each others work areas? 2. I wonder if Demonstration Projects fr om other campuses could post slides or mini-presentations on the Bamboo site? Perhaps they do already and I am missing this? But it relates in some ways to the story collection or recipe gathering that Project Bamboo is wor king into their system. 3. I would love to be able to demonstrate the Portable Tele-Immersion system sometime at a Project Bamboo conference, and see other practice/site/lab-based technologies as well demonstrated live. 4. I miss seeing a kind of live, body based interaction with technology within the Bamboo community, but this is my bias. I realize that this type of wor k is not

necessary considered shared. That is can we interact with technologies at Bamboo conferences more? We sometimes have web casts but what else can we work with live in real time? Thank you so much for funding my wor k as a Demonstration Project. I look f orward to being part of the Bamboo Community for a long time to come! Lisa Wymor e