HE-Book: A Prototype Haptic Interface for Immersive E-Book Reading Experience

Kazi Masudul Alam, Abu Saleh Md Mahfujur Rahman, Abdulmotaleb El Saddik

Multimedia Communications Research Lab University of Ottawa Ottawa, Canada e-mail:{malam@discover,ka@mcrlab,abed@mcrlab}.uottawa.ca

A BSTRACT This paper presents an intuitive approach of annotation based haptic interaction with traditional digital reading materials such as eBooks. The research targets to bring a multi-sensory interface consisting of perceptual, cognitive and vibrotactile interactions with the digital reading contents. It leverages a previously developed haptic jacket to receive haptic emotive signals wirelessly in the form of patterned vibrations of the actuators in order to pave ways for intimate reading experience for the readers in various eBook platforms. Index Terms: H.5.1 [Multimedia Information Systems]: Articial, augmented, and virtual realities; H.5.2 [User Interfaces]: Interaction styles; I.3.6 [Methodology and Techniques]: Interaction techniques 1 I NTRODUCTION Reading is a complex cognitive process, which intends to decode meanings from text by relating words with one existing knowledgebase and understanding. According to Dale [9] more use of modalities ensures effective learning. He described our learning phenomenon numerically 10% of what we read, 20% what we hear, 30% of what we see, 50% of what we hear and see, 70% of what we say and 90% of what we both say and do. Though these numerical labels are not widely accepted but the theory, Cone of Learning is more acknowledged in the research communities relating to analysis of the reading behaviours. Our ways of readings are continuously moulded with the advent of new technological innovations, devices and platforms. For example, eBook is continuously challenging the existence of print book and at times eBooks are replacing the former in schools [17]. In our daily life, we are getting more and more explored to electronic materials than printed books as now-a-days most of the advanced hand held mobile devices provide eBook reading facilities [22]. Haptics is considered to be both as independent and supplementary medium of communication channel [18]. Physical contacts are fundamental needs to mental and psychological development and hence their applications in various cognitive applications have attracted attention of many researchers around the world [12]. Recently handheld mobile devices are widely adopting tactile feedbacks in their touch screen based interfaces [5]. With the growing popularity of the haptic sensory feedbacks in the mobile handheld devices [14], the challenge is now to nd new avenues to intuitively incorporating the feature into mobile applications so that interactive storytelling becomes more appealing to the users. According to Mangent, Haptic perception is of vital importance to reading, and should be duly acknowledged. The reading process and experience of a digital text are greatly affected by the fact

IEEE World Haptics Conference 2011 21-24 June, Istanbul, Turkey 978-1-4577-0298-3/11/$26.00 2011 IEEE

that we click and scroll, in contrast to tactilely richer experience when ipping through the pages of a print book [17]. As todays user community is widely accepting haptic interfaces; the effect of tactile feedbacks from the reading materials has started to form a new genre of research interest. Recently, many eBook reader devices have introduced tactile as well as sound feedbacks while ipping a page. Motivated by some of these intuitive interaction styles we present a new approach to evaluate suitability of users acceptance of tactile feedbacks from eBook reader contents. This interaction will consider not only page ipping interaction but also different contents of the pages. This is particularly signicant as tactile feedback has been present in most of the handheld devices now-a-days [14]. In this regard, we present a preliminary prototype exploring the suitability of haptic feedbacks in eBook reading platforms that generates tactile feedbacks when a user reads an annotated part of the eBook content. The eBook reader application can read annotated digital materials e.g. Portable Document Format (PDF) type of eBooks that listens for touch based interactions from the user. When a user interacts with the annotated parts, the system provides distinct vibrotactile effect through a haptic device e.g. wearable haptic jacket [7]. We incorporated manual haptic annotation of the eBook content [19] with different types of haptic stimulations which also includes three from our previous work [21]. As an example, let us suppose that a user is reading paragraphs in a book that describe a terrible sea storm experience. While reading this part of the book using traditional eBook reader, the user is not provided with any immersive experience. Using augmented reality book [4] user can see this scene using some head mounted display or mobile screen but still do not receive any kind of haptic experience. The difference our system can bring to this user experience is that while reading that very eBook sitting on a haptic sofa, wearing a haptic jacket [7] user will receive various vibrotactile haptic feedbacks during the reading process. Such eBook reader facilitates user with immersive reading experience. And this service will not be limited to eBook reading devices like Kobo, Kindle, Sony etc. but will also include mobile phones or desktop eBook reading platforms. Our contribution in this article is three folds. First of all, we propose the annotation based haptic feedback integration in traditional digital reading contents. The haptic interaction is manually dened by analyzing the scenarios of the eBook content. Secondly, we present a generalized framework for such an HCI application that describes the haptic data annotation scheme and various interaction mechanisms with the annotated content. The annotation method used in the prototype allows the creation of separate annotation les, which can also be stored in remote annotation repository. Third, we present a prototype application that is capable of handling touch based interaction and generates haptic rendering to surrounding haptic devices such as a haptic jacket. Usability test results based on the developed prototype have been prepared to show the suitability of such an approach.

367

Figure 1: High level architecture of the proposed system

The remainder of this paper is organized as the following. In Section 2 we provide literature survey of existing or closely related applications. In Section 3, we illustrate various components of our system that facilitates the annotation of haptic in digital reading materials. Further in Section 4 we describe the implementation issues of the proposed work and present a usability study of the developed prototype. We conclude the paper in Section 5 and state some possible future work directions. 2 R ELATED S TUDY

According to a research article by Harrison [13], french visionaries Robida and Uzanne stated the future of books in 1895. According to those visionaries, written book contents will be transferred to sound wave and will be played in gramophone and subsequently gramophone will evolve to become pocket-sized player. A nice work by Harrison explored the history of book evolution upto 2000. This study mainly focused on reading usability, possible application areas and further development of interaction techniques. MagicBook [4] project was an early attempt of user transportation between reality and virtuality. When a person looked at the pages using an augmented reality display he or she could see 3D virtual models out of the pages. It also supported multiple user immersion in the same scene. Later this idea has been explored in many areas: gaming, chemistry, cultural heritage, biology etc. In their work Chen et al. introduced a new design of dual display for eBook reader and various supported interactions like folding, ipping, and fanning in their system. The main goal of this article is to replicate physical metaphors used in dealings to real book [8]. From these works we get an understanding of the layout of information, ease of navigation and importance of annotation in augmented reality book materials. In a long cherished development of mixed reality book, Grasset et al. have focused on design and development of mixed reality applications in broader sense. They have discussed semantics of mixed reality book, design space and user experience with such type of interfaces [10]. Card et al. [6] have proposed physical behaviourial simulation of a real book in the presentation of a 3D virtual book. They extended their prototype [15] to support virtual annotation or bookmarking. More recently, Welch et al. [23] extended the concept further to an immersive virtual reality book aimed at medical training for surgery. In another research,

Gupta et al. explored integration of projected imagery with physical book in order to introduce tangible interface to multimedia data. They have used camera and projector pair to introduce a tracking framework, which monitors 3D position of planar pages when user turns book pages back and forth. Their book can be loaded with various multimedia contents such as video, images, data etc. [11]. Several works on augmented books has explored relationship between real and digital book, various interaction techniques and technologies, audio-visual augmentation etc. But according to the best of our knowledge, none of them studied the opportunity of tactile feedback or haptics as an augmentation or annotation to the book content. In this paper we have introduced a new type of eBook called Haptic E-Book (HE-Book) that annotates eBook contents with various unique vibrotactile signals and later render the signals while user is reading that part of the book. To the best of our knowledge, haptic annotation to eBook contents is a new idea where a certain amount of research contribution is possible. For example, haptic annotation to augmented reality books can be an interesting genre to explore. In this paper we have presented our introductory prototype of the HE-Book. 3 H APTIC E BOOK S YSTEM

The HE-Book system consists of three separate modules and an interaction controller as depicted in Figure 1. One of the module is Ofine Annotation, which handles eBook annotation and annotation retrieval operations. Next module is Touch Based GUI that is responsible for user nger touch detection and processing. Last module is Haptic Interaction that handles content to haptic mapping according to the target output device. Operations of all the modules are centrally monitored and organized by the Interaction Controller. Workow of the system is, rst we create and store the haptic annotation le of an eBook. Later, when a user opens an annotated eBook using our HE-Book reader software and touches any annotated part, he/she will receive predened sets of haptic feedback in his/her surrounding haptic devices. In Section 3.1, we present the ofine annotation method and a detailed description of the annotation editor. Further, in Section 3.2 we describe the graphical user interface (GUI) interaction method in details. Later, in Section 3.3 we have illustrated haptic signal generation scheme. Finally, in Section 3.4, we present overall management method of various system modules by the interaction controller module.

368

3.1

Ofine Annotation

Ofle Annotation module is a combination of three submodules: Annotation Editor, Annotation File, and Annotation Retriever. Followings are the detail description of these three submodules. 3.1.1 Annotation Editor

Using a desktop based software system (Fig. 3) an operator will manually select a paragraph of an eBook i.e. PDF content and tag it using various haptic descriptions. For each paragraph there will be an XML tagging block, which is editable. Later the complete XML le will be stored for further usage by the HE-Book reader system. 3.1.2 Annotation File

Advantage of seperate XML based annotation le (Fig. 2) is that a HE-Book user can decide to either use the haptic extension system or avoid it. For the annotated paragraph of an eBook we create an XML element Page which has attribute ID that denotes the unique number of the page. Every Page element is divided in Para which also has ID that denotes the paragraph no. Granularity level of our annotation ends at Para i.e. vibrotactile feedback will start and end in a paragraph level, which we call scene or paragraph based annotation. Under the Para we have another element Haptic which is described using Type, TimeUnit, Pattern, Frequency and Delay. Inuenced by our previous works we manually annotated the eBook content with various haptic feedbacks [21] like Touch, Fun, Hug, Tickle and Kiss [20] in our prototype HE-Book system. Moreover we also extended haptic annotation to the basic emotion sets Love, Joy, Surprise, Anger, Sadness, Fear etc.

Figure 3: Annotation editor for the eBook document (i.e. PDF)

page through touch based interaction. In one of the prototypes, the eBook reader is deployed in an emulated touch mobile device. When a user touches an annotated part of the HE-Book the standard touch SDK is used to obtain the screen coordinate of the touched device. Afterwards the touched screen coordinate is matched with the eBook page paragraphs to obtain the current paragraph that the user is pointing at or touching. The eBook reader software continuously monitors the touched screen coordinate interaction. As soon as the user touches a paragraph of the page that has been previously annotated, the user is sent a motor vibration to conrm that a valid haptic annotation for the selected paragraph has been found. The GUI further initiates the haptic feedback retrieval process by sending the page ID and the corresponding paragraph ID to the Interaction Controller module. The Interaction Controller coordinates the retrieval operation and receives the appropriate haptic signal for the selected reading material. Further, the Interaction Controller takes necessary steps to initiate haptic signal rendering to the haptic jacket system as explained in the later section. 3.3 Haptic Interaction

Figure 2: XML based annotation le for the eBook document.

3.1.3

Annotation Retriever

This submodule (Fig. 1) is responsible of searching the annotation le when it is necessary and retrieve the requested XML block. In a nutshell this is an XML parser. For a request of {Page, Para} pair, this XML parser will return corresponding Haptic block from the XML le. 3.2 Touch Based GUI

The preliminary haptic eBook system works in a touch based device. The user can ip pages and scroll the paragraph of the

Based on the page and paragraph information of Touch Based GUI, Interaction Controller gets haptic description from the Annotation Retriever submodule and generates the described haptic signal for the target haptic device. Haptic Signal Generator plays an important role for device specic haptic signal generation. As the target haptic device can be heterogeneous such as haptic jacket, haptic armband, haptic sofa, haptic enabled mobile phones etc. hence it is important to consider device specic conguration. In our system architecture we have considered Bluetooth as a method of communication between our system and the corresponding haptic device. Such communication can also be extended to other possible personal network communication methods. In our prototype system we have used a Bluetooth enabled haptic jacket which has location based array of vibrotactile actuators. Location means arm, abdomen, shoulder, neck, chest etc. of human body. Our haptic jacket consists of an array of vibrotactile actuators that are placed in different portions of the jacket and their patterned vibration can stimulate touch in the users skin. Vibrotactile actuators communicate sound waves and create funnelling illusion when it comes into physical contacts with skin. A series of small actuator motors are placed in a 2D plane in the jacket in a certain manner. In order to produce touch feeling the actuators are activated

369

Figure 4: The Haptic jacket.

in a dened manner [3][7]. The jacket used in our prototype adhering to the aforementioned properties is shown in the Figure 4. 3.4 Interaction Controller The Interaction Controller plays the central role to organize and synchronize necessary work ow in the haptic eBook reading system. The Interaction Controller polls to acquire user touch inputs after certain interval. As soon as a touch interaction is performed the Interaction Controller coordinates the screen coordinate based paragraph identication of the eBook document. The Ofine Annotation module takes the page ID and the paragraph ID to retrieve the haptic signal associated with the paragraph and returns the signal data to the Interaction Controller. Further the Interaction Controller transmits the obtained haptic signal to haptic devices where the signal is rendered and the user feels the appropriated haptic feedback associated with the learning content of the current book. In addition to coordinating the aforementioned task Interaction Controller ensures that the system modules are not blocking or throttling the operation of the other modules. The Interaction Controller employs carrier sensing algorithm to determine the active/idle states of the other modules. For example, before the Interaction Controller transmits the haptic signal data to the haptic jacket system, it senses whether the haptic jacket system is still rendering an unnished haptic signal. Then, the Interaction Controller removes the haptic signal from the pool of pending haptic signal queue. In this manner the Interaction Controller can avoid repeated haptic signal generation for the same paragraphs unless it is otherwise desired by the eBook readers. 4 I MPLEMENTATION AND R ESULTS We present the implementation details of the haptic eBook system in Section 4.1. Further in Section 4.2 we present user study of the implemented prototype system to determine the suitability of the proposed system. 4.1 Implementation Details In this section we present the details of the implementation issues of our prototype system. One of our prototypes was developed for desktop eBook readers using Netbeans 6.5 IDE and the primary language was JAVA. In order to develop the eBook reader, we locally build the ICEPdf [2] open source JAVA PDF viewer. We have used a Dell touch screen monitor for our system and added the touch processing part to the ICEPdf. We also have added XML retrieving facilities to the ICEPdf for annotation searching. For serial port based Bluetooth communication we have used GPL based BlueCove [1] library which is very handy for J2SE based Bluetooth communication. Our prototype system communicates with a haptic jacket (Figure 4) when any annotated eBook part is

touched by the user. A Bluetooth device was connected with the PCs USB port, which was virtually congured with the COM port so that the Bluetooth device can send signals to the haptic jacket. Our desktop based test prototype was adequately responsive in a standard Pentium dual core 32-bit machine with 2 GB systems RAM. In order to annotate an eBook we have taken PDF as example and developed a PDF annotator modifying the ICEPdf that we have used as viewer also. We have added tagging window to the system where we have considered various tagging options. An operator can select a paragraph of a PDF le and tag it using our prototype editor of Fig. 3. For each tagging an XML block is created which is edited by the human operator for various haptic attributes. Later this XML le is stored to be further used by the ICEPdf viewer while reading annotated eBook (Figure 3). In our mobile based prototype, we have used the Android 2.1 based mobile emulator to implement the mobile version of the haptic eBook reader. In this case we have considered a text le as our candidate reading material. The touch based SDK was readily available to determine the screen coordinate and obtaining paragraph ID of the touched text and this was performed using standard selection operation. We used the Bluetooth Serial Port (SPP) devices with full support for VT-100/ANSI terminal emulation. Even though there is no such direct support in the emulator for Bluetooth we developed a bare-bone reimplementation of the android Bluetooth API where we run a Bluetooth signal processor server (telnet to localhost, port 8199) on our PC to which the emulators will connect through. Instead of using the classes in the package android.bluetooth, we needed to use the classes in the emulated packages. The reimplementation supports easy discovery of the devices, creating Bluetooth services and connecting to Bluetooth services. Now in order to use the simulator we call the BluetoothAdapter.SetContext(this) at some point in our activity/service. In general, the Bluetooth emulator server accepts commands on the form: comID paramname paramvalue . The command format is suitable for sending/receiving Bluetooth signal to the haptic jackets Parani ESD200 Bluetooth kit.

Figure 5: Usability study of the HE-Book system.

4.2 User Study We conducted usability tests to evaluate the users quality of experience with our HE-Book system and to justify the suitability of the proposed approach. The usability tests took place at the university laboratory in a controlled environment with twenty volunteers of different age groups and academic backgrounds. We have selected two groups of users who already had experienced

370

haptic and who did not. The users were requested to use the desktop based prototype and were given certain PDF documents to read which were previously annotated with haptic. Their activity is monitored throughout the experiment and noted for analysis. Afterwards, based on their interaction experience, they were asked to rate six assertions in Likert Scale [16]. The ratings of the assertions were in the range of 1-5 (the higher the rating, the greater is the satisfaction). Figure 5 shows the users responses for each given assertion. Above study considered only one interaction of a user to the prototype system. From the study, we nd that there is signicant number of users who answered neutral or negative to every assertion. But, more than 50% users have shown great interest to HE-Book system. We believe that after some round of tests on the same user set will change their view about the system as many of them were introduced rst time with haptic feedbacks. But, still our user study shows that people have a good tendency of accepting such technology. 5 C ONCLUSION
AND

F UTURE W ORKS

As computers have become more ubiquitous and invisible, we need new interfaces that introduce immersive experience and let users easily move between the physical and digital domains. The HEBook is a novel endeavour to attempt such a transitional interface for experiencing various haptic responses from the traditional eBook materials. The HE-Book supports manual haptic content annotation schemes. Although initial user feedback has been positive, we continue to improve the interface. In the future, we plan on exploring more intuitive ways to present haptic signals from the eBook reading materials rather than touching it. We are also working on automatic annotation process of the eBook text from which it would be possible to determine the appropriate haptic signals for the reading materials. R EFERENCES
[1] Bluecove, http://www.bluecove.org, June 2010. [2] Open source java pdf, http://www.icepdf.org, June 2010. [3] A. Barghout, J. Cha, A. El Saddik, J. Kammerl, and E. Steinbach. Spatial resolution of vibrotactile perception on the human forearm when exploiting funneling illusion. In Haptic Audio visual Environments and Games, 2009. HAVE 2009. IEEE International Workshop on, pages 19 23, 7-8 2009. [4] M. Billinghurst, H. Kato, and I. Poupyrev. The magicbook - moving seamlessly between reality and virtuality. Computer Graphics and Applications, IEEE, 21(3):6 8, 2001. [5] S. Brewster, F. Chohan, and L. Brown. Tactile feedback for mobile interactions. In Proceedings of ACM CHI, pages 159162, San Jose, CA, USA, 2007. ACM Press/Addison-Wesley Publishing Co. [6] S. K. Card, L. Hong, J. Mackinlay, E. H. Chi, and C. H. 3book: A scalable 3d virtual book. In In Extended abstracts of the 2004 conference on Human factors and computing systems (CHI, pages 10951098. ACM Press, 2004. [7] J. Cha, M. Eid, A. Barghout, A. S. M. M. Rahman, and A. El Saddik. Hugme: synchronous haptic teleconferencing. In MM 09: Proceedings of the seventeen ACM international conference on Multimedia, pages 11351136, New York, NY, USA, 2009. ACM. [8] N. Chen, F. Guimbretiere, M. Dixon, C. Lewis, and M. Agrawala. Navigation techniques for dual-display e-book readers. In Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, CHI 08, pages 17791788, New York, NY, USA, 2008. ACM. [9] E. Dale. Audio-Visual Methods in Teaching. Rinehart and Winstor, 1969. [10] R. Grasset, A. Dunser, and M. Billinghurst. The design of a mixedreality book: Is it still a real book? In Mixed and Augmented Reality, 2008. ISMAR 2008. 7th IEEE/ACM International Symposium on, pages 99 102, 2008.

[11] S. Gupta and C. Jaynes. The universal media book: tracking and augmenting moving surfaces with projected information. In Proceedings of the 5th IEEE and ACM International Symposium on Mixed and Augmented Reality, ISMAR 06, pages 177180, Washington, DC, USA, 2006. IEEE Computer Society. [12] A. Haans and W. IJsselsteijn. Mediated social touch: a review of current research and future directions. Virtual Real., 9(2):149159, 2006. [13] B. L. Harrison. E-books and the future of reading. IEEE Comput. Graph. Appl., 20:3239, May 2000. [14] E. Hoggan, S. A. Brewster, and J. Johnston. Investigating the effectiveness of tactile feedback for mobile touchscreens. In CHI 08: Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, pages 15731582, New York, NY, USA, 2008. ACM. [15] L. Hong, E. H. Chi, and S. K. Card. Annotating 3d electronic books. In CHI 05 extended abstracts on Human factors in computing systems, CHI 05, pages 14631466, New York, NY, USA, 2005. ACM. [16] R. Likert. A technique for the measurement of attitudes. Archives of Psychology, 140:155, 1932. [17] A. Mangent. Hypertext ction reading: haptics and immersion. Research in Reading, 31(4):404 419, nov. 2008. [18] T. Park, J. Hwang, and W. Hwang. Facilitating the design of vibration for handheld devices. In J. Jacko, editor, Human-Computer Interaction. Novel Interaction Methods and Techniques, volume 5611 of Lecture Notes in Computer Science, pages 496502. Springer Berlin / Heidelberg, 2009. [19] A. S. M. M. Rahman, J. Cha, and A. El Saddik. Authoring edutainment content through video annotations and 3d model augmentation. In IEEE International Conference on Virtual Environments, Human-Computer Interfaces and Measurement Systems (VECIMS), pages 370374, Hong Kong, China, May 2009. [20] A. S. M. M. Rahman, M. Eid, and A. El Saddik. Kissme: Bringing virtual events to the real world. In Virtual Environments, HumanComputer Interfaces and Measurement Systems, VECIMS 2008, pages 102105, Istanbul, Turkey, July 2008. [21] A. S. M. M. Rahman, S. A. Hossain, and A. El Saddik. Bridging the gap between virtual and real world by bringing an interpersonal haptic communication system in second life. volume 0, pages 228235, Los Alamitos, CA, USA, 2010. IEEE Computer Society. [22] B. Schilit, M. Price, G. Golovchinsky, K. Tanaka, and C. Marshall. The reading appliance revolution. Computer, 32(1):65 73, jan. 1999. [23] G. Welch, A. State, A. Ilie, K.-L. Low, A. Lastra, B. Cairns, H. Towles, H. Fuchs, R. Yang, S. Becker, D. Russo, J. Funaro, and A. van Dam. Immersive electronic books for surgical training. Multimedia, IEEE, 12(3):22 35, 2005.

371