Diane Gromala’s experimental book project (2000-present; a semi-finalist for Discover’s award in technological innovation).

“*The Living Book of Senses* is a new media form that extends the traditional book into a radically new sensorially interactive experience.

Users are able to see their physical surroundings while dynamically engaging with three-dimensional mixed realities which appear on their headsets. Users can interact with the book in dynamic ways that go beyond mere clicking and pointing. They can ask the book questions (via voice recognition), and can influence the book through their sensory (bio) feedback. Thus, the book becomes a powerful new sensory experience.

Users wear a headset/head-tracker/color camera system that enables them to see physical reality enhanced with a virtual reality overlay. The camera inputs images/patterns and feeds them back into the ARToolkit software which then displays digital information associated with the physical markers onto the headset. The ARToolkit can calculate camera position and orientation relative to physical markers in real time for video-mediated reality.

Each reader can view AR scenes from their own visual perspective. Users can fly into the immersive world and see each other represented as avatars in the same virtual scene. Readers remaining in the AR scene have a birds’-eye view of other readers as miniature avatars in the virtual scene displayed through their headset. User-controlled dialog with the book elicits responses/answers from the book (expressed in digital data: visual, textual, auditory). As the users simultaneously interact with the book in the physical and virtual realms, the book responds to individual and multiple physical states (via biofeedback) to express resulting changes in narrative. The narrative is a cultural history of the senses. The Living Book is an enhanced learning tool that enables users to become aware of their sensorial experience and bodily states. Collaborations using this book enable distance learning with multiple user interactions. The living book can be used in any narrative-based media to create dynamic communication between any number of people.â€? (From the web site).

Starter Links: Living Book of the Sense