Haptic or “Touch-Enabled” Imagery on the Web

Table of Contents:

I.Applications: Careers for Persons without Vision to Become Guides for Sighted Individuals as: Real-Estate Salespersons, Mount Rushmore National Park Tour Guides, Geography and Cartography Teachers.

II. Lessons and Sensory Limits Learned from Virtual Reality, Terrain Navigation and Medical Simulators.

III.Building the Cognitive Models for “Touch-enabled” or Haptic Image for a non-visual user or an Individual with Autism or Cognitive or Sensory Challenges:

IV.Building the 3-Dimensional Software Models of Crater Lake or Terrain and Topography:

V. Moving the Haptic or “Touch Enabled” Image Models to the Web:

VI. Adding Sound Direction the Sensory and Navigation Limitations of Haptic Navigation of Terrain and other “Touch Enabled” Models:

VII.In Closing:

Appendix A: Pictures of Haptic Model for Crater Lake National Park, Oregon

I.Applications: Careers for Persons without Vision to Become Guides for Sighted Individuals as: Real-Estate Salespersons, Mount Rushmore National Park Tour Guides, Geography and Cartography Teachers.

Why do we need to add haptics or touch-enabled applications to the World Wide Web? This is more than an academic question. Adding full access the imagery of the World Wide Web could provide career, academic, and recreational access for persons with vision impairment, and with cognitive disabilities to the pervasive imagery in the World Wide Web. These opportunities could include careers in areas such as: Salespersons for Real Estate, Guides for National Parks, and Teachers of Geography and Geology.

Additionally, Haptic Touch-Enabled Virtual Reality and Imagery navigation of Web images opens the opportunity for persons with visual and cognitive processing challenges to become free of the perception that they need to be led through images. In fact this would would enable them to become the guides or teachers for their “profoundly visual” (PV) peers. These multi-sensory teachers could teach their “profoundly visual” peers the multi-sensory elements of images which they might normally never consider – such as roughness of terrain on a satellite image, the “feeling” of distance across land and to water on satellite or topographical maps, and even to teach the features on the backside of 3-Dimensional models (such as medical simulations of the heart). This will also enrich the perception of praxis, motion and relative progress while navigating 3-Dimensional terrain maps and other environments for all users.

Virtual Reality – 3-D spatialized audio navigation and feature description, visual navigation and feature discrimination, and praxis or sense of motion.

II. Lessons and Sensory Limits Learned from Virtual Reality, Terrain Navigation and Medical Simulators.

The Benefits of Audio, and Visual Navigation in Virtual Reality Flight Simulators:

In our initial studies we built a “Virtual Reality Blind User and Autism Flight Simulator” which allowed a user with vision impairment, or children with autism to “fly” over the mountains and terrain of a simulated island, towards the directional (or spatialized) audio of a navigation beacon. These experience provided an enriched experience of navigation, the perception of motion and progress while “flying” through the (artificial) 3-Dimensional island, and the perception of praxis or body orientation.

The Virtual Reality Flight Simulator provided: a head tracker to provide the individual user orientation and directional motion in the Virtual world, and directional audio to provide scene orientation and directional navigation for persons with visual impairment.

The VR Flight Simulator and the Island scenario provided some ability to place iconography through the sound of the beacon. Non-visual users were drawn by a human voice saying “This is the northwest corner of the VR world” and which was voiced from a directional beacon to draw nonvisual users and provide audio navigation. The autistic users and cognitively challenged children were drawn by the sound of a (dragon’s stomach) rumbling – which was used to elicit both their attention and directional navigation for children with autism or cognitive challenges.

The island terrain scenario, however, included mountains – which could (not) be perceived by an individual with a single point of sound, emanating a spatialized or directional audio beacon on a mountain top.

This led us to investigate the use of haptics or “touch-enabled” applications with a Virtual Reality portrayal of actual terrain and topography of Crater Lake Oregon, as viewed from space, and projected onto a 3-Dimensional framework.

III.Building the Cognitive Models for “Touch-enabled” or Haptic Image for a non-visual user or an Individual with Autism or Cognitive or Sensory Challenges:

The first step for a non-visual user is to help them form a mental picture with audio descriptions. The next step is to build the context or view of Crater Lake’s rough terrain, flat mesa or table-like surface, depressed gullies and crevasses, and rising mountains, and then plummeting cliffs which cut off sharply down into the massive, round borders and surface of the lake below ( Examine Picture 4 with a Blind Teacher explaining the environment and guiding the hand motions of a six year old girl without vision). (Examine Picture 3 with an individual seated before a PC, holding a pen on a swivel arm which can travel across x, y, and z directions to feel the 3-dimensional features of the mountains, rough plains, and the lake in the center of the Crater Lake Satellite image). (Examine Picture 4, where the point of the pen will press against and follow the terrain of Crater Lake, rising the height of the mountains, sliding across the slipperiness of roads and paths, sinking into the depressions of gullies). (Examine Picture 1 to see the finger icon guided by the haptic pen pressing into and deforming and extending the flexible surface of the Lake).

IV.Building the 3-Dimensional Software Models of Crater Lake or Terrain and Topography:

Create the 3 dimensional map. This process used actual 3 dimensional Earth Views from Space: 3-Dimensional Geography and Topography Images of Crater Lake, Oregon (in Digital Elevation Model or DEM format from the U.S. Geological Survey. These elevation maps were downloaded then force feedback or haptic “touch enabled” model was crafted to demonstrate feature signatures for the haptic pen in the user’s hand to interact with or push against. This included flexible or deformable surfaces whose elastic surfaces flexed for the water of the Lake; hard surfaces for the land, hills and mountains; and slippery surfaces with minimal friction for roads and creek beds. Finally, a colored texture map was added for the Profoundly Visual users.

Integrate the terrain map with the PHANTOM hardware. The 3-dimensional map with the force-feedback or haptic “touch enabled” force model were then navigated by the user, feeling there way across the terrain with the ( Sensable ) PHANTOM haptic pen hardware which utilized 6 degrees of freedom. These 3 dimensional DEM terrain maps were modeled with physical features such as location, mass, friction, and stiffness.

V. Moving the Haptic or “Touch Enabled” Image Models to the Web:

The next development step is to make these models readily available to users with vision impairment, and autism or cognitive and sensory challenges – on the World Wide Web. This can be accomplished in two steps. The first step which has been prototyped already is to convert the DEM Digital Elevation Models to Virtual Reality Markup Language ( VRML ). The next step would be to parse the VRML into the GHOST Software Development Kit (SDK) which is provided by Sensable Corporation, and which can be used with PHANTOM Haptic Pen interface. This would then enable users to access the 3-Dimensional Terrain Models through the Web, and use the PHANTOM Haptic Pen interface to navigate the surface and features of the terrain models.

VI. Adding Sound Direction the Sensory and Navigation Limitations of Haptic Navigation of Terrain and other “Touch Enabled” Models:

Navigating across a 3 dimensional terrain map is not immediately intuitive to individuals with vision challenges or blindness, as reported by our researcher who is non-visual, Mr. Gonzalez. This would raise challenges where a terrain context was not first described in an auditory description.

One of the challenges to a non-visual user (in particular) or an autistic or cognitively or sensorily challenged individual navigating Crater Lake, or any other terrain model is the need for additional pre-recorded or voice synthesized (directive) information (including audio instructions such as “ You are on the rim of the cliffs surrounding Crater Lake, go to your left; you will move into the center of the lake and you will feel Wizard’s Island.”).

Additionally, one could add (self-directed) audio navigation cues – such as placing a directional or spatialized audio beacon into the VRML environment which such as the beacons used in the “Virtual Reality Flight Simulator.” This self-directed beacon could project a voice into the left ear of the user, stating “This is Wizard Island, located in the center of Crater Lake.” The non-visual or cognitively or sensorily challenged individual could then turn their head to the left, or move their hand to the left. The feedback from the haptic pen ( or a head tracker) would reorient the direction of the spatialized audio beacon so that the location information from Wizard Island would be heard directly ahead of the users face and the beacon would grow stronger and louder as the user approached Wizard Island with the Haptic Pen (such as the PHANTOM). This orientation, navigation, and traversal across the terrain is demonstrated with the “hand icon” or cursor which is visibly being pushed across the lake toward Wizard Island and is depicted on the computer monitor as well as to the PHANTOM Haptic Pen (shown in our Crater Lake Haptic Model in (Picture 2)).

VII.In Closing:

Education, science, career and personal independence can be afforded to persons with vision impairment, autism and sensory and communications challenges through applications using virtual reality and haptic “feeling or touch enabled” navigation of 3 dimensional environments – such as terrain. These haptic innovations can also enable and enhance macro and micro research, such as investigations into the composition and interactions of atomic structures, or investigation alongside the Mars Rover, or even the Comet Rider. Use of these new technologies can create new opportunities for Persons with Disabilities in existing and emerging sciences, while opening new sensory exploration for their co-workers and peers.

Appendix A: Pictures (1-4) of Crater Lake National Park, Oregon, USA

Picture 1: Image of a hand reaching down into water, and an index finger pressing into the surface of the lake. This is a satellite picture of Crater Lake Oregon.

Picture 2: Image of a hand hovering in the air over Crater Lake in Oregon.

Picture 3: This is a picture of a man sitting before a computer screen, holding

The haptic pen which is “pushing” a virtual hand down to feel the image of the Lake.

Picture 4: An adult teacher who is blind, teaches a six year old to form a mental image of 3 dimensional Virtual Reality geography through audio descriptions and by guiding her hands. The six year old girl then travels through the Virtual Reality to build the image through other senses of praxis and motion.

The End Or Fin