Voice and Multimodal Interaction Demonstration Session

Introduction
- Scott McGlashan, HP
Demonstrations
- Gerald McCobb, IBM: JK Insurance Demo
- Mikko Honkala, HUT: Speech Enabled XForms
- Dave Raggett, Canon/W3C: Visual and Aural Rendering of RSS new feeds
- Ewald Anderl, Kirusa: Mobile multimodal games
- Scott McGlashan, HP: VoiceXML and video
Panel
- With participants above
- Q+A on demos
- Discuss emerging issues; e.g. why many mobile applications use network components and custom clients rather than web browsers?

Introduction: industry perspective

Backend: ubiquitous web infrastructure in the network: HTTP, XML, web services, ...
Frontend: access to services by increasing variety of devices: desktop, mobile, TVs, fridges, etc.
- Device convergence: TV on desktop; video, TV, GPS on mobile
Desktop service interface: HTML predominates
- Increasingly complemented with Flash, JavaScript, ActiveX, IM, RSS, RTSP, SIP, ...
Mobile service interface: variety of
- ... devices and capabilities: cell phones, smartphones, PDAs
- ... interfaces: voice, SMS, XHTML, native (J2ME, Symbian)
- ... services: different needs when mobile
Emergence of multimodal services: service uses multiple interfaces at the same time

Introduction: multimodal interaction

The MMI working group is tasked with the development of specifications covering the following goals:
- Combination of input modes, including speech, keyboards, pointing devices, touch pads and electronic pens
- Combination of aural and visual output modes
- Combination of local and remote speech recognition services and streaming prerecorded speech and other media
- Local and remote processing of ink entered using a stylus, electronic pen or imaging device
- Varying degrees of coupling between visual and aural interaction modes
- Support the use of a remote voice dialog engine, e.g. a voice gateway running VoiceXML
- Coordination of interaction across more than one device, e.g. cellphone and wall mounted display

Introduction: Voice Browser and Multimodal Activities

Voice Browser
- VoiceXML: interactive voice dialogs
  - SRGS: speech recognition grammar
  - SSML: speech synthesis
  - SISR: semantic interpretation
- Lexicon: pronunciation for recognition and synthesis
- CCXML: call control
Multimodal Interaction:
- Interaction architecture: coordinate modalities
- EMMA: interpretation of modality input
- InkML: format for input entered with an electronic pen or stylus
- Composite input: grammar format for multiple modalities
- Authoring: multimodal authoring model
Convergence: state-based language for control and sequencing of modality components

Analysts - "Open standards programming will become mainstream, focused around VoiceXML"

VoiceXML will dominate the voice environment, due to its flexibility and eventual multimodal capabilities

Source Data Monitor, March 2004

Introduction: Fitting it all together

Web browser enhanced with VB/MMI capabilities: e.g.
- Adding speech capabilities (SRGS, SSML)
Coordinating web browser with voice browser: e.g.
- Web browser and voice browser on device
- Web browser on device, voice browser in network
Coordinating multiple browsers using emerging state-based language: e.g.
- Web browser triggers conference call browser and video browser display on TV

Demonstrations: People at Work and Play

JK Insurance Demo	Gerald McCobb (IBM)
Voice Enabling Forms	Mikko Honkala (HUT)
Visual and Aural rendering of RSS	Dave Raggett (Canon/W3C)
Mobile Multmodal Games	Ewald Anderl (Kirusa)
VoiceXML and Video	Scott McGlashan (HP)

JK Insurance Demo

The JK Insurance Enterprise
- Support for voice and multimodal access
- Synchronization of PIM, e-mail, and business application data
- Intelligent notification and instant messaging
- Server initiated actions
- Offline applications
Mobile Adjuster offline application
- Synchronizes a device with back-end server that contains customer information, filed claims, and parts for damage computation
- While on road adjuster processes claims on device
- When connected claims are synchronized with server

JK Insurance Enterprise Architecture

complex diagram

Mobile Adjuster submits new claim

The submitted claim is added to the embedded DB2e database. Additionally, the claim data is placed into an MQe message and sent to a Claims Processing Server, which listens for MQe messages and responds with claim status updates. The MQe message is queued while offline, and is transmitted when the device has an Internet connection.

complex diagram

JK Insurance Demo

Demo Technology
- HP iPAQ 5555
- X+V 1.2 for Multimodal interaction
  http://www.voicexml.org/specs/multimodal/x+v/12/
- Access NetFront Browser
- IBM Workplace Client Technology Micro Edition (WCTME)
Demo location: http://wea5.dfw.ibm.com/wps/portal

Speech Enabled XForms

Mikko Honkala

Helsinki University of Technology

Speech Enabled XForms

Introduction
- Motivation: XForms as the basis for multimodal user interfaces
Implementation
- Requirements
- Tools used
- Design
Demo
Conclusions

Motivation: XForms as the basis for multimodal user interfaces

Web is a channel for distributing the user interface to the client
In Web, the device accessing the application is not known
Problem: modality-dependent languages: VoiceXML, HTML
- Same UI must be defined again for each modality
- Ad-hoc modality changes not possible
Problem: HTML forms is more device-independent, but has severe problems
- Accessibility, Scripting, Mixing of presentation and content, abstraction level
A possible solution: raise the abstraction level of the UI definition language
- XForms does this
- Single XForms UI definition could be used with different modalities

XForms

Next generation web forms
Separates purpose and presentation
- Form can be accessed through several UIs
Important functionality for speech UIs
- Datatype association
- Dynamic forms (no round-trips, focus and UI state preserved)
- Higher level of abstraction: declarative language for UIs

Requirements

Dialog based navigation and form filling of XForms
Interaction with different types of form controls
- Selection lists
- Numbers, booleans, dates
- Pop-up messages, alerts

Implementation: Tools used

Speech recognition
- CMU Sphinx-4
- Pure-java ASR, open source
Speech synthesis
- FreeTTS
- Pure-java Text-To-Speech synthesizer, open source
Browser and XForms implementation
- X-Smiles from Helsinki University Of Technology
- Pure-java XML Browser supporting XForms and CSS, open source

Design

Speech implementation works on top of the XForms DOM
FocusManager
- Holds the current focus point
- Searches the dynamic DOM for focus points
Speech Widgets
- Generate speech questions and interpret the response

Conclusions

Current web technologies make applications usable only with one modality (speech vs. graphical)
Raising the abstraction level helps in delivering the same UI to different modalities
In this project XForms was speech-enabled
Idea seems to work
Pure java speech and Web tools are quite usable with modern computers
Future work
- Support for CSS speech properties
- Binding to custom speech form controls written in Voice XML

Visual and Aural Rendering of RSS News Feeds

Dave Raggett, W3C/Canon

RSS

Popular XML format for syndicating news
- BBC Worlds News
- Reuters
- CNN
- W3C
- etc.
RSS file lists details of a set of news items
- Title, Description and Link for each news item
- Title, Description and Link for the feed itself

Multimodal Rendering

Visual
- Widely supported by current Web browsers
- XHTML + CSS + Graphics + Scripting
Aural
- SSML: Speech Synthesis Markup Language
- Commonly used with VoiceXML for applications accessed via a telephone
- New separately installable extension to Firefox 1.0 Web browser that adds support for Speech Synthesis
- Demo uses Loquendo TTS for speech engine

Say It! Browser Extension

XPCom component written in C++
Exposes interface for speech synthesis
- Play(SSML), Pause(), Resume(), Stop()
Acts as client to TTS server written in C
- TTS Server supports request queue and works with Loquendo TTS
Can also use Say It! with open source Festival TTS server
- You send scheme commands to drive Festival TTS

XSLT Stylesheets

XSLT is an XML language for describing rules for transforming one XML format into another
Can be run on browser or server or both
I am using XSLT for styling
- Visual:
  - rss2html.xsl maps RSS into XHTML
- Aural
  - rss2ssml.xsl maps RSS into SSML
    - Demo has choice of three different aural style sheets

Goal

Support W3C Multimodal Interaction WG through experimental implementations
Next steps:
- Updating Mozilla to support CSS3 Speech
- CSS properties that are equivalent to XSLT & SSML
- Voice-family, voice-volume, voice-rate, ...
- Replacement for older ACSS properties
- Further extensions for Speech and Pen input
Hope to demo these in next year's Tech Plenary

Extensions will be available under W3C's open source software license

Credits

Thanks for Loquendo for use of their text to speech engine
- http://www.loquendo.com
Thanks to Johny Stenback, David Baron and Ian Hickson for information on working with Mozilla
Thanks to Max Froumentin, T.V. Raman and others for the idea of rendering RSS to speech
And now for the demo itself ....

Mobile Multimodal Games

Developed in partnership with France Telecom
Investigating novel Multimodal interaction ... new types of game experiences possible
- Timed (single player) - find all of the hidden words before the clock runs out
- Multiple categories; e.g. Sports, Animals, Countries
- Multiple levels: Beginner, Intermediate, Advanced
- The words arranged horizontal, vertical and diagonal, both left to right and right to left
Target mass market mobile devices
- Microsoft Smartphone, Symbian, etc.
- ASR & TTS resources Network Based
- Selective local audio resources
Multimodal Interactions - user has the choice to mix and match
- Visual selection of a hidden word with the joystick
- Vocal selection of a hidden word
- Feedback is both visual and audio

Multimodal Games — Multiple Standards in play

Web Based Game (Application)
HTML, XHTML, HTTP
- Extensive use of scripting on the device
VoiceXML 2.0
- Full use of VoiceXML available to developer
- Not constrained by device limitations
XML Grammars
XSLT
SIP
RTP

VoiceXML and video: why

Mobile handsets (e.g. 3G) with video capability
- Handset approach: upload/download video clips
- Network approach: video streaming with enhanced services (e.g. transcode, logo/banner insertion)
Video applications
- Video mail; including avatars
- Video portal: infotainment, gaming, instructions, ?
- Video conferencing ?
How to author video applications?
- SMIL: presentation containing audio and video clips
- VoiceXML?

VoiceXML and video: how

Video applications using VoiceXML 2.0 and browser supporting video media types:
- video playback: <audio src="myvideo.3gp"/>
- video recording: <record type="video/3gpp"/>
Video as a natural development of VoiceXML qua interactive media language
- Video interaction with all the benefits of VoiceXML (e.g. speech recognition, synthesis, DTMF, etc )
- Use existing VoiceXML application tools to build video services

VoiceXML and Video Demo

complex diagram

Panel Discussion

Participants
- Ewald Anderl, Kirusa
- Mikko Honkala, HUT
- Gerald McCobb, IBM
- Scott McGlashan, HP (moderator)
- Dave Raggett, Canon/W3C
Questions:
- Audience questions on demos
- why many mobile applications use network components and custom clients rather than web browsers?