What is "Multimodal Web Application"?

Let's think about car environment as an example

Input & Output Information: Steering Wheel

Input & Output Information: Speed Control

Transmission	Acceleration & Brake
Speed & Engine Revolution

Input & Output Information: Internal Status

Oil Temp., Pressure and Water Temp.

Mobilephone could be a Gateway to the Web

Navigation functionality is also available

e.g. from my home to the airport

Requirements for car environment

We have to concentrate upon driving, so requirements should include not only:

• Graphical display
• GPS information

but also:

• Speech I/O
• Gesture, tactile information capturing
• etc.

Therefore...

Extention of the Web technology is needed which allows

• Application users
  • to dynamically select the most appropriate interaction mode based on the current status
• Application developers
  • to provide an effective user interface for whichever modes the user selects

Various input/output methods should be allowed

• Input
  • speech
  • handwriting
  • keystrokes
  • etc.
• Output
  • displays
  • pre-recorded and synthetic speech
  • audio
  • tactile mechanisms
  • etc.

Multimodal Interaction Activity within the W3C

extends the Web technology to make it possible

• as a central point of coordination for multimodal related specifications
• collaborates with other related Activities:
  • Voice Browser, Compound Document Formats, Ubiquitous Web Applications, etc.

Goal of Multimodal Interaction

• To clearly define how to author concrete multimodal Web applications:
  • especially loose coupling of various modalities
    e.g.
    1. a local HTML and
    2. multiple remote VoiceXML forms/dialogs

Scope of Multimodal Interaction

• Architecture
  • Multimodal Architecture and Interfaces
• Data format
  • Extensible MultiModal Annotation markup language (EMMA)
• Handwriting and graphical capturing
  • Ink Markup Language (InkML)

MMI Architecture

An architecture for authoring multimodal Web applications

Interaction Manager

Core component of the MMI Architecture

• handles all events generated by the Modality Components
• candadates of Interaction Manager:
  • SCXML, CCXML, SMIL, ECMAScript, etc.

A talk on SCXML as the Interaction Manager will be given right after this talk

Modality Components

Controller for various input and output modalities on various devices

• details must be domain- and applicaton-specific

Possible examples:

• Graphical User Interface (HTML)
• Speech Interface (VoiceXML/SSML/SRGS)
• Hand writing and graphical data capturing (InkML)
• etc.

A talk on Speech Interface as a Modality Component will be given right after this talk

InkML

An XML-based markup for exchanging digital ink data either as captured data or software-generated

• dynamic information about pen movement:
  • force (pressure), pen tilt, etc.
• contextual information about capturing devices:
  • sample rate, resolution, uniformity, etc.

InkML Motivation

• growth in pen-enabled devices for mobile use
• allowing informal communication
  • you can communicate using your own hand writing ;-)
• appropriate for private and/or noisy environment
• flexibility and robustness of cross-modality redundancy

InkML Examples

• Hand written notes with text and figures
• Photo images can be tagged by hand writing
• Electronic notebook
• Electronic Form Filling: including signature
• Hand writing with voice input to specify destination: "My home is here"

But what is the concrete & detailed requirement???

• detailed requirement for the MMI Architecture must depend on devices, domains and/or services...
• need to identify and prioritize topics to be considered

A public quesionnaire will be soon issued

to gather requirements on following topics

• Devices
  • What client device(s) should be targeted for multimodal application specification languages?
• I/O methods
  • What input and output methods should be supported by multimodal application specification languages?
• Languages
  • What existing languages should continue to be supported to develop multimodal applications?
  • Should new language be developed to author multimodal applications?
• Reusability
  • How important is the ability to plug-in/reuse the modality components which is already developed?

We need your help

Please join us ;-)

• to add your requirements to the W3C specifications
• to participate in the development of MMI Architecture
• to get good publicity for your organization

providing comments/reviews/implementation reports using the public mailing list www-multimodal@w3.org would be also really appriciated

Thanks!

Some information is available at the W3C site:

• These slides
  • http://www.w3.org/2007/Talks/0511-mmi-ka/
• Resources on MMI Architecture
  • MMI Architecture Working Draft
  • SCXML Working Draft
  • Mike Bodell's slides on SCXML
  • Rafah Hosn's slides on SCXML as Interaction Manager
  • Jerry Carter's slides on Voice modalities
• And more ...
  • The Multimodal Interaction Working Group page
  • The Voice Browser Working Group
  • W3C Mission
  • Membershop Benefits

Related work within W3C

• Speech Interface Framework and VoiceXML 3.0 by The Voice Browser Working Group
• Rich Web Application Backplane by The Hypertext Coordination Group
• Compound Document by Reference Framework, WICD (Web Integration Compound Document) Core 1.0, etc. by The Compound Document Formats Working Group