Visual Web to Multimodal Web

- What is Multimodal Web?

• Not only usual GUI but also various I/O modalities like voice, gesture, etc.

e.g.) AT&T's Speak4it (demonstration on YouTube)

Like UIs in Schi Fi movies ?

(Well, just joking :)

Night Rider

Minority Report

Multimodal Mashup - serious again

Key technologies for Multimodal Web

1. MMI Architecture
   • Framework for handling various I/O devices on the Web using transparent and standardized manner
     • Modality component (MC): controller on/for each client device, e.g., HTML, VoiceXML
     • Interaction manager (IM): overall application logic, e.g., SCXML
2. Life-cycle events
   • Universal API for the communication between the above MC and IM
3. EMMA
   • Standard data format for Multimoda Web applications
   • Originally developed for speech recognition result
   • Being extended to device output and semantic information

Example of visual MCs

- HTML5 + JavaScript API:

• Web browsers
  • Getting platform for Web Applications: Canvas, Storage, Socket, etc.
  • Multimedia data: audio, video, etc.

- SVG Map + GPS:

• Navigation system on mobile phones

Possibly with distributed semantic information?

Difficulty with implementing Multimodal systems

Variety of...

• Input/Output methods and Natural language processing

Depending on...

• Device capability and user's preference

Possible modalities include:

• GUI, Voice, Gesture, Handwriting, printing
• Appliances, e.g., digital TV, video cam
• Sensors, e.g., GPS, thermometer, pressure meter

How to design concrete code for each MC?

Several prototype systems, e.g., Speak4it, available

• However, still need some more discussion about how to integrate MCs

Workshop on Conversational Applications

• Discussion on expected processing model for Conversational Multimodal Applications

Model-based UI design approach

• How could Model-based approach, e.g., MyMobileWeb, be applied to Multimodal systems?

Thanks!