MBUI for Multimodal Applications


Kazuyuki Ashimura <ashimura@w3.org>
13 May 2010


Visual Web to Multimodal Web

- What is Multimodal Web?

Like UIs in Schi Fi movies ?

(Well, just joking :)

Night Rider


Minority Report minority report

Multimodal Mashup - serious again

GUI as IM for home network

Key technologies for Multimodal Web

  1. MMI Architecture
    • Framework for handling various I/O devices on the Web using transparent and standardized manner
      • Modality component (MC): controller on/for each client device, e.g., HTML, VoiceXML
      • Interaction manager (IM): overall application logic, e.g., SCXML
  2. Life-cycle events
    • Universal API for the communication between the above MC and IM
  3. EMMA

Example of visual MCs

- HTML5 + JavaScript API:

- SVG Map + GPS:

Possibly with distributed semantic information?

MMI and Semantic Web with device info

Difficulty with implementing Multimodal systems

Variety of...

Depending on...

Possible modalities include:

How to design concrete code for each MC?

Several prototype systems, e.g., Speak4it, available

Workshop on Conversational Applications

Model-based UI design approach