Multimodal Interaction Activity


Max Froumentin

This talk is aimed at explaining what multimodal interaction is, how the W3C is standardising in that area, and what has happened since last year. A lot has happened actually, since it had just started last year.


Intro::What's MMI?

t68i phone p800 phone Nokia Phone Sony PDA


Described in Use Cases document.


a very complex problem...

... that's why there is quite a big working group...

Intro::The MMI Working Group

The MMI Framework

MMI Framework


input framework


output framework

Ongoing Work

Last year, the WG had just started and concentrated on the framework. Now more specific areas are worked on:

Ongoing Work::The Object Model

Going down one level, it is necessary to specify interfaces between components. The whole framework can be seen as a distributed DOM, or it can be seen as components passing messages between each other, in the form of markup.

Distributed DOM

Distributed DOM

Framework::The Object Model (cont'd)

Or message passing infrastructure

Message Passing

Ongoing Work::I/O Components

VIO - Requirements

Ink object, GUI object, etc.

Ongoing Work::Markup

Ongoing Work::Markup::Ink

Markup for pen-based input devices

     <channel name="X" type="decimal">
     <channel name="Y" type="decimal">
     <channel name="S1" type="boolean" default="F"/>
     <channel name="S2" type="boolean" default="F"/>

<trace id = "4525BCD">
1125 18432'23'43"7"-8 3-5+7  -3+6+2+6 8+3+6:T;+2+4:*T;+3+6+3-6:FF;

Ongoing Work::Markup::EMMA

Markup for input annotations

<emma:emma emma:version="1.0"
  <emma:one-of rdf:ID="r1">
    <emma:interpretation rdf:ID="int1">
    <emma:interpretation rdf:ID="int2">

    <!-- time stamps for date in first interpretation -->
    <rdf:Description rdf:about="#xpointer(id('int1')/date)" 


Conclusion::More Info