Various Methods for Web and Device Integration

• HTTP Query + CGI
• Plugins
• Ajax
• HTML5 (HTML + JavaScript APIs)
• WebIntents

Method1: HTTP Query String + CGI

Server-side solution

e.g.) http://example.com/tv?channel=xxx&start_time=yyy&end_time=zzz&action=record

Method2: Plugins

Client-side solution using proprietary plugins

Method3: Ajax

Client-Server collaboration without plugins

Method4: HTML5 (HTML + JavaScript APIs)

Client-side solution + Server-side solution

Method5: WebIntents

Inter-application collaboration

Issues with the Existing Methods

Actual mechanism (CGI script, plugins, service APIs, etc.) vary from vendor to vendor, and there is a burden for developers.

• Difficulties with defining APIs for various devices
• Difficulties with integration of distributed services
• Difficulties with dynamic selection of multiple modalities

→ A standard mechanism is required so that people can develop systems regardless their skills.

Why not MMI?

Inter-modality collaboration using standard interfaces

However HTML5 Browsers don't speak MMI :(

Question:

How to integrate various modality components with HTML5-based Web browsers then?

MMI over WebSocket (MoW)!

• A JavaScript library based on MMI Architecture
• makes it easy for Web developers to integrate variety of Web contents and input/output modalities with HTML5-based browsers

The Multimodal Interaction Japanese Chapter (MMI-JP)

• discussing use cases for actual multimodal Web applications based on MMI Architecture
• also tackling "MMI over WebSocket" to implement the library and put it into WebKit

Another Issue on Timing Management

Motivation & Goals

a bit researchy...

• interested in managing utterance timing and speech rate for dialog-based computer interface
• would like to reproduce the natural timing and rhythm of human speech to make computer interface even more friendly and easy-to-use

Issues

• Ordinary OS for PCs or smartphones can't handle presice timing due to multi-task prioritization and interruption :(
• Realtime OS should be used for user terminals to handle pricise timing.
• The whole framework should also handle timing and rhythm.

Possible Solution

• EMMA could be used to synchronize all the Modality Components in the system.
• And there should be an additional event exchange between Modality Components and the Interaction Manager about the "Genesis Timing" for the current session at the beginning of the session.
• This approach should be useful to synchronize more than one modality components in general, e.g., digital TV, recorder and gesture-based controller.

A Possible Use Case: Speech Jukebox

• I proposed a network-based system in 2002, but it was very difficult to implement this at that time :(
• However, it is getting feasible thanks to the Web and MMI Architecture :)
• Probably "Speech Timing Manager" below could be an Interaction Manager and all the other components could be Modality Components.

MMI-based Device Integration

Thanks!