Multimodal Interaction Working Group Charter

Scope

Background

The primary goal of this group is to develop W3C Recommendations that enable multimodal interaction with various devices including desktop PCs, mobile phones and less traditional platforms such as cars and intelligent home environments including digital TVs/connected TVs. The standards should be scalable to enable richer capabilities for subsequent generations of multimodal devices.

Users will be able to provide input via speech, handwriting, motion or keystrokes, with output presented via displays, pre-recorded and synthetic speech, audio, and tactile mechanisms such as mobile phone vibrators and Braille strips. Application developers will be able to provide an effective user interface for whichever modes the user selects. To encourage rapid adoption, the same content can be designed for use on both old and new devices. Multimodal access capabilities depend on the devices used. For example, users of multimodal devices which include not only keypads but also touch panel, microphone and motion sensor can enjoy all the possible modalities, while users of devices with restricted capability prefer simpler and lighter modalities like keypads and voice.Extensibility to new modes of interaction is important, especially considered in the context of the current proliferation of types of sensor input, for example, in biometric identification and medical applications.

Motivation

As technology evolves the ability to process modalities grows. The office computer is an example. From punch cards, to keyboards to mice and touch screens, the machine has seen a variety of modalities come and go. The mediums that enable interaction with a piece of technology continue to expand as increasingly sophisticated hardware is introduced into computing devices. The medium mode relationship is one of many to many. As mediums are introduced into a system, they may introduce new modes or leverage existing ones. For example: a system may have a tactile medium of touch and a visual medium of eye tracking. These two mediums may share the same mode, such as ink. Devices increasingly expand the ability to weave mediums and modes together. Modality components do not necessarily have to be devices. Taking into consideration the increasing pervasiveness of computing, the need to handle diverse sets of interactions introduces a need for a scalable multimodal architecture that allows for rich human interaction. Systems currently leveraging the multimodal architecture include TV, healthcare, automotive technology, heads up displays and personal assistants. MMI enables interactive, interoperable, smart systems.

MMI Ecosystem Example

The diagram below shows an example of an ecosystem of interacting Multimodal Architecture components. In this example, components are located in the home, the car, a smartphone, and the Web. User interaction with these components is mediated by devices such as smartphones or wearables. The MMI Architecture, along with EMMA, provides a layer that virtualizes user interaction on top of generic Internet protocols such as WebSockets and HTTP, or on top of more specific protocols such as ECHONET.

Example of Multimodal Interaction Ecosystem

As an example, a user in this ecosystem might be driving home in the car, and want to start their rice cooker.The user could say something like "start the rice cooker" or "I want to turn on the rice cooker". MMI Life Cycle events flowing among the smartphone, the cloud and the home network result in a command being sent to the rice cooker to turn on. To confirm that the rice cooker has turned on, voice output would be selected as the best feedback mechanism for a user who is driving. Similar scenarios could be constructed for interaction with a wide variety of other devices in the home, the car, or in other environments.

Note that precise synchronization of all the related entities is becoming more and more important, because network connection is getting very fast and there is a possibility of monitoring sensors on the other side of the world or even on an artificial satellite.

Possible Architecture of Advanced MMI-based Web Applications

The diagram below shows how the MMI-based user interface relates to other Web technologies including the following layers using various communication mechanisms, e.g., MMI Events, Indie UI Events, Touch Events and Web APIs:

• Presentation
• Application
• Device
• Session/Transport

Note that MMI is a generic mechanism to handle event exchange for advanced Web applications with distributed multiple entities.

Layer Cake of MMI-based Web Applications

Target Audience

The target audience of the Multimodal Interaction Working Group are vendors and service providers of multimodal applications, and should include a range of organizations in different industry sectors like:

Mobile and hand-held devices

As a result of increasingly capable networks, devices, and speech recognition technology, the number of existing multimodal applications, especially mobile applications, is rapidly accelerating.

Home appliances, e.g., TV, and home networks

Multimodal interfaces are expected to add value to remote control of home entertainment systems, as well as finding a role for other systems around the home. Companies involved in developing embedded systems and consumer electronics should be interested in W3C's work on multimodal interaction.

Enterprise office applications and devices

Multimodal has benefits for desktops, wall mounted interactive displays, multi-function copiers and other office equipment which offer a richer user experience and the chance to use additional modalities like speech and pens to existing modalities like keyboards and mice. W3C's standardization work in this area should be of interest to companies developing client software and application authoring technologies, and who wish to ensure that the resulting standards live up to their needs.

Intelligent IT ready cars

With the emergence of dashboard integrated high resolution color displays for navigation, communication and entertainment services, W3C's work on open standards for multimodal interaction should be of interest to companies working on developing the next generation of in-car systems.

Medical applications

Mobile healthcare professionals and practitioners of telemedicine will benefit from multimodal standards for interactions with remote patients as well as for collaboration with distant colleagues.

Speech is a very practical way for those industry sectors to interact with smaller devices, allowing one-handed and hands-free operation. Users benefit from being able to choose which modalities they find convenient in any situation. General purpose personal assistant applications such Apple Siri, Google Now, Google Glass, Anboto Sherpa, XOWi, Speaktoit for Android, Indigo for Android and Cluzee for Android are becoming widespread. These applications feature interfaces that include speech input and output, graphical input and output, natural language understanding, and back-end integration with other software. More recently, enterprise personal assistant platforms such as Nuance Nina, Angel Lexee and Openstream Cue-me have also been introduced. In addition to multimodal interfaces, enterprise personal assistant applications frequently include deep integration with enterprise functionality. Personal Assistant functionality is merging with earlier Multimodal Voice Search applications which have been implemented in applications by a number of companies, including Google, Microsoft, Yahoo, Novauris, AT&T, Vocalia, and Sound Hound. The Working Group should be of interest to companies developing smart phones and personal digital assistants or who are interested in providing tools and technology to support the delivery of multimodal services to such devices.

Work to do

To solicit industrial expertise for the expected multimodal interaction ecosystem, the group held the following two Webinars and a Workshop:

• W3C Webinar on "Developing Portable Mobile Applications with Compelling User Experience using the W3C MMI Architecture" on 31 January 2014
• W3C Workshop on "Rich Multimodal Application Development" on 22-23 July 2013
• W3C Webinar on "Discovery in Distributed Multimodal Interaction" on 24 September 2013

Those Webinars and Workshop were aimed at Web developers who may find it daunting to incorporate innovative input and output methods such as speech, touch, gesture and swipe into their applications, given the diversity of devices and programming techniques available today. During those events, we had great discussion on rich multimodal Web applications, and clarified many new use cases and requirements for enhanced multimodal user-experiences, e.g., distributed/dynamic applications depend on the ability of devices and environments to find each other and learn what modalities they support. Target industry areas include health care, financial services, broadcasting, automotive, gaming and consumer devices. Example use cases include:

• Advanced TV Applications/Services:
  • Authenticated, authorized users can watch TV from any device
  • Multiple devices in sync with each other, delivery mechanism for supplementary content
  • Invoking personalized TV home screen using facial recognition, customized program guide, content, applications
• Appliances and Cars:
  • Appliances connected to the cloud, e.g., rice cookers, fridges, microwaves and medical devices
  • Devices connected using NFC enabled smartphones
  • Cars connected to the cloud and road infrastructure giving traffic information, recommendations and music
• Interaction with Intelligent Sensors and Devices:
  • Devices dynamically become part of a group in the workplace
  • Sensor/camera input — integration with input from other modalities
  • Voice enabled personal assistant
  • Browsers take multiple/various inputs — GUI, gesture and sensor input

The main target of "Multimodal Interaction" used to be the interaction between computer systems and human, so the group have been working on human-machine interfaces, e.g., GUI, speech and touch. However, after holding the above events, it was clarified that MMI Architecture and EMMA are useful for not only human-machine interface but also integration with machine input/output. Therefore there is a need to consider how to integrate existing non Web-based embedded systems and Web-based systems, and the group would like to provide a solution to deal with the issue by tackling the following standardization work:

Multimodal Architecture and Interfaces (MMI Architecture)

During the previous charter period the group brought the Multimodal Architecture and Interfaces (MMI Architecture) specification to Recommendation. The group will continue maintaining the specification.

The group may expand the Multimodal Architecture and Interfaces specification by including the description of Modality Components Registration & Discovery Working Group Note so that the specification would fit the need of industory sectors.

Modality Components Registration & Discovery

The group has been investigating how various Modality Components in the MMI Architecture which are responsible for controlling the various input and output modalities on various devices can register their capabilities and existence with the Interaction Manager. The possible examples of registration information of Modality Components include capabilities of Modality Components such as Languages supported, ink capture and playback, media capture and playback (audio, video, images, sensor data), speech recognition, text to speech, SVG, geolocation and emotion.

The group's proposal for the description of multimodal properties focuses on the use of semantic Web services technologies over well known network layers like UPnP, DLNA, Zeroconf or the HTTP protocol. Intended to be an extension of the current MMI Architecture and Interfaces recommendation, the goal is to enhance the discovery of services provided by Modality Components on devices using well defined descriptions from a multimodal perspective. We believe that this approach will enhance the autonomous behavior of multimodal applications, providing a robust perception of the user-system needs and exchanges, and helping control the semantic integration of the human-computer interaction from a context and functioning environment perspective. As a result, the challenge of discovery is addressed using description tools provided by the field of web services, like WSDL 2.0 documents annotated with semantic data (SAWSDL) in owl or owl-s languages using the modelReference attribute. This extension adds functionalities for the discovering and registration of the Modality Components by using semantic WSDL descriptions of the contextual preconditions (QoS, price and other non functional information), processes and results provided by these components. These documents, following the WSDL 2.0 structure can either be expressed in xml format or in the lighter Json or Yaml formats.

The group generated a Working Group Note on Registration & Discovery, and based on that document the group will generate a document which describes the detailed definition of the format and process of Registration of Modality Components as well as the process for the Interaction Manager to discover the existence and current capabilities of Modality Components. This document is expected to become a recommendation track document.

Multimodal Authoring

The group will investigate and recommend how various W3C languages can be extended for use in a multimodal environment using the multimodal Life-Cycle Events. The group may prepare W3C Notes on how the following languages can participate in multimodal specifications by incorporating the life cycle events from the multimodal architecture: XHTML, VoiceXML, MathML, SMIL, SVG, InkML and other languages that can be used in a multimodal environment. The working group is also interested in investigating how CSS and Delivery Context: Client Interfaces (DCCI) can be used to support Multimodal Interaction applications, and if appropriate, may write a W3C Note.

Extensible Multi-Modal Annotations (EMMA) 2.0

EMMA is a data exchange format for the interface between different levels of input processing and interaction management in multimodal and voice-enabled systems. Whereas the MMI Architecture defines communication among components of multimodal systems, including distributed and dynamic systems, EMMA provides information used in communications with users. For example, EMMA provides the means for input processing components, such as speech recognizers, to annotate application specific data with information such as confidence scores, time stamps, and input mode classification (e.g. key strokes, touch, speech, or pen). It also provides mechanisms for representing alternative recognition hypotheses including lattice and groups and sequences of inputs. EMMA supersedes earlier work on the natural language semantics markup language (NLSML) in the Voice Browser Activity.

During the previous charter periods, the group brought EMMA Version 1.0 to Recommendation. Also the group generated Working Drafts of the EMMA Version 1.1 specification based on the feedback from the MMI Webinars and the MMI workshop in 2013.

In the period defined by this charter, the group will publish a new EMMA 2.0 version of the specification which incorporates new features that address issues brought up during the development of EMMA 1.1. The group won't do any more work on EMMA Version 1.1, but the work done on EMMA 1.1 will be folded into EMMA Version 2.0.

Note that the original EMMA 1.0 specification focused on representing multimodal user inputs, and EMMA 2.0 will add representation of multimodal outputs directed to users.

New features for EMMA 2.0 includes but not limited to the following:

• Broader range of semantic representation formats, e.g., JSON and RDF
• Handling machine output
• Incremental recognition results, e.g., speech and gesture

Semantic Interpretation for Multimodal Interaction

During the MMI Webinars and the MMI workshop, there was discussion on possible mechanism to handle semantic interpretation for multimodal interaction, e.g., gesture recognition, like the one for speech recognition, i.e., Semantic Interpretation for Speech Recognition (SISR).

The group will start to see how to apply SISR to semantic interpretation for non-speech Modality Components, and discuss concrete use cases and requirements. However, the group will not generate a new version of SISR specification.

Maintenance work

The group will be maintaining the following specifications:

• EMMA 1.0
• InkML
• MMI Architecture
• EmotionML

The group will also consider publishing additional versions of the specifications depending on feedback from the user commuinty and industry sectors.

Success Criteria

For each document to advance to proposed Recommendation, the group will produce a technical report with at least two independent and interoperable implementations for each required feature. The working group anticipates two interoperable implementations for each optional feature but may reduce the criteria for specific optional features.