Registration & Discovery of Multimodal Modality Components in Multimodal Systems: Use Cases and Requirements

Abstract

This document addresses people who want either to develop Modality Components for Applications that communicate with the user through different modalities such as voice, gesture, or handwriting, and/or to distribute them through a multimodal system using multi-biometric elements, multimodal interfaces or multi-sensor recognizers over a local network or "in the cloud". With this goal, this document collects a number of use cases together with their goals and requirements for describing, publishing, discovering, registering and subscribing to Modality Components in a system implemented according the Multimodal Architecture Specification. In this way, Modality Components can be used by automated tools to power advanced Services such as : more accurate searches based on modality, behavior recognition for a better interaction with intelligent software agents and an enhanced knowledge management achieved by the means of capturing and producing emotional data.

Status of This Document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

This is the 5 July 2012 W3C Working Group Note of "Registration & Discovery of Multimodal Modality Components in Multimodal Systems: Use Cases and Requirements". This W3C Working Group Note has been developed by the Multimodal Interaction Working Group of the W3C Multimodal Interaction Activity.

This document was published by the Multimodal Interaction Working Group as a Working Group Note. If you wish to make comments regarding this document, please send them to www-multimodal@w3.org (subscribe, archives). All comments are welcomed and should have a subject starting with the prefix '[dis]'.

Publication as a Working Group Note does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

This document was produced by a group operating under the 5 February 2004 W3C Patent Policy. W3C maintains a public list of any patent disclosures made in connection with the deliverables of the group; that page also includes instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy.

Table of Contents

• 1. Introduction
• 2. Domain Vocabulary
• 3. Use Cases
• 3.1. Use Cases SET 1 : Smart Homes
• 3.2. Use Cases SET 2 : Personal Externalized Interfaces
• 3.3. Use Cases SET 3 : Public Spaces
• 3.4. Use Cases SET 4 : Health Sensing
• 3.5. Use Cases SET 5 : Synergistic Interaction and Recognition
• 4. Requirements
• 4.1 Distribution
• 4.2 Advertisement
• 4.3 Discovery
• 4.4 Registration
• 4.5 Querying
• 4.6 Open Issues
• 5. Related Work (Informative)
• A. References

1. Introduction

User interaction with Internet Applications on mobile phones, personal computers, tablets or other electronic Devices is moving towards a multi-mode environment in which important parts of the interaction are supported in multiple ways. This heterogeneity is driven by Applications that compete to enrich the user experience in accessing all kinds of services. More and more, Applications need interaction variety, which has been proven to provide numerous concurrent advantages. At the same time, it brings new challenges in multimodal integration, which is often quite difficult to handle in a context with multiple networks and input/output resources.

Today, users, vendors, operators and broadcasters can produce and use all kinds of different Media and Devices that are capable of supporting multiple modes of input or output. In this context, tools for authoring, edition or distribution of Media for Application developers are well documented. Mature proprietary / open source tools and Services that handle, capture, present, play, annotate or recognize Media in multiple modes are available. Nevertheless, there is a lack of powerful tools or practices for a richer integration and semantic synchronization of all these media.

To the best of our knowledge, there is no standardized way to build a web Application that can dynamically combine and control discovered modalities by querying a registry based on user-experience data and modality states. This document describes design requirements that the Multimodal Architecture and Interfaces specification needs to cover in order to address this problem.

2. Domain Vocabulary

Interaction Context

For the purposes of this document an Interaction Context represents a single exchange between a system and one or multiple users across one or multiple interaction modes and covers the longest period of communication over which it would make sense for components to keep the information available. It can be as simple as a single period of audio visual content (e.g. a program ), a phone call or a web session. But it can be also a richer interaction combining for example, voice , gesture and a direct interaction with a light pointer or a shared whiteboard with an associated VoIP call that during the interaction evolves to a text chat. In these cases, a single context persists across various modality configurations. For more details see the multimodal context in the MMI architecture.

Multimodal System

For the purposes of this document a Multimodal System is any system communicating with the user through different modalities such as voice, gesture, or handwriting in the same interaction cycle identified by an unique context. In a Multimodal System the Application or the final user can dynamically switch modalities in the same context of information exchange. This is a bi-directional system with combined inputs and outputs in multiple sensorial modes (e.g. visual, acoustic, haptic, olfactive, gustative) and modalities (e.g. voice, gesture, handwriting, biometrics capture, temperature sensing, etc). This is also a system in which input and output data can be integrated [ See Fusion ] or dissociated [ See Fission ] in order to identify the meaning of the user's behavior or in order to compose a more adapted, relevant and pertinent returning message using multiple media, modes and modalities. [ See a Multimodal System Example ]

A Multimodal System can be any Infrastructure ( or any IaaS ), any Platform ( or any PaaS ) or any Software ( on any SaaS ) implementing human-centered multimodal communication. For more information : [ See the basic component's Description for a Multimodal Interaction Framework ]

Modality Component

For the purpose of this document, a modality is a term that covers the way an idea could be communicated or the manner an action could be performed. In some mobile multimodal systems, e.g. the primary modality is speech and an additional modality can be typically gesture, gaze, sketch, or any combination thereof. These are forms of representing information in a known and recognizable logical structure. For example, acoustic data can be expressed as a musical sound modality (e.g. a human singing) or as a speech modality (e.g. a human talking). Following this idea, in this document a Modality Component is a logic entity that handles the input and output of different hardware Devices (e.g. microphone, graphic tablet, keyboard) or software Services (e.g. motion detection, biometric changes) associated with the Multimodal System. Modality Components are responsible for specific tasks, including handling inputs and outputs in various ways, such as speech, writing, video, etc.. Modality Components are also loosely coupled software modules that may be either co-resident on a device or distributed across a network.

For example, a Modality Component can be charged at the same time of the speech recognition and the sound input management (i.e. some advanced signal treatment task like source separation). Another Modality Component can manage the complementary command inputs on two different devices: a graphics tablet and a microphone. Two modality components, can manage separately two complementary inputs given by a single device: a camcorder. Or finally, a Modality Component, can use an external recognition web service and only be responsible for the control of communication exchanges needed for the recognition task. [ See : this Modality Component Example ]

In all four cases the system has a generic Modality Component for the detection of a voice command input, despite the differences of implementation. Any Modality Component can potentially wrap multiple features provided by multiple physical Devices but also more than one Modality Component could be included in a single device. To this extent the Modality Component is an abstraction of the same kind of input handled and implemented differently in each case. For more information : [ See the Multimodal Interaction Framework input/output Description ]

Interaction Manager

For the purposes of this document, the Interaction Manager is also a logical component handling the multimodal integration and composition. It is responsible for all message exchanges between the components of the Multimodal System and the hosting runtime framework [ See : Architecture Components ]. This is a communication bus and also an event handler. Each Application can configure at least one Interaction Manager to define the required interaction logic. This is a controller at the core of all the multimodal interaction :

• It manages the specific behaviors triggered by the events exchanged between the various input and output components.
• It manages the communication between the modules and the client application.
• It ensures consistency between multiple inputs and outputs and provides a general perception of the application's current status.
• It is responsible for data synchronization.
• It is responsible for focus management.
• It manages communication with any other entity outside the system.

For more information : [ See the Multimodal Interaction Framework Description of an Interaction Manager ]

Data Component

For the purposes of this document a Data Component is a logic entity that stores the public and private data of any module in a Multimodal System. The data component's primary role is to save the public data that may be required by one or several Modality Components or by other modules (eg, a session component in the hosting framework ). The Data Component can be an internal module or an external module [ See : Example ]. This depends on the implementation chosen by each application. However, the Interaction Manager is the only module that has direct access to the Data Component and only Interaction Manager can view and edit the data and communicate with external servers if necessary. As a result, the Modality Components must use an Interaction Manager as a mediator to access any private or public data in the case of an implementation following the principle of nested dolls given by the MMI Architecture Specification [ See : MMI Recursion ]. For the storage of private data, each Modality Component can implement its own Data Component. This private Data Component accesses external servers and keeps the data that the Modality Components may require, for example, in the speech recognition task. For more information : [ See the Multimodal Interaction Framework Description of an Session Component ]

Application

For the purposes of this document, the term Application refers to a collection of events, components and resources which use server-side or client-side processing and the Multimodal Architecture Specification to provide sensorial, cognitive and emotional information [ See : Emotion Use Cases ] through a rich multimodal user experience. For example, a multimodal Application can be implemented to use in an integrated way, mobile Devices and cell phones, home appliances, Internet of Things objects, television and home networks, enterprise applications, web applications, "smart" cars or medical devices.

Service

For the purposes of this document, a Service is a set of functionalities associated with a process or system that performs a task and is wrapped in a Modality Component abstraction. While most of the industrial approaches for s ervice discovery use hardware Devices as networked Services and the majority of web services discovery approaches consider a Service as a software component performing a specific functionality, our definition covers both views and focuses on the Multimodal Component abstraction. Thus, a multimodal Service is any functionality wrapped in a Multimodal Component (e.g. a Modality Component or the Interaction Manager) and using one or multiple devices, device's services, Devices APIs or web services. For example, in the case of a Modality Component a Service is a functionality provided to handle input or output interaction in one or more devices, sensors, effectors, players (e.g. for virtual reality Media display), on-demand SaaS Recognizers (e.g. for natural language recognition) and on-demand SaaS User Interface Widgets (e.g. for geolocation display).

For the purposes of this document, we will use the term Service Description as a set of attributes (metadata) describing a particular service. The term Service Advertisement refers to the publication of the metadata and the Service itself by indexing the Service in some registry and making available the Service to the client's requests.

Device

For the purposes of this document, a Device is any hardware material resource wrapped in a Modality Component. The MMI Architecture does not describe how Modality Components are allocated to hardware devices, this is dependent on the implementation. A Device can act as an input sensor. In this case, for example, cameras, haptic devices, microphones, biometric devices, keyboards, mouse and writing tablets; are devices that provide input Services wrapped in one or many Modality Components. A Device can also act as an output effector. In this case displays, speakers, pressure applicators, projectors, Media players, vibrating Devices and even some sensor Devices (galvanic skins, switches, motion platforms) can provide effector Services wrapped in one or many Modality Components implemented in the Multimodal System.

Media

For the purposes of this document, the multimodal Media are resources depending on the semantics of the message, the user/author goals and the capabilities of the support itself. The multimodal Media adhere to a certain content mode (e.g. visual) and modality (e.g. animated image) and can be a document, a stream, a set of data or any other conventional logical entity to represent the information that is communicated through devices, services and networks. In a Multimodal System, Media can be played, displayed, recognized, touched, scented, heard, shaken, pushed, shared, adapted, fused, composed, etc.. and annotated to enhance integration, recognition, composition and interpretation [ See the use cases for MultiModal Media Annotation with EMMA ].

Fusion

For the purposes of this document, the multimodal Fusion mechanism is the integration of data from multiple Modality Components to produce specific and comprehensive unified information about the interaction. The goal of the multimodal Fusion is to integrate the data coming from a set of input Modality Components or to integrate data in the actions to be executed by a set of output Modality Components.

Fission

For the purposes of this document, the multimodal Fission mechanism refers to the media composition phenomenon: this is the process of realizing a single message on some combination of the available modalities for more than one sensorial mode. This process occurs before the processes that are dedicated to the information rendering or to the Media restitution. The goal is to generate an adequate message, according to the space (in the car, home, conference room), current activity (course, conference, brainstorm) or preferences and profile (chair of the conference, blind user, elderly). A Fission component determines which are the most relevant modalities, selects which media has the best content to return with the Modality Components available in the given conditions; and coordinates this final result [ See the MMI Architecture Standard Life-Cycle Events ]. In this way, the multimodal Fission mechanism handles the repartition of information among several Modality Components and resolves which part of the content will be generated within each Modality Component when the global multimodal content has been defined.

3. Use Cases

3.1 Use Cases SET 1 : Smart Homes

3.1.1 Audiovisual Devices Acting as Smartphone Extensions

3.1.2 Intelligent Home Apparatus

3.2 Use Cases SET 2 : Personal Externalized Interfaces

3.2.1 Smart Cars

3.3 Use Cases SET 3 : Public Spaces

3.3.1 Interactive Spaces

3.3.2 In-Office Events Assistance

3.4 Use Cases SET 4 : Health Sensing

3.4.1 Health Notifiers

3.5 Use Cases SET 5 : Synergistic Interaction and Recognition

3.5.1 Multimodal support to recognition

3.5.2 Discovery to enhance synergic interaction

4. Requirements

4.1 Distribution

4.2 Advertisement

• Information about the of the identity Modality Component :
  For example the unique identifier of the Modality Component, its name, address, port number, its embedded Device or service, constructor, version, transport protocol or lifetime.
  
  
• Information about the of the behavior of the Modality Component :
  The list of content or Media handled by the Modality Component; a list of the commands or actions (e.g. described as a service, functionality, operation or activity), to which the Modality Component responds; parameters or arguments for each action; and information variables that models the state of the Modality Component.
  
  
• Information about the of the context of use of the Modality Component :
  This may cover the name of the organization providing the Modality Component, its authorizations, authentication procedures, privacy policies, its business type (e.g. its UDDI businessService), its access point , complementary information about invocation policies or implementation details.
  
  
• Information about the semantics of the Modality Component for an specific domain :
  A Modality Component type that can be application-specific or can follow some specification (e.g. UPnP, ECHONET or IANA) depending on the implementation. The Modality Component's' federation group (e.g. the Zone in AppleTalk Name Binding Protocol or the DNS subdomain), its scope (e.g. like UA scopes in SLP), its category, its intent (e.g. like Implicit Intents in Android or Web Intents) or any other semantic metadata.

Figure 1: Manifest that can be created either at Design Time or at Load Time

4.3 Discovery

Figure 2: Interaction Manager sending a request via the StatusRequest/StatusResponse pair to the Modality Components

4.4 Registration

Figure 3: Modality Components using a StatusRequest/StatusResponse pair to register its description

4.5 Querying

Figure 4: Modality Components and Interaction Manager using StatusRequest/StatusResponse of ExtensionNotification events to query updates

4.6 Open Issues

During this work, the MMI Discovery and Registration Subgroup did not have time to explore topics like Fault tolerance in Modality Component nodes or in Modality Component's' communication during discovery, register or state updates. Consistency maintenance is a subject of extreme interest and the MMI Discovery and Registration Subgroup would suggest the MMI working group to further investigate this topic from the point of view of the MMI Architecture, eventually with the use of specified attributes like the RequestAutomaticUpdate or the Extension Notification and the use of Modality Component States.

5. Related Work

5.1 Web and TV Interest Group

The Web and TV Interest Group's work covers audio-visual content, e.g., broadcasting programs and Web pages, and related services delivered by satellite and terrestrial broadcasting, or via cable services, as well as delivery through IP. This group is currently working on services discovery for home-networking scenarios from the perspective of content providing.

5.2 Web Intents Task Force

The Web Intents Task Force works on a joint deliverable of the WebApps and Device APIs WGs that aims to produce a way for web applications to register themselves as handlers for services, and for other web applications to interact with such services (which may have been registered through Intents or in other implementation-specific manners) through simple requests brokered by the user agent (a system commonly known as Intents, or Activities). This Task Force is currently working on registering for a limited type of modalities and from a unimodal perspective oriented on process announcement and description.

5.3 Protocols and Formats Working Group

This working group covers the domain of web accessibility. Its objective is to explore how to make web content usable by persons with disabilities. The Accessible Rich Internet Applications Candidate Recommendation is primarily focused on developers of Web browsers, assistive technologies, and other user agents; developers of Web technologies (technical specifications); and developers of accessibility evaluation tools. WAI-ARIA provides a framework for adding attributes to identify features for user interaction, how they relate to each other, and their current state. WAI-ARIA describes new navigation techniques to mark regions and common Web structures as menus, primary content, secondary content, banner information, and other types of Web structures. This work is mostly oriented to web content, web UI components and web events annotation.

A. References

A.1 Key References

[ECMA-262]

ECMA International. ECMAScript Language Specification, Third Edition. December 1999. URL: http://www.ecma-international.org/publications/standards/Ecma-262.htm

[EMMA]

Michael Johnston. EMMA: Extensible MultiModal Annotation markup language. 10 February 2009. W3C Recommendation. URL: http://www.w3.org/TR/2009/REC-emma-20090210/

[MMI-ARCH]

Jim Barnett. Multimodal Architecture and Interfaces. 12 January 2012. W3C Candidate Recommendation. URL: http://www.w3.org/TR/2012/CR-mmi-arch-20120112/

[SLP]

E. Guttman et al. RFC 2608: Service Location Protocol. Version 2,1999. URL: http://www.ietf.org/rfc/rfc2608.txt

[WSDL]

Roberto Chinnici,Jean-Jacques Moreau, Arthur Ryman, Sanjiva Weerawarana. A Web Services Description Language WSDL, v.2.0, 26 June 2007. URL: http://www.w3.org/TR/wsdl20/

[XML]

Tim Bray, Jean Paoli, C.M. Sperberg-McQueen, Eve Maler, John Cowan, François Yergeau. XML Extensible Markup Language, XML 1.1, 4 February 2004. URL: http://www.w3.org/TR/xml11/

A.2 Other References

[A-INTENT]

Google. Android Intent Class Documentation Available at URL: http://developer.android.com/reference/android/content/Intent.html

[ARIA]

James Craig et al. Accessible Rich Internet Applications WAI-ARIA version 1.0. W3C Working Draft, 16 September 2010. Available at URL: http://www.w3.org/TR/wai-aria/

[ASN.1]

John Larmouth, Douglas Steedman, James E. White. ASN.1 International Telecommunication Union. ISO/IEC 8824-1 > 8824-4. Available at URL: http://www.itu.int/ITU-T/asn1/introduction/index.htm

[ECHONET]

Echonet Consortium. Detailed stipulation for Echonet Device Objects Available at URL: http://www.echonet.gr.jp/english/spec/pdf/spec_v1e_appendix.pdf

[EMOTION-USE]

Marc Schröder, Enrico Zovato, Hannes Pirker, Christian Peter, Felix Burkhardt. Emotion Incubator Group Use Cases Available at URL: http://www.w3.org/2005/Incubator/emotion/XGR-emotion/#AppendixUseCases

[IANA]

IANA.IANA Domain Name Service ICCAN - Internet Assigned Numbers Authority (IANA). Available at URL: http://www.iana.org/

[INS]

William Adjie-Winoto, Elliot Schwartz, Hari Balakrishnan, Jeremy Lilley. INS: Intentional Naming System Available at URL: http://nms.csail.mit.edu/projects/ins/

[JINI]

Sun Microsystems JINI / RIVER Apache River (formerly JINI). Available at URL: http://river.apache.org/about.html

[INS/TWINE]

Magdalena Balazinska, Hari Balakrishnan, David Karger INS / TWINE Apache River (formerly JINI). Available at URL: http://nms.csail.mit.edu/projects/twine/

[JXTA]

Sun Microsystems Juxtapose JXTA Protocols Specification v2.0 October 16, 2007. v2.0 . Available at URL: http://jxta.kenai.com/Specifications/JXTAProtocols2_0.pdf

[METEOR-S]

Kunal Verma, Karthik Gomadam, Amit P. Sheth, John A. Miller, Zixin Wu. METEOR-S: Semantic Web Services and Processes Technical Report, 06/24/2005. Available at URL: http://lsdis.cs.uga.edu/projects/meteor-s/

[OGSA]

I. Foster, Argonne et al. OGSA The Open Grid Services Architecture, 24 July 2006 Version 1.5. Available at URL: http://www.ogf.org/documents/GFD.80.pdf

[RES-DIS]

Reaz Ahmed et al. Resource and Service Discovery in Large-Scale Multi-Domain Networks.in Ieee Communications Surveys & Tutorials. 4Th Quarter 2007, Vol 9. Available at URL: http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=5473882&isnumber=5764312

[UDDI]

Luc Clement, Andrew Hately, Claus von Riegen, Tony Rogers. UDDI Spec Technical Committee Draft, Version 3.0.2. Dated 2004/10/19. Available at URL: http://uddi.org/pubs/uddi_v3.htm

[UPNP-DEVICE]

UPnP Forum. UPnP Device Architecture 1.0 Version 1.0, 15 October 2008. PDF document. URL: http://www.upnp.org/specs/arch/UPnP-arch-DeviceArchitecture-v1.0-20081015.pdf

[UPNP-CONTENT]

UPnP Forum. ContentDirectory:4 Service Standardized DCP. Version 1.0, 31 December 2010. PDF document.URL: http://www.upnp.org/specs/av/UPnP-av-ContentDirectory-v4-Service-20101231.pdf

[WEB-INTENTS]

Paul Kinlan. Web Intents. Available at URL: http://webintents.org/