W3C

Multimodal Architecture and Interfaces 1.0
Implementation Report

13 July 2012

Contributors:
Piotr Wiechno, France Telecom (Editor in chief)
Nagesh Kharidi, Openstream
Ingmar Kliche, Deutsche Telekom
B. Helena Rodriguez, Institut Telecom
Dirk Schnelle-Walka, TU Darmstadt
Deborah Dahl, Invited expert
Kazuyuki Ashimura, W3C

Table of Contents

1. Introduction

The Multimodal Architecture and Interfaces Specification entered the Candidate Recommendation period on 12 January 2012.

The planned date for entering Proposed Recommendation is 14 August 2012. This document summarizes the results from the Multimodal Architecture implementation reports received and describes the process that the Multimodal Working Group followed in preparing the report.

1.1 Implementation report objectives

  1. Must verify that the specification is implementable.

1.2 Implementation report non-objectives

  1. The IR does not attempt conformance testing of implementations.

2. Work during the Candidate Recommendation period

During the CR period, the Working Group carried out the following activities:

  1. Clarification and improvement of the exposition of the specification (http://www.w3.org/TR/2012/CR-mmi-arch-20120112/)
  2. Disposing of comments that were communicated to the WG during the CR period, detailed in the Disposition of Comments document for the CR period.
  3. Preparation of this Implementation Report.

3. Participating in the implementation report

Implementers were invited to contribute to the assessment of the W3C Multimodal Architecture and Interfaces 1.0 Specification by submitting implementation reports describing the coverage of their implementations with respect to the test assertions outlined in the table below.

Implementation reports, comments on this document, or requests made for further information were posted to the Working Group's public mailing list www-multimodal@w3.org (archive).

4. Entrance criteria for the Proposed Recommendation phase

The Multimodal Interaction Working Group established the following entrance criteria for the Proposed Recommendation phase in the Request for CR:

  1. Sufficient reports of implementation experience have been gathered to demonstrate that constituents of the Multimodal Architecture based on the specification are implementable and have compatible behavior.
  2. Specific Implementation Report Requirements (outlined below) have been met.
  3. The Working Group has formally addressed and responded to all public comments received by the Working Group.

All three of these criteria have been met. A total of 13 implementations were received from 5 organizations, one each from France Telecom, Openstream and JVoiceXML as well as 2 from Deutsche Telekom and 8 from Telecom ParisTech. The testimonials below indicate the broad base of support for the specification.

All of the required features of the Multimodal Architecture had at least two implementations. The only features which did not have enough implementations were those explicitly listed in the draft specification as "at risk of removal due to potential lack of implementations". These were:

Consequently, these two features have been removed from the final specification as stated in the draft version.

The maturity of the specification has been verified by an interoperability test done by Deutsche Telekom, France Telecom and Openstream. Results of this test have been documented and published as a W3C Working Group Note on 24 January 2012.

5. Implementation report requirements

5.1 Detailed requirements for implementation report

  1. Testimonials from implementers are included in the IR when provided to document the utility and implementability of the specification.
  2. The IR must cover all specified features in the specification. For each feature the IR should indicate:
  3. Feature status is a factor in test coverage in the report:

5.2 Notes on testing

  1. A implementation report must indicate the outcome of evaluating the implementation with respect to each of the test assertions. Possible outcomes are "pass", "fail" or "not-impl".  "not-impl" means the Multimodal Architecture constituent has not implemented the specific feature required by a test.

5.3 Out of scope

The Multimodal Architecture Implementation Report does not cover:

6. Systems

This section contains testimonials on the Multimodal Architecture from the 5 organizations that submitted implementation reports.

Deutsche Telekom

Exec Summary

Telekom Innovation Laboratories (Deutsche Telekom R&D) develops multimodal user interfaces for future telecommunication services and is pleased to see the Multimodal Architecture and Interfaces specification moving forward. We have implemented a distributed multimodal interaction management system completely relying on W3C standards: an Interaction Manager based SCXML (using Apache Commons SCXML interpreter), a Graphical Modality Component based on HTML and a Voice Modality Component based on CCXML and VoiceXML. EMMA is used for representation of user input throughout the system. All components are integrated using HTTP as described by the Multimodal Architecture and Interfaces specification. These components have been successfully used in the interoperability test activity with other MMI working group members, demonstrating a distributed multimodal application using components delivered by three independent parties.

Implementing multimodal applications requires to write a lot of markup code (e.g. SCXML, HTML, grammars), which has to be synchronized. To reduce complexity in application development we developed a Multimodal Application Builder, a graphical tool to model multimodal applications and to generate most parts of the code (stubs).

France Telecom

Exec Summary

Orange Labs (France Telecom R&D) researches and prototypes multimodal user interfaces on mobile devices. We use and promote open standards, and as such are particularly interested in the W3C Multimodal Architecture specification, which allows building multimodal applications out of specialized components delivered by multiple vendors. Speech, touch and other modalities can be managed by constituents hosted on local devices or on the web, all orchestrated into a single multimodal interaction through the Multimodal Architecture. Orange Labs has implemented an Interaction Manager based on the Commons SCXML application engine, available to Modality Components as an HTTP server. While this implementation is primarily designated for internal R&D work, it was also successfully used in interoperability tests with W3C partners, demonstrating a distributed multimodal application running on components delivered by three independent parties and integrated through the proposed standard.

JVoiceXML

Exec Summary

JVoiceXML is a free VoiceXML interpreter for JAVA with an open architecture for custom extensions. Demo implementation platforms are supporting JAVA APIs such as JSAPI and JTAPI.

Openstream

Exec Summary

Openstream, in its ongoing commitment to open and interoperable standards as a W3C Member, leads the development of multimodal platform and context-aware enablement of speech and other modalities in mobile applications. We are proud to be one of the early and active developers of the W3C Multimodal Interaction Architecture. We have implemented a reference architecture for MMI comprising of Interaction Manager(IM) and several modality components(MCs) as part of our context-aware multimodal platform Cue-me. Adopting the MMI architecture and markup languages such as SCXML, InkML, EMMA, etc. enabled us to easily incorporate and inter-operate with several different vendors' products, enabling us to offer our platform and solutions on a broad array of devices and systems.

Telecom ParisTech

Exec Summary

This report describes the implementation included in the SOA2M project of the MM Group- TSI Department (Institut Telecom-Telecom ParisTech). This research covers abstract multimodal user interfaces in Service Oriented Architectures for pervasive environemment, an particularly in Smart Conference Rooms. Our prototype is an implementation of a multimodal ambient system providing personalized assistance services to different profiles of users. With this prototype we are looking to test the intelligent automatic fusion and fission of modalities. We are interested on using semantics with the W3C Multimodal Architecture specification. The group has implemented a Flex/AIR Interaction Manager with an SCXML engine, available to Modality Components as a semantically annotated service (published as web or Bonjour service), and web-based RIA modality components at different levels: the basic ones (Pointer, Selector and Graphics IN/OUT) and the more complex (a voice synthesizer and a Carousel).

7. Test results

The aim of this section is to describe the range of test assertions developed for the Multimodal Architecture 1.0 Specification and summarize the results from the implementation reports received in the CR period. The table lists all the assertions that were derived from the Multimodal Architecture 1.0 Specification.

The Assert ID column uniquely identifies the assertion. The Feature column indicates the specific elements or attributes which the test assertion applies to. The Spec column identifies the section of the Multimodal Architecture Specification from which the assertion was derived. The Semantics column specifies the semantics of the feature or the constraint which must be met. The Result column will be annotated with the number of 'pass', 'fail', and 'not implemented' (P/F/NI) in the set of implementation reports.

7.1 MMI Lifecycle Event XML Schema conformance

A conforming Multimodal Achitecture implementation which uses XML to represent MMI Lifecycle Events must utilize documents that successfully validate with respect to the MMI Lifecycle Event XML Schema.

7.2 Multimodal Architecture test assertions

Common test assertions
SpecAssert IDFeature SemanticsResults
PFNI
5.2.1 144 The Interaction Manager All events that the Modality Components generate MUST be delivered to the Interaction Manager. 1201
5.2.1 145 All events that are delivered to Modality Components MUST be sent by the Interaction Manager 607
5.2.1 87 A modality component may contain its own interaction manager. 2011
5.2.4.1 118 The Event Transport Layer The event delivery mechanism MUST guarantee that events delivered from the same source are delivered in that order. 409
5.2.4.1 211 The event delivery mechanism MUST report an error if an event can not be delivered.    
6 89 Modality Components communicate with the Interaction manager via events. 1201
6 90 Modality Components communicate with the Interaction manager via asynchronous events. 1201
6 91 Constituents must be able to send events. 1300
6 92 Constituents must be able to handle events that are delivered to them asynchronously. 1300
6.1 32 All lifecycle events must use a common basic concept of 'context'. 1300
6.1 33 A context must represent a single extended interaction with zero or more users across one or more modality components. 1201
6.1 34 A context should cover the longest period of interaction over which it would make sense for components to store information. 1300
6.1.1 35 Context The Context attribute must be a URI that is unique for the lifetime of the system. 1210
6.1.1 36 All events relating to a given interaction must use the same context URI. 1300
6.1.1 146 Any two events that use different context URIs must be interpreted as parts of unrelated interactions. 1300
6.1.2 37 Source The Source attribute must be a URI representing the address of the sender of the event. 1300
6.1.3 38 Target The Target attribute must be a URI representing the address of the destination of the event. 1300
6.1.4 39 RequestID The RequestID attribute must be an identifier that is unique within the given context for each Request/Response pair. 1300
6.1.4 40 For any Request/Response event pair, the RequestID in the Response event must match the RequestID specified in the Request event. 1300
6.1.5 41 Status The Status attribute must be either 'success' or 'failure'. 1300
6.1.5 42 The Response event of a Request/Response pair must use the Status field to report whether it succeeded in carrying out the request. 1300
6.1.6 43 StatusInfo The Response event of a Request/Response pair MAY use the StatusInfo field to provide additional status information. 409
6.1.6 208 All constituents MUST be able to process Life-Cycle events that use the StatusInfo field to provide additional status information. 409
6.1.7 44 Data Any event MAY may use the Data field to contain arbitrary data. 1201
6.1.7 209 All constituents MUST be able to process Life-Cycle events that use the Data field to provide arbitrary data. 1201
6.1.8 147 Confidential Any event MAY use the Confidential field to indicate whether the contents of this event are confidential. 0013
6.1.8 148 The default value of the Confidential field must be 'false' 1012
6.1.8 149 If the value of Confidential is 'true', the Interaction Manager and Modality Component implementations MUST not log the information or make it available in any way to third parties unless explicitly instructed to do so by the author of the application. 0013
6.2.1 95 NewContext A modality component MAY send a NewContextRequest event to the interaction manager to request that a new context be created. 1102
6.2.1 151 The IM MAY create a new context without a previous NewContextRequest by sending a PrepareRequest containing a new context ID to the Modality Components. 508
6.2.1 152 The IM MAY create a new context without a previous NewContextRequest by sending a StartRequest containing a new context ID to the Modality Components. 409
6.2.2 154 Prepare The IM MAY send a PrepareRequest to allow the Modality Components to pre-load markup and prepare to run. 319
6.2.2 157 The Interaction Manager MAY send multiple PrepareRequest events to a Modality Component for the same Context before sending a StartRequest. 409
6.2.2 158 Each PrepareRequest the IM sends to a Modality Component MAY reference a different ContentURL or contain different in-line Content. 3010
6.2.2.1 160 The IM MAY use the same context value in successive PrepareRequest events when it wishes to execute multiple instances of markup in the same context. 409
6.2.2.1 161 The PrepareRequest event MAY contain a ContentURL field with the URL of the content that the Modality Component SHOULD prepare to execute. 409
6.2.2.1 162 The PrepareRequest event MAY contain a Content field with inline markup that the Modality Component SHOULD prepare to execute. 2011
6.2.2.1 164 The IM MAY leave both contentURL and content fields of a PrepareRequest event empty. 3010
6.2.3 168 The IM MAY include a value in the ContentURL or Content field of the StartRequest event. 517
6.2.3.1 171 The IM MAY use the same context value in multiple StartRequest events when it wishes to execute multiple instances of markup in the same context. 3010
6.2.3.1 172 The StartRequest event MAY contain a ContentURL field with the URL of the content that the Modality Component MUST attempt to execute 607
6.2.3.1 173 The StartRequest event MAY contain a Content field with inline markup that the Modality Component MUST attempt to execute. 3010
6.2.3.1 175 The IM MAY leave both contentURL and content fields of a StartRequest event empty. 409
6.2.5.1 17 The IM MAY send a CancelRequest to stop processing in the Modality Component. 508
6.2.6.1 31 The IM MAY send a PauseRequest to suspend processing by the Modality Component. 3010
6.2.7 1 Resume The IM MAY send the ResumeRequest to resume processing that was paused by a previous PauseRequest. 3010
6.2.8 61 ExtensionNotification The ExtensionNotification event MAY be generated by the IM to encapsulate application-specific events. 409
6.2.8 188 The ExtensionNotification event MAY be generated by the Modality Component to encapsulate application-specific events. 409
6.2.8 210 All constituents MUST be able to process ExtensionNotification events. 3010
6.2.10 74 Status The IM MAY send a StatusRequest event to a MC to inquire about the status of a specific user interaction (context) or the status of the MC itself. 607
6.2.10 79 The recipient of a StatusRequest event MUST respond with a StatusResponse event. 1300
6.2.10 190 A MC MAY send a StatusRequest event to the IM to inquire about the status of a specific user interaction (context) or the status of the IM itself. 508
6.2.10.1 75 The sender of a StartRequest event MAY use the Context field to specify the context for which the status is requested. 1201
6.2.10.1 78 The StatusRequest event MUST contain a RequestAutomaticUpdate field which is a boolean value. 1102
6.2.10.1 191 If the Context field is present in a StatusRequest message, the recipient MUST respond with a StatusResponse message indicating the status of the specified context. 1201
6.2.10.1 192 If the Context field is not present in a StatusRequest message, the recipient MUST send a StatusResponse message indicating the status of the underlying server. 904
6.2.10.1 195 If the RequestAutomaticUpdate field of a StatusRequest message is 'true', the recipient SHOULD send periodic StatusResponse messages without waiting for an additional StatusRequest message. 904
6.2.10.1 196 If the RequestAutomaticUpdate field of a StatusRequest message is 'false', the recipient SHOULD send one and only one StatusResponse message in response to this request. 1003
6.2.10.2 81 The sender of a StartResponse event MAY use the Context field to specify the context for which the status is being returned. 1300
6.2.10.2 194 The StatusResponse event MUST contain an AutomaticUpdate field which is a boolean value. 1102
6.2.10.2 198 If the AutomaticUpdate field of a StatusResponse message is 'true', the sender MUST keep sending StatusResponse messages in the future without waiting for another StatusRequest message. 1012
6.2.10.2 199 If the AutomaticUpdate field of a StatusResponse message is 'false', the sender MUST wait for a subsequent StatusRequest message before sending another StatusResponse message. 1102
6.2.10.2 200 If the Context field is present in a StatusResponse event, the response MUST represent the status of the specified context. 1201
6.2.10.2 201 If the Context field is not present in a StatusResponse event, the response MUST represent the status of the underlying server. 904
6.2.10.2 202 The Status field of a StatusResponse message is an enumeration of 'Alive' or 'Dead'. 1300
6.2.10.2 203 If the StatusResponse message specifies a context which is still active and capable of handling new life cycle events, the sender MUST set the Status field to 'Alive' 1201
6.2.10.2 204 If the StatusResponse message specifies a context which has terminated or is otherwise unable to process new life cycle events, the sender MUST set the Status to 'Dead'. 1210
6.2.10.2 205 If the StatusResponse message doesn't provide a Context field, and the sender is able to create new contexts, it MUST set the Status to 'Alive'. 904
6.2.10.2 206 If the StatusResponse message doesn't provide a Context field, and the sender is unable to create new contexts, it MUST set the Status to 'Dead'. 1003
IM-specific test assertions
SpecAssert IDFeature SemanticsResults
PFNI
6.2 150 The Interaction Manager must support the basic life-cycle events. 508
6.2.1 96 When the Interaction Manager receives a NewContextRequest event it MUST respond with a NewContextResponse event. 508
6.2.1 103 The NewContextResponse event MUST ONLY be sent in response to the NewContextRequest event. 1201
6.2.1.2 107 If IM has accepted the NewContextRequest, it MUST set the Status field to Success. 508
6.2.1.2 108 If the Status property of the NewContextResponse event is Failure then the context identifier MUST be empty. 1102
6.2.1.2 153 If the NewContextRequest Status field is set to Success, the IM MUST include a new context identifier. 508
6.2.2.1 163 The IM MUST NOT specify both the ContentURL and Content in a single PrepareRequest. 3010
6.2.3 166 Start To invoke a modality component, the IM MUST send a StartRequest. 508
6.2.3 174 The IM MUST NOT specify both the ContentURL and Content in a single StartRequest event. 508
6.2.7 185 The IM MUST NOT send the ResumeRequest to a context that is not paused due to a previous PauseRequest. 2011
6.2.9 65 Clear Context The IM SHOULD send a ClearContextRequest event to a MC to indicate that the specified context is no longer active and that any resources associated with it may be freed. 508
6.2.9 189 Once the IM has sent a ClearContextRequest to a Modality Component, it MUST NOT send the Modality Component any more events for that context. 508
MC-specific test assertions
SpecAssert IDFeature SemanticsResults
PFNI
6.2 94 All Modality Components must support the basic life-cycle events. 1102
6.2.2 155 Modality Components MUST return a PrepareResponse event in response to a PrepareRequest event. 904
6.2.2 156 Modality Components that return a PrepareResponse event with Status of 'Success' SHOULD be ready to run with close to 0 delay upon receipt of the StartRequest. 706
6.2.2 159 When it receives multiple PrepareRequests, the Modality Component SHOULD prepare to run any of the specified content. 706
6.2.2.1 165 If both contentURL and content of a PrepareRequest are empty, the Modality Component MUST revert to its default behavior. 706
6.2.2.1 207 If a MC receives a PrepareRequest containing a new context (without a previous NewContextRequest/Response exchange), it MUST accept the new context and return a PrepareResponse message. 706
6.2.3 167 The Modality Component MUST return a StartResponse event in response to a StartRequest event. 1102
6.2.3 169 If the IM includes a value in the ContentURL or Content field of the StartRequest event, the Modality Component MUST use this value. 607
6.2.3 170 If a Modality Component receives a new StartRequest while it is executing a previous one, it MUST either cease execution of the previous StartRequest and begin executing the content specified in the most recent StartRequest, or reject the new StartRequest, returning a StartResponse with status equal to 'failure'. 706
6.2.3.1 176 If the IM leaves both contentURL and content of a StartRequest event empty, the Modality Component MUST run the content specified in the most recent PrepareRequest in this context, if there is one. 607
6.2.3.1 177 If the Modality Component did not receive a PrepareRequest event prior to receiving a StartRequest event with empty Content and ContentURL fields, it MUST revert to its default behavior. 706
6.2.4 8 DoneNotification If a modality component finishes processing it MUST send a DoneNotification 805
6.2.5 178 Cancel A Modality Component MUST return a CancelResponse event in response to a CancelRequest event. 904
6.2.5 184 A Modality Component that receives a CancelRequest event MUST stop processing and then MUST return a CancelResponse. 904
6.2.5.2 27 CancelResponse MUST contain a Status field. 904
6.2.6 181 Pause Modality Components MUST return a PauseResponse once they have paused, or once they determine that they will be unable to pause. 508
6.2.6.2 186 Implementations that have received a ResumeRequest event even though they haven't paused MUST return a ResumeResponse with a Status of 'success'. 409
6.2.7 187 The 'Status' of a ResumeResponse event MUST be 'success' if the implementation has succeeded in resuming processing and MUST be failure otherwise 409
6.2.7.1 5 Implementations that have paused MUST attempt to resume processing upon receipt of this event and MUST return a ResumeResponse afterwards. 409
6.2.9 69 A MC MUST send a ClearContextResponse event in response to a ClearContextRequest event, even if doesn't take any particular action. 1102

Appendices

Appendix A - Acknowledgements

The Multimodal Interaction Working Group would like to acknowledge the contributions of all the people who allowed for the realization of this activity.