Introduction

• Multimodal user interfaces are frequently composed of loosely coupled components
• These may be based very different kinds of markup languages, e.g. VoiceXML and XHTML
• Alignment of general concepts such as events submission and data models is of great interest and value to the MMI WG

MMI and Backplane

• Backplane note, last para section 2, refers to
  • “loosely coupled” style of interaction with few dependencies on the internal design of components
• The MMI Architecture is a good practical realization of these goals
• A shared understanding should be helpful

MMI Architecture

• A loosely-coupled, event-based architecture for integrating multiple modalities into applications
• All communication is event-based
• Based on a set of standard life-cycle events
• Components can also expose other events as required
• Encapsulation protects component data and enhances extensibility to new modalities
• Can be used outside a web environment

MMI Architecture

See http://www.w3.org/TR/mmi-arch/

Defines five basic components of an MMI system

• Runtime Framework or Browser: initializes application and runs markup
• Interaction Manager: coordinates modality components and provides application flow
• Modality Components: provide modality capabilities such as speech, pen, keyboard, mouse
• Data Model: handles shared data
• DCI (Delivery Context and Interfaces): device properties and user preferences

Generic MMI Architecture

Principles

• Runtime Framework communicates with Modality Components through asynchronous events
• Modality Components don’t communicate directly with each other but indirectly through the Runtime Framework
• Components must implement basic life cycle events, may expose others
• Modality components can be nested (e.g. a Voice Dialog component like a VoiceXML <form>)
• Components need not be markup-based
• Use of EMMA for passing interpreted user input to Interaction Manager

Life Cycle Events

Differences in Perspective

• Processing vs Markup
  • Backplane focus on markup and documents
  • MMI Architecture focus on processing
• MMI Architecture describes components that may or may not be driven by markup
  • speech and handwriting recognition
  • natural language dialogue
• Would like to see more examples of multi-document applications in Backplane work
  • Combining procedural and declarative approaches

Submission

• MMI Architecture – modality components are inscrutable black boxes
• Modality components may independently submit data to server without going through central component
  • For example logging information for later analysis
• Does Backplane centralize submission?
  • suggested by Backplane note, section 3, figure 1

Data Models

• Backplane note characterizes the Data model
  • bridge between end users and business services
• MMI WG believes in more general role
  • Data models may include information used to support the interaction itself
    • interaction history at different levels of description
      • what was said, interpretations of user input, goals and tasks
    • user interface state as basis for synchronization across modalities
    • delivery context
      • user preferences, device capabilities, environmental conditions

Binding

• Backplane focus on model↔model updates
  • XForms use of model-view-controller pattern
• MMI Architecture
  • Interaction Manager owns global data model
    • can't be updated directly by modality components
    • modality components may have hidden data
  • Inherent uncertainty of recognition process
    • speech, handwriting, gestures, authentication
  • Differences in terminology across components
    • necessitates reformatting of information flows
  • Direct binding approach won't suffice