Traditional View of the Web

Someone sitting at a computer with a large colour display, full-sized keyboard and a high speed network connection.
The Web provides access to remote information and services. The Web Browser identifies web pages via their URI and down loads the markup, graphics and other resources from web servers.

But ...

Not everyone can see clearly or can operate the mouse etc.
- Need to design for accessible web sites
Many new kinds of devices with much smaller displays
- Need to adapt to small displays
- Need to design services for embedded or mobile context
Traditional telephone with audio + phone key pad
- VoiceXML: markup for speech dialogues
- Corporate front desk
  - Telephone number for call centre
  - Web address (URI) for website
  Both can utilize W3C web technologies

The Architecture of the Web

The World Wide Web (WWW, or simply Web) is an information space in which the items of interest, referred to as resources, are identified by global identifiers called Uniform Resource Identifiers (URI).

URI identifiers resource, data formats represent resources

See http://www.w3.org/TR/webarch/

Web pages, browsers and Web sites

Web Pages

Typically marked up in HTML
- But also XHTML, SVG, SMIL, MathML, XForms and combinations
Interpreted by Web browsers
Retrieved from Websites via HTTP (HyperText Transfer Protocol)
Behaviour is event driven and customizable via scripting
Presentation styled via CSS (Cascading Style Sheets)

Web Servers

Host websites
Are driven by client requests
- HTTP GET, PUT, POST
Map URIs to static files or scripts (CGI, Servlets)
Back-ended with databases and payment systems etc.

The role of scripting

The DOM

Platform and language neutral API's for manipulating XML documents with a rich model of events
Specialized API's for HTML and CSS
Widely used for customizing Web pages
- DHTML for dynamically configurable user interfaces
- XMLHTTP and live pages (AJAX)
Emerging opportunities for enhancing the DOM
- User preferences, device capabilities, device configuration and environmental conditions
- Device coordination and access to remote services
Layering APIs on protocols
- DOM property with get/set methods and events
- Local manifestation of a remote property
- Underlying implementation propagates changes
- Implemented on top of HTTP, SIP or other protocol

Freedom of Choice

New opportunities to give end users the means to select the modes of interaction best suited to their current needs, whilst also enabling the application developer to provide an effective end-user experience for whichever choices users make.

Multimodal Interaction

Giving users a choice of modes:

Speech, keypad or pen for input from user
- Speech recognition and recording
- Keystrokes and pointer gestures
- Handwriting recognition and trace capture
Aural, visual or tactile output to user
- Conventional display for visual output
- Speech synthesis and pre-recorded audio
- Phone vibrator and Haptic interfaces, e.g. BMW iDrive
Text as the common denominator and basis for accessibility
Need to dynamically adapt to the changing context
- Context Awareness
  - Personal preferences
  - Noisy environment
  - Restaurants and Theatres
  - Low battery
  - Poor reception
  - Geographical location (GPS)

Dialogues

For speech and for small displays, the user interface has to be broken down into a sequence of small steps.

Dialogue between the user and the application
- Expressed via markup, client and server-side scripts
Visual, aural, tactile or a combination of these
Who is in the driving seat?
Large displays offer users lots of choices
Small displays don't ...
Lead by the hand (Wizards)
User initiative for changing task
- Mouse click, key press or spoken command
Translating user input into a form that is meaningful to the application

Extensible Multi-Modal Annotations (EMMA)

XML format for data exchange between input processors and interaction management systems
Enables processors to annotate application specific data with things like confidence scores, time stamps, mode of input, alternative recognition hypotheses, partial results, etc.
Can be generated via combination of
- Speech Recognition Grammar Specification (SRGS)
- Semantic Interpretation for Speech Recognition (SISR)
If the user says: I want to fly to New York next Tuesday

The interpreted result can be represented in EMMA as follows:

<emma:emma version="1.0" xmlns:emma="http://www.w3.org/2003/04/emma"> 
  <emma:interpretation id="int1" emma:confidence="0.75"> 
    <destination>New York</destination>
    <date>28062005</date>  
  </emma:interpretation>
</emma:emma>

Can also be generated from text obtained via keystrokes or handwriting recognition (SRGS and SISR are text based)

Authoring Multimodal Applications

No complete standards based solution as yet
Various approaches have been considered
W3C Member contributions on SALT and X+V
W3C MMI Working Group is developing a flexible architecture that would encompass a range of solutions
Some other possible approaches
- XHTML + scripting
- XHTML + state transition networks
- Plan and rule based approaches
- Promise of XHTML + SVG
Link to more details

Multimodal Interaction Architecture

Modality components (MVC view)
- Support interaction with the user
- Can be simple or complex
- Single or multiple modalities
- Integrated or distributed across multiple devices/files
Examples:
- Speech component providing TTS and ASR services
- XHTML web page providing visual interaction with a form
- VoiceXML web page providing voice interaction with a form
- Compound markup supporting multimodal interaction
- Server based dialogue component encompassing multiple pages
Modality components communicate via asynchronous messages
- The DOM may hide the messaging layer/run-time framework
Shared data model (MVC model)
Optional Interaction Manager in coordination role (MVC controller)
- e.g. event driven state transition network for synchronizing visual and aural modalities

Distributed DOM

We need to focus on enabling distributed DOM's

Distributing DOM events
- User interface events (interpreted user input)
- Events signalling changes to user preferences, device configuration, device capabilities, environmental conditions, shared data model
- Could be raised by clients or servers
Synchronising DOM across multiple devices
- Propagating changes to shared data model
- Synchronising local copies of remote properties
  - ink levels in networked printer
  - geographical location of a mobile device
  - audio input and output levels and whether these are muted
- Updating the DOM for a remote user interface
Events and Updates can flow in both directions (in and out)
Take place in the context of an application session
- Both temporary and persistent sessions

Who does What?

W3C is an effective forum for the authoring layer
- Markup languages
- DOM: interfaces for properties, methods and events
- Application level view of sessions and resources
IETF is a natural home for work on bindings to IETF protocols
- Layering of distributed DOM on top of SIP, HTTP, etc.

Closing remarks

W3C seeks to enable royalty free implementations of its Recommendations
IETF work on protocol support for distributed DOM's would complement W3C's work on multimodal web applications
Coordination through people participating in both efforts and through W3C/IETF liaison contacts
- Dan Connolly, Philippe Le Hégaret, see also: public-ietf-w3c mailing list

W3C work relevant to remote user interfaces