Traditional View of the Web

Someone sitting at a computer with a large colour display, full-sized keyboard and a high speed network connection.
The Web provides access to remote information and services. The Web Browser identifies web pages via their URI and down loads the markup, graphics and other resources from web servers.

But ...

Not everyone can see clearly or can operate the mouse etc.
- Need to design for accessible web sites
Many new kinds of devices with much smaller displays
- Need to adapt to small displays
- Need to design services for embedded or mobile context
Traditional telephone with audio + phone key pad
- VoiceXML: markup for speech dialogues
- Corporate front desk
  - Telephone number for call centre
  - Web address (URI) for website
  Both can utilize W3C web technologies

The Architecture of the Web

The World Wide Web (WWW, or simply Web) is an information space in which the items of interest, referred to as resources, are identified by global identifiers called Uniform Resource Identifiers (URI).

URI identifiers resource, data formats represent resources

See http://www.w3.org/TR/webarch/

Web pages, browsers and Web sites

Web Pages

Typically marked up in HTML
- But also XHTML, SVG, SMIL, MathML, XForms and combinations
Interpreted by Web browsers
Retrieved from Websites via HTTP (HyperText Transfer Protocol)
Behaviour is event driven and customizable via scripting
Presentation styled via CSS (Cascading Style Sheets)

Web Servers

Host websites
Are driven by client requests
- HTTP GET, PUT, POST
Map URIs to static files or scripts (CGI, Servlets)
Back-ended with databases and payment systems etc.

Freedom of Choice

New opportunities to give end users the means to select the modes of interaction best suited to their current needs, whilst also enabling the application developer to provide an effective end-user experience for whichever choices users make.

Multimodal Interaction

Giving users a choice of modes:

Speech, keypad or pen for input from user
- Speech recognition and recording
- Keystrokes and pointer gestures
- Handwriting recognition and trace capture
Aural, visual or tactile output to user
- Conventional display for visual output
- Speech synthesis and pre-recorded audio
- Phone vibrator and Braille output strips
Text as the common denominator and basis for accessibility
Need to dynamically adapt to the changing context
- Context Awareness
  - Personal preferences
  - Noisy environment
  - Restaurants and Theatres
  - Low battery
  - Poor reception
  - Geographical location (GPS)

Differences between Speech and GUI

Speech brings new requirements for user interfaces

Speech is ephemeral
Recognition errors
Prompts and speech grammars
The role of dialogue
Traffic lights model for confirmations
Tapered prompts for progressively stronger guidance
Novice vs. power users

Small vs Large displays

You can't simply shrink the view!
Limits on how much information can be presented at any one time
Large applications need to be broken up into smaller pieces
Avoiding the temptation to always get the next piece from the server
This is about behaviour and not simply layout into multiple pages
The role of dialogue

Dialogues

Sequence of interactions between the user and the application
Visual, aural or a combination of the two
Who is in the driving seat?
Large displays offer users lots of choices
Small displays don't ...
Lead by the hand (Wizards)
User initiative for changing task
- Mouse click, key press or spoken command

Interpreting user input

Most input is essentially equivalent to text
Natural language understanding
- Numbers, dates, times, ...
- Named entities, e.g. "San Francisco", "John Smith", "pizza", "pepperoni"
- Verbs, adjectives, adverbs, prepositions, ...
- Deixis, Anaphora and Ellipsis
  - Deixis - referring to something external to the current utterance: I want that one
  - Anaphora - words that refer to something elsewhere in the utterance: When John called Mary, she was having her lunch.
  - Ellipsis - where part of the utterance has been dropped because it is "obvious" from the context
- Keeping it practical
Extracting data meaningful to the application
The role of EMMA

Extensible Multi-Modal Annotations (EMMA)

XML format for data exchange between input processors and interaction management systems
Enables processors to annotate application specific data with things like confidence scores, time stamps, mode of input, alternative recognition hypotheses, partial results, etc.
Can be generated via combination of
- Speech Recognition Grammar Specification (SRGS)
- Semantic Interpretation for Speech Recognition (SISR)
If the user says: I want to fly to New York next Tuesday

The interpreted result can be represented in EMMA as follows:

<emma:emma version="1.0" xmlns:emma="http://www.w3.org/2003/04/emma"> 
  <emma:interpretation id="int1" emma:confidence="0.75"> 
    <destination>New York</destination>
    <date>28062005</date>  
  </emma:interpretation>
</emma:emma>

Can also be generated from text obtained via keystrokes or handwriting recognition (SRGS and SISR are text based)

Authoring Multimodal Applications

No complete standards based solution as yet
Various approaches have been considered
W3C Member contributions on SALT and X+V
W3C MMI Working Group is developing a flexible architecture that would encompass a range of solutions
Some other possible approaches
- XHTML + scripting
- XHTML + state transition networks
- Plan and rule based approaches
- Promise of XHTML + SVG

Speech Application Language Tags (SALT)

Developed by Cisco, Comverse, Intel, Microsoft, Phillips and Speech Works
Basic SALT markup elements
- <listen> wrapper for other SALT markup
- <prompt> SSML prompt to guide what user says
- <grammar> SRGS grammar for recognition
- <DTMF> for touch-tone (DTMF) input
- <bind> assign data to form fields and JS variables
- <smex> message exchange e.g. for call control
- SALT is event driven
  - Choice of markup or JavaScript for event handlers
  - SALT markup is typically generated server-side from higher level dialog authoring framework

SALT Example

<body onload="intro.start();">

<salt:prompt  id="intro" onComplete=" askTravel.start();">
      Welcome to Ajax Travel 
</salt:prompt>

<salt:prompt id="askTravel" onComplete="lsnTravel.start();">
      Do you want to travel by air, rail, or boat? 
</salt:prompt>

<salt:listen id="lsnTravel" onreco="threeWayBranch();">
   <salt:grammar id="gram1" name="gram1">
      <grammar version="1.0" xml:lang="en-US"
      xmlns="http://www.w3.org/2001/06/grammar" root="travel">
         <rule id="travel" scope="public">
            <one-of>
               <item>air</item>
               <item>rail</item>
               <item>boa</item>
            </one-of>
         </rule>
      </grammar>
   </salt:grammar>
   <salt:bind targetelement ="saltdebug" value="/" />
   <salt:bind targetelement ="spokenValue" value="//" />
</salt:listen>

XHTML + Voice (X+V)

Developed by IBM, , Motorola and Opera Software
Extends XHTML 1.1 with subset of VoiceXML 2.0
Events can be used to initiate voice dialogs
- Use of XML Event syntax
- <sync> element synchronizes VoiceXML and XHTML form fields whether input is from speech or keyboard
- <cancel> element allows document author to abort a running speech dialog
  - Triggered as an event handler, e.g. for click event on an XHTML form field
- Typically implemented as separate XHTML and VoiceXML interpreters
  - Can be co-resident or distributed across different machines

X+V Example

<?xml version="1.0"?>
<html
 xmlns="http://www.w3.org/1999/xhtml"
 xmlns:vxml="http://www.w3.org/2001/vxml"
 xmlns:ev="http://www.w3.org/2001/xml-events"
 xmlns:xv="http://www.voicexml.org/2002/xhtml+voice"
>
  <head>
    <title>XHTML+Voice Example</title>
    <!-- voice handler -->
    <vxml:form id="sayHello">
      <vxml:block>
        <vxml:prompt xv:src="#hello"/>
      </vxml:block>
    </vxml:form>
  </head>
  <body>
    <h1>XHTML+Voice Example</h1>

    <p id="hello" ev:event="click" ev:handler="#sayHello">
      Hello World!
    </p>
  </body>
</html>

XHTML + Scripting

Speech capabilities exposed via script objects
Dialogues expressed as event driven scripts
For example, when user sets focus to a form field, the onfocus event handler plays audio prompt and activates speech grammar
The recognition event then fills out the field
If the user instead types in the value, the onchanged event can be used to abort the speech dialogue
This approach is very flexible
Requires minimal changes to the browser
but harder to maintain than declarative approaches

XHTML + State Transition Networks

Named states with event driven state transition rules
Have been extensively studied by researchers
We are looking at Harel State Charts, which offer nested states plus asynchronous execution of parallel states, and used by UML for modeling system behaviour, e.g. a watch:

See "On Visual Formalisms", David Harel, Communications of the ACM, May 1988.

Rule and Plan Based Approaches

These use rules to determine a set of goals and then formulate plans to satisfy those goals
Very interesting from a research perspective
But, unlikely to be suitable for standardization at least for the next few years
But opportunities for using such approachs on server-side, especially for large applications
Server uses plan to dynamically generate web page
Web page implements small part of overall dialogue and can be represented in conventional way
This enables embedded and mobile devices to be part of sophistocated applications

Embedded vs Distributed solutions

Size of speech engines
- State of the art commercial TTS takes about 100 to 200Mb per voice
- Speech grammars can be very large and impractical to download
- But embedded speech systems currently offer reasonable quality TTS, and an ability to use vocabularies of thousands of words for speech recognition, easily good enough for many applications
- Embedded speech avoids network latency issue (a problem even for 3G mobile networks)
XHTML + remote VoiceXML
- Opportunity to leverage existing network-based VoiceXML platforms to add multimodal capabilities to mobile devices
- But underpressure from integrated browser based solutions
- Long term we expect a mix of embedded and network-based speech recognition, and a mix of client and server-based dialogue handling

Where Next?

XML language for state charts (SCXML) is under development by W3C Voice Browser Working Group
Interfaces for context awareness are under development by W3C Device Independence Working Group
EMMA is due to enter Last Call within a few weeks
InkML an XML language for ink traces is expected to enter Last Call within a few months
Multimodal Architecture Working Draft will be refined over next few months
Pointer to UbiWeb talk later this afternoon