Dave Raggett,
W3C/Openwave Dave Raggett,
W3C/Openwave
A model is presented for representing web-based multimodal dialogs as sets of prioritized tasks. This is motivated by an analysis of VoiceXML and requirements for richer natural language interaction. The model facilitates mixed initiative across a set of narrow application focussed domains.
Setting the scene - my role in the web - the restricted nature of current voice-based human-machine dialogs - examples of a richer interaction style - the need for humility in the face of human intelligence - the opportunity for a modest extension in dialog capabilities.
I have been involved in the Web for many years, helping to drive the development of standards for HTML, HTTP and more recently work on voice browsing and multimodal interaction. HTML has enabled people to access content and services right across the world at the click of a button. HTML has been used to create a rich visual experience, but is not well suited for aural interaction. Work on aural style sheets has made it possible to style HTML when rendered to speech in combination with keyboard input, but the prevalence of table-based visual markup has made it difficult for people with visual impairments to easily browse visual web content. A better solution would help all of us when there is a need for hands and eyes free operation, or when we don't have access to a computer. At the time of writing there are well over a billion phones world-wide, could these be adapted to provide an effective means to access Web services? An affirmative answer would have a dramatic impact on the Web.
Speaker dependent speech recognition has been used for several years in dictation products, e.g. Scansoft's Dragon Dictate and IBM's ViaVoice. These products require the user to train the system to their voice to attain an adequate level of accuracy. More recently, speaker independent continuous speech recognition software has become available. This is made possible by using context free grammars to dramatically constrain the recognition task. The user is conditioned to respond within the scope of the grammar via carefully chosen prompts. This can be combined with word or phrase spotting techniques.
The need to write speech applications as complex programs is a powerful inhibitor for would be developers. As a result, a number of companies began to explore the use of markup as a means to reduce the effort needed from application developers. Some examples include, PML from AT&T and Lucent, SpeechML from IBM, VoxML from Motorola, and my own work at HP Labs on TalkML. These have focussed on menuing and form filling as metaphors for user interaction. AT&T, IBM, Lucent and Motorola subsequently pooled their efforts to merge their experience into a joint design for a new language called VoiceXML, This work was later picked up by W3C's Voice Browser working group and supplemented by additional work on markup specifications for speech grammars and speech synthesis, drawing upon work by Sun Microsystems.
The successful features, e.g. navigation links (main menu), form filling metaphor, tapered prompts, barge-in, traffic-lights model for confirmations. Flexibility through a judiscious mix of declarative and procedural elements. Mixed initiative in VoiceXML.
VoiceXML is being successfully deployed by wireless and wireline telephone network operators, and by companies for various kinds of call centers. A tutorial on VoiceXML is available on the W3C site. Users dial up to connect to a voice browser running a VoiceXML interpreter. This in turn contacts a web server to request the corresponding VoiceXML document. An application may extend across several VoiceXML documents. Developers are comfortable with markup and exploit their skills at dynamically generating markup on the fly, and providing for a division of labor between web servers and backend application servers.
VoiceXML supports global navigation links and form filling via the <link/> and <form>...</form> elements. VoiceXML supports the use of grammars for both speech recognition and DTMF (touch tone) input. For forms you can set form-level and field-level grammars. The results of speech recognition are treated either as activating a link or as setting the values of one or more named variables. There is no explicit model of dialog history. VoiceXML offers a judiscious mix of declarative and procedural features, with the ability to use ECMAScript for dynamically computed attribute values, and the ability to define event handlers in various scopes.
VoiceXML applications are generally based upon a system directed dialog where the application does most of the talking and the user responds with short simple utterances. As an example, here is a fictious application for ordering pizza: [play it]
Computer: Welcome to Joe's Pizza ordering service Computer: Select pizza size from large, medium or small? User: large Computer: what number of these pizzas do you want? User: two Computer: Select first topping from mozzarella, pepperoni and anchovies? User: mozzarella Computer: Do you want another topping, yes or no? User: yes Computer: Select second topping from mozzarella, pepperoni and anchovies? User: pepperoni Computer: Do you want any other pizzas, yes or no? ...
The prompts are designed to elicit very simple responses, thereby avoiding the difficulties of dealing with all the possible variations in responses such as "yeah sure, I er would like large pizzas". If the user doesn't answer in a reasonable time, the application repeats the prompt, perhaps rewording it. If the answer doesn't match the grammar, the application provides guidance, for example:
Computer: what number of these pizzas do you want? User: I reckon two would do the job Computer: please say the number on its own User: two Computer: Select first topping from mozarella, pepperoni and anchovies? ...
The dialog gets the job done, but is very rigid. With larger grammars, a more natural interaction style becomes possible, for example: [play it]
Computer: Welcome to Joe's Pizza Computer: What would you like? User: I would like two large pizzas with mozzarella and one small pizza with tomatoes and anchovies Computer: would you like any drinks with that? User: Sure, 3 large diet cokes, oh and add pepperoni to the large pizzas Computer: Is that all? User: yes Computer: Okay, that will be ready for you in 5 minutes User: thanks
In this example, the application starts with an open ended prompt. The context should be sufficient to guide the user to respond within the domain defined by the application. If the user's response can't be understood, the application provides guidance. Word spotting can be used as part of this process, where the presence of particular words triggers particular behaviors.
The example involves a structured data model going beyond the limits of flat lists of name/value pairs. The user's second response modifies information provided in the first response, necessitating some kind of query against the current state of the application data. This is something that would be hard to do with VoiceXML.
Visual interfaces based upon HTML are event driven and controlled by the user. This is very different from the system directed dialogs prevalent with VoiceXML. Microsoft's SALT proposal extends HTML to trigger speech prompts and activate speech grammars via HTML events, such as onload, onfocus, onmouseover and onclick etc. The results of speech recognition are handled in two steps. The first is for the recognizer to apply the speech grammar to the spoken utterance to create an annotated XML representation of the parse tree. The second step is to use an XPath expression to extract data from this tree and to insert it into a named variable.
SALT doesn't provide much in the way of declarative support for dialogs. As a result SALT applications tend to involve plenty of scripting. By contrast, VoiceXML is reasonably good for representing dialogs, but poor when it comes to event driven behavior. What is needed is a dialog model that supports the best of both approaches.
W3C's work on multimodal interaction aims to support synchronization across multiple modalities and devices with a wide range of capabilities. The vision of a multimodal interface to the Web in every pocket calls for an architecture suitable for low end devices. This necessitates a distributed approach with network based servers taking on tasks which are intensive in either computation, memory or bandwidth. Examples include speech recognition, pre-recorded prompts, speech grammars, concatenative speech synthesis, rich dialogs and natural language understanding.
W3C's vision of multimodal also includes the use of electronic ink as produced by a stylus, brush or other tool. IBM, Intel and Motorola have proposed an XML format for transferring ink across the network. This would enable the use of ink for text input, for gestures used as a means of control, for specialized notations such as mathematics, music and chemistry, and for diagrams and artwork. Ink is not restricted to flat two dimensional surfces, and in principle can be applied to curved surfaces or three dimensional spaces. It is thus a goal for multimodal dialog frameworks to address the use of ink.
Commercial offerings like General Magic's "Portico" and Orange's "Wildfire" provide users with personal assistants that allow you to browse mail boxes, listen to messages, compose and send messages, dial by name from your contact list, request and review appointments, listen to selected news channels and so forth.
This notion of a personal assistant can be considered as a group of intersecting application subdomains. In current systems, users are required to remember a set of navigation commands that move you from one subdomain to another. In some systems you have to say "main menu" to return to the top-level before issuing the command to move to the next subdomain of interest. A richer dialog model should allow you to move naturally between different subdomains without such restrictions.
VoiceXML supports the dialog model where you have permanently active navigation commands, together with task specific form filling dialogs, only one of which is active at any given time. It seems natural to consider a more flexible model whereby many tasks can be active at the same time, and waiting for the user to say something relevant to that task. Perhaps we can define a task based architecture as an evolutionary step beyond VoiceXML?
Navigation links and form fields in VoiceXML can be seen as examples of a more general notion of tasks, and suggests an approach involving a dialog interpreter that supports sets of active and pending tasks, where each task has a name and a priority ...
The previous sections have established the motivation for studying a more elaborate model for multimodal dialogs. Such a model doesn't spring fully formed out of the blue, so what follows should be considered as a preliminary sketch. Let's start with some ideas about tasks:
To make it easier for application developers, tasks should be represented declaratively. In the context of the Web this suggests markup. For instance, you could specify a task that is triggered by a mouse click, but which is only active between specified start and stop conditions. The corresponding markup could be derived from W3C's SMIL and XML Events specifications. The means to express actions will be discussed below following a consideration of how to approach natural language understanding.
To allow for richer voice interaction, a reasonable premise is for multiple grammars to be active at any time, and corresponding to different tasks. When the user says something that matches an active grammar, the utterance is handled by the task associated with that grammar. What if the utterance matches several grammars? This could happen because more than one task has activated the same grammar, or more likely, because the recognizer isn't quite sure what the user said. The solution is to prioritize tasks. The priorities can then be taken into account as part of the recognition process and combined with the recognition uncertainties to determine the most likely interpretation.
This is perhaps the most tricky area to deal with due to our very incomplete understanding of how the human brain operates. Language carries information at multiple levels and assumes a huge amount of knowledge about the world. Common sense is easy for people but intractable for machines, at least at the current state of technology. To get anywhere, it is critical to dramatically constrain natural language understanding to a narrow area that is amenable to a mechanical treatment, and within the scope of application developers.
Speech grammars define the set of expected utterances and are used to guide the recognizer. The output from the recognizer can be defined as an annotated natural language parse tree represented in XML. By defining the ouput of the recognizer as the most likely parse tree, there is a considerable loss of information compared with that available to the recognizer itself. This is a trade-off. A simplified representation makes it easier to apply subsequent stages of natural language processing, as compared with a richer representation giving the estimated likelihoods of a plurality of interpretations (for instance, a lattice of possible phoneme sequences).
Speech technology vendors have worked long and hard to improve the robustness of speech recognition for things like numbers, currency values, dates, times, phone numbers and credit card details. It therefore makes sense to incorporate the results of such processing into the output from the recognizer. The output is the most likely natural language parse tree, annotated with recognition confidence scores and the results of semantic preprocessing by recognizers. W3C has been working on an XML representation for this, called NLSML or natural language semantics markup language. This work is still at an early stage and may well change name by the time it is done.
The next step is to apply natural language understanding rules to interpret the utterance in the context of the current task and application state. The result is a sequence of actions to be performed. The actions cover such things as changing the application state, starting and stopping other tasks, following links, changing the dialog focus and so on. See the earlier section on tasks for other ideas. How should these natural language understanding rules be represented and what do they need to be capable of?
One posibility is support a sequence of if-then rules where the "if" part (the antecedent) operates on the output of the recognizer, the current application state, task specific data, and the dialog history. The "then" part (the consequent) specifies actions, but also can access information passed to it from the antecedent, and from the same sources as are available to the antecedent. These rules could be directly associated with grammar rules or could be bound to grammars at the task level. The rules could in turn invoke additional rule sets (modules).
The detailed representation of these rules is likely to be a contentious issue. XML experts will probably place a premium on consistency with existing XML specifications, for instance XPath and XSLT. Others who place a premium on simplicity for end-users may prefer a more consise and easier to learn syntax that is closer to conventional programming languages. For added flexibility it would be advisable to allow for breaking out to a general purpose scripting language such as ECMAScript, or a rule oriented language such as Prolog.
Tasks may provide locally scoped data. This corresponds to locally scoped variables in subroutines in common programming languages. This information is hidden from other tasks, unless exposed through defined methods. This assumes that tasks can be treated as objects with methods. An object-oriented approach blends declarative and procedural styles, and makes it straightforward for tasks to provide appropriate behaviors in response to a variety of events.
For many applications there will be a need for richly structured application information, whether this is for ordering pizza or for a personal assistant with access to mail boxes, contact lists and appointment calendars. Application developers will need a consistent interface to this data, and it is not unreasonable to do so via XML. This doesn't mean that data is expressed internally as XML files, but rather that the interface to the data can be handled via operations on XML structures.
In some cases, this may involve a time consuming transaction with a back-end system, e.g. a database on another server. Application developers need to be aware of such delays when designing the interaction with the end-user. For delays of about two seconds or longer, it is necessary to let the user know that some time consuming task is underway. A tick-tock sound effect is sometimes used as the aural equivalent of an hour glass. For longer delays, it is worth considering how to involve the user in some other activity until the task has been completed.
Sometimes the user might refer back to something mentionned earlier in the dialog. It may be possible to handle this in terms of a reference to the current application state, otherwise, it is necessary to maintain a representation of the sequence of prompts and responses. Observations of human short term memory suggest that only a small number turns need to be available. The dialog history can be represented at several levels, for instance:
The dialog history is accessible by the antecedents and consequents of the natural language understanding (NLU) rules. Linguistic phenomena such as anaphora, deixis, and ellipsis can be treated in terms of operations by the NLU rules on the current or preceding utterances. Anaphoric references include pronouns and definite noun phrases that refer to something that was mentioned in the preceding linguistic context, by contrast, deictic references refer to something that is present in the non-linguistic context. Ellipsis is where some words have been left out when the context makes it "obvious" what is missing. If the NLU rules aren't able to make sense of the utterance then application developers should provide some fall back behavior.
Application developers may want to allow the user to make responses that combine multiple modalities. One example is where the user is shown a street map centered around his/her current position. The user might ask how long it would take to walk to "here" while clicking on the map with a stylus. The NLU rules in this case would have to search the dialog history for positional information as recorded by the handler for the click event.
The need to support a mass market of low-end devices makes it imperative to provide a distributed architecture. The Web already has a model of events, as introduced into HTML, the next step will be to extend this across the network.
The events are divided into actions and notifications. Actions are events that cause a change of state, while notifications are events that are thrown as a result of such changes. Here are some examples:
Events can effect user interface specific features or modality independent abtractions. For example, when the user says a command to follow a link, this could be targeted at a button in the visual interface, resulting in this button appearing to depress momentarily. If the action is targeted at the page, the button won't be effected.
The XML Events specification describes markup for use in binding handlers to events following the model defined in the W3C DOM2 Recommendation. The framework needs to be extended to support the notion of action events, and to describe the representation of events as XML messages. This can be kept separate from the underlying transport protocols. In 2.5G and 3G mobile networks, the IETF SIP events specification looks like a natural fit.
In an asynchronous system, care needs to be taken to avoid inconsistencies arising. In one example, the user says something to select a choice from a menu, but then uses the stylus to tap on different choice on the same menu. In the time taken to recognize the speech and send the corresponding action, the visual interface will have already changed the value, based upon the stylus tap.
The simplest policy is to apply actions in the order they are received. An alternative would be to include a time stamp and to ignore an action that occurred before the latest action that was applied. If a more sophistocated approach is needed, it may be feasible to define script handlers that intercept the actions before they are applied.
Dialog models involving explicit turn taking provide a further basis for synchronization. The events are tagged with the turn, and this can be used to identify events that arrive out of turn. Further work is needed to understand how turn taking relates to the user interface model in XHTML.
One idea is to use an identifier corresponding to the web page. If an event is delivered after the page has changed, the event can be easily discarded or directed to an appropriate handler. For applications that last over multiple web pages, a session context seems appropriate, and fits with existing ideas for WML and VoiceXML.
When it comes to actions that change the structure and content of a document, then it would be interesting to compare and contrast approaches based upon transferring small scripts (scriplets) and more declarative approaches based upon markup. In both cases, it may be necessary to consider security mechanisms to avoid problems with hostile third parties intervening in the dialog between devices and servers.
This paper has presented an analysis of the requirements for multimodal dialogs and proposed a sketch of a task-based architecture using events for synchronization across modalities and devices. It is to be hoped that this paper will help to stimulate further discussion bridging the academic and commercial communities. Experience has shown that it takes several years to create Web standards. Now is the time to ensure that the next generation of Web user interfaces are grounded on solid review by both communities.