Playing Well Together

W3C Technologies as Consumers of Multimodal Interfaces

Doug Schepers, W3C

SVG

• Scalable Vector Graphics
  • like HTML for graphics
• Profiles for both desktop and mobile
  • recent focus on mobile
  • increased support in browsers
• Good technology for Rich Web Applications

Web API

• DOM (Document Object Model)
  • structure, events, objects, and interfaces
  • vital for scripting and dynamic content
• Various auxiliary technologies
  • XmlHttpRequest
  • Drag & Drop / Copy & Paste
  • Selectors
  • Window Object

CDF

• Compound Document Formats
• Mixing (X)HTML, SVG, MathML, DOM, CSS, etc.
• WICD
  • Web Integrated Compound Document
  • pronounced "wicked"
  • core W3C presentation languages

Consumers of MMI

• Implementers usually primary target
• … but other specs need love, too!
  • reduce duplication of effort
  • reduce friction between groups
  • present unified message to developers
• Mostly client-side functionality
  • using XHR to access server-side functionality
  • specify balance between client- and server-side

Pointer Pressure

• InkML
  • APIs, or just markup?
  • limited set of “channels” expressed in pointer event?
• Special uses in SVG
  • drawing applications
  • pressure to inform stroke-width or depth of color
  • threshold trigger for drag & drop
• UI responsiveness?

Voice

• How to interact with DOM?
  • text events?
  • IME?
• Speech recognition
  • “macro” (keyword) functionality
  • creating SVG shapes or other content via voice?

DCCI

• Geolocation
  • GPS vs. cell triangulation vs. IP address vs. manual input
• Battery life
• Volume, mute
  • decisions on what content to send down
• Wireless connectivity
  • strength
  • stability
  • offline Web Apps?
• Exposed in DOM via Window interface?

Multiple Pointers

• Apple iPhone and Microsoft Surface
  • new UI possibilities
    • zoom in and out
    • drag & drop
    • establishing connections between objects
• Multiple users vs. multiple touch points
  • Nintendo Wii
• Very challenging from DOM perspective
  • solutions for event model?
  • “tagging” event?

Two-Way Street

• Some functionality for MMI can be taken from other technologies
  • direct liaisons
  • inspiration and use cases through evaluation
• SVG as data/event source
  • predrawn shapes as input source
  • automation through declarative animation and script
    • custom events (DOM)
• Distributed DOM
  • REX (UbiWeb)

Where Is the Intersection?

• Common infrastructure
  • need compatible models
• Tight integration into DOM
  • events and markup
• Implementer focus
  • need to get multiple modalities in desktop browsers
  • joint “marketing” strategies… push into common use
• Implementer, developer, author, and user education

Next Steps

• Need to get desktop browser vendors involved to help specify MMI
• May need to create higher-level, simplified interfaces in Interaction Domain specs (HTML, DOM, SVG, etc.) to create compatible upgrade path to full functionality