From W3C Wiki
Jump to: navigation, search

Session on fusing user input and sensor data

The goal of the session was to identify use cases and requirements for integrating user input with sensor data. We identified 8 use cases and 5 requirements.

Use Cases

  1. Medical: relate medical sensor data (temperature, blood pressure, blood sugar, heart rate, etc.) with patient reports, patient's tone of voice
  2. Self-driving car: By listening to when people are speaking, interrupts passengers at the right time (to notify arrival, or to notify about interesting places en route)
  3. Weather information: relate weather sensor data (temperature, wind, barometric pressure, humidity) with user reports, use results on a weather web page or web signage
  4. Appliances: user reports ("the water is too hot") the application automatically changes the water temperature
  5. Instruction involving motion (sports, musical instruments, dancing): teacher/coach instructions + student behavior (as captured by motion sensors or camera). The teachers says "move your arm higher" and the system reports how the user moves his/her arm.
  6. Tagging images: integrate geotagging and image recognition information with speech/text input like "that's Tim" while pointing at a person in the image.
  7. Traffic: relate car speeds to (possibly multiple) user reports ("accident", "road work"). The system can then say "traffic is slow for the next five miles due to road construction."
  8. Biometrics: relate a claim of identity to a biometric ("I'm Debbie" + speaker verification)
  9. Smart TV: voice search for programs, control TV (volume, channel)


  1. representations for user input and sensor input
  2. ability to synchronize inputs in terms of time
  3. ability to represent inputs from multiple users and multiple sensors (for use cases like traffic)
  4. ability to track changes over time across inputs
  5. strategies for resolving inconsistencies

Possible tools

  • EMMA for representing user inputs
  • Gesture Markup Language (GML) for representing gestures
  • Behavior Markup Language (BML) for representing facial expressions