TPAC2016/session-fusing-user-and-sensor-input-summary
< TPAC2016
Session on fusing user input and sensor data
The goal of the session was to identify use cases and requirements for integrating user input with sensor data. We identified 8 use cases and 5 requirements.
Use Cases
- Medical: relate medical sensor data (temperature, blood pressure, blood sugar, heart rate, etc.) with patient reports, patient's tone of voice
- Self-driving car: By listening to when people are speaking, interrupts passengers at the right time (to notify arrival, or to notify about interesting places en route)
- Weather information: relate weather sensor data (temperature, wind, barometric pressure, humidity) with user reports, use results on a weather web page or web signage
- Appliances: user reports ("the water is too hot") the application automatically changes the water temperature
- Instruction involving motion (sports, musical instruments, dancing): teacher/coach instructions + student behavior (as captured by motion sensors or camera). The teachers says "move your arm higher" and the system reports how the user moves his/her arm.
- Tagging images: integrate geotagging and image recognition information with speech/text input like "that's Tim" while pointing at a person in the image.
- Traffic: relate car speeds to (possibly multiple) user reports ("accident", "road work"). The system can then say "traffic is slow for the next five miles due to road construction."
- Biometrics: relate a claim of identity to a biometric ("I'm Debbie" + speaker verification)
- Smart TV: voice search for programs, control TV (volume, channel)
Requirements
- representations for user input and sensor input
- ability to synchronize inputs in terms of time
- ability to represent inputs from multiple users and multiple sensors (for use cases like traffic)
- ability to track changes over time across inputs
- strategies for resolving inconsistencies
Possible tools
- EMMA for representing user inputs
- Gesture Markup Language (GML) for representing gestures
- Behavior Markup Language (BML) for representing facial expressions