Devices and Sensors WG TPAC F2F - Day 2/2

Welcome and introductions

anssik: I'm Anssi Kostiainen from Intel, co-chair of the group

reillyg: I'm Reilly Grant from Google, co-chair of the group

Flaki: I'm Flaki from Mozilla

Lars: Invited Expert of the group

Kenneth Christiansen: I'm Kenneth Christiansen from Intel

Yang_Gu: I'm Yang Gu from Intel

Mike_ONeill: I'm Mike ONeill, observer

Heejin_Chung: I'm Heejin Chung from Samsung

[more intro]

anssik: We'll discuss the findings from the W3C Workshop on Permissions and User Consent

Agenda bashing

Anssi: After that briefing, we have 2 hours to discuss Generic Sensor Level 2
… Bryan Bernhart pointed out @@ yesterday
… with Reilly we found out three high level stuff to discuss
… exposing sensors in service workers, which is highly controversial
… another high level topic is high frequency & batching
… Lars will share his findings about sensor discovery

reillyg: An additional topic is the relation to rAF

anssik: for Orientation Sensor, there's a proposal to provide Euler angles
… after lunch, we'll have discussions about WebDriver Extension API for Generic Sensor
… Reilly did some research of Web Bluetooth

anssik: We'll also discuss Shape Detection API, WebUSB, WebHID and Serial APIs
… they're kind of in the scope of the working group
… there're more people in the room, could you briefly introduce yourself?

Zoltan_Kis: I'm Zoltan Kis, working on Web of Things API

Sam_Weiler: I'm Sam Weiler from W3C, working on Security and Privacy

Thomas_Nattestad: I'm Thomas Nattestad from Google

anssik: Thomas will report the findings from the permissions workshop

Findings from the W3C Workshop on Permissions and User Consent

anssik: There were good browser vendors attendance in the workshop

<weiler> https://‌www.w3.org/‌Privacy/‌permissions-ws-2018/‌cfp.html

<weiler> https://‌www.w3.org/‌Privacy/‌permissions-ws-2018/‌schedule.html

<anssik> W3C Workshop on Permissions and User Consent Schedule

Thomas: We can go through the schedule

Thomas: @@
… another big problem re permissions is upfront notification about permissions
… how users should be able to control their settings
… how much we should be expecting from the users
… we had interesting discussions on XR
… ask permissions in contexts, e.g., in AR/VR experiences
… a "would you like this website to give you VR experience?" prompt
… the users would need to understand the implications

Anssi: the UI/UX of the permissions can be improved too
… similar to on-boarding of apps
… animation/interaction instead of a string
… can we have an alternative way to do permissions prompt?

johnpallett: a big challenge of VR in general is that users should understand what VR does
… understand what all the computer vision algorithms can do with data from camera

anssik: like time-bound permissions discussed yesterday

weiler: real-world geometry

<johnpallett> link to Immersive Web CG repo for privacy & security is here: https://‌github.com/‌immersive-web/‌privacy-and-security

<johnpallett> Explainer in progress capturing threat vectors and possible mitigations here: https://‌github.com/‌immersive-web/‌privacy-and-security/‌blob/‌master/‌EXPLAINER.md

anssik: we need to find a novel way to animate/visualize the permissions

reillyg: @@
… how privacy-sensitive low-level data makes sense in high level

John_Pallett: @@ bundling not just permissions, but also information the users should know

Thomas: there're two very different kind of permissions
… one is the really dangerous ones, blackmail etc.
… the other is more data
… geolocation is in the second camp
… it can be used for ad-targeting

reillyg: even for the geolocation case, it can be in the first camp
… some users do not want to share their location, and may not know that their geolocation data was used by the website

Thomas: currently, fingerprinting is fairly easy
… in the future we can use code analysis to detect if the website is using permissions they're not supposed to use
… instead of disabling the feature completely

Mike_ONeill: API of metadata of permissions

Thomas: what if people lie?

Mike_ONeill: then they break the law

Mike_ONeill: @@

anssik: the concept of URL doesn't exist for normal people
… e.g., iOS Safari only displays the domain name
… example.com uses [placeholder] data for [how long]

[scribe missed some parts]

anssik: you need to give the developers the incentive, otherwise they'll ask more than they need
… that's why better UX is needed

weiler: a question we must be asking, is if we should add this at all
… some users might not be able to understand the depth of risk they're taking here

Thomas: Electron as an example
… people use it because they need it
… but it's very insecure sometimes

Thomas: perhaps PWA installation is not as much trust as native installation

reillyg: in the Chrome extension manifest, there are required permissions, and optional permissions

<tomayac> Permissions could be listed in a `permissions` property in the Web App Manifest

<tomayac> And then grouped by level of “scariness”, which would allow less scary ones to be bundled in prompting

weiler: Nick Doty did write a guide for permissions
… it might help

https://‌lists.w3.org/‌Archives/‌Public/‌public-2018-permissions-ws/‌2018Oct/‌0000.html

<tomayac> “permissions”: [{“push_notifications”: {“rationale”: [“Inform the user of new content they have shown interest in”, “Alert the user of general breaking news”]}}]

weiler: changes to oauth about how oauth permissions might be used
… different trust of the browser chrome vs the web page
… there was a workshop about permissions in 2014

https://‌www.w3.org/‌2014/‌07/‌permissions/

anssik: is the guide created by Nick for spec authors?

weiler: yes
… spec authors, or browser vendors

weiler: for the privacy bits, which is mostly this is about
… helping their effort is probably the best way to push this forward

anssik: some privacy experts might not be UI/UX experts

<tomayac> The spec could add recommendations as to the bundling of permissions (based on “once”, “permanent”,…)

Resolved: Permissions being a cross-cutting concern across multiple groups we believe that we should find a forum for discussing these issues and that specifications should provide developers with affordances for expressing the scope of permission requests (i.e. duration and bundling) and the rationale for the permission request so that browser vendors can process that data to inform their UX.

Action: ThomasTheDane to connect Reilly with relevant Chrome PM to explore cross-group cutting group affordances

<trackbot> Error finding 'ThomasTheDane'. You can review and register nicknames at <https://‌www.w3.org/‌2009/‌dap/‌track/‌users>.

Ada: Ada from Samsung IWWG co-chair, interested in ALS

Ambient Light Sensor

Ada: interested in IW use cases to allow the model change according to the ambient light of the environment

Reilly: ALS API has an open PR for RGB support in addition to intensity
… encourage to note the IW use case in the respective issue https://‌github.com/‌w3c/‌ambient-light/‌issues/‌9

Ada: feedback from the developer is that this is highly requested
… without this support, need to extract this RGB information from camera

Reilly: if we didn't provide ALS as a dedicated API for RGB, you could attach an instance of the AmbientLightSensor to the XRDevice interface with that as an extension

<ada> https://‌github.com/‌immersive-web/‌proposals/‌issues/‌27

Reilly: also Gamepad could reuse e.g. Accelerometer

[agreement]

Resolved: promote RGB feature to Level 1 to address the Immersive Web use case

Generic Sensor Level 2

Generic Sensors Level 2

Reilly: when it comes to Workers, we generally agreed that adding support for dedicated workers has not issues, since they're attached to a frame
… benefit is that will help avoid jank

anssik: what is the implementation effort to expose to workers?

Reilly: not significant

Tom: what can you do in the background with WebBluetooth API?

Reilly: for WebUSB and WebBT we keep the connection open even if the page is not visible, provide UI indicators to keep the user informed
… unlike sensors, when you're connected to a device, there's an expectation there's shared state you do not want to interfere with
… a bi-direction connection is different from delivering events one-way

anssik: what are the other features exposed to dedicated workers?

Reilly: IndexedDB, Fetch, XHR at least
… within the limited scope of dedicated workers, as long as we can spec how that feature interacts with events in a frame, no known concern how that's done

anssik: this could be polyfilled, but that'd have worse performance

Resolved: Start work to expose sensor APIs to dedicated workers as they track an active frame and so there are no additional permissions questions.

Reilly: there's three categories for background work: 1) receiving the sensor events in the background, e.g. GPS logging app in realtime or batched 2) summarized view that is not available real-time, but can be queried e.g. fitness tracker 3) geofencing in which you set a trigger condition

Reilly: two different use cases: e.g. active tracking for navigation and one for geofencing

Tom: advise from Jake, long running tasks not applicable for service worker

Tom: you'd need to use system wake lock https://‌w3c.github.io/‌wake-lock/#dfn-system-wake-lock to keep the app running

Screen wake lock" and "System wake lock

Reilly: for geofencing we have a path to follow for Service Workers

(obsoleted) Geofencing API

<tomayac_> The paper I keep referring to: https://‌t.co/‌OxiuEg8uVL

Background sensor use cases

anssik: suggest we talk to Marijn to learn from the past

larsgk: how about combination of multiple events as a trigger?

Reilly: agreement?

[silence]

Resolved: Investigate further Geolocation and Wake Lock integration to allow active sensor tracking

Action: Reilly to talk to Marijn to understand why Geofencing API work was abandoned

<trackbot> Created ACTION-814 - Talk to marijn to understand why geofencing api work was abandoned [on Reilly Grant - due 2018-10-30].

Action: Tom to talk Geofencing at the Service Workers WG F2F, seek that group's feedback

<trackbot> Error finding 'Tom'. You can review and register nicknames at <https://‌www.w3.org/‌2009/‌dap/‌track/‌users>.

Sensor discovery

<larsgk> https://‌github.com/‌larsgk/‌imo/‌blob/‌master/‌GenericSensorDiscovery.md

larsgk: has been working with sensors and actuators for a number of years
… recently working with different companies helping them move from native stacks to the web, in medical and industrial space
… sensor discovery an issue, would be nice to have a system to add and remove sensors from the generic sensor context
… few cases: most mobile phones have sensors of varying quality, so you could detect and use sensors around you via Generic Sensor API, do sensor fusion
… vehicle monitoring, how to interact with sensors in cars, monitor the health of your car, integrate with weather services, optimize UX of connecting to external sensors
… able to interact with sensors e.g. in a supermarket dynamically

[discovery use cases described in https://‌github.com/‌larsgk/‌imo/‌blob/‌master/‌GenericSensorDiscovery.md#sensor-discovery]

<tomayac> Regarding my AI: https://‌github.com/‌w3c/‌ServiceWorker/‌issues/‌1303

<tomayac> Sorry, deep link got eaten: https://‌github.com/‌w3c/‌ServiceWorker/‌issues/‌1303#issuecomment-432195867

tapping NFC tag would be considered an implicit trust signal https://‌w3c.github.io/‌permissions/#implicit-signals

Reilly: unlikely browsers implement industrial sensor specific features, those requirements can be addressed with WebUSB and WebBT
… would be good to see whether we can get someone build a polyfill on top of WebBT

<wanming> anyone on the webex?

<rakuco> not yet, I think

<wanming> isn't the meeting started at 2pm?

<rakuco> we need to finish discussing some items from the previous session, and anssik's returning from lunch

<wanming> got, thanks!

<rakuco> let me know if it gets too late for you though

<wanming> it doesn't matter :)

<rakuco> wanming: can you hear us right now?

<wanming> no

<wanming> no one in the webex

<wanming> have you called the webex?

<rakuco> anssik will try to set up the bridge

<wanming> thanks

<wanming> seem it needs the host joined in, @xfq_

<xfq_> ok, joining

<wanming> i see you on the webex, but still hearing nothing..

<wanming> can anyone hear me

WebDriver Extension API for Generic Sensor

WebDriver Extension API for Generic Sensors presentation slides

foolip: come bug me for anything related

[we're fixing the webex bridge bear with us]

raphael: worked on multiple Chromium and Blink parts, and sensors, Wanming is part of the QA team
… problem is a generic one, testing hardware capabilities without actual sensors impossible in an automated fashion currently
… the problem generalized across other features that interface with hardware
… target to be able to test across multiple browsers

<wanming> a bit noise

<wanming> and the volume is a bit low

[showing slides]

WebDriver Extension API for Generic Sensor PR preview

[presentation finished, slides to be shared shortly]

raphael: any questions, comments?

foolip: this is perfect for what you want to test, WebDriver is unidirectional protocol so works in this case
… worth looking at mocking geolocation, how the API would look like

Reilly: a number of questions, WebUSB needs a bi-dir connection that's not currently supported
… the Chrome-specific protocol is bi-directional
… another issue around permissions: no permission support in WebDriver

foolip: we wrote the API for that, someone just needs to implement it

Action: foolip to connect Raphael and Wanming with ChromeDriver maintainers

High frequency & batching

Reilly: initially we attempted to sync with rAF loop, but Chromium implementation was pulled since there was not a very good model
… for example, 120 Hz frequency, batching two readings per rAF loop running at 60 Hz required
… initial thoughts align with whatever input stack (keyboards and such) is doing
… primary issue around sensor timing is Provide a way of tying sensor requests to animation frames #4

anssik: what are the most valuable use cases for high frequency (>60 Hz)?

Reilly: gaming, VR/XR
… if we want to provide higher frequency input, we could gate this into e.g. XR context

<mfoltzgoogle> Pointer Events level 2 Extensions: https://‌w3c.github.io/‌pointerevents/‌extension.html#dom-pointerevent-getcoalescedevents

Flaki: SLAM, motion capture might be use cases for high frequency

John: camera and sensor alignment for AR benefits from high frequency sampling

[brainstorming use cases for high frequency]

<johnpallett> I can take an AI to follow up with people using capture to see whether alignment with sensor data would help for AR use cases

Action: John to follow up with people using capture to see whether alignment with sensor data would help for AR use cases

<trackbot> Error finding 'John'. You can review and register nicknames at <https://‌www.w3.org/‌2009/‌dap/‌track/‌users>.

DeviceOrientation Event Specification

DeviceOrientation Event Specification

[reviewing open PRs]

Open PRs

Add screen-adjusted device orientation attributes to DeviceOrientationEvent interface

[silence]

Resolved: Close the issue #10

Add deviceorientation and devicemotion feature detection conformance requirement #12

[no objections]

Add the ondeviceorientationabsolute event handler attribute #34

Reilly: Chrome usage below 0.0007%, given this extremely low consider deprecating this in Chrome
… the question to the group is, if Chrome decides not to deprecate ondeviceorientationabsolute, should we merge the PR #34?

[agreement]

Resolved: If Chrome keeps this event then this PR should be modified to add a non-normative note about the additional event available in some user agents. If Chrome deprecates then this can be closed without merging.

update rotationRate alpha, beta, gamma description to be the same as implementation #43

Reilly: propose we merge this PR, since the spec should reflect the (sometimes messy) reality of implementations

This aligns with per https://‌www.w3.org/‌TR/‌html-design-principles/#priority-of-constituencies

Make security and privacy normative #49

[no concerns]

Remove '?' from dictionary members of DeviceMotionEventInit #55

Reilly: the diff between the implementation behavior will not surface in real-world use, unlikely to break existing content

[no concerns]

Action: Reilly to discuss with Chrome API owners whether to deprecate ondeviceorientationabsolute event

<trackbot> Created ACTION-815 - Discuss with chrome api owners whether to deprecate ondeviceorientationabsolute event [on Reilly Grant - due 2018-10-30].

Action: Anssi to take an action to rebase #49

<trackbot> Created ACTION-816 - Take an action to rebase #49 [on Anssi Kostiainen - due 2018-10-30].

kenneth: could we add a note redirecting readers to the actively being worked on spec

Action: Anssi to add a note to DeviceOrientationEvent that "Implementors need to be aware that the future work is now happening on the Generic Sensor-based specifications"

<trackbot> Created ACTION-817 - Add a note to deviceorientationevent that "implementors need to be aware that the future work is now happening on the generic sensor-based specifications" [on Anssi Kostiainen - due 2018-10-30].

MarkF: is the Generic Sensor work a superset of DeviceOrientationEvent

Reilly: yes, there's also a polyfill that implements a subset of Generic Sensor atop DeviceOrientationEvent https://‌github.com/‌kenchris/‌sensor-polyfills

MarkF: can we polyfill DeviceOrientationEvent on top of Generic Sensors

Reilly: yes

Shape Detection API

[Reilly to take his chair hat off]

Reilly: scope of the API is to expose platform provided, often hardware accelerated, shape detection capabilities to the Web

<xfq> https://‌wicg.github.io/‌shape-detection-api/

Reilly: detectors: face detection (not facial detection), QR/Barcode, Text

anssik: platform varies, how you handle that

Reilly: by providing feature detection mechanism
… all these use built-in platform capabilities, nothing browser-specific

Flaki: how does this relates to WebML work

Ningxin: Face Detection API uses models built-in and provided by the underlying platform, while WebML can make use of custom models pre-trained and deployed by web developers

Action: Reilly to investigate what popular computer vision JS libraries are available.

<trackbot> Created ACTION-818 - Investigate what popular computer vision js libraries are available. [on Reilly Grant - due 2018-10-30].

[reviewing open issues https://‌github.com/‌wicg/‌shape-detection-api/‌issues/]

Sangwhan: I have concerns re feature detection, API effectively doing platform-detection that's discouraged

Flaki: barcode reading in XR context? Could that work out

Reilly: I think someone might have even written a demo for doing that

WebUSB, WebHID and Serial APIs

Reilly: WebUSB shipped in Chrome 61, WebHID and Serial API are currently under development, targeting Origin Trials soon(TM)
… Serial API simple, bi-directional stream to/from device, readable/writable stream

Reilly: on top of both USB and BT you have diverse group of devices: keyboards, cameras etc., the web provides access to some of there abstractions, what is left out are long tail devices, e.g. 3D printers

Flaki: USB and BT concerns around sending malicious code to the device, reflash the device

Reilly: in WebHID use cases, we block a set of HID device types e.g. mouse and keyboard

[Julien from Logitech joined]

Reilly: WebUSB, WebBT pioneered the file picker model, site cannot enumerate the devices on its own, only after the user selection the site gets access to the particular device

Reilly: inviting other vendors to look at these specifications, feedback encouraged, WebHID and Serial are bleeding edge so can influence the API shape

<xfq> [adjourned]