Devices and Sensors WG - TPAC 2021 vF2F Day 1

27 October 2021


Anssi_Kostiainen, Arnaud_Mandy, Chris_Needham, François_Beaufort, Fuqiao_Xue, John, Juha_Vainio, Kenneth_Christiansen, Lars_Knudsen, Marcos Caceres, Matt_Reynolds, plh, Raphael_Kubo_da_Costa, Rayan_Kanso, Reilly_Grant, Sudeep, tomayac, Vincent_Scheib
anssik, reillyg
anssik, reillyg, xfq

Meeting minutes


anssik: Co-chair, welcome.

reillyg: Co-chair, welcome.

xfq: W3C staff, handle administrative work.

arma: Intel, worked on Generic Sensors and will be presenting Compute Pressure API.

Juha: Welcome.

kenneth: Intel, currently working on the Compute Pressure API.
… Also member of the TAG.

fbeaufort: Google, Web DevRel on device APIs.
… Interested in discussing the Screen Wake Lock API.

larsgk: Working in enterprise awareness of hardware integration capabilities.

marcosc: W3C staff, focusing on Screen Wake Lock API. Also Contact Picker API, as joint deliverable with WebApps WG which I co-chair.

mattreynolds: Google, work on Device APIs
… specification editor for WebHID and Gamepad API

rakuco: Intel, working on Screen Wake Lock API and Generic Sensors APIs
… Helping others at Intel work on specifications and implementations

anssik: Marcos rules the world as editor and implementer of all specifications in all browsers.
… (Joking.)

plh: W3C staff, will give update on rechartering process

rayankans: Specification editor for the Contact Picker API.

tomayac: Google, Web DevRel, jack of all trades, master of none.

scheib: Google, manager of reillyg and mattreynolds. Cares about device capabilities.

Charter update

plh: the charter does not have a decision yet
… we reviewed the charter in July, one objection received at the end of the charter review period
… we had a call on this objection with the objector in Aug and Sep and later in Sep the case was brought to AB/TAG Council
… good news is this case is #1 on the queue
… my latest prediction is AB/TAG is going to decide to take on the case next week and have resolution by the end of November
… Council is not official yet, we're experimenting with it to be prepared for the Director-free W3C
… part of the FO is a rehash of the previous objection, we want to understand how much precedence there is
… thee new APIs to add, we got consensus on one of theme, for two of them we're lacking implementation commitments
… we asked AC reps to go on record about implementation plans

anssik: The AC balloting tool is confusing.

plh: I followed up via email.

anssik: I think AC reps are busy people and overlooked some details re implementation plans in the AC review

plh: security and privacy reviews are another point in the FO

anssik: the concrete impact of this delay is we cannot publish new work on TR track

plh: that's correct

[DRAFT] Devices and Sensors Working Group Charter

Mozilla's comments

Contact Picker API

Contact Picker API

rayankans: New privacy-forward approach to the Contact Picker API.
… Previous attempts and other APIs grant persistent access to all contacts.
… This web approach allows users to select contact information that is shared to the site as a one-time action.


tomayac: Implemented behind a flag in Safari.

rayankans: Available in Chrome on Android.

anssik: How much does the specification talk about UX expectations for the picker? How different are the Chrome and Safari implementations?

rayankans: The specification only makes requirements about security & privacy.
… Origin must be shown.
… Selection of information must be presented to the user.

anssik: In general specifications stay away from proscribing UX. This specification seems to be innovating in UI. Curious how Safari compares to Chrome.

tomayac: Safari implementation doesn't seem to allow the user to opt out of sharing particular requested fields as the Chrome implementation does.

rayankans: Safari doesn't implement support for addresses and icons at the moment.
… The API is only available on mobile.

anssik: Marcos, what do you think this API did differently to get acceptance?

marcosc: This seems very similar (also a picker UI). Might be right place / right time.
… Filtering mechanism might not have been in the original version.

Contacts API (deprecated, for historical reference only)

marcosc: Even though the functionality is similar the new API is vastly improved in terms of API shape.
… Rayan, what challenges do you see moving forward on an API level?

rayankans: How will this work on desktop?
… This is designed to be extensible.

How well with this handle addresses? Currently based on how Payments API handles addresses.
… How well with this handle addresses? Currently based on how Payments API handles addresses.
… There is a feature request for adding contacts but we don't plan to look into that.

marcosc: Same discussion on address formats in the Web Payments WG.
… Current design maps well to Contacts API on macOS and seems to work well for Google Pay.

xfq: There have been a lot of discussions around HTML autocomplete attribute and address as well.


Compute Pressure API

[slides shared]

kenneth: Google started Compute Pressure effort, we at Intel think we have things like adaptive streaming network-bound
… moving use cases to the client such as face detectione etc. become compute-bound
… deeply interested in this effort, similar goal as to adaptive streaming, ensure great UX
… currently proposal exposes normalized CPU clock speed and utilization
… we're experimenting with a new metrics that are more useful

arno: there's an implementation of this on Chromium with Chrome Origin Trial completed
… with that only CPU speed and utilization exposed, using Linux familiar procstats etc.
… we thought we can extract the lower part ("backend") into a reusable library to offer more advanced metrics to better solve real-world use cases
… would also allow easier integration into web engines
… only the metrics are given to the browser, an opportunity is to use this with not just browser but also in other contexts such as Node.js
… have a small demo proof-of-concept

[video played displaying gauges for CPU speed, CPU utilization, CPU latency, Render engine utilization]

Lars: very interested in this, and how to make this more generic, e.g. offload to cloud use cases
… maybe it could be interesting to know the CPU util of the cloud

lars: use cases is to understand whether to run the workloads on the client or on the cloud

plh: are you monitoring other piece of hardware e.g. neural network accelerators?

kenneth: the scope is XPU, whatever hardware is there on the machine, we are not ruling out anything

marcos: wondering can you monitor the overall system load, not just your app?

kenneth: the use case is to allow do the same as native apps today, e.g. Zoom can do that
… bucketing is used to mitigate exposure
… sounds familiar to Battery API, where the web developer and OS are both trying to optimize

kenneth: Allows applications to avoid pressure in order to maintain better overall system performance.

anssik: Doesn't the OS handle this?

kenneth: OS know exactly what the app is doing, can lower priority, even stop the process
… this is a hint an app can utilize to optimize UX

kenneth: The OS doesn't know what your application is doing. The application knows which features it can disabled in order to reduce load gracefully.

reillyg: if the user is unlikely to notice start bitcoin mining use case was raised for Idle Detection API too
… detecting malicious usage of JS or compute is an active area of work for browsers
… think ad blockers fighting for the pieces of JS the users don't want to run
… we cannot design a programming language that does not allow doing math people don't want
… marcos asked on use cases how native apps are doing this


reillyg: WebKit position on Compute Pressure was interesting in that it makes a claim this kind of reaction to compute pressure is not the right way
… what specific signals are useful to the developer is a key question I think
… it is interesting data point that developers are using these signals
… also interesting to know WebKit engineers think developers should not use these signals
… some disagreement on what are the best practices, and probably Intel folks are in best position to tell how to best utilize a particular type of chip

sudeep: had a breakout on edge computing at TPAC, offloading web worker tasks

[edge here not meaning Edge the browser obviously :-)]

sudeep: wanted to bring this perspective on edge offload for this API

<plh> Simple Network Management Protocol

lars: I've done some work in SNMP, wondering if there's a connection to this API

reilly: the questions are which metrics are the most useful here, interesting are that in the Intel demo Cycles Per Instruction (CPI) is used
… addressing a claim that an app wants to use all the resources if available
… e.g. you don't want the web app to use all available disk space
… we should provide the capability to allow web apps implement the best perf they possibly can
… in order to do so you need information on the current system conditions
… maybe OS can do a better job here, maybe more generic pressure signals help

marcosc: I agree with Reilly, I'm running parallel apps and as a user I don't know who is using and needs more CPU cycles

<marcosc> MC: but how can you know what CPU % is being utilized by your particular app?

marcosc: how do you know what % of CPU is used by your app and how much pressure there is

arno: we are thinking of CPU vs. the whole platform available compute
… thinking of process-based metrics

reillyg: suggesting as a next step to get feedback from developers who are using compute pressure signals on other platforms
… build examples apps that make use of this to implement real-world use cases
… show the difference between an app that tries to keep load under control and another that uses as much as possible compute available

scheib: use cases are helpful, I'd suggest there are use cases that have optional workload and in those we see the most benefit
… "nice to have" functionality will be scaled if compute is available
… UA cannot know if something is optional and cannot solve that problem

Screen Wake Lock API

Screen brightness mode

rakuco: Strong signal from developers that there are use cases for this API.

tomayac: Surprising new use case for video player.
… Video players already take a screen wake lock.
… Consider splitting screen brightness from wake lock.
… Previous use cases: barcode scanning and illumination for camera

rakuco: Before Netflix case came up there was clarity that the use case was only for raising the brightness.

rakuco: Open questions: Tie this into Screen Wake Lock API?
… Tie this to fullscreen?
… Timeout?
… Permission model?
… Granularity?
… How bright is too bright?
… Should this be supported on non-mobile.

anssik: We should schedule a separate meeting to discuss all of these questions.
… Re the Netflix use case: Is this a common use case?

reillyg: I'm a bit unsure about the Netflix use case, is this really something you need an API for or should a browser provide a setting for this, maybe video players should integrate brightness into video controls
… the video playing use case seems different from use cases for which we have consensus on: this video needs to be brighter
… this hint is easier to spec that something for generic control over brightness
… are we building an API for something that already exists somewhere as a setting?
… we can think exposing screen brightness as video controls, maybe a declarative API is not need for that?

anssik: Francois can you pull in some Netflix people to understand this use cases in more detail whether e.g. video controls would address this?

Francois: on desktop, users do not see native controls but custom controls provided by web sites
… need some API to provide these custom controls

marcosc: agree with reilly and francois, in particular Netflix use case is interesting and I personally use it a lot, native UI control makes a lot of sense, needs discussion with HTML and Media groups at W3C as a generic thing
… QR code is important use case and needs to addressed

reillyg: brightness is system-wide control, while e.g. volume is app specific

rakuco: like to get help from the group to make progress with these issues

people interested in Screen brightness Anssi, Fuqiao, Reilly, Raphael, Francois, Thomas Marcos

ACTION: anssik to set up a follow-up meeting to discuss Brightness Mode, participants; Anssi, Fuqiao, Reilly, Raphael, Francois, Thomas, Marcos, Lars

Idle Detection API

reillyg: the update on Idle Detection API since our last meeting, there's a complete spec for this and it went through Origin Trial in Chrome
… working with partners interested in the API, result of the OT was that it worked as intended
… use case, if the user is not using the app on desktop the app cannot know if they should send notifications to the mobile or on the desktop
… not having this API caused a lot of user frustration per partner feedback
… response rate to messages remained the same, users got the messages but not unnecessary notifications
… the API is behind a permission prompt, and ongoing work is to try to implement adding fuzziness to the timing of timing events
… this hides global state
… mitigating cross-site vectors is work in progress, standard mitigations for events is to only allow sites visible receive the event
… the point of this API is a site that is not visible to get the event, so we add additional mitigations for that

tomayac: is there a plan to have this limited to an origin?
… I implemented this for a popular PWA and they do not need all the features of the API, only for their own origin

reillyg: my feeling is that since this can be polyfilled, it doesn't seem worthwhile to increase the complexity of the API

tomayac: I agree, brought this up because I heard developer feedback in this regard

anssik: Intel is interested this API and we are working on alternative backends

<tomayac> Closed https://github.com/WICG/idle-detection/issues/36#issuecomment-952619522

anssik: prosing we defer Network Information API to tomorrow, agenda updated accordingly


anssik: thank you everyone for joining us today, see y'all tomorrow at the same time!

Summary of action items

  1. anssik to set up a follow-up meeting to discuss Brightness Mode, participants; Anssi, Fuqiao, Reilly, Raphael, Francois, Thomas, Marcos, Lars
Minutes manually created (not a transcript), formatted by scribe.perl version 158 (Sun Oct 17 00:40:18 2021 UTC).