13:54:12 RRSAgent has joined #webmachinelearning 13:54:12 logging to https://www.w3.org/2022/08/11-webmachinelearning-irc 13:54:14 RRSAgent, make logs Public 13:54:15 please title this meeting ("meeting: ..."), anssik 13:54:20 Meeting: WebML WG Teleconference – 11 August 2022 13:54:44 Chair: Anssi 13:55:16 Agenda: https://github.com/webmachinelearning/meetings/blob/main/telcons/2022-08-11-wg-agenda.md 13:55:21 Scribe: Anssi 13:55:25 scribeNick: anssik 13:55:34 ghurlbot, this is webmachinelearning/webnn 13:55:34 anssik, OK 13:55:40 Present+ Anssi_Kostiainen 13:55:45 Regrets+ Dominique_Hazael-Massieux 13:56:18 anssik has changed the topic to: https://github.com/webmachinelearning/meetings/tree/main/telcons 13:58:47 RRSAgent, draft minutes 13:58:47 I have made the request to generate https://www.w3.org/2022/08/11-webmachinelearning-minutes.html anssik 14:02:04 ningxin_hu has joined #webmachinelearning 14:02:36 Present+ Ningxin_Hu 14:04:00 Bruce_Dai has joined #webmachinelearning 14:04:23 Present+ Bruce_Dai 14:04:34 Present+ Chai_Chaoweeraprasit 14:05:25 Chai has joined #webmachinelearning 14:12:51 RRSAgent, draft minutes 14:12:51 I have made the request to generate https://www.w3.org/2022/08/11-webmachinelearning-minutes.html anssik 14:13:22 anssik: Welcome back after the break! 14:13:54 ... we may consider adjusting out meeting time to be more Sydney friendly 14:14:23 Topic: Privacy review response and questions from PING 14:15:17 -> 14:15:17 Web Neural Network API 2022-06-30 > 2022-09-30 https://github.com/w3cping/privacy-request/issues/96 14:15:44 anssik: PING is very happy: "Thank you to the WG for such a thorough and detailed account of the work the group has undertaken on privacy since the last review. This is the model that all WGs should follow. Thank you." 14:16:27 anssik: today, I'd like us to review as as group Privacy Interest Group's review feedback #280 that is the most subtantial 14:16:28 https://github.com/webmachinelearning/webnn/issues/280 -> Issue 280 Items from privacy review (sandandsnow) 14:16:33 ... let's discuss this feedback in parts: 14:17:14 Subtopic: Fingerprinting via execution time analysis 14:17:19 "(1) Please keep us posted with the results of your research into the extent of the threat of fingerprinting via execution time analysis and potential mitigations." 14:17:40 anssik: This suggests no changes to the spec as of now. 14:17:56 Subtopic: The ethics & privacy implications of the identified and potential use cases of the API 14:18:04 "(2) Re: The ethics (and privacy implications) of the identified and potential use cases of the API. PING observed that a number of the use cases that the API would enable are highly privacy-invasive and, therefore, should include an analysis of the privacy implications of those use cases as well as mitigations (e.g. transparency, policy and/or user controls). (Note: While the described use cases are probably only a subset of use 14:18:04 cases that this API could be used for, the examples provided in the specification should help guide the use (or not use) of the API in other situations.)" 14:19:01 anssik: the concrete feedback here is: 14:19:10 "should include an analysis of the privacy implications of those use cases as well as mitigations (e.g. transparency, policy and/or user controls)" 14:19:37 anssik: our group's approach has been to discuss the ethical implementations in the separate deliverable we published and continue to refine 14:19:43 -> Ethical Principles for Web Machine Learning https://www.w3.org/TR/webmachinelearning-ethics/ 14:20:20 anssik: one approach to address this feedback would be to link the use cases enumerated in the spec with ethical and privacy implications expanded in the Ethical Principles W3C Note 14:20:33 ... to give a concrete example, we could link "Face Recognition" use case with an "Accuracy" ethical issue 14:20:37 -> Use case: Face Recognition https://www.w3.org/TR/webnn/#usecase-face-recognition 14:20:45 -> Ethical issue: Accuracy https://www.w3.org/TR/webmachinelearning-ethics/#accuracy 14:21:02 anssik: This type of Use Case-Ethical issue mapping couldn't be complete, but provide examples 14:21:08 ... thoughts? Would this be worth it? We could ask PING whether this approach would be preferred. 14:22:35 Chai: is the purpose to keep a record for future reference? 14:24:44 Subtopic: Software implementations to eliminate or reduce compute unit scheduling from being a fingerprinting risk? 14:25:00 "(3) What are the software implementations (or examples) that would eliminate or reduce compute unit scheduling from being a fingerprinting risk? (The text says: " Furthermore, software implementations can be used to further eliminate such artifacts.")" 14:25:30 anssik: here's the full context what we say in Privacy Considerations: 14:25:36 "The WebGPU API identifies machine-specific artifacts as a privacy consideration. Given the WebNN API defines means to record an ML worload onto a WebGPU-compatible GPUCommandBuffer, compute unit scheduling may under certain circumstances introduce a fingerprint. However, similarly to WebGPU, such fingerprints are identical across most or all of the devices of each vendor, mitigating the concern. Furthermore, software 14:25:36 implementations can be used to further eliminate such artifacts." 14:25:41 -> WebNN Privacy Considerations https://www.w3.org/TR/webnn/#usecase-face-recognition 14:25:50 anssik: This clause "software implementations can be used to further eliminate such artifacts" is inherited from WebGPU spec: 14:25:54 -> WebGPU Privacy Considerations https://gpuweb.github.io/gpuweb/#privacy-machine-artifacts 14:27:07 anssik: do we have a concrete software implementation (or an abstract example if it) we could refer to here to address PING's feedback? 14:27:23 Subtopic: Future device types and their fingerprint mitigation 14:27:33 "(3) [sic] Re: "If a future version of this specification introduces support for new a device type that can only support a subset of MLOperandTypes, that may introduce a new fingerprint", what mitigations does the WG envisage should this occur?" 14:28:16 anssik: thoughts on mitigations? I suppose the less fragmented the implementations are, the less fingerprinting is a concern. I suspect there are some learning from WebGL extensions 14:28:59 Chai: I think is is a good observation, I tend to believe even if you define an API people can detect when you fail and understand what happens underneath 14:29:21 ... that may suggest an abstraction is broken, even if the app wants to use NPU it should just work from app's PoV 14:29:47 ... how it works is an implementation detail, an impl can fall back to something that can produce the result and app can detect it in some manner 14:30:02 ... for example, try infer via runtime performance 14:30:23 ... an API that fails a lot on certain device types not a good sign, hard for apps to work with such an API 14:30:54 ... this is something we think at Microsoft, how to have a robust fall back in our driver model 14:31:15 ... this is the way how we view this, it is the platform's responsibility, not the app's responsibility 14:32:23 ... the bulk of the work we're doing is to make sure the app can benefit from WebNN, we define the ops so that they're common across frameworks, if the API keeps failing, it is impossible for frameworks to use the API 14:33:34 ... automatic fallback to other device types would mitigate this concern 14:34:05 ningxin_hu: we have an explicit requirement to have the graph compiled on specific device, would this fallback break this requirement? 14:34:26 Chai: it's never 100%, unless you want CPU, even on GPU today 14:34:41 ... there are some ops that do not run on GPU, reshape etc. layout adjustment 14:34:55 ... if you look how it's implemented, it's a few lines of CPU code 14:35:03 ... I see the point of not calling it a hint 14:35:25 ... to address Dom's feedback we want to not handwave, if we say CPU we want CPU 14:35:41 ... with NPU in the future, it may fail and fallback to either CPU or GPU 14:35:53 ... it'd not mean "everything will run on NPU" 14:37:15 ningxin_hu: generic question about fingerprinting and feature detection 14:37:42 ... how do we distinguish the two, we need feature detection for some use cases 14:37:56 ... is there any design principle? 14:41:39 anssik: feature detection can be used for legit use cases too, in which case we could make it explicit to allow UAs to e.g. ask the user for permission 14:42:00 Subtopic: WebGPU Privacy Considerations dependency 14:42:06 "(4) Re: "In general, implementers of this API are expected to be familiar with the WebGPU Privacy Considerations", it would be helpful to state that implementers are expected to apply WebGPU Privacy Considerations to their implementations." 14:42:39 anssik: this suggestions seems reasonable, and I'd suggest we reword our spec accordingly. 14:42:45 ... From: "In general, implementers of this API are expected to be familiar with the WebGPU Privacy Considerations." 14:42:52 ... To: "In general, implementers of this API are expected to apply WebGPU Privacy Considerations to their implementations where applicable." 14:43:06 ... comments? Sounds good? 14:43:24 +1 14:43:45 sgtm 14:44:03 Subtopic: Understanding "power preference" fingerprinting risk 14:44:16 "(5) "Power preference indicates preference as related to the power consumption and is considered a hint only and as such does not increase entropy of the fingerprint." - If power preference is set to default, what might the user agent do, and could that reveal information about the device and/or user? (We discussed this during the PING call, but we were not able to sort out whether the hint is fingerprintable. Could you explain 14:44:16 in more detail how the power preference works so we can understand the fingerprinting risk?)" 14:45:40 anssik: I'd suggest we respond that setting a power preference does not have web-facing observable side-effects that could be used as a fingerprint. This is the case as long there is no Web API to measure power consumption. If such a Web API emerged, then this interaction needs to be considered. 14:46:34 Chai: I'd add that, we have no API that the app can call and get the answer when they set the default option (part of Dom's earlier feedback to define device type as an explicit ask) 14:46:43 ... no way to query what device is picked up 14:48:14 ningxin_hu: Anssi you're right, no API to measure power consumption, can someone argue that based on performance (latency of graph compile) this could be fingerprinted? 14:48:51 ... timing could be used to distinguish different device 14:49:05 Chai: right, but depends on what they want to find out? 14:49:31 ... the power preference can be taken into account, but in the end there's no API to ask which device is in the end being used 14:50:17 ... the implementation will still pick e.g. GPU, but will not stop selecting the GPU, power preference will not change that, GPU could be in low power or high perf mode 14:50:59 ningxin_hu: other examples: low power GPU, high performance CPU 14:51:32 ... PING's first feedback was about timing attack, I'm trying to see if that and power preference are related, or have interactions 14:51:48 Chai: timing attacks are rather hard to fully mitigate 14:52:10 ... if an app wants to measure those metrics, implementers can not do too much, e.g. how long a conv op takes to execute 14:52:25 ... mostly thinking, does the API have ways to programmatically figure that out 14:52:37 ... I don't thing we have that 14:53:45 -> High Resolution Time https://www.w3.org/TR/hr-time-3/ 14:54:21 -> HR Time Note about resolution https://www.w3.org/TR/hr-time-3/#h-note 14:55:39 anssik: I will take an action to create a response to PING 14:56:59 Topic: WebGPU and WGSL wide review request 14:57:09 -> Wide Review for WebGPU and WGSL https://lists.w3.org/Archives/Public/public-webmachinelearning-wg/2022Jul/0000.html 14:57:17 anssik: WebGPU WG has approached us: 14:57:33 "The GPU for the Web Working Group has made significant progress on first 14:57:33 versions of the WebGPU and WebGPU Shading Language (WGSL) 14:57:33 specifications. The group is now firming up the specifications with a 14:57:33 view to requesting publication as Candidate Recommendations later this 14:57:34 year. 14:57:34 14:57:34 The Working Group would like to invite you to review the latest versions 14:57:35 of the specifications:" 14:57:40 - WebGPU: https://www.w3.org/TR/webgpu/ 14:57:45 - WGSL: https://www.w3.org/TR/WGSL/ 14:58:09 anssik: the WebGPU WG sets the following expectations in terms of stability and integration with WebNN: 14:58:21 "Working drafts are updated on a continuous basis but there should be no 14:58:21 more breaking changes at this stage. Feel free to review dated versions 14:58:21 of the specifications to avoid looking at a moving target. 14:58:21 14:58:21 Integration of WebGPU with WebNN is being tracked in [1], with possible 14:58:21 WebGPU adjustments considered for a future revision of the 14:58:23 specification. If breaking changes seem warranted to interoperate with 14:58:23 WebNN, the GPU for the Web Working Group would be keen on hearing 14:58:23 requirements." 14:58:31 anssik: DL for comments in 15 October 2022 via https://github.com/gpuweb/gpuweb/issues 14:58:37 ... I'd like to note the WebNN-WebGPU interop issue is explicitly mentioned in this request 14:58:55 ... I think this is a great opportunity to reinvigorate our dialogue with the WebGPU WG by updating #2500 with changes to the API since January 2022 and future plans 14:58:56 https://github.com/webmachinelearning/webnn/issues/2500 -> Issue 2500 [not found] 14:59:00 -> Investigation: how WebNN / WebGPU interop could be happening https://github.com/gpuweb/gpuweb/issues/2500 14:59:42 anssik: we also have a related issue open re CommandBuffer usage: 14:59:46 #264 14:59:47 https://github.com/webmachinelearning/webnn/issues/264 -> Issue 264 CommandBuffer usage clarification: internal, external, both? (bbernhar) 14:59:56 ... thoughts? 15:01:36 ningxin_hu: I opened webgpu#2500, after creating this issue Chai introduced CommandBuffer and polished integration story, we could give WebGPU WG an update on the latest developments, not feedback on the WebGPU API spec but introduce a plan for WebNN-WebGPU interop 15:01:37 https://github.com/webmachinelearning/webgpu/issues/2500 -> Issue 2500 [not found] 15:03:22 s/CommandBuffer/MLCommandEncoder/ 15:03:48 Chai: Rafael is working with Bryan to resolve #264 15:05:00 ningxin_hu: should I hold on updating #2500 until Rafael and Bryan have resolution on #264? 15:06:24 RRSAgent, draft minutes 15:06:26 I have made the request to generate https://www.w3.org/2022/08/11-webmachinelearning-minutes.html anssik 15:06:52 Topic: Define ULP (unit of least precision) tolerances for testing 15:07:01 #265 15:07:02 https://github.com/webmachinelearning/webnn/issues/265 -> Issue 265 Define ULP (unit of least precision) tolerances for Conformance testing of WebNN API (BruceDai) cr 15:07:36 -> https://github.com/web-platform-tests/wpt/pull/34287 Add WebNN API operations tests #34287 15:07:37 https://github.com/webmachinelearning/webnn/issues/34287 -> Issue 34287 [not found] 15:07:51 ... Bruce has an open PR for w-p-t that proposes new ULP-based assertions 15:08:02 ... it seems Bruce is blocked on w-p-t infra team to know whether testharness.js can be augmented with these new assertions? 15:08:22 ... If that is not possible, can be add these assertions in the WebNN specific test suite and upstream them to testharness.js later? 15:08:29 ... I know Dom and possibly also other reviewers are on vacation, so thus no response 15:09:16 ningxin_hu: I think there are two blockers for Bruce's work, one is ULP-based assertions 15:09:38 ... possible solution to look into whether the assertion can be added to WebNN-specific test suite per Anssi's proposal 15:10:16 ... another one is the issue itself, #265 where Bruce proposes ULP tolerances for certain ops, open for participants feedback 15:10:35 ... if we agree with the proposal Bruce will go implement the test cases based on the proposal 15:11:15 -> Proposed ULP tolerances https://github.com/webmachinelearning/webnn/issues/265#issuecomment-1168093010 15:11:51 Chai: I can help look at these tolerances 15:12:53 RRSAgent, draft minutes 15:12:53 I have made the request to generate https://www.w3.org/2022/08/11-webmachinelearning-minutes.html anssik 15:15:05 RRSAgent, draft minutes 15:15:05 I have made the request to generate https://www.w3.org/2022/08/11-webmachinelearning-minutes.html anssik 15:15:48 s/… Bruce has/anssik: Bruce has 15:15:50 RRSAgent, draft minutes 15:15:50 I have made the request to generate https://www.w3.org/2022/08/11-webmachinelearning-minutes.html anssik 16:34:01 Zakim has left #webmachinelearning