13:59:34 RRSAgent has joined #webmachinelearning 13:59:38 logging to https://www.w3.org/2024/10/31-webmachinelearning-irc 13:59:38 Meeting: WebML WG Teleconference – 31 October 2024 13:59:38 RRSAgent, make logs Public 13:59:39 please title this meeting ("meeting: ..."), anssik 13:59:41 Chair: Anssi 13:59:46 Agenda: https://github.com/webmachinelearning/meetings/blob/main/telcons/2024-10-31-wg-agenda.md 14:00:21 Scribe: Anssi 14:00:26 scribeNick: anssik 14:00:49 ningxin has joined #webmachinelearning 14:00:55 Present+ Anssi_Kostiainen 14:01:06 Present+ Dwayne_Robinson 14:01:11 Present+ Michael_McCool 14:01:17 Present+ Mike_Wyrzykowski 14:01:25 Regrets+ Ningxin_Hu 14:02:12 RRSAgent, draft minutes 14:02:13 I have made the request to generate https://www.w3.org/2024/10/31-webmachinelearning-minutes.html anssik 14:03:08 Present+ Bryan_Bernhart 14:03:57 Present+ Talha_Gorsi 14:04:24 McCool has joined #webmachinelearning 14:04:51 Present+ Austin_Sullivan 14:05:18 zkis has joined #webmachinelearning 14:05:39 Present+ Etienne_Noel 14:05:53 Present+ Zoltan_Kis 14:05:57 RRSAgent, draft minutes 14:05:58 I have made the request to generate https://www.w3.org/2024/10/31-webmachinelearning-minutes.html anssik 14:06:25 anssik: Happy Halloween! 14:06:41 ... To kick off the meeting, let me welcome our latest new participants: 14:06:51 ... Domenic Denicola and Kenji Baheux from Google 14:06:52 ... Islam El-Ashi from Dfinity Stiftung 14:06:56 ... Christian Liebel from Thinktecture 14:07:24 ... Domenic and Kenji are working on the task-specific API using built-in models proposed to the WebML Community Group, also contributing elsewhere 14:07:57 ... Christian has built solutions with these task-specific APIs and also wrote TypeScript definitions for them https://www.npmjs.com/package/@types/dom-chromium-ai 14:08:10 ... welcome all, we look forward to working with you! 14:08:32 Topic: WebML Community Group Charter update 14:08:39 gb, this is webmachinelearning/charter 14:08:39 anssik, OK. 14:09:10 anssik: I'd like to review and discuss the proposed WebML Community Group charter update to add new task-specific APIs (Writing Assistance APIs, Translator and Language Detector APIs, Prompt API) introduced at TPAC 2024 in scope 14:09:22 -> TPAC 2024 slides for task-specific APIs https://lists.w3.org/Archives/Public/www-archive/2024Sep/att-0008/TPAC_2024_Built-in_AI_APIs.pdf 14:09:39 MikeW has joined #webmachinelearning 14:09:42 anssik: I've prepared a PR #9 for the CG charter update 14:09:43 https://github.com/webmachinelearning/charter/pull/9 -> Pull Request 9 Update Community Group Charter for 2024-> (by anssiko) 14:09:50 Present+ Mike_Wyrzykowski 14:09:59 -> Charter Preview https://raw.githack.com/webmachinelearning/charter/refs/heads/charter-2024/index.html 14:10:07 Present+ Ningxin_Hu 14:10:12 RafaelCintron has joined #webmachinelearning 14:10:13 Talha has joined #webmachinelearning 14:10:19 anssik: review comments are welcome in the PR, on this call, or via email 14:11:03 ... I'd like to note the updated scope for proposed Community Group extends beyond the current scope of the Web Machine Learning Working Group 14:11:09 ... in a nutshell, WebML CG incubates new web spec proposals and WebML WG standardizes them when/if they gain traction and support 14:11:27 ... WebNN API is an example that was incubated in the CG and graduated to this WG 14:11:48 q+ 14:11:52 ... both the WebML CG and WG share many of the active participants and use the same GH infrastructure 14:12:00 ... this work mode enables smooth collaboration 14:12:09 MikeW9 has joined #webmachinelearning 14:12:34 ... one notable difference is the IPR policy: in CG a participant first make commitments over their own contributions, in a WG commitments are made based on the WG charter scope 14:13:06 ... other differences come from the W3C Process that WGs comply with, e.g. wide review for WG specs when hitting certain spec milestones 14:13:20 Subtopic: Changes overview 14:13:27 anssik: to give an overview of the proposed changes: 14:13:38 ... - Goals were updated to better explain the relationship of low-level APIs and higher-level task-specific APIs: 14:13:43 "Following the precepts of the Extensible Web Manifesto, higher-level task-specific APIs can be implemented in terms of the low-level APIs that use custom models downloaded over the network. To complement this approach, the Community Group will incubate selected task-specific APIs to enable reuse of the built-in models that are distributed as part of the browser or the underlying software platform." 14:14:29 anssik: - Task-specific APIs and Prompt API added to Deliverables: 14:14:33 -> Translator and Language Detector APIs https://github.com/WICG/translation-api 14:14:38 -> Writing Assistance APIs https://github.com/WICG/writing-assistance-apis 14:14:44 -> Prompt API https://github.com/explainers-by-googlers/prompt-api 14:14:56 ... - Noted WebNN has graduated to the WG 14:15:22 anssik: the next step is to conduct a 30-day vote for the CG participants on the proposed new charter, I'd like to kick off that review soon 14:15:38 ... any questions or comments? 14:15:50 q? 14:16:30 Etienne: Eng manager at Google for built-in AIs, working with Kenji and Domenic, also working with Natasha from Microsoft, 14:16:31 q? 14:16:53 Present+ Dominique_Hazael-Massieux 14:17:18 dom: during TPAC discussion there were discussion whether some of the APIs should be considered for the WGs charter 14:17:40 ... wanted to hear whether we indeed think adding them as potential deliverables in that new WG charter 14:18:01 ... IPR considerations differ between CG and WG charters 14:18:10 Present+ Austin_Sullivan 14:18:31 dom: if adding to the CG is the first step for getting them into the WG, would be good to know that soon 14:20:30 anssik: WG charter draft should be ready around January 14:21:03 q? 14:21:52 MikeW: I haven't yet looked at the proposals, can take a look 14:22:14 q? 14:22:32 ack dom 14:22:54 Topic: Device selection abstractions 14:23:00 gb, this is webmachinelearning/webnn 14:23:00 anssik, OK. 14:23:10 anssik: issue #749 14:23:11 https://github.com/webmachinelearning/webnn/issues/749 -> Issue 749 MLContextOptions.deviceType seems unnecessary outside of conformance testing (by mwyrzykowski) [device selection] 14:23:21 anssik: MikeW proposed a more generalized, abstract concept on how to address the issue of device selection 14:23:53 ... navigator.ml.opSupportLimitsPerAdapter() returns MLOpSupportLimits dictionary for each compute device on a system (CPU, GPU, NPU) 14:24:26 ... web developer/framework then passes the most appropriate MLOpSupportLimits to navigator.ml.createContext() to indicate to the implementation which limits will be required during the lifetime of the MLContext 14:24:42 ... implementation-specific constraints this concept attempts to address: 14:24:55 ... - DirectML needs to know which device the tensor will run on 14:25:05 ... - CoreML does not allow requiring the NPU 14:25:15 ... other considerations: 14:25:27 ... - implementation to decide how much information to disclose via opSupportLimitsPerAdapter 14:25:44 ... - this would expand the fingerprintable surface on top of what opSupportLimits() (PR #755) already exposes 14:25:44 https://github.com/webmachinelearning/webnn/pull/755 -> MERGED Pull Request 755 Define opSupportLimits() (by philloooo) 14:26:12 MikeW: excellent summary Anssi 14:26:34 ... not a concrete summary, but how to abstract the concept a bit so that DirectML gets what it needs 14:26:44 ... also cater to the fact that CoreML cannot require NPU 14:27:01 ... fingerprinting concern needs to be addressed 14:27:12 ... open to any feedback, this was a rough idea that I informally proposed 14:27:49 Zoltan: once we enumerate constraints and bind to a compute device you can identify which device you have 14:28:04 ... the main problem from current design is we have CPU, GPU, NPU, some combinations are not available on all platforms 14:28:28 ... why not pickyback on constraints query 14:28:49 ... I was wondering how to do this from the security point of view, without exposing too many details from the system 14:29:08 ... we can experiment with this and certainly it adds more code to the application, but it depends on the use case 14:29:33 ... we can do more work to map these to the application use cases 14:29:58 ... programmatically thinking I like the proposal 14:30:00 asully has joined #webmachinelearning 14:31:11 The question is what the USVString would be in record opSupportLimitsPerAdapter(); 14:31:43 dom: this is a topic to engage with the Privacy WG (was PING), in terms of when, it does not need to be fully fleshed out proposal, more likely what they're interested in is what additional fingerprinting surface this would add, what are the ways to mitigate 14:32:16 ... doing own research ahead of time, what mitigation would be realistic to deploy, drive-by fingerprinting, what are the tradeoffs to make during the design 14:32:17 q? 14:32:39 q+ 14:32:44 q? 14:33:15 dom: maybe writing an explainer for the new proposal is the best approach to bring it in front of the Privacy WG and document also Privacy considerations in that explainer 14:33:58 AramZS has joined #webmachinelearning 14:34:55 q+ 14:34:55 Zoltan: I think the problem now is NPU is not supported, if we find an algorithm that says choose NPU when available or closest mapping 14:35:20 ... if for other reasons we want to expose this proposed API that results the USVString and opSupportsLimits 14:35:33 ... should standardize the USVString that is the fingerprintable area 14:35:58 ... privacy people will not like device enumeration-style API 14:36:04 ack zkis 14:36:07 q? 14:36:11 ack MikeW9 14:37:02 MikeW: the existing API is an option, the concern from WebKit people was the concept of NPU, GPU, NPU being too hardware-specific for the web, if we'd abstract those terms it would help 14:37:51 q? 14:37:57 ack MikeW 14:38:06 Topic: MLTensor 14:38:12 anssik: MLTensor explainer has landed! 14:38:17 -> MLTensor Explainer https://github.com/webmachinelearning/webnn/blob/main/mltensor-explainer.md 14:38:24 ... Thank You all, this was a major effort, the PR #754 received 175(!) review comments 14:38:25 https://github.com/webmachinelearning/webnn/pull/754 -> MERGED Pull Request 754 Add MLTensor explainer (by a-sully) [webgpu interop] 14:39:02 anssik: special thanks to Austin for carefully iterating this explainer with review and contributions from Corentin, Domenic, Phillis, Rafael, Brandon, Bryan, Nignxin, Dwayne, others 14:39:19 ... our next step would be to prepare a spec patch to specify the feature normatively in spec language 14:39:25 ... another task would be to check that we've closed all "webgpu-interop" issues that this explainer (or its spec counterpart will) addressed: 14:39:31 -> webgpu-interop issues https://github.com/webmachinelearning/webnn/labels/webgpu%20interop 14:39:40 anssik: Austin, what are your thoughts on the spec update? I think you want to coordinate with the editors on that. 14:40:09 Austin: I'm very excited that this PR merged, the work has just began in terms of specifying the feature and prototype implementations in browsers 14:40:30 ... we're discussing whether we prototype first and then spec based on that implementation experience 14:41:06 ... leaning for the explainer as a source of trust for now and when especially the WebGPU interop implementation is better fleshed out advance to the spec update 14:42:07 q? 14:43:26 Austin: a draft PR might be a good idea for the spec patches, I will figure out how to go about it 14:43:43 Topic: Tensor primitives 14:43:51 -> TPAC 2024 slides https://lists.w3.org/Archives/Public/www-archive/2024Sep/att-0007/Tensor_Primitive_Ops_Proposal_-_TPAC.pdf 14:44:17 anssik: this topic is on the agenda to check if we're ready to discuss some of the requirements 14:44:24 ... I listed some possible subtopics in the agenda: 14:44:42 ... - Additional primitive ops: evolve core op set w/ MLIR linalg, PT Edge, TOSA 14:44:50 ... - Graph expressiveness: subgraphs, control flow 14:45:02 ... - Native runtime support: fusion, pattern matcher 14:45:08 ... Ningxin, would any of these topics benefit from group discussion? 14:46:05 ningxin: after TPAC we established two workstream, how to define additional primitive ops cross-referring other IRs, incorporating the feedback, beyond linalg, PT Edge, TOSA 14:46:56 ... another workstream is about subgraph definition by WebNN Builder API, this is being combined with the underlying implementation of fusion, pattern matcher 14:47:11 ... subgraph most useful for graph optimizer for pattern matching 14:47:59 ... we've started proof-of-concept based on Chromium to see how we can define this subgraph in WebNN and help the implementation do pattern matching, starting with multi-headed attention and look for other patterns LSTM, GRU 14:48:36 ... once we have data from this POC that will inform the WG on this design, at that stage we'll open a spec discussion for more discussion 14:49:00 ... we think control flow is out of scope for the first stage, because in some devices it is not supported natively, so we want to focus on the subgraph 14:49:04 q? 14:49:41 Dwayne: for comparison to others, I compiled a huge spreadsheet 14:50:36 ... next step would be to identify what is missing 14:50:57 ... should we use the core op set issue #573 for discussion these tensor primitives? 14:50:58 https://github.com/webmachinelearning/webnn/issues/573 -> Issue 573 Core operator set (by philloooo) [question] [opset] 14:51:09 573 is a good one 14:51:21 ... also after cross-referencing the custom ops issue #6 (from 2019) I think we should close it? 14:51:22 https://github.com/webmachinelearning/webnn/issues/6 -> Issue 6 Custom operations (by dsmilkov) [v2] [device selection] 14:51:34 ... it has great historical context but is not actionable 14:51:53 q? 14:52:04 q+ 14:52:07 ack ningxin 14:53:19 ningxin: re issue #6, we're explosing composed primitive, another way for WebGPU interop is for the framework to write a custom op, so issue #6 expands beyond composable custom op 14:53:54 q? 14:54:16 Topic: Open issues and PRs 14:54:21 anssik: as the usual, we'll discuss open issues and review PRs based on your feedback and progress: 14:54:29 -> All open issues https://github.com/webmachinelearning/webnn/issues 14:54:29 -> All open pull requests https://github.com/webmachinelearning/webnn/pulls 14:54:29 -> Recently merged PRs https://github.com/webmachinelearning/webnn/pulls?q=is%3Apr+is%3Amerged 14:54:33 Subtopic: Debrief on PRs merged recently 14:54:37 anssik: thanks to Austin and Bruce for PRs and everyone for reviews, since last call: 14:54:41 ... issue #666 fixed by PR #774 14:54:42 https://github.com/webmachinelearning/webnn/pull/774 -> MERGED Pull Request 774 Convert MLOperand methods into readonly attributes (by a-sully) 14:54:42 https://github.com/webmachinelearning/webnn/issues/666 -> CLOSED Issue 666 Reconsider `MLOperand` methods (by a-sully) [question] 14:54:44 ... issue #775 fixed by PR #776 14:54:45 https://github.com/webmachinelearning/webnn/pull/776 -> MERGED Pull Request 776 Bugfix: Fix the error of opSupportLimits for split op (by BruceDai) 14:54:45 https://github.com/webmachinelearning/webnn/issues/775 -> CLOSED Issue 775 The split's opSupportLimits should be of `MLSplitSupportLimits` (by BruceDai) 14:54:50 ... issue (various) fixed by PR #754 - MLTensor explainer 14:54:51 https://github.com/webmachinelearning/webnn/pull/754 -> MERGED Pull Request 754 Add MLTensor explainer (by a-sully) [webgpu interop] 14:55:00 ... https://github.com/w3ctag/design-reviews/issues/933 fixed by PRs #765 and #769 14:55:01 https://github.com/webmachinelearning/webnn/pull/769 -> MERGED Pull Request 769 Add Implementation Status to metadata (by anssiko) 14:55:01 https://github.com/webmachinelearning/webnn/pull/765 -> MERGED Pull Request 765 Add resource contention considerations (by anssiko) 14:55:01 https://github.com/w3ctag/design-reviews/issues/933 -> CLOSED Issue 933 Updated review of WebNN API (by dontcallmedom) [Priority: urgent] [Progress: review complete] [Review type: small delta] [Review type: horizontal review] [Venue: WebML CG] [Resolution: satisfied with concerns] [Mode: breakout] 14:55:03 … [Topic: Machine Learning] [Focus: Web architecture (pending)] [Focus: Security (pending)] [Focus: Privacy (pending)] 14:55:31 Subtopic: [feature request] LocalResponseNormalization 14:55:34 anssik: issue #228 14:55:35 https://github.com/webmachinelearning/webnn/issues/228 -> CLOSED Issue 228 Support for LocalResponseNormalization (LRN) operation (by MarkGHX) [opset] [feature request] 14:55:38 ... the group analyzed the proposal for a new op and recommends a decomposition path for LocalResponseNormalization 14:55:42 ... Dwayne's last comment documents the rationale: 14:55:47 "Using decomposition in higher layers (e.g. ORT's WebNN EP) for localResponseNormalization rather than a dedicated WebNN operator due to the rarity of the operator in models and the awkward backend differences." 14:55:57 +1 14:57:24 Subtopic: [operator specific] Consider adding int64/uint64 data type support for some reduce operators 14:57:28 anssik: issue #694 and PR #695 14:57:29 https://github.com/webmachinelearning/webnn/pull/695 -> Pull Request 695 Bugfix: Add missing 64-bit integers support for some reduction operators (by huningxin) [operator specific] 14:57:29 https://github.com/webmachinelearning/webnn/issues/694 -> Issue 694 Consider adding int64/uint64 data type support for some reduce operators (by lisa0314) [operator specific] 14:57:34 ... I wanted to clarify the PR status and possible blockers for this PR 14:57:49 ... MikeW suggests "we should start with the set of operations which is the intersection of which all native frameworks support today", Austin +1 14:57:54 ... Ningxin was asking a question re Apple platforms and 64-bit integers: 14:58:03 "if an implementation could map "cpu" MLContext to BNNS, "gpu" MLContext to MPS and "npu" MLContext to CoreML, would that mean only the "npu" MLContext not supporting 64bit integers? That device type specific difference can be detected through MLContext.opSupportLimits() interface." 14:58:51 ningxin: my questions is related to the device selection, so the question is whether this data type support can be detectable through opSupportsLimits in the current design? 14:58:52 q? 14:59:46 MikeW: I think technically this is feasible for the backends to map these, it does make implementation diverse substantially, not sure if anyone has feedback on that for CoreML 15:00:12 ... it might be confusing for frameworks or users, when they use 64-bit integer it'd not be possible to run on NPU 15:00:13 q? 15:00:43 ningxin: thanks Mike, we need to think this together with the device selection to find the right abstraction to give to developers 15:00:54 q? 15:01:48 RRSAgent, draft minutes 15:01:49 I have made the request to generate https://www.w3.org/2024/10/31-webmachinelearning-minutes.html anssik 15:06:11 Present+ Rafael_Cintron 15:07:03 s/from Microsoft,/from Microsoft 15:07:57 s/discussion there were/there was 15:08:21 s/WGs charter/next WG charter 15:10:57 s/pickyback/piggyback 15:14:06 s/two workstream/two workstreams 15:14:22 s/cross-referring/cross-referencing 15:27:14 RRSAgent, draft minutes 15:27:16 I have made the request to generate https://www.w3.org/2024/10/31-webmachinelearning-minutes.html anssik 15:28:05 s/a spec discussion/a spec issue 15:29:06 s/… should we/anssik: should we 15:29:17 RRSAgent, draft minutes 15:29:18 I have made the request to generate https://www.w3.org/2024/10/31-webmachinelearning-minutes.html anssik 15:30:12 s/should we use/anssik: should we use 15:30:19 RRSAgent, draft minutes 15:30:20 I have made the request to generate https://www.w3.org/2024/10/31-webmachinelearning-minutes.html anssik 15:34:42 s/also after cross-referencing/anssik: also after cross-referencing 15:35:00 s/it has great/anssik: it has great 15:35:03 RRSAgent, draft minutes 15:35:04 I have made the request to generate https://www.w3.org/2024/10/31-webmachinelearning-minutes.html anssik 15:36:18 s/diverse/diverge 15:36:33 RRSAgent, draft minutes 15:36:35 I have made the request to generate https://www.w3.org/2024/10/31-webmachinelearning-minutes.html anssik 17:30:45 Zakim has left #webmachinelearning