14:59:17 <RRSAgent> RRSAgent has joined #webmachinelearning
14:59:17 <RRSAgent> logging to https://www.w3.org/2022/01/13-webmachinelearning-irc
14:59:20 <Zakim> RRSAgent, make logs Public
14:59:21 <Zakim> please title this meeting ("meeting: ..."), anssik
14:59:29 <anssik> Meeting: WebML WG Teleconference – 13 January 2022
14:59:39 <anssik> Chair: Anssi
14:59:49 <anssik> Agenda: https://github.com/webmachinelearning/meetings/blob/main/telcons/2022-01-13-wg-agenda.md
14:59:55 <anssik> Scribe: Anssi
15:00:09 <anssik> scribeNick: anssik
15:00:19 <ningxin_hu> ningxin_hu has joined #webmachinelearning
15:00:28 <anssik> Present+ Anssi_Kostiainen
15:01:02 <dom> Present+
15:01:06 <anssik> Present+ Ningxin_Hu
15:02:11 <anssik> Present+ Rafael_Cintron
15:02:59 <anssik> Present+ Chai_Chaoweeraprasit
15:04:08 <anssik> Present+ Jonathan_Bingham
15:04:17 <anssik> scribe+ Jonathan_Bingham
15:04:30 <dom> scribe+
15:04:39 <Jonathan> Jonathan has joined #webmachinelearning
15:05:17 <anssik> scribe+ Jonathan
15:05:24 <RafaelCintron> RafaelCintron has joined #webmachinelearning
15:05:48 <anssik> Present+ James_Fletcher
15:06:04 <dom> Present+ Rachel_Yager
15:06:16 <dom> Present+ Geun-Hyung_Kim
15:06:40 <dom> Anssi: welcome to 2022!
15:06:52 <dom> ... exciting year for this WG with lots of milestones ahead of us
15:07:04 <dom> ... the 2021 work getting into the hands of developers
15:07:06 <anssik> Topic: WebNN API feature and change requests
15:07:22 <chai> chai has joined #webmachinelearning
15:07:23 <dom> Present+ Bruce
15:07:29 <chai> Present+
15:07:31 <anssik> Subtopic: Integration with real-time video processing
15:07:41 <anssik> -> https://github.com/webmachinelearning/webnn/issues/226 Integration with real-time video processing #226
15:07:51 <Rachel> Rachel has joined #webmachinelearning
15:08:08 <dom> Anssi: this emerged from the Web real-time communications WG
15:08:14 <Geun-Hyung> Geun-Hyung has joined #webmachinelearning
15:08:24 <Geun-Hyung> present+
15:08:28 <dom> ... the request is to develop a prototype that integrates the new mediacapture transform API in a worker context with the WebNN API
15:08:50 <dom> ... the most interesting use case driving this for WebRTC is background blur, a familiar feature in teleconferencing apps
15:09:02 <Bruce> Bruce has joined #webmachinelearning
15:09:13 <dom> ... in this issue, we have already had a discussion with the 2 of the WebRTC WG chairs providing their input
15:09:23 <dom> ... WebGPU co-chair (Corentin) also provided input
15:09:49 <anssik> -> https://github.com/miaobin/webnn-samples/tree/rnnoise-real-time/audio_stream Noise suppression (RNNoise) based on mediacapture-transform API
15:09:53 <anssik> -> https://honry.github.io/webnn-samples/semantic_segmentation/ Semantic segmentation based on mediacapture-transform API (preview)
15:09:59 <anssik> -> https://github.com/Honry/webnn-samples/tree/mediacapture-transform/semantic_segmentation Semantic segmentation based on mediacapture-transform API (source)
15:10:02 <dom> ... two initial prototypes were submitted, developed by Bin Miao and Wanming Lin - my thanks to them on behalf of the WG
15:10:22 <anssik> anssik: both the prototypes currently do processing in the main thread
15:10:51 <dom> anssik: it seems that the WG should initially focus on developing further the semantic segmentation prototype as the place to evaluate performance risks
15:11:06 <dom> ... proposed next steps as I understand them:
15:11:16 <JamesFletcher> JamesFletcher has joined #webmachinelearning
15:11:23 <dom> ... - we should migrate expensive processing to worker context
15:11:33 <dom> ... - build a full GPU-only pipeline to minimize GPU to CPU copies
15:11:42 <dom> ... - document any API gaps that emerge
15:11:46 <dom> q+
15:12:04 <anssik> q?
15:12:06 <anssik> ack dom
15:12:19 <anssik> dom: thanks Anssi for the summary
15:12:43 <anssik> ... the processing in worker is covered so that there are no gaps I think
15:13:25 <anssik> ... the real gaps are in the GPU-only pipeline approach, the gap seems to be in WebNN, interaction with WebGPU buffers
15:13:49 <anssik> ... specific questions on how to go back and forth between VideoFrame and GPUBuffer abstractions
15:14:01 <anssik> ... then, also stream-based processing
15:14:37 <anssik> ... in terms of specific next steps, would be great to get confirmation whether we can stay in the GPU-only pipeline
15:14:55 <chai> q+
15:15:10 <anssik> ... story for WebGPU and WebGL integration seems important take away from this experimentation
15:15:14 <anssik> q?
15:15:18 <anssik> ack chai
15:15:44 <dom> chai: good scenario to stress-test WebNN-on-GPU
15:16:04 <dom> ... already identified issues on CPU/GPU transitions which comes with high cost
15:16:13 <AramZS> AramZS has joined #webmachinelearning
15:16:22 <dom> ... as far as I know, the applications out there that are already doing background segmentation do it with GPU
15:16:43 <anssik> -> https://github.com/gpuweb/gpuweb/issues/2500 Investigation: how WebNN / WebGPU interop could be happening #2500
15:17:21 <dom> ningxin: the issue I raised summarize the current situation and our motivation for WebGPU integration
15:18:03 <dom> ... two use cases we want to support: custom ops in WebGPU shaders
15:18:17 <dom> ... and integration with real-time video processing
15:18:41 <anssik> -> https://github.com/w3c/webcodecs/pull/412 Define importing VideoFrame into WebGPU #412
15:18:45 <dom> ... we can expect VideoFrame to live in the GPU, and may be importable as a GPUTexture
15:18:56 <anssik> -> https://github.com/gpuweb/gpuweb/issues/2498 Where do we spec WebGPU-WebCodecs interaction? #2498
15:19:03 <dom> ... this would allow to use VideoFrame as input to compute
15:19:30 <dom> ... as Corentin mentioned, there needs more coordination between WebGPU/WebML WG to fulfill these capabilities
15:19:55 <dom> ... a good opportunity for the two groups to make progress, which I hope this issue will help with
15:20:54 <anssik> q?
15:20:59 <dom> anssi: similar questions are being explored at the intersection of WebGPU and WebCodecs
15:21:11 <dom> ... ningxin, you'll talk with WebGPU folks and report back to the group?
15:21:25 <dom> ningxin: the issue is the first step towards that
15:21:27 <dom> q+
15:21:53 <dom> ... I'm interested in input in how to best coordinate work across the two groups
15:22:13 <RafaelCintron> q+
15:22:30 <dom> anssik: if the topic would benefit from sync coordination, we could invite a subset of the gpu groups to our meeting (or vice versa)
15:23:22 <anssik> dom: talked with WebGPU WG staff contact Francois, his recommendation was to first have discussion on the GH issue
15:24:02 <anssik> ... it might be good to have a sync meeting with editors and chairs participating from WebML and WebGPU WGs
15:24:17 <dom> ack me
15:24:48 <ningxin_hu> no, i didn't
15:25:17 <anssik> q?
15:25:19 <anssik> ack RafaelCintron
15:25:46 <dom> RafaelCintron: I represent MS at the WebGPU WG since the beginning - they have 2 meetings, one for the shading language and one for the API
15:25:51 <anssik> RafaelCintron: rep of Msft in the WebGPU WG, two meetings, one for shading language, one for the API, bi-weekly cadence
15:26:08 <dom> ... they're focused on shipping ASAP based on interest e.g. from game engines, with a timeline toward May
15:26:23 <dom> ... I think they'd be open to speak with us
15:26:34 <dom> ... it could end up that we re-use some of they data types
15:26:56 <dom> ... or go through import/export
15:28:25 <dom> Anssi: so we need to be considerate of their time given their roadmap
15:28:42 <dom> RafaelCintron: but it's clearly a very compelling use case to have an ML-in-GPU pipeline
15:28:57 <anssik> q?
15:30:12 <anssik> dom: prototyping work might get stuck due to API gaps, but prototyping work needs to continue
15:31:22 <anssik> anssik: should we explore the worker context processing?
15:31:28 <ningxin_hu> q+
15:31:42 <anssik> dom: that probably will not have direct impact on performance, will inform the discussion in WebRTC WG
15:31:42 <RafaelCintron> q+
15:31:43 <anssik> ack ningxin_hu
15:32:12 <dom> ningxin_hu: in terms of perf evaluation, the current prototype is running on top of the polyfill
15:32:24 <dom> ... the TF backend runs on top of the WebGL backend
15:32:34 <anssik> q?
15:32:58 <dom> ... to see the real performance benefit, we would need the WebNN Chromium implementation, under engineering review (incl security & privacy)
15:33:23 <dom> ... will also need progress on the WebGPU/WebNN integration
15:33:36 <anssik> ack RafaelCintron
15:34:05 <dom> RafaelCintron: having working code will also help convincing people of the value of the work
15:34:31 <dom> anssik: wanming would likely be interested in continuing work on this
15:34:32 <chai> q+
15:34:48 <anssik> ack chai
15:35:08 <dom> chai: we're interested in helping with the prototype as well, it's a very exciting topic
15:35:27 <dom> anssi: let's continue discussion in the issue for the most impactful next steps
15:35:55 <dom> ... it can also help down the line with perf evaluation once the native implementation progresses
15:36:04 <anssik> q?
15:36:15 <anssik> Subtopic: Should restrict the sync APIs to only exist in Workers?
15:36:24 <anssik> -> https://github.com/webmachinelearning/webnn/issues/229 Should restrict the sync APIs to only exist in Workers? #229
15:36:57 <dom> Anssi: in summary, increasingly popular WASM-based frameworks need sync APIs, but blocking the main thread should be avoided
15:37:12 <anssik> q?
15:37:13 <dom> ... the proposed solution would be to restrict sync build/compute APIs in the worker context
15:37:27 <anssik> q?
15:37:30 <RafaelCintron> q+
15:37:34 <anssik> ack RafaelCintron
15:37:52 <anssik> RRSAgent, draft minutes
15:37:52 <RRSAgent> I have made the request to generate https://www.w3.org/2022/01/13-webmachinelearning-minutes.html anssik
15:38:00 <dom> RafaelCintron: when the API is working in the mode of doing everything on the CPU, I agree that sync should be restricted to worker
15:38:31 <dom> ... if you working on a different device (GPU or NPU...), then the sync API is only queuing commands not holding the main thread
15:39:12 <dom> ... that being said, passing around many structured objects is not great on the Web platform, as the babylon.js team has been reporting
15:39:25 <dom> s/objects/objects across threads/
15:39:36 <anssik> q?
15:39:55 <dom> ... this would negatively impact a worker-only API
15:40:28 <dom> anssik: it's easier to add API surface than removing it; so maybe we could start conservatively with a worker-only sync API
15:40:41 <dom> ... and extend it to main thread only later once we're more confident
15:40:54 <ningxin_hu> q+
15:40:56 <dom> ... removing a sync API later might be tricky, as illustrated by XHR
15:40:58 <anssik> q?
15:41:01 <anssik> ack ningxin_hu
15:41:34 <dom> ningxin_hu: ML Frameworks are primary consumers of the API - we should incorporate their feedback
15:41:44 <chai> q+
15:41:45 <dom> ... some run their WASM backend in the main thread
15:42:00 <dom> ... we should figure out the solution with them
15:42:32 <dom> anssik: most important frameworks with WASM-based backends?
15:42:42 <dom> ningxin: TF, ONNX
15:42:51 <dom> ... Ping & Emma might help us
15:43:13 <dom> anssik: let's tag them in the issue, and invite them to a future call if needed
15:43:14 <anssik> q?
15:43:17 <anssik> ack chai
15:43:45 <dom> chai: it's somewhat related to the previous topic - performance is key
15:44:04 <dom> ... the prototype on segmentation will be really important to drive progress on this
15:44:20 <dom> ... high-frame rate scenario requires as few CPU/GPU transition as possible
15:44:41 <dom> ... We also need to make sure that framework integration will work
15:45:12 <dom> ... from the API standpoint, conceptually, if the interop between WebNN and WebGPU allows to offload data upload/download to the WebGPU spec, we can free up WebNN to be more flexible
15:45:45 <dom> ... this would create an API that can be used in different contexts
15:45:59 <anssik> q?
15:46:22 <dom> q+
15:47:20 <anssik> ack dom
15:47:53 <anssik> dom: multiple aspects, 1) CPU vs GPU, WebNN could work with CPU backend with sync API
15:48:13 <anssik> ... not sure if we can make that distinction from WebIDL perspective
15:48:34 <anssik> ... the way we describe the ops are not specific enough
15:48:47 <anssik> ... e.g. compute(), no way to interpret it as non-blocking for main thread
15:48:59 <anssik> ... waits for compute to return
15:49:05 <anssik> ... even on a GPU basis
15:49:24 <anssik> ... getting input from frameworks what they need is important
15:49:55 <RafaelCintron> q+
15:49:55 <anssik> ... there will be improvements the spec will need to spell out, what parts of the API need to be sync and which can be async
15:50:24 <anssik> ... even if we have sync API on the worker, we'd like to have async API on the main thread, requires significant spec changes
15:50:35 <anssik> s/1)//
15:50:48 <anssik> dom: understanding the needs required to get the API into stable state
15:50:49 <anssik> q?
15:50:53 <anssik> ack RafaelCintron
15:51:28 <dom> RafaelCintron: the spec is not final yet - I'm hopeful we can improve it iteratively
15:51:37 <anssik> RRSAgent, draft minutes
15:51:37 <RRSAgent> I have made the request to generate https://www.w3.org/2022/01/13-webmachinelearning-minutes.html anssik
15:51:47 <dom> ... e.g. a non-CPU backend would return a different object that allows sync operations
15:52:07 <dom> ... WebGPU has the same issue, and creates two pipelines (sync & async)
15:52:17 <dom> ... we might have room for solving the issue differently
15:52:35 <dom> ... the problem with workers is that a lot of basic stuff doesn't work in a worker (e.g. the DOM)
15:53:05 <dom> ... we could start with everything async; but with a very pipelined-application, this will add a lot of latency in practice
15:53:12 <anssik> q?
15:53:36 <dom> anssik: so I hear let's check with framework folks on their perspectives before coming back to this issue
15:53:39 <anssik> Subtopic: Should WebNN support async APIs?
15:53:44 <anssik> -> https://github.com/webmachinelearning/webnn/issues/230 Should WebNN support async APIs? #230
15:54:42 <dom> anssik: having the API available in main thread helps with adoption
15:54:47 <anssik> q?
15:55:40 <dom> ningxin: most of this issue has been discussed in the context of the sync API and intersections with GPU
15:56:15 <dom> ... I've pinged Ping on this thread
15:56:28 <dom> ... TF.js has a WebGPU backend with async APIs
15:56:57 <anssik> q?
15:57:52 <anssik> dom: need to understand the requirements before making progress with this issue
15:58:21 <anssik> q?
15:58:29 <anssik> Topic: WebNN API open pull requests
15:58:42 <anssik> -> https://github.com/webmachinelearning/webnn/pulls Open PRs
15:59:54 <chai> I need to work on my PR backlog ^^
16:00:50 <anssik> q?
16:01:30 <ningxin_hu> ningxin_hu has joined #webmachinelearning
16:02:03 <anssik> Topic: WebNN API Candidate Recommendation
16:02:38 <dom> anssi: we're planning to reach Candidate Recommendation (CR) this year - CR is an important milestone:
16:02:50 <dom> ... it means the spec is feature complete and can be implemented as is
16:03:05 <anssik> -> https://github.com/webmachinelearning/webnn/issues/240 Candidate Recommendation readiness tracker #240
16:03:11 <dom> ... I've created a CR tracker for us to allow us to move toward this milestone in a transparent and coordinated fashion
16:03:25 <dom> ... it's managed as a github issue with checkboxes
16:03:43 <dom> ... I've summarized the 3 top level key CR requirements - as set by the W3C Process Document
16:04:06 <dom> ... the first requirement is to meet the WG's requirements, including the ones set in the WG charter
16:04:14 <dom> ... we need to demonstrate "adequate" implementation experience
16:04:34 <dom> ... there is room for interpretation there - formally the W3C Director will assess that
16:04:44 <dom> ... it doesn't mean the API need to ship to reach CR
16:04:52 <dom> ... We need to show that wide-review has been received
16:04:59 <anssik> -> https://github.com/webmachinelearning/webnn/issues/239 Wide review tracker #239
16:05:06 <dom> ... which I'm tracking in a separate tracker
16:05:39 <dom> ... we already completed TAG review, we started privacy review
16:05:53 <dom> ... this includes getting signals from other stakeholders, e.g. web developers
16:06:10 <dom> ... or the recommendation from the workshop
16:06:15 <dom> ... or signals from implementors
16:06:34 <anssik> -> https://github.com/webmachinelearning/webnn/issues?q=label%3Acr CR blocker issues (WIP)
16:10:08 <anssik> dom: good summary, we may need to perhaps discuss whether we have already satisfied all the use cases, and we should perhaps also note ethical considerations work that in ongoing
16:10:52 <anssik> ... the first people we need to convince is ourselves, what we want to demonstrate as "job well done"
16:11:29 <anssik> ... based on what I heard today, getting WebRTC and GPU story covered maybe we want to turn them into use cases
16:12:12 <anssik> q?
16:13:34 <dom> RRSAgent, draft minutes
16:13:34 <RRSAgent> I have made the request to generate https://www.w3.org/2022/01/13-webmachinelearning-minutes.html dom