14:45:07 RRSAgent has joined #webmachinelearning 14:45:07 logging to https://www.w3.org/2022/03/24-webmachinelearning-irc 14:45:10 RRSAgent, make logs Public 14:45:11 please title this meeting ("meeting: ..."), anssik 14:45:11 Meeting: WebML WG Teleconference – 24 March 2022 14:45:21 Chair: Anssi 14:45:21 Agenda: https://github.com/webmachinelearning/meetings/blob/main/telcons/2022-03-24-wg-agenda.md 14:45:26 Scribe: Anssi 14:45:30 scribeNick: anssik 14:45:34 scribe+ dom 14:45:41 Present+ Anssi_Kostiainen 14:45:52 RRSAgent, draft minutes 14:45:52 I have made the request to generate https://www.w3.org/2022/03/24-webmachinelearning-minutes.html anssik 15:00:56 ningxin_hu has joined #webmachinelearning 15:01:44 Present+ James_Fletcher 15:01:52 Present+ Ningxin_Hu 15:01:59 Present+ Dominique_Hazael-Massieux 15:02:49 Present+ Ping_Yu 15:03:05 Present+ Rachel_Yager 15:03:35 Present+ Dominique_Hazael-Massieux 15:04:06 Topic: Announcements 15:04:43 jonathanbingham has joined #webmachinelearning 15:05:00 Present+ Jonathan_Bingham 15:05:05 -> Ethical Principles for Web Machine Learning https://webmachinelearning.github.io/ethical-webmachinelearning/ 15:05:09 -> 10-minute intro video https://www.w3.org/2022/03/ml-ethics/ 15:05:29 -> https://github.com/webmachinelearning/ethical-webmachinelearning Repo for Ethical Principles for Web Machine Learning 15:05:39 james: consultation doc migrated to draft note, now migrated to github 15:05:50 ... it solidifies the principles (based on UNESCO) and the guidance attached to them 15:05:58 ... the guidance making the principles more concrete 15:06:21 ... risks & mitigations is the next steps - they'll be even more concrete and need y'all input 15:06:40 ... either via github, or through the live sessions week of Apr 4 which Dom is running 15:07:09 anssi: thanks, amazing work 15:07:22 ... I invite you all to watch James' intro video 15:07:56 q? 15:07:58 Present+ chai 15:08:11 Present+ Chai_Chaoweeraprasit 15:08:58 Topic: Security considerations - last call for review 15:09:25 -> PR: Update Security Considerations https://github.com/webmachinelearning/webnn/pull/251 15:09:33 anssi: I've been bugging you all to make progress on this work - security is a key component of our wide review commitment 15:09:49 ... thanks for the feedback received which I've incorporated in PR #251 15:10:06 anssik: changes per review feedback: 15:10:14 ... - Drop avoid reshaping tensors guideline 15:10:19 Rachel_ has joined #webmachinelearning 15:10:25 ... - Note shape inference is done during the graph building stage 15:10:39 ... - Note concrete device selection is left to the implementation 15:10:56 ... - Update timing attack mitigation considerations 15:11:51 anssi: should we go ahead and merge this for another round of review? 15:11:54 dom: +1 15:12:10 RafaelCintron has joined #webmachinelearning 15:12:18 +1 15:12:21 anssi: let's circle back with the chrome review and then bring it W3C security review 15:12:41 ... in any case, this is a first draft on which we'll keep iterating on as the rest of the document 15:12:42 q? 15:12:57 Topic: Context-based graph execution methods for different threading models 15:13:09 -> Context-based graph execution methods for different threading models https://github.com/webmachinelearning/webnn/pull/257 15:13:29 anssi: #257 is the pull request on which we agreed to converge during our previous meeting 15:14:38 chai: update from last time: there has been additional feedback and we're converging on the idea of making the default device option more explicit 15:14:40 Present+ Geunhyung_Kim 15:14:42 Geun-Hyung has joined #webmachinelearning 15:14:47 Present+ Rafael_Cintron 15:14:54 ... with the default being the CPU 15:14:59 present+ 15:15:09 ... this should resolve the problem of determining when the sync method can be used 15:15:30 ... Another progress is around the interop with WebGPU 15:15:59 ... Rafael, Brian and myself have spent quite a bit of time discussing this 15:16:26 ... we're converging on a somewhat more abstract layer that would be friendlier to implementations based on Vulkan, CoreML or linux 15:16:44 ... this hasn't been brought to the PR yet 15:17:12 ... it comes down to having the context taking the WebGPU queue and populating the ML workload the WebGPU queue 15:17:21 ... which would then be submitted and executed async 15:17:32 q? 15:18:04 q+ 15:18:04 Rafael: chai has been doing a great job pushing this forward 15:18:54 anssi: I notice Brian shared comments in PR #255 with a response from ningxin - not sure if that was incorporated 15:19:11 chai: yes, he's on board with our converged direction 15:19:19 q? 15:19:37 -> Device selection with MLDevicePreference and MLPowerPreference https://github.com/webmachinelearning/webnn/issues/169 15:19:46 anssi: re device selection, #169 is an existing issue on the topic 15:20:06 dom: I think Chai summarized my point re device selection 15:20:32 ... it has been an explicit choice to make device selection a hint due to privacy reasons 15:20:38 ... I'm thinking defaulting to CPU might be actually OK 15:20:53 ... if you ask for a CPU you'll get a CPU-backed context 15:21:31 -> "default" v.s. "auto" in MLDevicePreference and MLPowerPreference https://github.com/webmachinelearning/model-loader/issues/30 15:21:31 anssi: model loader is also considering reusing the device preference and power preference from our spec, with similar questions 15:21:45 q? 15:21:46 q? 15:21:52 ack ningxin_hu 15:22:11 ningxin_hu: thanks to chai, Rafael and brian to help moving the design forward 15:22:28 ... I've added a comment based on my investigation of the gpu-only processing pipeline 15:22:45 ... the WebGPU/WebNN interop plays a very important role there to make sure data stays on GPU for efficient processing 15:23:11 ... I've left some questions in PR #255 on resources sharing and execution order 15:23:17 ... I look forward to the updated PR 15:23:22 -> https://github.com/webmachinelearning/webnn/pull/255#issuecomment-1077444422 Ningxin's response based on prototype investigation findings (in context of the older PR #255) 15:23:38 ... my scenario is that the WebGPU compute shader is used for pre- and post-processing to the WebNN compute 15:24:15 ... my comment has pseudo-code to illustrate that usage in the WebGPU/WebNN background blur 15:25:21 anssi: let's land this in our spec before seeking WebGPU formal review 15:25:21 q? 15:25:41 q? 15:26:04 Topic: Integration with real-time video processing 15:27:10 -> Video processing with insertable streams main thread version https://huningxin.github.io/webrtc-samples/src/content/insertable-streams/video-processing/ 15:27:13 anssi: ningxin_hu has developed a media capture transform-based pipeline for background blur, one using WebGL, the other using WebGPU/WebNN that needs a prototype implementation of WebNN in Chromium 15:27:17 -> Video processing with insertable streams worker version https://huningxin.github.io/webrtc-samples/src/content/insertable-streams/video-processing-worker/ 15:27:25 -> Video processing with insertable streams main thread version https://huningxin.github.io/webrtc-samples/src/content/insertable-streams/video-processing/ 15:28:24 -> Details of the processing pipelines in issue #226 https://github.com/webmachinelearning/webnn/issues/226#issuecomment-1074968279 15:29:18 ningxin: this follows up to Dom's suggestion to investigate the integration of media capture transform (Process/generator design pattern that allows video processing via a VideoFrame object), as illustrated in webrtc-samples both in main thread and in a worker 15:30:09 ... the demo I created is based on these two examples to construct a pipeline that uses only GPU for ML processing, with here background blur as our example 15:30:25 ... As I explain in my description of the pipeline, it requires several steps: 15:30:35 ... - get a gpu buffer for a video frame 15:30:41 ... - a shader to blur an image 15:31:01 ... - a step to segment the input image between background from other objects, using machine learning 15:31:49 ... - based on the segmentation map that annotates background/foreground, another shader blurs the background and leaves the foreground alone 15:31:55 RRSAgent, draft minutes 15:31:55 I have made the request to generate https://www.w3.org/2022/03/24-webmachinelearning-minutes.html anssik 15:32:25 ... - this produces and output texture that can be drawn on an offscreencanvas to produce a videoframe fed into the media capture transform generator 15:32:46 ... that can be used for video playback or for sending on a webrtc connection 15:34:02 ... I implemented that pipeline with WebGL: WebGL shader and WebGL-TF.js backend for segmentation with TF.js DeepLabV3 model (it can be used on many objects, but this example focus on background detection) 15:34:08 ... this serves as a baseline 15:34:51 ... I initially wanted to make a WebGPU-only pipeline, but the WebGPU backend of TF.js has issues for segmentation (which I reported to the TF.js team) 15:35:11 ... once that resolved, I'll be able to update the sample with the WebGPU-only version 15:35:44 ... The final pipeline I published uses WebGPU shaders with a WebNN segmentation, which illustrates the interop with WebGPU 15:36:19 ... WebNN uses the output of the WebGPU shader as input to the compute 15:36:45 ... the output of WebNN compute is fed into another WebGPU shader for post-processing (blending input image with output segmentation map) 15:37:36 ... In this prototype, I still use the existing MLGraph.compute that uses GPUBuffer as input/output (found some related issue I documented in PR #255) 15:37:55 q? 15:37:56 ... as I alluded before 15:39:35 ... I also ran into issues - the WebGPU backend of TF.js limitation; a WebGPU implementation bug in Chromium; and on entry level GPUs, the WebGL pipeline can freeze the browser UI 15:39:58 Present+ Daniel_LaLiberte 15:40:21 ... Dom commented on the CPU usage, and asked whether main thread/worker will help reduce the CPU usage 15:40:31 ... it shouldn't based on my observation 15:41:02 ... the worker appraoch will mostly help with running sync version of the frameworks 15:41:27 q+ 15:41:31 q+ 15:41:54 ack RafaelCintron 15:42:13 RafaelCintron: congratulations ningxin_hu - this is really informative 15:42:40 ... being able to use seamlessly the GPU buffer from WebGPU to WebNN is a great signal 15:43:15 ack dom 15:43:20 anssi: +1 - getting interop right is harder but critical 15:43:38 dom: thanks for this amazing work, really impressive 15:44:50 ... I wonder if you have a sense whether this estimated CPU usage ~20 with WebGPU and ~40% with WebGL is what we should expect or whether there's optimizations to be done 15:44:53 q+ to ask pixel formats/color space 15:45:05 ningxu: re CPU usage, it probably depends on device driver & implementations 15:45:28 ... I didn't observe a difference of CPU usage between the two pipelines on my own setup 15:45:28 ningxin_hu: it depends on device drivers and implementation, I did not observe this big difference between the two pipelines 15:45:55 ... in terms of CPU usage, looking at WebGPU samples, they trigger similar CPU usage 15:46:11 ... e.g. when running the image blur sample 15:46:36 ... so this probably a question to discuss with WebGPU folks 15:47:05 ack dom 15:47:05 dom, you wanted to ask pixel formats/color space 15:47:53 ningxin: I observed significant performance benefits using WebNN 15:47:56 dom: did you get a sense whether WebNN does contribute to the better perf of the WebNN/WebGPU pipeline? 15:48:26 ningxin_hu: yes 15:48:27 ningxin_hu: the WebGL pipeline runs into freezes with the browser UI, in which case the FPS goes down to 0 or 1 15:48:41 ... with an entry level GPU 15:48:56 q? 15:49:09 ... WebGPU/WebNN runs with over 10 FPS in that situation 15:49:16 q+ to ask pixel formats/color space 15:49:44 q+ 15:49:52 q- later 15:49:56 ack RafaelCintron 15:50:24 RafaelCintron: if you run your analysis on windows, we can trace the source of cpu usage 15:50:36 ningxin_hu: that would be great - it is on windows; I'll follow up with you 15:50:39 ack dom 15:50:39 dom, you wanted to ask pixel formats/color space 15:51:45 dom: in issue 226 we discussed VideoFrame does not let you pick pixel format/color space you get from the camera, impact on shaders? 15:51:57 ningxin_hu: I noted that discussion in the thread 15:52:18 ... in my implementation, I use the WebGPU copy to external texture which turns it into RGB format 15:52:38 ... at this stage, this hasn't been a problem 15:53:11 q? 15:53:16 ... @@@ importExternalTexture @@@ will be a good next step to explore the performance implication of this 15:54:09 anssi: this started from a request from the WebRTC WG - is it time to report back to that WG? 15:54:55 dom: a good question, I'll bring this up with WebRTC WG chairs on our regular call next week, we should present outcome of this prototyping on WebRTC WG's mid-April call 15:55:22 ... we're pushing the limits of many bleeding edge web features in development with this prototyping 15:55:26 s/we should present/I'll suggest we present/ 15:55:37 q? 15:56:09 q? 15:56:23 Topic: Candidate Recommendation proposed new features 15:56:36 Subtopic: WebNN should support int8 quantized models 15:56:44 -> WebNN should support int8 quantized models https://github.com/webmachinelearning/webnn/issues/128 15:57:37 anssik: looking for input on int8 support as part of our first release of WebNN 15:58:17 chai: +1 - quantized int8 models for CR is important due to new NPU coming to the market; this would otherwise be a serious shortcoming 15:58:31 ... it also impacts device selection re NPU 15:58:40 ... they should be considered together 15:59:12 q? 15:59:16 anssik: not hearing pushback, I'll mark it for CR 15:59:19 Subtopic: WebNN / WebGPU interop 15:59:32 -> WebGPU issue: WebNN / WebGPU interop https://github.com/gpuweb/gpuweb/issues/2500 15:59:57 anssik: what are remaining investigation items once #257 lands? 16:01:19 ningxin_hu: my plan is to review Chai's update PR #257 and prototype that in Chromium, and then update the samples accordingly and that would be a good milestone for discussion with WebGPU WG 16:02:14 chai: I think one of the WebGPU topics is we're wondering how Vulkan Linux community and Apple CoreML will implement it, when we push for WebGPU review then we get explicit review from those communities I assume 16:02:20 q+ 16:02:43 ack RafaelCintron 16:03:15 RafaelCintron: in the WebGPU CG Apple is present and attends all meetings 16:03:38 ... no Vulkan reps in the CG, but people who are familiar with it are participants 16:04:39 q? 16:05:16 RRSAgent, draft minutes 16:05:16 I have made the request to generate https://www.w3.org/2022/03/24-webmachinelearning-minutes.html anssik 17:02:04 anssik has joined #webmachinelearning 18:06:42 Zakim has left #webmachinelearning