14:53:06 RRSAgent has joined #webmachinelearning 14:53:06 logging to https://www.w3.org/2022/12/15-webmachinelearning-irc 14:53:09 RRSAgent, make logs Public 14:53:09 please title this meeting ("meeting: ..."), anssik 14:53:25 Zakim, prepare meeting 14:53:26 RRSAgent, make logs Public 14:53:27 please title this meeting ("meeting: ..."), anssik 14:53:35 Meeting: WebML WG Teleconference – 15 December 2022 14:53:39 Chair: Anssi 14:53:43 Agenda: https://github.com/webmachinelearning/meetings/blob/main/telcons/2022-12-15-wg-agenda.md 14:53:55 Scribe: Anssi 14:53:55 scribeNick: anssik 14:54:22 ghurlbot, this is webmachinelearning/webnn 14:54:22 anssik, OK 14:54:33 Present+ Anssi_Kostiainen 14:54:37 Regrets+ Dominique_Hazael-Massieux 14:54:41 RRSAgent, draft minutes 14:54:41 I have made the request to generate https://www.w3.org/2022/12/15-webmachinelearning-minutes.html anssik 14:59:12 ningxin_hu has joined #webmachinelearning 14:59:20 Present+ Zoltan_Kis 14:59:25 Present+ Ningxin_Hu 15:00:05 bruce_dai has joined #webmachinelearning 15:00:16 Present+ Bruce_Dai 15:02:35 Topic: WebNN API Candidate Recommendation open issues 15:02:41 -> Current CR issues https://github.com/webmachinelearning/webnn/labels/cr 15:02:45 Subtopic: Web platform tests 15:02:53 anssik: WPT tests tracker issue is #265 15:02:53 https://github.com/webmachinelearning/webnn/issues/265 -> Issue 265 WPT tests tracker (BruceDai) cr 15:03:11 ... thanks Bruce for keeping the first comment in that issue updated. 15:03:18 ... Bruce reports in that issue: 15:03:30 ... - All submitted WPT WebNN tests PRs have been approved by Dwayne. 15:04:08 ... - 414 WPT WebNN operations tests covered 39 ops of total 40 ops for first wave models. 15:04:26 ... - Bruce is still working on last conv2dTranspose op tests. 15:04:36 Present+ Chai_Chaoweeraprasit 15:04:41 chai has joined #webmachinelearning 15:05:28 Subtopic: Web platform tests: WebNN API IDL Tests 15:05:32 ghurlbot, this is web-platform-tests/wpt 15:05:32 anssik, OK 15:05:37 ... let's first discuss WebNN API IDL Tests 15:05:52 anssik: first, IIUC WPT PR #37227 is the major blocker. 15:05:53 https://github.com/web-platform-tests/wpt/issues/37227 -> Pull Request 37227 Fixed CI failure of idlharness.js for unexposed overloaded functions. (BruceDai) infra, idlharness.js 15:06:12 ... PR #37227 fixes a CI failure introduced by the automated PR created by @webref/idl bot in PR #36908 15:06:12 https://github.com/webref -> @webref 15:06:13 https://github.com/web-platform-tests/wpt/issues/36908 -> Pull Request 36908 Sync interfaces/ with @webref/idl 3.22.3 (github-actions[bot]) interfaces 15:06:47 ... PR #37227 unblocks all the PRs in “1. WebNN API IDL Tests” category enumerated at https://github.com/webmachinelearning/webnn/issues/265#issue-1226221213 as well as all the Chromium CLs e.g. https://chromium-review.googlesource.com/c/chromium/src/+/4005570 that run wpt tests in Chromium CI. 15:07:14 ... to ensure we meet the CR readiness criteria for testing 15:07:18 ... I've asked Dom to help with WPT PR #37227 that lacks final touch review to unblock all WebNN API IDL Tests PRs and Chromium implementation CLs (blocked by Chromium CI that imports and runs wpt tests) 15:08:23 Bruce: #37227 also blocks WebNN op tests, I also asked Belem for help 15:09:13 Subtopic: Web platform tests: WebNN API JS Tests (testharness.js) for ops 15:09:18 ghurlbot, this is web-platform-tests/wpt 15:09:18 anssik, OK 15:09:35 anssik: thanks to Bruce for the heroic effort in creating these PRs and for Dwayne equally for carefully reviewing these PRs! 15:10:06 anssik: the following PRs are approved by Dwayne but at least 1 approving review is required by wpt repo reviewers with write access: 15:10:23 #37420 batchnorm 15:10:23 #36782 concat 15:10:23 #37067 clamp 15:10:23 #37275 conv2d 15:10:23 #37337 binary 15:10:23 https://github.com/web-platform-tests/wpt/issues/36782 -> Pull Request 36782 [webnn] Add tests for concat operation. (BruceDai) webnn, wg-s_webmachinelearning 15:10:24 https://github.com/web-platform-tests/wpt/issues/37420 -> Pull Request 37420 Add webnn batchnorm tests (BruceDai) webnn, wg-s_webmachinelearning 15:10:24 https://github.com/web-platform-tests/wpt/issues/37067 -> Pull Request 37067 Add webnn clamp tests (BruceDai) webnn, wg-s_webmachinelearning 15:10:24 #37351 unary 15:10:24 #37068 gemm 15:10:24 #37340 leakyRelu 15:10:24 https://github.com/web-platform-tests/wpt/issues/37275 -> Pull Request 37275 Add webnn conv2d tests (BruceDai) webnn, wg-s_webmachinelearning 15:10:25 #37379 pooling 15:10:25 #37069 matmul 15:10:25 #37380 reduce 15:10:25 #37070 reshape 15:10:25 #37071 relu 15:10:26 #37343 sigmoid 15:10:26 #37072 slice 15:10:26 #37254 softmax 15:10:27 https://github.com/web-platform-tests/wpt/issues/37337 -> Pull Request 37337 Add webnn binary tests (BruceDai) webnn, wg-s_webmachinelearning 15:10:27 #37398 split 15:10:27 #37352 squeeze 15:10:27 #37344 tanh 15:10:28 #37315 transpose 15:10:31 https://github.com/web-platform-tests/wpt/issues/37340 -> Pull Request 37340 Add webnn leakyRelu tests (BruceDai) webnn, wg-s_webmachinelearning 15:10:34 https://github.com/web-platform-tests/wpt/issues/37068 -> Pull Request 37068 Add webnn gemm tests (BruceDai) webnn, wg-s_webmachinelearning 15:10:38 https://github.com/web-platform-tests/wpt/issues/37351 -> Pull Request 37351 Add webnn unary tests (BruceDai) webnn, wg-s_webmachinelearning 15:10:41 https://github.com/web-platform-tests/wpt/issues/37069 -> Pull Request 37069 Add webnn matmul tests (BruceDai) webnn, wg-s_webmachinelearning 15:10:44 https://github.com/web-platform-tests/wpt/issues/37379 -> Pull Request 37379 Add webnn pooling tests (BruceDai) webnn, wg-s_webmachinelearning 15:10:48 https://github.com/web-platform-tests/wpt/issues/37071 -> Pull Request 37071 Add webnn relu tests (BruceDai) webnn, wg-s_webmachinelearning 15:10:51 https://github.com/web-platform-tests/wpt/issues/37070 -> Pull Request 37070 Add webnn reshape tests (BruceDai) webnn, wg-s_webmachinelearning 15:10:55 https://github.com/web-platform-tests/wpt/issues/37380 -> Pull Request 37380 Add webnn reduce tests (BruceDai) webnn, wg-s_webmachinelearning 15:10:58 https://github.com/web-platform-tests/wpt/issues/37343 -> Pull Request 37343 Add webnn sigmoid tests (BruceDai) webnn, wg-s_webmachinelearning 15:11:02 https://github.com/web-platform-tests/wpt/issues/37254 -> Pull Request 37254 Add webnn softmax tests (BruceDai) webnn, wg-s_webmachinelearning 15:11:05 https://github.com/web-platform-tests/wpt/issues/37072 -> Pull Request 37072 Add webnn slice tests (BruceDai) webnn, wg-s_webmachinelearning 15:11:08 https://github.com/web-platform-tests/wpt/issues/37398 -> Pull Request 37398 Add webnn split tests (BruceDai) webnn, wg-s_webmachinelearning 15:11:12 https://github.com/web-platform-tests/wpt/issues/37352 -> Pull Request 37352 Add webnn squeeze tests (BruceDai) webnn, wg-s_webmachinelearning 15:11:15 https://github.com/web-platform-tests/wpt/issues/37344 -> Pull Request 37344 Add webnn tanh tests (BruceDai) webnn, wg-s_webmachinelearning 15:11:19 https://github.com/web-platform-tests/wpt/issues/37315 -> Pull Request 37315 Add webnn transpose tests (BruceDai) webnn, wg-s_webmachinelearning 15:11:58 anssik: all these PRs have been reviewed by conformance testing expert and what is lacking is the sign-off from wpt repo reviewers 15:11:59 ... I can ask Dom again for help to mass-merge all of these. 15:12:13 ... to clarify, all of these PRs only touch files in the https://github.com/web-platform-tests/wpt/tree/master/webnn dir and its subdirs but not outside of it? 15:12:46 Bruce: yes 15:13:07 ... thanks again Bruce and Dwayne! 15:14:10 Subtopic: Web platform tests: open webnn-baseline PRs 15:14:13 ghurlbot, this is webmachinelearning/webnn-baseline 15:14:13 anssik, OK 15:14:13 anssik: webnn-baseline is the pure JS implementation of WebNN API we use for testing purposes. Let's discuss and review the open webnn-baseline PRs first. 15:14:16 -> Open webnn-baseline PRs https://github.com/webmachinelearning/webnn-baseline/pulls 15:14:21 #22 15:14:21 https://github.com/webmachinelearning/webnn-baseline/issues/22 -> Pull Request 22 [tool] Add scripts to generate test data for WPT WebNN operations tests (BruceDai) 15:14:53 anssik: PR #22 adds Node.js scripts `gen-.js` in the `tool` folder to generate test data for WebNN operations tests of web-platform-tests 15:14:53 ... approved by Dwayne, ready to merge? 15:15:19 Bruce: I will refine this PR according to the latest test data json file, I'm working on it and will ping folks for review after finished. 15:15:29 sgtm 15:15:46 anssik: In addition we have the following open PRs that generate test data for various tests, all approved by Dwayne with some comments 15:15:54 #23 matmul 15:15:54 #24 reshape 15:15:54 #25 relu 15:15:54 #26 slice 15:15:54 https://github.com/webmachinelearning/webnn-baseline/issues/23 -> Pull Request 23 generate test data for WebNN matmul tests of WPT (BruceDai) 15:15:54 https://github.com/webmachinelearning/webnn-baseline/issues/24 -> Pull Request 24 generate test data for WebNN reshape tests of WPT (BruceDai) 15:15:54 https://github.com/webmachinelearning/webnn-baseline/issues/25 -> Pull Request 25 generate test data for WebNN relu tests of WPT (BruceDai) 15:15:55 https://github.com/webmachinelearning/webnn-baseline/issues/26 -> Pull Request 26 generate test data for WebNN slice tests of WPT (BruceDai) 15:16:00 anssik: are we ready to merge also these? 15:16:34 Bruce: I will refine the scripts in these PRs too similarly 15:17:14 q? 15:17:38 Chai: Dwayne in on vaca until January, baseline feedback should be clear and easy to address 15:18:34 ... I reviewed Dwayne's feedback and they are not suggesting major changes, so Bruce can address them easily 15:18:45 ... once addressed we can merge, if approval needed I can do that 15:19:19 ... Huge thanks to our key test contributors Bruce and Dwayne, Ningxin for making the WebNN testing story more complete and compelling! 15:19:41 ... the improved test coverage provided by these PRs allow us to demonstrate implementation experience, an important CR readiness requirement 15:20:09 Subtopic: Use modern WebIDL conventions 15:20:12 ghurlbot, this is webmachinelearning/webnn 15:20:12 anssik, OK 15:20:34 anssik: another important area for CR readiness is to align the spec with modern WebIDL conventions and patterns 15:21:04 ... Thanks to Zoltan for spearheading this editorial refinement effort! 15:21:39 ... I've helped Zoltan by reviewing some of his PRs to ease the workload from our busy spec editors who are also delivering to other related commitments such as the Chromium implementation 15:21:56 ... I've maintained the living task list of editorial work broken down into: Done / WIP / TODO 15:22:01 -> Living task list https://github.com/webmachinelearning/webnn/issues/210#issuecomment-1326361748 15:22:16 anssik: The following editorial updates have been Done: 15:22:25 ... - #309 Add MLGraphBuilder constructor steps 15:22:26 https://github.com/webmachinelearning/webnn/issues/309 -> Pull Request 309 [closed] Add constructor steps to MLGraphBuilder (zolkis) 15:22:44 ... - #310 Improve createContext method steps and add method steps for createContextSync 15:22:44 https://github.com/webmachinelearning/webnn/issues/310 -> Pull Request 310 [closed] MlContext improvements (zolkis) 15:22:57 ... - #311 Improve spec prose on "exists" 15:22:58 https://github.com/webmachinelearning/webnn/issues/311 -> Issue 311 [closed] Improve spec prose on "exists" (zolkis) 15:23:08 ... - #314 Add method steps for createContextSync 15:23:08 https://github.com/webmachinelearning/webnn/issues/314 -> Issue 314 [closed] Handle errors according to Web IDL conventions (zolkis) 15:24:35 ... WIP tasks are: 15:24:42 ... - #308 Validate MLContext 15:24:42 https://github.com/webmachinelearning/webnn/issues/308 -> Issue 308 Validate MLContext (zolkis) 15:24:54 ... - #316 Review sync vs async compute differences 15:24:55 https://github.com/webmachinelearning/webnn/issues/316 -> Issue 316 Review sync vs async compute differences (zolkis) 15:25:08 ... Zoltan wanted to ask some questions re sync vs async compute differences to progress on this task 15:26:34 Zoltan: in #316 I noticed the following opportunities/differences: 15:26:43 ... Inputs and their validation is the same in both, so it can be factored out and referred from both (done). 15:26:49 ... The async steps currently are not really asynchronous. Also, they throw from a parallel job, so I assume that needs (minor) fixing (done). 15:26:59 ... There is only one difference between the sync and async steps (excluding promise-related): the async version compares the byte length of |outputTensor| with the length of the output descriptor corresponding to |key|, whereas the sync version compares the same with the byte length of |value|. If that is deliberate, we need an editor's note on why so - please point me to relevant background there. 15:28:54 ningxin_hu: there are two pointers, refer to the previous WebNN implementation hosted on GH, that one is somehow outdated and comment in there is invalid 15:29:33 ... I'll share links to the up to date implementation 15:30:08 Zoltan: interested in it sync or async steps we should follow, or should there be move checks? It looks like XNNPACK has more steps with checks 15:30:23 Ningxin: we should use XNNPACK inspired steps 15:30:35 ... implementation experience can inform the algorithm steps 15:30:48 ... some learnings there we can backport into the spec 15:30:59 Zoltan: I'm wondering if I'm looking at the latest code? 15:31:19 ningxin_hu: you're referring to the big CL for the Mojo implementation, that is not fully reviewed 15:31:47 ... I suggest we align on the XNNPACK backend at its current state because it is being merged into the Chromium code and split into smaller CLs and merged 15:32:02 ... I'll share mode details in the issue #316 15:32:15 Zoltan: execute and compute methods are on the graph, correct? 15:33:00 ningxin_hu: correct, IDL change Anssi mentioned was blocked by wpt infra PR and we are waiting for that PR to be merged for Chromium CI to fetch those tests, and then I plan to align the Chromium impl with the latest IDL 15:33:34 ... impl itself is waiting for XNNPACK, JS binding can be bound to that, we can focus on the MLGraph impl 15:33:44 Zoltan: intention is to move compute to context in the C++ code too 15:33:47 ningxin_hu: correct 15:33:59 Zoltan: current spec refers to impl as an internal slot 15:34:28 ... when we refer to impl it always means impl so we don't need an internal slot for that, but if there's another use for that internal slot we can use it 15:34:56 ningxin_hu: re internal slots, if you look at the Chromium impl, there's a dep in Blink on native lib to implement the native implementations 15:35:09 ... e.g. XNNPACK depends on XNNPACK lib to execute the ops 15:35:28 ... similar to DML backend in the big CL, there DML API will run the graph 15:35:46 ... XNNPACK or DML backend is represented by the implementation internal slot 15:35:54 ... maybe we can find a better way to define that 15:36:26 ... internal slot represents the native library or OS API for ML component 15:36:47 Zoltan: this is common for many Web APIs, but they do not have internal slot representing the implementation 15:36:57 ningxin_hu: fine to use a convention that is widely used 15:37:14 Zoltan: will establish a convention and put up a PR for that, work with Anssi 15:37:42 ... the rest of the algorithms mentioned in Anssi's tracking issue, Domenic's feedback was to document how inputs and outputs are handled towards the algorithms 15:38:00 ... we don't need to specify how e.g. "bubble sort" works to give an example 15:38:14 ... so expect short prose for those 15:38:24 ... TODO tasks are: 15:38:44 ... - Establish an algorithmic method steps convention for operations and apply that convention across all the ops. 15:39:28 s/... TODO/Anssi: ... TODO 15:40:07 ... Thanks Zoltan for your editorial work that helps align the WebNN API spec with modern WebIDL conventions 15:40:22 ... this work further increases our confidence we're reaching CR readiness 15:40:45 Zoltan: similarly tests should be aligned, so affects testing precision 15:41:10 ... MLGraph is an opaque object in script, what are the plans there, are we going to add methods to it? 15:41:30 ... I had some suggestions, but can discuss that later 15:41:55 Anssi: is MLGraph issue self-contained? 15:42:15 Zoltan: not yet clear, not clear for developers what it is 15:42:34 ... we have builder spawn from the context 15:42:51 MLGraph is an intermediary step that could we an internal slot 15:43:01 ... need to understand if there's a reason to keep it around 15:43:41 https://source.chromium.org/chromium/chromium/src/+/main:third_party/blink/renderer/modules/ml/webnn/ml_graph.idl 15:43:43 ningxin_hu: if you look at the Chromium code MLGraph is an opaque object so far, aligns with the current spec 15:43:47 ... ) 15:43:56 s/.. )/(pastes related code link) 15:44:14 Zoltan: in JS we don't usually have opaque interfaces, we hide it 15:44:53 ningxin_hu: MLGraph object represent a compiler computational graph, we should make this clear, e.g. in XNNPACK backend we inherit this interface 15:45:10 Zoltan: any methods we could expose on MLGraph such as shares or dimensions? 15:45:24 s/shares/shapes 15:45:33 ningxin_hu: I think that is a good idea, we could expose some methods to allow the developer to query input and output dimensions 15:45:41 ... currently not proposed for the WG for discussion 15:46:57 Anssi: let's not rush this new feature, open a new issue to discuss this in the WG 15:47:31 ... add an informative note that the opaque interface is there for future extensibility. 15:47:41 Zoltan: sounds good 15:48:13 ... another concern that we have WebGPU and user given context 15:48:33 ... why not have a single context that can be WebGPU, GPU, CPU 15:49:13 ... I did some experiments and a good abstraction could be to appear on the context level, have CPU context, GPU context, WebGPU context, AI accelerator context 15:49:54 https://github.com/webmachinelearning/webnn/issues/302 15:50:47 Anssi: I consider #302 v2 feature 15:50:48 https://github.com/webmachinelearning/webnn/issues/302 -> Issue 302 API simplification: context types, context options, createContext() (zolkis) 15:51:58 q? 15:52:05 Subtopic: Open PRs 15:52:10 ghurlbot, this is webmachinelearning/webnn 15:52:10 anssik, OK 15:52:16 anssik: we have two minor open PRs that emerged from Chromium implementation experience: 15:52:20 ... #294 Use unsigned long for size related options of conv2d, convTranspose2d and pooling operations 15:52:21 https://github.com/webmachinelearning/webnn/issues/294 -> Pull Request 294 Use unsigned long for size related options of conv2d, convTranspose2d and pooling operations (huningxin) 15:52:36 ... #306 Use unsigned long type for sizes option of resample2d 15:52:37 https://github.com/webmachinelearning/webnn/issues/306 -> Pull Request 306 Use unsigned long type for sizes option of resample2d (huningxin) 15:52:45 ... these PRs are pending Chai's review, can you PTAL? 15:53:20 Chai: I'll take a look 15:53:24 Anssi: Thanks! 15:53:45 Topic: WebML WG Charter 2023-2025 proposal ready for WG review 15:53:49 ghurlbot, this is w3c/machine-learning-charter 15:53:49 anssik, OK 15:54:10 anssik: I'm pleased to announce the Web Machine Learning Working Group Charter for 2023-2025 is now ready for WG review. 15:54:24 ... I'll now ask you to please review the draft PR and open issues, provide your comments and open new issues as appropriate to help shape the WG's technical scope. 15:54:28 -> Open charter issues https://github.com/w3c/machine-learning-charter/issues 15:54:34 -> Diff to the initial charter https://services.w3.org/htmldiff?doc1=https%3A%2F%2Fwww.w3.org%2F2021%2F04%2Fweb-machine-learning-charter.html&doc2=https%3A%2F%2Fpr-preview.s3.amazonaws.com%2Fw3c%2Fmachine-learning-charter%2Fpull%2F19.html 15:54:49 anssik: please see the Diff to the initial charter for changes 15:54:54 ... this is the summary of changes I proposed: 15:55:29 ... - Clarify in Out of Scope that "base linear algebra operations are out of scope" refers to the 15:55:29 Basic Linear Algebra Subprograms (BLAS) interface. (related #20) 15:55:29 https://github.com/w3c/machine-learning-charter/issues/20 -> Issue 20 WebGPU interoperability (anssiko) 15:55:49 ... - Note WebGPU interoperability expectation in Coordination (fix #20) 15:56:07 ... - Clarify WebRTC coordination (fix #21) 15:56:08 https://github.com/w3c/machine-learning-charter/issues/21 -> Issue 21 WebRTC coordination (anssiko) 15:56:37 ... - Add high-level to low-level op decompose path to facilitate progressive lowering by implementations (fix #22, related #28) 15:56:37 https://github.com/w3c/machine-learning-charter/issues/22 -> Issue 22 Level of abstraction for neural net operations (anssiko) 15:56:37 https://github.com/w3c/machine-learning-charter/issues/28 -> Issue 28 OpenXLA coordination (anssiko) 15:56:57 ... - Expand Motivation and Background and clarify priorities in Scope (fix #23) 15:56:57 https://github.com/w3c/machine-learning-charter/issues/23 -> Issue 23 Set of ops supported must be more comprehensive (anssiko) 15:57:07 ... - Add OpenXLA Project to Coordination (fix #28) 15:57:48 ... notably, based on my assessment, these changes do not alter the Scope of Work and do not add any new deliverables 15:58:12 ... to that end, this will minimize the need for internal legal reviews and my expectation is the current participants are not required to rejoin the WG when the new charter is operational and the transition is frictionless 15:58:36 ... any questions or comments for any of the changes or for rechartering in general? 15:59:02 ... Please provide your feedback by the end of the year. We are expected to start the AC review during Q1 to have an approval in time for a smooth transition. 15:59:10 q? 15:59:26 Topic: Happy Holidays! 15:59:33 anssik: Thank you all for a hugely productive 2022! 15:59:37 ... Looking back, we've accomplished a lot this year. 15:59:48 ... We've raced to bring the WebNN API spec to CR readiness quality for an early 2023 publication. 15:59:57 ... We've made substantial progress in implementation on multiple backends across OSes. 16:00:16 ... We have showcase WebNN-powered user experiences through running samples, and evolved the WebNN Polyfill and WPT tests and more! 16:00:31 ... As we now start to transition to the holiday season, I want to wish those whom it concerns, relaxing holiday. 16:00:36 ... please spend time with your family and friends. 16:00:50 ... We'll skip the next WG call that overlaps with the holiday season and will be back re-energized 12 January 2023! 16:00:57 ... See you in 2023! 16:01:19 RRSAgent, draft minutes 16:01:19 I have made the request to generate https://www.w3.org/2022/12/15-webmachinelearning-minutes.html anssik 18:00:51 Zakim has left #webmachinelearning