14:07:48 <RRSAgent> RRSAgent has joined #webmachinelearning
14:07:52 <RRSAgent> logging to https://www.w3.org/2023/11/02-webmachinelearning-irc
14:07:52 <Zakim> RRSAgent, make logs Public
14:07:53 <Zakim> please title this meeting ("meeting: ..."), anssik
14:07:53 <anssik> Meeting: WebML WG Teleconference – 2 November 2023
14:07:57 <anssik> Chair: Anssi
14:08:01 <anssik> Agenda: https://github.com/webmachinelearning/meetings/blob/main/telcons/2023-11-02-wg-agenda.md
14:08:05 <anssik> Scribe: Anssi
14:08:11 <anssik> scribeNick: anssik
14:08:21 <anssik> gb, this is webmachinelearning/webnn
14:08:21 <gb> anssik, OK.
14:08:25 <anssik> Present+ Anssi_Kostiainen
14:08:33 <anssik> Regrets+ Dominique_Hazael-Massieux
14:08:43 <anssik> Present+ Joshua_Lochner
14:08:48 <anssik> Present+ Ningxin_Hu
14:08:54 <anssik> Present+ Etienne_Noel
14:08:59 <anssik> Present+ Rafael_Cintron
14:09:05 <anssik> Present+ Dwayne_Robinson
14:09:12 <anssik> Present+ Chai_Chaoweeraprasit
14:09:16 <anssik> Present+ Vivek_Sekhar
14:09:17 <Joshua_Lochner> and Joshua Bell :)
14:09:31 <anssik> Present+ Joshua_Bell
14:09:38 <anssik> RRSAgent, draft minutes
14:09:39 <RRSAgent> I have made the request to generate https://www.w3.org/2023/11/02-webmachinelearning-minutes.html anssik
14:10:03 <dwayner> dwayner has joined #webmachinelearning
14:10:17 <anssik> Topic: WebNN v2: Review transformer ops spec contributions (continued)
14:10:17 <anssik> anssik: issue #375
14:10:18 <gb> https://github.com/webmachinelearning/webnn/issues/375 -> Issue 375 Support for transformers (by dontcallmedom) [v2] [operation set]
14:10:43 <anssik> ... continuing from our previous call, recapping with some resources first
14:10:55 <anssik> -> https://github.com/webmachinelearning/webnn/blob/main/CONTRIBUTING.md#proposing-and-adding-a-new-operation Guidelines for adding new operations
14:11:07 <Etienne> Etienne has joined #webmachinelearning
14:11:07 <anssik> ... we have identified use cases and sample models:
14:11:17 <anssik> -> https://huggingface.co/runwayml/stable-diffusion-v1-5 Text-to-image: stable-diffusion-v1-5
14:11:17 <anssik> -> https://github.com/facebookresearch/segment-anything Image segmentation: segment-anything
14:11:17 <anssik> -> https://huggingface.co/openai/whisper-tiny Speech-to-text: whisper-tiny
14:11:17 <anssik> -> https://huggingface.co/optimum/t5-small Text-to-text generation  (encoder-decoder): t5-small
14:11:17 <anssik> -> https://huggingface.co/facebook/m2m100_418M Text-to-text generation (encoder-decoder): m2m100_418M
14:11:19 <anssik> -> https://huggingface.co/meta-llama/Llama-2-7b Text-generation (decoder-only): Llama-2-7b
14:11:35 <anssik> ... we have done an op decomposition assessment:
14:11:40 <anssik> -> https://docs.google.com/spreadsheets/d/1ELfHuv2UqP2LoXWLgqsC0L8T_qqfBx48KxzFighl8d8/ Transformer Models Analysis)
14:12:02 <anssik> ... Chai, you indicated you've been working on a big PR for the new op spec definitions
14:12:09 <anssik> ... anything you'd like to bring to the group's attention in this meeting, or questions to ask from the group?
14:12:33 <anssik> ... I'm wondering whether it'd help if the big PR is split into smaller PRs so those pieces could be reviewed in parallel while new ops are still being defined? Or are there dependencies that make this not practical?
14:12:34 <anssik> q?
14:12:50 <anssik> q?
14:13:17 <anssik> Chai: this PR is huge change, but mostly additions, no breaking changes yet, still working on it, past midpoint
14:13:55 <anssik> ... some comments, the proposal makes sense, is a natural extension for what we have, there was a suggestion to combine a few ops that have nothing to do with the data itself, but update its shape
14:14:41 <anssik> ... e.g. squeeze, unsqueeze, flatten2d
14:15:37 <anssik> ... the other part is normalization, a good design example of design principles we have
14:16:40 <anssik> ... normalization ops are similar
14:17:32 <anssik> ... my preference to keep things in one PR, want to keep things consistent
14:18:55 <anssik> anssik: the PR does not need to be perfect, we'll review it in the group and perfect it together
14:19:29 <anssik> q?
14:19:42 <Ningxin_Hu> q+
14:20:42 <anssik> ack Ningxin_Hu
14:21:03 <anssik> Ningxin_Hu: update about Transformer Models Analysis
14:21:29 <anssik> ... int8 quantized models for whisper-tiny have been added, encoder and decoder columns for int8
14:21:43 <Joshua_Lochner> q+
14:21:49 <anssik> ... feedback from the community was quantized model is used in production
14:22:10 <anssik> ... four new ops used in the quantized model, conv integer, matmul integer
14:23:07 <anssik> ... dequantized liner, dynamic quantize linear
14:23:18 <Chai> q+
14:23:21 <anssik> q?
14:23:24 <anssik> ack Joshua_Lochner
14:24:07 <anssik> Joshua_Lochner: context for 8-bit weights, people wanted to use smaller weight with Transformers.js
14:24:33 <Joshua_Lochner> https://huggingface.co/spaces/Xenova/whisper-web
14:24:39 <Joshua_Lochner> https://i.imgur.com/oPCEtXs.png
14:24:39 <anssik> ... I experimented with this, text-to-speech not good perf, other way around  speech-to-text performs well
14:25:07 <anssik> ... as an example, whisper-tiny is 40 MB with 8-bit quantized
14:25:27 <anssik> ... very little quality degradation, able to run on mobile and low resource computers, surprising results
14:26:05 <Joshua_Lochner> https://github.com/huggingface/distil-whisper
14:26:17 <anssik> ... another thing, distilled version of whisper-medium and whisper-large to be released in an hour
14:26:38 <anssik> ... paper is also ready
14:28:07 <anssik> q?
14:28:16 <anssik> ack Chai
14:28:33 <anssik> Chai: noting the transformers ops will not inclulde quantized ops yet in the big PR
14:28:57 <anssik> ... regarding the size of the spec, it is pretty big, thinking factoring out some section to a more suitable medium
14:29:22 <anssik> ... e.g. implementation sections are about parameter validation, some parts covered by WPT tests
14:29:57 <anssik> ... I feel as the spec grows, it will be more important to keep only the relevant part of the ops in the document, and offload some of this validation in WPT in a more complete way
14:29:57 <anssik> q?
14:30:38 <Joshua_Lochner> q+
14:33:50 <anssik> Chai: implementation sections could be offloaded, e.g. param validation
14:33:51 <anssik> q?
14:33:54 <anssik> ack Joshua_Lochner
14:34:41 <anssik> Joshua_Lochner: these new models use the same architecture as the already existing, the distilled versions being released, internal decoder layers are removed, no new ops added
14:35:13 <anssik> ... any op in whisper-tiny encoder or decoder will be the same for 8-bit quantized versions of the models
14:35:14 <anssik> q?
14:35:47 <anssik> Topic: Enhancements
14:35:57 <anssik> Subtopic: 0D scalars
14:36:00 <anssik> anssik: #390
14:36:01 <gb> https://github.com/webmachinelearning/webnn/issues/390 -> Issue 390 0D scalars (by fdwr)
14:36:09 <anssik> ... zero-dimension scalars, we discussed this earlier this year and I'd like to drive this to resolution
14:36:27 <anssik> ... Dwayne identified a need for zero-dimension scalars while prototyping additional models we're now discussing
14:36:43 <anssik> ... there's data that suggests this enhancement should be baked into the API spec, consider:
14:36:58 <anssik> ... - every ML library represents 0D scalars via the shape e.g. Numpy, TF, PyTorch, ONNX, XNNPACK, SafeTensors file format too
14:37:13 <anssik> ... - the proposed spec enhancement is very small, deletion of one line in https://www.w3.org/TR/webnn/#api-mloperand-create check dimensions steps:
14:37:20 <anssik> ... "2. If dimensions.length is 0, return false."
14:37:40 <anssik> ... recently two related proposed changes were suggested for consideration by Nignxin if we add this 0D scalar support:
14:37:55 <anssik> ... - make MLOperandDescriptor.dimensions a required field to allow distinguish the scalar and 1D tensor
14:38:08 <anssik> ... - drop the scalar-specific MLGraphBuilder.constant(value, dataType) variant, discussed in issue #475
14:38:08 <gb> https://github.com/webmachinelearning/webnn/issues/475 -> Issue 475 Remove `builder.constant(value, dataType)` variant (by huningxin)
14:38:30 <anssik> ... it looks like the first (make MLOperandDescriptor.dimensions a required field) should be rolled into the PR that adds support for 0D scalars (deletes the one line from check dimensions steps), correct?
14:38:34 <anssik> ... issue #475 to be addressed in a separate PR?
14:38:57 <anssik> q?
14:39:20 <anssik> Dwayne: I sent a PR for this, not an enhancement per se, but the wording needs to be fixed
14:39:38 <anssik> PR #476
14:39:39 <gb> https://github.com/webmachinelearning/webnn/issues/476 -> Pull Request 476 Fix dimensions for 0D scalars (by fdwr)
14:39:49 <anssik> q?
14:40:17 <anssik> RRSAgent, draft minutes
14:40:18 <RRSAgent> I have made the request to generate https://www.w3.org/2023/11/02-webmachinelearning-minutes.html anssik
14:40:28 <anssik> Subtopic: Softmax axis absent
14:40:33 <anssik> anssik: issue #466
14:40:34 <gb> https://github.com/webmachinelearning/webnn/issues/466 -> Issue 466 Softmax axis absent (by fdwr)
14:40:59 <anssik> ... issue reported by Wanming (thanks!) in ONNX Runtime WebNN EP review
14:41:16 <AramZS> AramZS has joined #webmachinelearning
14:41:19 <anssik> ... this is another great example of how our diverse impl experience informs the spec development, including feedback from web engine and framework implementations
14:41:40 <anssik> ... Dwayne summarized the issue well, TF/PT/ONNX all take an axis parameter, but WebNN's softmax does not
14:41:46 <anssik> ... platform support is good:
14:41:54 <anssik> ... - Apple Metal Performance Shaders softMax has an axis
14:42:03 <anssik> ... - Apple Model Intermediate Language (MIL) activation.softmax supports an axis
14:42:09 <anssik> ... - DirectML's DML_ACTIVATION_SOFTMAX1_OPERATOR_DESC supports an arbitrary axis list and dimensions
14:42:19 <anssik> ... to be investigated is XNNPACK, currently limited to 2D input
14:42:42 <anssik> ... Dwayne proposed two solutions: 1) update XNNPACK to accept an axis or 2) use existing XNNPACK operator plus a reshape or transpose
14:42:51 <Chai> q+
14:42:52 <anssik> ... Dwayne also proposes the required IDL changes in the issue
14:42:55 <anssik> q?
14:43:02 <anssik> ack Chai
14:43:50 <Ningxin_Hu> q+
14:43:55 <anssik> Chai: proposing we tackle these issues from impl and prototypes
14:43:56 <anssik> q?
14:43:58 <anssik> ack Ningxin_Hu
14:44:52 <anssik> Ningxin_Hu: question to Chrome folks, can we send a CL to prototype this and inform the spec effort+
14:44:56 <anssik> q?
14:45:30 <anssik> Joshua_Bell: fine with that
14:45:57 <Ningxin_Hu> sounds great, thanks!
14:46:29 <anssik> anssik: good to move ahead with a CL to help inform the spec design
14:46:37 <anssik> q?
14:46:39 <anssik> RRSAgent, draft minutes
14:46:41 <RRSAgent> I have made the request to generate https://www.w3.org/2023/11/02-webmachinelearning-minutes.html anssik
14:46:47 <anssik> Subtopic: split() into sizes not widely supported
14:46:51 <anssik> anssik: issue #392
14:46:52 <gb> https://github.com/webmachinelearning/webnn/issues/392 -> Issue 392 `split` into sizes are not widely supported (by huningxin)
14:46:59 <anssik> ... this is an issue raised by Jiawei in implementation review (thanks!)
14:47:09 <anssik> ... WebNN split supports two variants:
14:47:19 <anssik> ... - "number of split" (when splits argument is an unsigned long)
14:47:27 <anssik> ... - "split into sizes" (when splits argument is a sequence<unsigned long>)
14:47:37 <anssik> ... however, some backends such as XNNPACK don't support "split into sizes"
14:47:45 <anssik> ... possible solutions proposed by Ningxin:
14:47:57 <anssik> ... - decompose split into multiple slices (per informative emulation path in the spec, see https://www.w3.org/TR/webnn/#api-mlgraphbuilder-split)
14:48:07 <anssik> ... - throw an error and leave the framework to handle
14:48:18 <anssik> ... Dwayne notes many frameworks support variable length splits: TensorFlow, PyTorch, ONNX
14:48:25 <anssik> ... and provides the following options expanding upon Ningxin's initial proposal:
14:48:31 <anssik> ... 1. Keep dedicated split
14:48:37 <anssik> ... 2. Backend decomposes variable lengths into slice calls
14:48:44 <anssik> ... 3. Front end decomposes variable lengths into slice calls
14:48:49 <anssik> ... 4. Throw error if backend doesn't support them
14:48:54 <anssik> RRSAgent, draft minutes
14:48:56 <RRSAgent> I have made the request to generate https://www.w3.org/2023/11/02-webmachinelearning-minutes.html anssik
14:49:35 <anssik> Dwayne: Does XNNPACK support variable size windows for concat, which is splits' symmetric pair operator?
14:50:06 <anssik> q?
14:50:15 <Ningxin_Hu> q+
14:50:18 <anssik> ack Ningxin_Hu
14:50:43 <Ningxin_Hu> https://bugs.chromium.org/p/chromium/issues/detail?id=1492036
14:50:52 <anssik> Ningxin_Hu: we also opened another Chromium issue to benchmark split vs slice for DirectML backend, this will inform the spec design decision
14:51:41 <anssik> ... decomp split may have an overhead, we want to understand that better, get data from the benchmark
14:52:12 <anssik> q?
14:52:39 <anssik> Subtopic: Clarify the restriction for minValue and maxValue of MLClampOptions
14:52:45 <anssik> anssik: issue #396
14:52:46 <gb> https://github.com/webmachinelearning/webnn/issues/396 -> Issue 396 Clarify the restriction for `minValue` and `maxValue` of `MLClampOptions` (by huningxin)
14:53:18 <anssik> ... Current behavior: "WebNN clamp limits the input tensor element-wise within a range specified by the minimum and maximum values. The minimum and maximum values could be specified by MLClampOptions"
14:53:33 <anssik> ... Google Security team's Alex asked "can min & max be == floats and is that ok or not?"
14:53:47 <anssik> ... The current Chromium implementation requires "min <= max"
14:53:51 <anssik> ... TF.js implements this restriction too
14:54:04 <anssik> ... however XNNPACK is stricter than that and requires "min < max"
14:54:29 <anssik> ... so the WebNN spec needs to clarify this restriction, Dwayne provided data on how other implementations are doing this, TF, PyTorch, ONNX, C++, DirectML
14:54:38 <anssik> ... and the result is all frameworks accept inverted ranges
14:54:44 <anssik> ... proposed solutions:
14:54:56 <anssik> ... 1) accept inverted ranges (and impl adjust min/max values before passing to backend)
14:55:08 <anssik> ... 2) reject min > max, let caller adjust min/max values before calling
14:55:27 <anssik> q?
14:56:08 <anssik> Dwayne: this issue implies if backend does not support something, we should limit WebNN similarly
14:56:45 <anssik> ... there are ways WebNN can have one policy and backends can still implement them
14:56:47 <Ningxin_Hu> q+
14:56:48 <jsbell> q+
14:56:55 <RafaelCintron> q+
14:56:58 <anssik> q?
14:57:04 <anssik> ack Ningxin_Hu
14:57:54 <anssik> Ningxin_Hu: frameworks behave differently, does that mean we should avoid different behavior?
14:58:39 <anssik> Dwayne: first I want to check if this issue is about empty range or inverted range?
14:58:50 <anssik> Ningxin_Hu: inverted range as in the examples
14:59:05 <Chai> q+
14:59:05 <anssik> Dwayne: for consistency, WebNN backend should clamp the min and max values
14:59:18 <anssik> ... running WebNN on different backends would otherwise give inconsistent results
14:59:19 <anssik> q?
14:59:32 <anssik> ack jsbell
15:00:02 <anssik> jsbell: we need to assume web developers won't test on every platform, so subtle differences will be considered bugs
15:00:42 <anssik> ... platform differences such as is this codec supported where one can query if a codec is supported, some level of feature detection, this we can expect from developers, not subtle differences
15:01:24 <anssik> ... WebNN is a bit different, developers are going through a framework, even there framework authors need to know about these subtle differences
15:01:46 <anssik> ... except in some cases where are high-level op may be supported or not, then feature detection may be OK
15:02:10 <anssik> ... we need to provide a lot of documentation to framework authors, we need to make sure the behaviour is consistent
15:02:10 <anssik> q?
15:02:28 <anssik> ack RafaelCintron
15:02:46 <anssik> RafaelCintron: I agree with jsbell, WebGPU is in a similar mode, high-level graphics API with multiple backends
15:03:15 <anssik> ... if backends behave differently, they discuss it in the group and usually majority wins, the minority backend gets polyfilled
15:03:38 <anssik> ... emulation may regress performance, so features are detectable
15:03:51 <Chai> ntd
15:03:55 <anssik> ... feature detection is the last resort in WebGPU API
15:04:00 <anssik> q?
15:04:30 <anssik> q- Chai
15:04:36 <anssik> Present+ Zoltan_Kis
15:04:50 <anssik> q?
15:05:37 <anssik> Dwayne: XNNPACK we can polyfill, ranges questions need to survey more backends
15:06:16 <anssik> anssik: is this blocking the implementation?
15:07:35 <anssik> Ningxin_Hu: current implementation behaviour need to be fixed in either XNNPACK or Chromium validation logic
15:08:53 <anssik> RRSAgent, draft minutes
15:08:54 <RRSAgent> I have made the request to generate https://www.w3.org/2023/11/02-webmachinelearning-minutes.html anssik