Meeting minutes
Anssi: welcome to Rattan Priya Bhasin from Microsoft joining us as a new participant since our last meeting
Web Neural Network API
Repository: webmachinelearning/webnn
WebNN-WebGPU zero-copy device selection
Anssi: PR #942
<gb> PR 942 Add initial WebNN-WebGPU Interop specification (by bbernhar) [Agenda+]
Anssi: we discussed this spec PR last time and I called for final review with an expectation we'll merge the PR after any remaining comments are addressed
… all comments are now marked as resolved
… thank you Bryan for the PR updates and everyone for reviewing
… the PR updates were the following during the review round:
… per MikeW's suggestion, acknowledged read-only storage use case for read-only WebGPU ops, to be added later, MikeW to open a new issue for this?
… per Ningxin's suggestion, added explicit cross-timeline ordering, a note to describe what happens when a GPUDevice is lost (stays exported), and spec'd explicit failure path when export fails, by throwing an exception
… Dwayne had comments and questions regarding naming and we landed on consensus that the current naming is appropriate given WebGPU nomenclature, we also agreed usage flags would be useful, to be discussed in a new issue, Dwayne to open?
Bryan: appreciate everyone's feedback and help on this PR
… with that, we can resolve to merge this PR now
<ningxin> +1
<Mike_Wyrzykowski> +1
RESOLUTION: Add initial WebNN-WebGPU interop support per PR #942 to WebNN API spec.
Anssi: next I'd like us to discuss v2 features
Anssi: zero-copy device selection for WebNN-WebGPU interop is issue #941
<gb> Issue 941 WebNN/WebGPU zero-copy device selection (by bbernhar) [webgpu interop] [Agenda+]
Bryan: we discussed this issue with Reilly, while prototyping
… API was not clear on how interop works in zero-copy situation, I proposed two options
… Reilly prefers the createContext({ interopDevice: gpuDevice }) option over createContext(GPUDevice) overload
Reilly: the idea is that depending on architecture GPU devices may interop as non-GPU devices, we should separately talk about interop device and power and performance
… interop device and power and performance are related, but we don't have to make GPU specific createContext
Ningxin: questions related to the PR, new createExportableTensor, that interface also takes GPUDevice, I'd like to understand how these two GPUDevices are related?
Bryan: it's the source and destination device, when exporting a tensor, that's the destination, source device could be NPU, or the same GPU in which case could do zero-copy
… could be a different device and behind the scenes would do zero-copy
… we spec'd this like this because the tensor is created differently depending on the device for compatibility's sake
MikeW: I will take a look at the proposal
Anssi: next v2 feature proposal is read-only storage use case for read-only WebGPU ops
MikeW: I created and issue for this, real-only is possible today already, but perhaps you want to use exportedMLTensor as an indirect buffer
<Mike_Wyrzykowski> webmachinelearning/
<gb> Issue 954 MLTensor export to GPUBuffer with custom usages? (by mwyrzykowski)
Anssi: another v2 feature proposal is usage flags for createExportableTensor from Dwayne
… Dwayne, do you want to introduce the usage flags idea?
Dwayne: I don't have a list of flags that would be applicable right now, will create an issue for this
https://
<gb> PR 942 Add initial WebNN-WebGPU Interop specification (by bbernhar) [Agenda+]
Bryan: flags feel related to MikeW's proposal in issue #954, we need to explicitly pass WebGPU flags and see how they correspond, good symmetry between the two
<gb> Issue 954 MLTensor export to GPUBuffer with custom usages? (by mwyrzykowski) [webgpu interop]
Accuracy for reductions
Anssi: issue #947
<gb> Issue 947 reduceLogSumExp reference decomposition contradicts conformance test; accuracy requirements missing (by mtavenrath) [Agenda+]
Anssi: MarkusT notes accuracy for reductions is generally underspecified
… I'd like the group to review the proposed fix for reduceLogSumExp
… and second, agree on general guidelines for other reductions
Mikhail: there's diversion between spec and what happens on real HW, we can fix this either by fixing WPT tests so they represent reality, not really looking into this
… second option is to modify reference decomposition in the spec, prototyped in RustNN and it works fine, propose we go with option (b)
Anssi: proposed fix to "replace the reference decomposition with the numerically stable formulation"
Dwayne: just adding this improved decomposition is used in frameworks
Reilly: the reference decompositions in the spec are not reference implementations to be mandated
… very often the spec does not specify exact algorithms
Anssi: what do we want to do with other reductions?
Dwayne: other reductions may not have more accurate representations
Anssi: no need to adjust other reduction decompositions then
<DwayneR> +1
<mklimenko> +1
RESOLUTION: Update the spec to use numerically more stable reduceLogSumExp decomposition. (issue #947)
<reillyg> Yes, he's from Google's LiteRT team.
Subgraphs
Anssi: issue #907
<gb> Issue 907 Composite operators / subgraphs (by fdwr) [opset] [Agenda+]
Anssi: we've received new feedback for this issue since we touched this last time, that was also a while ago
<DwayneR> Changming used to be on ONNX Runtime.
Anssi: to recap, this issue is exploring how to allow WebNN express "big ops" (attention, gelu, RMSNorm) as composite operators aka subgraphs
… the recent feedback on GH from Changming seems to argue the proposed design wouldn't work for LLMs
… MarkusT commented that he disagrees with that assessment and gives an example of GroupQueryAttention experiment by Ningxin that demonstrated clear performance benefits
… MarkusT suggests a standardized subgraph construction API (option C) would provide the best path forward to avoid fragile pattern matching approaches, avoid scope creep and minimize maintenance burden, and enable maximum performance
Anssi: do we need more performance data on how the proposed subgraph API would perform with LLMs?
Ningxin: I want to share two aspects, we should look at GenAI and LLM trends, we delivered good GenAI model support with Transformer ops
… now we see some challenges with new LLMs, e.g. attention for flow control
… llama.cpp has a few ops and gets good performance, Joshua published well-optimized WebGPU kernels recently, should look into these efforts
… did some experiments, but received some pushback that pattern matching is complex, should consider some ahead-of-time optimization to reduce complexity imposed on framework authors
Reilly: I generally agree with MarkusT's comment, Changming's observation is also correct, however custom kernels cannot be done inside WebNN
… what we can express is grouped graph nodes and give them to the underlying implementations
MikeW: I'll look at this issue
Dwayne: experiments that Ningxin did helped, not sure we need more of that, would like to see the shape of the possible new subgraph API
<mklimenko> +1
<DwayneR> +1
Ningxin: I think the subgraph API drafting is good approach, also example usages would be good complement, we did example decomposition experiment, could provide some decomposition in current form using JS pseudocode, to share one GQA example, so we can exercise the new graph API
<ningxin> +1
<Mike_Wyrzykowski> +1
RESOLUTION: Draft a subgraph API proposal and JS usage example code for group review and feedback. (issue #907)
Full-sequence output validation
Anssi: issue #949
<gb> Issue 949 Validate LSTM/GRU full-sequence output regardless of the `returnSequence` flag (by shiyi9801) [security-tracker] [Agenda+]
Anssi: the group puts high priority in resolving any security related issues, and the last three are particularly relevant in this context
… I labeled these issues with [security-tracker] to keep track of security related issues we've discussed in the WG and their mitigations
… many mitigations happen at the implementation level, but where applicable, we mitigate at the spec level and/or provide guidance for implementers to allow all implementations benefit from the latest security best practices
… Shiyi reports certain backends allocate the full-sequence output tensor internally regardless of the returnSequence flag, but do not validate the size
… this might cause an out-of-bounds write
… proposed fix is to always validate the full-sequence output descriptor so that oversized shapes are rejected before reaching the backend
… does the group have any concerns for adding this fix to the spec?
… Chromium implementation experience validates this approach in practice
… next step would be to create a spec PR with the proposed fix to align with that
… and I believe we also want to update WPT tests to validate the fix
Reilly: I'm curious from other implementers, to provide more context, this is a case where implementing a security mitigation changes so that web developers can observe the mitigation is in place
<DwayneR> +1
<ningxin> +1
<reillyg> +!
RESOLUTION: Validate LSTM/GRU full-sequence output descriptor to mitigate OOB writes, prepare a spec PR for review. (issue #949)
Formulaic mitigation for integer overflows
Anssi: issue #928
<gb> Issue 928 Consider specify lower limit for conv2d/pool2d kernel sizes, dilations, strides (by philloooo) [security-tracker] [Agenda+]
Anssi: for this issue, we resolved to extend opSupportLimits() to be able to express limits on the size of inputs
… recently we received implementation experience from Chromium for mitigation that rejects conv2d/pool2d/convTranspose2d strides, dilations and filters that exceed the usable extent:
https://
Anssi: we also landed corresponding WPT updates:
Anssi: what remains is to update the spec to reflect these limits and ensure all implementations adhere to them
Reilly: this one is interesting, a case where an op is well defined, but with a very large value we can wrap around
… the proposal is to make passing these large values invalid, so the developer needs to pass smaller values
… this allow WebNN implementation to have more control over values to be passed to underlying framework
… WebNN could do some of this conversion by itself
Dwayne: do we have clear limits?
… instead of having hard coded limits prefer to have relational limits
Reilly: I agree with Dwayne, proposal we've made have run into this issue
Ningxin: rather than propose the limit, can we propose algorithmic steps to compute the tensor/buffer size, would this be an alternative way to address this?
Dwayne: that would be more robust and would address my concern
Ningxin: I will provide my suggestion in the issue