Meeting minutes
welcome to the new participants joining us since our last meeting:
… Jay Mok from PayPal
… Mikhail Klimenko from NVIDIA
… Thomas Steiner and Rezvan Mahdavi Hezaveh from Google
… welcome all!
New incubation: Cross-Origin Storage
Repository: wicg/cross-origin-storage
Thomas: content-addressable browser cache, shares files across origins by hash
… currently, caches are partitioned in all browsers
… huge potential for sharing commonly used resources, e.g. AI models, Wasm files, Web fonts, Game engines, libraries
… the proposal is called Cross-Origin Storage (COS), Blob-based, resource hashes, extends FileSystemFileHandle, visibility limited to certain origins per allow-list
[proposed API code sample for reading, imperative]
[proposed API code sample for writing, imperative]
Thomas: declarative approach using <link> and <script> extensions
… declarative approach for CSS also
… integration with fetch() with extensions
Thomas: demo available for testing
… Chromium build available with COS API implemented including DevTools support:
Thomas: HuggingFace, MLC AI WebLLM, wllama, LiteRT-ML JS, Flutter, Emscripten etc. as initial customers
… Chrome team would like to make this a proper API
[drum roll, COS demo unveiling]
Thomas: patched webnn-samples repo for this COS WebNN demo
… cache miss/hit shown in the Chrome UI
… this is true cross-origin shared resource, the AI model used for the style transfer sample
https://
https://
tomayac/
Thomas: DevTools support allows to inspect cached files, AI models in this case
Thomas: this proposal is general purpose so it will likely go to more general purpose group for standardization
… WebNN is one important customer for this API
MarkusT: very cool, does the file have to match the hash, do I have to compute hash first?
Thomas: HF publishes hashes for the models they host, there's a public hash list on their site
MarkusT: if you store, from where does the hash come from?
Thomas: hash comes from Hugging Face Hub
MarkusT: would be nice if the UI would provide a mechanism for the user to clean up cache as appropriate
Thomas: in Chrome you can remove the site's data, does not differentiate on a file basis
… a regular user does not need to care, the implementation handles that for the user
Rafael: first question was, request blob and you'll get a hash, how does the implementation know what to look for given the hash?
Thomas: there needs to be a URL that is mapped to the hash initially
… URL is to deliver the resource, then forgotten
Rafael: contents of the file does not matter?
Thomas: if the file content changes, the hash changes
Rafael: internally the browser keeps a checksum?
Thomas: right, this ensures integrity
Rafael: about cross-origin, when you list URLs, can I use wildcards (*)?
Thomas: our design philosophy is "people will do stupid things"
… core idea is you need to use public hashes
https://
Web Neural Network API
Repository: webmachinelearning/webnn
WebGPU interop
Anssi: PR #942
<gb> PR 945 Add dynamic shape explainer (by miaobin)
Anssi: in our June meeting we resolved to port over the necessary features to the WebNN API spec from the MLTensor explainer
https://
Anssi: Bryan has shared with the group PR #942 for the proposed spec updates, ready for review
<gb> PR 942 Add initial WebNN-WebGPU Interop specification (by bbernhar) [Agenda+]
Anssi: the PR adds the following features to the WebNN API spec:
Anssi: createExportableTensor() creates an MLTensor associated with this MLContext which may be exported to the given GPUDevice using exportToGPU()"
Anssi: exportToGPU() exports an exportable MLTensor to its associated GPUDevice as a GPUBuffer, and rents it to that GPUDevice until the returned GPUBuffer is destroyed
… these spec changes follow the explainer with minor naming adjustment (createTensor -> createExportableTensor):
https://
Anssi: Bryan, please walk us through the PR and any open questions you'd like the group to discuss
… after that, we'll discuss the feedback shared via GH from MikeW, Ningxin, and any other comments from the group
Bryan: this is an initial spec PR we've used in our implementation, there are likely some edge cases, open questions on multi-adapter scenarios
… in terms of basic handoff API, zero-copy enabling, it is more or less there
… to support more formats is future work
… this is the first draft for basic interop, the spec PR is self-explanatory, uses strictly WebGPU types
… interop tensor looks a lot like normal tensor and you can use it in WebGPU directly
… not a lot of additional API exposed to web developers
Bryan: feedback on the PR directly welcome
Rafael: I had a couple of questions about the PR, sometimes middleware developers don't fully know how their middleware is used, e.g. ThreeJS in WebGPU world
… make all tensors in their world exportable, can have performance implications
… should we give advice to web developers about this
… what if the chosen adapter does not match?
Bryan: not in developers' hands, use interop when you need it
… we can warn developers of performance cliffs
Rafael: on the surface it'd be cool if you could say high power, and if you could pass GPU device and have it work, have the browser copy for you
… performance cliff possible if browser picks a wrong adapter that needs copies
Bryan: high performance is overloaded, an open issue to spec for the source and destination devices to match
… a proposal how device selection is influenced, orthogonal, current spec v1 assumes this just works
Ningxin: there are some webgpu-interop open issues, could the spec PR embed those issues?
… about narrow scoped usage, when exporting a tensor, you can still read and write to it, should we restrict that?
Bryan: for read/write, we could, but would require developers to go to WebGPU for that
Ningxin: are there performance implications to make a tensor read/write, use a different memory region?
Bryan: if you specify a GPU device and we make it not readable and/or writable, there are performance implications
… creating MLContext, how it can influence the destination buffer
Rafael: I will review the PR
Reilly: I'm happy to see this spec'd, agree with earlier discussion on performance cliff mitigation, we can probably make rules that address this issue
Accumulation precision
Anssi: issue #948
<gb> Issue 948 Accumulation precision is unspecified for matmul/gemm and reduction operations (by mtavenrath) [Agenda+]
Anssi: MarkusT noted the spec does not say anything about what precision intermediate accumulations should use
MarkusT: matmul and gemm the most significantly affected, but also other ops that sum values internally are affected e.g. reduceSum, conv2d, gru, lstm etc.
… what is the minimum precision the group would support?
… is it exactly same as that of inputs?
… or should we add an extra attribute that specified the accumulator type
MarkusT: A) explicit upcast
… cast inputs to float32, perform the operation, and cast back
MarkusT: B) explicit accumulator type per operation
… add an optional accumulatorType field to operations that have internal accumulation
MarkusT: which approach is preferred?
Dwayne: I like what I see, will provide more input
Ningxin: I think this is important issue, we see this in e.g. diffusion models
… if you spec proposal B, is there composition or emulation path available?
MarkusT: we don't need any emulation, this should be minimum accumulation type
Ningxin: makes sense, if e.g. polyfill sees this option, what does it do, should it go through the cast pass to emulate this?
MarkusT: not sure about that case
Ningxin: I will comment on the issue
MarkusT: considering WebGPU, could we have WebGPU implementation working on an exact accumulator type
… how to ensure people can test the model runs in such a scenario
… we need to define the WebGPU implementation uses minimum accumulator type anywhere