WebML WG Virtual Meeting at TPAC 2021 - Day 1

26 October 2021


Anssi_Kostiainen, AramZS, Aram_Zucker-Scharff, Belem_Zhang, Binmiao, BruceDai, Chair, Chrisine_Runnegar, christine (PING co-chair), CorentinWallez, Dan_Appelquist, dom, EricMeyer, EricMwobobia, FrancesBaum, Ganesan_Ramalingam, Geun-Hyung_Kim, GeunHyungKim, jonathanBingham, JunWai, KirstenChapman, Kristen_Chapman, MattWilson, MaudStiernet, MichalKarzynski, NickDoty, NingxinHu, npdoty, PetrPenzin, PhilippeLeHégaret, plh, RachelYager, RafaelCintron, Raviraj, SamWeiler, SangwhanMoon, weiler, wolfgang, Wolfgang_Schindler, Wonsuk_Lee, WonsukLee, ZoltanKis
anssik, Dom

Meeting minutes

Welcome, intros, setting the context

<christine> +christine (PING co-chair)

Anssi: glad to see so many new people here, well in the TPAC spirit
… we have identified a number of topics that can benefit from wider perspectives
… we have invited guests to this meeting and here their perspectives
… we won't go into the detailed discussions of issues and pull requests
… hopefully that will make it easier for newcomers to discover the work of this group

Anssi: we have meetings spread over 3 days
… today, three topics: policy on adding new operations to WebNN
… versioning and web compatibility (with caveat our presenter on the topic won't be able to join)
… and finally, privacy and security

<AramZS> We are reviewing: https://github.com/webmachinelearning/meetings/issues/18

Rationale/criteria for adding new ops to the WebNN API

<dka> Intro: I'm Dan and I'm co-chair of the TAG group and work for Samsung. Happy to talk about our design principles doc and also interested in the privacy & security topic.

Anssi: a few people have voiced interest on the topic - chai, jonathan and ping from TF.js
… Michal from ONNX project
… he co-leads the operators special interest group there
… we want to discuss what criteria to use to decide adding new operations to the WebNN API
… and learn from other approaches in this space

WebNN informal process for adding new ops

Anssi: in our current workmode, the Web machine learning WG has documented a bunch of use cases, directly in the spec
… based on needs from JS machine learning frameworks

<Michal_K> I have a few slides I can share about our experience with ONNX ops.

Anssi: we have written a bunch of experimental implementations that we make available through polyfill initially
… and a x-platform implementation called WebNN-Native that validates implementability across major platforms

WebNN use cases

First-wave models

WebNN running code experiments

webnn-native cross-platform implementation

Anssi: we haven't really done good work in documenting that process, so no good reference to share with newcomers

How to select operators for inclusion into ONNX

Slideset: https://lists.w3.org/Archives/Public/www-archive/2021Nov/att-0000/W3C_Adding_new_Operators.pdf

[ Slide 1 ]

Michal_K: thanks for inviting me; together with Rama from Microsoft we co-chair the operators special group in ONNX
… wanted to share the ONNX perspectives and the experience we gained through a couple of years of evolving the spec
… and the type of problems we ran into

Proposing a new op to ONNX

[ Slide 2 ]

Michal_K: As a word of introduction, the goals of ONNX project are slightly different from WebNN:
… we want to provide a format to exchange models across different frameworks
… we want to be flexible and expressive
… while having a limited scope to limit the cost of implementation
… ONNX doesn't have a reference implementation as part of the standard - we have some in tests only
… ONNX started by inheriting its first set of operations from Caffe2 and CNTK
… there wasn't a clearly defined style guide

[ Slide 3 ]

Michal_K: The main criteria we use to add an operator to ONNX: it has to be used, preferably in popular models
… with existing support in some frameworks
… we also pay attention to how the operation is defined - the definition must be clear enough for implementors of ONNX backend to implement it
… we want to see unit tests, and we welcome reference implementations

[ Slide 4 ]

Michal_K: Some other considerations that we use: we don't want to add too many operations in ONNX
… ONNX can define functions where more complex tasks can be defined in terms of simpler primitives
… these complex operations can but don't have to be implemented by backends for optimization
… that's a useful addition to ONNX
… if an operation CAN be decomposed into primitives, we recommend defining the primitives instead (and have the complex op defined as a function)
… we prefer static attributes over dynamic input - that allows optimization on backends
… with each operation that we add, we want to have the ability to infer the shape of a model that we're dealing with
… (usually possible unless the op is fully dynamic)

[ Slide 5 ]

Michal_K: There is a tradeoff between a very strict and a more relaxed mode of operator definition
… compare the image resize operator - deceptively simple given the number of interop strategies that exist, with different set of attributes
… in the first version of ONNX was defined as a very simple operator, but ended being up quite complex in more recent versions
… on the other hand, we have a pretty relaxed definition of the random number generator
… different backends are not expected to give necessarily the same value out of a given seed
… that's because different frameworks don't necessarily have the same random number algorithms
… this is all linked to our goal with ONNX: making models portable across frameworks
… if portability is not the main concern, strictness is preferable in general

[ Slide 6 ]

Michal_K: I would recommend having a set of guidelines for common things that are part of the definition of many operators
… this includes things like broadcasting rules for things of different shape, axes (support for negative values)
… default values for stride, dilation and other common attributes
… ONNX has a growing number of input types which requires adjusting which operators support which input types
… you may want to consider how to treat input type (with group of types, or abstract types)
… When you have different operators that form families, it's good to keep them consistent
… they should share the same interface

[ Slide 7 ]

Michal_K: in ONNX, we also have operator set - that allows us to evolve the spec over time
… new operators replace old ones
… this adds complexity to the spec - to support ONNIX, you have to support 4 different versions of resize
… we now have added a requirement to add downgrade/upgrade paths for your ops

[ Slide 8 ]

Michal_K: Finally, the best addition to any operator definition is to have lots of tests - makes it easier for implementors, converters, adapters
… in ONNX, our tests aren't very well suited for code review, because they're based on binary blobs
… this is an area for improvement that you may also consider
… ONNX don't have a reference implementation, so we only get to run these tests after an implementation has been made
… and sometimes realize only at that stage that a test was wrong
… Having a reference implementation to exercise tests is also helpful

[ Slide 9 ]

Michal_K: The number of operators keep growing as the number of apps is evolving and the deep learning field is evolving too
… We need to keep track of these evolutions

Anssi: thanks, great set of advices
… seems like a set of common insights with WebNN: composability, testing
… and similar challenges with regard to the constant evolution of the field

Chai: the ONNX challenges are very similar to WebNN
… I would add that generally speaking that the purpose of WebNN is to define the backend set of operators for frameworks used on the Web
… there are a number of those
… How do we define a set of core operators that can be implemented to support this breadth of framework operators?
… ONNX interoperability is a reference

JonathanB: from Michal's slides, the criteria are very sensible - a good framework to thinking about it and a reasonable approach
… it remains subjective at that level: what counts as widely used? or popular frameworks?
… at a high level, there are a couple of very different approaches we could take for WebNN:
… - let's everything in - add anything with minimal level of usage outthere
… - another approach would to pick X JS ML frameworks, and only import operators used by these frameworks - this would be the most restrictive
… - something in between these two extremes
… Google has taken different approaches across different products
… e.g. TF has been very inclusive in accepting new operators
… that wouldn't work for the Web where these operators would need to stay forever, without clear need
… on Android, we have taken a middle path - but there is always pressure to add more operators
… it's probably useful to make some of these subjective assessments a bit tighter
… and ask people to join the group to make their case

anssi: great closing statement - there can't be a perfect decision process, there will always be subjective aspects
… and join the group to make your case

Versioning and web compatibility

Anssi: the goal of this session is how versioning fits (or not) in the Web platform
… and what techniques to enable conditional running code
… in the ML context, we can use eager execution for conditionally running code

<dka> https://www.w3.org/TR/design-principles/#feature-detect

DKA: our design principles doc describe feature detection which is relevant to this topic
… the W3C Technical Archtiecture Group (TAG) which I co-chair conducts design reviews of specifications (in W3C and others, like WHATWG, IETF, CGs)
… beside these reviews, we develop documents including this design principles document which we update constantly
… we spend about a quarter of our time updating documents like this one or the ethical principles document

Web Platform Design Principles: New features should be detectable

DKA: the design principles are meant to be broadly applicable to features on the Web platform

<dka> https://www.w3.org/2001/tag/doc/ethical-web-principles/

W3C TAG Ethical Web Principles

DKA: we also have another document which we produce: the ethical Web Principles that tries to anchor the specificities of the Web in an ethical framework and architectural considerations
… one of the key design principles (1.2) is that it should be safe to visit a Web page
… derived from an ethical principle about privacy and security
… Web users should feel safe visiting a Web page, a differentiator of the Web compared to other platforms
… On feature detection in particular, we're asking people to make new features detectable based on how people build Web applications
… trying to build a platform in such a way that when a Web app requests a feature or makes use of a feature, that it can degrade gracefully if that feature isn't available

Anssi: there is a tradeoff with fingerprintability if feature detection can identify specific devices

DKA: we always encourage to minimize fingerprintability when designing their features
… that's also covered in the design principles
… this includes paying attention to the "private mode" detection
… or don't reveal the existence of assistive technology
… there are trade-offs to be made

<npdoty> feature detection may not always increase entropy of fingerprinting surface. if features are generally not specific to users or hardware or configuration, they may just be revealing which browser is in use and what it supports

Anssi: on the Web platform, we don't have versioning (à la Web APIs 1.0, 2.0), but instead use feature detection to allow adaptability
… in other non-Web projects related to ML, there are explicit enforced policies around versioning
… e.g. ONNX versioning

<npdoty> and active feature detection is also detectable, which is a less severe type of fingerprintability

Sangwhan: I believe the TAG raised concerns about the evolution of the API going forward, in particular with regard to hardware binding
… the spec seems to be bound to current hardware capabilities with the idea of making it evolve over time
… but that doesn't quite fit with the Web model
… once the API has shipped, it will be very difficult to change the API based on early assumptions we've made
… Some APIs could be made more extensible by taking objects as parameters to function calls

Anssi: would the TAG prefer to come to you again with an update? how should we follow up on this concern?

Sangwhan: a new review request would be helpful

<sangwhan> One thing to note is that we have made attempts to version APIs in the past in the platform and haven't been particularly successful at this.

chai: sangwhan, if you have concrete suggestions, we would love to see those - we believe we have addressed previously raised concerns on hardware, and would be happy to continue the conversation

Privacy and security discussion

Anssi: we have Christine, co-chair of Privacy Interest Group, Nick (editor of fingerprinting guidance)
… Corentin, one of the chairs of the WebGPU WG

<Michal_K> Operator versioning in ONNX is very explicit and baked in from the start. It adds complexity, but allows for API changes with backward compatibility of older models.

Anssi: the idea here is to look at the broader space of privacy discussions across similar features, specifically WebGPU
… WebGPU has done a fantastic job in documenting their privacy and security considerations
… I would to reuse these learnings to the extent possible
… and would like to discuss Nick's fingerprinting guidance and look for best practices that would apply well to WebNN
… and hear from Jonathan on the model loader API security
… Christine, do you want to give a high level of overview of the PINg work mode and how we can effectively work with you?
… We've already received valuable feedback from PING

Christine: 1st of all, thanks for coming to us early - it's very difficult to retrofit privacy
… we have formal tooling that helps us keep track - we can take offline how to request privacy reviews
… you already know of the self-review questionnaire
… and Nick's excellent document

Mitigating Browser Fingerprinting in Web Specifications

Anssi: the fingerprinting guidance documents 10 best practices - Nick, any high level in the context of this group?

Nick: in general, we're not looking at eliminating all the things JS can learn about a device, but we can make it detectable, or less accessible, less permanent, less available x-origin
… that might be significant for the ML context
… e.g. fingerprinting based on feature detection - it could be made detectable

Best Practices Summary

Nick: or not making it available to all Web sites

Anssi: we'll evaluate our group's work with this document

<npdoty> and here is the section on evaluating severity of fingerprinting surface: https://www.w3.org/TR/fingerprinting-guidance/#identifying-fingerprinting-surface-and-evaluating-severity

Anssi: is it integrated in the PING questionnaire?

Nick: it is referred from the questionnaire; once we start looking into fingerprinting, we would start evaluating these best practices

weiler: in terms of what PING is asking for: we're not asking for verbatim answers to the questionnaire, instead we want a narrative based on the answers and mitigations
… we want to know what analysis has already been done

<npdoty> I think we should also consider privacy considerations that are more specific to machine learning in general. what are the capabilities of inference or manipulation that are enabled by adding ML into the browser?

<Zakim> sangwhan, you wanted to bring up hardware capabilities/acceleration (esp. with software support, e.g. cuDNN for desktop) as an extra entropy fingerprinting surface

WebGPU Security and Privacy Considerations

<Michal_K> sangwham We keep a changelog for operator changes and each change creates a new operator version. https://github.com/onnx/onnx/blob/master/docs/Changelog.md

Anssi: Corentin, how did the WebGPU group develop its security and privacy considerations (to great results)?

WebGPU Security considerations

Corentin: for WebGPU, we had a good starting point for the security side of things due to the experience with WebGL

WebGPU Privacy considerations

<npdoty> with regrets, I have another group discussing privacy review specifically starting now. I'll definitely read the webgpu security write-up later

<plh> [it would be good for PING to confirm that webgpu is a good example to follow]

Corentin: fingerprinting is a harder issue, because you can measure hardware that has different behaviour
… you may want to expose some information about hardware, ML workloads, exposing this information is a privacy information
… these issues have been hard to solve in WebGPU too
… at the choice of the browser to have their own policies, for example, exposing the name of the adapter

<sangwhan> "GPU bug workarounds"

Corentin: use case is to switch content execution based on adapter name

<sangwhan> Living proof: https://chromium.googlesource.com/chromium/src/+/HEAD/gpu/config/gpu_driver_bug_list.json

Corentin: from the Chromium experience it is important to be able to workaround driver bugs

<RafaelCintron> See https://github.com/gpuweb/gpuweb/issues/2191 for current discussion about exposing adapter naming.

sangwhan: there's an extra fingerprinting surface if you enable neural network acceleration on mobile with co-processor, separate fingerprinting surface
… e.g. cuDNN has different perf characteristics depending on which hardware it is run, this is extra fingerprinting surface

<sangwhan> So you have "GPU name" x "GPU dot product acceleration library version" x ("Coprocessor" x "Coprocessor software version") (parens for when you have a neural coprocessors that is not a GPU)

<sangwhan> (and you can throw in GPU driver version I guess too)

RafaelCintron: the WebGPU is discussing how much of names to expose, on Windows quite a lot information is exposed

<sangwhan> There is a tradeoff, and something that warrants investigation and mention in the spec.

<RafaelCintron> http://browserleaks.com/webgl

<Michal_K> Thank you for inviting me.

<ningxin_hu> we have one topic on the agenda: Discuss Model Loader API security

Minutes manually created (not a transcript), formatted by scribe.perl version 149 (Tue Oct 12 21:11:27 2021 UTC).