13:52:15 RRSAgent has joined #webmachinelearning 13:52:15 logging to https://www.w3.org/2021/10/26-webmachinelearning-irc 13:52:17 RRSAgent, make logs Public 13:52:19 please title this meeting ("meeting: ..."), anssik 13:52:20 Meeting: WebML WG Virtual Meeting at TPAC 2021 - Day 1 13:52:24 Chair: Anssi 13:52:29 Agenda: https://github.com/webmachinelearning/meetings/issues/18 13:57:11 AramZS has joined #webmachinelearning 13:59:04 wolfgang has joined #webmachinelearning 13:59:13 Present+ Anssi_Kostiainen 13:59:16 Present+ 13:59:17 scribe+ 13:59:26 Scribe: Dom 13:59:30 scribeNick: dom 13:59:56 scribe+ anssik 14:00:13 present+ wolfgang 14:00:13 Present+ Ganesan_Ramalingam 14:00:16 ningxin_hu has joined #webmachinelearning 14:00:30 dka has joined #WebMachineLearning 14:00:34 Present+ Belem_Zhang 14:00:42 present+ Dan_Appelquist 14:00:43 Present+ Wolfgang_Schindler 14:00:50 plh has joined #webmachinelearning 14:00:53 Michal_K has joined #webmachinelearning 14:00:54 Present+ Wonsuk_Lee 14:00:56 present+ 14:01:09 present+ 14:01:30 Present+ Aram_Zucker-Scharff 14:01:54 Present+ Chrisine_Runnegar 14:01:57 christine has joined #webmachinelearning 14:01:59 Present+ EricMeyer 14:02:01 wonsuk has joined #webmachinelearning 14:02:02 Rama has joined #webmachinelearning 14:02:03 Present+ Raviraj 14:02:05 belem_zhang has joined #webmachinelearning 14:02:09 Present+ Chair 14:02:15 Present+ Binmiao 14:02:18 Topic: Welcome, intros, setting the context 14:02:22 Present+ BruceDai 14:02:38 Present+ jonathanBingham 14:02:40 +christine (PING co-chair) 14:02:55 Present+ christine (PING co-chair) 14:02:57 Anssi: glad to see so many new people here, well in the TPAC spirit 14:03:01 Chai has joined #webmachinelearning 14:03:09 ... we have identified a number of topics that can benefit from wider perspectives 14:03:24 ... we have invited guests to this meeting and here their perspectives 14:03:36 ... we won't go into the detailed discussions of issues and pull requests 14:03:49 ... hopefully that will make it easier for newcomers to discover the work of this group 14:03:54 Present+ JunWai 14:03:58 Present+ KirstenChapman 14:04:15 npdoty has joined #webmachinelearning 14:04:21 Present+ MichalKarzynski 14:04:24 Present+ NickDoty 14:04:27 Present+ RachelYager 14:04:30 present+ 14:04:32 Present+ CorentinWallez 14:04:41 Present+ NingxinHu 14:04:45 Present+ RafaelCintro 14:04:50 s/tro/tron 14:04:56 Present+ SamWeiler 14:05:01 Present+ PhilippeLeHégaret 14:05:08 Present+ WonsukLee 14:05:17 Anssi: we have meetings spread over 3 days 14:05:34 ... today, three topics: policy on adding new operations to WebNN 14:05:49 ... versioning and web compatibility (with caveat our presenter on the topic won't be able to join) 14:06:00 ... and finally, privacy and security 14:06:06 q+ 14:06:09 ack anssik 14:06:09 Present+ MaudStiernet 14:06:12 We are reviewing: https://github.com/webmachinelearning/meetings/issues/18 14:06:25 weiler has joined #webmachinelearning 14:06:30 RafaelCintron has joined #webmachinelearning 14:06:41 Present+ GeunHyungKim 14:07:01 Topic: Rationale/criteria for adding new ops to the WebNN API 14:07:20 Geun-Hyung_Kim has joined #webmachinelearning 14:07:25 present+ 14:07:34 Intro: I'm Dan and I'm co-chair of the TAG group and work for Samsung. Happy to talk about our design principles doc and also interested in the privacy & security topic. 14:07:44 Anssi: a few people have voiced interest on the topic - chai, jonathan and ping from TF.js 14:07:46 ... Michal from ONNX project 14:08:07 ... he co-leads the operations special interest group there 14:08:32 ... we want to discuss what criteria to use to decide adding new operations to the WebNN API 14:08:45 ... and learn from other approaches in this space 14:08:50 Subtopic: WebNN informal process for adding new ops 14:09:18 Anssi: in our current workmode, the Web machine learning WG has documented a bunch of use cases, directly in the spec 14:09:28 ... based on needs from JS machine learning frameworks 14:09:34 I have a few slides I can share about our experience with ONNX ops. 14:09:42 ... we have written a bunch of experimental implementations that we make available through polyfill initially 14:09:56 ... and a x-platform implementation called WebNN-Native that validates implementability across major platforms 14:10:00 -> https://www.w3.org/TR/webnn/#usecases WebNN use cases 14:10:06 -> https://github.com/webmachinelearning/webnn/blob/main/op_compatibility/first_wave_models.md First-wave models 14:10:12 -> https://webmachinelearning.github.io/webnn-samples-intro/ WebNN running code experiments 14:10:16 -> https://github.com/webmachinelearning/webnn-native webnn-native cross-platform implementation 14:10:24 Anssi: we haven't really done good work in documenting that process, so no good reference to share with newcomers 14:10:37 s/operations special/operators special/ 14:10:41 Subtopic: How to select operators for inclusion into ONNX 14:11:29 Michal_K: thanks for inviting me; together with Rama from Microsoft we co-chair the operators special group in ONNX 14:11:46 ... wanted to share the ONNX perspectives and the experience we gained through a couple of years of evolving the spec 14:11:51 ... and the type of problems we ran into 14:11:56 -> https://github.com/onnx/onnx/blob/master/docs/AddNewOp.md Proposing a new op to ONNX 14:12:04 ... As a word of introduction, the goals of ONNX project are slightly different from WebNN: 14:12:15 ... we want to provide a format to exchange models across different frameworks 14:12:22 ... we want to be flexible and expressive 14:12:31 ... while having a limited scope to limit the cost of implementation 14:12:46 ... ONNX doesn't have a reference implementation as part of the standard - we have some in tests only 14:13:07 ... ONNX started by inheriting its first set of operations from Caffe2 and CNTK 14:13:16 ... there wasn't a clearly defined style guide 14:13:37 ... The main criteria we use to add an operator to ONNX: it has to be used, preferably in popular models 14:13:45 ... with existing support in some frameworks 14:14:07 ... we also pay attention to how the operation is defined - the definition must be clear enough for implementors of ONNX backend to implement it 14:14:19 ... we want to see unit tests, and we welcome reference implementations 14:14:40 ... Some other considerations that we use: we don't want to add too many operations in ONNX 14:14:59 ... ONNX can define functions where more complex tasks can be defined in terms of simpler primitives 14:15:17 ... these complex operations can but don't have to be implemented by backends for optimization 14:15:25 ... that's a useful addition to ONNX 14:15:44 ... if an operation CAN be decomposed into primitives, we recommend defining the primitives instead (and have the complex op defined as a function) 14:15:58 ... we prefer static attributes over dynamic input - that allows optimization on backends 14:16:24 ... with each operation that we add, we want to have the ability to infer the shape of a model that we're dealing with 14:16:42 ... (usually possible unless the op is fully dynamic) 14:16:56 ... There is a tradeoff between a very strict and a more relaxed mode of operator definition 14:17:20 ... compare the image resize operator - deceptively simple given the number of interop strategies that exist, with different set of attributes 14:17:45 ... in the first version of ONNX was defined as a very simple operator, but ended being up quite complex in more recent versions 14:17:59 ... on the other hand, we have a pretty relaxed definition of the random number generator 14:18:12 ... different backends are not expected to give necessarily the same value out of a given seed 14:18:31 ... that's because different frameworks don't necessarily have the same random number algorithms 14:18:38 q? 14:19:00 ... this is all linked to our goal with ONNX: making models portable across frameworks 14:19:15 ... if portability is not the main concern, strictness is preferable in general 14:19:34 ... I would recommand having a set of guidelines for common things that are part of the definition of many operators 14:20:00 ... this includes things like broadcasting rules for things of different shape, axes (support for negative values) 14:20:07 s/command/commend 14:20:16 ... default values for stride, dilation and other common attributes 14:20:38 ... ONNX has a growing number of input types which requires adjusting which operators support which input types 14:20:58 ... you may want to consider how to treat input type (with group of types, or abstract types) 14:21:11 ... When you have different operators that form families, it's good to keep them consistent 14:21:16 ... they should share the same interface 14:21:29 ... in ONNX, we also have operator set - that allows us to evolve the spec over time 14:21:51 ... new operators replace old ones 14:22:10 ... this adds complexity to the spec - to support ONNIX, you have to support 4 different versions of resize 14:22:25 RRSAgent, draft minutes 14:22:25 I have made the request to generate https://www.w3.org/2021/10/26-webmachinelearning-minutes.html anssik 14:22:27 ... we now have added a requirement to add downgrade/upgrade paths for your ops 14:22:57 ... Finally, the best addition to any operator definition is to have lots of tests - makes it easier for implementors, converters, adapters 14:23:15 ... in ONNX, our tests aren't very well suited for code review, because they're based on binary blobs 14:23:29 ... this is an area for improvement that you may also consider 14:23:46 ... ONNX don't have a reference implementation, so we only get to run these tests after an implementation has been made 14:23:56 ... and sometimes realize only at that stage that a test was wrong 14:24:07 ... Having a reference implmeentation to exercise tests is also helpful 14:24:15 s/plmee/pleme/ 14:24:41 ... The number of operators keep growing as the number of apps is evolving and the deep learning field is evolving too 14:24:50 q? 14:24:53 ... We need to keep track of these evolutions 14:25:13 Anssi: thanks, great set of advices 14:25:25 ... seems like a set of common insights with WebNN: composability, testing 14:25:38 ... and similar challenges with regard to the constant evolution of the field 14:26:03 Chai: the ONNX challenges are very similar to WebNN 14:26:22 ... I would add that generally speaking that the purpose of WebNN is to define the backend set of operators for frameworks used on the Web 14:26:26 ... there are a number of those 14:26:48 ... How do we define a set of core operators that can be implemented to support this breadth of framework operators? 14:27:08 ... ONNX interoperability is a reference 14:29:09 JonathanB: from Michal's slides, the criteria are very sensible - a good framework to thinking about it and a reasonable approach 14:29:25 ... it remains subjective at that level: what counts as widely used? or popular frameworks? 14:29:45 ... at a high level, there are a couple of very different approaches we could take for WebNN: 14:30:01 ... - let's everything in - add anything with minimal level of usage outthere 14:30:31 ... - another approach would to pick X JS ML frameworks, and only import operators used by these frameworks - this would be the most restrictive 14:30:37 ... - something in between these two extremes 14:30:48 ... Google has taken different approaches across different products 14:30:57 ... e.g. TF has been very inclusive in accepting new operators 14:31:12 ... that wouldn't work for the Web where these operators would need to stay forever, without clear need 14:31:31 ... on Android, we have taken a middle path - but there is always pressure to add more operators 14:32:00 ... it's probably useful to make some of these subjective assessments a bit tighter 14:32:06 ... and ask people to join the group to make their case 14:32:21 q? 14:32:28 anssi: great closing statement - there can't be a perfect decision process, there will always be subjective aspects 14:32:38 ... and join the group to make your case 14:32:48 Topic: Versioning and web compatibility 14:34:01 Anssi: the goal of this session is how versioning fits (or not) in the Web platform 14:34:12 ... and what techniques to enable conditional running code 14:34:34 ... in the ML context, we can use eager execution for conditionally running code 14:34:47 https://www.w3.org/TR/design-principles/#feature-detect 14:35:05 DKA: our design principles doc describe feature detection which is relevant to this topic 14:35:31 ... the W3C Technical Archtiecture Group (TAG) which I co-chair conducts design reviews of specifications (in W3C and others, like WHATWG, IETF, CGs) 14:35:50 ... beside these reviews, we develop documents including this design principles document which we update constantly 14:36:07 ... we spend about a quarter of our time updating documents like this one or the ethical principles document 14:36:22 -> https://www.w3.org/TR/design-principles/#feature-detect Web Platform Design Principles: New features should be detectable 14:36:28 ... the design principles are meant to be broadly applicable to features on the Web platform 14:36:33 https://www.w3.org/2001/tag/doc/ethical-web-principles/ 14:36:48 -> https://www.w3.org/2001/tag/doc/ethical-web-principles/ W3C TAG Ethical Web Principles 14:37:18 ... we also have another document which we produce: the ethical Web Principles that tries to anchor the specificities of the Web in an ethical framework and architectural considerations 14:37:34 ... one of the key design principles (1.2) is that it should be safe to visit a Web page 14:37:49 ... derived from an ethical principle about privacy and security 14:38:11 ... Web users should feel safe visiting a Web page, a differentiator of the Web compared to other platforms 14:38:42 ... On feature detection in particular, we're asking people to make new features detectable based on how people build Web applications 14:38:43 q? 14:39:00 RRSAgent, draft minutes 14:39:00 I have made the request to generate https://www.w3.org/2021/10/26-webmachinelearning-minutes.html anssik 14:39:06 ... trying to build a platform in such a way that when a Web app requests a feature or makes use of afeature, that it can degrade gracefully if that feature isn't available 14:39:09 s/afe/a fe/ 14:39:34 Present+ ZoltanKis 14:39:47 Present+ SangwhanMoon 14:39:59 Present+ EricMwobobia 14:40:07 Present+ FrancesBaum 14:40:16 Present+ MattWilson 14:41:05 presnt+ 14:41:08 present+ 14:41:16 s/presnt+// 14:42:10 Anssi: there is a tradeoff with fingerprintability if feature detection can identify specific devices 14:42:26 DKA: we always encourage to minimize fingerprintability when designing their features 14:42:35 ... that's also covered in the design principles 14:42:50 ... this includes paying attention to the "private mode" detection 14:43:11 ... or don't reveal the existence of assistive technology 14:43:33 ... there are trade-offs to be made 14:44:03 feature detection may not always increase entropy of fingerprinting surface. if features are generally not specific to users or hardware or configuration, they may just be revealing which browser is in use and what it supports 14:44:04 Anssi: on the Web platform, we don't have versioning (à la Web APIs 1.0, 2.0), but instead use feature detection to allow adaptability 14:44:21 q? 14:44:23 ... in other non-Web projects related to ML, there are explicit enforced policies around versioning 14:44:24 q+ 14:44:28 ... e.g. ONNX versioning 14:44:30 and active feature detection is also detectable, which is a less severe type of fingerprintability 14:44:36 q? 14:44:40 ack sangwhan 14:45:10 Sangwhan: I believe the TAG raised concerns about the evolution of the API going forward, in particular with regard to hardware binding 14:45:32 ... the spec seems to be bound to current hardware capabilities with the idea of making it evolve over time 14:45:42 ... but that doesn't quite fit with the Web model 14:46:00 ... once the API has shipped, it will be very difficult to change the API based on early assumptions we've made 14:46:29 ... Some APIs could be made more extensible by taking objects as parameters to function calls 14:47:22 Anssi: would the TAG prefer to come to you again with an update? how should we follow up on this concern? 14:47:30 q+ 14:47:33 Sangwhan: a new review request would be helpful 14:47:52 ack chai 14:48:01 One thing to note is that we have made attempts to version APIs in the past in the platform and haven't been particularly successful at this. 14:48:36 q? 14:48:44 chai: sangwhan, if you have concrete suggestions, we would love to see those - we believe we have addressed previously raised concerns on hardware, and would be happy to continue the conversation 14:48:45 Topic: Privacy and security discussion 14:49:21 Anssi: we have Christine, co-chair of Privacy Interest Group, Nick (editor of fingerprinting guidance) 14:49:35 ... Corentin, one of the chairs of the WebGPU WG 14:49:46 q+ to bring up hardware capabilities/acceleration (esp. with software support, e.g. cuDNN for desktop) as an extra entropy fingerprinting surface 14:49:47 Operator versioning in ONNX is very explicit and baked in from the start. It adds complexity, but allows for API changes with backward compatibility of older models. 14:49:58 ... the idea here is to look at the broader space of privacy discussions across similar features, specifically WebGPU 14:50:13 ... WebGPU has done a fantastic job in documenting their privacy and security considerations 14:50:21 ... I would to reuse these learnings to the extent possible 14:50:38 ... and would like to discuss Nick's fingerprinting guidance and look for best practices that would apply well to WebNN 14:50:48 ... and hear from Jonathan on the model loader API security 14:51:15 ... Christine, do you want to give a high level of overview of the PINg work mode and how we can effectively work with you? 14:51:23 ... We've already received valuable feedback from PING 14:51:42 Christine: 1st of all, thanks for coming to us early - it's very difficult to retrofit privacy 14:51:56 ... we have formal tooling that helps us keep track - we can take offline how to request privacy reviews 14:52:02 ... you already know of the self-review questionnaire 14:52:08 ... and Nick's excellent document 14:53:11 -> https://www.w3.org/TR/fingerprinting-guidance/ Mitigating Browser Fingerprinting in Web Specifications 14:53:45 Anssi: the fingerprinting guidance documents 10 best practices - Nick, any high level in the context of this group? 14:54:33 Nick: in general, we're not looking at eliminating all the things JS can learn about a device, but we can make it detectable, or less accessible, less permanent, less available x-origin 14:54:41 ... that might be significant for the ML context 14:55:10 ... e.g. fingerprinting based on feature detection - it could be made detectable 14:55:12 -> https://www.w3.org/TR/fingerprinting-guidance/#bp-summary Best Practices Summary 14:55:19 ... or not making it available to all Web sites 14:55:22 q+ 14:55:49 Anssi: we'll evaluate our group's work with this document 14:55:56 and here is the section on evaluating severity of fingerprinting surface: https://www.w3.org/TR/fingerprinting-guidance/#identifying-fingerprinting-surface-and-evaluating-severity 14:55:56 ... is it integrated in the PING questionnaire? 14:55:59 q- 14:56:09 q+ 14:56:27 q+ 14:56:45 Nick: it is referred from the questionnaire; once we start looking into fingerprinting, we would start evaluating these best practices 14:57:01 q? 14:57:08 q- 14:57:12 q? 14:57:20 ack weiler 14:57:58 weiler: in terms of what PING is asking for: we're not asking for verbatim answers to the questionnaire, instead we want a narrative based on the answers and mitigations 14:58:07 q+ to ask if PING can point to an example 14:58:10 ... we want to know what analysis has already been done 14:58:11 I think we should also consider privacy considerations that are more specific to machine learning in general. what are the capabilities of inference or manipulation that are enabled by adding ML into the browser? 14:59:06 ack sangwhan 14:59:06 sangwhan, you wanted to bring up hardware capabilities/acceleration (esp. with software support, e.g. cuDNN for desktop) as an extra entropy fingerprinting surface 14:59:15 Subtopic: WebGPU Security and Privacy Considerations 14:59:15 sangwham We keep a changelog for operator changes and each change creates a new operator version. https://github.com/onnx/onnx/blob/master/docs/Changelog.md 14:59:55 Anssi: Corentin, how did the WebGPU group develop its security and privacy considerations (to great results)? 15:00:12 -> https://www.w3.org/TR/webgpu/#security WebGPU Security considerations 15:00:15 Corentin: for WebGPU, we had a good starting point for the security side of things due to the experience with WebGL 15:00:29 -> https://www.w3.org/TR/webgpu/#security-privacy WebGPU Privacy considerations 15:00:31 with regrets, I have another group discussing privacy review specifically starting now. I'll definitely read the webgpu security write-up later 15:01:59 q- 15:02:30 [it would be good for PING to confirm that webgpu is a good example to follow] 15:04:06 Corentin: fingerprinting is a harder issue, because you can measure hardware that has different behaviour 15:04:24 ... you may want to expose some information about hardware, ML workloads, exposing this information is a privacy information 15:04:30 q+ 15:04:38 ... these issues have been hard to solve in WebGPU too 15:05:09 ... at the choice of the browser to have their own policies, for example, exposing the name of the adapter 15:05:39 "GPU bug workarounds" 15:06:02 ... use case is to switch content execution based on adapter name 15:06:08 Living proof: https://chromium.googlesource.com/chromium/src/+/HEAD/gpu/config/gpu_driver_bug_list.json 15:06:47 ... from the Chromium experience it is important to be able to workaround driver bugs 15:07:15 See https://github.com/gpuweb/gpuweb/issues/2191 for current discussion about exposing adapter naming. 15:07:46 q? 15:07:58 ack sangwhan 15:08:43 sangwhan: there's an extra fingerprinting surface if you enable neural network acceleration on mobile with co-processor, separate fingerprinting surface 15:09:25 ... e.g. cuDNN has different perf characteristics depending on which hardware it is run, this is extra fingerprinting surface 15:09:36 q? 15:10:02 q? 15:10:03 zkis has joined #webmachinelearning 15:10:52 So you have "GPU name" x "GPU dot product acceleration library version" x ("Coprocessor" x "Coprocessor software version") (parens for when you have a neural coprocessors that is not a GPU) 15:11:07 (and you can throw in GPU driver version I guess too) 15:11:13 RafaelCintron: the WebGPU is discussing how much of names to expose, on Windows quite a lot information is exposed 15:11:44 There is a tradeoff, and something that warrants investigation and mention in the spec. 15:11:52 http://browserleaks.com/webgl 15:12:31 q? 15:12:40 q+ 15:12:43 Thank you for inviting me. 15:12:59 we have one topic on the agenda: Discuss Model Loader API security 15:14:04 Present+ PaulAdenot, CarineBournez, FrancoisDaoust, HaraldAlvestrand, BernardAboba, ChrisChunningham, RandellJesup, SteveLee, Tove 15:14:16 topic: Next Generation Audio Codecs 15:14:20 [slide 50] 15:14:51 Present- PaulAdenot, CarineBournez, FrancoisDaoust, HaraldAlvestrand, BernardAboba, ChrisChunningham, RandellJesup, SteveLee, Tove 15:31:21 RRSAgent, draft minutes 15:31:21 I have made the request to generate https://www.w3.org/2021/10/26-webmachinelearning-minutes.html anssik 15:32:51 Present+ Kristen_Chapman 15:33:34 Present+ PetrPenzin 15:34:20 RRSAgent, draft minutes 15:34:20 I have made the request to generate https://www.w3.org/2021/10/26-webmachinelearning-minutes.html anssik 15:34:59 s/topic: Next Generation Audio Codecs// 15:35:06 s/[slide 50]// 15:35:10 RRSAgent, draft minutes 15:35:10 I have made the request to generate https://www.w3.org/2021/10/26-webmachinelearning-minutes.html anssik 16:34:17 zkis has joined #webmachinelearning 16:35:37 zkis has joined #webmachinelearning 16:37:03 zkis has joined #webmachinelearning 16:39:33 zkis_ has joined #webmachinelearning 17:58:18 Zakim has left #webmachinelearning 18:24:40 zkis_ has joined #webmachinelearning