14:00:11 RRSAgent has joined #webmachinelearning 14:00:16 logging to https://www.w3.org/2023/10/19-webmachinelearning-irc 14:00:16 RRSAgent, make logs Public 14:00:17 please title this meeting ("meeting: ..."), anssik 14:00:19 Meeting: WebML WG Teleconference – 19 October 2023 14:00:22 Phillis_Tang has joined #webmachinelearning 14:00:24 dom has joined #webmachinelearning 14:00:25 Chair: Anssi 14:00:45 Agenda: https://github.com/webmachinelearning/meetings/blob/main/telcons/2023-10-19-wg-agenda.md 14:00:49 Scribe: Anssi 14:00:53 scribeNick: anssik 14:01:01 gb, this is webmachinelearning/webnn 14:01:01 anssik, OK. 14:01:04 present+ Rachel Yager 14:01:05 Present+ Anssi_Kostiainen 14:01:09 Regrets+ Dominique_Hazael-Massieux 14:01:17 Present+ Dominique_Hazael-Massieux 14:01:25 Regrets- Dominique_Hazael-Massieux 14:02:20 RRSAgent, draft minutes 14:02:22 I have made the request to generate https://www.w3.org/2023/10/19-webmachinelearning-minutes.html anssik 14:02:39 asully has joined #webmachinelearning 14:02:58 Present+ Reilly_Grant 14:03:12 Present+ Austin Sullivan 14:03:15 ningxin_hu has joined #webmachinelearning 14:04:22 phillis has joined #webmachinelearning 14:04:23 Deepti has joined #webmachinelearning 14:04:36 Present+ Phillis_Tang 14:04:43 Present+ Ningxin_Hu 14:05:09 Present+ Austin_Sullivan 14:05:20 Present+ Deepti_Gandluri 14:05:45 Present+ Reilly_Grant 14:06:09 Present+ Vivek_Sekhar 14:06:19 Present+ Zoltan_Kis 14:06:24 RRSAgent, draft minutes 14:06:25 I have made the request to generate https://www.w3.org/2023/10/19-webmachinelearning-minutes.html anssik 14:06:32 Present+ Rachel_Yager 14:07:13 anssik: we again had a number of new folks joining the WG since our last meeting, let me introduce them 14:07:44 ... please welcome Reilly Grant from Google Chrome; Reilly co-chairs W3C Devices and Sensors WG with me and works on device-related features in Chromium, he is very knowledgeable in spec authoring having edited many specs 14:08:53 .. leading a new team with Phillis that is looking at Web APIs such as WebNN 14:09:10 s/.. leading/... leading 14:09:18 zkis has joined #webmachinelearning 14:10:20 Reilly: I've worked on Shape Detection API a while ago, also APIs for high-level ops such as Media Capture extensions 14:10:34 chai has joined #webmachinelearning 14:10:55 ... also looking at ML from computation perspective as a new workload, how it works with Wasm, WebGPU, we see devices coming to the market with more purpose-built accelerators 14:11:04 Present+ Chai_Chaoweeraprasit 14:11:55 AramZS has joined #webmachinelearning 14:12:13 anssik: similarly, please welcome Christian Weyer, CTO of Thinktecture AG, a SW service and consultancy company working in web tech space; I've worked with some of his team in W3C context, e.g. Christian Liebel who I met at TPAC recently to discuss this WG's recent progress 14:12:29 ... also please welcome to Austin Sullivan from Google who has worked on File System related web features 14:13:03 Austin: worked on File System features, moving into the new space with Reilly 14:14:15 Phillis: Phillis Tang, working with Reilly on this new team 14:15:03 anssik: I want to note BlinkOn conference is currently on 14:15:13 ... on Day 1 Oct 17 a breakout session "WebNN implementation on DirectML" was on the BlinkOn agenda, those who attended feel free to share your key takeaways 14:15:38 Chai: BlinkOn is also today, Rafael is there at this very moment 14:16:01 ... we had one breakout session on WebNN, it was well-attended 14:16:14 ... chatted with Deepti there at BlinkOn 14:16:48 ... went well and the recording is posted on YT on BlinkOn channel, as a summary, this is replay of the talk we gave at Intel Innovation conference a month ago 14:16:58 Enjoyed the talk, thanks for giving it Chai! 14:17:21 ... a bit more discussion, after the talks and break we talked with a number of Google people, I'm happy to share the status of the work we're doing here with a number of people 14:17:29 ... new faces here, very encouraging! 14:17:47 ... this is an active area with interest from many companies in this space, 14:18:10 ... the keynote was on the challenges of running AI in the browser by Jim Bankoski, I had a good chat with him 14:18:31 q? 14:18:45 Ningxin: Chai did a great job represent WebNN at BlinkOn 14:19:21 Topic: WebNN v2: Review transformer ops spec contributions 14:19:34 anssik: issue #375 14:19:35 https://github.com/webmachinelearning/webnn/issues/375 -> Issue 375 Support for transformers (by dontcallmedom) [v2] [operation set] 14:20:00 ... the WG has actively worked on transformer op definitions as its priority effort during Q3 and now Q4 14:20:16 ... we have followed the guidelines for adding new operations: 14:20:22 -> https://github.com/webmachinelearning/webnn/blob/main/CONTRIBUTING.md#proposing-and-adding-a-new-operation 14:20:57 ... we have identified the use cases and sample models (thanks Dwayne, JoshuaL!): 14:21:12 ... - Text-to-image: https://huggingface.co/runwayml/stable-diffusion-v1-5 14:21:20 ... - Image segmentation: https://github.com/facebookresearch/segment-anything 14:21:29 ... - Speech-to-text: https://huggingface.co/openai/whisper-tiny 14:21:41 ... - Text-to-text generation (encoder-decoder): https://huggingface.co/t5-small and https://huggingface.co/facebook/m2m100_418M 14:21:49 ... - Text-generation (decoder-only): https://huggingface.co/meta-llama/Llama-2-7b 14:22:13 ... we have done the initial op decomposition aka Transformer Models Analysis (thanks Wanming!): 14:22:18 -> https://docs.google.com/spreadsheets/d/1ELfHuv2UqP2LoXWLgqsC0L8T_qqfBx48KxzFighl8d8/ Transformer Models Analysis 14:22:25 q+ 14:22:43 Vivek has joined #webmachinelearning 14:22:44 dwayner has joined #webmachinelearning 14:23:39 ... based on Google's feedback this spreadsheet now includes also TOSA and StableHLO mapping contributed by Dwayne, feedback welcome on this mapping in particular from Google's TOSA and StableHLO experts 14:24:30 Dwayne: there's roughly 1:1 mapping for all except one, triangularMatrix 14:25:07 ... the next step for the WG is to investigate: 14:25:26 ... - Cross-framework support. Is an identical or similar operation supported by multiple popular frameworks? What are they? 14:25:55 q+ 14:25:57 ... - Cross-platform implementability. Is the operation implementable in more than one platform? What are they? 14:25:59 q? 14:26:07 ack chai 14:27:05 Chai: I think in the BlinkOn session there were a few good questions re op coverage, I explained the methodology how we look at these overlaps across frameworks and design principles re small and big ops we're talked about in this WG for a while 14:27:59 ... ongoing balance, tradeoff, trying to find things that are useful to everyone, discussion is alive and well in the minds of people, transformers have good overlap across frameworks 14:28:21 q? 14:28:26 ack dom 14:29:12 dom: how are the conversations around transformers and LLMs, should we look at storage and caching of large models? 14:30:16 anssik: we had a dedicated session to discuss storage APIs and caching on our call a month ago 14:30:25 https://www.w3.org/2023/09/21-webmachinelearning-minutes.html 14:30:45 q? 14:30:49 q+ 14:30:57 ack reillyg 14:31:42 Reilly: I think that for storage there's potential integrations with some of the storage APIs 14:32:20 ... can load constants from buffer views, could also do that from storage objects such as blobs directly 14:32:50 ... on the general subject of integrations with other APIs, I saw MLCommandEncoder and it sounds nice for WebGPU integration 14:33:14 ... Wasm is getting better support for loading additional ArrayBuffers to share immediate values with the WebNN API possibly 14:33:17 q? 14:34:49 anssik: we are also starting to work on spec definitions for these ops in parallel 14:35:05 ... we've also committed to follow the security guidelines for new operations to ensure proper considerations are given to the design 14:35:12 -> https://www.w3.org/TR/webnn/#security-new-ops Security guidelines for new operations 14:35:33 anssik: I want to make sure the WG addresses any security issues reported or identified as a top priority 14:35:48 ... on today's agenda there's a separate topic to discuss the recently published WebGPU security technical report to distill any learnings to apply to WebNN 14:35:54 anssik: so that's the current status of transformer ops effort 14:36:06 ... now I want us to discuss and review any spec PRs or contributions for the proposed transformer ops informed by the op breakdown exercise 14:36:19 ... I know Chai is working on a spec PR for op definitions, I think it needs a bit more work before we can review it, Chai? 14:36:45 Chai: it is a big PR, I've been working on it actively over the last few week, I hope to be able to share the PR soon 14:36:57 ... can push it out and have people look at it 14:37:30 ... what is outlined in the issue #375 is pretty complete picture, the expected spec definition PR addresses the issue pretty closely 14:37:30 https://github.com/webmachinelearning/webnn/issues/375 -> Issue 375 Support for transformers (by dontcallmedom) [v2] [operation set] 14:37:51 ... maybe we can collapse some of the ops to reduce [no pun] the API surface 14:38:13 ... design principles around small and big ops applied to the design 14:38:53 ... I've also discussed with Zoltan and he will make time available in Q4 to help the WG with these spec PRs and supports reviews, align new ops with the modern spec conventions as needed 14:38:54 q? 14:39:15 Zoltan: I try to be useful for the WG, I got great help from Joshua Bell 14:39:30 q+ 14:39:36 ack reillyg 14:40:53 Reilly: some of the earlier documentation on v2 plans, where the group is in terms of removing some ops? 14:41:13 q+ 14:41:22 ack chai 14:41:41 -> https://www.w3.org/TR/webnn/#security-new-ops 3.1. Guidelines for new operations 14:41:53 Chai: great question, versioning is an important topic, v1/v2 is what we coined 14:42:16 ... the changes to op set need to be taken with case, we've done some name changes that have been done carefully 14:42:49 ... in my day job, working on Windows, we are very sensitive to API breakage, want it not to happen 14:43:23 ... if there's something that is redundant we can composite the semantics in terms of other ops 14:43:40 q+ 14:43:48 ... transformers change will be one of the bigger ones, I'll pay a lot of attention to any possibly breaking changes 14:43:51 ack ningxin_hu 14:44:16 ningxin_hu: in prototyping, we work with Google and Microsoft on Chromium implementation, DML and XNNPACK backends 14:44:41 ... this implementation experience informs the spec design 14:44:57 ... we actively open new spec issues as we find issues in implementation 14:45:29 ... op set simplification is informed by implementation experience too, more data points for the WG 14:45:44 q+ 14:45:55 q? 14:45:57 ack Deepti 14:46:18 Deepti: wanted to touch on the API surface and possible changes to it 14:46:51 ... it is my understanding it is still early in browser adoption, we have experimental implementation in Chromium but not yet other public web engine implementations 14:47:16 ... the WG should take that into consideration, want to know what the requirements are, how to minimize API breakage 14:47:18 q? 14:48:17 q+ 14:48:27 ack reillyg 14:48:55 q+ 14:48:57 reillyg: I've looked at CoreML API with Phillis, and it does a bit of digging to find it, they have a list of ops supported by their platform 14:49:25 ... the WG should start looking to include those ops into the analysis 14:49:43 ack chai 14:50:47 phillis_ has joined #webmachinelearning 14:51:21 Chai: before the spec reached initial CR we talked with our Apple contacts and they reviewed the spec and the latest feedback was they liked the spec design as backend API, graph API, also noting it is implementable on their platform 14:52:13 ... for op comparison, we can look at different frameworks, we want to be extensive in our research, want to say op coverage is still a living area, maybe not as fast as earlier 14:52:41 ... there will be changes to native OS-level APIs too, but it is a never-ending story 14:53:43 RRSAgent, draft minutes 14:53:45 I have made the request to generate https://www.w3.org/2023/10/19-webmachinelearning-minutes.html anssik 14:53:51 q? 14:54:04 Topic: WebGPU security technical report 14:54:12 anssik: Jiewei (thanks!) brought to the WG's attention the recently published WebGPU security technical report: 14:54:16 -> https://chromium.googlesource.com/chromium/src/+/main/docs/security/research/graphics/webgpu_technical_report.md WebGPU security technical report 14:54:49 ... this well-written report (kudos to authors tiszka, bookholt, mattdr) outlines how WebGPU works through the mind of an attacker, Chrome team's vulnerability research methodologies, and thought processes in some of the more difficult research areas. 14:55:04 ... I encourage all WG participats to read this report, in particular I'd ask the WG to identify any hardening opportunities for the WebNN API based on these WebGPU security insights. 14:55:13 ... from the report I gather WebGPU introduces two unique attack surfaces to Chrome: 14:55:24 ... - the WebGPU API implementation which was added to the GPU process & renderer process; and 14:55:35 ... - the WGSL (pronounced "wig-sal"?) shader compiler added to the GPU process 14:55:50 ... at the end there's a section summarizing systemic concerns 14:55:54 -> https://chromium.googlesource.com/chromium/src/+/main/docs/security/research/graphics/webgpu_technical_report.md#systemic-concerns WebGPU systemic concerns 14:56:10 anssik: I'm glad the Chrome Offensive Security team shared this report publicly so other APIs can learn from these insights 14:56:22 ... any findings anyone else would like to surface for discussion in the context of WebNN API? 14:56:23 RRSAgent, draft minutes 14:56:25 I have made the request to generate https://www.w3.org/2023/10/19-webmachinelearning-minutes.html anssik 14:56:31 q? 14:56:36 q+ 14:56:40 ack reillyg 14:56:46 AramZS_ has joined #webmachinelearning 14:57:04 Reilly: I did not yet read the report, but have worked on similar API that enable this type of interaction with renderer and service processes 14:57:42 ... the common theme is that anytime you give untrusted content the ability to affect the lifetime of trusted context, you may provide exploitable surface 14:58:04 ... e.g. referring to platform resources in privileged processes in WebNN 14:58:13 q? 14:58:24 q+ 14:58:42 anssik: any other similar reports? 14:58:58 reillyg: Project Zero has made good publications in this space 14:59:15 q? 14:59:32 ack ningxin_hu 15:00:01 ningxin_hu: want to add that because WebNN supports multiple devices, CPU, GPU, NPU, different backends have different security design considerations 15:00:10 ... XNNPACK CPU backend runs in renderer sandbox 15:00:17 ... different from GPU backend in GPU process 15:00:38 sorry need to drop. 15:00:38 ... WebNN is HW-agnostic and has different security considerations 15:00:38 q? 15:00:55 q? 15:01:49 RRSAgent, draft minutes 15:01:51 I have made the request to generate https://www.w3.org/2023/10/19-webmachinelearning-minutes.html anssik 15:14:59 zkis has joined #webmachinelearning 15:34:58 zkis has joined #webmachinelearning 16:08:50 AramZS has joined #webmachinelearning 16:10:03 AramZS has joined #webmachinelearning 17:06:16 Zakim has left #webmachinelearning 18:03:24 zkis has joined #webmachinelearning 22:21:00 AramZS has joined #webmachinelearning