14:01:06 <RRSAgent> RRSAgent has joined #webmachinelearning
14:01:06 <RRSAgent> logging to https://www.w3.org/2020/10/22-webmachinelearning-irc
14:01:10 <Zakim> Zakim has joined #webmachinelearning
14:01:16 <anssik> RRSAgent, make logs public
14:01:20 <anssik> Meeting: WebML CG Virtual Meeting at TPAC 2020 – 22 October 2020
14:01:24 <anssik> Chair: Anssi
14:01:32 <kleber> kleber has joined #webmachinelearning
14:01:34 <anssik> Agenda: https://github.com/webmachinelearning/meetings/tree/master/2020-10-22-tpac2020
14:01:38 <ningxin_hu> ningxin_hu has joined #webmachinelearning
14:01:59 <vagner_> vagner_ has joined #webmachinelearning
14:02:13 <dom> dom has joined #webmachinelearning
14:02:17 <chb> chb has joined #webmachinelearning
14:02:17 <Chai> Chai has joined #webmachinelearning
14:02:18 <GeunHyung_Kim> present+ GeunHyung_Kim
14:02:22 <RafaelCintron> RafaelCintron has joined #webmachinelearning
14:02:28 <Erika_Miguel> Erika_Miguel has joined #webmachinelearning
14:02:38 <anssik> Present+ Chai_Chaoweeraprasit
14:02:39 <dom> Present+ Dominique_Hazael-Massieux
14:02:43 <RafaelCintron> present+ RafaelCintron
14:02:49 <Erika_Miguel> present+ Erika_Miguel
14:02:50 <anssik> Present+ Anssi_Kostiainen
14:02:52 <vagner_> present Vagner NIC.br
14:02:54 <martin> present+ Martin_Alvarez
14:02:54 <kleber> present+ Michael_Kleber
14:02:55 <takio> Present+ Takio_Yamaoka
14:02:55 <ningxin_hu> Present+ Ningxin_Hu
14:02:57 <mehmetoguzderin> Present+ Mehmet_Oguz_Derin
14:03:01 <BarbaraH> BarbaraH has joined #webmachinelearning
14:03:07 <Chai> present+
14:03:09 <vagner_> present+ Vagner NIC.br
14:03:14 <anita_chen> anita_chen has joined #webmachinelearning
14:03:19 <hcai> hcai has joined #webmachinelearning
14:03:19 <piwanczak> sorry to bother again - could I kindly ask for call in details? The w3c list is unaccessible for me unfortunately
14:04:02 <anssik> Scribe: Anssi
14:04:06 <anssik> scribeNick: anssik
14:04:06 <belem> belem has joined #webmachinelearning
14:04:08 <br-rtbhouse> @piwanczak – same for me, I'm getting either 'waiting...' or 'unauthorised'
14:04:44 <anssik> RRSAgent, draft minutes v2
14:04:44 <RRSAgent> I have made the request to generate https://www.w3.org/2020/10/22-webmachinelearning-minutes.html anssik
14:05:03 <anssik> RRSAgent, make logs public
14:05:24 <anssik> Topic: Workshop proceedings
14:05:54 <br-rtbhouse> Is the w3c website working fine for anyone else? I still cannot access this page with the call-in details :/
14:06:05 <anssik> anssik: W3C organized a workshop on Web and Machine Learning over the course of August and September 2020. This workshop brought together web platform and machine learning practitioners to enrich the Open Web Platform with better foundations for machine learning.
14:06:10 <ehsan> I still can't as well
14:06:11 <anssik> ... Today's meeting is to discuss the W3C Workshop on Web and Machine Learning key outcomes with a focus on proposed near-term and long-term standardization directions and next steps.
14:06:16 <anssik> Subtopic: Materials
14:06:20 <SandeepGupta> SandeepGupta has joined #webmachinelearning
14:06:29 <anssik> -> https://www.w3.org/2020/06/machine-learning-workshop/report.html
14:06:29 <anssik> https://www.w3.org/2020/06/machine-learning-workshop/report.html  Workshop report
14:06:35 <anssik> -> https://www.w3.org/2020/06/machine-learning-workshop/presentations.html Workshop presentations
14:06:43 <anssik> -> https://github.com/w3c/machine-learning-workshop/issues Workshop discussions on GitHub
14:06:51 <BarbaraH> Zoom logistics   https://mit.zoom.us/j/233956702?pwd=R0RCejUzWjhuWDlVK1UyQW9kcGxWZz09
14:06:54 <anssik> -> https://www.w3.org/2020/06/machine-learning-workshop/proceeding.html Workshop minutes
14:06:55 <kim_wooglae> present+ kim_wooglae
14:07:06 <br-rtbhouse> present+
14:07:33 <anssik> Subtopic: Next steps
14:07:43 <anssik> anssik: Next Steps from the workshop broken into three categories:
14:07:44 <ehsan> thanks
14:07:45 <jonasz_> present+
14:07:48 <anssik> -> https://www.w3.org/2020/06/machine-learning-workshop/report.html#next Next Steps
14:07:54 <ping_yu> ping_yu has joined #webmachinelearning
14:07:56 <anssik> anssik: Next Steps in Standardization
14:08:01 <anssik> ... Next Steps in Incubation
14:08:07 <anssik> ... Other exploratory work
14:08:13 <anssik> anssik: Let's focus on the first one, next steps in standardization:
14:08:29 <anssik> -> https://w3c.github.io/machine-learning-charter/charter.html Web Machine Learning Working Group Charter
14:08:37 <dom> https://github.com/w3c/machine-learning-charter/issues
14:08:50 <anssik> anssik: some issues raised for the Charter
14:08:59 <anssik> -> https://github.com/w3c/machine-learning-charter Web Machine Learning Working Group Charter - issues
14:09:13 <anssik> ... 1) Is a graph of operations the right level of abstraction for a web standard?
14:09:25 <anssik> https://github.com/w3c/machine-learning-charter/issues/2
14:09:31 <anssik> ... 2) Detailed explainer for Web NN API
14:09:37 <anssik> https://github.com/w3c/machine-learning-charter/issues/3
14:09:49 <anssik> anssik: We believe these issues are being addressed by the explainer update
14:09:53 <anssik> -> https://github.com/webmachinelearning/webnn/pull/109 Explainer update PR
14:09:57 <anssik> -> https://github.com/webmachinelearning/webnn/blob/wchao/fill_explainer/explainer.md WebNN API Explainer preview (in review, in staging)
14:10:33 <anssik> anssik: Inviting Chai to respond to the two concerns, starting with 1, Is a graph of operations the right level of abstraction for a web standard?
14:11:37 <anssik> RRSAgent, draft minutes v2
14:11:37 <RRSAgent> I have made the request to generate https://www.w3.org/2020/10/22-webmachinelearning-minutes.html anssik
14:11:40 <fengdai> fengdai has joined #webmachinelearning
14:13:07 <anssik> Chai: A summary of what has been discussed in CG is condensed into the first section of the explainer
14:13:42 <anssik> ... The diagram in the explainer depicts where the WebNN API sits
14:14:18 <anssik> ... WebNN API sits between OS and web browsers, because similarly to WebGL/GPU it provides a set of ML APIs that the browser can implement so that the web apps or frameworks  sit atop of
14:15:13 <anssik> ... why do we need this API then? In the past 5 year ago there has been a lot of innovation in hw ecosystem, e.g. in GPU also specialized AI accelerators have emerged tailored for ML workloads
14:16:05 <anssik> ... many web frameworks have also been created, TF.js, ONNX.js, so the problem is how to bridge this development on the hardware side with the developments on the web regardless of the OS
14:16:28 <anssik> ... anyone should be able to execute ML models on any browser and on any OS
14:17:20 <anssik> ... also in our early prototype, we've seen that in terms of performance, WebNN being able to connect to hardware, executes much faster than generic APIs such as WebGL/GPU
14:17:39 <anssik> ... with this interface, the web browser can support these experiences more efficiently
14:17:51 <anssik> ... another aspect, why does this need to run on OS?
14:18:20 <anssik> ... as a DirectML lead, I've learned the integrity is also very important
14:19:43 <anssik> [Chai walking through the example code]
14:19:54 <anssik> Chai: first non-goal is we do not intend to define a model format
14:20:06 <anssik> ... format can be built on top of the API
14:20:25 <anssik> ... people using ML know there are many formats out there, and the idea of WebNN API is to enable all these formats
14:20:52 <anssik> ... speaking to both model format, and serialization format, including encryption, packaging
14:21:02 <anssik> ... we do not define the delivery mechanism
14:21:22 <anssik> ... third, we do not define media formats, the web is already rich in terms of media type definitions, we want to interop with those types
14:22:46 <ningxin_hu> The WebNN POC perf num Chai just mentioned: https://www.w3.org/2020/06/machine-learning-workshop/talks/access_purpose_built_ml_hardware_with_web_neural_network_api.html#slide-10
14:22:49 <Jonathan_Bingham> Thanks for the updates to the Explainer, Chai. We'll take a look.
14:23:14 <anssik> Ehsan: What is the level of explainability and transparency for the ML models here?
14:24:15 <dom> https://github.com/w3c/machine-learning-workshop/issues/108 (in which Ehsan chimed in)
14:24:55 <jonasz> jonasz has joined #webmachinelearning
14:25:44 <anssik> anssik: 1) Is a graph of operations the right level of abstraction for a web standard?
14:25:44 <anssik> 
14:25:44 <anssik> https://github.com/w3c/machine-learning-charter/issues/2
14:25:54 <anssik> ... 2) Detailed explainer for Web NN API
14:25:54 <anssik> 
14:25:54 <anssik> https://github.com/w3c/machine-learning-charter/issues/3
14:26:30 <anssik> Chai: Not going to all the details, but responding to 1) re abstraction level
14:26:38 <anssik> ... we spent most of the time in CG discussing this topic
14:27:02 <anssik> ... the rationale is 1) if we look across all ML frameworks, every single one produces a graph
14:27:37 <anssik> ... job #1 is to understand what abstractions people are using
14:28:14 <anssik> Chai: the second thing is a bit deeper, what would be the abstraction that compose that graph
14:28:22 <anssik> ... these are many ideas discussing this topic
14:28:39 <anssik> ... looking across all the platform supporting ML today, iOS, macOS, Windows, Linux, Android ...
14:28:55 <anssik> ... the common currency is operator-level, conv, gem, reductions, all these important math functions
14:29:13 <anssik> ... in the context of studying these existing platforms and frameworks we found a lot of commonality
14:29:57 <anssik> ... we started with conv and gem and found other highly reusable functions used everywhere
14:30:06 <anssik> ... this is the common currency across all these platforms
14:30:35 <anssik> ... this looks reasonable for a contract, since #1 it is supported by consumer OSes, #2 it is already produced by the frameworks on the frontend level
14:31:02 <anssik> ... I hope that answers the questions raised why the API is defined as a graph of ops
14:33:05 <anssik> Jonathan: Thanks Chai for the updates to the explainer, we'll take a look
14:33:18 <anssik> ... also thank for review of how we ended up with a graph API
14:33:32 <anssik> ... sounds familiar to the Google's discussions, having created NN API
14:33:50 <anssik> ... we have a lot of users for our NN API, we learned something during that process
14:34:25 <anssik> ... the second this about the level of abstraction, all the ML frameworks do on some level, construct a graph, does not necessarily mean web platform should expose the same level
14:34:54 <anssik> ... that's not the real point, there are many ops that are common across ML models
14:35:04 <anssik> ... this is the reason we went in this direction with Android NN API
14:35:27 <anssik> ... in out experience from NN API in Android, and TF, is that there is no end to the number of ops that can be useful
14:35:35 <anssik> ... we have to major sets of ops for TF
14:35:47 <anssik> ... on the one hand we have TF that is research focused
14:36:05 <anssik> ... engaging with the research community, thus adding new ops almost daily, lots of churn
14:36:18 <anssik> ... TF has not over 1000 ops, historically growing double-digit per yera
14:36:24 <anssik> s/yera/year
14:36:49 <anssik> Jonathan: TF Lite focuses on a smaller set of ops
14:37:09 <anssik> ... that smaller set of ops has grown from 30-40 ops MVP to 120 ops today
14:37:17 <anssik> ... that's pretty similar to ONNX opset
14:37:24 <anssik> ... could be the outcome of WebNN API too
14:37:45 <anssik> ... but even there, we already feel pulled to too many operations and there are so many models people want to run and cannot be expressed
14:37:58 <anssik> ... Google has been working on NN API next version taking a different approach
14:38:19 <anssik> ... we try to find lower level building blocks for that
14:38:37 <anssik> ... I prefer to call them instructions, where ops are higher level than instructions
14:38:44 <anssik> ... instructions end up in a graph
14:39:09 <anssik> ... so discussion is whether we have a graph of ops, graph of instructions, or a mix of both
14:39:25 <anssik> ... we're like to be cautions to go too high-level for a web standards
14:39:42 <anssik> ... we're wanting to make sure we have some up with a low-level set of instructions
14:39:59 <anssik> ... that could be potentially be done by a graph API, maybe also with a model loader API
14:40:18 <anssik> ... as people in this group have said, there approaches are complementary
14:41:28 <anssik> Anssi: do we have an explainer doc for the low-level instructions proposal?
14:41:38 <anssik> Jonathan: we're very early with this proposal
14:41:45 <anssik> ... we have Tensor Compute Primitives
14:41:51 <anssik> ... that's one initiative
14:41:59 <anssik> ... another one is XLA-SLO
14:42:38 <ping_yu> TF-RISC
14:42:49 <anssik> ... internally also working on TF Reduced Instruction Set
14:43:33 <anssik> Jonathan: implications to WebNN, in the case of TF Lite, it has 120 ops, the current version of TF-RISC only has ~20% of those
14:43:41 <anssik> ... it is quite different
14:45:57 <anssik> Anssi: How about moving ahead with WebNN API now and see what comes in the future and adapt as needed?
14:46:24 <anssik> Jonathan: That is a valid questions, we would need to find out how much value WebNN API does have for web developers it is worth doing
14:47:19 <anssik> ... to answer that questions, I'd like to ask for an update from Ningxin on some performance benchmark, WebNN API vs. WebGPU/GL/Wasm, what is the performance benefit pushing this through now, even if the WebNN API might not be the forever there API?
14:47:52 <ningxin_hu> https://www.w3.org/2020/06/machine-learning-workshop/talks/access_purpose_built_ml_hardware_with_web_neural_network_api.html#slide-10
14:47:59 <Chai> q+
14:48:03 <anssik> Ningxin: I can drop a link to my workshop talk, that has some PoC data in it, it can be used as a reference for performance gains
14:48:36 <anssik> ... Slide 11 has the performance numbers on a smartphone, we tested on two devices, PC using OpenVINO, on smartphone we use NN API
14:48:57 <anssik> [Ningxin shares slide 10]
14:49:36 <anssik> Ningxin: performance numbers on PC using CPU, GPU, AI accelerator
14:50:27 <anssik> ... if we look at the difference, for Wasm with SIMD+Threads, there's 8x speedup with WebNN with delegation to native ML API undernearth
14:50:58 <anssik> ..compared to WebGL, WebNN ~5x faster
14:51:10 <anssik> s/..compared/... compared
14:51:14 <Jonathan_Bingham> Are these performance numbers for latency of the first inference ? Or for subsequent inferences? Because there are startup costs.
14:51:52 <anssik> ningxin_hu: we use VNNI and int8 with WebNN, not apples to apples comparison to Wasm using float32
14:52:20 <anssik> Jonathan: Is this latency for the first inference, or running multiple times?
14:52:36 <anssik> ningxin_hu: multiple inference runs
14:52:50 <anssik> Jonathan: I'd like to get our engineer to work with you on this data, thanks!
14:53:44 <mehmetoguzderin> but
14:53:45 <anssik> ningxin_hu: On Slide 11, using Android smartphone, on CPU 2.4X faster using WebNN, using GPU 4.5X, with DSP lower precision int8 inference, 10X speedup
14:53:57 <mehmetoguzderin> (sorry, accidental message)
14:54:02 <mehmetoguzderin> Haha no no, so sorry!
14:54:04 <anssik> s/but//
14:56:33 <anssik> -> https://w3c.github.io/machine-learning-charter/charter.html Web Machine Learning Working Group Charter
14:56:33 <anssik> 
14:56:33 <anssik> anssik: some issues raised for the Charter
14:56:33 <anssik> 
14:56:33 <anssik> -> https://github.com/w3c/machine-learning-charter Web Machine Learning Working Group Charter - issues
14:56:49 <dom> q+
14:57:02 <anssik> q?
14:57:14 <anssik> ack Chai
14:57:21 <BarbaraH> BarbaraH has joined #webmachinelearning
14:58:11 <anssik> Chai: Listening to Jonathan, a lot of what Jonathan said sounds very familiar to me as the lead of WinML used by industry widely
14:58:31 <anssik> ... we are not necessarily making a decision between short vs long-term investment
14:59:00 <anssik> ... we work closely with all ISV, and all these vendors have been super focused on improving their hardware
14:59:23 <anssik> ... the performance numbers shared by Ningxin demonstrate what the hardware is capable and what the frameworks utilize
14:59:43 <anssik> ... the web is already a lot of behind in this sense, we can never know what might come in the future
15:00:09 <anssik> ... by defining this set of ops to be forwarded to the OS and hardware is the way to close the gap, as the gap gets wider every year
15:00:48 <anssik> ... speaking with my first-hand experience working on this space for many years, with billions of devices out these running Windows leveraging ML capabilities
15:01:06 <anssik> ... I don't think WebNN will be a stopgap solution
15:01:09 <anssik> q?
15:01:17 <anssik> ack dom
15:01:48 <anssik> Dom: on the short vs long term is a very complex decision to make, and I'm happy the community is pursuing this question
15:02:15 <anssik> ... in the past we have cases such as XHR replaced by Fetch, WebGL being replaced with WebGPU possibly
15:02:45 <anssik> ... what would be useful for me re WG formation is the timeline it takes to get to agreement on this question
15:02:54 <anssik> ... I'd be grateful if you could share the timeline with me
15:04:01 <anssik> Jonathan: we saw the explainer today, we saw the proposal the WG creation is happening during the workshop
15:04:21 <anssik> ... I expect that over the course of next couple of weeks we take a look at the materials available to understand this better
15:04:32 <anssik> ... before we do that prepwork we cannot give a timeline
15:06:09 <anssik> Dom: Two weeks making plans around this is totally fine.
15:06:24 <anssik> ... the tension is making sure the platform has the right tools for developers
15:06:28 <anssik> ... this is delicate balance
15:06:36 <anssik> q?
15:07:34 <anssik> Topic: Adjourn
15:07:42 <anssik> RRSAgent, draft minutes v2
15:07:42 <RRSAgent> I have made the request to generate https://www.w3.org/2020/10/22-webmachinelearning-minutes.html anssik
15:07:50 <Joshue108> Joshue108 has joined #webmachinelearning
15:08:16 <belem> belem has left #webmachinelearning
15:08:20 <Joshue108> present+
15:08:38 <takio> takio has left #webmachinelearning
15:08:41 <anssik> s/R0RCejUzWjhuWDlVK1UyQW9kcGxWZz09/redacted
15:08:45 <anssik> RRSAgent, draft minutes v2
15:08:45 <RRSAgent> I have made the request to generate https://www.w3.org/2020/10/22-webmachinelearning-minutes.html anssik
15:10:13 <anssik> Present+ Jonathan_Bingham
15:10:26 <anssik> Present+ Anita_Chen
15:10:44 <anssik> Present+ Belem
15:11:09 <anssik> Present+ Hong_Cai
15:11:48 <anssik> RRSAgent, draft minutes v2
15:11:48 <RRSAgent> I have made the request to generate https://www.w3.org/2020/10/22-webmachinelearning-minutes.html anssik
15:14:32 <anssik> anssik: Thank you all for attending!
15:14:33 <anssik> RRSAgent, draft minutes v2
15:14:33 <RRSAgent> I have made the request to generate https://www.w3.org/2020/10/22-webmachinelearning-minutes.html anssik
15:44:47 <zkis> zkis has joined #webmachinelearning
15:55:33 <myles> myles has joined #webmachinelearning
17:33:25 <Zakim> Zakim has left #webmachinelearning
18:32:54 <zkis> zkis has joined #webmachinelearning