13:54:24 RRSAgent has joined #webmachinelearning 13:54:24 logging to https://www.w3.org/2020/04/16-webmachinelearning-irc 13:54:26 Zakim has joined #webmachinelearning 13:54:31 RRSAgent, make logs public 13:54:37 Meeting: WebML CG Teleconference – 16 April 2020 13:54:43 Chair: Anssi 13:54:51 Agenda: https://github.com/webmachinelearning/meetings/blob/master/telcons/2020-04-16-agenda.md 13:54:52 Scribe: Anssi 13:54:57 scribeNick: anssik 13:55:07 Present+ Anssi_Kostiainen 13:55:15 Present+ Rafael_Cintron 13:56:46 Present+ Greg_Whitworth 13:57:46 RafaelCintron has joined #webmachinelearning 13:58:17 scribenick: gregwhitworth 13:59:25 Rama has joined #webmachinelearning 13:59:37 ningxin_hu has joined #webmachinelearning 14:00:08 Present+ Ganesan_Ramalingam 14:00:13 Present+ Ningxin_Hu 14:00:18 baul_eun has joined #webmachinelearning 14:00:56 Present+ Baul_Eun 14:02:22 Present+ Rama 14:03:23 Nikhil has joined #webmachinelearning 14:03:35 Present+ Nikhil_Thorat 14:04:52 Topic: New ops for WebNN API 14:05:13 anssik: as you know we have added conf and matmul to the API 14:05:27 daniel_smilkov has joined #webmachinelearning 14:05:28 anssik: now is the time to discuss what the next ops we think we should add 14:05:52 anssik: the proposal is, prioritize ops that address identified use cases 14:05:55 -> https://webmachinelearning.github.io/webnn/#usecases WebNN API use cases 14:06:48 paul_mcdaniel_msft has joined #webmachinelearning 14:06:50 anssik: the space of usecases is expanding rapidly, we should also maintain these use cases that inform the API priorities to ensure we target the future and not the past 14:07:25 anssik: currently the usecases informantly reference papers from publicly available locations 14:07:42 anssik: we should continue to bring to the group's attention the latest developments in this space 14:08:00 anssik: with that background, let's hear proposals for new ops to add to the WebNN API 14:08:26 anssik: the floor is open 14:08:36 ack anssik 14:08:36 anssik, you wanted to ask 14:09:09 q+ 14:09:15 ack ningxin_hu 14:09:48 ningxin_hu: I think I like the approach to identify the models and usecases and we can use operators that are used in those models 14:10:09 https://github.com/huningxin/webnn/wiki/WebNN-first-wave-models-and-ops 14:10:11 ningxin_hu: with that, I recommend to do some homework and make a table with the operators for the models 14:10:54 ningxin_hu: on each row there is an operator and models 14:11:10 ningxin_hu: this does not fully cover all usecases but does cover a lot of them 14:11:58 ningxin_hu: add, multiply and some activations as well as pooling ops (avg pool) 14:12:09 ningxin_hu: there are some that we already support, like convulution 14:12:28 s/convulution/convolution 14:12:40 ningxin_hu: I left some questions open in the table as well. That's my initial try to identify the ops 14:12:45 q? 14:13:40 q+ 14:13:49 anssik: thank you ningxin_hu, this is very helpful 14:14:12 anssik: I think we could add this content into the spec as informative content 14:14:47 ACTION: ningxin_hu submit a PR for model ops table into spec or wiki 14:15:00 q? 14:15:26 Nikhil: this is a bigger question, we've been spending quite a bit of time about what's going on with TF ops 14:15:47 Nikhil: is there any other standard that is going to make its way into an open body 14:16:05 Nikhil: I'm slowly thinking we should target a lower level of abstraction 14:16:44 Nikhil: we'll be chasing the ops forever and they'll be iterating at a high rate. I know this is kind of a left turn but this at about the level of XLA 14:17:08 Nikhil: the 80 or so ops within XLA covers the majority of neural nets they've seen 14:17:20 Nikhil: I'd challenge us to consider lower level abstraction 14:17:34 Nikhil: there are only 80 where as in TF there are 1K 14:17:51 Nikhil: there are tensor compute primitives and XLA, another Google one... 14:18:03 Nikhil: something at the same level of XLA 14:18:36 Nikhil: that's my current thinking, but to answer your question regarding ops. I have intuitions but I think looking at real models and consider the next steps 14:18:59 Nikhil: interested in hearing people's thoughts 14:19:03 q? 14:19:08 ack RafaelCintron 14:20:07 RafaelCintron: I acknowledge what Nikhil is saying, my worry if we get too low level we at the browser can't make them performant. But we'd have to see what they are, you say matmul and conf2d are included. What would be the other ones? 14:20:24 s/conf2d/conv2d 14:20:50 Nikhil: that's a great point, what you mentioned. I'm not saying this is 100% what we do, but I think it will be easier if we have a smaller opset 14:21:09 Nikhil: If we ride the wave of another standard looking at ops 14:21:45 q+ 14:22:03 Nikhil: there's the other approach is that you would never go upwards, you'd ship that lower level ops and it would go directly to x86, arm, etc. If we have to call something higher than that - that's an interesting question 14:22:08 q+ 14:22:24 RafaelCintron: looking at the ops that ningxin_hu proposed are those too high level? 14:23:12 ack paul_mcdaniel_msft 14:23:37 paul_mcdaniel_msft: Nikhil one thing that may help, low level vs high level. Some scenarios we've hit in the past if we go too low you hit real issues with optimizations 14:24:16 paul_mcdaniel_msft: there is an lstm sitting in there but there was no way of knowing this and inferring the structure. Keras added to the TF model the information to allow for optimization 14:24:58 Nikhil: yeah, there are a lot of problems here and that's a valid point. There is no golden bullet. That said it is valid to have lower level, smaller opset that is pretty solid and not moving 14:25:28 paul_mcdaniel_msft: the ONNX group has been trying to do, is function which allows you to go down to these lower level ops while retaining the higher level concept of what it is 14:25:57 paul_mcdaniel_msft: ningxin_hu is talking about activation but you don't want to know its a mul and add because the GPU can do things when it knows it's an activation 14:26:14 daniel_smilkov: just wanted to add something to Nikhil suggestion 14:26:26 Present+ Daniel_Smilkov 14:26:51 daniel_smilkov: I don't think we want to go extremely low level, we want to find intermediate so that we don't have to add ops every 3 months 14:27:14 daniel_smilkov: XLA for example has LTSM information to allow for optimization on GPU/TPU, etc 14:27:16 FWI: how the linux foundation is starting to craft "how to propose new ops" 14:27:18 https://github.com/onnx/onnx/blob/master/docs/AddNewOp.md 14:27:52 daniel_smilkov: issue with high level is we'll need to keep adding ops. Whatever we standardize today, a year from now we'll have a new architecture that will have more opsets that we'll need to add. It's just the growth 14:28:01 daniel_smilkov: my worry is us keeping up 14:28:09 q+ 14:28:12 RRSAgent, draft minutes v2 14:28:12 I have made the request to generate https://www.w3.org/2020/04/16-webmachinelearning-minutes.html anssik 14:28:24 daniel_smilkov: we'll have to add custom ops if we don't which we've lost the hardware benefit 14:28:28 paul_mcdaniel_msft: that's a good point 14:28:51 paul_mcdaniel_msft: the ONNX foundation, we were worried that we'd have unchecked ops - but it's slowed down and we've started to deprecate ops 14:29:12 paul_mcdaniel_msft: I don't know if we're at steady state, but we were worried about that as well and I don't think we've seen that 14:29:14 q+ to re-ask this question from RafaelCintron: looking at the ops that ningxin_hu proposed are those too high level? 14:29:19 onnx model architectures 14:29:19 https://github.com/onnx/models 14:29:23 ack gregwhitworth 14:29:35 daniel_smilkov: the issue is we're trying to guess it 14:29:47 paul_mcdaniel_msft: I'm saying one step stronger, we're not guessing in ONNX 14:30:01 paul_mcdaniel_msft: we have over two dozen model arch that have fed the opsets in ONNX 14:30:15 paul_mcdaniel_msft: it's worth looking at as a case study 14:31:06 daniel_smilkov: it would be a good exercise to go back a year and half ago to understand what models wouldn't run due to missing ops in ONNX 14:31:18 anssik: is that something you all can look at? Or folks in ONNX 14:31:24 daniel_smilkov: yep we can look at that 14:31:33 Jonathan has joined #webmachinelearning 14:31:56 ACTION: daniel_smilkov to look at the opsets from a year ago to see what couldn't be ran on ONNX opsets 14:32:20 Nikhil: the goal wasn't to derail this, just to propose this thought because others at Google is considering this 14:32:36 q? 14:32:41 ack anssik 14:32:41 anssik, you wanted to re-ask this question from RafaelCintron: looking at the ops that ningxin_hu proposed are those too high level? 14:33:07 https://github.com/huningxin/webnn/wiki/WebNN-first-wave-models-and-ops 14:33:57 ACTION: Nikhil and daniel_smilkov to compare ningxin_hu ops and XLA to see what, if any, are too high level 14:34:04 q+ 14:34:13 ack ningxin_hu 14:34:40 XLA HLO: https://www.tensorflow.org/xla/operation_semantics 14:35:14 ningxin_hu: want to mention that I, the operator list, I did initial investigation into XLA. I reviewed that and there are some overlaps for example convolution and matmul 14:35:35 ningxin_hu: there are some other things such as batch normalization and concat, elementwise, etc 14:35:54 NIkhil do you have a copy to that other lower level op you mentioned? was it "TCP"? thx ! 14:35:56 ningxin_hu: we have something common - so my initial take is to grow that set slowly doing our due diligence by studying 14:36:12 ningxin_hu: so we can bring in other element wise ops 14:36:44 ningxin_hu: ningxin_hu will make a PR that will allow Nikhil and daniel_smilkov comment on 14:36:45 q? 14:36:56 ack RafaelCintron 14:37:33 RafaelCintron: just the question we have from earlier, ONNX is not in the thousands but in the hundreds and XLA seems to be similar. Maybe we can look at an intersection of those 14:37:56 RafaelCintron: that we can agree on that allow for hardware optimizations but not high enough where we're adding numerous ops every year 14:39:51 Rama: I feel that actually it is helpful to look at it as tensor and scalar ops 14:40:03 Rama: you can identify key categories 14:40:26 Rama: you end up covering a large aspect of ops 14:40:42 Rama: one way to limit it is to identify the op categories 14:41:09 Rama: I think if we do that exercise we'll find the commonalities across XLA and ONNX 14:42:24 Rama: the question that arises is, that if you express it this way can you reference them at a higher level? 14:42:44 Rama: there is value in doing that because of the existing implementations 14:43:13 Rama: hand crafted compilers is normally better, while there is innovation in MLIR, custom is normally better 14:43:22 Rama: so there is value in defining higher level ops 14:46:15 ACTION: paul_mcdaniel_msft to do an intersection between XLA & ONNX 14:46:56 RRSAgent, draft minutes v2 14:46:56 I have made the request to generate https://www.w3.org/2020/04/16-webmachinelearning-minutes.html anssik 14:47:09 TOPIC: Inference API to load and run a model 14:47:35 Jonathan has joined #webmachinelearning 14:47:44 anssik: to quickly summarize where we are 14:48:32 https://github.com/webmachinelearning/model-loader 14:48:50 anssik: we reached consensus on incubating this idea in this group 14:49:01 anssik: next steps is to start drafting the actual spec text 14:49:11 anssik: the use cases are already covered in the existing explainer 14:49:50 q+ 14:49:56 anssik: can anyone speak to prototyping? Possibly folks on the call 14:50:04 ack RafaelCintron 14:50:19 RafaelCintron: we can help with the mission of the load model API 14:50:58 RafaelCintron: I think the group needs to define a canonical format, we don't want to have fragmentation which is not good for web developers. 14:51:09 RafaelCintron: that obviously will bring us full circle to the ops 14:51:41 anssik: yeah that's the good long term goal but the API shape doesn't require us to have that 14:52:06 anssik: our current charter blocks us from making a declaration of a model format 14:52:23 RafaelCintron: really, we can't define a format but we can define an opset 14:52:24 https://webmachinelearning.github.io/charter/#out-of-scope 14:53:55 https://github.com/webmachinelearning/charter/issues 14:54:08 gregwhitworth: I recommend we do start the charter changes, but yes it shouldn't block the API 14:54:22 ACTION: anssik create a strawman spec for others to iterate on 14:54:52 TOPIC: Virtual workshop agenda building 14:55:27 Sorry, my internet is down. Catching up on meeting notes. @Anssi 14:55:29 -> https://bit.ly/webml-workshop-talks Virtual workshop agenda 14:55:34 anssik: I'd like this group to help figure out what to discuss on the virtual workshop 14:55:42 Jonathan: no worries :) 14:56:01 Hard to wfh with no internet and poor cell phone reception 14:56:02 anssik: Nikhil it would be good to have a talk around different abstractions, such as XLA, MLIR, etc 14:56:39 anssik: we would have offline short talks that would allow for async consumption 14:56:49 anssik: Nikhil & daniel_smilkov do you all think that's a good topic? 14:57:10 -> https://bit.ly/webml-workshop-talks Virtual workshop agenda 14:57:18 ACTION: Nikhil and daniel_smilkov to put together abstraction talk proposal 14:57:41 anssik: I'm trying to figure out which parts of the topics would make good for a virtual workshop 14:58:25 anssik: I'd expect us to try and cover 30-50% of the content. We would have 4 90minute sessions to discuss these topics 14:58:44 anssik: for example have folks from Google & XLA to answer questions and have a discussion around these topics 14:59:01 anssik: if you can just vote for which talks you think should be created for offline consumptions 14:59:32 anssik: for now just add a comment of support, if necessary I'll create a more formal voting process 15:00:04 anssik: we have settled on Zoom + Slack for infra 15:00:10 -> https://www.w3.org/Guide/meetings/zoom.html Zoom @ W3C 15:00:30 anssik: On that page there are tips how to mitigate some of the known security/privacy issues raised recently 15:01:03 anssik: Slack has been used in the past workshops with success, it is more accessible to people than IRC 15:01:32 ... I will setup our next CG call with Zoom so we can dogfood the system and iron out any kinks 15:01:52 anssik: we're trending for a time in June 15:03:16 TOPIC: Adjourn 15:03:25 anssik: thanks Greg for scribing!! 15:03:30 RRSAgent, draft minutes v2 15:03:30 I have made the request to generate https://www.w3.org/2020/04/16-webmachinelearning-minutes.html anssik 16:58:25 Zakim has left #webmachinelearning 18:25:18 zkis_ has joined #webmachinelearning 19:58:58 zkis_ has joined #webmachinelearning 20:08:30 zkis__ has joined #webmachinelearning