IRC log of webmachinelearning on 2019-08-08

Timestamps are in UTC.

14:00:43 [RRSAgent]

RRSAgent has joined #webmachinelearning

14:00:43 [RRSAgent]

logging to https://www.w3.org/2019/08/08-webmachinelearning-irc

14:00:44 [Zakim]

Zakim has joined #webmachinelearning

14:00:54 [anssik]

RRSAgent, make logs public

14:01:14 [anssik]

Meeting: WebML CG Teleconference – 8 August 2019

14:01:19 [anssik]

Chair: Anssi

14:01:26 [anssik]

Agenda: https://github.com/webmachinelearning/meetings/blob/master/telcons/2019-08-08-agenda.md

14:01:27 [Ningxin_Hu]

Ningxin_Hu has joined #webmachinelearning

14:01:34 [kainino]

kainino has joined #webmachinelearning

14:01:34 [anssik]

Scribe: Anssi

14:01:45 [anssik]

scribeNick: anssik

14:01:51 [anssik]

Regrets+ Thomas_Steiner

14:01:56 [anssik]

Present+ Anssi_Kostiainen

14:02:00 [anssik]

Present+ Rafael_Cintron

14:02:02 [Rafael]

Rafael has joined #webmachinelearning

14:02:02 [Ningxin_Hu]

Present+ Ningxin_Hu

14:02:04 [anssik]

Present+ Ganesan_Ramalingam

14:02:09 [Rafael]

present+

14:02:11 [anssik]

Present+ Paul_McDaniel

14:02:23 [anssik]

Present+ Gabe_Esteven

14:02:39 [anssik]

Present+ Jonathan_Bingham

14:03:11 [anssik]

Present+ Kai_Ninomiya

14:03:31 [anssik]

Present+ Greg_Whitworth

14:03:38 [anssik]

RRSAgent, draft minutes v2

14:03:38 [RRSAgent]

I have made the request to generate https://www.w3.org/2019/08/08-webmachinelearning-minutes.html anssik

14:03:53 [Nikhil]

Present+ Nikhil Thorat

14:03:56 [Nikhil]

Present+ Daniel Smilkov

14:04:04 [Nikhil]

Present+ Nikhil_Thorat

14:04:07 [Nikhil]

Present+ Daniel_Smilkov

14:04:10 [Nikhil]

:)

14:04:27 [anssik]

RRSAgent, draft minutes v2

14:04:27 [RRSAgent]

I have made the request to generate https://www.w3.org/2019/08/08-webmachinelearning-minutes.html anssik

14:04:40 [anssik]

TOPIC: Define the set of operations and their specification

14:04:51 [anssik]

-> https://github.com/webmachinelearning/webnn/issues/17 Define the set of operations and their specification #17

14:05:17 [anssik]

anssik: we had a review of the proposed resolution and receiver good feedback we need to resolve, let's discuss that now.

14:05:50 [anssik]

... the objective of this call is to resolve objections raised for the proposed resolution and clarify proposed resolution based on feedback where appropriate

14:06:38 [anssik]

To start, I captured the following questions from issue #17 we need to resolve:

14:07:06 [anssik]

nsthorat: "An important part of this specification will be ensuring this set of ops are compatible with the major ML JavaScript frameworks [...] it's not possible for us to move forward with this resolution without understanding compatibility."

14:07:41 [anssik]

jbingham: "what's the plan for dealing with versioning?"

14:07:55 [anssik]

jbingham: "How are custom ops defined and included in the graph?"

14:08:09 [anssik]

walrusmcd: "How many ops?"

14:08:29 [anssik]

jbingham: "Decide if a graph API is the right thing to standardize on"

14:09:13 [anssik]

anssik: To summarize, need to choose a set of operations to be included in the API that enables adequate compatibility in the major ML frameworks

14:09:56 [anssik]

q+ to ask about something

14:10:04 [Nikhil]

q+ to talk about something

14:10:19 [anssik]

q+

14:10:22 [anssik]

q?

14:10:22 [Nikhil]

q+ to talk about onnx & tf lite compatibility doc: https://docs.google.com/document/d/1RXCkZ9mliWbqSakYvNlWhsRH4yFtnpe1YQQNFAIRZo8/edit

14:10:25 [jonathan]

jonathan has joined #webmachinelearning

14:10:37 [anssik]

ack Nikhil

14:10:37 [Zakim]

Nikhil, you wanted to talk about something and to talk about onnx & tf lite compatibility doc: https://docs.google.com/document/d/1RXCkZ9mliWbqSakYvNlWhsRH4yFtnpe1YQQNFAIRZo8/edit

14:11:04 [anssik]

Nikhil: shared doc on the chat, please take a look

14:11:25 [anssik]

... spend time looking at compat, started with basic 2 ops, tried to understand diff

14:11:27 [jdarpinian]

jdarpinian has joined #webmachinelearning

14:12:16 [gabe]

gabe has joined #webmachinelearning

14:12:23 [anssik]

... starting with low number of ops is our preference and grow that over time to understand compat issues of each op

14:13:00 [Rafael]

q+

14:13:33 [kainino]

Present+ James_Darpinian

14:14:02 [anssik]

danielsmilkov: this is about diffing libs, looking into possible compat issue, 1) when comparing with NN API e.g. some ops allow fusing, we propose separate ops no fused kernel, under the hood implementer could fuse the ops so that it runs great on a particular hardware

14:14:39 [anssik]

... ONNX is opinionated regarding the layout, TF lite wants channels come last, different hw depend channels first or channels last

14:14:49 [anssik]

... which layout is better changes over time

14:16:31 [anssik]

Nikhil: would prefer to start very small with a POC that works, and have a plan how to grow that set of ops

14:16:39 [anssik]

q?

14:17:24 [anssik]

... probably need a way to deal with custom ops, have a way in app space to describe custom ops share memory with matmul

14:17:26 [anssik]

ack?

14:17:29 [anssik]

ac?

14:17:31 [anssik]

q?

14:17:49 [anssik]

ack Rafael

14:18:27 [anssik]

Rafael: I agree with a plan to keep this hardware agnostic

14:18:39 [Nikhil]

awesome! that would be great.

14:18:47 [Nikhil]

regarding the script to convert onnx / tensorflow

14:20:10 [anssik]

Paul: everything Rafael basically yes, goal is to be hw agnostic

14:21:10 [anssik]

... ONNX has done work on channel formats and hit these same issues, proposed solutions

14:21:18 [anssik]

q?

14:21:21 [Ningxin_Hu]

q+ to talk about op set & use cases

14:21:22 [anssik]

ack anssik

14:21:22 [Zakim]

anssik, you wanted to ask about something and to

14:21:24 [anssik]

ack?

14:21:29 [anssik]

ack Ningxin_Hu

14:21:29 [Zakim]

Ningxin_Hu, you wanted to talk about op set & use cases

14:22:17 [anssik]

Ningxin_Hu: thanks for the efforts of Nikhil and Daniel, great work! Agree with approach of starting with a small set of ops and validate compat with JS libs

14:23:26 [anssik]

... proposal how to grow the op set: add ops that are needed to implement identified use cases

14:23:42 [anssik]

https://webmachinelearning.github.io/webnn/#usecases

14:23:50 [Nikhil]

q+ want to talk about custom ops technical details

14:24:26 [Ningxin_Hu]

op set and use cases: https://github.com/webmachinelearning/webnn/issues/17#issuecomment-508426036

14:24:29 [anssik]

[silence, agreement]

14:24:38 [anssik]

ack Nikhil

14:25:32 [anssik]

q+ Nikhil

14:26:10 [anssik]

Paul: we took an approach where we selected the ops that benefit from hw acceleration

14:26:28 [anssik]

... a bit similar approach to CUDA

14:27:43 [anssik]

Ningxin_Hu: if we only select expensive ops that benefit from hw, that may impose perf penalty when doing context switching

14:28:19 [anssik]

Paul: I agree, it might be worth prototyping that now, assumption we're proposing is this hybrid approach (w/ WebGL) is viable

14:28:25 [jonathan]

What other ML frameworks should review each op, like Daniel did for TensorFlow, and confirm compatibility before we finalize the definition?

14:30:15 [anssik]

Ningxin_Hu: agree with Paul's comments, interleaving with Wasm in POC, overhead was significant

14:30:51 [anssik]

Rafael: CPU readback is slow, staying with GPU compute shaders should work pretty well

14:30:57 [anssik]

q?

14:31:29 [RRSAgent]

I have made the request to generate https://www.w3.org/2019/08/08-webmachinelearning-minutes.html anssik

14:32:02 [anssik]

q?

14:32:24 [anssik]

ack Nik

14:32:27 [anssik]

ack Nikhil

14:32:29 [anssik]

q?

14:32:30 [jdarpinian]

q+

14:32:36 [anssik]

ack jdarpinian

14:33:02 [anssik]

jdarpinian: I'm on the Chrome team and think custom ops based on WebGL can work, but will be very complex to implement

14:33:33 [Nikhil]

We think it's important to be able to have custom operations share memory with conv2d / matmul without doing a readback. for cpu-accelerators, share the buffer with WASM, for gpu-accelerators share the buffer with WebGL

14:33:52 [anssik]

... portability between custom ops between different systems, CPU and GPU not very good

14:33:56 [Ningxin_Hu]

q+ talk about mem layout reordering overhead between custom ops and hw accelerated ops

14:34:05 [Ningxin_Hu]

q+

14:34:14 [anssik]

q?

14:34:34 [Nikhil]

this allows us to grow the spec slowly and not have tail-end ops be bottlenecks and the webnn accelerated ops can get quick wins by accelerating the bottleneck ops (conv2d, matmul, etc)

14:34:37 [anssik]

Paul: I think Ningxin_Hu posted an architecture diagram

14:34:51 [anssik]

-> https://github.com/webmachinelearning/webnn/issues/17#issuecomment-518915131 arch diagram

14:35:28 [anssik]

Paul: frameworks will do the heavy lifting, web developer won't see the complexity

14:36:01 [anssik]

Nikhil: we think the same, but not all devices have WebGL backend so fallback to Wasm for example

14:36:07 [anssik]

q?

14:36:14 [anssik]

ack Ningxin_Hu

14:36:42 [anssik]

Ningxin_Hu: about custom ops, folks talked about memory transfer overhead

14:38:08 [anssik]

... even long SIMD instructions on CPU can require tensor memory re-layout, an expensive operation

14:38:19 [anssik]

q?

14:38:29 [Nikhil]

q+

14:39:35 [anssik]

anssik: it was asked on the issue whether graph is the right abstraction?

14:40:31 [jdarpinian]

q+

14:40:45 [anssik]

jonathan: what are the other JS frameworks we need to take into compatibility study?

14:41:32 [anssik]

Paul: in ONNX we considered all frameworks that matter, they have a voice in ONNX project

14:42:28 [anssik]

... in ONNX we have considered PyTorch, Caffe, Intel's frameworks, Microsoft's frameworks, TensorFlow, we have ONNX to TF converter, Apple's CoreML

14:42:50 [anssik]

... CoreML was part of the opset 1 compatibility

14:42:55 [anssik]

q?

14:43:31 [anssik]

Nikhil: specifically interested in JS ML frameworks

14:43:40 [anssik]

... for compatibility

14:44:06 [anssik]

... for example, Brain.js

14:44:54 [anssik]

Paul: we don't want to have two bodies managing op schema, right?

14:45:18 [anssik]

Nikhil: we want to grow slowly, right?

14:45:40 [anssik]

... focus on web stuff to figure out an intersection of JS ML libraries, does that sounds reasonable?

14:46:41 [anssik]

Paul: ONNX does have namespace and versioning concepts, so we could create our own ONNX namespace for the ops references by Web NN API

14:47:22 [anssik]

Rafael: it is up to us to decide how many ops to adopt, the op definitions themselves would come from ONNX standards body

14:48:11 [anssik]

danielsmilkov: that makes sense, want to be clear, because of portability issues and JS libs as users, some changes needed to ONNX may be needed e.g. memory layout

14:48:43 [anssik]

Paul: that's fairly reasonable, ONNX community would certainly welcome that

14:48:58 [anssik]

danielsmilkov: relaxing, not breaking existing ONNX behaviour

14:49:13 [anssik]

... going to custom ops

14:50:09 [anssik]

... we deal with real models every way, need to add ops to TF, interoperability important for e.g. pre and post-processing of media, video

14:50:22 [anssik]

q?

14:50:48 [anssik]

ack jdarpinian

14:51:21 [anssik]

jdarpinian: also need to look into hardware we want to support, there's a lot of hardware out these and new coming up, e.g. neural engines coming up in ARM chips

14:51:44 [kainino]

q+

14:51:58 [anssik]

Nikhil: that's a good point, e.g. for matmul would be good to do homework checking how that works across all hardware

14:55:28 [anssik]

anssik: Daniel and Nikhil could you move your doc https://docs.google.com/document/d/1RXCkZ9mliWbqSakYvNlWhsRH4yFtnpe1YQQNFAIRZo8/edit#heading=h.n1gbg8k8lggq into a GH issue

14:55:37 [anssik]

Nikhil: yes, we'll do that

14:56:29 [anssik]

danielsmilkov: GH issue #17 there's a comment where Ningxin_Hu proposed 14 ops, we could do the work to split these 14 ops into 3-4 GH issues with some logical bundling

14:57:24 [anssik]

PROPOSED RESOLUTION: The specification will reference the ONNX operations and if there are any improvements desired for ONNX the work should be there.

14:57:36 [Ningxin_Hu]

14 ops proposal: https://github.com/webmachinelearning/webnn/issues/17#issuecomment-512651711

14:58:41 [anssik]

PROPOSED RESOLUTION: The specification will reference a subset of the ONNX operations, starting small, adding more ops when compatibility with major ML JavaScript frameworks has been validated

14:58:48 [kainino]

q+ re: jdarpinian, want to point out it's important to not only understand the current and upcoming hardware, but since the browser runs in userspace we also need to run on top of the userspace apis (NNAPI, CoreML, DirectML) so we are constrained by how they expose things

14:59:31 [Nikhil]

q+ AI for custom ops

14:59:34 [anssik]

q?

14:59:43 [anssik]

ack want

14:59:43 [Zakim]

want, you wanted to talk about custom ops technical details

14:59:46 [anssik]

q?

14:59:51 [anssik]

ack AI

14:59:51 [Zakim]

AI, you wanted to discuss custom ops

14:59:56 [anssik]

ack kainino

14:59:56 [Zakim]

kainino, you wanted to discuss jdarpinian, want to point out it's important to not only understand the current and upcoming hardware, but since the browser runs in userspace we

15:00:00 [Zakim]

... also need to run on top of the userspace apis (NNAPI, CoreML, DirectML) so we are constrained by how they expose things

15:00:35 [anssik]

kainino: we want to point out it's important to not only understand the current and upcoming hardware, but since the browser runs in userspace

15:00:43 [anssik]

... we also need to run on top of the userspace apis (NNAPI, CoreML, DirectML) so we are constrained by how they expose things

15:00:49 [anssik]

q?

15:00:52 [anssik]

ack Nikhil

15:01:14 [anssik]

Nikhil: sharing memory with custom ops needs to be better understood

15:01:37 [anssik]

... can you Ningxin_Hu do that investigation?

15:01:57 [anssik]

Ningxin_Hu: with help from james or kai we could make progress with custom ops issue

15:02:25 [anssik]

Rafael: have bandwith to help, but not time to drive

15:02:32 [anssik]

jdarpinian: the same, can help not drive

15:02:41 [anssik]

Ningxin_Hu: I can take the lead, with help from others

15:03:08 [anssik]

PROPOSED RESOLUTION: The specification will reference a subset of the ONNX operations, starting small, adding more ops when compatibility with major ML JavaScript frameworks has been validated

15:03:13 [kainino]

Ningxin_Hu: Please reach out to us as needed

15:03:23 [kainino]

oops that's supposed to be @Ningxin_Hu

15:03:39 [Ningxin_Hu]

thanks @kainino

15:05:10 [anssik]

https://www.w3.org/2019/09/TPAC/

15:06:35 [anssik]

any concerns with the amended proposed resolution?

15:06:43 [anssik]

[hearing no concerns]

15:06:48 [anssik]

RESOLUTION: The specification will reference a subset of the ONNX operations, starting small, adding more ops when compatibility with major ML JavaScript frameworks has been validated

15:06:59 [anssik]

RRSAgent, draft minutes v2

15:06:59 [RRSAgent]

I have made the request to generate https://www.w3.org/2019/08/08-webmachinelearning-minutes.html anssik

15:08:42 [anssik]

TOPIC: Adjourn

15:09:08 [anssik]

RRSAgent, draft minutes v2

15:09:08 [RRSAgent]

I have made the request to generate https://www.w3.org/2019/08/08-webmachinelearning-minutes.html anssik

17:08:27 [Zakim]

Zakim has left #webmachinelearning