14:53:53 <RRSAgent> RRSAgent has joined #webmachinelearning
14:53:53 <RRSAgent> logging to https://www.w3.org/2021/01/21-webmachinelearning-irc
14:53:55 <Zakim> RRSAgent, make logs Public
14:53:57 <Zakim> please title this meeting ("meeting: ..."), anssik
14:54:08 <anssik> Meeting: WebML CG Teleconference – 21 January 2021
14:54:11 <anssik> Chair: Anssi
14:54:15 <anssik> Agenda: https://github.com/webmachinelearning/meetings/blob/master/telcons/2021-01-21-agenda.md
14:54:19 <anssik> Scribe: Anssi
14:54:24 <anssik> scribeNick: anssik
14:58:26 <ningxin_hu> ningxin_hu has joined #webmachinelearning
14:58:58 <anssik> Present+ Anssi_Kostiainen
14:59:00 <anssik> Present+ Ningxin_Hu
14:59:52 <anssik> RRSAgent, draft minutes v2
14:59:52 <RRSAgent> I have made the request to generate https://www.w3.org/2021/01/21-webmachinelearning-minutes.html anssik
14:59:57 <anssik> Present+ Rafael_Cintron
15:00:11 <anssik> Present+ Chai_Chaoweeraprasit
15:00:53 <anssik> Present+ Ganesan_Ramalingam
15:01:42 <chai> chai has joined #webmachinelearning
15:01:54 <anssik> Present+ Zoltan_Kis
15:02:06 <RafaelCintron> RafaelCintron has joined #webmachinelearning
15:02:26 <anssik> RRSAgent, draft minutes v2
15:02:26 <RRSAgent> I have made the request to generate https://www.w3.org/2021/01/21-webmachinelearning-minutes.html anssik
15:03:28 <anssik> TOPIC: webnn-native
15:03:37 <zkis> zkis has joined #webmachinelearning
15:05:11 <anssik> anssik: I asked Ningxin to present the proposed standalone native implementation of the WebNN API spec called webnn-native that complements the JS implementation webnn-polyfill. Both the implementations help inform the API spec development, validate implementability atop native APIs (initial targets OpenVINO and DirectML, extensible with other backends), and provide a performance benchmark.
15:05:47 <Rama> Rama has joined #webmachinelearning
15:05:58 <anssik> ... Discuss the proposal to adopt webnn-native as a CG deliverable similarly to webnn-polyfill. No CG Charter changes needed, is considered "Other Software" per the existing charter text.
15:06:03 <anssik> -> https://webmachinelearning.github.io/charter/#test-suites CG Charter: Test Suites and Other Software
15:06:07 <anssik> "The group MAY produce test suites to support the Specifications. Please see the GitHub LICENSE file for test suite contribution licensing information."
15:06:47 <anssik> ningxin_hu: I would like to add about the problems we want to solve with webnn-native
15:07:19 <anssik> ... many discussions in this group are around native API implementability and compatibility, e.g. what ops can be mapped to native APIs, what params and options can be implemented
15:07:34 <anssik> ... also performance prediction, what perf we can get by offloading to native APIs
15:07:59 <ping_yu> ping_yu has joined #webmachinelearning
15:08:08 <anssik> ... we haven't had a good tool to get answers to these questions, this proposal is to help get those answers and inform the API spec development
15:08:12 <anssik> Present+ Ping_Yu
15:08:35 <anssik> ningxin_hu: while webnn-polyfill is based on TF.js, this will be based on native APIs
15:08:59 <Sandeep> Sandeep has joined #webmachinelearning
15:08:59 <anssik> ... header file can be mapped to WebIDL from the WebNN API spec
15:09:10 <anssik> ... arepresented as C/C++ interfaces
15:09:17 <anssik> s/arepresented/represented
15:09:37 <anssik> ... explainer enumerated backend targets, DirectML, OpenVINO initially
15:09:49 <anssik> ... sample code can be written in C and C++
15:10:06 <anssik> ... we target a few important models first -- that's the basic idea
15:10:21 <anssik> ... also this project should help with WebML-Wasm coordination
15:11:00 <anssik> ... the plan is to have the code be compiled into Wasm
15:11:15 <anssik> ... proposal here is to create a repo under the CG's GH org
15:11:27 <anssik> ... this would be Apache 2.0 licensed
15:12:06 <anssik> ... we can have some MVP deliverables for the first wave, e.g. minimal LeNet sample running with two backends
15:12:26 <anssik> Present+ Sandeep_Gupta
15:12:35 <zkis> Could a container be provided as well?
15:12:40 <anssik> q?
15:13:31 <anssik> zkis: Could a container be provided as well?
15:13:39 <anssik> ningxin_hu: good questions, we can discuss in GH
15:13:52 <anssik> ... primarily a source code release
15:14:11 <anssik> zkis: asking because OpenVINO is already available as a container
15:15:07 <anssik> zkis: another reason for containerization, Wasm also prefers that deployment model
15:15:24 <anssik> q?
15:16:53 <anssik> anssik: Chromium fork was hard to maintain, this proposal solves that issue I suppose, is these a plan to integrate this proposal with Chromium as e.g. a 3rd party dep?
15:18:03 <anssik> ningxin_hu: once matured, other projects could adopt this, but that's beyond the initial goal of the project
15:18:09 <anssik> q?
15:18:22 <anssik> proposed RESOLUTION: Adopt webnn-native as "Other Software" deliverable, repo hosted at webmachinelearning/webnn-native
15:18:46 <chai> +1 its a good idea
15:18:53 <zkis> +1
15:18:58 <RafaelCintron> +1
15:19:34 <anssik> RESOLUTION: Adopt webnn-native as "Other Software" deliverable, repo hosted at webmachinelearning/webnn-native
15:19:40 <anssik> RRSAgent, draft minutes v2
15:19:40 <RRSAgent> I have made the request to generate https://www.w3.org/2021/01/21-webmachinelearning-minutes.html anssik
15:20:17 <chai> i can help with the reviews
15:20:44 <anssik> TOPIC: WebNN conv2d layout parameter TensorFlow incompatibility
15:21:13 <anssik> anssik: Discuss and gather further feedback on TF, TF Lite, and TF.js preference for input and filter layout approach in WebNN API
15:21:19 <anssik> -> https://github.com/webmachinelearning/webnn/issues/125 TensorFlow conv2d expects channel_last filter layout regardless of input layout format #125
15:21:29 <anssik> anssik: there seems to be a PR for this by Chai
15:21:35 <anssik> -> https://github.com/webmachinelearning/webnn/pull/130 Support a separate filter layout formats and the optional bias tensor. #130
15:23:13 <anssik> Chai: more flexible to have separate filter layout for conv2d, also lacks bias 1D tensor added as the last step of the output
15:23:24 <anssik> ... appreciate review from folks
15:23:38 <anssik> ... this is not a breaking change, adds to the existing API
15:24:10 <anssik> ... would like to move optional parameters to the struct, if we need to do this in the future it'd be a breaking change
15:24:45 <ping_yu> I will review the PR as well
15:25:25 <anssik> TOPIC: Numerical precision in conformance testing
15:25:50 <anssik> anssik: Discuss and solicit insights on difference between TF.js WebGL and CPU/Wasm backends
15:25:57 <anssik> -> https://github.com/webmachinelearning/webnn-polyfill/pull/32 Add SqueezeNet model test
15:25:57 <anssik> #32
15:26:02 <Jonathan_> Jonathan_ has joined #webmachinelearning
15:26:12 <anssik> anssik: Ningxin has done some work and will give us an update?
15:26:27 <anssik> Present+ Jonathan_Bingham
15:26:54 <anssik> ningxin_hu: adds one 1st wave model as a test case SqueezeNet
15:27:32 <anssik> ... adapts the SqueezeNet model, two opens for the group:
15:28:47 <anssik> ... discussed in detail in the PR
15:29:06 <ningxin_hu> https://github.com/webmachinelearning/webnn-polyfill/pull/32#issuecomment-763825323
15:30:37 <anssik> ningxin_hu: second open regarding TF.js backend, supports multiple backends CPU Wasm, WebGL, but numerical precision differs
15:30:51 <anssik> ... the CPU backend passes the two models in TF Lite and ONNX formats
15:31:08 <anssik> ... GPU backend has some issues, leads to the failure of the two models
15:31:20 <anssik> ... which backend we should use for the numeric accuracy test?
15:31:38 <ping_yu> we have two settings, for webGL 2 epsilon set to e-3 and webGL 1 as e-1
15:31:51 <anssik> ... my proposal is to use the accuracy setting, and use the CPU backend to test the numerical accuracy
15:32:02 <chai> q+
15:32:24 <anssik> Ping: Ningxin is right in that for CPU we have better accuracy
15:32:35 <anssik> ... WebGL1 no float32 e.g. for iOS
15:33:23 <anssik> ... leads to precision loss, for webGL 2 epsilon set to e-3 and webGL 1 as e-1
15:33:27 <anssik> q?
15:33:29 <anssik> ack chai
15:34:33 <anssik> Chai: for the polyfill testing it might be OK, but from the spec itself, the reason to have conformance test is to ensure accuracy, there are so many frameworks using different criteria, so with multiple backends this becomes an issue
15:34:57 <anssik> ... the whole point of conformance is universality, that's the statement from the principles
15:35:36 <anssik> ... my second point, a data point, when we test we test on two levels: 1 model level, 2 op level
15:38:15 <ningxin_hu> q+
15:38:20 <anssik> ack ningxin_hu
15:38:43 <anssik> ningxin_hu: I'd like to comment about the test against the polyfill
15:39:22 <chai> 1. two level conformance -- models and operators
15:39:32 <anssik> ... polyfill is used as a tool to run the code, depending on the implementation of the polyfill e.g. with CPU backend accuracy satisfies the conformance test criteria
15:39:34 <ping_yu> +1 Good to have requirement on both tolerance on op level and model level
15:40:05 <chai> 2. for standard conformance, having a tolerance defined over standard double precision value ensures longevity
15:40:24 <anssik> Ping: I agree with Chai we should have different tolerance level for op and model
15:40:32 <anssik> ... target to have accurate output for the model
15:40:36 <chai> 3. try to define tolerance in term of ULP instead of absolute distances
15:41:24 <anssik> Ping: these ops may have certain tolerance,  if people can not pass the conformance test, they might be in trouble
15:41:29 <chai> 4. have different tolerance for different operation. dont use a single tolerance across everything -- that will be a least common denominator
15:41:42 <anssik> q?
15:42:12 <anssik> q?
15:42:56 <anssik> ningxin_hu: I propose to merge #32
15:42:59 <chai> <away from keyboard>
15:43:19 <anssik> ... this PR uses different tolerance for two different models
15:43:56 <anssik> ... tried to reference the respective native settings from ONNX and TF
15:44:12 <chai> <back on keyboard>
15:44:39 <anssik> Chai: PR#32 is fine to merged
15:44:59 <ping_yu> I will take a closer look
15:45:19 <anssik> anssik: after Ping's review, can merge
15:45:31 <anssik> q?
15:45:42 <anssik> TOPIC: NSNet2 sample and TF.js memory leak
15:45:58 <anssik> anssik: Check TF.js upstream blocker status and discuss any other opens blocking PR
15:46:05 <anssik> -> https://github.com/webmachinelearning/webnn-samples/pull/22 Add NSNet2 sample #22
15:46:17 <anssik> anssik: memory leak issues seems to be solved now thanks to Ping and TF.js folks!
15:46:22 <anssik> ... anything else blocking this PR?
15:46:46 <ping_yu> yes, I will do that todya
15:46:53 <ping_yu> sorry, I have to drop off now
15:47:04 <anssik> anssik: Ping to look at PR#22, can merge once that's completed
15:47:28 <anssik> TOPIC: Proposals for future work
15:47:48 <anssik> anssik: Continue discuss proposals submitted for consideration for future work:
15:47:54 <anssik> -> https://github.com/webmachinelearning/proposals/issues/1 Data processing by Wenhe Li
15:48:30 <anssik> -> https://github.com/webmachinelearning/proposals/issues/2 Operation-specific APIs #2 by Jonathan
15:48:37 <anssik> anssik: Chai had some feedback in this issue
15:49:25 <anssik> Chai: I was just asking for more information so we can better understand the proposal
15:49:58 <anssik> ... exactly what is the proposal, one way to clarify this is to define some samples or prototype functions
15:50:18 <anssik> ... for example, looking at the WebNN graph API, there the currency is an Operand
15:50:29 <anssik> ... that will eventually construct a Tensor
15:51:06 <anssik> ... prototype functions would help us better understand what is the currency flowing through these functions
15:51:42 <anssik> ... or are we talking of e.g. WebGL buffers?
15:52:01 <anssik> Jonathan: agree pseudo code or proto example would make this clearer
15:52:14 <anssik> ... started a conversation internally, we just haven't done that yet
15:52:44 <anssik> ... I take an action item to deliver a pseudo code or prototype example
15:53:44 <anssik> q?
15:54:53 <anssik> TOPIC: Adjourn
15:54:58 <anssik> RRSAgent, draft minutes v2
15:54:58 <RRSAgent> I have made the request to generate https://www.w3.org/2021/01/21-webmachinelearning-minutes.html anssik