13:56:54 <RRSAgent> RRSAgent has joined #webmachinelearning
13:56:54 <RRSAgent> logging to https://www.w3.org/2020/09/03-webmachinelearning-irc
13:57:00 <Zakim> Zakim has joined #webmachinelearning
13:57:04 <anssik> RRSAgent, make logs public
13:57:08 <anssik> Meeting: WebML CG Teleconference – 3 September 2020
13:57:19 <anssik> Chair: Anssi
13:57:19 <anssik> Agenda: https://github.com/webmachinelearning/meetings/blob/master/telcons/2020-09-03-agenda.md
13:57:23 <anssik> Scribe: Anssi
13:57:28 <anssik> scribeNick: anssik
13:57:37 <anssik> Present+ Anssi_Kostiainen
13:59:43 <Rama> Rama has joined #webmachinelearning
14:00:14 <anssik> Present+ Rafael_Cintron
14:00:19 <anssik> Present+ Ningxin_Hu
14:00:27 <anssik> Present+ Ganesan_Ramalingam
14:00:54 <RafaelCintron> RafaelCintron has joined #webmachinelearning
14:01:02 <ningxin_hu> ningxin_hu has joined #webmachinelearning
14:01:41 <anssik> Present+ Chai_Chaoweeraprasit
14:02:13 <anssik> TOPIC: Workshop presentations and GH discussions
14:02:21 <anssik> anssik: Workshop presentations are actively being discussed on GitHub. CG participants feedback has been appreciated, further engagement welcome.
14:02:34 <anssik> -> https://www.w3.org/2020/06/machine-learning-workshop/presentations.html Workshop presentations
14:02:42 <anssik> -> https://github.com/w3c/machine-learning-workshop/issues Workshop GH discussions
14:03:30 <anssik> TOPIC: WebNN polyfill and samples
14:03:50 <anssik> anssik: Discuss review feedback and suggestions for the foundational implementation
14:03:58 <anssik> -> https://github.com/webmachinelearning/webnn-polyfill/pull/1 Add the foundation implementation (PR #1)
14:04:01 <Chai> Chai has joined #webmachinelearning
14:05:19 <anssik> ningxin_hu: thanks for the review comments
14:06:11 <anssik> ... during the last two week we've been addressing the feedback and have done improvements per Ping's suggestions from TF.js API usage, for example
14:06:34 <anssik> ... WebNN is a graph builder so need some clarifications how to map WebNN graph to TF operators
14:06:52 <jonathan_> jonathan_ has joined #webmachinelearning
14:06:58 <anssik> ... some bug fixing to be done in that area, another improvement is documentation, added are JS docs for the public API of the polyfill
14:07:10 <paul_mcdaniel_msft> paul_mcdaniel_msft has joined #webmachinelearning
14:07:15 <anssik> ... it is lightweight since we have spec in place, rather than duplicate spec text
14:07:21 <anssik> Present+ Paul_McDaniel
14:07:45 <anssik> ningxin_hu: polyfill tracks the spec closely, so JS docs complement the spec, not duplicate
14:07:59 <anssik> ... also unit tests based on Mocha were added
14:08:28 <anssik> ... it would be good if Chai could review the unit tests, they are not perfect, but directional guidance welcome
14:08:48 <Ping_Yu> Ping_Yu has joined #webmachinelearning
14:08:56 <anssik> ... Anssi suggested hosting JS docs and unit tests via GH so those are added
14:09:01 <anssik> Present+ Ping_Yu
14:09:33 <anssik> q?
14:10:15 <anssik> ningxin_hu: Ping had a question re API design itself, so we agreed to open a design issue for the spec instead of polyfill issue, any API design issues should be discussed in the spec repo
14:11:08 <anssik> ... I will keep the first polyfill PR aligned with the current spec, so the proposal is to discuss any changes in WebNN spec repo first and then follow up with polyfill change that keeps up with the spec
14:12:30 <anssik> anssik: are the mocha tests automatically or manually generated?
14:12:40 <anssik> ningxin_hu: manually created
14:13:01 <anssik> anssik: interested in figuring out if we can reuse mocha tests for w-p-t tests
14:15:55 <anssik> ningxin_hu: worth opening issue about w-p-t testing plan
14:16:12 <Ping_Yu> It would be good to unified the op correctness test at the platform level.
14:17:47 <anssik> Ping_Yu: so my comment is for the op correctness, would be good to unify, polyfill is just one implementation, going forward there will be multiple  native browser implementations that need to be tested
14:18:17 <anssik> https://web-platform-tests.org/
14:19:33 <anssik> anssik: Discuss and review LeNet sample for handwritten digit recognition using WebNN
14:19:38 <anssik> -> https://github.com/webmachinelearning/webnn-samples/pull/1 Add LeNet example (PR #1)
14:20:58 <anssik> ningxin_hu: I acknowledge this example had a UI issue on narrow screens, will fix that
14:21:38 <anssik> ... in this sample, I display the compilation time for the graph, also execution time (inference time actually)
14:22:03 <anssik> ... this example can show the difference in performance if we run it on a native implementation
14:23:53 <Ping_Yu> sorry, I have not had a chance to check it out
14:24:30 <paul_mcdaniel_msft> sorry, have also not had a chance to review LeNet on my end either.
14:24:32 <Ping_Yu> one related question, what is the plan for benchmark?
14:25:40 <anssik> Ping_Yu: in the long run it'd be good to have a benchmark for all the implementations
14:26:33 <Chai> q+
14:26:45 <anssik> ningxin_hu: I'm only aware of  framework-specific benchmarks
14:26:47 <anssik> ack Chai
14:27:07 <anssik> Chai: I think the benchmark is a good idea when the API becomes more stable
14:27:27 <anssik> ... normally conformance and benchmark are done when you have a release, so you can test against a stable base
14:28:48 <Ping_Yu> we shall look into mlperf standard for mobile or NNAPI benchmark
14:28:52 <anssik> ... most trustworthy benchmarks are done by 3rd parties
14:28:57 <paul_mcdaniel_msft> https://mlperf.org/ is a common one
14:30:49 <Ping_Yu> There is a team in google working on mlperf, I can chat with them on possible web mlperf
14:31:29 <Chai> i'll take a look at lenet sample. took some time off last week. still catching up.
14:32:07 <anssik> TOPIC: GRU and corresponding ops
14:32:40 <anssik> anssik: Fill the operator gaps to support noise suppression first-wave models
14:33:00 <anssik> ... Adding GRU and GRUcell operators to support GRU recurrent network. Defining a cell operator in addition to the network operator for added customization flexibility e.g. to support stacked cell recurrent network, etc.
14:33:06 <anssik> Review PR: https://github.com/webmachinelearning/webnn/pull/83
14:33:13 <anssik> Proposal to close noise suppression issue with PR #83:
14:33:18 <anssik> https://github.com/webmachinelearning/webnn/issues/66
14:33:47 <anssik> anssik: are we fine to close issue #66 now that PR #83?
14:33:52 <anssik> Chai: will take a look
14:34:43 <paul_mcdaniel_msft> looks great !
14:34:50 <anssik> TOPIC: First-wave models and ops delta with WebNN API definition
14:35:01 <anssik> anssik: Review the delta between the first-wave models ops and WebNN API surface:
14:35:09 <anssik> [ ] clamp
14:35:09 <anssik> [ ] globalAveragePool - Lowering to reducewindow, add and div?
14:35:09 <anssik> [x] gru
14:35:09 <anssik> [x] sigmoid
14:35:10 <anssik> [ ] split - Lowering to slice?
14:35:10 <anssik> [x] squeeze
14:35:40 <anssik> anssik: Discuss which ops to add to the spec definition considering e.g. major platform support and performance implications.
14:35:46 <Chai> q+
14:35:51 <anssik> ack Chai
14:36:04 <anssik> Chai: globalAveragePool already supported
14:36:11 <anssik> ... split is new
14:36:20 <anssik> ... clamp need to be defined
14:36:48 <anssik> ... with this delta, I'd like to ask again, how do we think about adding new ops before closing on 1st wave models and what's the criteria to add new ops?
14:38:11 <anssik> anssik: my though is to work from use cases, to models, toward ops
14:38:11 <paul_mcdaniel_msft> https://github.com/webmachinelearning/webnn/blob/master/op_compatibility/first_wave_models.md
14:38:20 <anssik> ... should review our use cases that they're still good
14:38:44 <anssik> Chai: frameworks have model zoos that could be used as inspiration
14:39:07 <anssik> ... 1st wave ops set is around 50, which is about half way, the remaining ops being fillers
14:39:15 <anssik> ... so the questions is how to cut the first release
14:39:51 <paul_mcdaniel_msft> just FYI .. ONNX model zoo is here:  https://github.com/onnx/models
14:41:30 <Ping_Yu> Is there a guideline on what criteria the committee follows to determine what ops get to be added to the spec
14:41:51 <anssik> RafaelCintron: no strong opinion regarding release scoping
14:41:56 <paul_mcdaniel_msft> again just FYI .  tf.js model zoo is here (i think): https://www.tensorflow.org/js/models
14:42:14 <anssik> paul_mcdaniel_msft: similarly, no strong opinion on release scoping
14:50:22 <Ping_Yu> I have some comments
14:50:34 <anssik> ack Ping_Yu
14:51:17 <anssik> Ping_Yu: release without proper tests, people cannot be sure of conformance
14:51:29 <Chai> q+
14:51:54 <RafaelCintron> +1 to having test suites and conformance
14:52:15 <Chai> +1
14:52:19 <ningxin_hu> +1
14:52:19 <paul_mcdaniel_msft> agreed.  i would really like to have a test suite AND some reference implementations (backends) so that we know this really is viable before we publish our v1 api draft
14:52:23 <anssik> anssik: we should aim to have a test suite for the opset we are about to release
14:52:28 <Chai> this is why we also starting polyfill, samples and unit tests
14:53:36 <Chai> q+
14:53:48 <anssik> ack Chai
14:53:49 <anssik> q?
14:54:41 <anssik> Chai: from execution point of view, maybe good to say we're opening a proposal for adding new ops for 1st wave, but define a timebox  and process for adding new ops
14:56:34 <RafaelCintron> q+
14:57:24 <Ping_Yu> q+
14:58:20 <anssik> PROPOSED RESOLUTION: Define criteria for adding new ops to the spec
14:58:48 <anssik> PROPOSED RESOLUTION: Define criteria for adding support for new models
14:59:23 <anssik> RESOLUTION: Define criteria for adding support for new models
14:59:27 <anssik> q?
14:59:37 <anssik> ack RafaelCintron
14:59:39 <Ping_Yu> I would suggest define criteria for use case first instead of model
14:59:53 <anssik> RafaelCintron: want to say, maybe soonish we should request TAG review
15:00:10 <anssik> PROPOSED RESOLUTION: Request TAG feedback for WebNN API
15:00:15 <paul_mcdaniel_msft> (thinking out loud) it's almost like: step1 - the first wave of models identified.  step2: tests and samples and polyfils for wave1.  step3:  Proof-of-concept implementations for wave1; backend (DirectML) and frontends (onnx.js / tf.js) to make sure our api holds for implementation.   step4:  publish our findings .
15:00:59 <anssik> RESOLUTION: Request TAG feedback for WebNN API
15:01:20 <anssik> ack?
15:01:23 <anssik> q?
15:01:26 <anssik> ack Ping_Yu
15:02:01 <anssik> Ping_Yu: if our releases are MVP style, then people can start implementation, propose we minimize the models we include
15:02:25 <anssik> ... also minimize the ops involved, so people can try out sooner, rather than expand the scope
15:02:42 <anssik> ... another comment is, instead of focusing on models, focus on use cases
15:02:52 <anssik> ... models come and go, there may be new architectures in the future
15:03:44 <Chai> q+
15:03:51 <anssik> ack Chai
15:04:55 <anssik> Chai: aiming for MVP is a good idea, but we need to also balance the appearance of the first draft of the API, since if the functionality is too minimal impact is not there, this is delicate balance, how to define a reasonable set of models, ops, so that it covers sufficient set of use cases
15:05:11 <anssik> ... while allow implementers to start implementation early
15:05:27 <anssik> ... I assume MVP here means  the coverage we can successfully implement
15:06:09 <anssik> ...  the first milestone for WebNN, we're already past that, since we've worked on the API for over a year now, we should set the bar higher
15:06:14 <Ping_Yu> My comment is on the first wave of models level not API level
15:07:16 <Chai> we derive op set from the models
15:07:19 <paul_mcdaniel_msft> i like use cases -> model architectures -> ops.
15:07:24 <anssik> q?
15:09:05 <anssik> TOPIC: Adjourn
15:09:17 <anssik> RRSAgent, draft minutes v2
15:09:17 <RRSAgent> I have made the request to generate https://www.w3.org/2020/09/03-webmachinelearning-minutes.html anssik
15:10:32 <anssik> RRSAgent, make logs public
15:14:22 <anssik> s/TAG feedback/TAG review/
15:14:25 <anssik> RRSAgent, draft minutes v2
15:14:25 <RRSAgent> I have made the request to generate https://www.w3.org/2020/09/03-webmachinelearning-minutes.html anssik
17:29:42 <Zakim> Zakim has left #webmachinelearning