Meeting minutes
WebNN first wave models and ops
anssik: Discuss WebNN first wave models and ops, action review
anssik: Review PR
first_wave_models.md (HTML preview)
anssik: models in this proposal target image classification and object detection use cases
… models are SqueezeNetV1.1, MobileNetV2, ResNetV2, TinyYOLOV2
… 13 ops: Add, AveragePool, BatchNormalization, Concat, Conv, Gemm, GlobalAveragePool, LeakyRelu, MaxPool, Mul, Relu, Reshape, Softmax
anssik: related actions:
… 1. Compare with XLA to see what ops, if any, are too high level (Nikhil & Daniel)
… 2. Review opsets from a year ago to see what couldn't be ran on ONNX opsets (Daniel)
… 3. Intersection between XLA & ONNX (Paul)
anssik: Daniel's review comments to fix action 1 was to compare which ops are too high level for XLA HLO (see PR#52 for the context), feedback:
Daniel: AveragePool, GlobalAveragePool, and MaxPool can be expressed in XLA HLO via ReduceWindow which is lower level because it can take an arbitrary function f: T->T of type XLAComputation.
Daniel: Gemm can be covered by Matmul and Add. If the underlying hardware supports Gemm, the fusion of matmul+add --> gemm should be done by an internal optimization pass.
Daniel: LeakyRelu, Relu and Softmax can all be lowered into other XLA HLO ops.
https://github.com/pytorch/xla/blob/master/torch_xla/csrc/ops/softmax.cpp#L15
Chai: I think no need to map 1:1 to XLA-HLO, but to make sure the set is not too large, should consider break them up
… compiler is specific to the browser, so my concern is how to make sure the compiler has enough context to handle the big group as one
… want to understand how to preserve this context, maybe fusion container could be contained within the graph
… with this information compiler can use it as a clue to check whether the hardware actually support fusion, otherwise fall back and handle ops one by one
… maybe that'd be a good compromise between XLA and ONNX, allow retaining context to allow optimizations
daniel_smilkov: that's a great point, just for a full disclosure, I did not comment on batch normalization
… I like the label idea that gives a safe fallback
… no need for complex heuristics
… I looked at ONNX and other high-level op sets
Chai: not obvious at authoring time, what kind of hardware would run the model, even with the composite idea there's a chance that compiler walks the graphs and can do a second level fusion, on the heuristics side of the compiler
Paul: I also like the idea of labels
… having heuristics is so difficult, we've learned that hard way
… on the path of labels, they probably need to be structured, have some semantics with them, the bottom knows what's on top
<pingyu> From tooling point of view, MLIR has a multi-level representation of ML graph. If WebML can be defined as a dialect, the composite can be presented as function of low level HLO ops
Chai: for the label, we need to define what the label means, it cannot be arbitrary
… need to define what the subgraph is, no need to define what to do if we fail, the graph is the source of truth, label is just a hint for fusion
<pingyu> TFLite uses labeling before for fusing but it has been bit hacky.
Action: Chai to open an issue for the label hint for defining semantics for op fusion
Action: Chai to open an issue for the label hint for defining semantics for op fusion
ningxin_hu: thanks for discussion
ningxin_hu: regarding the PR, daniel commented on particular ops, how about we go forward with elementwise add and multiply?
… concat and reshape also well supported
… these primitives supported in both XLA and ONNX
PROPOSED RESOLUTION: Add elementwise add, elementwise multiply, concatenation, and reshape ops to the WebNN API
<daniel_smilkov> looks good
ningxin_hu: LGTM
<Chai> LGTM
Resolution: Add elementwise add, elementwise multiply, concatenation, and reshape ops to the WebNN API
paul_mcdaniel_msft: for the ntersection between XLA & ONNX, I have something but maybe better for me to share those on our next call?
anssik: SGTM
<Jonathan> @paul, at google we can open xls files :)
<paul_mcdaniel_msft> :)
<paul_mcdaniel_msft> lol
<Jonathan> interop ftw
anssik: are we ready to merge your PR?
ningxin_hu: OK to keep this PR open and iterate on it a bit, then come back with a revised version
Model Loader API
anssik: Review spec strawman, please open issues for your feedback
<Rama> Elementwise ops can also benefit by introducing a single higher-order op "elementwise"
anssik: thanks Jonathan for volunteering to edit the spec
<Rama> which takes a scalar function f as a parameter
anssik: explainer is in a good shape, documents use cases, FAQ, discusses design considerations
<ningxin_hu> @Rama, SGTM
anssik: this repo to be used to iterate on the API and evolve examples with it
… repo welcomes issues and PRs
… Jonathan, do you have some open questions in mind you'd like to ask the group for feedback? (sees zero open issues)
Jonathan: thanks Ningxin for proposing extending ML namespace
… first thing to do is to review the spec examples section
… when first drafted, it was pseudo code
… especially interested from folks how to improve it
… another update, I've talked to Googlers working on MLIR, and we're trying to figure out how to get engineering staffing behind this
<paul_mcdaniel_msft> welcome ping !!
<pingyu> thanks
Jonathan: discussed with Android and TF teams as well
<ningxin_hu> welcome ping!
anssik: pingyu welcome!
pingyu: working with Nikhil on TF.js
… looking forward to contributing to this group
Nikhil: excited to have pingyu in this group, a lot of experience with MLIR
Nikhil: with Daniel we're moving from TF.js to another project in around 6 months
daniel_smilkov: nothing changes in TF.js we've already reduced our direct involvement in TF.js over the last months so no slowdowns for that project
Jonathan: for next steps, need to have discussion in GH on engineering details, as Ping ramps up, my hope is he can coordinate with Ningxin
<pingyu> Jonathan, thanks for the suggestion
ningxin_hu: SGTM, happy to work with Ping on that
Jonathan: also the prototype, but also the future of how this should look like
Virtual workshop
anssik: wanted to discuss virtual workshop prepwork status
… on our last call heard interest for a session to discuss XLA and MLIR
… I created a "Domain-specific compilers for ML" high-level topic to discuss these topics, if not accurate name help me perfect it
Nikhil: high-level topic name looks good to me
… I can poke people for lightning talks
Review workshop agenda details
anssik: survey maps 1:1 to the proposed agenda
… asks to rate high-level topics on scale 1-5 from not important to very important
… the idea is to send to through your personal networks to gather feedback on scoping. Couple of open questions at the end.
… The Program Committee would build an agenda based on its assessments of what topics would benefit to be covered (starting from https://bit.ly/webml-workshop-talks).
… To gather feedback from the community on the workshop scoping and focus areas, we prepare a survey (starting from https://bit.ly/webml-workshop-survey) to be sent to the prospective participants.
anssik: does this survey look good to you? would you respond? too long, too short, asking wrong questions? LGTM?
anssik: we'll discuss this on the workshop program committee call right after this meeting, you're feedback is extremely valuable.
<Jonathan> Sure, happy to introduce the model-loader API