<scribe> scribenick: kaz

First hour

McCool: my OCF device is still down
... but want to talk about speech device over google hangout
... would work on OCF devices as well
... merged Oracle's update
... now 6 vendors

draft implementation report (local file on McCool's PC is newer than this online version)

McCool: consistent terminology
... generator or producer
... this (Node-WoT with Oracle Binding) is interesting

<McCool> mlagally, are you on mute? we can't hear you

<McCool> https://github.com/mmccool/wot-thing-description/blob/updated-test-results/testing/report.html

McCool: wanted characterize all the implementations
... consumer/producer
... this (Device Model Converter) is both consumer/producer

Lagally: yes
... Oracle Digital Twin Simulator exposes TDs

McCool: goes through Oracle's implementations
... btw, this "Oracle Asset Monitoring" doesn't produce or consume TDs, do it?
... a couple of things to restructure the interoperability table

Lagally: it's not a simulator
... integration via node-wot
... connecting TDs to devices

McCool: we should be clear about whether double counting or not

Lagally: count based on code-base?

McCool: we count systems with mostly independent code bases

Lagally: to be clear, code base of the asset management is significantly bigger
... so would say it's a separate implementation

McCool: also we should be clear which each implementation is, producer or consumer
... moving on

Lagally: McCool, please create a PR with the proposed changes to the Oracle input

McCool: unfortunately, Ege is not here
... but used my time for playground as well

<McCool> https://github.com/w3c/wot-thing-description/issues/321

McCool: issue 321
... problem with assertion stated
... TD arrays
... required and enum
... created an issue for TD
... also
... another problem
... issue 322

<McCool> https://github.com/w3c/wot-thing-description/issues/322

McCool: readOnly and writeOnly
... are optional
... but for JSON Schema, they're required
... if you look at the testfest-1 TD version
... writeOnly is mandatory
... this version has writable and writeOnly
... so there are mistakes
... we have a decision to make
... modify the schema to be up-to-date with the testfest-1 tagged version?

Lagally: would advocate fixing the schema

McCool: would agree
... encourage people to update the schema
... would take an action to fix it

<McCool> https://github.com/mmccool/thingweb-playground/blob/dec-2018-testfest/WebContent/td-schema.json

McCool: remove writable and make readOnly/writeOnly not required

Lagally: one more thing
... on the assertions
... version in Ege's assertion is mandatory

McCool: there are bunch of versions
... optional in the tagged version TD
... that's good
... the Assertions
... we have a problem
... 3 outcomes
... pass/fail/not-implemented
... but how to handle those 3 options?
... version is good example
... property object
... there is another properties tag here
... type of the members properties and items MUST be serialized as a JSON bject
... exist and has to have property
... the problem is
... in my test cases
... problem with camera
... bunch of properties
... simple integer type
... for brightness
... actually has sub property but not implemented

<McCool> https://github.com/w3c/wot/blob/master/testfest/2018-12-online/TDs/Intel/SimpleWebCamera.jsonld

McCool: one of my properties
... updated schema

<McCool> https://github.com/w3c/wot/blob/master/testfest/2018-12-online/TDs/Intel/BetterWebCamera.jsonld

McCool: we now have security definition
... list of values
... and writable is wrong
... to be fixed
... did in contrast
... mixture of properties
... some of them have types
... JSON rule says property expressed as object is required
... only some of the properties are expressed
... properties sub field
... this TD should be a PASS
... but it turns to be fail or not-impl
... was scratching my head...
... work around is
... (dec-2018... branch)
... TD validator for JSON Schema
... special option called $comment
... callback function with this value
... particular rule match
... for schema
... I can write a rule
... if it matches and valid, it's pass
... if it matches but invalid, it's fail
... good news is can tell how to fix
... bad news is bunch of things to do

<McCool> https://github.com/epoberezkin/ajv

<McCool> https://github.com/epoberezkin/ajv#options

Kaz: sorry but have started to wonder if it would make sense to try manual test first, and then this automatic testing mechanism as the second stage

McCool: kind of agree
... but
... we should have some automatic tooling mechanism for the next testfest

Kaz: yeah

McCool: technically each implementer can provide implementation report
... manually creating the CSV input is one possible option
... do we have any gaps?

Kaz: you've already found one possible solution, so we can try that at some point
... but maybe we should start manual check first

McCool: yeah
... can try to solve the problem by Friday
... worth while trying both approaches
... manually filling the CSV files and tooling

<McCool> https://github.com/mmccool/wot-thing-description/blob/updated-test-results/testing/inputs/results/template.csv

McCool: basically I need a CSV file for each implementation
... that's kind of homework for people

Lagally: wondering about a couple of TDs by others
... do we expect the other companies' TDs as well?

McCool: Siemens is expected
... Hitachi does TD consumer only
... don't know about Dave's implementation
... my main concern is
... we may not have gaps
... do we have 2 implementations for security features?
... might show up down the vocabulary
... may see 0 here

Lagally: what is the implication to have only one implementation?

Kaz: if we can get only one implementation for some features, we need to drop the features
... and we have to republish a Candidate Rec after dropping them
... but we can identify some of the difficult features as "features at risk" when we transtion to CR
... and we can drop those "features at risk" and transition to PR without going back to CR

Lagally: how are optional features treated? What happens if they are not implemented at all?

Kaz: the trend is "even optional features should have 2 or more implementations"

McCool: also wondering about default values

Kaz: btw, the 2nd assertion "property, action, event MUST have..." should be split into 3 separate assertions, e.g., "property MUST have...", "action MUST have..." and "event MUST have..."

McCool: also thinking about that point
... using child assertions
... can create some extra assertions
... and consolidate all the child assertions

Lagally: why don't we define the combined assertions separately in the spec and generate the tables

McCool: also can do so

Lagally: that might be easier

Kaz: that said, the more important at the moment is checking the coverage of features
... so that we can tell which features are possibly at risk

McCool: right
... at the moment, people should manually generate the CSV reports

Lagally: the CSV format would work even after the TD spec is updated?

McCool: as long as the feature ID doesn't change
... we need example and counter example as well
... these assertions should not change much
... btw, I went through the test spec at the appendix as well
... need to go through all the details and see counter examples
... parent vs sub assertions
... the parent assertion is true if all the child assertions are true
... need a bit more work
... a few assertions are still vague
... confusing assertions to be away

Kaz: we should ask the others, Koster, Sebastian, Toumura-san, etc., to provide their TDs and reports

McCool: recap would be better

Koster: would like to recap

Sebastian: yeah
... also would like to talk about the f2f in Princeton

McCool: date of f2f and IG Charter as well
... Kaz and I had discussion

<McCool> https://w3c.github.io/wot/charters/wot-ig-2019.html

McCool: we should send out a Call for Consensus message to the whole IG for the updated draft IG Charter
... we should wrap up the discussion by Friday
... regarding the f2f dates https://doodle.com/poll/5d4gbht3wm9v66dy
... result here
... 2 weeks
... 2 opposing conflicts
... Dave's conflict is he'll miss Thursday
... not sure about Taki
... or Sebastian

Sebastian: definitely OK
... just in favor with later meeting
... prefer latter one

McCool: yeah
... maybe Matthias also has difficulty with the earlier option
... more preparation would be better
... Taki, what is your trouble?

Taki: north carolina for IIC

McCool: Taki can't make the latter option at all
... Dave just has conflict with Thursday

Lagally: what about Matthias?

McCool: in the worst case, he can't make either of the options
... given Taki's situation, my reincarnation is the first option

Sebastian: let's make the decision by the end of this week

<McCool> 28 Jan - 2 Feb 2019

McCool: ok
... we'll finalize the date by the end of this week
... should send a message for CfC out
... should nail down by Friday

Second hour

McCool: recap

Koster: most of the test cases are validating TD

<McCool> https://github.com/w3c/wot/blob/master/testing/criteria.md

McCool: either TD producer or a TD consumer
... you have a device on the network and have a TD which describes the device
... automatically generated TD or manually generated TD
... proxies take a TD and possibly change it
... somewhere here (in the report.html)
... we have to count distinct implementations
... and what is "distinct implementations"?

<McCool> https://github.com/mmccool/wot-thing-description/tree/updated-test-results/testing

McCool: e.g., even though there are three implementation by Intel for security features, they're based on the same code-base (node-wot), so should be counted as one implementation
... we have 6 organizations at the implementations/ subdirectory
... would have the other implementers as well
... (and then shows optional interop/ area)
... arcs within the connection diagram should be described here
... producer/consumer pairs

Kaz: yeah, so at some point, we should split that interop part
... and create a separate WG Note

McCool: yeah, maybe an interoperability Note
... (and then shows the implementation result CSV files)

https://github.com/mmccool/wot-thing-description/tree/updated-test-results/testing/inputs/results

Kaz summrizes what we should do:
1. submit TDs at https://github.com/w3c/wot/tree/master/testfest/2018-12-online/TDs
2. submit CSV-based reports: https://github.com/mmccool/wot-thing-description/tree/updated-test-results/testing/inputs/results
3. implementation descriptin: https://github.com/mmccool/wot-thing-description/tree/updated-test-results/testing/inputs/implementations

McCool: that is kind of described within the manual at: https://github.com/w3c/wot/tree/master/testfest/2018-12-online
... but the automatic tool is not available yet

Koster: ok
... the CSV file is the result of assertions

McCool: (updates the testfest manual)

https://github.com/w3c/wot/tree/master/testfest/2018-12-online

McCool: not-impl means no example
... null will be ignored

Koster: question about Ege's tool
... using npm
... AssertionTester

McCool: there are 2 tools
... playground validator
... td-schema.json is almost updated for the testfest-1 TD version

<McCool> https://github.com/mmccool/thingweb-playground/tree/dec-2018-testfest/WebContent

McCool: we need to get it updated
... have a PR
... Ege also has a PR
... waiting for those PRs to be merged
... playground has web interface and command interface
... and then AssertionTester
... ajv schema validator
... some of my assertions here
... (McCool revisits his assertions)
... (and explains the issue with handling pass/fail/not-impl)

Koster: TD schema with ajv hooks

Sebastian: need to leave
... wondering about the goal of the testfest this week

Kaz: thinks the goal of this first testfest effort is simply looking the implementability of each feature, i.e., can get one or more implementation for each feature (rather than 2 or more)

McCool: would add some more extra assertions
... if you would like to split some of them, we can do so
... this is a snapshop as of today
... if node-wot is to be updated, that's fine

Sebastian: another question about TD slot on Friday
... will we have a regular TD meeting?

McCool: other than a few issues, you can use the rest for the TD discussion

Sebastian: we have to make decision about some of the TD issues

Kaz: using the first 30m for testfest wrap-up
... and using the remaining 1.5h for TD discussion
... is that ok?

Sebastian: ok

Toru: going back to the interop test
... pass/fail/secure, etc.
... is it possible to handle partial result?

McCool: (visits interop/ area)
... right now we have producer, consumer, security, status
... you can mention two rows and merge them
... hundred passes and one failure would be failure
... purpose of the summary table
... maybe more information to be captured
... you can add more columns
... my tool will ignore the information
... e.g., you can add columns like cause, comment, etc.
... some failure may be caused by "authentication invalid" with "expired certificate"
... we can keep track those columns

Toru: maybe just one additional comment column would work

McCool: ok
... probably will become an interoperability document

Lagally: one suggestion
... overview document
... private repos
... very confusing
... and Ege not on the call

(Ege just joined)

McCool: Ege, have you checked my PR?
... Ege's PR is the main one
... and I also have a PR

Ege: now I'm free to work on that

McCool: discovered bunch of issues
... fundamental issue

Kaz: would suggest we resolve that issue via the PR/Issue since we've already discussed during this call twice :)

Lagally: which of the repos to have the latest tools, schemas, etc.?

McCool: here is the suggestion
... nothing is in a good shape at the moment...
... we need some more time to stabilize the situation
... (summarize the issue with schema)

Ege: updated the schema Monday evening

McCool: Ege will update the JSON Schema
... will also add much bashing
... within the next two days

Lagally: thingweb resources also on ege's private repo

Ege: will merge it but still need to check

https://github.com/w3c/wot/tree/master/testfest/2018-12-online

Kaz: so we'd like to use only the wot official repo for our testfest (if possible)
... note that TD files and CSV reports are already to be copied under w3c/wot repo
... and so I was wondering about the implementation description HTML

McCool: that can be also copied under the official w3c repo

<McCool> https://github.com/w3c/wot/blob/master/testfest/2018-12-online/implementations/README.md

McCool: people can install their implementation description HTML as well to the above official area
... and please submit your generated CSV report under: https://github.com/w3c/wot/tree/master/testfest/2018-12-online/results
... each organization should create a subdirectory there
... and copy your generated CSV files to that subdirectory

Kaz: is it OK to start with this manual approach on the w3c/wot repo?

Lagally: ok
... so we're encouraged to copy the implementation description HTML as well to the w3c/wot repo

McCool: yes
... can copy a template for your help
... and I'll copy all the resources to my mmccool repo to regenerate the report.html

Lagally: ok

Taki: particular assertion, e.g., property, action and event
... not sure we really should split it

McCool: yeah
... can have one combined assertion as a parent with several child assertions
... can just generate them based on the current TD

Taki: can also have some discussion on Friday

McCool: yeah

Ege: working on the schema fix

McCool: taki, maybe you can merge Ege's PR

<McCool> https://github.com/w3c/wot-thing-description/pull/323

Taki: issue on the JSON Schema

McCool: yeah
... writable
... ah, even older which just include writable
... another PR

https://github.com/w3c/wot-thing-description/pull/324

Taki: can merge it

McCool: anything else for today?

(none)

McCool: please work on your CSV files
... will continue to work on the automation
... Ege, we need further collaboration

[the official testfest call adjourned]