<McCool> https://www.w3.org/WoT/IG/wiki/PlugFest_WebConf#Agenda_21.11.2018

<scribe> scribenick: mkovatsc

Update on TD Test Plan

<kaz> PR 290

McCool: new section on what counts as implementation
... added implementation section for Intel
... mk added implementation section for Siemens
... need to figure out how implementation is counted when a sw component is shared

<McCool> https://github.com/mmccool/wot-thing-description/tree/updated-test-results/testing

McCool: now using terms "TD Consumer" and "TD Producer"
... added several CSV files
... changes to the document need to go into template.html
... it imports other files
... implementations folder contains HTML blocks for each Member, listing and describing their implementations and a leading testimonial
... please give each implementation a unique id ("impl-<member>-<impl>")

<Zakim> kaz, you wanted to mention we should identify if each implementation is based on different code-base

Kaz: two points:
... 1. implementation should relate to specific code-base
... 2. should we really use this repo for this?

McCool: code-base aspect is known
... my repo is only used to speed up things (merge into master requires review)
... testing focuses on TD, thus it is in TD repo

Kaz: you could become moderator for TD repo to speed up

McCool: it is more about authorization by the Editors, as spec was touched by PR
... best we wait until PR merge before people open PRs for impl information
... best start by retrieving the template from my repo and make PR to master

Toru: "TD Producer" has the notion of an automatic generator. What about manual TDs?

<kaz> Matthias: we're learning more and more about the actual procedure now

<kaz> ... the current assertion focused on whether it's correct or not

<kaz> ... need to see proper terms

<inserted> kaz: During the WoT Chairs call today, I also mentioned "consumedThing" and "exposedThing" as possible terms, but we can revisit the concrete terminology later.
... The point at the moment is the terminology for the Implementation Report should be consistent with the definition within the WoT Architecture.

McCool: I understand my implementation with a manual TD still a unique code base and an implementation that produces a TD

Lagally: how stable is the tooling?

McCool: changes are planned to be backward-compatible
... overall the testing plan is still "in flux", as we need more information
... plan is to stabalize in two weeks for the online TestFest
... data files will not change much, assertions will need to change

Lagally: How can I play with this to check the resulting document for correctness?

McCool: adapt data files. script is supposed to pull them in automatically at some point.

Lagally: it is confusing that there are multiple testing directories

Matthias: note that w3c/wot" repo is IG!

McCool: IG material is general planning. Deliverable repos have concrete test documents for the REC process.

Lagally: is it a valid assumption that w3c/wot is not required for REC process?

McCool: yes
... using CSV for test results as they are easy to edit and render nicely on GitHub
... CSVs can be broken down in multiple files, as each entry uses an ID
... tool will colorcode the results
... red is critical, e.g., only one implementation for security
... prefix your test results with your member identifier (e.g., "intel-")

Ege: having reports per assertions will require to change the TD Playground, so that each assertion is in its own JSON Schema
... Playground can only check syntax

Matthias: statements during today's chairs call said that tests must focus on behavior, e.g., if a parser must ignore unknown terms or must throw an error
... need to clarify what kind of assertions are actually required from us

Kaz: I don't think we confirmed that during the chairs call today.
... the W3C process document requires us to show evidence for implementability

McCool: there is a file "extra-asserts.html" to file new/changes assertions
... @all, please have a look at your implementations in the context of testing and report to us what you have
... need to think about interop and penetration testing and how it fits into the report
... (something about two-stage process for the test plan impl)

Kaz: As I (repeatedly) mentioned, CR exit criteria defined by the W3C Process is showing that a specification is sufficiently clear, complete, and relevant.
... On the other hand, I can understand people are interested in testing interoperability between various implementations itself and describing concrete behavior of each implementation (e.g., TD consumer/TD producer).
... That's why I'm planning to talk with W3M to confirm what is required from us

McCool: charter states security testing
... please clarify what we need to provide

Ege: isn't interoperability shown when both client and server fulfills the assertions?

McCool: interoperability would go as report into appendix
... AoB?

Toru: Could you please check my PR?

<inserted> wot PR 596

McCool: merging
... EoM