<McCool> https://www.w3.org/WoT/IG/wiki/PlugFest_WebConf#Agenda_10.12.2018_-_Test_Fest_Day_1

<kaz> scribenick: taki

TestFest logistics

<kaz> https://github.com/w3c/wot/tree/master/testfest/2018-12-online

McCool explaining logistics...

McCool: testfest-1 tagged version is used today.
... There is diff between the published version (oct. 21) and the testfest version (Nov. 29) also.
... Wednesday, we will use Editors call WebEx.
... Google hangout is ready. I and Panasonic are in there.
... Ege created tools using JSON Schema to validate implementations.

McCool: Test involving validating TDs.
... I documented process.
... Three repos. Ege's playground repo. TD repo where test tools are in, and WoT repo where we collect results.

<ege> https://github.com/egekorkan/thingweb-playground/tree/assertionTest is the repo and branch of the assertion tester

McCool: Run Ege's tool on TD, results goes into result directory.
... Copy results into testfest directory.
... Copy results to csv file.
... Merge multiple results for a single implementation.
... Copy all results into WoT directory.
... Intel has several TDs today. README is available.
... Panasonic is on google hangout. Described in README.
... Intel has initial results. generated by Ege's tools.
... Ege's tool currently has three tests.

Lagally: There is also Oracle TD.
... instance has pump instance.
... also a few simulators.
... New server setup. New URL and TDs.
... same credentials, just on different machine.

McCool: Please fill out result directory, and use Ege's tool.

McCool is explaining how to use Ege's tool....

McCool: It can pass, fail, or not-implemented.

Kaz: Which result goes into which repository/directory?

McCool: Put the real result in WoT/testfest/2018-12-online/results.

<kaz> https://github.com/w3c/wot/tree/master/testfest/2018-12-online/results

McCool: Raw results should be merged using scripts.

<kaz> everybody is encouraged to submit their reports to the above master repo under their own sub directory (after creating those sub directory, e.g., "Intel"

McCool: For example, Intel/intel-ocf.csv.
... For certain tests, we need to do manual test.
... testspec.html in TD repo lists assertions and how to test them.
... We need to figure out which ones we can use JSON-Schema.
... We can generate new tests.
... ERCIM, Intel, Panasonic, Siemens implementation descriptions are available.

Lagally: I will provide Oracle's implementation description.

McCool: Your entire simulator is one implementation, right?
... Fujitsu and Hitachi, please write them up.

McCool is showing report...

McCool: I am going to update the result in the report with real result.
... Color red indicates feature at risk.
... As we finish test, we can convert them into green.
... Test Specification is in Appendix A.

Ege: we have version, action-objects and additional-context tests.

McCool: Mark everything in red for ones we do not have test yet.
... For security binding test, we need to do manual HTTP test.

Lagally: Can you show us one whole process for validation test?

Ege: I am going to show.

Ege is preparing for that...

McCool: In the meantime, I am going to update the report.

Ege is showing screen...

Ege: Playground repo. AssertionTester/Assertions folder has JSON-Schemas.
... td-action-objects.json, and also for context and version. There are three JSON-Schemas now.
... Run each with "node".
... It generates result.csv. Details are in result.json.

McCool: You can add lines in result.csv. Tools will ignore them.

Ege: @context is tricky to test.

Ege showing td-action-objects test result...

Ege: if output in TD is removed, it causes an error.

McCool: mergeresults.js merges multiple result csv into one.

https://github.com/w3c/wot/blob/master/testfest/2018-12-online/README.md

Ege: I have result.csv and result2.csv.

McCool: It takes two or more test results as input, and writes out to std output.

Ege is merging results...

Ege shows the merged results...

McCool: Multiple results for one implementation. If you have multiple implementations, keep them as separate results.

Lagally: playground still uses old syntax and old schema.

Ege: It is in plan to update.

Ege showing thingweb-playground.

Ege: I will update.

McCool: Let's take a 10 minutes break.
... We can do maintenance work during that.
... We will resume in 10 minutes.

<kaz_> [break]

TestFest reports and TD files

Updated report

McCool is showing report.

McCool: One assertion passed.
... Everything in pink, we need to test.
... For assertions that we do not have tests, I marked everything in red.
... How do we generate JSON-Schemas.

Ege: Write schemas if we can test them with schemas.

McCool: atrisk.csv contains the list.
... We can divide the work among people.
... Would that accelerate the work?
... We can create more schemas before Wednesday.

<McCool> https://github.com/mmccool/wot-thing-description/blob/updated-test-results/testing/inputs/atrisk.csv

McCool: I can split the work with Ege and work on those.

<McCool> https://raw.githubusercontent.com/mmccool/wot-thing-description/updated-test-results/testing/report.html

McCool: We should have a more fleshed-out set of tests.
... We will run those tests on Wednesday.

Lagally: there is a bug in setup instruction. In the main README in test fest.
... Section 2. Now I see it is fixed.

McCool: People should go off and try to work on this.
... For non-schema test, we will need to do network tests.
... security tests probably need network test.

Ege: We can also use testbench.

McCool: how can we check security assertions? e.g. basic-authentication.

Ege: Penetration test is another.

McCool: Elena sent me some instructions.
... Burp sends stimulus and gets response. need to check if it can do Authencation test.
... I will talk with Elena later.
... Adversarial testing is required by charter.
... This week, we will finish JSON-Schema based testing.
... at least for the current set of assertions.
... I can do more work before Wednesday.

Ege: I will update playground before Wednesday.

McCool: I will start from the the bottom of the list, Ege work from the top.

Toumura: Where should I upload files?

<McCool> https://github.com/mmccool/wot-thing-description/blob/updated-test-results/testing/inputs/atrisk.csv

<McCool> https://github.com/mmccool/wot-thing-description/tree/updated-test-results

McCool: Please make a PR against McCool's fork.

<McCool> please make sure to do it against the updated-test-results branch

<McCool> https://github.com/mmccool/wot-thing-description/tree/updated-test-results/testing/inputs/implementations

Toumura: I will upload files. Thank you.

<McCool> https://github.com/w3c/wot/tree/master/testfest/2018-12-online/TDs

McCool: Check TD in to the repo. WoT/testfest/2018-12-online/TDs
... My TD, e.g. security part needs to be updated to the new TD version.
... By Friday, we can fill in most of Result column in the report.

Lagally: Is the JSON-Schema up-to-date?

McCool: Date is wrong.
... It has some of the changes. Not sure if it is all up-to-date.

Lagally: Please make sure Ege's schema is consistent with the spec JSON-Schema.

Ege: Playground has batch test.
... We can run with valid TDs and invalid TDs for JSON-Schema debugging.
... It still does not response field.

McCool: missing response in JSON-schema. case-insensitive mediatype and content type.
... We will create PR to fix the JSON-schema by Friday.
... Let's meet on Wednesday. Starting Editor's call time (4am US Pacific, 7am US Eastern, noon UK, 1pm Europe, 8pm China, 9pm Japan).

Lagally: 1pm in Germany.

McCool: We may not need three hours.
... anything else?
... We can continue to use hangout.
... let's adjourn. I will give time back. Please continue to work.