<kaz> scribenick: dape

Agenda

<kaz> Agenda: https://www.w3.org/WoT/IG/wiki/Test_WebConf#Agenda

McCool: Any additions to the agenda?
... we have plan
... 1. step, markup spec with normative assertions

<kaz> plan

McCool: RFC 2119 keywords
... anyone started with normative assertions ?

Kaz: think Sebastian and Zoltan started with it...

McCool: Section "Conformance" needs to be added
... particular words MAY; MUST, ... etc need to be used

<kaz> scripting api draft

Kaz: Zoltan added conformance section as Section 9

<kaz> section 9

McCool: will be protocol binding separate document till to the end?

Koster: plan to provide normative statements for TD

McCool: for security it is the same
... assume normative statements are a big chunk

Categories of "testing"

McCool: 1. specification validation (required by W3C)
... spec implementable and consistent
... 2. Implementation testing
... check implementation satisfy requirements
... a) scripting API
... b) TD validation
... general useful tool
... c) network interface described by TD
... would like to have an automated thing that tests this
... wonder which is actually necessary

Koster: do they need to be reference implementation

kaz: usually we don't have "reference" implementations
... just need 2 implementations for each feature

McCool: 3. category is interop testing
... difficult problem

Reference implementations?

Koster: we can have reference implementation
... reference implementation helps to get started

McCool: Could be online tool

Koster: PlugFest could be venue

Kaz: usual W3C specifications are related to Web browsers
... so the W3C Testing Framework mainly handles Web browser testing
... on the other hand, we need server side also for WoT
... that's why, reference impl for server side is useful

Levels of "testing"

Daniel: Wonder whether w3c testing only means spec validation ?

Kaz: the main purpose of "testing" for Candidate Recommendation transition is spec validation

McCool: we can add levels
... and we missed security testing

Koster: there is our spec we test against but there is more (e.g., in security)

McCool: we can just have recommendation
... provide framework how to test

Kaz: maybe we might want to rename "security testing" to "advanced testing" which include security testing

McCool: I believe we have statement in charter that requires us to test security
... in our tooling we can point to how to improve security

Kaz: ah, that's true

<kaz> WG Charter, fyi
[[
In order to enhance the security of WoT systems, we will also generate and implement a security testing plan which will include both functional and adversarial testing of the proposed standards and their implementations. We will only recommend an implementation of the proposed standards for use in production once it has passed such testing.
]]

McCool: I think we need validation tool
... "functional testing" could be basic authentication ...
... hope we can reuse things that already exists

Matthias: w.r.t. CoAP there is some penetration testing
... for CoAP we broke it down by feature
... test it pair-wise
... also test it with security enabled (difficult and maybe also impossible with all combinations)

McCool: test common combination
... w.r.t. reference implementation
... we need 2 separate implementations
... for example in the case of scripting we might just have a limited number
... need to focus breaking things down by feature
... after that we can create table with common combinations
... maybe PlugFest could be used to implement test cases

Koster: in physical realization: we need system under test and test tool
... can have scripts for exposed thing and consumed thing using node-wot

McCool: for PlugFest (or after) I would like to have a skeleton testing framework in place
... implies that before the PlugFest we need test description (before that test assertions)

Koster: need to talk about how to implement
... what new tools we need

Actions

McCool: Next, will create section on wiki for "Actions"
... normative assertions
... test descriptions
... each group should think about this
... we might want to have MD template

Testing manager?

McCool: Q: Is there any test expert who could help us/me?

Matthias: mainly time issue
... talked to Ege Korkan
... he could volunteer
... will ask him

Time slot for the testing call?

Koster: should also re-consider time slot

McCool: Kaz, can you do a doodle?

Kaz: will do
... BTW, we might want to think about collaboration between PlugFest and Testing
... PlugFest call should start again
... possibly done within this testing call, but maybe need a separate call

<kaz> Koster: right. plugfest has a bit different logistics cycle

<kaz> [adjourned]