<kaz> scribenick: McCool

Agenda

two topics: plugfest prep and testing

McCool: I suggest we do Fredrico's pres on testing first, so he can leave if he has to

F-interop, by Federico

Federico: presentation, then a little demonstration on what is being done for CoAP
... WoT may be different, but it will give you an idea
... F-Interop is an H2020 project
... almost done, Oct 2018, but tools will be maintained
... main objectives is to develop and provide online testing tools
... including interoperabilty testing
... members into INRIA, ETSI, etc.
... currently, SoA is F2F events ("plugfests")
... but F2F plugfests have issues...
... short, if you hit a bug, can wreck entire event
... so F-interop wants to provide online and remote process
... so can do continuous testing
... system description
... runs in cloud, can connect both through Web GUI and to IUT (Item under test)
... can also have simulated IUTs to connect to running in cloud
... various tools: communication, including tunneling (bypass NATs and Firewalls); coordinating tests based on test descriptions; traffic sniffing; dissecting messages; traffic analysis
... results -> issue PASS/FAIL/INCONCLUSIVE based on test description
... there is a tutorial available online that you can look at
... initially looked at a couple of IoT standards: CoAP and 6TiSCH
... started with ETSI plugfest test descriptions
... now looking at extensions: oneM2M, OMA LwM2M, 6LoWPAN, and... hopefully WoT
... Ex: CoAP test case (test spec)
... did not make these up, used ETSI test specs
... gives objectives, configuration, pretest condition, then test sequence
... in this case, test sequence starts with a stimulus, then a set of checks on responses
... so what are WoT test cases?
... could not find anything similar to CoAP
... would be nice to have things more formal

Ege: looks a bit like invoke simple action
... but autogenerated from TD
... can generate payload from description
... can send payload and check response
... for example, can check that output satisfies schema

Federico: that makes sense...
... but not predefining coverage, looking at capabilities of device, making sure those work

Ege: test description basically is given by the TD

Federico: nice approach, but at the end of the day doing the same kind of tests that were in the BJ plugfest
... but a more advanced automation

McCool: time check, only 5-10m more

Federico: ok, I'd like to take the time to do a demo
... can access for free

https://go.f-interop.eu

Federico: can do both single user and user-to-user tests
... or can use a reference-based test
... pick device, and test case
... there is also a playground to launch requests without a predefined test
... start it, deploys instance in Docker containers and runs tests
... need to set up environment, communication, etc.
... communication can include a virtual network
... including support for UDP
... based on very formal set of test cases
... at the end get a report

scribenick: kaz

McCool: we should clarify which part is opensource and which is not

scribenick: McCool

McCool: also, I think a small team (Ege, Federico, Dave) should go off and put together a small proof-of-concept using just the OSS F-Interop tools and then report back

scribenick: kaz

Kaz: agree we should clarify which is open and which is not.
... also we should think about levels of testing again, and consider which level/part is this work (F-Interop) should be applied
... regarding the smaller call, would suggest at least all the co-Chairs and the staff contacts should be included

scribenick: McCool

McCool: I was thinking a working meeting with just the implementors, but let's discuss via email

Koster: we should get in the habit of publishing an agenda
... so people who are just interested in the pf, for instance, know when to join

PlugFest prep

Koster: did prepare one slide as a discussion point

<ryuichi> https://github.com/w3c/wot/blob/master/plugfest/2018-bundang/preparation.md

Matsukura: want to first explain preparation.md structure
... sections: first is intro, info from last pf
... second section is new information
... 2.1 is a table for participants to fill out
... please insert your information in this table
... 2.2 is checking points
... these should also be discussed in the results.md for your report-out
... can follow this template and carry information forward
... then 2.3 are other issues
... if any suggestions to improve please make them
... section 4 are logistical requirements: network, power, etc.
... I also made an example, fujitsu-preparation.md
... information is the same as last plugfest

Koster: table is useful, people can just add a row
... checkpoints are also useful for results.md
... but not everyone used the same template
... or maybe it was
... but edited
... section 2.3, other issues... my material
... kind of a mixture of different things; experimental features, validation, binding templates, services, etc.
... this section will go away/be integrated

McCool: I think we should prioritize, e.g. security should be mandatory
... also, "other" maybe should be broken into "extras" and "experimental features". Latter are things that are candidates for inclusion in the standard but need so experimentation

Sebastian: I think we need fewer "demos" and more "plugging"
... we really need to check how things can work together
... that would really help to improve the standard
... we need to actually focus on testing interoperability and the standard
... would like to see more "client" implementations
... a lot of devices, not enough systems trying to use other devices

Koster: agree. we also need more formal test plans and better record keeping
... agree also that we need more applications
... we are building a system, we need to have all the components present
... need so see more interactions

Sebastian: rather than bringing a lot of things, we should focus on the content and interactions
... we should concentrate on the functionality

Federico: table of features that implementations should implement
... at the end of the day, pulling tables to see what features are being tested

Koster: when you have a set of test cases, it can stand in for a standard application
... but we also need some ad-hoc applications to test integration
... and use cases
... for example, using semantic search to find a bunch of sensors and analyse them for some purpose
... in other words, system-level applications
... first cover functional, then cover system
... functional: list features you want to test, and then generate tests against them
... then, assuming functional tests done, how do we orchestrate them to solve a higher-level problem
... we don't really know what "interoperability" is yet

McCool: I think the second part is more like "what is it good for", eg. use case analysis
... functional testing is just "does the implementation match the spec"
... system testing is "does the spec have the right things"

scribenick: kaz

Kaz: was on the queue and wanted to mention something like McCool :)
... probably we should have a combined demo scenario on the main preparation.md, and clarify what kind of use cases/features are included in that. And then clarify details in the company-preparation.md and test/report if it works after the plugfest

scribenick: McCool

Koster: so maybe we need to think about to structure this in the preparation.md file
... would still like to do system-level scenario exploration
... but should think about how to structure it better

Matsukura: adding a "use case" section might be useful
... added, please insert your idea

Koster: my presentation... more of the same
... we are building a system, e.g. an application that orchestrates multiple things
... but also have components and roles

Koster: to test components need to have them in context, eg. in a system (application)
... several components are not servients...
... or are they?
... Servients use the scripting API, interacts with Thing Directory
... if it only exposes a TD, e.g. is an endpoint device, then it's not a servient
... system architecture includes both local and remote parts
... I myself am going to bring fewer things but focus on applications
... but applications have a bunch of common components and patterns, e.g. local/remote directories, proxies, etc.
... also some new concepts, like "Bridge Servient"; can also think of as a "role"
... also notable is that proxy-proxy connection doesn't matter
... but need to go through the proxy to get to local devices from the cloud
... other way does not necessarily need the proxy
... want to list the patterns separately
... but would like to take an application-layer approach
... register things locally
... tell the proxy I want to make available
... can use the information in the TD to figure out how to implement
... missing an arrow here, some extra orchestration

McCool: noticed that the directory and the proxy are paired
... and also notable is that TD may have to be modified in various ways before being registered in the could TDir
... could be the other way around: could register in the TDir, then proxy can pick it up

Koster: another point: we want to think about applications and how to deploy them
... I am thinking about using node-wot to deploy applications
... what I suggest we start thinking about
... we need goals that are bigger than just turning a light on
... what are the higher-level problems we are trying to solve, and what are their requirements?
... for example, what are the security requirements

Sebastian: don't think we need to spend so much time on the application
... I am thinking more that we just do more 1:1 testing
... and try to use all the interaction patterns
... we could go further and do mash-ups
... even that simple approach exposes many issues

Koster: I was thinking about how to accomplish what you say...
... can create a control panel in a client that lets you connect to all the

McCool: could be a simple tool that generates a web dashboard given a TD

Kaz: can clarify use cases, brief, in preparation.md
... so people can understand the "story"

Sebastian: if you recall in BJ
... we tried to identified what worked well and what didn't
... I am a little worried about our timeline
... we only have a couple more plugfests to get experience with TD in practice
... then we are supposed to be done
... not convinced we are testing things completely enough
... we need to at least do all the basic component tests

scribenick: kaz

Kaz: would agree with your concern. however, each use case doesn't have to be very fancy but can be something simple, and the main preparation use case scenario simply refer to them and clarify concrete demo scenarios. sometimes in parallel and sometimes sequentially using dashboard, etc.

scribenick: McCool

Koster: this is definitely a different level
... we also need the basic testing at the plugfest
... but... the low-level testing is obviously a higher priority
... although we also need to move forward with system testing too

Toru: do we really need scripting API?

McCool: my opinion is that the minimum is that it "consumes" a TD
... if it only exposes a TD it's an endpoint device

Koster: I do want to motivate using the TDir as well
... but, yeah...

<Zakim> kaz, you wanted to ask about the availability of Koster's slides :)

Matsukura: our implementation doesn't use scripting API, so... but we need to better focus on "applications"
... our implementation doesn't use scripting API, though

Kaz: can you make slides available?

Koster's slides

Lagally: suggestion
... suggest to add a legend to the architecture slides, to be able to understand the meaning of the colours of the boxes

Koster: sure

Matsukura-san: AOB?

ok, adjourn