Plan and expectation

McCool: we should talk about our plan/expectation first
... kaz previously mentioned test framework
... myself think about interoperability testing
... more focused consumer viewpoint
... conformance, etc.
... generally need to think about features
... what are the features and related assertions
... for the Helsinki meeting, we should have test framework
... should use this time for sorting out it
... would like to state the issues
... what kind of test would be appropriate, etc.

Sebastian: what do you mean by "framework"?
... set of tools?

McCool: tools and procedure
... various tools, things, etc.
... data to be used for multiple implementations
... interoperability testing
... formal output like CSV
... we're half of that
... also should think about behavioral testing

Kaz: right
... we should provide resources for non-WoT people to understand WoT standards and generate implementations

McCool: yes
... how we communicate with implementation reports, etc., as well

Sebastian: our assertion table should be a good starting point

McCool: yes
... don't think we have to have automated test
... but might need some additional APIs for testing
... how do we test everything?
... should clarify our plan

Kunihiko: many specs are in incubation phase
... we should investigate the gaps
... not only test/interoperability but also we can use plugfest for feasibility check

McCool: meaning we need some prototyping?

Kunihiko: right

McCool: should split into testing part and prototyping part
... if we have a test suite, people can test implementations
... and we can focus on gaps
... ambiguity, etc.
... generally speaking, we should hold plugfest/test calls regularly
... would have a separate repository
... also should clarify we use the same webex as the main call
... regarding conformance, we don't have any description about conformant implementations
... kind of marketing topic as well

Koster: some kind of online tools people can do something
... we always need to look at files
... but also have some idea of online things
... like eclipse foundation's tools

McCool: virtual devices with online urls
... so that we can play with them
... there is testing and kind of like examples and documentation
... but not well-organized yet
... we care about test coverage
... someone is doing multi-language system
... if we put WoT testing repo in one place
... might be cleaner to navigate people

Koster: developer journey will start
... how do I get involved?
... download tools to check conformance

McCool: good point
... what to be supported in each step of the procedure
... two steps, examples and how to see if it's correct
... different stages

Koster: a lot of hints for developers
... engage the hands-on side

McCool: one is prototyping to check new ideas
... and evangelism for new comers
... then testing for conformance check
... one event handler or multiple handlers?

Koster: those are questions I have as well

McCool: would be good if we could have a hackathon
... should have experts there
... probably should consider both training and prototyping

Koster: during OCF ones, they work together to validate test cases
... should be online?

McCool: for testing the test cases
... do we have same conception as the spec?
... ambiguity about the meaning
... f2f time is good for validation
... pragmatically what would be our actions?

Kaz: should we have multiple tasks including prototyping, conformance test and documentation?

McCool: would create a dedicated GitHub repo for the work

Kaz: what should be the title for the repo?

McCool: "wot-testing"
... what do you think?

Sebastian: ok

McCool: specifically testing, planning for test cases, etc.
... landing point for the work

Koster: test content should be self-contain
... a lot of different resources there
... before we create a new repo, we should clean them up

McCool: a lot of old contents
... would suggest we create a new repo and move necessary contents to there
... and then make update

Sebastian: makes sense
... but who to handle what?

McCool: can volunteer
... and work with Ege

Kaz: what about JP participants?

McCool: can issue a call-for-participants
... also we can have discussion during the marketing call as well
... should have a resolution for "wot-testing" repo?

proposal: we'll create the "wot-testing" GitHub repo for further testing/plugfest work

(no objections)

RESOLUTION: we'll create the "wot-testing" GitHub repo for further testing/plugfest work

<scribe> ACTION: kaz to create "wot-testing" repo

Next calls

Kaz: we'll have a marketing call tomorrow at 14UTC
... and the next test/plugest call will be right after the main call next Wednesday

[adjourned]