PlugFest WebConf
From Web of Things Interest Group
The WoT Plugfest/Testing task force is responsible for organizing the Web of Things (WoT) Plugfests and also for organizing general testing procedures, data collection, and implementation report tooling for WoT deliverables.
WebConf Information
- Quick start guide for W3C teleconferences
- Table of all the WoT calls on our Web page
- Main WoT WebConf
- Scripting APIs | Security | Discovery | Marketing | Use Cases | PlugFest/Testing | TD | Architecture
Repositories
Testing and plugfest materials are located at https://github.com/w3c/wot-testing
Wednesdays at 9am US Eastern / 3pm Europe / 11pm Japan
Note: Due to Daylight Savings in US, the call will start one hour earlier in Europe between October 15 March and 26 March. Calls are all scheduled in US Eastern Time, European daylight savings starts on a different date, and some places do not have Daylight Savings at all, so the following changes will need to be observed:
- In Europe, calls will start one hour earlier after March 15 and then go back one hour later (=usual time) after March 29.
- In China, Japan and Russia, where Daylight Savings is not observed, calls will start one hour earlier after March 15.
See email here for further information.
WebEx
This call does not have its own WebEx, instead the main call is continued.
Note:
- Calls are all scheduled in US Eastern Time
IRC
- IRC #wot-pf channel
- Custom clients: irc.w3.org, port 6667
The IRC is used for the minutes, speaker queue, and sharing links etc.
Actions
- Organize wot-test repo (McCool)
- Copy over existing material from wot repo
- Generate feature lists to abstract assertions, cover "informative" features, like use of MQTT, etc.
- Copy over and improve implementation report generation scripts from wot-thing-description
- Improve and generalize implementation report generation scripts (McCool)
- Capture implementation descriptions in formal CSV tables
- Include and generate report on interoperability testing
- Plan Helsinki plugfest
- Include formal data capture plans, including for conformance and interoperability
- Finalize technical requirements
- Set up remote access links, eg WebEx, Google hangouts, etc.
- Capture more examples that can be shared
- Link to wotify.org and other examples from wot-testing
- Add links to formal implementation CSV files
- Capture information on both producers and consumers, e.g. mashups in Node-RED
- Even just screenshots
- Improve test tools
- E.g. scripts to "exercise" a device, use UI to confirm actions, etc.
- See https://github.com/tum-esi/testbench - connect to impl/plugfest report
- Standardize setup for implementations so they can be reproduced easily
- Use semantics to guide testing; special vocab for testing?
- Create procedure/template for Plugfest implementation preparation
- Passwords, network setup, how to document implementations (incl authentication), behavior, etc.
- Add link to documentation from CSV table, archive documentation in wot-testing
Action Items
- Organize 2022.07 testfest
- Update manual.csv template (Fady)
- Regenerating Results (mccool)
- update script to capture results from previous years/plugfests
- Will keep old results separate, add prefix to results files with dates, etc.
- install playground
- run batch script
- copy results to TD repo (keep files separate for older results)
- regen IRs
- see process.md for instructions (ege to update)
- update script to capture results from previous years/plugfests
- https://github.com/w3c/wot-testing/issues/265
Agendas
Agenda 07.12.2022
Chair: McCool
Scribe: Fady
- Review Minutes
- Nov 30, 2022 Minutes (defer)
- Test fest preparation
- Review instructions
- CSV change log script
- CSV format updates
- AOB
Agenda 30.11.2022
Chair: McCool
Scribe: Sebastian
- Review Minutes
- test fest preparation
- logistics
- testing data organization (CSV files, etc)
- scenarios
Agenda 16.11.2022
Chair: Lagally
Scribe: Sebastian
- Review Minutes
- Overall testing structure
- Continue the testing conversation wrt. profile
- manual vs. automated testing
- test fest preparation - scenarios
Agenda 09.11.2022
Chair: Lagally
Scribe: tbd.
- Review Minutes
- Implementation Reports
- Check at risk items
- Testfest/Plugfest planning
- Schedule: Dec 12-16, times tbd.
- Goals
- Scenarios
- AOB
Agenda 02.11.2022
Chair: McCool
Scribe: Kaz
- Review Minutes
- Implementation Reports
- Testfest/Plugfest planning
- Schedule: Dec 12-16, times
- Goals
- Scenarios
- AOB
Agenda 12.10.2022
Chair: McCool
Scribe: Ege
- Review Testfest minutes
- Review Implementation Reports
- TD - Latest Draft Implementation Report
- Discovery - Latest Draft Implementation Report
- Architecture - Latest Draft Implementation Report
- Profile - Latest Draft Implementation Report
- PRs
- Issues
- AOB
Agenda 07.10.2022
Testfest
Chair: McCool
Scribe: Sebastian
- PRs
- Possible useful tool: https://github.com/simonw/csv-diff
- Implementation Report Reviews
- Wrapup and next steps
Agenda 06.10.2022
Testfest
Chair: McCool
Scribe: Cristiano
- Tooling issues (Ege)
- Two new assertions : tm-versioning-2 and tm-tmOptional-resolver
- There is a PR in the TD spec that needs merging asap at https://github.com/w3c/wot-thing-description/pull/1715 since it helps with the tooling.
- merged the automatic PR at https://github.com/w3c/wot-thing-description/pull/1699 since that was causing one part of the discrepancy with manual.csv
- When we are generating the manual.csv, we need to put quotes around “everything”. For example, when we generate the word null for the value of Status, if we do not encapsulate it in quotes, it is understood as the JS specific null and nothing is printed to the manual.csv status value. Do quotes break your tools?
- PRs
- Implementation Reports
- Review latest versions
- Architecture 1.1
- Thing Description 1.1
- Discovery
- Profile
Agenda 05.10.2022
Testfest
Chair: McCool
Scribe: Ege
- PRs
- Implementation Reports
- Review latest versions
- Architecture 1.1 (rendered)
- Thing Description 1.1 (rendered)
- Discovery (not updated - no new results)
- Profile (not updated - no new results)
Agenda 04.10.2022
Testfest
Chair: McCool
Scribe: Lagally
- PRs
- Implementation Reports
- Review latest versions
- Architecture 1.1
- Thing Description 1.1
- Discovery
- Profile
- Profile
- Overview of implementations
- Interoperability testing
Agenda 03.10.2022
Testfest
Chair: McCool
Scribe: Cristiano
- PRs
- Organization
- https://github.com/w3c/wot-testing/tree/main/events/2022.09.Online
- README
- Results
- Templates
- Implementation Descriptions
- Schedule
- AOB
Agenda 28.09.2022
Chair: McCool
Scribe: Cristiano
- Review Minutes
- October Testfest planning
- AOB
Agenda 31.08.2022
Chair: McCool
Scribe: Kaz
- Review Minutes
- Architecture IR
- Post-TPAC Testfest
- Week of Sept 26
- Focus on Profile *testing*
- Possible "consumer" testings for all TDs (e.g. node-wot and Node-RED trying to read all TDs).
- Architecture should not need *additional* testing (in theory)
- AOB
Agenda 24.08.2022
Canceled.
Agenda 10.08.2022
Chair: McCool
Scribe: Ege
- Review Minutes
- Planning
- Prep for CR Transition
- Additional testing prior to CR/PR -> data/input_2022
- Updated instructions (README, last testfest)/email (McCool)
- Periodic merging and IR updates: daily
- How to deal with changes to manual.csv e.g. when TD spec changes? Warnings from tooling for missing/obsolete/auto-manual/new changes?
- batchValidation.sh had many missing -m options
- still need many implementation descriptions (McCool drafts... then implementers can do fixes via PR suggestions)
- Pre-TPAC Meeting
- Sept 7 - resolutions for CR transition
- Sept 5 should be deadline for CR transition IR and test results
- Post-TPAC Testfest
- Options: Sept 26-30; Oct 3-7; Oct 10-14; Oct 17-21 -> doodle
- Prep for CR Transition
- Review recent testing results
- AOB
Agenda 03.08.2022
Chair: McCool
Scribe: Fady
- Review Minutes (Testfest)
- Status Review
- TD
- Discovery
- AOB
Agenda 29.07.2022
Testfest - Day 5
Chair: McCool
Scribe: Cristiano
- Review Minutes
- deferred
- PRs
- Implementation Reports
- Next Steps
- Implementation Descriptions
- Testimonials
- Update todo.csv
- Manual assertions
- Assertion Review
- Architecture
- Security and Privacy
- AOB
Agenda 28.07.2022
Testfest - Day 4
Chair: McCool
Scribe: Ege
- Review Minutes
- deferred
- PRs
- Implementation Reports
- Assertion Review
- Architecture
- Security and Privacy
- AOB
Agenda 27.07.2022
Testfest - Day 3
Chair: McCool
Scribe: Sebastian
- Review Minutes
- deferred
- PRs
- Implementation Reports
- Assertion Review
- Architecture
- Security and Privacy
- AOB
Agenda 26.07.2022
Testfest - Day 2
Chair: McCool
Scribe: Sebastian
- Review Minutes
- deferred
- PRs
- PRs with new data already merged
- Implementation Reports
- TD IR updated yesterday
- Now running validation script on new inputs
- Assertion Review
- Architecture
- Security and Privacy
- AOB
Agenda 25.07.2022
Testfest - Day 1
Chair: McCool
Scribe: Kaz
- Review minutes
- Will defer all until next week
- Logistics
- PRs
- Planning
- Security and Privacy
- Architecture
- AOB
Agenda 20.07.2022
Chair: McCool
Scribe: Ege
- Review minutes
- PRs
- Logistics
- Implementation Reports
Agenda 13.07.2022
Chair: McCool
Scribe: Philipp
- Review minutes
- PRs
- Logistics
- July 25 - Testfest
- August 29 - Plugfest
- Implementation Reports
- Still missing implementation descriptions and testimonials
Agenda 06.07.2022
Chair: McCool
Scribe: Cristiano
- Review minutes
- PRs
- Logistics
- July 25 - Testfest
- August 29 - Plugfest
- Implementation Reports
- Still missing implementation descriptions and testimonials
Agenda 29.06.2022
Chair: McCool
Scribe: Fady
- Review minutes
- PRs
- Plugfest logistics
- Move to Week of July 25
- Implementation Reports
- Manual test fixes in Discovery
- TM testing issues still?
- To Dos
- Still missing implementation descriptions and testimonials
Agenda 22.06.2022
Chair: McCool
Scribe:
- Review PF minutes
- PRs
- Plugfest logistics
- Week of July 18
- Need to avoid DTDL mtg during Use Case slot...
- Implementation Reports
- Updates to boilerplate text (in Discovery, at least)
- span/div issues fixed
- trailing spaces problem fixed
- issues with at-risk markup fixed
- at-risk if less than 2 implementations whether or not optional
- assertion fixes (e.g. due to missing markup, multiple RFC keywords, etc)
- To Dos
- Discovery todo.csv updated
- Still problem with manual.csv
- Still missing implementation descriptions, testimonials, manual assertions
- Discovery todo.csv updated
- "Policy" Assertions
- Assumption: Treat these as manual assertions.
Agenda 15.06.2022
Chair: McCool
Scribe: Fady
- Review Implementation Reports
- To Dos
Agenda 10.06.2022
Testfest Day 5
Chair: McCool
Scribe: Fady
- PRs
- Last-minute updates
- Implementation Report Drafts
- TD
- Discovery
- At-Risk Items
- Markup
- Tables
- Responsibilities
- AOB
Agenda 09.06.2022
Testfest Day 4
Chair: McCool
Scribe: Ege/Kaz
- Updates
- Implementation Report Drafts
- Gaps
- TD PR on testing
- Testing PRs
- Issues
- AOB
Agenda 08.06.2022
Testfest Day 3
Chair: McCool
Scribe: Fady
- Updates
- Implementation Report Drafts
- Gaps
- TD PR on testing
- PRs
- Issues
- AOB
Agenda 07.06.2022
Testfest Day 2
Chair: McCool
Scribe: Ege
- Updates
- Implementation Report Drafts
- Gaps
- PRs
- Issues
- AOB
Agenda 06.06.2022
Testfest Day 1
Chair: McCool
Scribe: Jan
- Organization
- TD is ready, see README
- Discovery still needs updates to template.csv; will be done by tomorrow
- Instructions
- Links to example PRs and issues (for work on addressing gaps)
- PRs
- Intel examples in TD
- Issues
- Intel example for gap-issue
- AOB
Agenda 01.06.2022
Chair: McCool
Scribe: Jan
- Review PF minutes
- TestFest org
- Calendar entries, WebExes, etc
- Goals:
- CR-Ready implementation report for TD
- CR-Ready implementation report for Discovery
- To Do (see README for event)
- Testing inputs
- Manual assertions
- Implementation Descriptions
- Documentation
- PRs
- Issues
- Summer Testfest?
- AOB
Agenda 25.05.2022
Chair: McCool
Scribe: Ege
- Review PF minutes
- Action Items
- TestFest org
- Chair: Fady -> Fady/McCool co-chair, both TF and Testfests/Plugfests
- Time: https://doodle.com/meeting/participate/id/bmZp0L3d
- Proposal:
- 8am Eastern/12pm UTC Monday and Tuesday
- 7am Eastern/11am UTC Wednesday
- 10am Eastern/2pm UTC Thursday, and Friday
- Separate testfests and plugfests?
- Testing is very time-critical and should be priority
- TestFest org
- PRs
- Issues
Agenda 18.05.2022
Chair: McCool
Scribe: Fady
- Review PF minutes
- Action Items
- TestFest org
- PRs
- Issues
Agenda 11.05.2022
Chair: McCool
Scribe: Fady
- Review PF minutes
- Action Items
- PRs
- Issues
Agenda 04.05.2022
Chair: McCool
Scribe: Fady
- Review PF minutes
- PRs
- Issues
Agenda 27.04.2022
Chair: McCool
Scribe: Philipp
- Review PF minutes
- 3xx Error Codes - Profile
- Implementation Report updates
- PRs
- Testfest planning
- Proposed Testfest 4-8 June
- https://github.com/w3c/wot-testing/issues/297
- Issues
- TPAC
- Demos?
- How to handle remote connections?
- Videos?
- Presentations?
- What do we already have?
- In joint with a Plugfest?
- Vancouver is now a go - hybrid event
- Demos?
Agenda 20.04.2022
Chair: McCool
Scribe: Ege
- Review PF minutes
- Implementation Report updates
- Results for TD still need to be updated, then IR to update
- Discovery are some new results - but IR not updated yet
- PRs
- https://github.com/w3c/wot-testing/pull/293 - merged, discovery test results
- Issues
Agenda 13.04.2022
Chair: McCool
Scribe: Philipp
- Review PF minutes
- Implementation Report updates
- PRs
- Guidelines for manual.csv
- Ditto TDs
- Issues
- Organization
- Home Assistant
Agenda 06.04.2022
Chair: Fady
Scribe: TBD
- Review PF minutes
- Manual CSV demo
Agenda 30.03.2022
Chair: Fady
Scribe: Kaz
- Review PF minutes
- Manual CSV demo next week
- Streamlining generation of assertions and diffs
- Review Issues
Agenda 23.03.2022
Chair: Fady
Scribe: Cristiano
- Review PF minutes
- https://www.w3.org/2022/03/09-wot-pf-minutes.html
- Also last Friday's plugfest minutes: https://www.w3.org/2022/03/18-wot-pf-minutes.html
- Testfest followup
- ESPHome/Retail demo
Agenda 09.03.2022
Chair: Fady
Scribe: TBD
- Review PF minutes
- Status of VLAN and VPN
- PRs:
Agenda 02.03.2022
Chair: Fady
Scribe: Ege
- Review PF minutes
- Offline meeting
- PRs:
- WIP: Oracle tools
- #211
- #210
- Updates on the assertion tester
Agenda 23.02.2022
Chair: Fady
Scribe: McCool
- Review PF minutes
Agenda 16.02.2022
Chair: Fady
Scribe: Ege
- Review PF minutes
- Small update regarding assertion tester tool
Agenda 09.02.2022
Chair: Fady
Scribe: Cristiano
- Review PF minutes
- Discuss needs for next Testfest
Agenda 02.02.2022
Chair: Fady
Scribe: McCool
- Review PF minutes
- Look at schedule issue
- Considerations for Testfest/Plugfest in March
Agenda 26.01.2022
Chair: Fady
Scribe: Kaz
- Review PF minutes
- Review PRs
- Considerations for Testfest in March
Agenda 19.01.2022
Chair: Fady
Scribe: Cristiano
- Review PF minutes
- Plugfest schedule
- Merging PRs
