TPAC2016/WebPlatformTests

Minutes

https://www.w3.org/2016/09/22-testing-minutes.html

Intro

On the Thursday and Friday, we have a room for testing-related discussions, under the Web Platform Tests banner. This is currently (as of early September) scheduled to be in room 1.10. We're planning on running this in a participant-driven way, with no hard agenda; we realise that we'll have to give times for certain topics given people have other commitments on the two days.

Intended topics for discussion

Feel free to add anything to this list; if you don't have a W3C account with which to edit the wiki feel free to drop into #testing on IRC and ask for it to be added!

Outcomes are being tracked in this document

Thursday

Starting at 10am.

Overriding topic is getting browsers running more tests and improving interoperability in general.

Overview of how each engine is leveraging web-platform-test today, and the process for their devs to contribute to the tests.
Merging CSS WG tests into Web Platform Tests
Adding CI integration of WPT tests to Blink/WebKit/Edge builds (i.e., Mozilla does for both Gecko and Servo builds/CI)
Using WebDriver to automate tests that otherwise need to be run manually
APIs for automating input (eg. webdriver-actions)
Testing Bluetooth, USB, and other awkward APIs
Tool to easily run a set of tests on a bunch of browsers, automate bug-filing. Make TDD spec work more practical.
Where to best focus WPT efforts to reduce web developer interop pain?
Mine real-world browser site compat reports looking for areas to increase test coverage?
What is standing in the way of engines adding a "interop test results" requirement for every launch? (eg. in the blink intent-to-ship-template)
Are there other areas we could be collaborating in order to better reduce interop pain? Should we have some other interop-focused communication channel?

Friday

Starting at 10am.

Overriding topic is structure of tests and using the tests to meet CR exit criteria.

Using web-platform-tests to meet CR exit criteria
Publishing (publicly) of latest WPT test results by all browser-engine projects (possibly also history / trend lines)
Extension of “Test the Web Forward” project to include spec version information in test cases.
Include metadata in JSON created by testharness
Add support for 'assertions' in tests that have links into specs (and spec support for assertions via ReSpec and Bikeshed)
Better process for enhancing and releasing enhancements to wptreport, testharness, wptserver, etc.
Expanding the definition of "web platform" beyond the browser (other agents / plugins that support services)
Allowing for additional result types from tests (SKIPPED, Further Information Provided, etc)
Supporting tests that require facilities outside of the WPT server
Better support for tests of "optional" features (philosophically controversial)
Option to allow all assertions in a 'test' to run even if one fails (apparently discussed in the past; could use some work-around)
Passing parameters to the test runner via the URI
Passing parameters to the *tests* via the test runner URI (for tests that might need tuning)