The web-platform-tests Project

Back in 2013, Tobie Langel wrote about testing the Open Web Platform. The vision was simple: W3C was launching an unprecedented effort to scale up its test offering and the journey was 1% finished. Many things have been happening since then and this week is a great opportunity to look at the project again.

In 2014, the project moved to become a consensus-driven open-source project, using the web-platform-tests (aka WPT) as its central repository on GitHub. James Graham spent significant time on bringing the infrastructure together (see his GTAC 2014: Automation for a Better Web talk), as well as improving the level of reliability for the tests, with the help of Ms2ger. The wptrunner was introduced, originally designed to allow fully automated running of the tests in various Mozilla products.

In 2015, multiple improvements were added. Reftests were made more flexible to align with the CSS testing. A lint tool was introduced to facilitate the work of reviewers; The server got new features, such as storing cross-request state.

In 2016, to help improve the stability of the tests, a stability checker was brought in. It automatically runs new test 10 times on Firefox and Chrome to avoid problems when imported into the browser CI infrastructures. The Pointer Events Working Group adopted a test-driven approach, where changes to the specification are generally expected to have a corresponding pull request in WPT.

In 2017, the convergence of all browser engines around WPT increased. The Chromium project adopted WPT as their primary shared test suite. WebKit is actively looking at doing the same. The WHATWG started to use a WPT test-driven approach. The WebDriver specification reached Candidate Recommendation status. Sam Sneddon migrated the CSS tests to WPT. Earlier this month, the Web Performance Working Group adopted the WPT test-driven approach as well.

This week, Bob Holt extended the stability checks to include Edge and Safari (see also Diving Into the Web Platform Tests). This brings the project closer to producing and storing test results.

As Tobie wrote 4 years ago, there’s plenty more we’d like to tackle in the future and the community is welcome to help. I'm looking forward to leverage the progress of the testing project for the various W3C specifications in development (over 200 nowadays!).