Testing the Open Web Platform

As the Web has grown from a documentation sharing platform targeted essentially at desktop users to a fully applicative one available on all kinds of devices, so have its testing requirements. The test suites published by a number of W3C Working Groups in order to move their specifications along the Recommendation track are no longer sufficient. The kinds of applications built by developers require a level of conformance, quality of implementation, performance and interoperability that can only be guaranteed by much more thorough testing. In truth, the testing requirements are such that a concerted effort involving the industry as a whole is necessary in order to meet them. Unfortunately, so far, our infrastructure has been cruelly lacking, which has really impeded any such effort.

But this is about to change.

W3C is launching an unprecedented effort to scale up its test offering. And the good news is this effort is backed up by significant financial and human contributions from the W3C Membership. For the next year, my employer, Facebook, will be covering my salary and expenses while I focus, full time, on driving this effort at W3C. Some W3C Members have already committed human resources, others are poised to contribute tests or significant financial support.

Over the course of the next year, we’re going to take a number of steps to improve our testing infrastructure and documentation, simplify our processes, and increase test coverage.

First, we’re going to move all tests to a central repository on GitHub. This will be an opt-in process for Working Groups, so don’t expect change overnight. A number of Working Groups have already moved or are about to do. Other groups may require additional time, especially if they have invested in infrastructure built on top of Mercurial. There will be migration costs, but I’m positive that the benefits of a common repository, tooling, and processes will outweigh them.

Second, we’ll be streamlining test contribution and review processes. Writing a test, getting it reviewed and approved shouldn’t be more complicated than forking the repository on GitHub and making a pull request. There will still be a contribution license, but the process for agreeing to the license will be simpler than it is now.

Third, we’re creating a dedicated resource center for all things testing. There, you will find documentation, tutorials, test coverage, browser support, and other valuable data points, as well as information on how to download and run the test suites locally.

Fourth, we are overhauling the test framework to cater for the different use cases we’ve collected, such as running subsets of test suites or being able to install a local copy of the framework to test prototype devices or unreleased browsers. We will also set up an instance of the test framework on a public server so you can run tests on various devices over the Web. And we’ll publish test results to give developers a clear picture of support for a feature (e.g. IndexedDB) or even a subset of a feature (e.g. blob support in IndexedDB).

Where will the tests come from? We are investigating more direct solutions to increase test contributions and speed up reviews. Browser vendors are key contributors in this area and we’ve started discussing ways to simplify upstreaming their tests and syncing to our test repository. That’s not a trivial undertaking, so it could take some time before any of this is in place. Once set up, however, this could be a significant step forward.

Developers have also become an important source of test cases, thanks to great efforts such as Test the Web Forward. We’d like to see this trend continue and to foster a strong developer community around testing. I hope that the full slate of improvements we are making will move us in that direction.

It is also conceivable that we outsource the development of tests for some parts of the platform, for example to meet a particular timetable (such as HTML5 becoming a Recommendation in 2014). The changes described above will also help us manage this scenario more effectively.

There’s plenty more we’d like to tackle in the future (integration testing and performance benchmarking immediately come to mind), but I think the above focus is a solid starting point. I invite people to follow our work on the wiki and our mailing list. If you or your organization want to help us in any way, please contact me either directly or through the mailing list.

This journey 1% finished.

One thought on “Testing the Open Web Platform

  1. I’m glad that there will be common tests to work with, I’d hate to be a browser developer working directly from the written language of the spec, myself.

    Still, I sometimes wonder how we will ever get truly new rendering browsers entering the browser landscape, as the requirements of being a modern browser get more and more complex. Canvas? CSS animations? Document outlining? SVG? Backwards compatibility with old attributes and elements?

    Now, many of these things can degrade gracefully simply by not implementing them and not relying on them, but how can we ever expect html5 (a spec) to supplant flash (effectively a binary from a single source) when we can’t rely on things like canvas implementations to be the same cross browser? Like, how can different implementations ever be the same?

    Having centralized tests is great, and it’s a great step on the path to compatibility, but at the same time, the more complex the tests and requirements become, the harder it is going to be for new, innovative people to enter the browser space. Opera has given up (not that I blame them). I can’t imagine a lot of new browser groups taking on the herculean task of doing something out-of-the-box (out of the webkit?) and pushing innovation that way any more.

    I’m just glad I’m not a browser developer.

Comments are closed.