Goal

A website that can be used to compare the results of running web-platform-tests across multiple different browsers.

Background Information

web-platform-tests is a cross-browser testsuite for the web platform. It is designed to improve interoperability between different web browsers by allowing them to check their implementations of various specs against a testsuite. Full documentation of web-platform-tests is available in

In order to run web-platform-tests in browsers a couple of tools have been developed. The first is a semi-automatic in-browser runner that produces JSON output by default. The second is wptrunner, a fully automated runner that produces mozlog format messages as output by default. The latter is being used to run the test in several browser CI systems (notably those for Gecko and Servo).

At the moment there isn't a good system for visualizing the results of a specific test run (e.g. to tell where a browser is not matching the spec) or for comparing between different browsers (in order to tell where tests are broken across many browsers possibly indicating a test bug or a spec bug). Note: it it *not* a goal to provide coarse grained "scores" for each result set (e.g. percentage of tests passed), since these can be very misleading and lead to bad incentives for people contributing tests.

Specific project goals

A server-side component that can ingest data from a wptrunner run (and perhaps an in-browser runner run) and store the results in a database, retaining the information that is required for the subsequent display. Initially this can just require manual specification of the log files (note that in the case of e.g. gecko there is > 1 log produced per run due to test chunking; this needs to be supported). Some consideration needs to be given of the metadata required to identify the browser version that was being used and the version of web-platform-tests that the browser was being run against.
A web front end that can display the results of a particular run in a specific browser. This must be able to work efficiently with a subset of the data i.e. it must not be necessary to download and display the results of all the tests in order to see the results of a subset of tests.
Extensions to the web front end to allow comparisons between various sets of results e.g. to see results that changed between two different runs, or all tests that fail in 2 or more browsers from a given set of runs.

Required Skills

Server-side web programming (Python)
Some database knowledge.
Client side web programming (HTML, Javascript, etc.)