Test Framework for SysApps

From W3C Wiki

This is a collection of notes on testing for normative assertions in the specifications developed by the System Applications Working Group. The aim is to align with best practices for testing at W3C.

Would it help to appoint someone to head up testing for SysApps with an overall responsibility for managing testing of SysApps specifications and liaison with the W3C testing activity spearheaded by Tobie Langel?

Robin Berjon writes:

You definitely need to coordinate your work with Tobie; but again don't block on that. You can start writing tests, and then we can figure out how to integrate. If you follow some basic guidelines you'll be fine.


Guidelines for specification editors

Mobile Web Initiative Test Suites Working Group Note from 28 January 2010:

Do we want to recommend the use of ReSpec, and what further guidelines are appropriate?

Robin Berjon writes:

Using ReSpec is of course great, but at this point it's entirely unrelated to testing so I wouldn't cross the beams there.

Generating Tests

Some tests can be full automated, whilst others require human involvement, either to perform an action or to interpret the result.

  • Techniques for generating tests from WebIDL in specifications, e.g. Wakaba's wttjs, and Robin Berjon's WebIDL parser
  • Community contributed tests and the process for their submission, review and approval
  • Issue tracking for tests

Robin Berjon writes:

You will soon realise that the primary bottleneck in test writing isn't the writing itself but rather the reviewing. So make that lightweight: a test that seems roughly correct is good enough, if there's a problem it can be backed out. If you use GitHub, which you really, really should, then do test reviewing through pull requests (and never, ever use the submitted/approved system that WebApps used to have). Anyone in the group should have review powers, including people from the same company as the person submitting.

See also Testing the Web Forward for an initiative for facilitating community contributions for tests through hackathons. These would become useful once we have specifications, implementations, and enough of a community to attend.

Test Harness

What are the implications for using packaged and hosted apps for test frameworks?

Robin Berjon writes:

Definitely use testharness.js. For hosted apps it should always be linked as /resources/testharness.js. For packaged apps you probably want to come up with the same convention. The way we set it up in the web-platform-tests repo is that I have a submodule for https://github.com/w3c/testharness.js installed at the root and called "resources". This makes the repo self-sufficient for installation at a web root.

Test repository

Some groups have used w3c-test.org, but the current plan is to switch to a central repository on GitHub.

This will be an opt-in process for Working Groups. This set of test suites is targeted at specs implemented in the browser, so may not be the right target for SysApps. Do we (SysApps) want to opt into this central repository or do we want to maintain our own?

Robin Berjon writes:

Using w3c-test.org and GitHub aren't alternative choices, they work together. The web-platform-tests repo is synched to w3c-test.org. One is for managing the test suite, the other for publishing it.

I would have to check with others but I don't think that at this point in time SysApps tests would be welcome to the web-platform-tests repo. That's because tests that are there are intended for the browser platform. I know there's overlap, and I certainly hope that there will be enough convergence to mix the test suites, but at this point I'd start with a different repo. But please use the same structure (e.g. one directory per spec), review process, etc. We can of course set up automatic publication to w3c-test.org. The more we are aligned, the more you will be able to use the tools that we develop. And we're in the process of getting a decent budget for tool writing so it's really worth it.

Community based testing

Do we want to provide a means for people to run tests on a variety of devices and automatically report the results to W3C, along with a report generation tool for viewing individual and aggregated results?

  • This should work well on a range of different devices, e.g. through responsive design, so that tests can be run on desktops, smart phones, tablets and connected TVs, etc.

Importing W3C's test suite into implementer's test suites

What do we need to do to facilitate this?