HTML Testsuite Requirements
Improved Test Runner
- Produces a machine-readable report in some format (could be current XML or some other possibly non-XML format).
- Loads tests automatically based on test manifest files containing metadata about test uri, type, etc.
- Allows all or a subset of tests to be run
- Allows reftests to be run by humans, e.g., by automatically switching between test view and ref view several times per second and asking the user if they see flickering (automatic running of reftests will require browser-specific code and is explicitly out of scope)
- Allows manual tests to be run by humans
- Tests that require a top level browsing context
Annotation of Spec
In order to know which areas of the spec are well tested and hence have a sense for (an upper bound on) the completeness of the testsuite as well as the areas where it would be most profitable to direct new testing effort, it would be beneficial to produce an annotated version of the spec documenting tests that cover given requirements.
- Map each test onto a piece of spec
- Fine grained definition of "piece"; some sections are long and contain many normative requirements so paragraph-level is probably the minimum useful level
- Good behaviour in the face of spec modifications, deletions, insertions and rearragements.
Test Review Tool
Most test review is currently done informally via the mailing list. This doesn't work so well, especially for for large testsuites. Maybe there is an existing tool that can help us here.
- Allow anyone to easily give feedback on tests, not just named reviewers or people with W3C accounts
- Reasonable hg integration