Warning:
This wiki has been archived and is now read-only.
Testing/Infrastructure/Requirements/
Contents
HTML Testsuite Requirements
Improved Test Runner
The current test runner is optimised mainly for manual tests. An in-browser test runner that deals automatically with the results of javascript tests is essential. It should also deal with self-describing reftests and manual tests in a sane way.
Requirements
- Produces a machine-readable report in some format (could be current XML or some other possibly non-XML format).
- Loads tests automatically based on test manifest files containing metadata about test uri, type, etc.
- Allows all or a subset of tests to be run
- Hooks into testharness to extract the results of the javascript tests without human intervention
- Allows reftests to be run by humans, e.g., by automatically switching between test view and ref view several times per second and asking the user if they see flickering (automatic running of reftests will require browser-specific code and is explicitly out of scope)
- Allows manual tests to be run by humans
Possible considerations
- Tests that require a top level browsing context
Annotation of Spec
In order to know which areas of the spec are well tested and hence have a sense for (an upper bound on) the completeness of the testsuite as well as the areas where it would be most profitable to direct new testing effort, it would be beneficial to produce an annotated version of the spec documenting tests that cover given requirements.
Requirements
- Map each test onto a piece of spec
- Fine grained definition of "piece"; some sections are long and contain many normative requirements so paragraph-level is probably the minimum useful level
- Good behaviour in the face of spec modifications, deletions, insertions and rearragements.
Test Review Tool
Most test review is currently done informally via the mailing list. This doesn't work so well, especially for for large testsuites. Maybe there is an existing tool that can help us here.
Requirements
- Allow anyone to easily give feedback on tests, not just named reviewers or people with W3C accounts
- Reasonable hg integration