Testing/Infrastructure/Requirements/

From HTML WG Wiki
Jump to: navigation, search

HTML Testsuite Requirements

Improved Test Runner

The current test runner is optimised mainly for manual tests. An in-browser test runner that deals automatically with the results of javascript tests is essential. It should also deal with self-describing reftests and manual tests in a sane way.

Requirements

  • Produces a machine-readable report in some format (could be current XML or some other possibly non-XML format).
  • Loads tests automatically based on test manifest files containing metadata about test uri, type, etc.
  • Allows all or a subset of tests to be run
  • Hooks into testharness to extract the results of the javascript tests without human intervention
  • Allows reftests to be run by humans, e.g., by automatically switching between test view and ref view several times per second and asking the user if they see flickering (automatic running of reftests will require browser-specific code and is explicitly out of scope)
  • Allows manual tests to be run by humans

Possible considerations

  • Tests that require a top level browsing context

Annotation of Spec

In order to know which areas of the spec are well tested and hence have a sense for (an upper bound on) the completeness of the testsuite as well as the areas where it would be most profitable to direct new testing effort, it would be beneficial to produce an annotated version of the spec documenting tests that cover given requirements.

Requirements

  • Map each test onto a piece of spec
  • Fine grained definition of "piece"; some sections are long and contain many normative requirements so paragraph-level is probably the minimum useful level
  • Good behaviour in the face of spec modifications, deletions, insertions and rearragements.

Test Review Tool

Most test review is currently done informally via the mailing list. This doesn't work so well, especially for for large testsuites. Maybe there is an existing tool that can help us here.

Requirements

  • Allow anyone to easily give feedback on tests, not just named reviewers or people with W3C accounts
  • Reasonable hg integration