Revision as of 21:31, 8 December 2012 by Abarsto (Talk | contribs)

Jump to: navigation, search

The primary focus of this document is to describe the test case submission process for the Web Applications WG (aka WebApp). In particular, the document describes the WG's test depository structure and how WG members and non-WG members may contribute test cases.

Comments on this document are welcome and should be submitted to the WG's public-webapps-testsuite e-mail list (archive).

Test Licensing

All of the tests in WebApps' test suites are intended to be released under both the W3C Test Suite license and the 3-clause BSD license.

If a Contributor is a member of WebApps, there is no need to complete any license forms. However, if a Contributor is not a member of WebApps, the Contributor must grant permission to distribute their contribution under these two licenses, and this requires completing the W3C's license grant form.

Test Repository Structure

The WG uses the W3C's Hg (Mercurial) source control tool for its tests. WebApps' Hg and Mirror roots are as follows:

The structure of the repository, is:

  • .../webapps/[SPECIFICATION]/
  • .../webapps/[SPECIFICATION]/tests/Status.html
  • .../webapps/[SPECIFICATION]/tests/approved/
  • .../webapps/[SPECIFICATION]/tests/submissions/[CONTRIBUTOR]/


  • SPECIFICATION is a short name of a specification e.g. XMLHttpRequest, WebWorkers
  • CONTRIBUTOR is the name of the submitter / contributor e.g. CompanyA, ContributorName
  • The Status.html file is described in Approval
  • The submissions directory will have a different directory for each Contributor that submits test cases.
  • The approved directory only includes those test cases that have been approved as described in Approval. Contributors should not put test cases in this directory without prior consent.

The WG's repository also includes the general directories for testing-related resources e.g. resources that are shared among the various test suites:

  • .../webapps/tests/
  • .../webapps/tests/resources/

Test Case Requirements

The group has agreed that all tests need to be written in the following manner:

  1. Tests should use the JavaScript harness described in Harness
  2. If a test is a visual test (not API specific) a Reference Test (aka RefTest) must be created. Note that -ref must be added to the testcase name to identify the reference page. For some information on RefTests, see:
    1. RefTests by the CSS WG
    2. Creating reftest-based unit tests by Mozilla
  3. The final option is a self-describing test and should be used only when a JavaScript or a RefTest is not appropriate.

How to Submit a New Test - The Short Version

A Contributor can submit tests by:

  1. Submitting the tests directly to WebApp's Hg repository. However, if you are not a W3C member and do not have write access to this repository, then any public location is ok.
  2. Notifying the WG of the submission or the intent to submit by sending an e-mail to the mailing list, describing the test cases submitted or the test cases available for sumbmission, and, if the Contributor is not a WG member, granting permission for their use as described in the licensing section above.

Please make sure submissions target one of WebApps' specifications.

All tests are subject to the approval process as described in Approval.

Contributors may be asked to make updates to their submitted test cases, if, for example, a test case does not meet the test case requirements listed above, the test case has a bug, etc.

How to Submit New Tests - The Long Version

  1. Install Mercurial
  2. Pull The Respository
    1. hg clone
  3. Prepare a Directory in the Staging Area
    1. Go to the [SPECIFICATION]/tests/submissions/ directory in the staging area.
    2. If you already have a directory for yourself or your group, use it; otherwise create a new directory.
  4. Write Your Test Case
    1. Write a test case @@@ TBD: add a link to an example
  5. Commit your test case
    1. Set your W3C username in your global mercurial configuration.
    2. In the [paths] section of the .hg/hgrc file inside the top level of your clone of the repository, add the line: default-push = https://[USERNAME]
    3. Use hg add to add the new files. (See hg help add. Or, if you want to be fancy, you can use hg addremove; see hg help addremove.)
    4. Run hg diff to see what you're going to commit and then use hg commit (with a good commit message, the -m option). (For more information, see hg help commit and hg help diff.)
    5. Run hg outgoing to see what you're about to push and then hg push to push to the repository.
      1. If you're told that the push creates new remote heads, you need to merge with them. At this point, you have two options:
        1. hg pull, hg merge, and hg commit -m"Merge.", and then try pushing again.
        2. rebase your changesets either using the rebase extension or using mq
  6. Notify the WG of the submission by sending an e-mail to the mailing list, describing the test cases submitted.