If a Contributor is a member of WebApps, there is no need to complete any license forms. However, if a Contributor is not a member of WebApps, the Contributor must grant permission to distribute their contribution under these two licenses, and this requires completing the W3C's license grant form.
Test Repository Structure
The WG uses the W3C's web-platform-tests repository on GitHub for its tests.
- The repository:
- Mirror Root:
The structure of the repository, is:
[specification]is the short-name of a specification as used when the spec is published as a Technical Report e.g.
[section]is the section number of the specification the tests cover/test. The maximum number of section numbers is three levels deep.
NOTE: using section numbers as subdirectories is not required. However, if the test folder for the specification you're writing a test for uses the sections, then your submission (Pull Request) should also use section directories.
All tests that exist in the test repository are considered approved and reviewed. You can find current submissions by looking at the open pull requests of the repository.
Test Case Requirements
The group has agreed that all tests need to be written in the following manner:
- If a test is a visual test (not API specific) a Reference Test (aka RefTest) must be created. Note that -ref must be added to the testcase name to identify the reference page. For some information on RefTests, see:
The tests should preferably include the metadata:
<link rel="author" title="Name of Author" href="mailto: or http://"/> <link rel="help" href="http://example.com/relevant-spec#section"/> <meta name="assert" content="The lines of what the test asserts"/>
How to Submit a New Test - The Short Version
The way to contribute is just the usual GitHub way:
- Fork the GitHub repository (and make sure you're still relatively in sync with it if you forked a while ago)
- Create a branch for your changes,
git checkout -b submission/your-name
- Make your changes
- Push that to your GitHub repository
- Send in a pull request based on the above
You might get asked about completing W3C's license grant form if your employer is not already a member and you have not done so before. Someone will review your test and you might have to do some changes based on that review.
You are also welcome to ping the public-webapps-testsuite mailing list for any reason. Such as explicitly asking for review, discuss or announce your submission.
How to Submit New Tests - The Long Version
GitHub itself has a nice long help document explaining how to contribute to projects.
If you are brand new to Git and GitHub, you should probably read the getting started guide from the start. It's nice and friendly. The GitHub help pages has lots of other great information and guides. And for everything else you are welcome to send an email to the consortium's Public testing mail list
public-test-infra e-mail list (archive).
Test Review and Approval Workflow
The general workflow for test reviews and approval follows the GitHub contribution model and is summarized here:
- Pre condition: we assume everyone that is interested in reviewing and/or approving WebApps' tests will set a "Watch" at GitHub so they will be notified when a Pull Request (PR) is made.
- To initiate a review, make a Pull Request to the main
web-platform-tests/masteron GitHub (this will notify all subscribers of the PR). The review initiator may also notify WebApps of the PR via its public-webapps mail list.
- Reviewers should review the test code and reply accordingly. If no issues are found, the reviewer should state that to assure there is a paper trail a review was done.
- After all of the review issues are addressed, the PR will be merged into the repository. The merge may be done by the spec's Test Facilitator or someone that is overseeing the test repository. ^1
- The merged tests are considered Approved.
^1 If you are a member of WebApps, and would like to be able to merge Pull Requests after a successful review, ask one of the W3C Staff members to add your GitHub account into the Team group. Good candidates are Yves (@ylafon), Cindy (@siusin), Philippe (@plehegar), Robin (@darobin), James (@jgraham), Simon (@zcorpan) and Mike (@MikeSmith).
Information in this document refers to the following roles:
- Contributor / Submitter - someone that contributes one or more test cases
- Test Facilitator - the person(s) responsible for maintaining a specification's test suite, including updating test cases, making sure Pull Requests are reviewed, etc. Normally, this role is assumed to the specification's Editor(s). Test Facilitators are expected to work with Contributors e.g. to resolve quality issues, to eliminate redundant/overlapping asserts, etc. A Test Facilitator should create an Implementation Report (see WebApps Interops for example Implementation Reports.
- Testing Coordinator - the person responsible for coordinating all of WebApps' testing efforts. By default, this role is taken by a WG Chair, a WebApps Team Contact or a member of the WG. (In practice, this role may be shared by more than one person.)
Test Suite Status
The easiest way to determine the status of a spec's test suite is to simply ask on the
firstname.lastname@example.org mai list. Additionally, some test suite status information including the spec's Test Facilitator is available in WebApps Publication Status document.
How to Report a Test Case Bug
A bug report for a test case can be reported via any of the following, in order of preference (highest to lowest):
- Make a Pull Request with a patch to fix the bug
- Create a new GitHub Issue using the spec's directory name as the GitHub label. Include the name of the file and test case(s) that has a bug and clearly describe the bug. If the Test Facilitator's Github handle is known, assign the Issue to them.
- Send an e-mail to the
email@example.com that describes the bug and identifies the test file.