Revision as of 23:40, 19 November 2011 by Abarsto (Talk | contribs)

Jump to: navigation, search

This document describes the the Web Application WG's (aka WebApps) test case Review and Approval process.

See the Submission document for information about the WG's test repository and how to submit test cases to the repository.

Comments on this document are welcome and should be submitted to the WG's public-webapps-testsuite e-mail list (archive).


Information in this document refers to the following roles:

  • Contributor - someone that contributes one or more test cases
  • Test Facilitator - the person(s) responsible for maintaining a specification's test suite, including: updating test cases, maintaining the test suite's status, etc. Normally, this role is assumed to the specification's Editor(s). A Test Facilitator will be designated as the Default Assignee for the test suite's bug component in WebApp's Bugzilla database. Test Facilitators are expected to work with Contributors e.g. to resolve quality issues, to eliminate redundant/overlapping asserts, etc.
  • Testing Coordinator - the person responsible for coordinating all of WebApps' testing efforts, including: assuring only approved tests are put in .../approved/ directories, making sure the test suites' status are current, etc. By default, this role is taken by a WG Chair, a WebApps Team Contact or a member of the WG. (In practice, this role may be shared by more than one person.)

Test Case Review and Approval

Early review of test cases is encouraged and expected. Test case review can be started by a Contributor, Test Facilitator or Testing Coordinator by sending a Request for Review (RfR) to the public-webapps-testsuite list that identifies the test case(s) to review. The RfR should include a deadline for comments and if the deadline is problematic for a reviewer, the reviewer should respond to the RfR as soon as possible and propose a later date.

If any issues are raised during the review, the group will use a consensus-based approach to resolve the issues.

If no issues for a test case are raised during the review, a Test Facilitator will copy the test case to the test suite's .../approved/ directory and update the test suite's status according to the information in the Test Suite Status section.

Test Suite Status

To facilitate determining the up-to-date status of a test suite, a Test Facilitator is responsible for maintaining the test suite's status. Each test suite includes a Status file i.e. .../tests/Status.html and this file includes at least the following information:

  1. Test Facilitator(s): <names ...>
  2. Approval status: normally, one of: a) no test cases are approved; b) test cases in the .../approved/ directory are approved by members of WebApps' Testing Group; c) the entire WG has formally approved all of the test cases in the .../approved/ directory.

Formal Test Suite Approval

After a test suite has been approved by WebApps' Testing Group via the process described above, and the WG considers the specification stable, a WG-wide Call for Consensus to formally approve the test suite will be sent to the public-webapps list.

A formal review should include:

  • Analyzing each test case for correctness e.g. does the test case actually test what it says it tests
  • Determining if the test cases cover all of the specification's assertions (or a subset of the specification depending on the set of test cases being reviewed)
  • Reporting test case bugs (as described below)

If any testing-related issues are found during the review, they will be addressed using the same consensus-based process the WG uses to address specification issues and this may result in a new CfC to approve a subset of a test suite's test cases. (See WebApps WorkMode wiki for more information about the WG's CfC process).

If the group agrees the review resulted in identifying an issue with a specification, a bug will be filed against the specification using the WG's Bugzilla database

The duration of the CfC will vary, depending on factors such as the number of test cases to be reviewed.

Note: to be consistent with all of the WG's other Calls for Consensus, silence on CfCs to review test cases will be considered Approval.

How to Report a Test Case Bug

The process for reporting a test case bug depends on whether the test case has been approved or not ...

To report a bug in a test case that has Not been Approved (i.e. the test case is not in a specification's .../approved/ directory), send an e-mail to and identify the test case(s) (including the URI) that has a bug and clearly describe the bug. Contributors and Test Maintainers are expected to update their non-approved test cases without sending such a notification.

To report a bug in a test case that has been Approved (i.e. the test case is in the specification's .../approved/ directory), file a Bugzilla bug against the appropriate test suite component in WebApps' Bugzilla database. Please identify the test case(s) that has a bug and clearly describe the bug. (If you do not have write access to the bug database, follow the instructions above for non-approved tests).

Updating an Approved Test Cases

An update to an approved test case is only permitted after the group that approved the test case (i.e. the WebApps Testing Group or the entire WebApps WG) approves the update.

If a bug is found in an Approved test case, please file a bug report against the test case as described above.