Difference between revisions of "Approval"

From WEBAPPS
Jump to: navigation, search
m (How to Report a Test Case Bug)
(This doc is now obsolete; some info moved to http://www.w3.org/wiki/Webapps/Testing)
 
Line 1: Line 1:
This document describes the the Web Application WG's (aka WebApps) test case Review and Approval process.
+
'''This document has been obsoleted, replaced by WebApps' new [http://www.w3.org/wiki/Webapps/Testing Github-based Testing wiki].
 
+
'''
See the [[Submission]] document for information about the WG's test repository and how to submit test cases to the repository.
+
 
+
Comments on this document are welcome and should be submitted to the WG's <code>[mailto:public-webapps-testsuite@w3.org public-webapps-testsuite]</code> e-mail list ([http://lists.w3.org/Archives/Public/public-webapps-testsuite/ archive]).
+
 
+
 
+
== Roles ==
+
 
+
Information in this document refers to the following ''roles'':
+
 
+
* Contributor - someone that contributes one or more test cases
+
 
+
* '''Test Facilitator''' - the person(s) responsible for maintaining a specification's test suite, including: updating test cases, maintaining the test suite's status, etc. Normally,  this role is assumed to the specification's Editor(s). A Test Facilitator will be designated as the ''Default Assignee'' for the test suite's bug component in WebApp's [http://www.w3.org/Bugs/Public/describecomponents.cgi?product=WebAppsWG Bugzilla database]. Test Facilitators are expected to work with Contributors e.g. to resolve quality issues, to eliminate redundant/overlapping asserts, etc.
+
 
+
* '''Testing Coordinator''' - the person responsible for coordinating all of WebApps' testing efforts, including: assuring only approved tests are put in <code>.../approved/</code> directories, making sure the test suites' status are current, etc. By default, this role is taken by a WG Chair, a WebApps ''Team Contact'' or a member of the WG. (In practice, this role may be shared by more than one person.)
+
 
+
* '''WebApps' Testing Group''' - WG members subscribed to the WG's <code>[http://lists.w3.org/Archives/Public/public-webapps-testsuite/ public-webapps-testsuite]</code> list
+
 
+
== Test Case Review and Approval ==
+
 
+
Early review of test cases is encouraged and expected. Test case review can be started by a Contributor, Test Facilitator or Testing Coordinator by sending a '''Request for Review''' (RfR) to the <code>[mailto:public-webapps-testsuite@w3.org public-webapps-testsuite]</code> list that identifies the test case(s) to review. The RfR should include a deadline for comments and if the deadline is problematic for a reviewer, the reviewer should respond to the RfR as soon as possible and propose a later date.
+
 
+
If any issues are raised during the review, the group will use a consensus-based approach to resolve the issues.
+
 
+
If no issues for a test case are raised during the review, a Test Facilitator will copy the test case to the test suite's <code>.../approved/</code> directory and update the test suite's status according to the information in the Test Suite Status section.
+
 
+
== Test Suite Status ==
+
 
+
To facilitate determining the up-to-date status of a test suite, a Test Facilitator is responsible for maintaining the test suite's status. Each test suite includes a '''Status''' file i.e. <code>.../tests/Status.html</code> and this file includes at least the following information:
+
 
+
# Test Facilitator(s): <names ...>
+
# Approval status: normally, one of: a) no test cases are approved; b) test cases in the <code>.../approved/</code> directory are approved by members of WebApps' Testing Group; c) the entire WG has formally approved all of the test cases in the <code>.../approved/</code> directory.
+
 
+
== Formal Test Suite Approval ==
+
 
+
After a test suite has been approved by WebApps' Testing Group via the process described above, and the WG considers the specification stable, a WG-wide Call for Consensus to formally approve the test suite will be sent to the <code>[http://lists.w3.org/Archives/Public/public-webapps/ public-webapps]</code> list.
+
 
+
A formal review should include:
+
 
+
* Analyzing each test case for correctness e.g. does the test case actually test what it says it tests
+
* Determining if the test cases cover all of the specification's assertions (or a subset of the specification depending on the set of test cases being reviewed)
+
* Reporting test case bugs (as described below)
+
 
+
If any testing-related issues are found during the review, they will be addressed using the same consensus-based process the WG uses to address specification issues and this may result in a new CfC to approve a subset of a test suite's test cases. (See WebApps [[WorkMode]] wiki for more information about the WG's CfC process).
+
 
+
If the group agrees the review resulted in identifying an issue with a specification, a bug will be filed against the specification using the WG's [http://www.w3.org/Bugs/Public/describecomponents.cgi?product=WebAppsWG Bugzilla database]
+
 
+
The duration of the CfC will vary, depending on factors such as the number of test cases to be reviewed.
+
 
+
<span style="color:#ff0000">'''Note: to be consistent with all of the WG's other Calls for Consensus, silence on CfCs to review test cases will be considered Approval.'''</span>
+
 
+
== How to Report a Test Case Bug ==
+
 
+
The process for reporting a test case bug depends on whether the test case has been approved or not ...
+
 
+
To report a bug in a test case that has '''Not''' been ''Approved'' (i.e. the test case is not in a specification's <code>.../approved/</code> directory), send an e-mail to <code>[mailto:public-webapps-testsuite@w3.org public-webapps-testsuite@w3.org]</code> and identify the test case(s) (including the URI) that has a bug and clearly describe the bug. Contributors and Test Maintainers are expected to update their non-approved test cases without sending such a notification.
+
 
+
To report a bug in a test case that has been ''Approved'' (i.e. the test case is in the specification's <code>.../approved/</code> directory), file a Bugzilla bug against the appropriate test suite component in  [http://www.w3.org/Bugs/Public/describecomponents.cgi?product=WebAppsWG WebApps' Bugzilla database]. Please identify the test case(s) that has a bug and clearly describe the bug. (If you do not have write access to the bug database, follow the instructions above for non-approved tests).
+
 
+
== Updating an Approved Test Cases ==
+
 
+
'''An update to an approved test case is only permitted after the group  that approved the test case (i.e. the WebApps Testing Group or the entire WebApps WG) approves the update.'''
+
 
+
If a bug is found in an ''Approved'' test case, please file a bug report against the test case as described above.
+

Latest revision as of 16:46, 17 May 2013

This document has been obsoleted, replaced by WebApps' new Github-based Testing wiki.