Copyright © 2003-004 W3C® (MIT, ERCIM, Keio), All Rights Reserved. W3C liability, trademark, document use and software licensing rules apply.
This document is an appendix QA Framework: Test Guidelines [QAF-TEST] . It provides a tabular checklist of all checkpoints from the specification guidelines, sorted by their priorities. Please refer to QA Framework: Test Guidelines [QAF-TEST] for the full statement and description of the test guidelines and checkpoints, as well as references to related documents and full credits and acknowledgements of contributors to the test guidelines work.
This section describes the status of this document at the time of its publication. Other documents may supersede this document. The latest status of this document series is maintained at the W3C.
This document is derived from and is an appendix to QA Framework: Test Guidelines [QAF-TEST] , which document is a W3C Working Draft (WD), made available by the W3C Quality Assurance (QA) Activity for discussion by W3C members and other interested parties. For more information about the QA Activity, please see the QA Activity statement. Please see the "Status of this document" section of the corresponding specification guidelines [QAF-TEST] , for complete details about the status of the specification guidelines version from which this is extracted and which it accompanies.
Please send comments to www-qa@w3.org, the publicly archived list of the QA Interest Group [QAIG]. Please note that any mail sent to this list will be publicly archived and available. Do not send information you wouldn't want to see distributed, such as private data.
Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or made obsolete by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress".
A list of current W3C Recommendations and other technical documents can be found at http://www.w3.org/TR.
This checklist includes all checkpoints from the QA Framework: Test Guidelines [QAF-TEST] presented in a tabular format. The checkpoints are presented by order of their priorities, which makes it an appropriate Implementation Conformance Statement for the guidelines.
The presentation is intended to be convenient for organizers and evaluators of QA projects in W3C Working Groups, to facilitate assessing test materials against the checkpoints. The table includes spaces for scoring each checkpoint, "yes" (satisfied), "no" (not satisfied), "n/a" (not applicable).
For a description of the meaning of the priorities in the following table, please consult the conformance clause of the test guidelines ([QAF-TEST], chapter 4) .
To be A-conformant with the guidelines, the following Priority 1 checkpoints must be fulfilled:
Nbr | Checkpoint | Yes | No | N/A |
---|---|---|---|---|
Guideline 1. Define Testing Strategy | ||||
1.1 | Identify the specifications to be tested. | |||
1.2 | Define the scope of the test suite. | |||
1.3 | Define a testing approach. | |||
Guideline 2. Identify testable assertions | ||||
2.2 | Tag assertions with essential metadata. | |||
Guideline 3. Support test material metadata | ||||
3.1 | Define the metadata to be associated with test materials. | |||
Guideline 4. Define the process for executing tests | ||||
4.1 | Define the test execution process. | |||
4.2 | Specify where test results may not be repeatable or reproducible. | |||
4.3 | Allow for filtering and selection of tests to be executed. | |||
Guideline 5. Document, package, and test the test materials before publication | ||||
5.1 | Include test review status in test materials metadata. | |||
5.3 | Package the test materials into a test suite. | |||
5.4 | Publish the results of testing the test suite. | |||
5.6 | Provide versioning of the test suite. | |||
Guideline 6. Define the mechanisms for reporting and publishing test results | ||||
6.1 | Tests must report their outcome in a consistent manner. |
To be AA-conformant with the guidelines, the following Priority 2 checkpoints must be fulfilled (in addition to the above Priority 1 checkpoints):
Nbr | Checkpoint | Yes | No | N/A |
---|---|---|---|---|
Guideline 2. Identify testable assertions | ||||
2.1 | Identify and list testable assertions. | |||
Guideline 3. Support test material metadata | ||||
3.2 | Provide coverage information. | |||
3.3 | Automate the test materials management process. | |||
Guideline 4. Define the process for executing tests | ||||
4.4 | Automate the test execution process. | |||
4.5 | Integrate results reporting into the automated test execution process. | |||
Guideline 5. Document, package, and test the test materials before publication | ||||
5.2 | Document the test materials. | |||
5.5 | Define and publish a mechanism for obtaining feedback on the test materials. | |||
Guideline 6. Define the mechanisms for reporting and publishing test results | ||||
6.2 | Tests must report information on the reason for failure. | |||
6.3 | Encourage publishing the results of test execution runs. |
To be AAA-conformant with the guidelines, the following Priority 3 checkpoints must be fulfilled (in addition to the above Priority 1 and Priority 2 checkpoints):
Nbr | Checkpoint | Yes | No | N/A |
---|