This document is an appendix QA Framework: Test Guidelines [QAF-TEST]
. It provides a tabular checklist of all
checkpoints from the specification guidelines, sorted by their priorities. Please refer to QA Framework: Test Guidelines [QAF-TEST]
for the full statement and description of
the test guidelines and checkpoints, as well as references to related
documents and full credits and acknowledgements of contributors to the
test guidelines work.
This section describes the status of this document at the time of its
publication. Other documents may supersede this document. The latest status
of this document series is maintained at the W3C.
This document is derived from and is an appendix to QA Framework: Test Guidelines [QAF-TEST]
, which document is a W3C Working Draft
(WD), made available by the W3C
Quality Assurance (QA) Activity for discussion by W3C members and other
interested parties. For more information about the QA Activity, please see
the QA Activity statement. Please
see the "Status of this document"
section of the corresponding specification guidelines
[QAF-TEST]
, for complete details about the status of the
specification guidelines version from which this is extracted and which it
accompanies.
Please send comments to www-qa@w3.org,
the publicly
archived list of the QA Interest
Group [QAIG]. Please note
that any mail sent to this list will be publicly archived
and available. Do not send information you wouldn't want to see distributed,
such as private data.
Publication of this document does not imply endorsement by the W3C, its
membership or its staff. This is a draft document and may be updated,
replaced, or made obsolete by other documents at any time. It is
inappropriate to use W3C Working Drafts as reference material or to cite them
as other than "work in progress".
A list of current W3C Recommendations and other technical documents can be
found at http://www.w3.org/TR.
This checklist includes all checkpoints from the QA Framework: Test Guidelines [QAF-TEST]
presented in a
tabular format.
The checkpoints are presented by order of their priorities, which makes it an appropriate Implementation Conformance Statement for the guidelines.
The presentation is intended to be convenient for organizers and
evaluators of QA projects in W3C Working Groups, to facilitate assessing
test materials against the checkpoints. The table includes spaces for
scoring each checkpoint, "yes" (satisfied), "no" (not satisfied), "n/a" (not
applicable).
For a description of the meaning of the priorities in the following table,
please consult
the conformance
clause
of the test guidelines ([QAF-TEST], chapter 4)
.
To be A-conformant with the guidelines, the following Priority 1 checkpoints must be fulfilled:
| Nbr | Checkpoint | Yes | No | N/A |
|---|
Guideline 1. Define Testing Strategy |
| 1.1 | Identify the specifications to be tested. | | | |
| 1.2 | Define the scope of the test suite. | | | |
| 1.3 | Define a testing approach. | | | |
Guideline 2. Identify testable assertions |
| 2.2 | Tag assertions with essential metadata. | | | |
Guideline 3. Support test material metadata |
| 3.1 | Define the metadata to be associated with test
materials. | | | |
Guideline 4. Define the process for executing tests |
| 4.1 | Define the test execution process. | | | |
| 4.2 | Specify where test results may not be repeatable or
reproducible. | | | |
| 4.3 | Allow for filtering and selection of tests to be
executed. | | | |
Guideline 5. Document, package, and test the test materials before
publication |
| 5.1 | Include test review status in test materials
metadata. | | | |
| 5.3 | Package the test materials into a test suite. | | | |
| 5.4 | Publish the results of testing the test suite. | | | |
| 5.6 | Provide versioning of the test suite. | | | |
Guideline 6. Define the mechanisms for reporting and publishing test
results |
| 6.1 | Tests must report their outcome in a consistent
manner. | | | |
To be AA-conformant with the guidelines, the following Priority 2 checkpoints must be fulfilled (in addition to the above Priority 1 checkpoints):
| Nbr | Checkpoint | Yes | No | N/A |
|---|
Guideline 2. Identify testable assertions |
| 2.1 | Identify and list testable assertions. | | | |
Guideline 3. Support test material metadata |
| 3.2 | Provide coverage information. | | | |
| 3.3 | Automate the test materials management process. | | | |
Guideline 4. Define the process for executing tests |
| 4.4 | Automate the test execution process. | | | |
| 4.5 | Integrate results reporting into the automated test
execution process. | | | |
Guideline 5. Document, package, and test the test materials before
publication |
| 5.2 | Document the test materials. | | | |
| 5.5 | Define and publish a mechanism for obtaining feedback
on the test materials. | | | |
Guideline 6. Define the mechanisms for reporting and publishing test
results |
| 6.2 | Tests must report information on the reason for
failure. | | | |
| 6.3 | Encourage publishing the results of test execution
runs. | | | |
To be AAA-conformant with the guidelines, the following Priority 3 checkpoints must be fulfilled (in addition to the above Priority 1 and Priority 2 checkpoints):
- QAF-TEST
- QA Framework: Test Guidelines, P. Curran, D. Dimitriadis, Eds., W3C Working Draft, 25
February 2004, available at http://www.w3.org/TR/2004/WD-qaframe-test-20040225/.