Checklist of Checkpoints for "QA Framework: Test Guidelines"

This version:: http://www.w3.org/TR/2004/WD-qaframe-test-20040225/qaframe-test-checklist
This document is an appendix to:: http://www.w3.org/TR/2004/WD-qaframe-test-20040225/
Latest version of "QA Framework: Test Guidelines":: http://www.w3.org/TR/qaframe-test/
Editors:: Dominique Hazaël-Massieux (dom@w3.org); Lofton Henderson (lofton@rockynet.com)
Contributors:: See Acknowledgments in QA Framework: Test Guidelines.

Abstract

This document is an appendix QA Framework: Test Guidelines [QAF-TEST] . It provides a tabular checklist of all checkpoints from the specification guidelines, complete with their priorities. Please refer to QA Framework: Test Guidelines [QAF-TEST] for the full statement and description of the test guidelines and checkpoints, as well as references to related documents and full credits and acknowledgements of contributors to the test guidelines work.

Status of this document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. The latest status of this document series is maintained at the W3C.

This document is derived from and is an appendix to QA Framework: Test Guidelines [QAF-TEST] , which document is a W3C Working Draft (WD), made available by the W3C Quality Assurance (QA) Activity for discussion by W3C members and other interested parties. For more information about the QA Activity, please see the QA Activity statement. Please see the "Status of this document" section of the corresponding specification guidelines [QAF-TEST] , for complete details about the status of the specification guidelines version from which this is extracted and which it accompanies.

Please send comments to www-qa@w3.org, the publicly archived list of the QA Interest Group [QAIG]. Please note that any mail sent to this list will be publicly archived and available. Do not send information you wouldn't want to see distributed, such as private data.

Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or made obsolete by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress".

A list of current W3C Recommendations and other technical documents can be found at http://www.w3.org/TR.

Introduction

This checklist includes all checkpoints from the QA Framework: Test Guidelines [QAF-TEST] presented in a tabular format. The checkpoint priorities are included, and the order of the test guidelines document is maintained.

The presentation is intended to be convenient for organizers and evaluators of QA projects in W3C Working Groups, to facilitate assessing test materials against the checkpoints. The table includes spaces for scoring each checkpoint, "yes" (satisfied), "no" (not satisfied), "n/a" (not applicable).

Priorities

For a description of the meaning of the priorities in the following table, please consult the conformance clause of the test guidelines ([QAF-TEST], chapter 4) .

Checklist table

Nbr	Checkpoint	Priority	Yes	No	N/A
Guideline 1. Define Testing Strategy
1.1	Identify the specifications to be tested.	[Priority 1]
1.2	Define the scope of the test suite.	[Priority 1]
1.3	Define a testing approach.	[Priority 1]
Guideline 2. Identify testable assertions
Nbr	Checkpoint	Priority	Yes	No	N/A
2.1	Identify and list testable assertions.	[Priority 2]
2.2	Tag assertions with essential metadata.	[Priority 1]
Guideline 3. Support test material metadata
Nbr	Checkpoint	Priority	Yes	No	N/A
3.1	Define the metadata to be associated with test materials.	[Priority 1]
3.2	Provide coverage information.	[Priority 2]
3.3	Automate the test materials management process.	[Priority 2]
Guideline 4. Define the process for executing tests
Nbr	Checkpoint	Priority	Yes	No	N/A
4.1	Define the test execution process.	[Priority 1]
4.2	Specify where test results may not be repeatable or reproducible.	[Priority 1]
4.3	Allow for filtering and selection of tests to be executed.	[Priority 1]
4.4	Automate the test execution process.	[Priority 2]
4.5	Integrate results reporting into the automated test execution process.	[Priority 2]
Guideline 5. Document, package, and test the test materials before publication
Nbr	Checkpoint	Priority	Yes	No	N/A
5.1	Include test review status in test materials metadata.	[Priority 1]
5.2	Document the test materials.	[Priority 2]
5.3	Package the test materials into a test suite.	[Priority 1]
5.4	Publish the results of testing the test suite.	[Priority 1]
5.5	Define and publish a mechanism for obtaining feedback on the test materials.	[Priority 2]
5.6	Provide versioning of the test suite.	[Priority 1]
Guideline 6. Define the mechanisms for reporting and publishing test results
Nbr	Checkpoint	Priority	Yes	No	N/A
6.1	Tests must report their outcome in a consistent manner.	[Priority 1]
6.2	Tests must report information on the reason for failure.	[Priority 2]
6.3	Encourage publishing the results of test execution runs.	[Priority 2]

References

QAF-TEST: QA Framework: Test Guidelines, P. Curran, D. Dimitriadis, Eds., W3C Working Draft, 25 February 2004, available at http://www.w3.org/TR/2004/WD-qaframe-test-20040225/.