W3C

Checklist of Checkpoints for "QA Framework: Test Guidelines"

This version:
http://www.w3.org/TR/2003/WD-qaframe-test-20030516/qaframe-test-ics
This document is an appendix to:
http://www.w3.org/TR/2003/WD-qaframe-test-20030516/
Latest version of "QA Framework: Test Guidelines":
http://www.w3.org/TR/qaframe-test/
Editors:
Dominique HazaŽl-Massieux (dom@w3.org)
Lofton Henderson (lofton@rockynet.com)
Contributors:
See Acknowledgments in QA Framework: Test Guidelines.

Abstract

This document is an appendix QA Framework: Test Guidelines [QAF-TEST] . It provides a tabular checklist of all checkpoints from the specification guidelines, sorted by their priorities. Please refer to QA Framework: Test Guidelines [QAF-TEST] for the full statement and description of the test guidelines and checkpoints, as well as references to related documents and full credits and acknowledgements of contributors to the test guidelines work.

Status of this document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. The latest status of this document series is maintained at the W3C.

This document is derived from and is an appendix to QA Framework: Test Guidelines [QAF-TEST] , which document is a W3C Working Draft (WD), made available by the W3C Quality Assurance (QA) Activity for discussion by W3C members and other interested parties. For more information about the QA Activity, please see the QA Activity statement. Please see the "Status of this document" section of the corresponding specification guidelines [QAF-TEST] , for complete details about the status of the specification guidelines version from which this is extracted and which it accompanies.

Please send comments to www-qa@w3.org, the publicly archived list of the QA Interest Group [QAIG]. Please note that any mail sent to this list will be publicly archived and available. Do not send information you wouldn't want to see distributed, such as private data.

Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or made obsolete by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress".

A list of current W3C Recommendations and other technical documents can be found at http://www.w3.org/TR.


Introduction

This checklist includes all checkpoints from the QA Framework: Test Guidelines [QAF-TEST] presented in a tabular format. The checkpoints are presented by order of their priorities, which makes it an appropriate Implementation Conformance Statement for the guidelines.

The presentation is intended to be convenient for organizers and evaluators of QA projects in W3C Working Groups, to facilitate assessing test materials against the checkpoints. The table includes spaces for scoring each checkpoint, "yes" (satisfied), "no" (not satisfied), "n/a" (not applicable).

Priorities

For a description of the meaning of the priorities in the following table, please consult the conformance clause and the priorities definition section of the test guidelines ([QAF-TEST], chapter 3 and section 1.6) .

Checklist table

Degree A Conformance

To be A-conformant with the guidelines, the following Priority 1 checkpoints must be fulfilled:

Nbr Checkpoint Yes No N/A

Guideline 1. Perform a functional analysis of the specification and determine the testing strategy to be used.

1.1

Define the test suite scope

     
1.2

Identify the specification(s) to be tested.

     
1.3

Analyze the structure of the specification, partition it as appropriate, and determine and document the testing approach to be used for the test suite as a whole and for each partition.

     

Guideline 2. Identify and tag testable assertions within the specification.

2.1

Identify and list assertions within the specification.

     

Guideline 3. Provide a test management system

3.1

The Working Group must provide a test management system

     
3.2

The test management system must associate tests with metadata.

     

Guideline 4. Provide a test framework

4.1

Provide a test framework

     

Guideline 5. Provide results reporting

5.1

Test Materials must support results reporting

     
5.3

Results reporting framework must indicate result status of each test.

     

Guideline 6. Plan for conformance testing

6.1

Organize conformance testing activities.

     

Degree AA Conformance

To be AA-conformant with the guidelines, the following Priority 2 checkpoints must be fulfilled (in addition to the above Priority 1 checkpoints):

Nbr Checkpoint Yes No N/A

Guideline 2. Identify and tag testable assertions within the specification.

2.2

Metadata must be associated with test assertions, enabling test developers, the test-management system, the test-execution framework, and the results-reporting process to make useful distinctions between groups of tests.

     

Guideline 3. Provide a test management system

3.3

Test management system must allow filtering based on metadata.

     
3.4

Test management system must support results.

     

Guideline 4. Provide a test framework

4.2

Prototype the test framework

     
4.3

Automation of testing is encouraged.

     

Guideline 5. Provide results reporting

5.2

Results reporting framework should generate a unified report.

     
5.4

Results reporting framework should allow filtering based on metadata

     

Degree AAA Conformance

To be AAA-conformant with the guidelines, the following Priority 3 checkpoints must be fulfilled (in addition to the above Priority 1 and Priority 2 checkpoints):

Nbr Checkpoint Yes No N/A

Guideline 6. Plan for conformance testing

6.2

Encourage Vendors to publish test results.

     

References

QAF-TEST
QA Framework: Test Guidelines, P. Curran, D. Dimitriadis, P. Fawcett, K. Gavrylyuk, L. Henderson, M. Skall, Eds., W3C Working Draft, May 2003, available at http://www.w3.org/TR/qaframe-test/.