Checklist of Checkpoints for "QA Framework: Test Guidelines"

This version:: http://www.w3.org/TR/2002/WD-qaframe-test-20021220/qaframe-test-checklist
This document is an appendix to:: http://www.w3.org/TR/2002/WD-qaframe-test-20021220/
Latest version of "QA Framework: Test Guidelines":: http://www.w3.org/TR/qaframe-test/
Editors:: Dominique Hazaël-Massieux (dom@w3.org); Lofton Henderson (lofton@rockynet.com)
Contributors:: See Acknowledgments in QA Framework: Test Guidelines.

Abstract

This document is an appendix QA Framework: Test Guidelines [QAF-TEST] . It provides a tabular checklist of all checkpoints from the specification guidelines, complete with their priorities. Please refer to QA Framework: Test Guidelines [QAF-TEST] for the full statement and description of the test guidelines and checkpoints, as well as references to related documents and full credits and acknowledgements of contributors to the test guidelines work.

Status of this document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. The latest status of this document series is maintained at the W3C.

This document is derived from and is an appendix to QA Framework: Test Guidelines [QAF-TEST] , which document is a W3C Working Draft (WD), made available by the W3C Quality Assurance (QA) Activity for discussion by W3C members and other interested parties. For more information about the QA Activity, please see the QA Activity statement. Please see the "Status of this document" section of the corresponding specification guidelines [QAF-TEST] , for complete details about the status of the specification guidelines version from which this is extracted and which it accompanies.

Please send comments to www-qa@w3.org, the publicly archived list of the QA Interest Group [QAIG]. Please note that any mail sent to this list will be publicly archived and available. Do not send information you wouldn't want to see distributed, such as private data.

Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or made obsolete by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress".

A list of current W3C Recommendations and other technical documents can be found at http://www.w3.org/TR.

Introduction

This checklist includes all checkpoints from the QA Framework: Test Guidelines [QAF-TEST] presented in a tabular format. The checkpoint priorities are included, and the order of the test guidelines document is maintained.

The presentation is intended to be convenient for organizers and evaluators of QA projects in W3C Working Groups, to facilitate assessing @@@ against the checkpoints. The table includes spaces for scoring each checkpoint, "yes" (satisfied), "no" (not satisfied), "n/a" (not applicable).

Priorities

For a description of the meaning of the priorities in the following table, please consult the conformance clause of the test guidelines ([QAF-TEST], chapter 3) .

Checklist table

Guideline 1. Provide analysis of the specification(s).
Nbr	Checkpoint	Priority	Yes	No	N/A
1.1	Identify the target set of specifications that are being tested.	[Priority 1]
1.2	Identifytest assertions from the target set of specifications.	[Priority 1]
1.3	Group test assertions by levels, profiles, modules, etc.	[Priority 1]
1.4	Identify those test assertions that are part of the conformance criteria	[Priority 1]
1.5	Identify all the discretionary choices defined in the specification	[Priority 1]
1.6	Identify optional behaviors in the specification	[Priority 2]
1.7	Identify behaviors that are undefined or defined ambiguously in the specification	[Priority 2]
1.8	Identify explicitly undefined behaviors in the specification	[Priority 2]
1.9	Identify contradictory behaviors in the specification	[Priority 2]
1.10	List the user scenarios for the specification	[Priority 1]
Guideline 2. Declare the structure of the test suite.
Nbr	Checkpoint	Priority	Yes	No	N/A
2.1	Document the structure for the test suite.	[Priority 1]
2.2	Provide mapping between the test suite structure and the specification structure.	[Priority 1]
Guideline 3. Document the testing methodology.
Nbr	Checkpoint	Priority	Yes	No	N/A
3.1	For each test area, identify the testing approach	[Priority 1]
3.2	Identify publicly available testing techniques. List the publicly available testing techniques that have been reused	[Priority 1]
Guideline 4. Provide the test automation and framework
Nbr	Checkpoint	Priority	Yes	No	N/A
4.1	List available test frameworks and applicable automation. Identify available test frameworks used. If none, justify why new frameworks are needed and existing ones could not be used.	[Priority 1]
4.2	Ensure the framework and automation are platform independent. Demonstrate on 3 platforms. Ensure that the framework and automation are built using open standards.	[Priority 1]
4.3	Ensure the framework and automation are applicable to any product or content that implements the specification. Demonstrate with three products or contents. Ensure that the framework and automation are built using open standards.	[Priority 2]
4.4	Ensure the framework makes it easy to add tests for any of the specification areas. Demonstrate, through an example, how tests are added to a specification area.	[Priority 2]
4.5	Ensure the ease of use for the test automation. Document how the test automation is used.	[Priority 1]
4.6	Ensure the framework allows for specification versioning and errata levels. Explain how specification versioning and errata levels are accommodated by the test framework	[Priority 2]
4.7	Ensure the framework accounts for choices allowed by discretionary choices in the specification. Explain how discretionary behaviors are accommodated by the framework.	[Priority 2]
4.8	Ensure the framework includes tests for the optional behaviors defined in the specification. Explain how optional behaviors are accommodated by the framework.	[Priority 3]
4.9	Ensure the test framework accommodates profiles, modules, product classes, and levels if they are used in the specification. For each dimension of variability used in the specification, demonstratehow the framework allows tests to be filtered by the dimension.	[Priority 2]
4.10	Ensure the framework accommodates the conformance policy defined in the specification. Demonstrate how the framework allows tests to be filtered by levels.	[Priority 1]
4.11	Ensure the framework supports test results verification. Demonstrate results verification by testing three products.	[Priority 1]
4.12	Ensure the framework allows the tests to be documented. Explain how to document the tests within the framework.	[Priority 2]
4.13	Ensure the framework has proper test case management. Demonstrate how at least one of the following test case management functions is accomplished within the framework: managing additions, managing removals, filtering by various criteria	[Priority 3]
4.14	Ensure the framework allows specification coverage to be measured. Demonstrate the above by mapping a list of tests to the list of test assertions, grouped by areas	[Priority 2]
Guideline 5. Provide the results reporting framework
Nbr	Checkpoint	Priority	Yes	No	N/A
5.1	Ensure that the test framework supports reporting the results.	[Priority 1]
5.2	Ensure the ease of use for results reporting. Demonstrate that the results reporting has sorting and filtering capabilities.	[Priority 1]
5.3	Document how the results reporting allows results to be exported in a self-contained format suitable to publication on the web.	[Priority 2]
5.4	Demonstrate that the results reporting provides details on failures (logs) sufficient to investigate.	[Priority 3]
5.5	Document how the results reporting allows for history/storing analysis of comments	[Priority 3]
Guideline 6. Plan for tests development
Nbr	Checkpoint	Priority	Yes	No	N/A
6.1	Define priorities for the test areas	[Priority 2]
6.2	Introduce a mechanism for the early feedback on the test suite architecture and test framework. Document and publish the mechanism to the intended audience.	[Priority 3]
6.3	Ensure regular specification coverage analysis. Provide the schedule for specification coverage analysis.	[Priority 2]
Guideline 7. Plan for conformance testing
Nbr	Checkpoint	Priority	Yes	No	N/A
7.1	Document the plan for engaging vendors of implementations in conformance testing activities.	[Priority 1]
7.2	Encourage Vendors to publish test results for their products by reserving a special space where information pertaining to test results can be maintained.	[Priority 3]

Guideline 1. Provide analysis of the specification(s).

Nbr

Checkpoint

Priority

Yes

N/A

1.1

Identify the target set of specifications that are being tested.

[Priority 1]

1.2

Identifytest assertions from the target set of specifications.

[Priority 1]

1.3

Group test assertions by levels, profiles, modules, etc.

[Priority 1]

1.4

Identify those test assertions that are part of the conformance criteria

[Priority 1]

1.5

Identify all the discretionary choices defined in the specification

[Priority 1]

1.6

Identify optional behaviors in the specification

[Priority 2]

1.7

Identify behaviors that are undefined or defined ambiguously in the specification

[Priority 2]

1.8

Identify explicitly undefined behaviors in the specification

[Priority 2]

1.9

Identify contradictory behaviors in the specification

[Priority 2]

1.10

List the user scenarios for the specification

[Priority 1]

Guideline 2. Declare the structure of the test suite.

Nbr

Checkpoint

Priority

Yes

N/A

2.1

Document the structure for the test suite.

[Priority 1]

2.2

Provide mapping between the test suite structure and the specification structure.

[Priority 1]

Guideline 3. Document the testing methodology.

Nbr

Checkpoint

Priority

Yes

N/A

3.1

For each test area, identify the testing approach

[Priority 1]

3.2

Identify publicly available testing techniques. List the publicly available testing techniques that have been reused

[Priority 1]

Guideline 4. Provide the test automation and framework

Nbr

Checkpoint

Priority

Yes

N/A

4.1

List available test frameworks and applicable automation. Identify available test frameworks used. If none, justify why new frameworks are needed and existing ones could not be used.

[Priority 1]

4.2

Ensure the framework and automation are platform independent. Demonstrate on 3 platforms. Ensure that the framework and automation are built using open standards.

[Priority 1]

4.3

Ensure the framework and automation are applicable to any product or content that implements the specification. Demonstrate with three products or contents. Ensure that the framework and automation are built using open standards.

[Priority 2]

4.4

Ensure the framework makes it easy to add tests for any of the specification areas. Demonstrate, through an example, how tests are added to a specification area.

[Priority 2]

4.5

Ensure the ease of use for the test automation. Document how the test automation is used.

[Priority 1]

4.6

Ensure the framework allows for specification versioning and errata levels. Explain how specification versioning and errata levels are accommodated by the test framework

[Priority 2]

4.7

Ensure the framework accounts for choices allowed by discretionary choices in the specification. Explain how discretionary behaviors are accommodated by the framework.

[Priority 2]

4.8

Ensure the framework includes tests for the optional behaviors defined in the specification. Explain how optional behaviors are accommodated by the framework.

[Priority 3]

4.9

Ensure the test framework accommodates profiles, modules, product classes, and levels if they are used in the specification. For each dimension of variability used in the specification, demonstratehow the framework allows tests to be filtered by the dimension.

[Priority 2]

4.10

Ensure the framework accommodates the conformance policy defined in the specification. Demonstrate how the framework allows tests to be filtered by levels.

[Priority 1]

4.11

Ensure the framework supports test results verification. Demonstrate results verification by testing three products.

[Priority 1]

4.12

Ensure the framework allows the tests to be documented. Explain how to document the tests within the framework.

[Priority 2]

4.13

Ensure the framework has proper test case management. Demonstrate how at least one of the following test case management functions is accomplished within the framework: managing additions, managing removals, filtering by various criteria

[Priority 3]

4.14

Ensure the framework allows specification coverage to be measured. Demonstrate the above by mapping a list of tests to the list of test assertions, grouped by areas

[Priority 2]

Guideline 5. Provide the results reporting framework

Nbr

Checkpoint

Priority

Yes

N/A

5.1

Ensure that the test framework supports reporting the results.

[Priority 1]

5.2

Ensure the ease of use for results reporting. Demonstrate that the results reporting has sorting and filtering capabilities.

[Priority 1]

5.3

Document how the results reporting allows results to be exported in a self-contained format suitable to publication on the web.

[Priority 2]

5.4

Demonstrate that the results reporting provides details on failures (logs) sufficient to investigate.

[Priority 3]

5.5

Document how the results reporting allows for history/storing analysis of comments

[Priority 3]

Guideline 6. Plan for tests development

Nbr

Checkpoint

Priority

Yes

N/A

6.1

Define priorities for the test areas

[Priority 2]

6.2

Introduce a mechanism for the early feedback on the test suite architecture and test framework. Document and publish the mechanism to the intended audience.

[Priority 3]

6.3

Ensure regular specification coverage analysis. Provide the schedule for specification coverage analysis.

[Priority 2]

Guideline 7. Plan for conformance testing

Nbr

Checkpoint

Priority

Yes

N/A

7.1

Document the plan for engaging vendors of implementations in conformance testing activities.

[Priority 1]

7.2

Encourage Vendors to publish test results for their products by reserving a special space where information pertaining to test results can be maintained.

[Priority 3]

Checklist of Checkpoints for "QA Framework: Test Guidelines"

Abstract

Status of this document

Introduction

Priorities

Checklist table

References