W3C

QA Framework: Test Guidelines

W3C Working Draft 13 December 2002

This version:
http://www.w3.org/QA/WG/2002/12/qaframe-test-20021205
Latest version:
http://www.w3.org/QA/WG/qaframe-test
Previous version:
(None, this is the first published WD)

http://www.w3.org/QA/WG/2002/10/qaframe-test-20021006

Editors:
Kirill Gavrylyuk (kirillg@microsoft.com)
Dimitris Dimitriadis (dimitris@ontologicon.com)
Lofton Henderson (lofton@rockynet.com)
Mark Skall (mark.skall@nist.gov)
Peter Fawcett (pfawcett@real.com)
Contributors:
See Acknowledgments.

Abstract

This document defines a set of common guidelines for conformance test materials for W3C specifications. This document is one in a family of Framework documents of the Quality Assurance (QA) Activity, which includes the other existing or in-progress specifications QA Framework: Introduction,QA Framework: Operational Guidelines, and QA Framework: Specification Guidelines.

Status of this document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. The latest status of this document series is maintained at the W3C.

This document is a W3C Working Draft (WD), made available by the W3C Quality Assurance (QA) Activity for discussion by W3C members and other interested parties. For more information about the QA Activity, please see the QA Activity statement.

This version is the first published Working Draft. It is expected that updated WD versions of this document will be produced regularly, along with other members of the Framework documents family. Future progression of this document beyond Working Draft is possible, but has not yet been determined.

This part of the Framework document family will eventually have an informative accompanying QA Framework: Test Examples and Techniques document. It will illustrate ways in which the guidelines and checkpoints of this document might be satisfied.

The QA Working Group Patent Disclosure page contains details on known patents related to this specification, in conformance with W3C policy requirements.

At this time, the QAWG (QA Working Group) has addressed the contents of the document at a high level, agreeing to the concepts and principles as well the coverage of the guidelines. The QAWG has not, at this time, addressed and achieved consensus of the priority levels. A future version of this document will be accompanied by a "Specification Examples & Techniques" document, which will illustrate the guidelines and checkpoints with case studies, and explain how to satisfy the checkpoints.

Please send comments to www-qa@w3.org, the publicly archived list of the QA Interest Group [QAIG]. Please note that any mail sent to this list will be publicly archived and available, do not send information you wouldn't want to see distributed, such as private data.

Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress".

A list of current W3C Recommendations and other technical documents can be found at http://www.w3.org/TR/.

Table of contents


Introduction

Scope and goals

Class of product and audience

Motivation and expected benefits

Relationship to other specifications

Understanding and using this document

Checkpoint priorities

Terminology

Definitions

This section contains the definitions for all the critical terms used in the guidelines below. This does not substitute the QA Glossary [QA-GLOSSARY], but rather focuses on the most important terms for the Testing guidelines. Some terms in this section have been borrowed or adapted from other specifications.

Contradictory behaviors
Two or more different behaviors unconditionally prescribed by a specification for the same class of implementations under the same circumstances.
Discretionary choices
A value or behavior may be chosen from a well-defined enumerated set of two or more possibilities.
Optional behaviors
A well-defined feature may be supported or not (if supported, then the requirements are clear and unambiguous).
Explicitly undefined behaviors
Specification states that it is open ended and undefined, what set of values an element or attribute may have, or the behaviors of a product that implements a feature.
Test Assertion
A set of premises that are known to be true by definition in the spec.
Test Area
A minimal compound unit in the test suite structure.
Test Framework
A set of utilities, stylesheets and documentation that describe and facilitate development, documentation and use of the tests.
Results Verification
A common testing practice used to determine if a test passes or fails by verification of the test result or output against the expected one.

Guidelines

High level functional analisis of spec to determine stratigy of test develpment. (was G2-G3)

Checkpoints:

Analize the specification and determine how to structure test materials. determine what testing areas the specification is composed of.
Determine how to cover each area. Is only one approach going to be used or will there be more than one.
1.10 develop user scenarios for the specification.

Deep analisys the spec and extract what needs to be tested and how (was parts of G1)

Checkpoints:

Extract assertions/normative langauge and tag according to catigory.

Extract assertions/normative langauge and tag according to catigory - Using catigories provided by patrick In otherwords rather than having explicit checkpoints for each required, optional, depreciated, discressionary, underdefined, etc asserts, have a checkpoint that has all asserts or normative language extracted and then grouped by catigory. It's the same basic idea but it takes 4 checkpoints into one.

Determine level of coverage (depth/bredth) and priority of coverage (what's most important).

Test managemet system (was part of G4)

Checkpoints:

Have a test management system
The system must support meta data like documentation, pass/fail criteria, coverage level, state of test, association back to asserts, and domensions of variability.

Test suite/cases development (was G6)

Checkpoints:

prototype test framework (6.2)

Test execution. (was part of G4)

Checkpoints:

The metadata from the management system must provide sufficient information to allow tests to be exicuted in a perditicble and consistent way.
Automation is encouraged (cross platform stuff goes to extec.)
System must allow for tests to be filtered based on metadata criteia. (This is where the DOV really enters in for a test suite rather than in the analisys part. In the analisys you want to identify them but here is where you really care)
The test execution process should save output from tests for analisys (pass/fail, what cases failed, and logs if relevent).

Result reporting (G5)

Checkpoints:

must support result reporting (5.1)
should create reports as a unified package (for example like a web page (5.3))
must indicate what passed and failed.
should be automated if possible
should supported filtering and comparison of results

Conformance testing. (G7)

Checkpoints:

Encourage vendors to use the test suite. (p1)
Encourage vendors to publish results (p3)

Conformance

This section defines conformance of Working Group processes and operations to the requirements of this specification. The requirements of this specification are detailed in the checkpoints of the preceding "Guidelines" chapter and apply to the Working Group QA-related documents and deliverables required by this specification.

Conformance definition

This section defines three levels of conformance to this specification:

A Working Group conforms to the "QA Framework: Test Guidelines" at Level X (A, AA, or AAA) if the Working Group meets at least all Conformance Level X requirements.

To make an assertion about conformance to this document, specify:

Example:

The Test Suite for X module, version 2.1 of this Working Group, conforms to the W3C's 'QA Framework: Test Guidelines' version 1.0, available at http://www.w3.org/TR/2002/qaframe-test/, Level AA as determ ined on January 1, 2003.

Conformance disclaimer

The checkpoints of this specification present verifiable conformance requirements about the quality of the test materials developed or adopted by the Working Group. As with any verifiable test requirements, users should be aware that:

  1. Passing all of the requirements to achieve a given conformance level -- A, AA, or AAA -- does not guarantee that the quality of the subject test materials are well-suited to or will achieve their intended purposes.
  2. Failing to achieve level A conformance does not mean that the subject test materials are necessarily deficient to their intended purposes. It means that the test materials fail one or more checkpoints that best-practice experience has shown to facilitate and enable successful development, maintenance and use of the test materials.

Acknowledgments

The following QA Working Group and Interest Group participants have contributed significantly to the content of this document:

References

EXTERN-TA
QA activity email thread about third-party participation in test materials production, available at http://lists.w3.org/Archives/Public/www-qa/2001Oct/0060.html.
MATRIX
W3C-wide conformance activity survey covering all the Working Groups, "The Matrix", available at http://www.w3.org/QA/TheMatrix.
PROCESS
W3C Process Document, 19 July 2001, available at http://www.w3.org/Consortium/Process-20010719/.
TAXONOMY
QA Activity test taxonomy, a classification scheme for conformance test materials, available at http://www.w3.org/QA/Taxonomy.
QA-GLOSSARY
A comprehensive glossary of QA terms, maintained by the QA Working Group. (Initial version under construction.)
QAIG
Quality Assurance Interest Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/IG/.
QAWG
Quality Assurance Working Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/WG/.
DOM Working Group TS
Process document for DOM Working Group Test suite, available at http://www.w3.org/2002/01/DOMConformanceTS-Process-20020115.
REC-TRACK
Stages and milestones in the W3C Recommendation Track, per the Process Document (Process Document is available at http://www.w3.org/Consortium/Process-20010719/, see section 5.2).
RFC2119
Key words for use in RFCs to Indicate Requirement Levels, March 1997, available at http://www.ietf.org/rfc/rfc2119.txt.
SVGTEST
SVG Working Group's test suite resource page, which may be found at http://www.w3.org/Graphics/SVG/Test/.
WCAG10
Web Content Accessibility Guidelines, version 1.0, W3C Recommendation, 5 May 1999, available at http://www.w3.org/TR/WCAG10/.
WG-QA-RANGE
Email proposal by David Marston, on the QA public mail list, for range of Working Group commitment levels to conformance test materials production, available at http://lists.w3.org/Archives/Public/www-qa/2001Apr/0004.html.
XMLTEST
OASIS XML Conformance TC's XML test suite resource page, which may be found at http://www.oasis-open.org/committees/xml-conformance/.
XSLT-TEST
OASIS XML Conformance TC's XSLT/Xpath test suite resource page, which may be found at http://www.oasis-open.org/committees/xml-conformance/.
QAF-TEST
QA Framework: Test Materials Guidelines (not yet published).
QAF-SPEC
"QA Framework: Specification Guidelines", Working Draft companion version to this document, available at [...].
XQuery-use-case
"XML Query Use Cases
SOAP-test
SOAP Version 1.2 Specification Assertions and Test Collection
SOAP12-1
SOAP Version 1.2 Part 1
SOAP12-2
SOAP Version 1.2 Part 2
XSD-TEST
W3C XML Schema Test Collection
XSLT-DISC
List of Discretionary items produced by OASIS XSLT/XPath Technical Committee

Change History

3-12-2003

Added new outline to document commented out a number of sections that need editing.

12-05-2002

Edited, improved the Introduction(goals, motivation, document's structure

Updated the definition of the checkpoint's Priorities

corrected abstract, SOT

Changed the goal of the document and wording of the checkpoints/guidelines to focus it on testing strategy, moving all the tips on tactics to be EX-TECH

Added examples to most of the checkpoints

incorporated all the editors comments

Rewrote several checkpoints in the Gd 5 - defined results verification and reporting as part of the test framework to remove redundant checkpoints

Rewrote guideline 6 and 7 to focus on strategy of test development and testing, rather then on tactics.

Updated the conformance section

09-12-2002

Expanded introduction, added motivation, etc...

Added examples to the checkpoints in the Gd1,2,3

[MS] Changed the text of many checkpoints to make them verifiable

[DD] First pass on Introduction, added more text to the checkpoints in the Gd 3-5

07-01-2002

Fixed definitions of priorities

Fixed the glitch with the "Test Areas" guideline

Added clarification to Ck 1.1, 1.2, 1.5 (removed "vague"), 1.6

06-17-2002

Added short prose to each checkpoint

06-12-2002

First draft outline