QA Framework: Specification Guidelines

W3C Working Draft 11 March 2002

This version:
Latest version:
Previous version:
(N/A, this is the first published version)
Lynne Rosenthal (lynne.rosenthal@nist.gov)

Dimitris Dimitriadis (dimitris@ontologicon.com)

Lofton Henderson (lofton@rockynet.com)

See Acknowledgments.


This document provides a common framework for specifying general requirements and definitions concerning conformance and related issues and for representing the anatomy of the document as schemata to facilitate the generation of test materials. It presents traditional best-practice guidelines for writing clearer, more implementatable, and better testable technical reports. This document is one in a family of Framework documents of the Quality Assurance (QA) Activity, which includes the other existing or in-progress specifications: Introduction; Operational Guidelines; and, Test Materials Guidelines.

Status of this document

This version is a WG-only draft.

This section describes the status of this document at the time of its publication. Other documents may supersede this document. The latest status of this document series is maintained at the W3C.

This document is a W3C Working Draft (WD), made available by the W3C Quality Assurance (QA) Activity for discussion by W3C members and other interested parties. For more information about the QA Activity, please see the QA Activity statement.

This version is the first preliminary draft and incorporates the discussions at the QA face-to-face meetings, plus several subsequent QA Working Group [QAWG] teleconferences. There are no other previous drafts. It is expected that updated WD versions of this document will be produced regularly, along with other members of the Framework documents family. Future progression of this document beyond Working Draft is possible, but has not yet been determined.

Please send comments to www-qa@w3.org, the publicly archived list of the QA Interest Group [QAIG]. Please note that any mail sent to this list will be publicly archived and available, do not send information you wouldn't want to see distributed, such as private data.

Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress".

A list of current W3C Recommendations and other technical documents can be found at http://www.w3.org/TR.

Table of contents

1. Introduction
1.1 Motivation for this Guidelines Document
1.2 Navigating through this document
1.3 Terminology
1.4 Major Themes
2. Guidelines
G 1. Include a conformance clause
G 2. Identify what needs to conform
G 3. Specify how to make a conformance claims
G 4. discretionary behaviors
G 5. deprecation
G 6. profiles
G 7. levels
G 8. extensions
G 9. optional features
G 10. other stuff?
G 11. anatomy stuff
3. Conformance
4. Acknowledgments
5. References

1. Introduction

This document provides a common framework for specifying general requirements and definitions concerning conformance and related issues and for representing the structure of the document as schemata to facilitate the generation of test materials. The primary goal is to assist W3C Working Groups (WGs) by providing guidelines for writing clearer, more implementable, and better testable technical reports and specifications. Good specifications lead to better implementations and foster the development of conformance test suites and tools. This document identifies the conformance requirements and statements to be included or addressed in specifications as well as addressing the anatomy of the specification. Conformance requirements are the expression that conveys the criteria to be fulfilled in an implementation of a specification. The conformance requirements are stated in a conformance clause or statements within the specification. The anatomy of the specification pertains to the method for writing the specification using schemata. A specification represented by an XML grammar with sufficient granularity of the information conveyed facilitates the generation of test materials by providing the ability to point to, extract, query, manipulate and/or automatically generate test materials.

This document is part of a family of QA Framework documents designed to improve the quality of W3C specifications as well as their implementation by solidifying and extending current quality practices within the W3C. The QA Framework documents are:

The guidelines are intended for all Working Groups as well as developers of conformance materials for W3C specifications. Not only are the Working Groups the consumer of these guidelines they are also key contributors. The guidelines capture the experiences, good practices, activities, and lessons learned of the Working Groups and present them in a comprehensive, cohesive set of documents for all to use and benefit from. The objective is to reuse what works rather than reinvent and to foster consistency across the various Working Group quality activities and deliverables.

1.1 Motivation for this Guidelines Document

Good specifications lead to better implementations.

In particular, it is easier to implement clear specifications, safeguarding the quality of implementations. In addition, if care is given to the specifications, interoperability between W3C technologies is easier to track and assert.

[@@Something about W3C and better specifications.]

Something like The W3C needs to ensure that the deliverables of its WGs are clear, implementable and sound technical specifications with testable assertions and easy generation of Quality Assurance material (tests and suites).

Implementers and user understand what is needed in order to claim conformance

Foster the development of conformance materials

Benefits of testable specifications.

Invest in writing granular specifications - allows for detailed control over information extraction, generating tests directly from specs, traceability, graphical interface to specs. (already being done in DOM, XML). efficient and effective test development.

In particular

use of a formal (or formal-like) specification language may enable 'validate' specification as well as auto generation of tests or test components

synchronize writing specs with building test materials provides a feedback loop. testing can help find ambiguities, inconsistencies, holes in spec.

In particular

1.2 Navigating through this document

This document not preclude the need to apply Style Manual or Pubrules - rather it focuses on writing testable specifications.

[@@need to read Ops - commitment from WG, staffing and resource allocation, determine test materials, etc.]

This document about what goes into the sepcification with respect to conformance and conformance topics, followed by the anatamy of the specification to enable test development as well as better, testable specifications.

This document employs the WAI (Web Accessibility Initiative) model for representing guidelines or general principles for the development of conformance materials. See, for example, Web Content Accessibility Guidelines. Each guideline includes:

The checkpoint definitions in each guideline define the processes and operations that need to be implemented in order to accomplish the guideline. Each checkpoint definition includes:

Each checkpoint is intended to be specific enough so that someone can implement the checkpoint as well as verify that the checkpoint has been satisfied.

1.3 Terminology

The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" will be used as defined in RFC 2119 [RFC2119].

Unusual terms in these framework documents are defined when first used, and most generally useful QA-specific terms will eventually be in the QA Glossary [QA-GLOSSARY].

1.4 Major Themes

Two themes to this document: conformance related content and considerations in specification and anatomy of specification

There is a link between the type of test material to be produced and the anatomy

there is a link between the conformance statements - e.g., who/what must conform and the type of test material (validator, test suite)

Theme 1:Conformance content stuff: general requirements and definitions concerning conformance and related issues. It is intended to contribute towards mutual understanding amongst developers of specifications and conformance test suites and tools. Does not define specific conformance requirements for any specific specification - this is the responsibility of the WG chartered to develop specification. For the developers of specifications to help enable them to develop conformance requirements and to create testable, unambiguous specifications. A conformance clause:

Theme 2: Anatomy: invest in writing granular specification with careful modeling of the information conveyed

2. Guidelines

Guideline 1. Include a conformance clause

A conformance clause is a part or collection of parts of a specification that defines the requirements, criteria, or conditions to be satisfied by an implementation or application in order to claim conformance. Typically the conformance clause is a high-level description of what is required of implementations and applications. The objective of the conformance clause and its related conformance statements is to provide clear and unambiguous statements, so that the reader knows what is required in order to claim conformance and what is optional.


Use conformance keywords (RFC2119) consistently and appropriately

Create a separate conformance section

Have the conformance clause exist as a separate section within the specification, so that it is clearly identifiable, allowing a reader to find all conformance provisions from a single starting point.

Checkpoint 1.1. Generate entry for conformance clause table of contents [Priority 2]

Guideline 2. Identify what needs to conform

The conformance clause identifies the "class of products" (i.e., object of the claim) that will be developed, where "class of product" is an implementation, application, service and/or protocol (e.g., content, user agent, authoring tool). In addition to identifying what conforms (i.e., class of products), is describing how conformance can be met. This would be a description of the conformance requirements, conditions and/or criteria for each class of product.


Checkpoint 2.1. Specify all classes of products [Priority 1]

Checkpoint 2.2. For each class of product, describe the conditions to be satisfied in order to claim conformance [Priority 1]

In addition to specifying the criteria for claiming conformance, it may be appropriate to specify that which is not a requirement.

Checkpoint 2.3. If a class of product is comprised of an integrated set of components, indicate whether conformance to the class of product is equivalent to conformance of all required components considered individually. [Priority 3]

A class of product can consist of several integrated components rather than a single piece of software.

Checkpoint 2.4. If a class of product is comprised of an integrated set of components, specify any restrictions or constraints on the number or types of components that make up the class of product. [Priority 3]

Guideline 3. Specify how to make a conformance claims

A specification may differentiate conformance claims by designating different degrees of conformance in order to apply and group requirements according to profiles or levels or priorities or to indicate the permissibility of extensions.

Checkpoint 3.1. Identify and define all designations of conformance [Priority 1]

(for example, WAI designates 3 levels of conformance A, AA, AAA.....)

Checkpoint 3.2. Provide specific wording of the claim [Priority 2]

Checkpoint 3.3. Impose no restrictions about who can make a claim [Priority 1]

Guideline 4. discretionary behaviors

Guideline 5. deprecation

Guideline 6. profiles

Guideline 7. levels

Guideline 8. extensions

Guideline 9. optional features

Guideline 10. other stuff?

Guideline 11. anatomy stuff

3. Conformance

This section defines conformance of Working Group processes and operations to the requirements of this specification. The requirements of this specification are detailed in the checkpoints of the preceding "Guidelines" chapter of this specification, and apply to the Working Group QA-related documents and deliverables required by this specification.

This section defines three levels of conformance to this specification:

A Working Group conforms to the "QA Framework: Process & Operational Guidelines" at Level X (A, AA, or AAA) if the Working Group meets at least all Conformance Level X requirements.

To make an assertion about conformance to this document, specify:


"This specification conforms to W3C's 'QA Framework: Specification Guidelines', available at http://www.w3.org/TR/2002/qaframe-spec/, Level AA."

4. Acknowledgments

The following QA Working Group and Interest Group participants have contributed significantly to the content of this document:

5. References

QA Activity Issues List, maintained by the QA Working Group, available at http://www.w3.org/QA/WG/#issues.
W3C-wide conformance activity survey covering all the WGs, "The Matrix", available at http://www.w3.org/QA/TheMatrix.
"QA Framework: Operational Examples and Techniques", Working Draft companion version to this document, available at [...].
W3C Process Document, 19 July 2001, available at http://www.w3.org/Consortium/Process-20010719/.
Comprehensive glossary of QA terminology. (Under construction.)
"QA Framework: Operational Guidelines", Working Draft companion version to this document, available at [...].
"QA Framework: Test Materials Guidelines", not yet published.
Quality Assurance Interest Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/IG/.
Quality Assurance Working Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/WG/.
Key words for use in RFCs to Indicate Requirement Levels, March 1997, available at http://www.ietf.org/rfc/rfc2119.txt.
"QA Framework: Specification Examples and Techniques", not yet published.
"W3C Manual of Style", summarizing the style and publication rules for W3C technical reports, available at http://www.w3.org/2001/06/manual/.
QA Activity test taxonomy, a classification scheme for conformance test materials, available at http://www.w3.org/QA/Taxonomy.
"QA Framework: Test Materials Examples and Techniques", not yet published.
Location of all published W3C technical reports, see http://www.w3.org/TR/.
Web Content Accessibility Guidelines, version 1.0, W3C Recommendation, 5 May 1999, available at http://www.w3.org/TR/WCAG10/.