QA Framework: Specification Guidelines

W3C Working Draft 11 March 2002

This version:

http://www.w3.org/QA/WG/2002/framework-20020311/qaframe-spec

Latest version:

http://www.w3.org/QA/WG/qaframe-spec

Previous version:

(N/A, this is the first published version)

Editors:

Lynne Rosenthal (lynne.rosenthal@nist.gov)

Dimitris Dimitriadis (dimitris@ontologicon.com)

Lofton Henderson (lofton@rockynet.com)

Contributors:

See Acknowledgments.

Abstract

This document provides a common framework for specifying general requirements and definitions concerning conformance and related issues and for representing the anatomy of the document as schemata to facilitate the generation of test materials. It presents traditional best-practice guidelines for writing clearer, more implementatable, and better testable technical reports. This document is one in a family of Framework documents of the Quality Assurance (QA) Activity, which includes the other existing or in-progress specifications: Introduction; Operational Guidelines; and, Test Materials Guidelines.

Status of this document

This version is a WG-only draft.

This section describes the status of this document at the time of its publication. Other documents may supersede this document. The latest status of this document series is maintained at the W3C.

This document is a W3C Working Draft (WD), made available by the W3C Quality Assurance (QA) Activity for discussion by W3C members and other interested parties. For more information about the QA Activity, please see the QA Activity statement.

This version is the first preliminary draft and incorporates the discussions at the QA face-to-face meetings, plus several subsequent QA Working Group [QAWG] teleconferences. There are no other previous drafts. It is expected that updated WD versions of this document will be produced regularly, along with other members of the Framework documents family. Future progression of this document beyond Working Draft is possible, but has not yet been determined.

Please send comments to www-qa@w3.org, the publicly archived list of the QA Interest Group [QAIG]. Please note that any mail sent to this list will be publicly archived and available, do not send information you wouldn't want to see distributed, such as private data.

Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress".

A list of current W3C Recommendations and other technical documents can be found at http://www.w3.org/TR.

1. Introduction
    1.1 Motivation for this Guidelines Document
    1.2 Navigating through this document
    1.3 Terminology
    1.4 Major Themes
2. Guidelines
         G 1. Include a conformance clause
         G 2. Identify what needs to conform
         G 3. Specify how to make a conformance claims
         G 4. discretionary behaviors
         G 5. deprecation
         G 6. profiles
         G 7. levels
         G 8. extensions
         G 9. optional features
         G 10. other stuff?
         G 11. anatomy stuff
3. Conformance
4. Acknowledgments
5. References

1. Introduction

This document provides a common framework for specifying general requirements and definitions concerning conformance and related issues and for representing the structure of the document as schemata to facilitate the generation of test materials. The primary goal is to assist W3C Working Groups (WGs) by providing guidelines for writing clearer, more implementable, and better testable technical reports and specifications. Good specifications lead to better implementations and foster the development of conformance test suites and tools. This document identifies the conformance requirements and statements to be included or addressed in specifications as well as addressing the anatomy of the specification. Conformance requirements are the expression that conveys the criteria to be fulfilled in an implementation of a specification. The conformance requirements are stated in a conformance clause or statements within the specification. The anatomy of the specification pertains to the method for writing the specification using schemata. A specification represented by an XML grammar with sufficient granularity of the information conveyed facilitates the generation of test materials by providing the ability to point to, extract, query, manipulate and/or automatically generate test materials.

This document is part of a family of QA Framework documents designed to improve the quality of W3C specifications as well as their implementation by solidifying and extending current quality practices within the W3C. The QA Framework documents are:

Introduction

Operational Guidelines

Specification Guidelines (this document)

Test Materials Guidelines

The guidelines are intended for all Working Groups as well as developers of conformance materials for W3C specifications. Not only are the Working Groups the consumer of these guidelines they are also key contributors. The guidelines capture the experiences, good practices, activities, and lessons learned of the Working Groups and present them in a comprehensive, cohesive set of documents for all to use and benefit from. The objective is to reuse what works rather than reinvent and to foster consistency across the various Working Group quality activities and deliverables.

1.1 Motivation for this Guidelines Document

Good specifications lead to better implementations.

In particular, it is easier to implement clear specifications, safeguarding the quality of implementations. In addition, if care is given to the specifications, interoperability between W3C technologies is easier to track and assert.

[@@Something about W3C and better specifications.]

Something like The W3C needs to ensure that the deliverables of its WGs are clear, implementable and sound technical specifications with testable assertions and easy generation of Quality Assurance material (tests and suites).

Implementers and user understand what is needed in order to claim conformance

Foster the development of conformance materials

Benefits of testable specifications.

Straightforward to assert conformance
Fairly easy to generate tests and test suites
Given that specification authoring is standardized, it will be straightforward to test interdependencies between specifications (for example it is easier to point to relevant parts of other specifications with regard to technical details where specifications share common features) as well as test interoperability between implementations of different W3C specifications.

Invest in writing granular specifications - allows for detailed control over information extraction, generating tests directly from specs, traceability, graphical interface to specs. (already being done in DOM, XML). efficient and effective test development.

In particular

Information from the specifications is easier to extract if specification authoring involves a standardized set of specification schemas
Tests can be generated directly from the specifications, where needed, to generate primers fot test suites
An added bonus is that a graphical representation of the specification can be generated in parallel to the normative HTML version
Specification granularity allows for more concise reporting when running test suites by pointing directly to the relevant part of the specification

use of a formal (or formal-like) specification language may enable 'validate' specification as well as auto generation of tests or test components

synchronize writing specs with building test materials provides a feedback loop. testing can help find ambiguities, inconsistencies, holes in spec.

In particular

Ambiguities and misinterpretations are easier caught and can be resolved early in the specification writing process
Given the possibility to automatically generate primers for test suites, coverage can be tracked fairly easily and therefore enhanced. Corner casesa re easy to identify and write tests for.

1.2 Navigating through this document

This document not preclude the need to apply Style Manual or Pubrules - rather it focuses on writing testable specifications.

[@@need to read Ops - commitment from WG, staffing and resource allocation, determine test materials, etc.]

This document about what goes into the sepcification with respect to conformance and conformance topics, followed by the anatamy of the specification to enable test development as well as better, testable specifications.

This document employs the WAI (Web Accessibility Initiative) model for representing guidelines or general principles for the development of conformance materials. See, for example, Web Content Accessibility Guidelines. Each guideline includes:

The guideline number

The statement of the guideline

The rationale behind the guideline

A list of checkpoint definitions.

The checkpoint definitions in each guideline define the processes and operations that need to be implemented in order to accomplish the guideline. Each checkpoint definition includes:

The checkpoint number
The statement of the checkpoint
The priority of the checkpoint. Priority 1 checkpoints are highlighted through the use of style sheets.
Optional informative notes, clarifying examples, and cross references to related guidelines or checkpoints.

Each checkpoint is intended to be specific enough so that someone can implement the checkpoint as well as verify that the checkpoint has been satisfied.

1.3 Terminology

The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" will be used as defined in RFC 2119 [RFC2119].

Unusual terms in these framework documents are defined when first used, and most generally useful QA-specific terms will eventually be in the QA Glossary [QA-GLOSSARY].

1.4 Major Themes

Two themes to this document: conformance related content and considerations in specification and anatomy of specification

There is a link between the type of test material to be produced and the anatomy

there is a link between the conformance statements - e.g., who/what must conform and the type of test material (validator, test suite)

Theme 1:Conformance content stuff: general requirements and definitions concerning conformance and related issues. It is intended to contribute towards mutual understanding amongst developers of specifications and conformance test suites and tools. Does not define specific conformance requirements for any specific specification - this is the responsibility of the WG chartered to develop specification. For the developers of specifications to help enable them to develop conformance requirements and to create testable, unambiguous specifications. A conformance clause:

promotes a common understanding of conformance and what is required to claim conformance to a specification

facilitates consistent application of conformance within a specification
facilitates consistent application of conformance across related specifications,
promotes interoperability and open interchange,
encourages the use of applicable conformance test suites,
promotes uniformity in the development of conformance test suite

Theme 2: Anatomy: invest in writing granular specification with careful modeling of the information conveyed

allows for detailed control over information extraction from the spec

allows for generating tests directly from the specs

allows for easy graphic interface to specification

2. Guidelines

Guideline 1. Include a conformance clause

A conformance clause is a part or collection of parts of a specification that defines the requirements, criteria, or conditions to be satisfied by an implementation or application in order to claim conformance. Typically the conformance clause is a high-level description of what is required of implementations and applications. The objective of the conformance clause and its related conformance statements is to provide clear and unambiguous statements, so that the reader knows what is required in order to claim conformance and what is optional.

Checkpoints

Use conformance keywords (RFC2119) consistently and appropriately

Create a separate conformance section

Have the conformance clause exist as a separate section within the specification, so that it is clearly identifiable, allowing a reader to find all conformance provisions from a single starting point.

Checkpoint 1.1. Generate entry for conformance clause table of contents [Priority 2]

Guideline 2. Identify what needs to conform

The conformance clause identifies the "class of products" (i.e., object of the claim) that will be developed, where "class of product" is an implementation, application, service and/or protocol (e.g., content, user agent, authoring tool). In addition to identifying what conforms (i.e., class of products), is describing how conformance can be met. This would be a description of the conformance requirements, conditions and/or criteria for each class of product.

Checkpoints

Checkpoint 2.1. Specify all classes of products [Priority 1]

Checkpoint 2.2. For each class of product, describe the conditions to be satisfied in order to claim conformance [Priority 1]

In addition to specifying the criteria for claiming conformance, it may be appropriate to specify that which is not a requirement.

Checkpoint 2.3. If a class of product is comprised of an integrated set of components, indicate whether conformance to the class of product is equivalent to conformance of all required components considered individually. [Priority 3]

A class of product can consist of several integrated components rather than a single piece of software.

Checkpoint 2.4. If a class of product is comprised of an integrated set of components, specify any restrictions or constraints on the number or types of components that make up the class of product. [Priority 3]

Guideline 3. Specify how to make a conformance claims

A specification may differentiate conformance claims by designating different degrees of conformance in order to apply and group requirements according to profiles or levels or priorities or to indicate the permissibility of extensions.

Checkpoint 3.1. Identify and define all designations of conformance [Priority 1]

(for example, WAI designates 3 levels of conformance A, AA, AAA.....)

Checkpoint 3.2. Provide specific wording of the claim [Priority 2]

Checkpoint 3.3. Impose no restrictions about who can make a claim [Priority 1]

Guideline 4. discretionary behaviors

Guideline 5. deprecation

Guideline 6. profiles

Guideline 7. levels

Guideline 8. extensions

Guideline 9. optional features

Guideline 10. other stuff?

Guideline 11. anatomy stuff

3. Conformance

This section defines conformance of Working Group processes and operations to the requirements of this specification. The requirements of this specification are detailed in the checkpoints of the preceding "Guidelines" chapter of this specification, and apply to the Working Group QA-related documents and deliverables required by this specification.

This section defines three levels of conformance to this specification:

Conformance Level "A": all Priority 1 checkpoints are satisfied;
Conformance Level "AA": all Priority 1 and Priority 2 checkpoints are satisfied;
Conformance Level "AAA": all Priority 1, Priority 2, and Priority 3 checkpoints are satisfied;

A Working Group conforms to the "QA Framework: Process & Operational Guidelines" at Level X (A, AA, or AAA) if the Working Group meets at least all Conformance Level X requirements.

To make an assertion about conformance to this document, specify:

The guidelines title: "QA Framework: Specification Guidelines"
The guidelines URI: http://www.w3.org/TR/2002/qaframe-ops/
The conformance level satisfied: "Level A", "Level AA", or "Level AAA".

Example:

"This specification conforms to W3C's 'QA Framework: Specification Guidelines', available at http://www.w3.org/TR/2002/qaframe-spec/, Level AA."

4. Acknowledgments

The following QA Working Group and Interest Group participants have contributed significantly to the content of this document:

Daniel Dardailler (W3C),
Dimitris Dimitriadis (Ontologicon)
Karl DuBost (W3C),
Peter Fawcett (RealNetworks),
Kirill Gavrylyuk (Microsoft),
Dominique Hazael-Massieux (W3C)
Lofton Henderson (CGM Open)
Ian Jacobs (W3C)
David Marston (Lotus Development Corp)
Jack Morrison (Sun Microsystems)
Olivier Thereaux (W3C),
Lynne Rosenthal (NIST).
Mark Skall (NIST)
Andrew Thackrah (Open Group)

5. References

ISSUES-LIST: QA Activity Issues List, maintained by the QA Working Group, available at http://www.w3.org/QA/WG/#issues.
MATRIX: W3C-wide conformance activity survey covering all the WGs, "The Matrix", available at http://www.w3.org/QA/TheMatrix.
OPS-EXTECH: "QA Framework: Operational Examples and Techniques", Working Draft companion version to this document, available at [...].
PROCESS: W3C Process Document, 19 July 2001, available at http://www.w3.org/Consortium/Process-20010719/.
QA-GLOSSARY: Comprehensive glossary of QA terminology. (Under construction.)
QAF-OPS: "QA Framework: Operational Guidelines", Working Draft companion version to this document, available at [...].
QAF-TEST: "QA Framework: Test Materials Guidelines", not yet published.
QAIG: Quality Assurance Interest Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/IG/.
QAWG: Quality Assurance Working Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/WG/.
RFC2119: Key words for use in RFCs to Indicate Requirement Levels, March 1997, available at http://www.ietf.org/rfc/rfc2119.txt.
SPEC-EXTECH: "QA Framework: Specification Examples and Techniques", not yet published.
STYLE-MAN: "W3C Manual of Style", summarizing the style and publication rules for W3C technical reports, available at http://www.w3.org/2001/06/manual/.
TAXONOMY: QA Activity test taxonomy, a classification scheme for conformance test materials, available at http://www.w3.org/QA/Taxonomy.
TEST-EXTECH: "QA Framework: Test Materials Examples and Techniques", not yet published.
W3C-TR: Location of all published W3C technical reports, see http://www.w3.org/TR/.
WCAG10: Web Content Accessibility Guidelines, version 1.0, W3C Recommendation, 5 May 1999, available at http://www.w3.org/TR/WCAG10/.