W3C

QA Framework: Specification Guidelines

W3C Working Draft 18 October 2002

This version:
http://www.w3.org/QA/WG/2002/10/qaframe-spec-20021025
Latest version:
http://www.w3.org/QA/WG/qaframe-spec
Previous version:
http://www.w3.org/QA/WG/2002/10/qaframe-spec
Editors:
Dominique HazaŽl-Massieux (dom@w3.org)
Lofton Henderson (lofton@rockynet.com)
Lynne Rosenthal (lynne.rosenthal@nist.gov)
Dimitris Dimitriadis (dimitris@ontologicon.com)
Kirill Gavrylyuk (kirillg@microsoft.com)
Contributors:
See Acknowledgments.

Abstract

The principal goal of this document is to help W3C Working Groups to write clearer, more implementable, and better testable technical reports. It both provides a common framework for specifying conformance requirements and definitions, and also addresses the representation of specifications (technical reports) as schemata, both of which facilitate the generation of test materials. The material is presented as a set of organizing guidelines and verifiable checkpoints. This document is one in a family of Framework documents of the Quality Assurance (QA) Activity, which includes the other existing or in-progress specifications: Introduction; Operational Guidelines; and, Test Guidelines.

Status of this document

This is the WG version of the SpecGL document. It's set for discussions among the WG members towards a future publication in TR space.

This version implements the changes discussed during 2002/10/21 Teleconf, and some remaining decisions from the F2F. Besides, levels has been moved from GL 7 to 5 to be adjacent to modules and profiles. And finally, the introduction has been much reworked, integrating Lofton's proposal.. The draft changes are more visible in the running editor's version.

This section describes the status of this document at the time of its publication. Other documents may supersede this document. The latest status of this document series is maintained at the W3C.

This document is a W3C Working Draft (WD), made available by the W3C Quality Assurance (QA) Activity for discussion by W3C members and other interested parties. For more information about the QA Activity, please see the QA Activity statement.

A future version of this document will be accompanied by a QA Framework: Specification Examples & Techniques document, which will illustrate the guidelines and checkpoints with case studies, and explain how to satisfy the checkpoints.

The QA Working Group Patent Disclosure page contains details on known patents related to this specification, in conformance with W3C policy requirements.

Please send comments to www-qa@w3.org, the publicly archived list of the QA Interest Group [QAIG]. Please note that any mail sent to this list will be publicly archived and available. Do not send information you would not want to see distributed, such as private data.

Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or made obsolete by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress".

A list of current W3C Recommendations and other technical documents can be found at http://www.w3.org/TR.

Table of contents

  1. Introduction
    1. Scope and Goals
    2. Motivation and expected benefits
    3. Relationship to other specifications
    4. Understanding and using this document
    5. Terminology
    6. Checkpoint priorities
    7. Roadmap to guidelines
    8. Dimensions of Variability
  2. Guidelines
    1. Define Scope.
    2. Identify what needs to conform and how.
    3. Address the use of profiles to divide the technology.
    4. Address the use of modules to divide the technology.
    5. Address the use of functional levels to divide the technology.
    6. Specify conformance policy.
    7. Identify the relation between deprecated features and conformance.
    8. Define discretionary items.
    9. Allow extensions or NOT!
    10. Provide a conformance clause.
    11. Specify how to make conformance claims.
    12. Publish an Implementation Conformance Statement proforma.
    13. Support general document conformance conventions.
    14. Provide test assertions.
  3. Conformance
    1. Test assertions
    2. Conformance definition
    3. Conformance disclaimer
  4. Definitions
  5. Acknowledgments
  6. References (non-normative)
  7. Change history

An appendix to this document [SPEC-CHECKLIST] presents all checkpoints in a tabular form, for convenient reference.


1. Introduction

1.1. Scope and Goals

The principal goal of this document is to define a framework to assist the W3C Working Groups (WGs) in writing specifications that:

Within this Specifications Guidelines document, the term "specifications" is specifically limited to W3C Technical Reports, even though these guidelines could be used along other documents.

This document describes what goes into the specification with respect to conformance and conformance topics, followed by rules for specification anatomy that should facilitate test development as well as produce better, more testable specifications.

The set of checkpoints as a whole has been written in anticipation that the checkpoints are being applied to new, in-development specifications. It is not expected that existing specifications that pre-date these guidelines will be able to satisfy all checkpoints as they are stated. Many legacy specifications may indirectly comply with the spirit or intent of some checkpoints, without actually satisfying all requirements in those checkpoints.

The target audience of these specification guidelines is:

It is a design goal of these guidelines the WGs can apply them in a common-sense and workable manner.

1.2. Motivation and expected benefits

Good specifications lead to better implementations and foster the development of conformance test suites and tools. Conforming implementations lead to interoperability.

The quality of the specification has direct impact on the quality of implementations. Quality encompasses utility which refers to the usefulness of the specification to the intended users and objectivity which focuses on the whether the specification is presented in an accurate, clear, complete, and unbiased manner.

Providing requirements and definitions about conformance topics, as well as guidance in the structure and anatomy of specifications, will foster a mutual understanding amongst developers of specifications, implementations, and conformance test materials. Specifically, well-structured specifications with clear and comprehensive conformance requirements:

1.3. Relationship to other specifications

This document is part of a family of QA Framework documents designed to help the WGs improve all aspects of their quality practices. The QA Framework documents are:

The QA Framework documents are interrelated and complement each other. For example, the anatomy of a specification is related to the type of test materials that will be built, hence there is interrelationship between this document and the Test Guidelines. The reader is strongly encouraged to be familiar with the other documents in the family.

The Framework as a whole is intended for all Working Groups, as well as developers of conformance materials for W3C specifications. Not only are the Working Groups the consumers of these guidelines, they are also key contributors. The guidelines capture the experiences, good practices, activities, and lessons learned of the Working Groups and present them in a comprehensive, cohesive set of documents for all to use and benefit from. The objective is to reuse what works rather than reinvent and to foster consistency across the various Working Group quality activities and deliverables.

This document does not preclude the need to apply the W3C Manual of Style [STYLE-MAN] and to conform to the Publication Rules [PUBRULES ] (member-only). It is intended to complement those specifications.

1.4. Understanding and using this document

This document employs the WAI (Web Accessibility Initiative) model for representing guidelines or general principles for the development of conformance materials. See, for example, Web Content Accessibility Guidelines 1.0 [WCAG10]. Each guideline includes:

The checkpoints in each guideline define the processes and operations needed to fulfill the purpose of the guideline. Each checkpoint definition includes:

Each checkpoint is intended to be specific enough so that someone can implement the checkpoint as well as verify that the checkpoint has been satisfied. A checkpoint will contain at least one, and may contain multiple individual requirements , that use RFC2119 normative keywords. See the "Conformance" chapter for further discussion of requirements and test assertions.

A separate appendix to this document [SPEC-CHECKLIST] presents all checkpoints in a tabular form, for convenient reference. This is an Implementation Conformance Statement (ICS) pro-forma for this specification. (See GL12.)

1.5. Terminology

The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY ", and "OPTIONAL" will be used as defined in RFC 2119 [RFC2119].

Unusual terms in these framework documents are defined when first used. This document contains a "Definitions" chapter. Generally useful QA-specific terms will eventually be in the QA Glossary [QA-GLOSSARY]. If terms are in the QA Glossary, their definition herein will refer to that QA Glossary entry with a link. The definitions herein may supplement or build on that generic definition with other information that is useful or helpful in the specification guidelines context. They will not contradict the generic definitions.

1.6. Checkpoint priorities

Some checkpoints are more critical than others for producing a high quality, testable standard that is a sound basis for successfully interoperable products. Therefore each checkpoint is assigned a priority level based on the checkpoint's impact on the quality of the specifications produced by the Working Groups.

[Priority 1]
Critical/Essential. These checkpoints are considered to be basic requirements for ensuring an acceptable level quality and testability in the specification.
[Priority 2]
Important/desirable. Satisfying these checkpoints, in addition to the priority 1 checkpoints, should significantly improve the clarity, quality, and testability, of the standard, as well as its suitability as a basis for building quality test materials.
[Priority 3]
Useful/beneficial. Satisfying these checkpoints, on top of all the others, will further improve the quality, usability, and testability of the specification.

1.7. Roadmap to guidelines

There are 14 guidelines:

  1. Scope and use cases (often in Requirements Document)
  2. Specification categories and product classes
  3. Profiles (hardware, environment, etc.)
  4. Modules
  5. Levels
  6. Conformance policy
  7. Deprecation policy
  8. Discretionary items
  9. Extensibility
  10. Conformance clause
  11. Claims of conformance
  12. Proforma
  13. Document conventions
  14. Granular grammar
  15. Test assertions

The guidelines are of two general types:

1.8. Dimensions of Variability

Each of the latter set of eight guidelines GL2 through GL9 addresses a way in which the conformance policy of a specification might allow variation amongst conforming implementations. For example, a specification might allow implementations to choose between one of two well defined behaviors for a given functionality, thus two conforming implementations might vary on that aspect.

For this reason, these eight guidelines are collectively called the "dimensions of variability (DoV)". The eight dimensions of variability recognized by this document are:

The above are not necessarily all orthogonal to one another. There are many possible associations, dependencies, and interrelationships. As a general policy, these specification guidelines do not attempt to legislate correct or proper relationships amongst the DoV. Rather, they try to clarify the nature of each dimension, and require specification to make deliberate and well documented choices. Some discussion of possible interrelationships, including examples will be found in the Specification Examples & Techniques.

The dimensions of variability are one of the principal concepts in this guidelines document to help organize, classify and assess the conformance characteristics of W3C specifications. The eight DoV get special attention because, since they are at the core of the definition of a specification's conformance policy, there is significant potential for negative interoperability impacts if they are handled carelessly or without careful deliberation.

As a general principle, variability complicates interoperability.. In theory, interoperability is best when there are numerous identical, complete, correct implementations. However, in practice, the net effect of conformance variability is not necessarily negative in all cases, when compared to the alternatives. For example profiles — subdivisions of the technology targeted at specific applications communities — introduce variability amongst implementations. Some will implement Profile ABC, some will implement Profile XYZ, and the two might not intercommunicate well if ABC and XYZ are fairly different. However, if ABC and XYZ are subsets of a large monolithic specification — too large for many implementors to tackle in toto -- and if they are well targeted at actual application sectors, then subdivision by profiles may actually enhance interoperability.

Different sorts of variability have different negative and positive impacts. The principal danger is "excessive" variability - variability which goes beyond that needed for positive interoperability tradeoffs, and which unnecessarily complicates the conformance landscape.Specification writers need to carefully consider and justify any conformance variability allowed, do so by reference to the project requirements and use cases, and explicitly document the choices made.

2. Guidelines

Guideline 1. Define Scope.

When writing specifications it is critical to understand their primary purpose and scope. Clearly defined scope helps to keep the specification content focused and unambiguous.

If the specification describes content requirements (for example, [WCAG10], [UAAG10]), understanding of their purpose is the key to defining the minimal sufficient set. If the specification describes a protocol or Application Programmer Interface (API) — examples include XML Protocol, DOM -- there should be a clear understanding of the primary use cases. For the purposes of this document, a use case is a specification mechanism or technique that captures the ways a specification would be used, including the set of interactions between the user and the specification as well as the services, tasks, and functions the specification is required to perform. A complete set of use cases specifies all the different ways to use the system (@@@what does this parenthesis mean? specification).

Readers of the specification face similar difficulties to writers. To understand what the document says, the reader needs to know the context of the author, and the scenarios the author had in mind. In case of protocols and APIs specifications developers try to assess whether the specifications cover the use cases the product needs to cover. Having no use cases described in the specification invites misuse of the specification itself.

Checkpoints

Checkpoint 1.1. Define the scope of the specification. [Priority 1]

To fulfill this checkpoint, a specification MUST define what scenarios are in scope and SHOULD identify out of scope scenarios.

Rationale: it helps both the writer and the reader to know what are the limits of what is specified in a normative document.

Checkpoint 1.2. Provide User Scenarios. [Priority 2]

A User Scenario is an instance of a use case, representing a single path through the use case. Thus, there may be a scenario for the main flow through the use case and other scenarios for each possible variation of flow through the use case (e.g., representing each option).

To fulfill this checkpoint, a specification MUST include or link to User Scenarios, SHOULD have an extensive list of the user scenarios that the authors have in mind

Rationale: User scenarios help to assess what features are missing and what features are superfluous in the specification.

Checkpoint 1.3. Provide examples. [Priority 1]

To fulfill this checkpoint, a specification MUST include or link to at least one example (@@@ Would this be acceptable to make it a little stronger?) and SHOULD provide an example for each section that public feedback has shown to be unclear or hard to understand.

Checkpoint 1.4. Provide an example for each formal description. [Priority 2]

@@@ Definition of formal description needed

To fulfill this checkpoint, a specification MUST provide an example for each identified formal description.

It is hard to understand the formal descriptions of content without informative interpretations.

For instance, the recent complex specifications like XML Schema [XML-SCHEMA] and XML Protocol have shown the interest of having a "Primer" part or section to illustrate how to use the specification.

Guideline 2. Identify what needs to conform and how.

Categorizing the specification provides a basis for classifying the software that may be affected by the specification — and thus, provides the answer to "what needs to conform?". To answer this question, it helps to look at the nature of the specification and categorize it. Most specifications can be classified into one of the following categories:

  1. foundation or abstract (e.g., Infoset),
  2. content/data (e.g., MathML, SVG),
  3. protocols (e.g., SOAP),
  4. processors (e.g., XSLT, XML Query),
  5. APIs (e.g., DOM),
  6. notation/syntax (e.g., XPath),
  7. set of events (e.g., one part of XForms)
  8. rules for deriving profiles (e.g., part of SMIL)

The categories indicate what the specification describes. One specification could potentially fall into more than one category.

From this categorization of specifications, the WG can identify the class of products that are affected by the specification. Classes of products can be generalized as either or producers or consumers, or as content itself. Identifying which are producers and consumers is clear for a protocol-type specification, the two parties to the dialog are the targets. For a processor-type specification, the processor is the consumer of an XML vocabulary defined in the specification. For content-type specifications, there may be one or more consumers that take the content and 'play' it in some way.

The following is a non-exhaustive list of classes of products for W3C specifications.

One specification could define more than one player. For example, MathML could address the behavior of display of math notation and also a consumer that parses the MathML as a formula and uses it for mathematical processing.

The conformance clause identifies the "class of products" (i.e., object of the claim) that is the target of the specification. In addition to identifying what conforms (i.e., class of products), it describes how conformance can be met. This would be a description of the conformance requirements, conditions and/or criteria for each class of product.

Checkpoints

Checkpoint 2.1. Identify all classes of product. [Priority 1]

To fulfill this checkpoint, a specification:

Example. Within the SMIL 2.0 Language Profile ([SMIL20], chapter 13), there are 2 classes of products: documents and basic user agents. Within Mathematical Markup Language (MathML) 2.0 [MATHML20] there are output-compliant, authoring tools and input-compliant, rendering/reading tools.

Checkpoint 2.2. For each class of product, define the conformance requirements. [Priority 1]

The conformance requirements indicate the conditions to be satisfied in order to claim conformance. @@@ Should this be moved to ET? In addition to specifying the requirements for claiming conformance, it may be appropriate to specify that which is not a requirement. It is likely that these conformance definitions will reference normative text within the specification or in other related specifications.

To fulfill this checkpoint, a specification MUST define the conformance requirements for each class of product. @@@ Here, the test assertion is pretty much a duplicate of the CP title... Is there something we can do about this?

Checkpoint 2.3. Identify which of the categories of object are specified in the document as a whole. [Priority 3]

To fulfill this checkpoint, a specification MUST identify in the Introductory section which of the types of object are specified in the document as a whole.

Rationale: doing so helps keep the scope of the specification in focus, both for the reader, and for the author(s) who must define the classes of product and their conformance criteria.

Some specifications define more than one of the enumerated object types. XForms is an example. It defines: Content, via an XML vocabulary; Content, via named datatypes; Syntax, in the form of a set of functions to supplement the XPath core set; behavior of a processor; behavior of a user agent; a set of events.

Guideline 3. Address the use of profiles to divide the technology.

A profile is a subset of the technology that supports a particular functional objective or a subset of a set of technologies defining how they are required to operate together (e.g., XHTML plus MathML plus SVG).

Profiles can be based on hardware considerations associated with target product classes — for example, SVG Tiny is aimed at mobile phones — or they may be driven by other functional requirements of their target constituencies — for example, a graphical profile tailored for technical illustrations in aircraft maintenance manuals.

The use of profiles to divide the technology is described in the specification, and may or may not be reflected and paralleled by the structure and organization of the specification.

Specifications may define individual profiles, or may define rules for profiles, or both. An individual profile defines the requirements for classes of products that conform to that profile. Rules for profiles define validity criteria for profiles themselves — i.e., if others (users, applications, or other standards) define their own profiles of the standard, then rules for profiles define the constraints that those profiles must satisfy in order to be considered valid profiles of the specification.

For example, XHTML Modularization ([XHTML-MOD], section 3) and Synchronized Multimedia Integration Language (SMIL 2.0), [SMIL20] specifications both define rules for profiles -- what constraints must a profile meet in order to be classified as a "Host Language Profile" or an "Integration Set Profile." SMIL further defines some specific profiles, using the modularization. Separate recommendations -- XHTML Basic [XHTML-BASIC] and XHTML 1.1 [XHTML11] — define specific profiles based on the XHTML modularization.

Checkpoints

Checkpoint 3.1. Include a table of contents entry. [Priority 2]

To fulfill this checkpoint, a specification MUST have a link in the table of contents to informations about the use of profiles. It is not applicable on specification that do not use profiles. @@@ What about specs that don't have TOC? and to which spec (the profile itself, the standard profiled, the rules for profiles?) does that apply?

The reader should be able to easily identify and locate information about profiles. A link from the table of contents (TOC) provides this ability. The form of TOC linkage can vary, depending on the style of the specification — some have only a high-level TOC, some have a detailed TOC, and some have both.

Checkpoint 3.2. Indicate whether or not the use of profiles is mandatory for conformance. [Priority 1]

To fulfill this checkpoint, a specification

It is not applicable if profiles are not used.

For example, is content required to conform to one of the profiles, or is there a concept of conformance of content independent of conformance to one of the profiles? Is a producer (of content) conforming if it generates content that is otherwise valid but does not conform to a profile?

Note: If there is a "full" profile defined — for example, incorporating all of the defined functionality of the specification, including extensibility features — then any valid content, as well as any correct producers and fully capable consumers, might seem to be automatically using that profile. However, profiles (e.g., of content) often include self-identification requirements, and these would have to be observed for conformance of valid content to even a "full" profile.

An example of additional conditions on profiles would be to require that only one profile can be implemented at a time.

Checkpoint 3.3. If profiles are chosen, define any minimal requirements. [Priority 2]

To fulfill this checkpoint, a specification MUST define for each profile the minimal required features/support for each class of product. It is not applicable if profiles are not used.

Rationale: a profile places requirements on each class of product that is affected by the profile's definition. @@@ so ?

To illustrate, SMIL 2.0 [SMIL20] has a SMIL 2.0 Language Profile for user agents but also provides a SMIL 2.0 Basic Profile for wireless and embedded devices. The SMIL 2.0 Language Profile requires that a user agent implement the BasicAnimation module but not the SplineAnimation Module. The SMIL 2.0 Basic Profile on the other hand does not require implementation of any of the Animation modules.

Checkpoint 3.4. If profiles are chosen, define their relationships and interaction with other dimensions of variability. [Priority 2]

To fulfill this checkpoint, a specification MUST identify all the relations and interactions between profiles and the other dimensions of variability. It is not applicable if profiles are not used.

Dependency or interrelationship between profiles and modules is common -- XHTML [XHTML-MOD], SMIL [SMIL20], SVG 1.1 [SVG11]. Less often, deprecated features, levels, discretionary choices, or extensions could depend on profiles.

Checkpoint 3.5. If profiles are chosen, address rules for profiles. [Priority 2]

if the specification allows the creation of derived profiles, to fulfill this checkpoint a specification MUST provide rules for derived profiles and these rules MUST be testable @@@ what is a testable rule?. It is not applicable if profiles are not used.

It is RECOMMENDED that rules for derived profiles impose at least these two rules on derived profiles: derived profiles SHOULD be specified in a way that meets all the pertinent checkpoints of this document (QA Framework: Specification Guidelines); and, derived profiles SHOULD NOT clash with pre-defined profiles, if there are any in the base specification @@@ What do we mean by clash?.

Rationale: well-designed rules for profiles will facilitate that derived profiles are well-specified, and testability will enable an independent tester to verify conformance of a derived profile to the rules.

Guideline 4. Address the use of modules to divide the technology.

Modules are non-hierarchical, discrete divisions or functional groupings of the technology.

For example, SMIL 2.0 [SMIL20] defines a module as follows:

A module is a collection of semantically-related XML elements, attributes, and attribute values that represents a unit of functionality. Modules are defined in coherent sets.

Modules generally can be implemented independently of one another — e.g., audio vs. video module. That notwithstanding, it is possible for one module's definition (and therefore implementation) to have explicit dependency upon another module. It is not only possible, but common to implement multiple modules.

Checkpoints

Checkpoint 4.1. Include a table of contents entry. [Priority 2]

To fulfill this checkpoint, a specification MUST have a link in the table of contents to informations about the use of modules. It is not applicable on specification that do not use modules.

Rationale: the reader should be able to easily identify and locate modules information. A link from the table of contents (TOC) provides this ability.

Checkpoint 4.2. If modules are chosen, indicate any mandatory conditions or constraints on their usage. [Priority 1]

To fulfill this checkpoint, a specification MUST document any mandatory conditions or constraints on their usage. Such conditions include,

It is not applicable if modules are not used.

The conditions or constraints normally will be tailored according to class of product.

Checkpoint 4.3. If modules are chosen, define their relationships and interaction with other dimensions of variability. [Priority 2]

To fulfill this checkpoint, a specification MUST specify any relationships and interaction with other dimensions of variability. It is not applicable if modules are not used.

Rationale: often there is dependency or interrelationship among modules, on the one hand, and profiles or discretionary choices on the other. Modules may have levels or deprecated features. Extensions could be defined based on modules.

Guideline 5. Address the use of functional levels to divide the technology.

Functional levels — or in common usage, simply "levels" — are used to group functionality into nested subsets, ranging from minimal or core functionality to full or complete functionally. Level 1 is the minimum or core of the technology. Level 2 includes all of level 1 plus additional functionality. This nesting continues until level n, which consists of the entire technology.

Levels may result from progressive historical development and enrichment of the technology in a series of specifications, as in the case of CSS and DOM. Levels could also be defined explicitly in a single edition of the specification, but no examples of this are found in W3C specifications. Rather, it is more common in current W3C practice to use profiles to accomplish this. For example, SVG Mobile [SVG-MOBILE] defines three nested profiles — Tiny, Basic, Full — which are each targeted at a specific graphics hardware community (mobile phone, hand-held computer, desktop computer).

See Guideline 3 for full discussion of profiles, including comments on possible profiles-levels relationships. See Guideline 5 for a full discussion of modules, including possible modules-levels relationships.

Checkpoints

Checkpoint 5.1. Include a table of contents entry. [Priority 2]

To fulfill this checkpoint, a specification MUST have a link in the table of contents to informations about the use of levels. It is not applicable if the specification doesn't use levels.

Rationale: the reader must be able to easily identify and locate levels information. A link from the table of contents (TOC) provides this ability.

Checkpoint 5.2. If levels are used, define their relationships and interaction with other dimensions of variability. [Priority 2]

To fulfill this checkpoint, a specification MUST specify any relationships and interaction with other dimensions of variability. It is not applicable if levels are not used.

Levels can be dependent on, or apply to, modules. Less often, there can be a relationship between levels, on the one hand, and profiles or deprecated features on the other.

Guideline 6. Specify conformance policy.

A look at various W3C Technical Reports shows that the term "conformance" is often qualified, resulting in more than one type of conformance. It is important to convey an understanding of what is meant by conformance and where the products of a class are allowed to have more or less functionality than defined in the specification. If the specification defines behavior for more than one class of product, there may be a separate conformance policy for each class. Often, the specification will allow discretionary choices, such as the choice of one or more natural languages for verbal content and messages, but require a conforming product to make a choice only within the allowable range. (See Guideline 8 for more discussion.)

Sometimes a product developer can choose to implement certain modules. There may be per-module conformance requirements that apply if and only if the developer chooses to implement a particular module.

Where all products of a class must be substantially alike, it should be clear that a "strict conformance" policy is in effect for that product class.

Strict conformance is defined as conformance of an implementation that employs only the requirements and/or functionality defined in the specification and no more (i.e., no extensions to the specification are implemented).

Sometimes a product developer can choose to implement extensions. There may be conformance requirements for non-interference of extensions. (See Guideline 9 for more discussion.)

Overall, the intent of the WG should be clear. In particular, a reader intending to implement a product in one of the product classes addressed by the specification should know what the WG wants for interoperability among products in the class. The developer should understand what forms, if any, of "product differentiation" are allowed among conformant products. The specification may need to explain how the rules apply and possibly provide examples.

Guideline 10, "conformance clause" is related to this guideline. This Guideline 6 focuses on the establishment and scope of definition of a conformance policy, while Guideline 10 focuses on where and how to document it. That is, the verification of these checkpoints will require looking at the Conformance Clause.

Checkpoints

Checkpoint 6.1. Specify any universal requirements for minimum functionality. [Priority 1]

To fulfill this checkpoint, a specification MUST include a normative section detailing any universal requirements for minimum functionality. It is not applicable if there isn't any universal requirements.

Rationale: the reader must be able to recognize any minimum that applies to all conforming products of each class.

If levels are used (see Guideline 5), the lowest level may represent the minimum set of requirements. If profiles are used there may be different minima for each profile. If modules are used, there may be different minima for each module. Furthermore, a module may itself be a minimum (i.e., required) for a particular class of product.

Checkpoint 6.2. Identify strict conformance requirements. [Priority 1]

To fulfill this checkpoint, a specification MUST state in its conformance section if the conformance requirements are strict or not.

Rationale: the reader must be able to recognize when a policy of "strict conformance" applies. As defined above, this implies that all conformant products of a class behave essentially the same way.

@@@ Does that really illustrate this CP? Wouldn't this belong to DOV discussion instead? If profiles are used, each profile may have its own conformance boundaries. If modules are used, each module may have its own conformance boundaries.

Use the definition provided above (or in the QA Glossary [QA-GLOSSARY]). The definition may be modified to adjust its scope, for example when applying it to modules, profiles or levels. In such cases, the scope of "requirements [...] defined in the specification" is narrowed from specification to the module, profile, or functional level that is the target of the statement.

Checkpoint 6.3. Distinguish requirements from product-specific extra features [Priority 1]

To full this checkpoint, a specification MUST distinguish any requirements from product-specific extra features.

Rationale: @@@ This is not very convincing, IMHO... Any good reason we have this CP? The reader must be able to recognize conformance requirements as distinct from allowable extra functionality.

@@@ This is rather long and cannot be normative; should this be moved to ET? If profiles are used (see Guideline 3), make it clear whether extra capabilities of the platform may be exploited. If modules are used (see Guideline 4), there may be different provisions for extra features applying to each. If deprecation applies (see Guideline 7), make it clear whether support of the obsolete features is optional, part of a level, or required. If levels are used (see Guideline 5), make it clear whether the highest level may be exceeded with additional features or functionality. If discretionary choices are allowed (see Guideline 8), make it clear if more than one may be implemented, when it is technically possible to do so.

Checkpoint 6.4. If special conformance terms are used, include a definition in the specification. [Priority 1]

To fulfill this checkpoint, a specification MUST either use conformance terms as defined in this document or define any other conformance terms used in it and reference them from the conformance clause.

Rationale: it is necessary to define terms that govern application of the conformance provisions. Ideally, all terms are from QA documents and other existing literature and need only be cited. If special terms are constructed, such as to combine modules and levels or modules and discretionary choices, they need to be defined in the specification.

Guideline 7. Identify the relation between deprecated features and conformance.

A deprecated feature is a feature whose use is discouraged because it has been outdated by newer constructs or is no longer viable.

After the initial publication of a specification, specification developers may consider the deprecation of a feature (e.g., function argument, element or attribute) defined in the specification. Deprecated features may become obsolete and no longer defined in future versions of the specification. Deprecation of a feature may warn implementers that the feature was a bad idea and it may be withdrawn in the future. Specification developers need to consider the effect of deprecation on all the classes of products that implement the specification (e.g., authoring tools, user agents) as well as the conformance consequences on each class of product. For the purpose of backward compatibility, it may be necessary to specify different requirements for the support of deprecated features for each class of product. For example, authoring tools (producers) would not use the feature, but user agents (consumers) would continue to support it.

Checkpoints

Checkpoint 7.1. Identify each deprecated feature. [Priority 1]

To fulfill this checkpoint, a specification MUST identify in a normative section each deprecated feature. It is not applicable if there is no deprecated feature.

Checkpoint 7.2. For each class of product, specify the degree of support required for each deprecated feature and the conformance consequences of the deprecation. [Priority 1]

To fulfill this checkpoint, a specification MUST specify the degree of support required for each deprecated feature for each product class and the conformance consequences of the deprecation @@@ one more TA whose prose is equivalent to the CP... sigh. It is not applicable if there is no deprecated features.

For example, a deprecated-features section of MathML 2.0 ([MATHML20], section 7.2.1.2) describes, about deprecated MathML 1.x features, that MathML-output-compliant authoring tools may not generate MathML markup containing deprecated features; whereas MathML-input-compliant rendering/reading tools must support deprecated features.

Checkpoint 7.3. Include an explanation for the deprecation. [Priority 3]

To fulfill this checkpoint, a specification MUST document each deprecated features with a rationale for deprecation. It is not applicable if there is no deprecated features.

Rationale: providing the rationale for deprecating a feature helps implementers and users to understand the motivation for the deprecation, the impact and consequences on current and future implementations, and perhaps insight into its eventual disappearance from the specification.

Checkpoint 7.4. Include examples to illustrate how to avoid using deprecated features. [Priority 3]

@@@ This probably only applies to "producers", not to "consumers". Is this OK? What if a spec only defines consumers and not producers

To fulfill this checkpoint, a specification MUST provide an example for each deprecated feature showing how to avoid using it. It is not applicable if there is no deprecated features.

Rationale: examples are helpful in providing alternatives or better ways to get the same results. By showing what can be done in place of the deprecated feature will help to get implementers to discontinue use of the deprecated feature.

Checkpoint 7.5. Include a table of contents entry. [Priority 2]

To fulfill this checkpoint, a specification MUST have a link in the table of contents to informations about deprecated features. It is not applicable if there is no deprecated features.

Rationale: the reader must be able to easily identify and locate information about deprecated features. A link from the table of contents (TOC) provides this ability.

Guideline 8. Define discretionary items.

Discretionary items are defined as deliberate and explicit grants of discretion by the specification, to the implementations, that describe or allow optionality of behavior, functionality, parameter values, error handling, etc.

Discretionary items are often made available in specifications, to give implementers of the technology the opportunity to decide from alternatives when building applications and tools. Discretionary items may be considered necessary because of environmental conditions (e.g., hardware limitations or software configuration, or external systems), locality (e.g., time zone or language), optional choices providing flexibility of implementation, dependence on other specifications, etc.

Discretionary items come in three basic variants:

discretionary choices
a value or behavior may be chosen from a well-defined enumerated set of two or more possibilities;
optional features
a well-defined feature may be supported or not (if supported, then the requirements are clear and unambiguous)
implementation dependent values (or features)
it is open ended and undefined, what set of values an element or attribute may have, or the behaviors of a product that implements a feature, etc

Checkpoints

Checkpoint 8.1. State the circumstances for when discretionary items are allowed [Priority 1]

To fulfill this checkpoint, a specification MUST indicate the rationale for the discretionary items by providing a reference or link to its use cases and/or project requirements and SHOULD identify by labeling all discretionary items. It is not applicable for specifications that don't have discretionary items.

Doing this helps readers, implementers and testers to find these discretionary items and understand the need for them.

Checkpoint 8.2. For implementation dependencies, address the allowable differences between implementations [Priority 1]

To fulfill this checkpoint, a specification MUST describe any permitted variations or constraints for how the implementation dependency is realized by implementations.

Examples of permitted variations or constraints to be addressed include:

Checkpoint 8.3. Indicate any constraints on the number of choices/options that can be implemented [Priority 2]

To fulfill this checkpoint, a specification MUST indicate whether zero, exactly one, or a multiple of choices/options are allowed to be implemented. It is not applicable for specifications that don't use discretionary items or for implementation dependent values among them.

Examples of constraints include mandating that an implementation implement only one choice, implement every choice, or be allowed to implement none of the choices.

Checkpoint 8.4. Promote consistent handling of discretionary choices. [Priority 2]

To fulfill this checkpoint, the specification MUST state that given identical conditions, the effect of a discretionary choice is consistent within a single implementation.

Rationale. Users have an expectation of what to expect and should be able to count on getting the same results under the same conditions with a given implementation.

Checkpoint 8.5. Include a table of contents entry. [Priority 2]

To fulfill this checkpoint, a specification MUST have a link in the table of contents to information about the use of discretionary items. It is not applicable on specifications that do not use discretionary items.

The reader must be able to easily identify and locate information about discretionary items. A link from the table of contents (TOC) provides this ability.

Guideline 9. Allow extensions or NOT!

An extension to a specification is a mechanism to incorporate functionality beyond what is defined in the specification.

Allowing extensions affects how conformance is defined as well as what conformance claims can be made. Exercise caution in determining the extent to which extensions are allowed or not allowed. Since extensions can seriously compromise interoperability, specification writers should carefully consider whether extensions should be allowed.

Extensions may be private (often vendor specific) or public (a full description of the extension is public). Private extensions are usually truly private, i.e., valid for a specific implementation or are only known by prior agreement between implementations. Public extensions are extensions in which the syntax, semantics, identifiers, etc. are defined and published allowing anyone to implement the extended functionality.

Specifications allow extensions for various reasons. Extensions allow implementers to include features that they consider to be required by their customers. Also, extensions, often define new features that may migrate into future versions of the specifications. However, the use of extensions can have a severe negative impact on interoperability. Some methods for enabling extension have less impact on interoperability than other methods. For example, a specification that allows private extensions (e.g., proprietary) is highly likely to impede interoperability, whereas a specification than permits only registered extensions partially mitigates the negative impacts.

Checkpoints

Checkpoint 9.1. Indicate if extensions are disallowed [Priority 1]

To fulfill this checkpoint, a specification MUST state that extensions are disallowed and MUST indicate the conditions for when this holds true.

Rationale: if the specification writer wants to enhance interoperability by constrining implementer extensions, wording in the specification must indicate this.

If extensions are not allowed, then it should be clear to the reader that not only are extensions not allowed, but the circumstances under which they are not allowed. This is strict conformance. Strict conformance is often imposed on applications or content (e.g., a software program or document instance). This prohibition of extensions could be applied to a specific profile or module, rather than to the entire specification.

Checkpoint 9.2. Indicate if extensions are allowed. [Priority 1]

To fulfill this checkpoint, a specification MUST state

and MUST document the rationale for allowing extensions by referencing use cases and/or project requirements. It is not applicable if the specification doesn't allow extensions.

Rationale: readers should be able to understand the motivation for the inclusion of an extension and its intended use. Documenting the scope and rationale for extensions helps assess the impact of extensions on interoperability

Checkpoint 9.3. Prevent extensions from contradicting the specification. [Priority 1]

To fulfill this checkpoint, the specification MUST state that extensions can not negate or change support for required functionality. It is not applicable if extensions are not allowed.

Requirements in the specification can not be compromised or contradicted by adding extensions.

Checkpoint 9.4. Use a standard mechanism to define the extension. [Priority 3]

To fulfill this checkpoint, a specification MUST provide a standard way of defining the extension. It is not applicable if extensions are not allowed.

This helps to ensure predictable handling of extensions, that is, its recognition as such and the appropriate actions (i.e., to ignore or to implement).

Checkpoint 9.5. Require that extensions be published. [Priority 3]

To fulfill this checkpoint a specification MUST require that the syntax and semantics of the extension be publicly documented. It is not applicable if extensions are not allowed.

Rationale: public availability with a full description allows the extension to be implemented by anyone without prior arrangement. This enhances interoperability by allowing the same functionality to be uniformly implemented across different implementations.

Checkpoint 9.6. Require implementations to include a way to "turn off" extensions. [Priority 3]

To fulfill this checkpoint, a specification MUST indicate via conformance requirements that implementations provide a mode under which they produce only conforming content. This checkpoint is not applicable if extensions are not allowed.

Rationale: This checkpoint can be used to ensure that there is a way to produce (generate) only conforming content. It is applicable to specifications that identify producer of content as one of its classes of products. @@@ Shouldn't this be moved in the applicability clause instead?

Checkpoint 9.7. Include a table of contents entry. [Priority 2]

To fullfill this checkpoint, a specification MUST have a link in the table of contents to informations about any extensions mechanism.

Rationale: the reader must be able to easily identify and locate information about extensions and extensibility. A link from the table of contents (TOC) provides this ability.

Guideline 10. Provide a conformance clause.

A conformance clause is a part or collection of parts of a specification that defines the requirements, criteria, or conditions to be satisfied by an implementation or application in order to claim conformance. Typically the conformance clause is a high-level description of what is required of implementations and applications.

Guideline 6, "conformance policy" is related to this guideline. Guideline 6 focuses on the establishment and scope of definition of a conformance policy, while this Guideline 10 focuses (among other topics) on how and where to document it.

Checkpoint 10.8. Include a conformance clause. [Priority 1]

To fulfill this checkpoint, a specification MUST document its conformance policy and specific conformance requirements.

As used in this checkpoint, "clause" does not necessarily imply a specific single document section or location (see next checkpoint).

Checkpoint 10.9. Include a conformance section. [Priority 2]

To fulfill this checkpoint, a specification MUST document its conformance policy and specific conformance requirements in a dedicated section of the document.

Rationale: having the conformance clause exist as a separate section within the specification makes it clearly identifiable, allowing a reader to find the overall conformance policy, as well as all specific conformance provisions from a single starting point.

Checkpoint 10.10. Include a conformance clause entry in the table of contents. [Priority 2]

To fulfill this checkpoint, a specification MUST have a link in the table of contents to its conformance clause.

Rationale: it must be possible for the reader to start at the table of contents and find all aspects of the conformance requirements, including the overall policy.

Checkpoint 10.11. Make normative reference to specifications on which the current specification depends [Priority 2]

A specification depends on another specification when it relies on or requires functionality (or behavior) from the other specification in order to work (function) correctly. This other specification provides a necessary condition or functionality.

To fulfill this checkpoint, a specification MUST have normative references to any specification it depends on and MUST describe the relationship between the specifications and any conformance implications

Rationale: dependence on other specifications affects the conformance boundaries of the current specification, and thereby affects the requirements on conformant products.

The linking parts of the Manual of Style ([STYLE-MAN], section 11.5.1) describe the recommended way of citing an external reference from the prose of a specification, as well as how to construct an entry in its References section.

For example, SVG 1.0 requires that the class of product called "user agent" be consistent with the XML 1.0 Recommendation [XML10] and (conditionally) support Cascading Style Sheets, level 2 [CSS2].

Guideline 11. Specify how to make conformance claims.

A specification may differentiate conformance claims by designating different degrees or types of conformance in order to apply and group requirements according to modules, profiles, levels, or priorities. When a conformance claim is linked to functionality, impact and/or incremental degrees of implementation, the term 'conformance level' is often used to indicate the varying degrees of conformance. The WG includes in the specification the way they want people to claim their conformance.

Checkpoints

Checkpoint 11.1. Identify and define all conformance designations. [Priority 1]

To fulfill this checkpoint, a specification MUST identify and define all conformance designations.

In current W3C practice, a number of different naming conventions are used to label conformance, in cases where there is not a single, monolithic (strict) conformance definition. The naming convention used to label the conformance can provide useful information. Degrees, for example, implies incremental importance or difficulty. This Specification Guidelines document uses "degrees" for example, to refer to three successively more demanding degrees of conformance (A, AA, AAA).

Commonly used conformance designations include categories, degrees, and levels. Use of "conformance levels" is discouraged in new specifications, because of the potential for confusion with "functional levels".

Checkpoint 11.2. Provide specific wording of the claim. [Priority 3]

To fulfill this checkpoint, a specification MUST provide specific wording of the claim and the specific wording MUST at least include the specification name, its version, the conformance level satisfied and information about the subject that which is claiming conformance and the date of the claim.

Checkpoint 11.3. Provide a conformance disclaimer. [Priority 3]

To fulfill this checkpoint, a specification MUST provide a conformance disclaimer.

Rationale: although it is possible to prove with certainty when something does not conform, the reverse is not necessarily true. Especially for functional specifications, where a claim goes beyond syntax testing, a claim of conformance is not a guarantee that the claimant is 100% conforming with the specification. A disclaimer can help clarify the meaning of a conformance claim as well as its limitations. For example, this document contains a conformance disclaimer.

Checkpoint 11.4. Impose no restrictions about who can make a claim or where claims can be published. [Priority 1]

To fulfill this checkpoint, a specification MUST NOT include any restriction about who can make a claim nor where claims can be published.

Claimants (or relevant assuring parties) are solely responsible for the validity of their claims, keeping claims up to date, and proper use of the conformance icons.

Checkpoint 11.5. Include a table of contents entry. [Priority 2]

To fulfill this checkpoint, a specification MUST have a link in the table of contents to informations on conformance claims.

Rationale: the reader should be able to easily identify and locate the information on how to make conformance claims. A link from the table of contents (TOC) provides this ability.

Guideline 12. Publish an Implementation Conformance Statement proforma.

An Implementation Conformance Statement (ICS) provides a mechanism whereby a supplier of an implementation of the specification provides information about the implementation in a standardized manner. It is used to indicate which capabilities and options have been implemented, as well as the limitations of the implementation. A ICS usually takes the form of a questionnaire where product implementors report on the conformance of their product to the named specification.

An ICS is useful in disclosing optional functionality and discretionary behavior and values. The results of the ICS can be used to identify the subset of test cases from a conformance test suite that are applicable to the implementation to be tested. This will allow the implementation to be tested for conformance against only the relevant requirements.

The basic and detailed information that an ICS provides can also be used to assess and deduce the interoperability potential of two or more products.

Checkpoints

Checkpoint 12.1. Provide an Implementation Conformance Statement proforma. [Priority 3]

To fulfill this checkpoint, a specification MUST include or have a reference to an Implementation Conformance Statement.

@@@ How an ICS can be normative? Is an implementation that provdes an ICS not exactly equal to the one of the spec not conformant? On what class of product is this requirement set? If an ICS is included as part of the specification, indicate whether it is a normative or informative part of the specification.

@@@ Again, since W3C isn't in the business of certification, I'm not sure what's the interest of having anything normative in specs about conformance claims...

Checkpoint 12.2. Require the ICS be completed as part of the conformance claim. [Priority 3]

To fulfill this checkpoint, a specification MUST require that @@@ I don't know which product class is concerned to formulate an assertion here.

Rationale: an ICS provides specific information about the implementation and can be helpful in substantiating the conformance claim.

Guideline 13. Support general document conformance conventions.

There is a lot to be said for consistency and clarity within a document - it facilitates the understanding and comprehension of the document. Authors and editors of specifications should already be familiar with the W3C Manual of Style [STYLE-MAN] and Publication Rules (member-only) [PUBRULES], which help to achieve this. With respect to conformance, it is important to provide clear and unambiguous statements, so that the reader knows what is required in order to claim conformance and what is optional. To achieve this objective, throughout the document, employ uniformity of structure and style, and consistency of terminology and phraseology.

Checkpoints

Checkpoint 13.1. Use conformance key words. [Priority 1]

To fulfill this checkpoint, a specification MUST use RFC 2119 keywords to denote wheter or not requirements are mandatory, optional, or suggested @@@ Should we add something on capitalization?

Rationale: Using these keywords helps to identify the testable statements in a specification.

Checkpoint 13.2. Distinguish normative and informative text. [Priority 2]

Normative statements are the prescriptive parts of the specification whereas informative statements are for informational purposes and assist in the understanding or use of the specification.

To fulfill this checkpoint, a specification MUST distinguish normative text from informative text.

Rationale: it is important that the reader be able to distinguish between normative and informative statements in order to know what is required to claim conformance. SMIL 2.0 is an example, indicating within every subsection whether it is normative or informative, and even separately labelling pieces of subsections that contain both kinds of text.

Checkpoint 13.3. Use consistent terminology. [Priority 1]

To fulfill this checkpoint, a specification MUST use identical wording to express identical provisions, and analogous wording to express analogous provisions @@@ Is that testable?.

Guideline 14. Provide test assertions.

A test assertion is a statement of behavior, action or condition that can be measured or tested. It is derived from the specification's requirements and bridges the gap between the narrative of the specification and the test cases. Each test assertion is an independent, complete, testable statement for requirements in the specification. Each test assertion results in one or more test cases. Multiple test assertions can be combined to form a test case, in this case one tests multiple facets of a particular behavior.

Checkpoints

Checkpoint 14.1. Provide test assertions [Priority 1]

To fulfill this checkpoint, a specification MUST include or reference normatively a list of test assertions stated in it. @@@ Here (or in an adjacent CP) should come probably the discussion on normativity and conflict resolution?

In order to enable pointing to test assertions from tests as well as to give a map of the specification from the point of view of tests, use a mechanism for making explicit test assertions in the specification.

Rationale: providing test assertions facilitates and promotes the development of test materials. Tests can point directly to the test assertion. Specific benefits include:

Checkpoint 14.2. Provide a mapping between the specification and the test assertions list [Priority 2]

To fulfill this checkpoint, a specification MUST provide a mechanism linking each test assertion to the part of the specification it is stated.

Rationale: this allows both to ensure consistency between the specification and the test assertions list and to facilitate building a test suite framework based on the test assertions list.

Isn't an "intented behavior" no more than a requirement put on a specific product class? How is that different from a test assertion?

Checkpoint 14.3. Identify intended behavior in the specification using markup. [Priority 2]

There are different types of technologies that are standardized by W3C specifications, and this checkpoint applies in different ways to those. For API specifications, such as DOM, the intended behavior is the algorithm output; the specification contains wording on what the expected result of applying a particular method on an interface is. On user-centric specifications, such as WAI guidelines specifications, the intended behavior is not so much a question of testing algorithms, but rather how user agents should behave given different input or how markup should be designed. In this case, intended behavior is more a question of parsing documents. On processor specifications, such as XML, the intended behavior is, for example, preserving or expanding entity references, and the intended behavior is a particular state after processing a document.

Checkpoint 14.4. Supply prose description of intended behavior together with each test assertion. [Priority 2]

Following the guideline below, group test assertions and statements of intended behavior in the specification, or group the pointer to the test assertions together with the statement of intended behavior in the specification (in case the test assertion is, for example, given in an appendix).

3. Conformance

This section defines conformance of Working Group specifications — i.e., technical reports — to the requirements of this QA Framework guidelines specification. The requirements of this guidelines specification are detailed in the checkpoints of the preceding "Guidelines" chapter, and apply to the technical reports produced by Working Groups.

3.1. Test assertions

The test assertions of this Specification Guidelines document are found in the prioritized checkpoints. A checkpoint will contain at least one, and may contain multiple individual requirements. These requirements are the test assertions of this specification. A checkpoint is satisfied by satisfying all of the individual requirements. Failing one individual requirement means that the checkpoint is not satisfied.

3.2. Conformance definition

This section defines three degrees of conformance to this guidelines specification:

A specification conforms to the QA Framework: Specification Guidelines at degree X (A, AA, or AAA) if the Working Group meets at least all degree X conformance requirements.

To make an assertion about conformance to this document, specify:

Example:

This specification conforms to W3C's QA Framework: Specification Guidelines, available at http://www.w3.org/TR/qaframe-spec/, AA-Conforming.

3.3. Conformance disclaimer

The checkpoints of this guidelines specification present verifiable conformance requirements about the specifications (technical reports) that Working Groups produce. As with any verifiable test requirements, it is also true of these specification requirements that:

  1. Passing all of the requirements to achieve a given degree of conformance — A, AA, or AAA — does not guarantee that the subject specification is well-suited to or will achieve its intended purposes, nor does it guarantee the quality or suitability of test materials produced from the specification.
  2. Failing to achieve conformance degree of at least A-conforming does not mean that the subject specification is necessarily deficient to its intended purposes, nor does it mean that it is an unacceptable basis for the development of quality test materials. It means that the specification has failed one or more checkpoints that best-practice experience has shown to improve the testability and usability of specifications, and to facilitate the timely and successful development and maintenance of quality test materials based on the specification.

4. Definitions

[@@Ed note. under construction.] This section contains terms used in this specification, with functional or contextual definitions appropriate for this specification. See also [QA-GLOSSARY]. Some terms in this section have been borrowed or adapted from other specifications.

conditional conformance
[@@Ed note. we don't use it yet, but do use its companion "strict conformance"]
conformance clause
a part or collection of parts of a specification that defines the requirements, criteria, or conditions to be satisfied by an implementation or application in order to claim conformance
conformance level
a variety of conformance designation. Other designations include conformance category, conformance degree, conformance xxx,... "Conformance level" is discouraged in new specifications, because of confusion with "functional level".
dimensions of variability
the ways in which different products that are conformant to a specification may vary among themselves. In this Specification Guidelines document, the dimensions of variability are used to help organize, classify and assess the conformance characteristics of W3C specifications
discretionary items
deliberate and explicit grants of discretion by the specification, to the implementations, that describe or allow optionality of behavior, functionality, parameter values, error handling, etc.
functional level
a technology subset that is one of a hierarchy of nested subsets, ranging from minimal or core functionality to full or complete functionally.
implementation conformance statement (ICS)
a mechanism for providing standardized information about an implementation of a named specification, usually in the form of a questionnaire on which product implementers report about the product's conformance to the specification. An ICS is used to indicate which requirements, capabilities and options have and have not been implemented.
informative text
text in a specification whose purpose is informational or assistive in the understanding or use of the specification, and which contains no test assertions or conformance requirements.
level
a commonly used shorthand for functional level.
module
a collection of semantically-related elements, attributes, and attribute values that represents a unit of functionality. Modules are non-hierarchical, discrete divisions that are defined in coherent sets.
normative text
text in a specification which is prescriptive or contains conformance requirements.
profile
a subset of a technology that is tailored to meet specific functional requirements of a particular application community. A profile may address a single technology; or, a profile can also group a set of technologies (i.e., from different specifications) and define how they operate together. Profiles may be based on hardware considerations associated with target product classes, or they may be driven by other functional requirements of their target communities.
profiling
a method for defining subsets of a technology by identifying the functionality, parameters, options, and/or implementation requirements necessary to satisfy the requirements of a particular community of users.
strict conformance
conformance of an implementation that employs only the requirements and/or functionality defined in the specification and no more (i.e., no extensions to the specification are implemented).
test assertion
any sentence or equivalent assemblage of words and phrases that prescribes the behavior that must be obtained when a stimulus occurs under a certain set of conditions. (See also QA Glossary [QA-GLOSSARY].)
unconditional conformance
[Ed note. @@we don't use it yet, but do use its companion "strict conformance"]
use case
a specification mechanism or technique that captures the ways a specification would be used, including the set of interactions between the user and the specification as well as the services, tasks, and functions the specification is required to perform.
user scenario
an instance of a use case, that represents a single path through the use case. Thus, there may be a scenario for the main flow through the use case and another scenarios for each possible variation of flow through the use case (e.g., representing each option).

5. Acknowledgments

The following QA Working Group and Interest Group participants have contributed significantly to the content of this document:

6. References (non-normative)

CSS2
Cascading Style Sheets, Level 2, W3C Recommendation, 12 May 1998, available at http://www.w3.org/TR/REC-CSS2.
MATHML20
Mathematical Markup Language (MathML) Version 2.0, W3C Recommendation, 21 February 2001, available at http://www.w3.org/TR/MathML2/.
PUBRULES
W3C Publication Rules, available at http://www.w3.org/Guide/pubrules.html (member-only).
QA-GLOSSARY
Comprehensive glossary of QA terminology. (Under construction, at http://www.w3.org/QA/Glossary.)
QAF-INTRO
QA Framework: Introduction, L. Henderson, D. Dimitriadis, K. Gavrylyuk, L. Rosenthal, Eds., W3C Working Draft, May 2002, companion version to this document, available at http://www.w3.org/QA/WG/2002/framework-20020826/qaframe-intro.
QAF-OPS
QA Framework: Operational Guidelines, K. Gavrylyuk, D. Dimitriadis, L. Henderson, L. Rosenthal, Eds., W3C Working Draft, May 2002, companion version to this document, available at http://www.w3.org/QA/WG/2002/framework-20020826/qaframe-ops.
QAF-TEST
QA Framework: Test Guidelines, not yet published, see Table of Seven (at http://www.w3.org/QA/WG/#docs), for most-recent links to any WG-only or published version(s).
QAIG
Quality Assurance Interest Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/IG/.
QAWG
Quality Assurance Working Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/WG/.
RFC2119
Key words for use in RFCs to Indicate Requirement Levels, March 1997, available at http://www.ietf.org/rfc/rfc2119.txt.
SMIL20
Synchronized Multimedia Integration Language (SMIL 2.0), W3C Recommendation, 07 August 2001, available at http://www.w3.org/TR/smil20/.
SPEC-CHECKLIST
An appendix to this specification guidelines document presents all checkpoints in tabular form. Available at http://www.w3.org/QA/WG/2002/framework-20020826/qaframe-spec-checklist.
SPEC-EXTECH
QA Framework: Specification Examples & Techniques, not yet published, see Table of Seven (at http://www.w3.org/QA/WG/#docs), for most-recent links to any WG-only or published version(s).
STYLE-MAN
W3C Manual of Style, summarizing the style and publication rules for W3C technical reports, available at http://www.w3.org/2001/06/manual/.
SVG11
Scalable Vector Graphics (SVG) 1.1 Specification, D. Jackson, J. Ferraiolo, J. Fujisawa, Eds., W3C Candidate Recommendation 30 April 2002, available at http://www.w3.org/TR/SVG11/.
SVG-MOBILE
Mobile SVG Profiles: SVG Tiny and SVG Basic, T. Capin, Editor, W3C Candidate Recommendation, 30 April 2002, available at http://www.w3.org/TR/SVGMobile/.
TEST-EXTECH
"QA Framework: Test Examples & Techniques", not yet published, see Table of Seven (at http://www.w3.org/QA/WG/#docs), for most-recent links to any WG-only or published version(s).
UAAG10
User Agent Accessibility Guidelines 1.0, I. Jacobs, J. Gunderson, E. Hansen, Eds., W3C Candidate Recommendation, 12 September 2001, available at http://www.w3.org/TR/UAAG10/.
W3C-TR
Location of all published W3C technical reports, see http://www.w3.org/TR/.
WCAG10
Web Content Accessibility Guidelines 1.0, W. Chisholm, I. Jacobs, G. Vanderheiden, Eds., W3C Recommendation, 5 May 1999, available at http://www.w3.org/TR/WCAG10/.
XHTML-MOD
Modularization of XHTML, M. Altheim, F. Boumphrey, S. Dooley, S. McCarron, S. Schnitzenbaumer, T. Wugofski,Eds., W3C Recommendation, 10 April 2001, available at http://www.w3.org/TR/xhtml-modularization/.
XHTML-BASIC
XHTML Basic, M. Baker, M. Ishikawa, S. Matsui, P. Stark, T. Wugofski, T. Yamakami, Eds., W3C Recommendation, 19 December 2000, available at http://www.w3.org/TR/xhtml-basic/.
XHTML11
XHTML 1.1 - Module-based XHTML, M. Altheim, S. McCarron, Eds., W3C Recommendation, 31 May 2001, available at http://www.w3.org/TR/xhtml11/
XML10
Extensible Markup Language (XML) 1.0 (Second Edition), T. Bray, J. Paoli, C. M. Sperberg-McQueen, E. Maler, Eds., W3C Recommendation, 6 October 2000, available at http://www.w3.org/TR/REC-xml.
XML-NAMES
Namespaces in XML, T. Bray, D. Hollander, A. Layman, Eds., W3C Recommendation, 14-January-1999, available at http://www.w3.org/TR/1999/REC-xml-names-19990114/.
XML-SCHEMA
XML Schema Part 1: Structures, W3C Recommendation, 2 May 2001, available at http://www.w3.org/TR/xmlschema-1/.
XPATH10
XML Path Language (XPath) Version 1.0, W3C Recommendation, 16 November 1999, available at http://www.w3.org/TR/xpath.
XPOINTER
XML Pointer Language (XPointer) Version 1.0, W3C Candidate Recommendation, 11 September 2001, available at http://www.w3.org/TR/2001/CR-xptr-20010911/
XSLT10
XSL Transformations (XSLT) Version 1.0, W3C Recommendation, 16 November 1999, available at http://www.w3.org/TR/xslt.

7. Change history

2002-08-26, second published WD

Significantly reorganized and revised the first published WD. This version produced as a series of editor's drafts. The changes below are reverse chronological (most recent first), so more recent ones may build on older ones.

2002-05-15 version

First published WD.