Abstract

The principal goal of this document is to help W3C Working Groups to write clearer, more implementable, and better testable technical reports. It provides both a common framework for specifying conformance requirements and definitions, and also addresses how a specification might allow variation among conforming implementations, both of which facilitate the generation of test materials. The material is presented as a set of organizing guidelines and verifiable checkpoints. This document is one in a family of Framework documents of the Quality Assurance (QA) Activity, which includes the other existing or in-progress specifications: Introduction, Operational Guidelines, and Test Guidelines.

Status of this document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

This document is a W3C Candidate Recommendation made available by the W3C Quality Assurance (QA) Activity for discussion by W3C members and other interested parties. For more information about the QA Activity, please see the QA Activity statement.

The QA Working Group has made changes and improvements in response to Last Call comments, as documented in Change History. At a high level, the normative requirements of the guidelines and checkpoints are substantially the same. However, much of the text has been edited for clarity, precision and simplicity, and the guidelines have been reorganized for better usability. New structuring and CSS styling have been applied to all of the guidelines and checkpoints. See also the disposition of last call comments.

The QA Working Group resolved to seek transition of Specification Guidelines to Candidate Recommendation in order to gather trial application experience from the W3C Working Groups. Based on this experience, the QA Working Group will decide whether to request to advance to Proposed Recommendation or to publish as a Working Group Note. To allow time to gather experience, the QAWG does not expect to request to advance to the next state sooner than 10 May 2004. Feedback from the Working Groups will be used to update the preliminary implementation report.

The QAWG expects to show two implementations of each checkpoint in "QA Framework Specification Guidelines" before requesting to advance to the next state.

Publication as a Candidate Recommendation does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

The QA Working Group Patent Disclosure page contains details on known patents related to this specification, in conformance with W3C policy requirements.

You may email comments on this document to www-qa@w3.org, the publicly archived list of the QA Interest Group [QAIG]. Please note that comments that you make will be publicly archived and available, do not send information you would not want to see distributed, such as private data.

Table of contents

1. Introduction
   • 1.1. Scope and Goals
   • 1.2. Class of Product and Audience
   • 1.3. Motivation and expected benefits
   • 1.4. Relationship to other specifications
   • 1.5. Understanding and using this document
   • 1.6. Usage of terminology in this document
   • 1.7. Checkpoint priorities
   • 1.8. Use cases
2. Concepts
   • 2.1. Dimensions of Variability (DoV)
   • 2.2. Specification category and class of product
   • 2.3. Profiles, Modules, Levels
   • 2.4. Addressing relationships among Dimensions of Variability used
3. Guidelines
   • Guideline 1. Define Scope.
   • Guideline 2. Identify what needs to conform and how.
   • Guideline 3. Address the use of profiles, modules and levels to divide the technology.
   • Guideline 4. Identify the relation between deprecated features and conformance.
   • Guideline 5. Define discretionary items.
   • Guideline 6. Allow extensions or not
   • Guideline 7. Clearly identify conformance requirements.
   • Guideline 8. Document the conformance policy.
   • Guideline 9. Specify how to make conformance claims.
   • Guideline 10. Provide test assertions.
4. Conformance
   • 4.1. Normative parts
   • 4.2. Extensibility
   • 4.3. Conformance requirements and test assertions
   • 4.4. Conformance definition
   • 4.5. Conformance claims
   • 4.6. Conformance disclaimer
5. Definitions
6. Acknowledgments
7. References
   • 7.1. Normative
   • 7.2. Informative
8. Change history

0.1. Appendices

1. Checklist of Checkpoints sorted by guidelines
2. Checklist of Checkpoints sorted by priority
3. Normative list of Test Assertions

1. Introduction

1.1. Scope and Goals

The scope of this document is a set of requirements for W3C Technical Reports (TRs) that if satisfied, will enhance the clarity, implementability, and testability of TRs. It describes what goes into a TR with respect to conformance and conformance topics, dealing with how a TR establishes, defines, and presents its conformance policy. This document addresses the most basic and critical topics with respect to conformance and testabily; more advanced topics will be addressed in future versions. This document includes:

• This introduction, which provides for context understanding the requirements in section 3, as well as use scenarios or examples of how this document could be used.
• Section 2 explains the main concepts used throughout the guidelines.
• Section 3 contains ten guidelines. Each guideline consists of one or more verifiable "checkpoints". Each checkpoint includes the conformance requirements that a specification would need to meet in order to satisfy the checkpoint. There are two general types of guidelines, those that deal with the document features and conventions of the specification and those that deal with establishing and defining conformance policy.
• Section 4 explains how to make conformance claims that W3C TRs satisfy the requirements of section 2. It defines conformance for this document.
• An appendix lists all the guidelines and checkpoints into an Implementation Conformance Statement (ICS) pro forma for this specification, which is required to be completed as part of the conformance claim to this document.

A separate document, entitled "Specification Examples and Techniques 1.0" (the "ExTech document" from here on) provides suggestions and examples of how each checkpoint might be satisfied. The techniques in the ExTech document are informative examples only, other strategies may be used or required to satisfy the checkpoints. The ExTech document is expected to be updated more frequently than the current guidelines. Developers, W3C Working Groups, users, and others are encouraged to contribute techniques and examples.

1.2. Class of Product and Audience

The specification category for this specification is "guideline". The class of product or target of this specification is "specification", in particular W3C Technical Reports. Within this Specification Guidelines document, the term "specifications" is specifically limited to W3C Technical Reports, even though these guidelines could be applied to other documents.

The primary target audience of this document is specification authors, however, it is applicable to a broader audience including:

• those who review specifications during their development,
• implementers of specifications,
• those who use specifications to build test materials.

These guidelines are designed so that the WGs can apply them in a common-sense and workable manner.

1.3. Motivation and expected benefits

Good specifications lead to better implementations and foster the development of conformance test suites and tools. Conforming implementations lead to interoperability.

The quality of the specification has direct impact on the quality of implementations. Quality encompasses utility which refers to the usefulness of the specification to the intended users and objectivity which focuses on the whether the specification is presented in an accurate, clear, complete, and unbiased manner.

Providing requirements and definitions about conformance topics, as well as guidance in the structure and anatomy of specifications, will foster a mutual understanding among those developing specifications, implementations, and conformance test materials. Specifically, well-structured specifications with clear and comprehensive conformance requirements:

• promote a common understanding of conformance and what is required to claim conformance to a specification,
• facilitate consistent application of conformance within a specificationand across related specifications,
• promote better interoperability and open interchange,
• promote uniformity in the development of conformance test suites,
• enhance test assertion identification and test traceability,
• promote automated test generation directly from the specifications.

1.4. Relationship to other specifications

This document is part of a family of QA Framework documents designed to help the WGs improve all aspects of their quality practices. The QA Framework documents are:

• QA Framework: Introduction [QAF-INTRO].
• QA Framework: Operational Guidelines [QAF-OPS].
• QA Framework: Specification Guidelines (this document).
• QA Framework: Test Guidelines [QAF-TEST].

Although the QA Framework documents are interrelated and complement each other, they are independently implementable. For example, the anatomy of a specification is related to the type of test materials that will be built, hence there is an interrelationship between this document and the Test Guidelines. The reader is strongly encouraged to be familiar with the other documents in the family.

The Framework as a whole is intended for all Working Groups, as well as developers of conformance materials for W3C specifications. Not only are the Working Groups the consumers of these guidelines, they are also key contributors. The guidelines capture the experiences, good practices, activities, and lessons learned of the Working Groups and present them in a comprehensive, cohesive set of documents for all to use and benefit from. The objective is to reuse what works rather than reinvent and to foster consistency across the various Working Group quality activities and deliverables.

This document does not preclude the need to apply the W3C Manual of Style [STYLE-MAN] and to conform to the Publication Rules [PUBRULES] (Member-only). It is intended to complement those resources.

1.5. Understanding and using this document

This document employs the WAI (Web Accessibility Initiative) model for representing guidelines or general principles for the development of conformance materials. See, for example, Web Content Accessibility Guidelines 1.0 [WCAG10]. Each guideline includes:

• the guideline number,
• the statement of the guideline,
• the rationale behind the guideline,
• a collection of one or more checkpoints.

The guidelines are of two general types:

• those that deal solely with the document features and conventions of the specification — (GL 1, GL 7, GL9-10),
• those that, in addition to documentation aspects, deal with how specifications should establish and define the conformance policy for the specification's technology, including ways in which the technology may be subdivided for conformance purposes —(GL 2-6 and GL 8).

The checkpoints in each guideline define specification characteristics and requirements needed to fulfill the purpose of the guideline. Each checkpoint definition includes:

• the checkpoint number,
• the statement of the checkpoint (its "title"),
• the priority of the checkpoint,
• a statement of the conformance requirements of the checkpoint, i.e., how to fulfill the checkpoint,
• optional informative notes, clarifying examples, and cross references to related guidelines or checkpoints,
• a link to the corresponding section in Specification Examples & Techniques.

Each checkpoint is intended to be specific enough so that someone can implement the checkpoint as well as verify that the checkpoint has been satisfied. A checkpoint will contain at least one, and may contain multiple individual requirements, that use RFC2119 normative keywords. See the "Conformance" chapter for further discussion of requirements and test assertions.

Two separate appendices to this document [SPEC-CHECKLIST] and [SPEC-ICS] present all checkpoints in a tabular form sorted in their original order and sorted by their priorities, for convenient reference. The latter is an Implementation Conformance Statement (ICS) pro forma for this specification. (See GL9.).

1.6. Usage of terminology in this document

The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY ", and "OPTIONAL" will be used as defined in RFC 2119 [RFC2119]. When used with the normative RFC2119 meanings, they will be all uppercase. Occurrences of these words in lowercase comprise normal prose usage, with no normative implications.

Unusual terms in these framework documents are defined when first used. When used in this specification, terms have the meaning assigned in the "Definitions" chapter and the QA Glossary [QA-GLOSSARY]. Terms in Definitions may supplement or build on the definitions in the generic QA Glossary, further refining the definition in the context of the Specifications guidelines. Terms herein that also appear in the QA Glossary include a link to the QA definition.They will not contradict the generic definitions.

1.7. Checkpoint priorities

Some checkpoints are more critical than others for producing a high quality, testable standard that is a sound basis for successfully interoperable products. Therefore each checkpoint is assigned a priority level based on the checkpoint's impact on the quality of the specifications produced by the Working Groups.

[Priority 1]

Critical/Essential. These checkpoints are considered to be basic requirements for ensuring an acceptable level of quality and testability in the specification.

[Priority 2]

Important/desirable. Satisfying these checkpoints, in addition to the priority 1 checkpoints, should significantly improve the clarity, quality, and testability of the standard, as well as its suitability as a basis for building quality test materials.

[Priority 3]

Useful/beneficial. Satisfying these checkpoints, on top of all the others, will further improve the quality, usability, and testability of the specification.

1.8. Use cases

The Specification Guidelines are intended to be used in the following typical cases:

• A Working Group is developing a new specification or a new edition of a specification. The WG gets familiar with the principles in the Specification Guidelines as early as possible, since it can influence what goes into the specification as well as the organization, structure, and presentation of content in the specification.
• A Working Group is concerned by interoperability issues discovered during the implementations phase (e.g. during the Candidate Recommendation), and tries to address them by making the conformance requirements more constraining. It uses the Specification Guidelines to identify the common interoperability issues explained in the document.
• A Working Group develops a specification that it wants to move to Candidate Recommendation in the near future and knows that it will develop a Test Suite during this phase; to ensure that the building of the Test Suite will be as efficient and accurate as possible, it uses the Specification Guidelines to have a clearer idea on how to make the specification testable.
• A Working Group has committed in its charter to the Specification Guidelines for its specifications (as required for A conformance to the Operational Guidelines, for instance), and has picked the AA conformance level; it uses the Specification Guidelines to assess the conformance of its specifications at different stages of the specification life.
• A Working Group asks others to review their specification. The Working Group indicates its claim of conformance to the Specification Guidelines, i.e., the degree of conformance met. The reviewers use the Specification Guidelines as part of their evaluation criteria and ensure that the appropriate checkpoints in the Specification Guidelines are satisfied.

2. Concepts

This section details some of the concepts that are used in the following guidelines. The reader is invited to become familiar with them before further reading.

2.1. Dimensions of Variability (DoV)

Several of the guidelines, GL2 through GL6, address a way in which the conformance policy of a specification might allow variation among conforming implementations. For example, a specification might allow implementations to choose between one of two well defined behaviors for a given functionality, thus two conforming implementations might vary on that aspect.

The Working Group has identified seven ways in which a specification can allow variability, that are refered to as dimensions of variability (DoV). The seven dimensions of variability recognized by this document are:

• classes of product(Guideline 2),
• profiles (Guideline 3),
• modules (Guideline 3),
• levels (Guideline 3),
• deprecation (Guideline 4),
• discretionary items (Guideline 5),
• extensibility (Guideline 6).

The above are not necessarily all orthogonal to one another. There are many possible associations, dependencies, and interrelationships. As a general policy, these specification guidelines do not attempt to legislate correct or proper relationships among the DoV. Rather, they try to clarify the nature of each dimension and require a specification to make deliberate and well documented choices. Some discussion of possible interrelationships, including examples, will be found in the Specification Examples & Techniques.

The dimensions of variability are one of the principal concepts in this guidelines document to help organize, classify and assess the conformance characteristics of W3C specifications. The seven DoV get special attention because they are at the core of the definition of a specification's conformance policy, and thus there is significant potential for negative interoperability impacts if they are handled carelessly or without careful deliberation.

As a general principle, variability complicates interoperability. In theory, interoperability is best when there are numerous identical, complete, correct implementations. However, in practice, the net effect of conformance variability is not necessarily negative in all cases, when compared to the alternatives. For example profiles — subdivisions of the technology targeted at specific applications communities — introduce variability among implementations. Some will implement Profile ABC, some will implement Profile XYZ, and the two might not intercommunicate well if ABC and XYZ are fairly different. However, if ABC and XYZ are subsets of a large monolithic specification — too large for many implementors to tackle in total -- and if they are well targeted at actual application sectors, then subdivision by profiles may actually enhance interoperability.

Different sorts of variability have different negative and positive impacts. The principal danger is "excessive" variability - variability which goes beyond that needed for a positive interoperability trade-off, and which unnecessarily complicates the conformance landscape. Specification writers need to carefully consider and justify any conformance variability allowed, do so by reference to the project requirements and use cases, and explicitly document the choices made.

Note that the variability addressed by the so called dimensions of variability is only considered with regard to conformance to a well-defined specification. As such, the changes introduced in the conformance requirements between two versions or two editions of the specification are not considered as dimensions of variability.

2.2. Specification category and class of product

Specification category

To answer the question "what needs to conform?" it helps to first look at the nature of the specification and categorize it and then look at the types of products that would implement the specification.Categorizing the specification provides a basis for classifying the software that may be affected by the specification. The specification category is the generic name for the type of specification and the technology it describes.

The following is a list of some of the most common specification categories:

• foundation or abstract (e.g., Infoset),
• content/data (e.g., MathML, SVG),
• protocols (e.g., SOAP),
• processors (e.g., XSLT, XML Query),
• APIs (e.g., DOM),
• notation/syntax (e.g., XPath),
• set of events (e.g., one part of XForms),
• rules for deriving profiles (e.g., part of SMIL),
• guidelines (e.g., WCAG, QA Operational Guidelines).

The categories indicate what the specification describes. One specification could potentially fall into more than one category. This list does not exhaust all possibilities. Specifications may have to define their own specification category if none of these fits.

Classes of products

From this categorization of specifications, the WG can identify the class of products that are affected by the specification. Classes of products can be generalized as either producers or consumers, or as content itself.

For example, identifying which are producers and consumers is clear for a protocol-type specification: the two parties to the dialog are the targets. For a processor-type specification, the processor is the consumer of an XML vocabulary defined in the specification. For content-type specifications, there may be one or more consumers that take the content and 'play' it in some way.

The following is a list of the most common classes of products for W3C specifications:

• content (of type, meaning, and format as defined in the specification),
• producer of content (may be divided into initiators and modifiers),
• player (read-only consumer, conveys content in non- XML way),
• consumer in a one-way pipeline,
• responding agent (e.g., server) of API (consumer and producer),
• processor (consumer of its vocabulary/instructions),
• module that hosts the processor,
• producer of instructions/commands to processor,
• profile derived from the specification's Rules for Profiles,
• specification (guidelines).

This list does not exhaust all possibilities. Specifications may have to define their own classes of product if none of these fits.

2.3. Profiles, Modules, Levels

Profiles, modules and levels are three ways to subdivide a specification into related groups of conformance requirements. Because these three dimensions of variability define subsets of a technology, they share some characteristics in the way they affect conformance and interoperability.

A profile is a subset of the technology that supports a particular functional objective or a subset of a set of technologies defining how they are required to operate together (e.g., XHTML plus MathML plus SVG).

Profiles can be based on hardware considerations associated with target product classes — for example, SVG Tiny is aimed at mobile phones — or they may be driven by other functional requirements of their target constituencies — for example, a graphical profile tailored for technical illustrations in aircraft maintenance manuals.

The use of profiles to divide the technology is described in the specification, and may or may not be reflected and paralleled by the structure and organization of the specification.

Specifications may define individual profiles, or may define rules for profiles, or both. An individual profile defines the requirements for classes of products that conform to that profile. Rules for profiles define validity criteria for profiles themselves — i.e., if others (users, applications, or other standards) define their own profiles of the standard (called derived profiles of the specification), then rules for profiles define the constraints that those derived profiles must satisfy in order to be considered valid profiles of the specification.

For example, XHTML Modularization ([XHTML-MOD], section 3) and Synchronized Multimedia Integration Language (SMIL 2.0), [SMIL20] specifications both define rules for profiles -- what constraints must a profile meet in order to be classified as a "Host Language Profile" or an "Integration Set Profile." SMIL further defines some specific profiles, using the modularization. Separate recommendations -- XHTML Basic [XHTML-BASIC] and XHTML 1.1 [XHTML11] — define specific profiles based on the XHTML modularization.

Modules are discrete divisions or functional groupings of the technology and do not necessarily fit in a simple hierarchical structure.

Modules generally can be implemented independently of one another — e.g., audio vs. video module. That notwithstanding, it is possible for one module's definition (and therefore implementation) to have explicit dependency upon another module. It is not only possible, but common to implement multiple modules.

Functional levels — or in common usage, simply levels — are used to group functionality into nested subsets, ranging from minimal or core functionality to full or complete functionally. Level 1 is the minimum or core of the technology. Level 2 includes all of level 1 plus additional functionality. This nesting continues until level n, which consists of the entire technology.

Levels may result from progressive historical development and enrichment of the technology in a series of specifications, as in the case of CSS and DOM. Levels could also be defined explicitly in a single edition of the specification, but no examples of this are found in W3C specifications. Rather, it is more common in current W3C practice to use profiles to accomplish this. For example, SVG 1.1 [SVG11] together with SVG Mobile [Mobile [SVG-MOBILE] define three nested profiles — Tiny, Basic, Full — which are each targeted at a specific graphics hardware community (mobile phone, hand-held computer, desktop computer).

2.4. Addressing relationships among Dimensions of Variability used

As stated above, variability affects interoperability and it is important for the specification writers to have this in mind while designing the specification. But it is even more important to take into account the multiplicative effect on variability created by combining several dimensions of variability.

Hence, each pair of dimensions of variability used in a specification needs to be assessed with regard to the variability it creates; the writers should document the limited ways an implementation can combine two dimensions. For instance, deprecated features in HTML 4.01 [HTML4] are allowed in the Transitional profile and forbidden in the Strict profile.

3. Guidelines

Guideline 1. Define Scope.

When writing specifications it is critical to understand their primary purpose and scope. A clearly defined scope helps to keep the specification content focused and unambiguous.

Checkpoints

Guideline 2. Identify what needs to conform and how.

Identifying what needs to conform to the specification is essential for the specification writers as well as for the implementors. This guideline relies on the concepts of classes of products and specification categories.

Classes of product is a dimension of variability (DoV).

Checkpoints

Guideline 3. Address the use of profiles, modules and levels to divide the technology.

To address various needs (re-usability, evolution, scaling, ...), technologies are often divided in different ways. These divisions have an important impact on the conformance to a specification. This guideline relies on the concepts of profiles, modules and levels that the QA WG has identified as being the three main division types in use.

Profiles, modules and levels are each a dimension of variability (DoV).

Checkpoints

Guideline 4. Identify the relation between deprecated features and conformance.

A deprecated feature is an existing feature that has been outdated by newer constructs or is no longer viable. Deprecated features should be avoided (e.g. for a format language, not be used by the authors and authoring tools) and may be removed in some future version, at which time the feature becomes obsolete.

After the initial publication of a specification, specification developers may consider the deprecation of a feature (e.g., function argument, element or attribute) defined in the specification. Deprecated features may become obsolete and no longer defined in future versions of the specification. Deprecation of a feature may warn implementers that the feature was a bad idea and it may be withdrawn in the future. Specification developers need to consider the effect of deprecation on all the classes of products that implement the specification (e.g., authoring tools, user agents) as well as the conformance consequences on each class of product. For the purpose of backward compatibility, it may be necessary to specify different requirements for the support of deprecated features for each class of product. For example, authoring tools (producers) would not use the feature, but user agents (consumers) would continue to support it.

Deprecation is a dimension of variability (DoV).

Checkpoints

Guideline 5. Define discretionary items.

Discretionary items are defined as deliberate and explicit grants of discretion by the specification to the implementations, that describe or allow optionality of behavior, functionality, parameter values, error handling, etc.

Discretionary items are often made available in specifications, to give implementers of the technology the opportunity to decide from alternatives when building applications and tools. Discretionary items may be considered necessary because of environmental conditions (e.g., hardware limitations or software configuration, or external systems), locality (e.g., time zone or language), optional choices providing flexibility of implementation, dependence on other specifications, etc.

Discretionary items are a dimension of variability (DoV).

Checkpoints

Guideline 6. Allow extensions or not

An extension to a specification is a mechanism to incorporate functionality beyond what is defined in the specification. Extensions can be implemented individually (e.g., one function at a time) or by defining new profiles, modules, or levels that incorporate additional functionality beyond what is defined in the specification.

Allowing extensions affects how conformance is defined as well as what conformance claims can be made. Exercise caution in determining the extent to which extensions are allowed or not allowed. Since extensions can seriously compromise interoperability, specification writers should carefully consider the degree to which extensions are be allowed.

Disallowing extensions for any part of the specification enables strict conformance: strict conformance is defined as conformance of an implementation that employs only the requirements and/or functionality defined in the specification and no more (i.e., no extensions to the specification are implemented). Dimensions of variability (e.g., modules, profiles, levels) are not extensions if the specification defines them or allows them to be defined.

Extensions may be private (often vendor specific) or public (a full description of the extension is public). Private extensions are usually truly private, i.e., valid for a specific implementation or are only known by prior agreement between implementations. Public extensions are extensions in which the syntax, semantics, identifiers, etc. are defined and published allowing anyone to implement the extended functionality.

Specifications allow extensions for various reasons. Extensions allow implementers to include features that they consider to be required by their customers. Also, extensions often define new features that may migrate into future versions of the specifications. However, the use of extensions can have a severe negative impact on interoperability. Some methods for enabling extension have less impact on interoperability than other methods.

Extensions are a dimension of variability (DoV).

Checkpoints

Guideline 7. Clearly identify conformance requirements.

The checkpoints of this guideline aim to ensure that normative content and conformance requirements are easily identifiable in specifications. Some of the desirable characteristics of conformance requirements are that they are mutually independent from other requirements, express a minimal requirement (i.e., are atomic), and are distinguishable and labeled (e.g., tagged). It is important that specifications provide unambiguous statements, that clearly define what is required in order to claim conformance and what is optional. It is important also that the conformance statements are easily identifiable as such, and easily locatable. Clarity is fostered by uniformity of structure and style, and consistency of terminology and phraseology.

There is a lot to be said for consistency and clarity within a document - it facilitates the understanding and comprehension of the document. Authors and editors of specifications should already be familiar with the Publication Rules (Member-only) [PUBRULES] and W3C Manual of Style [STYLE-MAN], which help to achieve clarity and uniformity. This guideline builds on those general document conventions, with a particular focus on the presentation and identification of conformance information.

Checkpoints

Guideline 8. Document the conformance policy.

A look at various W3C Technical Reports shows that the term "conformance" is often qualified, resulting in more than one type of conformance. It is important to convey an understanding of what is meant by conformance and how it applies to each class of product as well as each dimension of variability (e.g., modules) if applicable. For example, if the specification defines behavior for more than one class of product, there may be a separate conformance policy for each class. Similarly, if the specification defines modules, there may be a different conformance policy for each module.

In particular, a reader intending to implement a product in one of the product classes addressed by the specification should know what the WG wants for interoperability among products in the class. The developer should understand what forms, if any, of "product differentiation" are allowed among conformant products. The specification may need to explain how the rules apply and provide examples.

A conformance clause is a part or collection of parts of a specification that defines the requirements, criteria, or conditions to be satisfied by an implementation or application in order to claim conformance. Typically the conformance clause is a high-level description of what is required of implementations and applications.

Checkpoints

Guideline 9. Specify how to make conformance claims.

A specification may differentiate conformance claims by designating different degrees or types of conformance in order to apply and group requirements according to modules, profiles, levels, or priorities. When a conformance claim is linked to functionality, impact and/or incremental degrees of implementation, the term 'conformance level' is often used to indicate the varying degrees of conformance. The use of this term, conformance level, is discouraged and should be replaced by the term, 'degree of conformance'. The WG includes in the specification the way they want people to claim their conformance.

An Implementation Conformance Statement (ICS) provides a mechanism whereby a supplier of an implementation of the specification provides information about the implementation in a standardized manner. It is used to indicate which capabilities and options have been implemented, as well as the limitations of the implementation. An ICS usually takes the form of a questionnaire where product implementors report on the conformance of their product to the named specification.

An ICS is useful in disclosing optional functionality and discretionary behavior and values. The results of the ICS can be used to identify the subset of test cases from a conformance test suite that are applicable to the implementation to be tested. This will allow the implementation to be tested for conformance against only the relevant requirements.

The basic and detailed information that an ICS provides can also be used to assess and deduce the interoperability potential of two or more products.

Checkpoints

Guideline 10. Provide test assertions.

A test assertion is a statement of behavior, action, or condition that can be measured or tested. It is derived from the specification's requirements and provides a normative foundation from which test cases can be built. Each test assertion is an independent, complete, testable statement for requirements in the specification. Each test assertion results in one or more test cases. Multiple test assertions can be combined to form a test case, in this case one tests multiple facets of a particular behavior. It is recommended that test assertions be available by the time a specification enters Proposed Recommendation.

Checkpoints

4. Conformance

This section defines conformance of Working Group specifications — i.e., technical reports — to the requirements of this QA Framework guidelines specification. The requirements of this guidelines specification are detailed in the checkpoints of the preceding "Guidelines" chapter, and apply to the technical reports produced by Working Groups.

4.1. Normative parts

The following parts ofthis document are normative:

• Statements prefaced by the keywords,"Conformance Requirements:",
• Statements using an uppercase keyword from RFC 2119,
• Any non-applicability statement(s) associated with a checkpoint,
• Section 4, Conformance.

Text that is designated as normative is directly applicable to achieving conformance to this document. Informative parts of this document consist of examples, extended explanations, terminology, and other matter that contains information that should be understood for proper implementation of this document.

4.2. Extensibility

This specification is extensible. That is, adding conformance related information and structure to the document in ways beyond what is presented in this specification is allowed and encouraged. Extensions to this specification MUST not contradict nor negate the requirements of this specification (see also the checkpoint about not contradicting a specification through extensions).

The rationale for allowing Working Groups to define extensions to these specficiation guidelines is that these requirements are considered to be the minimal requirements for writing testable technical reports. Doing more than the minumum is not only acceptable, but beneficial. Extensions also allow Working Groups to tailor their specifications more closely to the technology and their specific needs.

4.3. Conformance requirements and test assertions

Within each prioritized checkpoint there is at least one conformance requirement.These are prefaced by "Conformance requirements" and highlighted in a different style. This Specification Guidelines enumerates a normative list of test assertions [QA-TA], which were derived from each MUST requirement.

4.4. Conformance definition

This section defines three degrees of conformance to this guidelines specification:

• A-Conforming: all Priority 1 checkpoints are satisfied,
• AA-Conforming: all Priority 1 and Priority 2 checkpoints are satisfied,
• AAA-Conforming: all Priority 1, Priority 2, and Priority 3 checkpoints are satisfied.

A checkpoint is satisfied by fulfilling all of the mandatory conformance requirements. Mandatory requirements are those that use the conformance keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", or "SHALL NOT".

Note that it is allowed (even encouraged) to implement checkpoints in addition to those required to satisfy a particular degree of conformance (A, AA, or AAA). The more checkpoints that are satisfied, the better. However, there is no additional conformance designation for such intermediate collections of checkpoints (i.e., for all checkpoints of a given level plus some, but not all, of the checkpoints of the next levels).

The minimum recommended conformance to this specification is A-conforming -- all Priority 1 (critical/essential) checkpoints satisfied.

4.5. Conformance claims

A specification conforms to the QA Framework: Specification Guidelines at degree X (A, AA, or AAA) if the Working Group meets at least all degree X conformance requirements.

An assertion of conformance to this specification -- i.e., a conformance claim -- MUST specify:

• the URI of a frozen dated version of the specification that is claiming conformance,
• the guidelines title: QA Framework: Specification Guidelines,
• the guidelines dated URI: http://www.w3.org/TR/2003/CR-qaframe-spec-20031110/,
• the date of the claim,
• the degree of conformance satisfied: A-Conforming, AA-Conforming, or AAA-Conforming.

Example:

On 22 September 2003, W3C's QA Framework: Operational Guidelines (http://www.w3.org/TR/2003/CR-qaframe-ops-20030922), dated 22 September 2003 conforms to W3C's QA Framework: Specification Guidelines, available at http://www.w3.org/TR/2003/WD-qaframe-spec-20030912/, Conformance level AA.

The checklist for this specification ([SPEC-ICS]) is the Implementation Conformance Statement (ICS) pro forma for this specification. Any assertion of conformance to this specification MUST link to a completed ICS.

4.6. Conformance disclaimer

The checkpoints of this guidelines specification present verifiable conformance requirements about the specifications (technical reports) that Working Groups produce. As with any verifiable conformance requirements, users should be aware that:

Passing all of the requirements to achieve a given degree of conformance — A, AA, or AAA — does not guarantee that the subject specification is well-suited to or will achieve its intended purposes, nor does it guarantee the quality or suitability of test materials produced from the specification.

5. Definitions

This section contains terms used in this specification, with functional or contextual definitions appropriate for this specification. See also [QA-GLOSSARY]. Some terms in this section have been borrowed or adapted from other specifications.

class of product

the generic name for the group of products that would implement, for the same purpose, the specification, (i.e., target of the specification). The class of product is the object of the conformance claim. A specification may identify several classes of products.

conformance clause

a part or collection of parts of a specification that defines the requirements, criteria, or conditions to be satisfied by an implementation or application in order to claim conformance.

conformance level

a variety of conformance designation. Other designations include conformance category, conformance degree, conformance xxx. "Conformance level" is discouraged in new specifications, because of confusion with "functional level".

conformance requirement

a condition for conformance of an implementation to a specification. Conformance requirements can have different levels of necessity: mandatory, recommended, or optional.

deprecated

an existing feature that has become outdated by a newer construct or is no longer viable. Deprecated features should be avoided, since they may be removed in some future version.

derived profile

a profile that is created from a set of profile rules, where these profile rules provide instructions for building profiles (i.e., defining profiles) and the rules are defined in a specification.

dimensions of variability (DoV)

the ways in which different products that are conformant to a specification may vary among themselves. In this Specification Guidelines document, the dimensions of variability are used to help organize, classify and assess the conformance characteristics of W3C specifications.

discretionary items

deliberate and explicit grants of discretion by the specification to the implementations, that describe or allow optionality of behavior, functionality, parameter values, error handling, etc.

functional level

a technology subset that is one of a hierarchy of nested subsets, ranging from minimal or core functionality to full or complete functionally.

implementation conformance statement (ICS)

a mechanism for providing standardized information about an implementation of a specification, usually in the form of a questionnaire in which product implementers report the product's conformance to the specification. An ICS is used to indicate which requirements, capabilities, and options have and have not been implemented.

informative text

text in a specification whose purpose is informational or assistive in the understanding or use of the specification, and which contains no conformance requirements or test assertions.

level

a commonly used shorthand for functional level.

module

a collection of semantically-related elements, attributes, and attribute values that represents a unit of functionality. Modules are non-hierarchical, discrete divisions that are defined in coherent sets.

normative text

text in a specification which is prescriptive or contains conformance requirements.

obsolete

feature that is no longer defined in the specification. A feature is often deprecated before becoming obsolete.

profile

a subset of a technology that is tailored to meet specific functional requirements of a particular application community. A profile may address a single technology; or, a profile can also group a set of technologies (i.e., from different specifications) and define how they operate together. Profiles may be based on hardware considerations associated with target product classes, or they may be driven by other functional requirements of their target communities.

profiling

a method for defining subsets of a technology by identifying the functionality, parameters, options, and/or implementation requirements necessary to satisfy the requirements of a particular community of users.

specification category

the generic name for the type of specification and the technology it describes.

strict conformance

conformance of an implementation that employs only the requirements and/or functionality defined in the specification and no more (i.e., no extensions to the specification are implemented).

test assertion

a statement of behavior, action, or condition that can be measured or tested. It is derived from the specification's requirements and provides a normative foundation from which test cases can be built. (See also QA Glossary [QA-GLOSSARY].)

use case

a specification mechanism or technique that captures the ways a specification would be used, including the set of interactions between the user and the specification as well as the services, tasks, and functions the specification is required to perform.

---

6. Acknowledgments

The following QA Working Group and Interest Group participants have contributed significantly to the content of this document:

• Patrick Curran (Sun Microsystems)
• Daniel Dardailler (W3C)
• Dimitris Dimitriadis (Ontologicon)
• Karl Dubost (W3C)
• Peter Fawcett (RealNetworks)
• Kirill Gavrylyuk (Microsoft)
• Dominique Hazael-Massieux (W3C)
• Lofton Henderson (CGM Open)
• Ian Jacobs (W3C)
• Sandra Martinez (NIST)
• David Marston (IBM Research)
• Lynne Rosenthal (NIST)
• Alex Rousskov (Measurement Factory)
• Mark Skall (NIST)
• Andrew Thackrah (Open Group)
• Olivier Thereaux (W3C)
• Vanitha Venkatraman (Sun Microsystems)

7. References

7.1. Normative

[QA-GLOSSARY]

Comprehensive glossary of QA terminology. (Under construction, at http://www.w3.org/QA/glossary.)

[RFC2119]

Key words for use in RFCs to Indicate Requirement Levels, March 1997, available at http://www.ietf.org/rfc/rfc2119.txt.

[QA-TA]

Test assertions for "QA Framework: Specification Guidelines", 10 November 2003, Mark Skall Editor, available at http://www.w3.org/TR/2003/CR-qaframe-spec-20031110/qaframe-spec-ta .

7.2. Informative

[CSS2]

Cascading Style Sheets, Level 2, W3C Recommendation, 12 May 1998, available at http://www.w3.org/TR/REC-CSS2/.

[HTML4]

HTML 4.01 Specification, W3C Recommendation, 24 December 1999, available at http://www.w3.org/TR/html401/.

[MATHML20]

Mathematical Markup Language (MathML) Version 2.0, W3C Recommendation, 21 February 2001, available at http://www.w3.org/TR/MathML2/.

[PUBRULES]

W3C Publication Rules, available at http://www.w3.org/Guide/pubrules.html (Member-only).

[QAF-INTRO]

QA Framework: Introduction, L. Henderson, L. Rosenthal, Eds., W3C Working Draft, September 2003, companion version to this document, available at http://www.w3.org/TR/2003/NOTE-qaframe-intro-20030912/ .

[QAF-OPS]

QA Framework: Operational Guidelines, L. Henderson, D. Hazaël-Massieux, L. Rosenthal, K. Gavrylyuk, Eds., W3C Candidate Recommendation, September 2003, companion version to this document, available at http://www.w3.org/TR/2003/CR-qaframe-ops-20030922/.

[QAF-TEST]

QA Framework: Test Guidelines, P. Curran, P. Fawcett, K. Gavrylyuk, D. Dimitriadis, L. Henderson, M. Skall, Eds, W3C Working Draft, 16 May 2003, companion version to this document, available at http://www.w3.org/TR/2003/WD-qaframe-test-20030516/ .

[QAIG]

Quality Assurance Interest Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/IG/.

[QAWG]

Quality Assurance Working Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/WG/.

[SMIL20]

Synchronized Multimedia Integration Language (SMIL 2.0), W3C Recommendation, 07 August 2001, available at http://www.w3.org/TR/smil20/.

[SPEC-CHECKLIST]

An appendix to this specification guidelines document presents all checkpoints in tabular form sorted in their original order. Available at http://www.w3.org/TR/2003/CR-qaframe-spec-20031110/qaframe-spec-checklist.

[SPEC-ICS]

An appendix to this specification guidelines document presents all checkpoints in tabular form sorted by priorities. Available at http://www.w3.org/TR/2003/CR-qaframe-spec-20031110/qaframe-spec-ics.

[SPEC-EXTECH]

QA Framework: Specification Examples & Techniques, available at http://www.w3.org/QA/WG/2003/09/qaframe-spec-extech .

[STYLE-MAN]

W3C Manual of Style, summarizing the style and publication rules for W3C technical reports, available at http://www.w3.org/2001/06/manual/.

[SVG11]

Scalable Vector Graphics (SVG) 1.1 Specification, D. Jackson, J. Ferraiolo, J. Fujisawa, Eds., W3C Recommendation 14 January 2003, available at http://www.w3.org/TR/2003/REC-SVG11-20030114/.

[SVG-MOBILE]

Mobile SVG Profiles: SVG Tiny and SVG Basic, T. Capin, Editor, W3C Recommendation, 14 January 2003, available at http://www.w3.org/TR/2003/REC-SVGMobile-20030114/.

[WCAG10]

Web Content Accessibility Guidelines 1.0, W. Chisholm, I. Jacobs, G. Vanderheiden, Eds., W3C Recommendation, 5 May 1999, available at http://www.w3.org/TR/WCAG10/.

[XFORMS]

XForms 1.0, M. Dubinko, L. L. Klotz Jr., R. Merrick, T. V. Raman, Eds., W3C Candidate Recommendation, 12 November 2002, available at http://www.w3.org/TR/2002/CR-xforms-20021112/.

[XHTML-MOD]

Modularization of XHTML, M. Altheim, F. Boumphrey, S. Dooley, S. McCarron, S. Schnitzenbaumer, T. Wugofski,Eds., W3C Recommendation, 10 April 2001, available at http://www.w3.org/TR/xhtml-modularization/.

[XHTML-BASIC]

XHTML Basic, M. Baker, M. Ishikawa, S. Matsui, P. Stark, T. Wugofski, T. Yamakami, Eds., W3C Recommendation, 19 December 2000, available at http://www.w3.org/TR/xhtml-basic/.

[XHTML11]

XHTML 1.1 - Module-based XHTML, M. Altheim, S. McCarron, Eds., W3C Recommendation, 31 May 2001, available at http://www.w3.org/TR/xhtml11/.

[XML10]

Extensible Markup Language (XML) 1.0 (Second Edition), T. Bray, J. Paoli, C. M. Sperberg-McQueen, E. Maler, Eds., W3C Recommendation, 6 October 2000, available at http://www.w3.org/TR/REC-xml.

8. Change history

2003-11-10, Candidate Recommendation

This version integrated the feedback received on the previous intermediate draft, where only minor changes were requested:

• The glossary section and the reference to the QA Glossary are now normative
• the test assertion definition has been completed
• plus a few other trivial fixes

2003-09-12, intermediate WD

This version integrated the resolutions of the issues raised during the Last Call review; the document has been reordered and some of the guidelines have been consolidated; furthermore, many checkpoints have been reworded for sake of clarification or adjustment.

• a new "Concepts" section (section 2) gathers the most important concepts used throughout the document
• Former GL 4,5 and 6 have been consolidated into GL 3; during the operation, former CP 4.3, 5.2 and 6.1 have been merged, and most of the verbiage of those GL have been merged in the section 2.3
• Former GL 3 and 10 have been merged into GL 8
• Former GL 11 and 12 have been merged into GL 9
• GL 13 have been moved up as GL 7
• Test assertions derived from the specification are now available
• CP 9.1 and 9.2 have been merged
• The conformance section has been completed
• Rationales have been added to some CP

The detailed changes are more visible in the editor's version of SpecGL.

2003-02-10, Last Call WD

This version integrates the resolutions taken by the WG on the issues raised since the last Working Draft; a good set of them have been taken during the QA WG F2F in Seattle. The structure hasn't really changed since the last draft, except that some checkpoints were removed, some were added.

• the scope part of the introduction has been reworded for clarification
• added an explicit section on normative sections
• created a list of checkpoints ordered by priority, that is used as the ICS for the specification
• the conformance requirements are now explicitly labeled as such
• the following CP have been removed (using numbers of the previous version): CP 1.4
• the following CP have been added (using numbers of this version): CP 2.4, 3.2, 3.3, 3.5, 7.3, 8.5, 9.7
• the following sections have been significantly modified: GL 3 verbiage, GL 13 verbiage, CP 1.1, 1.2, 8.3, 8.4, 9.4

2002-11-08, third published WD

While keeping most of the principles behind the second published WD, this version has re-ordered the guidelines in a more logical way, and takes a more formal approach in the design of CP, where all CP has a set of test assertions. Besides, repetitions have been diminished through factorization, and non-testable or arguable CP have been removed or marked as to be moved as examples and techniques. Finally, it integrates all the issues resolutions agreed during the QA WG Tokyo face-to-face meeting and the weeks following.

• the introduction integrates explicit scope and goals as required by the GL and has been cleaned of its repetitions
• usage of RFC 2119 has been clarified with the usage of uppercase keywords
• the dimensions of variability have now their own section in the introduction, regrouping the various bits previously scattered around
• ex-GL 14 on granular grammars has been removed, some of its content being integrated in the GL on test assertions
• ex-GL 6 on conformance policy has been moved ahead on GL 3, ex-GL 7 on levels has been grouped with its related GL on profiles and modules
• every CP has now a set of test assertions using RFC 2119 keywords; most of them have an explicit rationale and some of them an example; the style sheet has been modified to highlight the test assertions
• the following CP (using numbering of the previous version) have been removed: 1.4 (Ensure that every test assertion is covered by an example), 2.3 (For each class of product, indicate minimal support requirements.), 3.1 (Choose whether or not to have profiles.), 3.2 (Include a table of contents entry.), 4.1 (Choose whether or not to have modules.), 4.2 (Include a table of contents entry.), 6.5 (Include a table of contents entry.), 7.1 (Address whether the specification uses or will use functional levels.), 7.2 (Include a table of contents entry.), 9.7 (Include a table of contents entry.), 10.3 (Include a conformance clause entry in the table of contents), 10.5 (Identify all dimensions of variability that are not used.) 11.5 (Include a table of contents entry.), 13.3 (Follow Web Accessibility Initiative and Internationalization Guidelines.)
• CP 13.4 (Provide a fast way to find conformance information) is new
• overall, an effort has been put into using a consistent terminology across the specification and avoiding unclear or untestable statement
• the verbiage on each DoV has been vastly reduced or factorized in the DoV discussion in the introduction

2002-08-26, second published WD

Significantly reorganized and revised the first published WD. This version produced as a series of editor's drafts. The changes below are reverse chronological (most recent first), so more recent ones may build on older ones.

• Handle negative disclaimers for DoV's via new CK10.5.
• First cut at distinguishing specifications from technologies that they describe (esp in profiles, modules, levels.)
• Add concept of SpecGL test assertions to "Conformance" chapter.
• Define "use case" and "user scenario", revise GL1 & CK1.2 to use them accordingly, clarify CK1.3 (Provide examples), add rigorous CK1.4 (examples--TAs).
• Clean up "atomicity of modules" discussion, add atomicity to CK4.3
• Harmonize priorities of CK8.1 & 8.2, 9.1 & 9.2.
• All GL14 and GL15 checkpoints to priority 2, except CK15.1; tighten descriptions.
• All TOC checkpoints to priority 2, except Conformance Clause (p1).
• Accommodate "Rules for Profiles" in GL2 categories and classes; new CK3.6 and changes to 2.3.
• Add caveat to "Understanding and using.." (sec. 1.2), about specifications that pre-date SpecGL.
• Rewrite CK11.1 to replace 'levels' in "conformance levels" w/ degrees, categories, etc (confusion with "functional levels").
• Change "levels" to "degrees" in SpecGL conformance clause.
• Change "discretionary behaviors" to "discretionary items" in GL8, define three variants in GL8 verbiage.
• Extensive new prose in GL3 (profiles), GL4 (modules), GL7 (levels), describing potential relationships.
• Fixed definition of strict conformance, clarified definition and usage of "conformance clause".
• Added "Definitions" section (in-spec glossary).
• Enhanced wording of all TOC checkpoints.
• For each of CK3.x (profiles), 4.x (modules), 7.x (levels), x>1, added a "not applicable" disclaimer if the feature is not used. (Except for the TOC checkpoints.)
• Change wording of CK7.x, eliminating "chosen", and add some new explanation to 7.1
• Delete CK5.4 (conformance policy in conformance clause) and move its verbiage to GL10 (conformance clause) and its checkpoints.
• Fix "Conformance" chapter, add "Checkpoint Priorities" section.
• Change chapter title of "Navigating through this document" to "Understanding and using this document"
• New text for Checkpoint 3.4
• Added warnings about "excessive variability" to all dimension-of-variability guidelines.
• Added to several DoV checkpoints a requirement to explain/justify any variability dimension by use cases and requirements.
• Fix separation between last two guidelines, "Granular Grammars" and "Test Assertions".
• Introduce and motivate "dimensions of variability" (DoV) in sec 1.5
• Reordered guidelines (GL), split GL.5 into three new ones, GL.3, GL.4, and GL.7, customized checkpoints to their topics (profiles, modules, levels).
• Added new p3 checkpoint ("categorize specification") to (new) GL.2
• Rewrote (old) GL.3, "conformance flavors", in terms of "specify conformance policy, (new) GL.5.
• Split CK1.3 into two, wrote new rationale and removed CK1.4.
• Replaced CK9.1 and 9.2 with proposed wording from Issue 106.
• Modified CK13.4, adding 'direct access' and a reference to DoV checkpoints.
• Deleted CK10.1 and changed priority of CK10.2.
• Added new section on extensibility of SpecGL in the Conformance section.

2002-05-15 version

First published WD.