W3C

QA Framework: Specification Guidelines

W3C Working Draft 12 September 2003

This version:
http://www.w3.org/TR/2003/WD-qaframe-spec-20030912/
Latest version:
http://www.w3.org/TR/qaframe-spec/
Previous version:
http://www.w3.org/TR/2003/WD-qaframe-spec-20030210/
Editors:
Dominique HazaŽl-Massieux (dom@w3.org)
Lofton Henderson (lofton@rockynet.com)
Lynne Rosenthal (lynne.rosenthal@nist.gov)
Dimitris Dimitriadis (dimitris@ontologicon.com)
Kirill Gavrylyuk (kirillg@microsoft.com)
Contributors:
See Acknowledgments.

This document is also available in this non-normative format: single HTML file.


Abstract

The principal goal of this document is to help W3C Working Groups to write clearer, more implementable, and better testable technical reports. It both provides both a common framework for specifying conformance requirements and definitions, and also addresses how a specification might allow variation among conforming implementations,the resentation of specifications (technical reports) as schemata both of which facilitate the generation of test materials. The material is presented as a set of organizing guidelines and verifiable checkpoints. This document is one in a family of Framework documents of the Quality Assurance (QA) Activity, which includes the other existing or in-progress specifications: Introduction, Operational Guidelines, and Test Guidelines.

Status of this document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

This document is a W3C Working Draft (WD), made available by the W3C Quality Assurance (QA) Activity for discussion by W3C Members and other interested parties. For more information about the QA Activity, please see the QA Activity statement.

This Working Draft consolidates changes and editorial improvements undertaken in response to feedback received during the Last Call publication of the Specification Guidelines which began on February 10, 2003. A list of the Last Call issues addressed by the Working Group is also available. This document aims at gathering feedback on the resolutions to these issues before to enter in Candidate Recommendation phase.

This part of the Framework document family has a companion QA Framework: Specification Examples & Techniques. That informative companion document is currently being progressed as a W3C Note. At least until this guidelines document stabilizes, that Examples & Techniques companion will be maintained and frequently updated in QA Working Group Web space. Although that document is not the principal subject of this review, the QAWG welcomes feedback on it as well.

The QA Working Group Patent Disclosure page contains details on known patents related to this specification, in conformance with W3C policy requirements.

Please email your comments to www-qa@w3.org, the publicly archived list of the QA Interest Group [QAIG]. Please note that comments that you make will be publicly archived and available, do not send information you would not want to see distributed, such as private data.

Publication as a Working Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

Table of contents

Two appendices to this document [SPEC-CHECKLIST] and [SPEC-ICS] present all checkpoints in a tabular form sorted in their original order and sorted by priority, for convenient reference. A third appendix presents the test assertions derived from the conformance requirements set by this specification.

Introduction

Scope and Goals

The scope of this document is a set of requirements for W3C Technical Reports (TRs) that if satisfied, will enhance the clarity, implementability, and testability of TRs. It describes what goes into a TR with respect to conformance and conformance topics, dealing with how a TR establishes, defines, and presents its conformance policy. This document addresses the most basic and critical topics with respect to conformance and testabily; more advanced topics will be addressed in future versions. This document includes:

A separate document, entitled "Specification Examples and Techniques 1.0" (the "ExTech document" from here on) provides suggestions and examples of how each checkpoint might be satisfied. The techniques in the ExTech document are informative examples only, other strategies may be used or required to satisfy the checkpoints. The ExTech document is expected to be updated more frequently than the current guidelines. Developers, W3C Working Groups, users, and others are encouraged to contribute techniques and examples.

This document is applied and conformance (to this document) achieved as new TRs are being written. New TRs include those in-development or being revised. As for legacy specification, they may indirectly comply with the spirit or intent of some checkpoints, without actually satisfying all requirements in those checkpoints.

Class of Product and Audience

The specification category for this specification is "guideline". The class of product or target of this specification is "specification", in particular W3C Technical Reports. Within this Specification Guidelines document, the term "specifications" is specifically limited to W3C Technical Reports, even though these guidelines could be applied to other documents.

The primary target audience of this document is specification authors, however, it is applicable to a broader audience including:

These guidelines are designed so that the WGs can apply them in a common-sense and workable manner.

Motivation and expected benefits

Good specifications lead to better implementations and foster the development of conformance test suites and tools. Conforming implementations lead to interoperability.

The quality of the specification has direct impact on the quality of implementations. Quality encompasses utility which refers to the usefulness of the specification to the intended users and objectivity which focuses on the whether the specification is presented in an accurate, clear, complete, and unbiased manner.

Providing requirements and definitions about conformance topics, as well as guidance in the structure and anatomy of specifications, will foster a mutual understanding among those developing specifications, implementations, and conformance test materials. Specifically, well-structured specifications with clear and comprehensive conformance requirements:

Relationship to other specifications

This document is part of a family of QA Framework documents designed to help the WGs improve all aspects of their quality practices. The QA Framework documents are:

Although the QA Framework documents are interrelated and complement each other, they are independently implementable. For example, the anatomy of a specification is related to the type of test materials that will be built, hence there is an interrelationship between this document and the Test Guidelines. The reader is strongly encouraged to be familiar with the other documents in the family.

The Framework as a whole is intended for all Working Groups, as well as developers of conformance materials for W3C specifications. Not only are the Working Groups the consumers of these guidelines, they are also key contributors. The guidelines capture the experiences, good practices, activities, and lessons learned of the Working Groups and present them in a comprehensive, cohesive set of documents for all to use and benefit from. The objective is to reuse what works rather than reinvent and to foster consistency across the various Working Group quality activities and deliverables.

This document does not preclude the need to apply the W3C Manual of Style [STYLE-MAN] and to conform to the Publication Rules [PUBRULES] (Member-only). It is intended to complement those resources. specifications

Understanding and using this document

This document employs the WAI (Web Accessibility Initiative) model for representing guidelines or general principles for the development of conformance materials. See, for example, Web Content Accessibility Guidelines 1.0 [WCAG10]. Each guideline includes:

The guidelines are of two general types:

The checkpoints in each guideline define specification characteristics and requirements needed to fulfill the purpose of the guideline. Each checkpoint definition includes:

Each checkpoint is intended to be specific enough so that someone can implement the checkpoint as well as verify that the checkpoint has been satisfied. A checkpoint will contain at least one, and may contain multiple individual requirements, that use RFC2119 normative keywords. See the "Conformance" chapter for further discussion of requirements and test assertions.

Two separate appendices to this document [SPEC-CHECKLIST] and [SPEC-ICS] present all checkpoints in a tabular form sorted in their original order and sorted by their priorities, for convenient reference. The latter is an Implementation Conformance Statement (ICS) pro forma for this specification. (See GL9.).

Usage of terminology in this document

The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY ", and "OPTIONAL" will be used as defined in RFC 2119 [RFC2119]. When used with the normative RFC2119 meanings, they will be all uppercase. Occurrences of these words in lowercase comprise normal prose usage, with no normative implications.

Unusual terms in these framework documents are defined when first used. When used in this specification, terms have the meaning assigned inThis document contains a the "Definitions" chapter. Generally useful QA-specific terms will eventually be in and the QA Glossary [QA-GLOSSARY]. If terms are in the QA Glossary, their definition herein will refer to that QA Glossary entry with a link. The definitions herein Terms in Definitions may supplement or build on the definitions in the generic QA Glossary, further refining the definition in the context of the Specifications guidelines. Terms herein that also appear in the QA Glossary include a link to the QA definition.that generic definition with other information that is useful or helpful in the specification guidelines context.They will not contradict the generic definitions.

Checkpoint priorities

Some checkpoints are more critical than others for producing a high quality, testable standard that is a sound basis for successfully interoperable products. Therefore each checkpoint is assigned a priority level based on the checkpoint's impact on the quality of the specifications produced by the Working Groups.

[Priority 1]
Critical/Essential. These checkpoints are considered to be basic requirements for ensuring an acceptable level of quality and testability in the specification.
[Priority 2]
Important/desirable. Satisfying these checkpoints, in addition to the priority 1 checkpoints, should significantly improve the clarity, quality, and testability of the standard, as well as its suitability as a basis for building quality test materials.
[Priority 3]
Useful/beneficial. Satisfying these checkpoints, on top of all the others, will further improve the quality, usability, and testability of the specification.

Use cases

The Specification Guidelines are intended to be used in the following typical cases:

Concepts

This section details some of the concepts that are used in the following guidelines. The reader is invited to become familiar with them before further reading.

Dimensions of Variability (DoV)

Several of the Each of the latter set of eight guidelines, GL2 through GL6and GL8, address a way in which the conformance policy of a specification might allow variation among conforming implementations. For example, a specification might allow implementations to choose between one of two well defined behaviors for a given functionality, thus two conforming implementations might vary on that aspect.

For this reason, these seven guidelines are collectively called the "dimensions of variability (DoV)". The Working Group has identified seven ways in which a specification can allow variability, that are refered to as dimensions of variability (DoV). The seven dimensions of variability recognized by this document are:

The above are not necessarily all orthogonal to one another. There are many possible associations, dependencies, and interrelationships. As a general policy, these specification guidelines do not attempt to legislate correct or proper relationships among the DoV. Rather, they try to clarify the nature of each dimension and require a specification to make deliberate and well documented choices. Some discussion of possible interrelationships, including examples, will be found in the Specification Examples & Techniques.

The dimensions of variability are one of the principal concepts in this guidelines document to help organize, classify and assess the conformance characteristics of W3C specifications. The seven DoV get special attention because they are at the core of the definition of a specification's conformance policy, and thus there is significant potential for negative interoperability impacts if they are handled carelessly or without careful deliberation.

As a general principle, variability complicates interoperability. In theory, interoperability is best when there are numerous identical, complete, correct implementations. However, in practice, the net effect of conformance variability is not necessarily negative in all cases, when compared to the alternatives. For example profiles — subdivisions of the technology targeted at specific applications communities — introduce variability among implementations. Some will implement Profile ABC, some will implement Profile XYZ, and the two might not intercommunicate well if ABC and XYZ are fairly different. However, if ABC and XYZ are subsets of a large monolithic specification — too large for many implementors to tackle in total -- and if they are well targeted at actual application sectors, then subdivision by profiles may actually enhance interoperability.

Different sorts of variability have different negative and positive impacts. The principal danger is "excessive" variability - variability which goes beyond that needed for a positive interoperability trade-off, and which unnecessarily complicates the conformance landscape. Specification writers need to carefully consider and justify any conformance variability allowed, do so by reference to the project requirements and use cases, and explicitly document the choices made.

Note that the variability addressed by the so called dimensions of variability is only considered with regard to conformance to a well-defined specification. As such, the changes introduced in the conformance requirements between two versions or two editions of the specification are not considered as dimensions of variability.

Specification category and class of product

Specification category

To answer the question "what needs to conform?" it helps to first look at the nature of the specification and categorize it and then look at the types of products that would implement the specification.Categorizing the specification provides a basis for classifying the software that may be affected by the specification. — and thus, provides the answer to "what needs to conform?". To answer this question, it helps to look at the nature of the specification and categorize it.The specification category is the generic name for the type of specification and the technology it describes.

The following is a list of some of the Most specifications can be classified into one of the most common specification categories:

The categories indicate what the specification describes. One specification could potentially fall into more than one category. This list does not exhaust all possibilities. Specifications may have to define their own specification category if none of these fits.

Classes of products

From this categorization of specifications, the WG can identify the class of products that are affected by the specification. Classes of products can be generalized as either orproducers or consumers, or as content itself.

For example, identifying which are producers and consumers is clear for a protocol-type specification: the two parties to the dialog are the targets. For a processor-type specification, the processor is the consumer of an XML vocabulary defined in the specification. For content-type specifications, there may be one or more consumers that take the content and 'play' it in some way.

The following list is a non-exhaustive list of classes of products for W3C specifications.The following is a list of the most common classes of products for W3C specifications:

This list does not exhaust all possibilities. Specifications may have to define their own classes of product if none of these fits.

One specification could define more than one player. For example, MathML could addresses the behavior of display of math notation and also a consumer that parses the MathML as a formula and uses it for mathematical processing.

Profiles, Modules, Levels

Profiles, modules and levels are three ways to subdivide a specification into related groups of conformance requirements. Because these three dimensions of variability define subsets of a technology, they share some characteristics in the way they affect conformance and interoperability.

A profile is a subset of the technology that supports a particular functional objective or a subset of a set of technologies defining how they are required to operate together (e.g., XHTML plus MathML plus SVG).

Profiles can be based on hardware considerations associated with target product classes — for example, SVG Tiny is aimed at mobile phones — or they may be driven by other functional requirements of their target constituencies — for example, a graphical profile tailored for technical illustrations in aircraft maintenance manuals.

Diagram showing how profiles were used in SVG 1.1
Diagram illustrating profiles used to adapt the SVG Technology to different platforms.

The use of profiles to divide the technology is described in the specification, and may or may not be reflected and paralleled by the structure and organization of the specification.

Specifications may define individual profiles, or may define rules for profiles, or both. An individual profile defines the requirements for classes of products that conform to that profile. Rules for profiles define validity criteria for profiles themselves — i.e., if others (users, applications, or other standards) define their own profiles of the standard (called derived profiles of the specification), then rules for profiles define the constraints that those derived profiles must satisfy in order to be considered valid profiles of the specification.

For example, XHTML Modularization ([XHTML-MOD], section 3) and Synchronized Multimedia Integration Language (SMIL 2.0), [SMIL20] specifications both define rules for profiles -- what constraints must a profile meet in order to be classified as a "Host Language Profile" or an "Integration Set Profile." SMIL further defines some specific profiles, using the modularization. Separate recommendations -- XHTML Basic [XHTML-BASIC] and XHTML 1.1 [XHTML11] — define specific profiles based on the XHTML modularization.

Modules are non-hierarchical, discrete divisions or functional groupings of the technology and do not necessarily fit in a simple hierarchical structure.

Diagram showing how modules were used in XHTML 1.1
Diagram illustrating modules used to divide XHTML 1.1 in re-usable components.

For example, SMIL 2.0 [SMIL20] defines a module as follows:

A module is a collection of semantically-related XML elements, attributes, and attribute values that represents a unit of functionality. Modules are defined in coherent sets.

Modules generally can be implemented independently of one another — e.g., audio vs. video module. That notwithstanding, it is possible for one module's definition (and therefore implementation) to have explicit dependency upon another module. It is not only possible, but common to implement multiple modules.

Functional levels — or in common usage, simply levels — are used to group functionality into nested subsets, ranging from minimal or core functionality to full or complete functionally. Level 1 is the minimum or core of the technology. Level 2 includes all of level 1 plus additional functionality. This nesting continues until level n, which consists of the entire technology.

Diagram showing how levels were used in the DOM
Diagram illustrating levels used to build up the Document Object Model.

Levels may result from progressive historical development and enrichment of the technology in a series of specifications, as in the case of CSS and DOM. Levels could also be defined explicitly in a single edition of the specification, but no examples of this are found in W3C specifications. Rather, it is more common in current W3C practice to use profiles to accomplish this. For example, SVG 1.1 [SVG11] together with SVG Mobile [Mobile [SVG-MOBILE] define three nested profiles — Tiny, Basic, Full — which are each targeted at a specific graphics hardware community (mobile phone, hand-held computer, desktop computer).

Addressing relationships among Dimensions of Variability used

As stated above, variability affects interoperability and it is important for the specification writers to have this in mind while designing the specification. But it is even more important to take into account the multiplicative effect on variability created by combining several dimensions of variability.

Hence, each pair of dimensions of variability used in a specification needs to be assessed with regard to the variability it creates; the writers should document the limited ways an implementation can combine two dimensions. For instance, deprecated features in HTML 4.01 [HTML4] are allowed in the Transitional profile and forbidden in the Strict profile.

Guidelines

Define Scope.

When writing specifications it is critical to understand their primary purpose and scope. A clearly defined scope helps to keep the specification content focused and unambiguous.

Checkpoints

Include the a scope statement. of the specification
Definition

The scope describes the areas covered by the specification, thereby indicating the intent or limits of applicability of the specification. or particular parts of it. It does not specify requirements and is worded as a series of statements of fact.

Conformance requirements

The specification MUST define the subject matter of the specification, and SHOULD include a statement of objectives and/or goals. This information MUST appear at the beginning of the document.

Rationale

It helps the writer and the reader know the limits of what is specified in a normative document. It tells the reader how the specification could be used and by whom.

Discussion

Note that defining the classes of product as required by the checkpoint 2.1 contributes to defining the scope of a document.

Illustrate what is inscope.
Conformance requirements

The specification MUST provideinclude examples or use cases illustrating what is in or out of scope.

Rationale

A set of broadexamples and/or use cases helps to clarify the specification's scope. It gives the reader additional information in understanding the purpose, audience, and applicability of the specification.

Discussion

It is up to the Working Group to determine what the best way to illustrate the scope, i.e., to use examples or use cases or both. , illustrating what is in and/or out of scope. Use cases are a valuable way to describe the different ways a specification can be used. These examples and use cases may be included in the specification or may be in a separate document referenced by the specification.

Illustrate functional details. Provide examples.
Conformance requirements

The specification MUST provide one or more examples of the functionality, concepts, and behaviors of the specification.

Rationale

It is difficult to understand some concepts, behaviors, or functionality without informative interpretations to aid the reader. These examples are specific in nature and are used to make clear a concept, behavior, interaction, etc. that is innately complex or for which the WG has had to explain to its members or the public. aids the reader in interpreting the specification.

Discussion

The nature of the specification will influence the type of examples that are relevant. For example, for markup specifications, provide at least one example of each markup construct; for transformation specifications, illustrate each transformation capability with an example showing input and output.

Provide Usage Scenarios.

AUsage Scenariois an instance of a use case, representing a single path through the use case. Thus, there may be a scenario for the main flow through the use case and other scenarios for each possible variation of flow through the use case (e.g., representing each option).

Conformance requirements: the specification MUST provide Usage Scenarios.

Usage scenarios provide more specific information regarding the authors' view of the specification's applicability. The additional information given in a usage scenario assists the understanding or use of the specification.

Identify what needs to conform and how.

Identifying what needs to conform to the specification is essential for the specification writers as well as for the implementors. This guideline relies on the concepts of classes of products and specification categories.

Classes of product is a dimension of variability (DoV).

The conformance clause identifies the "class of products" (i.e., object of the claim) that is the target of the specification. In addition to identifying what conforms (i.e., class of products), it describes how conformance can be met. This would be a description of the conformance requirements, conditions and/or criteria for each class of product.

Checkpoints

Identify allclasses of product.
Conformance requirements

A specification:

  • MUST list the classes of product for which it defines conformance requirements, it addresse
  • MUST use the above enumerated classes names when they match those of the specification,
  • MUST define and describe any identified class of product that does not match the enumerated set.
Rationale

Doing so helps define the scope of the specification. It also helps the reader know what is being targeted by the specification (i.e., what needs to conform to the specification).

Discussion

The list of classes of products enumerate the most common classes of products. If your class of product matches one or more terms in the list, then use the terms in the list. If no term is an appropriate match, then define your own.

Example

Example. Within the SMIL 2.0 Language Profile ([SMIL20], chapter 13), there are 2 classes of products: documents and basic user agents. Within Mathematical Markup Language (MathML) 2.0 [MATHML20] there are output-compliant, authoring tools and input-compliant, rendering/reading tools.

For each class of product, define the conformance requirements.
Definition

The conformance requirements indicate the conditions to be satisfied in order to claim conformance. It is likely that these conformance requirements will reference normative text within the specification or in other related specifications.

Conformance requirements

The specification MUST define the conformance requirements for each identifiedclass of product identified in checkpoint 2.1.

Identify the applicablewhich of the specification categories. of object are specified in the document as a whole.
Conformance requirements

The specification:

Rationale

Understanding the kind of specification category that is being written contributes towards understanding the classes of productsof the specification, i.e., the applicable classes of products, as required by the preceding checkpoint. Moreover, it helps both the author(s) and readers understand the scope of the specification and its focus.doing so helps keep the scope of the specification in focus, both for the reader, and for the author(s) who must define the classes of product and their conformance criteria.

Discussion

If your specification matches one or more a categories in the list, then use those categories. If no category is an appropriate match, then define your own.

Example

Some specifications define more than one of the enumerated category types. XForms [XFORMS] is an example. It defines: Content, via an XML vocabulary; Content, via named datatypes; Syntax, in the form of a set of functions to supplement the XPath core set; behavior of a processor; behavior of a user agent; a set of events.

If there are several classes of products, define their relationships and interaction with other dimensions of variability.
Conformance requirements

The specification MUST address and discuss the relations and interactions between classes of product and the other dimensions of variability. This checkpoint is not applicable if there is only one class of product.

Discussion

Be aware that excessive variability harms interoperability.

Address the use of profiles, modules and levels to divide the technology.

To address various needs (re-usability, evolution, scaling, ...), technologies are often divided in different ways. These divisions have an important impact on the conformance to a specification. This guideline relies on the concepts of profiles, modules and levels that the QA WG has identified as being the three main division types in use.

Profiles, modules and levels are each a dimension of variability (DoV).

Checkpoints

Indicate whether or not the use of profiles is mandatory for conformance.
Conformance requirements

A specification:

  • MUST state whether conformance of each identified class of product in the specification is only defined within the context of profiles, or whether there can be conforming products outside of the bounds of profiles.
  • MUST describe additional conditions associated with a particular profile or collection of profiles for each identified class of product.

This checkpoint is not applicable if profiles are not used.

Example

For example, content can be required to conform to one of the profiles, or it may be conformant to the specification independently of conformance to one of the profiles. The questions arises also for a producer (of content): is it conforming if it generates content that is otherwise valid but does not conform to a profile?

An example of additional conditions on profiles would be to require that only one profile can be implemented at a time.

Discussion

Note: If there is a "full" profile defined — for example, incorporating all of the defined functionality of the specification, including extensibility features — then any valid content, as well as any correct producers and fully capable consumers, might seem to be automatically using that profile. However, profiles (e.g., of content) often include self-identification requirements, and these would have to be observed for conformance of valid content to even a "full" profile.

If profiles are chosen, define any minimal requirements.
Conformance requirements

The specification MUST define for each profile the minimal required features/support for each identified class of product. This checkpoint is not applicable if profiles are not used.

Rationale

Because a profile places explicit requirements on each class of product that have specific and often limited operating environments, these requirements must be defined for each class of product that is affected.

Example

To illustrate, SMIL 2.0 [SMIL20] has a SMIL 2.0 Language Profile for user agents but also provides a SMIL 2.0 Basic Profile for wireless and embedded devices. The SMIL 2.0 Language Profile requires that a user agent implement the BasicAnimation module but not the SplineAnimation Module. The SMIL 2.0 Basic Profile on the other hand does not require implementation of any of the Animation modules.

If profiles are chosen, define their relationships and interaction with other dimensions of variability.

Conformance requirements: the specification MUST address and document identified relations and interactions between profiles and the other dimensions of variability. This checkpoint is not applicable if profiles are not used.

Dependency or interrelationship between profiles and modules is common -- XHTML [XHTML-MOD], SMIL [SMIL20], SVG 1.1 [SVG11]. Less often, deprecated features, levels, discretionary choices, or extensions could depend on profiles.

If profiles are chosen, address rules for profiles.
Conformance requirements

If the specification allows the creation of derived profiles, the specification MUST provide requirements for derived profiles and these requirements MUST be testable. This checkpoint is not applicable if profiles are not used. This checkpoint is not applicable if the specification does not allow the creation of derived profiles.

Rationale

Well-designed rules for profiles will facilitate the creation of derived profiles that are well-specified and testability will enable an independent tester to verify conformance of a derived profile to the rules.

Discussion

Note that previous experience in W3C (e.g., the creation of WebCGM) and by other organizations who have created derived profiles or defined rules for profiles, shows that requirements for derived profiles should impose at least these two rules on derived profiles: (1) derived profiles should be specified in a way that meets all the pertinent checkpoints of this document (QA Framework: Specification Guidelines); and, (2) derived profiles should not contradict predefined profiles, if there are any in the base specification.

Define the relationships between profiles, modules, levels and other dimensions of variability
Conformance Requirements

The specification MUST address and discuss the relationships between profiles, modules, levels and the other dimensions of variability used.

Address the use of modules to divide the technology.

Checkpoints

If modules are chosen, indicate any mandatory conditions or constraints on their usage.
Conformance requirements

The specification MUST document any identified mandatory conditions or constraints on the usage of modules. Such conditions include,

  • atomicity of modules,
  • any mandatory module(s),
  • any modules that are optional,
  • dependencies among modules,
    • modules that require and build on functionally related modules,
    • modules that require modules from other functional areas,
  • constraints against combined occurrence of particular pairs of modules,
  • any other — there may be conditions and constraints other than just enumerated, and they MUST be described.

This checkpoint is not applicable if modules are not used.

Discussion

The conditions or constraints normally will be tailored according to class of product.

If modules are chosen, define their relationships and interaction with other dimensions of variability.

Conformance requirements: the specification MUST address and document any identified relationships and interaction with other dimensions of variability. This checkpoint is not applicable if modules are not used.

often there is dependency or interrelationship among modules, on the one hand, and profiles or discretionary choices on the other. Modules may have levels or deprecated features. Extensions could be defined based on modules.

Address the use of functional levels to divide the technology.

See Guideline 4 for full discussion of profiles, including comments on possible profiles-levels relationships. See Guideline 6 for a full discussion of modules, including possible modules-levels relationships.

Checkpoints

If profiles, modules or levels are used, define their relationships and interaction with other dimensions of variability.
Conformance requirements

The specification MUST address and discuss any identified relationships and interaction between profiles, modules, levels and with other dimensions of variability. This checkpoint is not applicable if none of the three DoV — profile, modules, and levels — are used.

Example

Dependency or interrelationship between profiles and modules is common -- XHTML [XHTML-MOD], SMIL [SMIL20], SVG 1.1 [SVG11].

Discussion

Often there is dependency or interrelationship among modules, on the one hand, and profiles or discretionary choices on the other. Modules may have levels or deprecated features. Extensions could be defined based on modules.

Levels can be dependent on, or apply to, modules. Less often, deprecated features, levels, discretionary choices, or extensions could depend on profiles.

Be aware that excessive variability harms interoperability.

Identify the relation between deprecated features and conformance.

A deprecated feature is an existing feature that has been outdated by newer constructs or is no longer viable. Deprecated features should be avoided (e.g. for a format language, not be used by the authors and authoring tools) and may be removed in some future version, at which time the feature becomes obsolete.

After the initial publication of a specification, specification developers may consider the deprecation of a feature (e.g., function argument, element or attribute) defined in the specification. Deprecated features may become obsolete and no longer defined in future versions of the specification. Deprecation of a feature may warn implementers that the feature was a bad idea and it may be withdrawn in the future. Specification developers need to consider the effect of deprecation on all the classes of products that implement the specification (e.g., authoring tools, user agents) as well as the conformance consequences on each class of product. For the purpose of backward compatibility, it may be necessary to specify different requirements for the support of deprecated features for each class of product. For example, authoring tools (producers) would not use the feature, but user agents (consumers) would continue to support it.

Deprecation is a dimension of variability (DoV).

Checkpoints

Identify each deprecated feature.
Conformance requirements

The specification MUST document in a normative section each deprecated feature. This checkpoint is not applicable if there are no deprecated features.

Rationale

It helps the reader find the deprecated features by presenting them as a collection in one place rather than distributed throughout the document. This collection may be derived from distributed deprecated features and may take the form of a list of links to where the features appear in the document.

For each class of product, specify the degree of support required for each deprecated feature and the conformance consequences of the deprecation.
Conformance requirements

The specification MUST specify the degree of support required for each deprecated feature for each class of product and the conformance consequences of the deprecation. This checkpoint is not applicable if there is areno deprecated features.

Example

For example, a deprecated-features section of MathML 2.0 ([MATHML20], section 7.2.1.2) describes, about deprecated MathML 1.x features, that MathML-output-compliant authoring tools may not generate MathML markup containing deprecated features; whereas MathML-input-compliant rendering/reading tools must support deprecated features.

If deprecated features exist, deprecation is used, define their relationships and interaction with other dimensions of variability.
Conformance requirements

The specification MUST indicate, for each deprecated feature, either (1) that it is independent of all other dimensions of variability or (2) discuss the relationship between the feature and each of the other DoV.address and discuss the relations and interactions between deprecated features and all the otherDoV.

Rationale

Depending of the breadth of the feature being deprecated, it may impact other DoV. The effect may be contained (e.g., within a single class of product or module), be wide ranging (e.g., across all classes of products), be a dimension of variability (e.g., deprecate an entire module), or introduce changes to a dimension of variability (e.g., spawn new discretionary choices).

Be aware that excessive variability harms interoperability.

Include an explanation for the deprecation.
Conformance requirements

The specification MUST document each deprecated feature with a rationale for the deprecation. This checkpoint is not applicable if thereis are no deprecated features.

Rationale

Providing the rationale for deprecating a feature helps implementers and users to understand the motivation for the deprecation, the impact and consequences on current and future implementations, and perhaps insight into its eventual disappearance from the specification.

Include examples to illustrate how to avoid using deprecated features.
Conformance requirements

The specification MUST provide an example for each deprecated feature showing how to avoid using it. This checkpoint is not applicable if there are isno deprecated features. This checkpoint is only applicable to specifications that identify "producer of content" as one of its classes of product.

Rationale

Examples are helpful in providing alternatives or better ways to get the same results. Showing what can be done in place of the deprecated feature will help get implementers to discontinue use of the deprecated feature.

Discussion

Note that this checkpoint only makes sense for specifications that define the behavior of a producer.

Identify each obsolete feature.
Conformance requirements

The specification MUST document each obsolete feature. This checkpoint is not applicable if there are no obsolete features.

Rationale

Obsolete features are listed for historical purposes. There is no guarantee of support for obsolete featrues by implementations of the specification.

Example

Obsolete features can be listed in the Change section of a specification, e.g., HTML 4.01.

Define discretionary items.

Discretionary items are defined as deliberate and explicit grants of discretion by the specification to the implementations, that describe or allow optionality of behavior, functionality, parameter values, error handling, etc.

Discretionary items are often made available in specifications, to give implementers of the technology the opportunity to decide from alternatives when building applications and tools. Discretionary items may be considered necessary because of environmental conditions (e.g., hardware limitations or software configuration, or external systems), locality (e.g., time zone or language), optional choices providing flexibility of implementation, dependence on other specifications, etc.

Discretionary items come in three basic variants:

discretionary choices
a value or behavior may be chosen from a well-defined enumerated set of two or more possibilities,
optional features
a well-defined feature may be supported or not (if supported, then the requirements are clear and unambiguous).
implementation dependent values (or features)
a value an element or attribute may have or the behavior of a product that implements a feature that is open ended and undefined.

Discretionary items are a dimension of variability (DoV).

Checkpoints

State the circumstances for when discretionary items are allowed.
Conformance requirements

The specification MUST indicate the rationale for the discretionary items and MUST identify by labeling all discretionary items. This checkpoint is not applicable for specifications that do not have discretionary items.

Rationale

Test creators will want to discover all choices and variability allowed to implementers. Having a name for each discretionary item, and a link target for each one, allows systematic reference to individual items from test cases as well as elsewhere in the specifications.Doing this helps readers, implementers and testers to find these discretionary items and understand the need for them.

Discussion

Note that references to use cases and project requirements are usually a good way to justify discretionary items.

For implementation dependent values or features, address the allowable differences between implementations.
Conformance requirements

The specification MUST describe any permitted variations or constraints for how the implementation dependent value or feature is realized by implementations.

Example

Examples of permitted variations or constraints to be addressed include:

  • implementation dependent ranges, data, minimum or maximum values,
  • environmental resources (e.g., memory or disk limitations),
  • environmental values (i.e., language and local settings).
Indicate any constraints on the number of choices/options that can be implemented for discretionary items.
Conformance requirements

The specification MUST indicate, for each identified discretionary item, whether zero, exactly one, or several of the choices/options are allowed to be implemented. If the allowable number is dependent on other dimensions of variability, the dependencies MUST be stated. This checkpoint is not applicable for specifications that do not use discretionary items.

Rationale

A test framework for the specification could provide a tailoring mechanism so that each implementation is tested with a set of tests tailored to the choices made by the implementer. The tailoring mechanism for choose-exactly-one items would not be the same as the tailoring mechanism for choose-one-or-more items. The tailoring mechanism would need to accommodate modules, profiles, and levels, when they are used, which may affect the range of choices on some discretionary items (and may drop the range to zero in some situations).

Example

Examples of constraints include mandating that an implementation implement only one choice, implement every choice (allowing the user to choose), be allowed to implement none of the choices, or be required to implement one specified value and as many additional values as they wish from an open-ended list. An example of a constraint being dependent on another dimension of variability is a rule that there must be exactly one instance of the item for each profile that is supported.

It is possible for a discretionary item to affect the range of available options on another item, or to render it moot. For example, if a discretionary choice addresses serialization of characters as entities, one option of that choice may render moot a grant of discretion regarding entity names for character entities.

Promote consistent handling of discretionary choices.
Conformance requirements

The specification MUST provide rules for consistent terminology about discretionary items and MUST provide a rationale for discretionary items that do not follow the rules.

The specification MUST document the identified policies for handling discretionary choices

Rationale

This helps identifying where the specification could actually factorize these policies, so that implementations could consistently handle discretionary choices - users have an expectation of what to expect and should be able to count on getting the same results under the same conditions with a given implementation.

This helps to ensure that implementations can consistently handle discretionary choice. Users have an expectation of what to expect and should be able to count on getting the same results under the same conditions with a given implementation. It also helps to propagate the rules about discretionary items onto the implementations, especially when an implementation could offer choices to the user.

Names of discretionary items, and names of choices if used, should use verbiage consistently in order to satisfy Checkpoint 7.3. Identical words should be used for identical behaviors.

Discussion

Note that the Working Group believes that given identical conditions, the effect of a discretionary choice should be consistent within a single implementation, and thus, that specifications should enforce it.

Specifications should encourage the implementation and any associated documentation to use the specification's terminology when discretionary choices are visible to the user.

If discretionary items are used, define their relationships and interaction with other dimensions of variability.
Conformance requirements

The specification MUST address and discuss the relations and interactions between profiles discretionary items and all the other dimensions of variability.

Where multiple discretionary items can be related to each other, the specification MUST justify making them independent of each other. In other words, when individual details are subject to discretion but can be summarized as a policy-level decision, the specification MUST present one named discretionary item for the policy OR present a justification for dividing the item into smaller items. This checkpoint is not applicable for specifications that do not have discretionary items.

Rationale

The use cases begin the process of setting expectations of how implementations will be used. When further refined, the use cases set expectations for variation or flexibility of implementations. Any variability that is too detailed for the use cases is probably of low value to the user, but it hurts interoperability.

Discussion

Be aware that excessive variability harms interoperability.

Allow extensions or not

An extension to a specification is a mechanism to incorporate functionality beyond what is defined in the specification. Extensions can be implemented individually (e.g., one function at a time) or by defining new profiles, modules, or levels that incorporate additional functionality beyond what is defined in the specification.

Allowing extensions affects how conformance is defined as well as what conformance claims can be made. Exercise caution in determining the extent to which extensions are allowed or not allowed. Since extensions can seriously compromise interoperability, specification writers should carefully consider the degree to which extensions are be allowed.

Disallowing extensions for any part of the specification is calledenables strict conformance: strict conformance is defined as conformance of an implementation that employs only the requirements and/or functionality defined in the specification and no more (i.e., no extensions to the specification are implemented). Dimensions of variability (e.g., modules, profiles, levels) are not extensions if the specification defines them or allows them to be defined.

Extensions may be private (often vendor specific) or public (a full description of the extension is public). Private extensions are usually truly private, i.e., valid for a specific implementation or are only known by prior agreement between implementations. Public extensions are extensions in which the syntax, semantics, identifiers, etc. are defined and published allowing anyone to implement the extended functionality.

Specifications allow extensions for various reasons. Extensions allow implementers to include features that they consider to be required by their customers. Also, extensions often define new features that may migrate into future versions of the specifications. However, the use of extensions can have a severe negative impact on interoperability. Some methods for enabling extension have less impact on interoperability than other methods.

Extensions are a dimension of variability (DoV).

Checkpoints

Indicate if the specification is extensible.

Conformance requirements: the specification MUST state the conditions under which extensions are allowed and disallowed.

Implementers and readers of a specification need to know if the specification is extensible. It is equally important to convey the circumstances (i.e., conditions) under which the extensions are allowed or disallowed.

Indicate if the specification is extensible, and if extensions are allowed, define their scope and constraints.
Conformance requirements

The specification MUST state the conditions under which extensions are allowed and disallowed, and if allowed, MUST state:

  • the scope of the extensions,
  • their effect on conformance claims,
  • any limitations or restrictions on the use of the extension,
  • the rationale for allowing extensions by referencing use cases and/or project requirements.This checkpoint is not applicable if the specification does not allow extensions.
Rationale

Readers should be able to understand the motivation for the inclusion of an extension and its intended use. Documenting the scope and rationale for extensions helps assess the impact of extensions on interoperability.

If the specification writer wants to enhance interoperability by constraining implementer extensions, wording in the specification must indicate this.

If extensions are not allowed, then it should be clear to the reader that not only are extensions not allowed, but the circumstances under which they are not allowed. This is strict conformance. Strict conformance is often imposed on applications or content (e.g., a software program or document instance). This prohibition of extensions could be applied to a specific profile or module, rather than to the entire specification.

Prevent extensions from contradicting the specification.
Conformance requirements

The specification MUST state that extensions cannot negate or change support for required functionality. This checkpoint is not applicable if extensions are not allowed.

Discussion

Requirements in the specification cannot be compromised or contradicted by adding extensions.

Define a uniform mechanism to create an extension.
Conformance requirements

The specification MUST provide a uniform way to define that extensibility is being invoked and MUST provide the syntax to be used to indicate the extension. This checkpoint is not applicable if extensions are not allowed.

Rationale

This helps to ensure predictable handling of extensions, that is, its recognition as such and the appropriate actions to take (i.e., to ignore or to implement).

Require that extensions be published.
Conformance requirements

The specification MUST require that the syntax and semantics of the extension be publicly documented. This checkpoint is not applicable if extensions are not allowed.

Rationale

Public availability with a full description allows the extension to be implemented by anyone without prior arrangement. This enhances interoperability by allowing the same functionality to be uniformly implemented across different implementations.

Mitigate the impact of extensions on interoperability.
Conformance requirements

The specification MUST define a policy about implementation requirements for mitigation of the interoperability impacts of extensions. This checkpoint is not applicable if extensions are not allowed. This checkpoint is only applicable to specifications that identify "producer of content" as one of its classes of products.

Rationale

Extensions can have a negative affect on implementation interoperability. This checkpoint can be used to impose conformance requirements on producer implementations to minimize the consequences of the extension. For example, the specification could include a requirement that implementations have a no-extensions mode; or could include a requirement that implementations include equivalent alternative (standard) content with any extensions; or could explicitly state that there are in fact no implementation requirements for mitigation of interoperability impacts of extensions. This checkpoint can be used to ensure that there is a way to produce (generate) only conforming content. This checkpoint is applicable to specifications that identify producer of content as one of its classes of products.

If extensions are allowed, define their relationships and interaction with other dimensions of variability
Conformance requirements

The specification MUST address and discuss the relations and interactions between extensions and all the other dimensions of variability.

Example

For example, a specification could define a strict conformance policy for one class of product, while allowing extensions for another class.

Discussion

Be aware that excessive variability harms interoperability.

Clearly identify conformance requirements.

The checkpoints of this guideline aim to ensure that normative content and conformance requirements are easily identifiable in specifications. Some of the desirable characteristics of conformance requirements are that they are mutually independent from other requirements, express a minimal requirement (i.e., are atomic), and are distinguishable and labeled (e.g., tagged). It is important that specifications provide unambiguous statements, that clearly define what is required in order to claim conformance and what is optional. It is important also that the conformance statements are easily identifiable as such, and easily locatable. Clarity is fostered by uniformity of structure and style, and consistency of terminology and phraseology.

There is a lot to be said for consistency and clarity within a document - it facilitates the understanding and comprehension of the document. Authors and editors of specifications should already be familiar with the Publication Rules (Member-only) [PUBRULES] and W3C Manual of Style [STYLE-MAN], which help to achieve clarity and uniformity. This guideline builds on those general document conventions, with a particular focus on the presentation and identification of conformance information.

Checkpoints

Use conformance key words.
Conformance requirements

The specification:

  • MUST define the way conformance requirements can be identified,
  • SHOULD use the method of RFC 2119 keywords,
  • if not RFC2119 keywords, MUST state why RFC2119 keywords were not used.
Rationale

Using the RFC 2119 keywords helps to identify the testable statements in a specification as well as those statements that are desirable or allowed. This specification does not place any requirements on the formatting or rendering of these keywords.Guidance on the proper usage of these keywords is given in RFC 2119.

If for some reason, the RFC 2119 keywords cannot be used in the specification, it is important that the reader can identify the conformance requirements explicitly and unambiguously.

Distinguish normative and informative textcontent.
Definition

Normative content is the prescriptive part of the specification whereas informative content is for informational purposes and assists in the understanding or use of the specification.

Conformance requirements

The specification MUST distinguish normative text from informative content. text

Rationale

It is important that the reader be able to distinguish between normative and informative contents in order to know what is required to claim conformance. For example, Synchronized Multimedia Integration Language (SMIL 2.0) [SMIL20] indicates within every subsection whether the section is normative or informative, and even separately labels pieces of subsections that contain both kinds of content.

Discussion

Content includes not only text, but also examples, illustrations, and use cases.

Use consistent terminology.
Conformance requirements

The specification MUST use identical wording to express identical provisions, and analogous wording to express analogous provisions.

Rationale

Being consistent and following established conventions and W3C editorial practices will enhance the readability and comprehensibility of the specification.

Provide a fast way to find conformance information.
Conformance requirements

The specification MUST provide at least one navigation mechanism that allows the reader to locate by direct access, all conformance-related information that is relevant to the specification. The mechanism MUST minimally locate:

  • the conformance section,
  • unambiguous statements about those dimensions of variability that the specification employs, from among the seven defined in this specification,
  • requirements for conformance claims.
Rationale

A reader must be able to easily identify and locate all the information necessary to understand the conformance policy and related conformance information without having to read the document from cover to cover. A navigation mechanism adds to the usability of the specification.

Discussion

A table of contents entry is one way to accomplish this. In addition to the minimal required set above, other conformance related information such as the ICS, location of test suites, etc., may be helpful to users and implementers. Also see checkpoints requiring conformance information about the Dimensions of Variability that are used (e.g., Checkpoint 6.1).

Note that having a navigation mechanism available to hardcopy rendered specifications can be very useful for test suite developers and users.

Make normative reference to specifications on which the current specification depends
Definition

A specification depends on another specification when it relies on or requires functionality (or behavior) from the other specification in order to work (function) correctly. This other specification provides a necessary condition or functionality.

Conformance requirements

The specification MUST have normative references to any identified specification it depends on and MUST describe the relationship between the specifications and any identified conformance implications.

Rationale

Dependence on other specifications affects the conformance boundaries of the current specification, and thereby affects the requirements on conformant products.

Discussion

The linking parts of the Manual of Style ([STYLE-MAN], section 11.5.1) describe the recommended way of citing an external reference from the prose of a specification as well as how to construct an entry in its References section.

Example

For example, SVG 1.0 requires that the class of product called "user agent" be consistent with the XML 1.0 Recommendation [XML10] and (conditionally) support Cascading Style Sheets, level 2 [CSS2].

Document the conformance policy.

A look at various W3C Technical Reports shows that the term "conformance" is often qualified, resulting in more than one type of conformance. It is important to convey an understanding of what is meant by conformance and how it applies to each class of product as well as each dimension of variability (e.g., modules) if applicable. For example, if the specification defines behavior for more than one class of product, there may be a separate conformance policy for each class. Similarly, if the specification defines modules, there may be a different conformance policy for each module.

Overall, the intent of theWGshould be clear. In particular, a reader intending to implement a product in one of the product classes addressed by the specification should know what the WG wants for interoperability among products in the class. The developer should understand what forms, if any, of "product differentiation" are allowed among conformant products. The specification may need to explain how the rules apply and possibly provide examples.

A conformance clause is a part or collection of parts of a specification that defines the requirements, criteria, or conditions to be satisfied by an implementation or application in order to claim conformance. Typically the conformance clause is a high-level description of what is required of implementations and applications.

The conformance policy is a dimension of variability (DoV).

Checkpoints

Specify any universal requirements for minimum functionality.
Conformance requirements

The specification MUST include a normative section enumerating the minimal requirements that apply across all identifiedconforming products of a class.

Rationale

The reader must be able to recognize any minimum functionality complexity, or support that applies to conforming products of a specific class. These minimum requirements can be considered a starting point for a checklist to be used by a team developing a conforming product. It helps the reader find these requirements by presenting them as a collection, in one place.

Discussion

Usually, the individual requirements comprising the minima all occur elsewhere, as conformance requirements distributed throughout the document. In principle, the collection could therefore be derived from the distributed requirements, although such a derivation could be difficult. The collection of minimal requirements may apply to a single class of product or across classes of products.the reader must be able to recognize any minimum functionality, complexity or support that applies to all conforming products of a specific class.

Example

If levels are used (see Guideline 3), the lowest level may represent the minimum set of requirements. If profiles are used (see Guideline 3), there may be different minima for each profile. If modules are used (see Guideline 3), there may be different minima for each module. Furthermore, a module may itself be a minimum (i.e., required) for a particular class of product.

If special conformance concepts terms are used, include a definition in the specification.
Conformance requirements

Any conformance concepts used in the specification MUST be defined, either by reference or by including the definition in the text.

Rationale

It is necessary to define concepts terms that govern application of the conformance provisions. Ideally, all concepts terms are from QA documents and other existing literature and need only be cited. If special conceptsterms are constructed, such as to combine modules and levels or modules and discretionary choices, they need to be defined in the specification.

Justify any usage of a dimension of variability.
Conformance requirements

A specification MUST justify each dimension of variability the specification uses.

Rationale

Dimensions of variabilityadd complexity to a specification. Explaining the usage of each dimension of variability can help implement and test them better and can help reviewers understand their necessity.

Discussion

References to usage scenarios and requirements are usually a good way to justify the uses of dimensions of variability.

Guideline 3, "conformance policy" is related to this guideline. Guideline 3 focuses on the establishment and scope of definition of a conformance policy, while this Guideline 10 focuses (among other topics) on how and where to document it.

Include a conformance section.
Conformance requirements

The specification MUST document its conformance policy and specific conformance requirements in a dedicated section of the document.

Rationale

Having the conformance clause exist as a separate section within the specification makes it clearly identifiable, allowing a reader to find the overall conformance policy, as well as all specific conformance provisions from a single starting point.

Identify and define all conformance designations.
Conformance requirements

The specification MUST identify and define all the conformance designations used.

Example

In current W3C practice, a number of different naming conventions are used to label conformance, in cases where there is not a single, monolithic (strict) conformance definition. The naming convention used to label the conformance can provide useful information. Degrees, for example, implies incremental importance or difficulty. This Specification Guidelines document uses "degrees" for example, to refer to three successively more demanding degrees of conformance (A, AA, AAA).

Discussion

Commonly used conformance designations include categories, degrees, and levels. Use of "conformance levels" is discouraged in new specifications, because of the potential for confusion with "functional levels".

Specify how to make conformance claims.

A specification may differentiate conformance claims by designating different degrees or types of conformance in order to apply and group requirements according to modules, profiles, levels, or priorities. When a conformance claim is linked to functionality, impact and/or incremental degrees of implementation, the term 'conformance level' is often used to indicate the varying degrees of conformance. The use of this term, conformance level, is discouraged and should be replaced by the term, 'degree of conformance'. The WG includes in the specification the way they want people to claim their conformance.

An Implementation Conformance Statement (ICS) provides a mechanism whereby a supplier of an implementation of the specification provides information about the implementation in a standardized manner. It is used to indicate which capabilities and options have been implemented, as well as the limitations of the implementation. An ICS usually takes the form of a questionnaire where product implementors report on the conformance of their product to the named specification.

An ICS is useful in disclosing optional functionality and discretionary behavior and values. The results of the ICS can be used to identify the subset of test cases from a conformance test suite that are applicable to the implementation to be tested. This will allow the implementation to be tested for conformance against only the relevant requirements.

The basic and detailed information that an ICS provides can also be used to assess and deduce the interoperability potential of two or more products.

Checkpoints

Provide specific wording of the claim.
Conformance requirements

The specification MUST provide specific wording of the claim and the specific wording MUST include at least:

  • the specification name,
  • the specification version,
  • the degree of conformance satisfied,
  • information about the subject which is claiming conformance,
  • the date of the claim.
Provide a conformance disclaimer.
Conformance requirements

The specification MUST provide a conformance disclaimer.

Rationale

Although it is possible to prove with certainty when something does not conform, the reverse is not necessarily true. Especially for functional specifications, where a claim goes beyond syntax testing, a claim of conformance is not a guarantee that the claimant is 100% conforming with the specification. A disclaimer can help clarify the meaning of a conformance claim as well as its limitations. For example, this document contains a conformance disclaimer.

Impose no restrictions about who can make a claim or where claims can be published.
Conformance requirements

The specification MUST NOT include any restriction about who can make a claim nor where claims can be published.

Discussion

Claimants (or relevant assuring parties) are solely responsible for the validity of their claims, keeping claims up to date, and proper use of any the conformance icons.

Provide an Implementation Conformance Statement proforma.
Conformance requirements

The specification MUST either:

Rationale

The inclusion of an ICS may not be applicable to all specifications. A specification's ICS can be thought of as providing a minimal list of items supported by the implementation, capturing information the WG deems necessary to support conformance claims.

Discussion

An ICS that is part of a specification does not preclude other organizations such as certification organizations, from having their own ICS.

Require the ICS be completed as part of the conformance claim.
Conformance requirements

The specification MUST include the ICS as part of its conformance claim requirements. This checkpoint is not applicable, if an ICS is not applicable to the specification.

Rationale

The ICS is coupled with the requirements for making a conformance claim (checkpoint 9.1), thus providing specific information about the implementation and substantiating the conformance claim.

Provide test assertions.

A test assertion is a statement of behavior, action, or condition that can be measured or tested. It is derived from the specification's requirements and provides a normative foundation from which test cases can be built.bridges the gap between the narrative of the specification and the test cases. Each test assertion is an independent, complete, testable statement for requirements in the specification. Each test assertion results in one or more test cases. Multiple test assertions can be combined to form a test case, in this case one tests multiple facets of a particular behavior. It is recommended that test assertions be available by the time a specification enters Proposed Recommendation.

Checkpoints

Provide test assertions
Conformance requirements

The specification MUST provide a normative list of test assertions. stated in it

Rationale

Providing test assertions facilitates and promotes the development of test materials. Tests can point directly to the test assertion, which in turn are grounded in the specification. Specific benefits include:

  • Publicly reviewed and consensus based test assertions when the test assertions are provided in the specification,
  • Detailed reporting when testing an implementation using the test,
  • Detailed control over what parts of a specification an implementation supports, grouped by behavior, technical aspect (method/interface) or other conceptual grouping.
Discussion

The list of test assersions can be included in the specification or in a separate document that is referenced by the specification. Test assertions may be developed by the specification authors, others members of the WG, or people external to the WG.

Provide a mapping between the specification and the test assertions list.
Conformance requirements

The specification MUST provide a mechanism linking each test assertion to the part of the specification from which it was derived.

Rationale

This ensures consistency between the specification and the test assertions list and facilitates the building of a test suite based on the test assertions list. In order to enable traceability from the tests back to the specification, use a mechanism to make the test assertions identifiable (i.e., labeled or tagged) in the specification.

Conformance

This section defines conformance of Working Group specifications — i.e., technical reports — to the requirements of this QA Framework guidelines specification. The requirements of this guidelines specification are detailed in the checkpoints of the preceding "Guidelines" chapter, and apply to the technical reports produced by Working Groups.

Normative parts sections

The following sections are normative in parts ofthis document are normative:

Text that is designated as normative is directly applicable to achieving conformance to this document. Informative parts of this document consist of examples, extended explanations, terminology, and other matter that contains information that should be understood for proper implementation of this document.Any other section not explicitly marked as normative is assumed to be informative.

Extensibility

This specification is extensible. That is, adding conformance related information and structure to the document in ways beyond what is presented in this specification is allowed and encouraged. Extensions to this specification MUST not contradict nor negate the requirements of this specification.

The rationale for allowing Working Groups to define extensions to these specficiation guidelines is that these requirements are considered to be the minimal requirements for writing testable technical reports. Doing more than the minumum is not only acceptable, but beneficial. Extensions also allow Working Groups to tailor their specifications more closely to the technology and their specific needs.

For each degree of conformance claimed (I.e., A, AA or AAA), it is allowable to implement more than the checkpoint required to satisfy that degree of conformance. This may be achieved by either satisfying some, but not all of the checkpoints of the next degree of conformance or by implementing additional conformance related features beyond what is specified in this document. For example, claiming to be A-conforming but also satisfying some of the Priority 2 checkpoints and some Priority 3 checkpoints. The rationale for allowing extensions is to enable specification writers to add conformance information or to structure their documents in a way that fits their own specific needs more closely. The guidelines of this specification may not be sufficient enough to meet the needs of all WGs and thus was built to be extended.

Conformance requirements and test assertions

Within each prioritized checkpoint there is at least one conformance requirement.These are prefaced by "Conformance requirements" and highlighted in a different style. This Specification Guidelines enumerates a list of test assertions, which were derived from each MUST requirement.

Conformance definition

This section defines three degrees of conformance to this guidelines specification:

A checkpoint is satisfied by fulfilling all of the mandatoryindividual conformance requirements. Failing one individual mandatory requirement means that the checkpoint is not satisfied.Mandatory requirements are those that use the conformance keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", or "SHALL NOT".

Note that it is allowed (even encouraged) to implement checkpoints in addition to those required to satisfy a particular degree of conformance (A, AA, or AAA). The more checkpoints that are satisfied, the better. However, there is no additional conformance designation for such intermediate collections of checkpoints (i.e., for all checkpoints of a given level plus some, but not all, of the checkpoints of the next levels).

The minimum recommended conformance to this specification is A-conforming -- all Priority 1 (critical/essential) checkpoints satisfied.

Conformance claims

A specification conforms to the QA Framework: Specification Guidelines at degree X (A, AA, or AAA) if the Working Group meets at least all degree X conformance requirements.

An assertion of conformance to this specification -- i.e., a conformance claim -- MUST specify:

Example:

On 12 September 2003, W3C's QA Framework: Operational Guidelines (http://www.w3.org/TR/2003/CR-qaframe-ops-20030912), dated 12 September 2003 conforms to W3C's QA Framework: Specification Guidelines, available at http://www.w3.org/TR/2003/WD-qaframe-spec-20030912/, Conformance level AA. -Conforming.

The checklist for this specificationlist of checkpoints of this specification ordered by priority ([SPEC-ICS]) is the Implementation Conformance Statement (ICS) pro forma for this specification. Any assertion of conformance to this specification claiming a degree of conformance with this specification MUST link to a completed ICS.

Conformance disclaimer

The checkpoints of this guidelines specification present verifiable conformance requirements about the specifications (technical reports) that Working Groups produce. As with any verifiable conformance requirements, users should be aware it is also true of these specification requirements that:

Passing all of the requirements to achieve a given degree of conformance — A, AA, or AAA — does not guarantee that the subject specification is well-suited to or will achieve its intended purposes, nor does it guarantee the quality or suitability of test materials produced from the specification.

Definitions

This section contains terms used in this specification, with functional or contextual definitions appropriate for this specification. See also [QA-GLOSSARY]. Some terms in this section have been borrowed or adapted from other specifications.

class of product
the generic name for the group of products that would implement, for the same purpose, the specification, (i.e., target of the specification). The class of product is the object of the conformance claim. A specification may identify several classes of products.
conformance clause
a part or collection of parts of a specification that defines the requirements, criteria, or conditions to be satisfied by an implementation or application in order to claim conformance.
conformance level
a variety of conformance designation. Other designations include conformance category, conformance degree, conformance xxx. "Conformance level" is discouraged in new specifications, because of confusion with "functional level".
conformance requirement
a condition for conformance of an implementation to a specification. Conformance requirements can have different levels of necessity: mandatory, recommended, or optional.
deprecated
an existing feature that has become outdated by a newer construct or is no longer viable. Deprecated features should be avoided, since they not be used andmay be removed in some future version.
derived profile
a profile that is created from a set of profile rules, where these profile rules provide instructions for building profiles (i.e., defining profiles) and the rules are defined in a specification.
dimensions of variability (DoV)
the ways in which different products that are conformant to a specification may vary among themselves. In this Specification Guidelines document, the dimensions of variability are used to help organize, classify and assess the conformance characteristics of W3C specifications.
discretionary items
deliberate and explicit grants of discretion by the specification to the implementations, that describe or allow optionality of behavior, functionality, parameter values, error handling, etc.
functional level
a technology subset that is one of a hierarchy of nested subsets, ranging from minimal or core functionality to full or complete functionally.
implementation conformance statement (ICS)
a mechanism for providing standardized information about an implementation of a specification, usually in the form of a questionnaire in which product implementers report the product's conformance to the specification. An ICS is used to indicate which requirements, capabilities, and options have and have not been implemented.
informative text
text in a specification whose purpose is informational or assistive in the understanding or use of the specification, and which contains no conformance requirements or test assertions.
level
a commonly used shorthand for functional level.
module
a collection of semantically-related elements, attributes, and attribute values that represents a unit of functionality. Modules are non-hierarchical, discrete divisions that are defined in coherent sets.
normative text
text in a specification which is prescriptive or contains conformance requirements.
obsolete
feature that is no longer defined in the specification. A feature is often deprecated before becoming obsolete.
profile
a subset of a technology that is tailored to meet specific functional requirements of a particular application community. A profile may address a single technology; or, a profile can also group a set of technologies (i.e., from different specifications) and define how they operate together. Profiles may be based on hardware considerations associated with target product classes, or they may be driven by other functional requirements of their target communities.
profiling
a method for defining subsets of a technology by identifying the functionality, parameters, options, and/or implementation requirements necessary to satisfy the requirements of a particular community of users.
specification category
the generic name for the type of specification and the technology it describes.
strict conformance
conformance of an implementation that employs only the requirements and/or functionality defined in the specification and no more (i.e., no extensions to the specification are implemented).
test assertion
a statement of behavior, action, or condition that can be measured or tested (See also QA Glossary [QA-GLOSSARY].)
unconditional conformance
use case
a specification mechanism or technique that captures the ways a specification would be used, including the set of interactions between the user and the specification as well as the services, tasks, and functions the specification is required to perform.
usage scenario
an instance of a use case, that represents a single path through the use case. Thus, there may be a scenario for the main flow through the use case and another scenarios for each possible variation of flow through the use case (e.g., representing each option).

Acknowledgments

The following QA Working Group and Interest Group participants have contributed significantly to the content of this document:

References

Normative

[RFC2119]
Key words for use in RFCs to Indicate Requirement Levels, March 1997, available at http://www.ietf.org/rfc/rfc2119.txt.

Informative

[CSS2]
Cascading Style Sheets, Level 2, W3C Recommendation, 12 May 1998, available at http://www.w3.org/TR/REC-CSS2/.
[HTML4]
HTML 4.01 Specification, W3C Recommendation, 24 December 1999, available at http://www.w3.org/TR/html401/.
[MATHML20]
Mathematical Markup Language (MathML) Version 2.0, W3C Recommendation, 21 February 2001, available at http://www.w3.org/TR/MathML2/.
[PUBRULES]
W3C Publication Rules, available at http://www.w3.org/Guide/pubrules.html (Member-only).
[QA-GLOSSARY]
Comprehensive glossary of QA terminology. (Under construction, at http://www.w3.org/QA/glossary.)
[QAF-INTRO]
QA Framework: Introduction, L. Henderson, L. Rosenthal, Eds., W3C Working Draft, September 2003, companion version to this document, available at http://www.w3.org/TR/2003/NOTE-qaframe-intro-20030912/ .
[QAF-OPS]
QA Framework: Operational Guidelines, L. Henderson, D. HazaŽl-Massieux, L. Rosenthal, K. Gavrylyuk, D. Dimitriadis, Eds., W3C Candidate Recommendation, September 2003, companion version to this document, available at http://www.w3.org/TR/2003/CR-qaframe-ops-20030912/.
[QAF-TEST]
QA Framework: Test Guidelines, P. Curran, P. Fawcett, K. Gavrylyuk, D. Dimitriadis, L. Henderson, M. Skall, Eds, W3C Working Draft, 16 May 2003, companion version to this document, available at http://www.w3.org/TR/2003/WD-qaframe-test-20030516/ .
[QAIG]
Quality Assurance Interest Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/IG/.
[QAWG]
Quality Assurance Working Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/WG/.
[SMIL20]
Synchronized Multimedia Integration Language (SMIL 2.0), W3C Recommendation, 07 August 2001, available at http://www.w3.org/TR/smil20/.
[SPEC-CHECKLIST]
An appendix to this specification guidelines document presents all checkpoints in tabular form sorted in their original order. Available at http://www.w3.org/TR/2003/WD-qaframe-spec-20030912/qaframe-spec-checklist.
[SPEC-ICS]
An appendix to this specification guidelines document presents all checkpoints in tabular form sorted by priorities. Available at http://www.w3.org/TR/2003/WD-qaframe-spec-20030912/qaframe-spec-ics.
[SPEC-EXTECH]
QA Framework: Specification Examples & Techniques, available at http://www.w3.org/QA/WG/2003/09/qaframe-spec-extech.
[STYLE-MAN]
W3C Manual of Style, summarizing the style and publication rules for W3C technical reports, available at http://www.w3.org/2001/06/manual/.
[SVG11]
Scalable Vector Graphics (SVG) 1.1 Specification, D. Jackson, J. Ferraiolo, J. Fujisawa, Eds., W3C Recommendation 14 January 2003, available at http://www.w3.org/TR/2003/REC-SVG11-20030114/.
[SVG-MOBILE]
Mobile SVG Profiles: SVG Tiny and SVG Basic, T. Capin, Editor, W3C Recommendation, 14 January 2003, available at http://www.w3.org/TR/2003/REC-SVGMobile-20030114/.
[WCAG10]
Web Content Accessibility Guidelines 1.0, W. Chisholm, I. Jacobs, G. Vanderheiden, Eds., W3C Recommendation, 5 May 1999, available at http://www.w3.org/TR/WCAG10/.
[XFORMS]
XForms 1.0, M. Dubinko, L. L. Klotz Jr., R. Merrick, T. V. Raman, Eds., W3C Candidate Recommendation, 12 November 2002, available at http://www.w3.org/TR/2002/CR-xforms-20021112/.
[XHTML-MOD]
Modularization of XHTML, M. Altheim, F. Boumphrey, S. Dooley, S. McCarron, S. Schnitzenbaumer, T. Wugofski,Eds., W3C Recommendation, 10 April 2001, available at http://www.w3.org/TR/xhtml-modularization/.
[XHTML-BASIC]
XHTML Basic, M. Baker, M. Ishikawa, S. Matsui, P. Stark, T. Wugofski, T. Yamakami, Eds., W3C Recommendation, 19 December 2000, available at http://www.w3.org/TR/xhtml-basic/.
[XHTML11]
XHTML 1.1 - Module-based XHTML, M. Altheim, S. McCarron, Eds., W3C Recommendation, 31 May 2001, available at http://www.w3.org/TR/xhtml11/.
[XML10]
Extensible Markup Language (XML) 1.0 (Second Edition), T. Bray, J. Paoli, C. M. Sperberg-McQueen, E. Maler, Eds., W3C Recommendation, 6 October 2000, available at http://www.w3.org/TR/REC-xml.

Change history

2003-09-12, intermediate WD

This version integrated the resolutions of the issues raised during the Last Call review; the document has been reordered and some of the guidelines have been consolidated; furthermore, many checkpoints have been reworded for sake of clarification or adjustment.

  • a new "Concepts" section (section 2) gathers the most important concepts used throughout the document
  • Former GL 4,5 and 6 have been consolidated into GL 3; during the operation, former CP 4.3, 5.2 and 6.1 have been merged, and most of the verbiage of those GL have been merged in the section 2.3
  • Former GL 3 and 10 have been merged into GL 8
  • Former GL 11 and 12 have been merged into GL 9
  • GL 13 have been moved up as GL 7
  • Test assertions derived from the specification are now available
  • CP 9.1 and 9.2 have been merged
  • The conformance section has been completed
  • Rationales have been added to some CP

The detailed changes are more visible in the editor's version of SpecGL.

2003-02-10, Last Call WD

This version integrates the resolutions taken by the WG on the issues raised since the last Working Draft; a good set of them have been taken during the QA WG F2F in Seattle. The structure hasn't really changed since the last draft, except that some checkpoints were removed, some were added.

  • the scope part of the introduction has been reworded for clarification
  • added an explicit section on normative sections
  • created a list of checkpoints ordered by priority, that is used as the ICS for the specification
  • the conformance requirements are now explicitly labeled as such
  • the following CP have been removed (using numbers of the previous version): CP 1.4
  • the following CP have been added (using numbers of this version): CP 2.4, 3.2, 3.3, 3.5, 7.3, 8.5, 9.7
  • the following sections have been significantly modified: GL 3 verbiage, GL 13 verbiage, CP 1.1, 1.2, 8.3, 8.4, 9.4
2002-11-08, third published WD

While keeping most of the principles behind the second published WD, this version has re-ordered the guidelines in a more logical way, and takes a more formal approach in the design of CP, where all CP has a set of test assertions. Besides, repetitions have been diminished through factorization, and non-testable or arguable CP have been removed or marked as to be moved as examples and techniques. Finally, it integrates all the issues resolutions agreed during the QA WG Tokyo face-to-face meeting and the weeks following.

  • the introduction integrates explicit scope and goals as required by the GL and has been cleaned of its repetitions
  • usage of RFC 2119 has been clarified with the usage of uppercase keywords
  • the dimensions of variability have now their own section in the introduction, regrouping the various bits previously scattered around
  • ex-GL 14 on granular grammars has been removed, some of its content being integrated in the GL on test assertions
  • ex-GL 6 on conformance policy has been moved ahead on GL 3, ex-GL 7 on levels has been grouped with its related GL on profiles and modules
  • every CP has now a set of test assertions using RFC 2119 keywords; most of them have an explicit rationale and some of them an example; the style sheet has been modified to highlight the test assertions
  • the following CP (using numbering of the previous version) have been removed: 1.4 (Ensure that every test assertion is covered by an example), 2.3 (For each class of product, indicate minimal support requirements.), 3.1 (Choose whether or not to have profiles.), 3.2 (Include a table of contents entry.), 4.1 (Choose whether or not to have modules.), 4.2 (Include a table of contents entry.), 6.5 (Include a table of contents entry.), 7.1 (Address whether the specification uses or will use functional levels.), 7.2 (Include a table of contents entry.), 9.7 (Include a table of contents entry.), 10.3 (Include a conformance clause entry in the table of contents), 10.5 (Identify all dimensions of variability that are not used.) 11.5 (Include a table of contents entry.), 13.3 (Follow Web Accessibility Initiative and Internationalization Guidelines.)
  • CP 13.4 (Provide a fast way to find conformance information) is new
  • overall, an effort has been put into using a consistent terminology across the specification and avoiding unclear or untestable statement
  • the verbiage on each DoV has been vastly reduced or factorized in the DoV discussion in the introduction
2002-08-26, second published WD

Significantly reorganized and revised the first published WD. This version produced as a series of editor's drafts. The changes below are reverse chronological (most recent first), so more recent ones may build on older ones.

  • Handle negative disclaimers for DoV's via new CK10.5.
  • First cut at distinguishing specifications from technologies that they describe (esp in profiles, modules, levels.)
  • Add concept of SpecGL test assertions to "Conformance" chapter.
  • Define "use case" and "user scenario", revise GL1 & CK1.2 to use them accordingly, clarify CK1.3 (Provide examples), add rigorous CK1.4 (examples--TAs).
  • Clean up "atomicity of modules" discussion, add atomicity to CK4.3
  • Harmonize priorities of CK8.1 & 8.2, 9.1 & 9.2.
  • All GL14 and GL15 checkpoints to priority 2, except CK15.1; tighten descriptions.
  • All TOC checkpoints to priority 2, except Conformance Clause (p1).
  • Accommodate "Rules for Profiles" in GL2 categories and classes; new CK3.6 and changes to 2.3.
  • Add caveat to "Understanding and using.." (sec. 1.2), about specifications that pre-date SpecGL.
  • Rewrite CK11.1 to replace 'levels' in "conformance levels" w/ degrees, categories, etc (confusion with "functional levels").
  • Change "levels" to "degrees" in SpecGL conformance clause.
  • Change "discretionary behaviors" to "discretionary items" in GL8, define three variants in GL8 verbiage.
  • Extensive new prose in GL3 (profiles), GL4 (modules), GL7 (levels), describing potential relationships.
  • Fixed definition of strict conformance, clarified definition and usage of "conformance clause".
  • Added "Definitions" section (in-spec glossary).
  • Enhanced wording of all TOC checkpoints.
  • For each of CK3.x (profiles), 4.x (modules), 7.x (levels), x>1, added a "not applicable" disclaimer if the feature is not used. (Except for the TOC checkpoints.)
  • Change wording of CK7.x, eliminating "chosen", and add some new explanation to 7.1
  • Delete CK5.4 (conformance policy in conformance clause) and move its verbiage to GL10 (conformance clause) and its checkpoints.
  • Fix "Conformance" chapter, add "Checkpoint Priorities" section.
  • Change chapter title of "Navigating through this document" to "Understanding and using this document"
  • New text for Checkpoint 3.4
  • Added warnings about "excessive variability" to all dimension-of-variability guidelines.
  • Added to several DoV checkpoints a requirement to explain/justify any variability dimension by use cases and requirements.
  • Fix separation between last two guidelines, "Granular Grammars" and "Test Assertions".
  • Introduce and motivate "dimensions of variability" (DoV) in sec 1.5
  • Reordered guidelines (GL), split GL.5 into three new ones, GL.3, GL.4, and GL.7, customized checkpoints to their topics (profiles, modules, levels).
  • Added new p3 checkpoint ("categorize specification") to (new) GL.2
  • Rewrote (old) GL.3, "conformance flavors", in terms of "specify conformance policy, (new) GL.5.
  • Split CK1.3 into two, wrote new rationale and removed CK1.4.
  • Replaced CK9.1 and 9.2 with proposed wording from Issue 106.
  • Modified CK13.4, adding 'direct access' and a reference to DoV checkpoints.
  • Deleted CK10.1 and changed priority of CK10.2.
  • Added new section on extensibility of SpecGL in the Conformance section.
2002-05-15 version

First published WD.