The principal goal of this document is to help W3C Working Groups to write clearer, more implementable, and better testable technical reports. It both provides a common framework for specifying conformance requirements and definitions, and also addresses the representation of specifications (technical reports) as schemata, both of which facilitate the generation of test materials. The material is presented as a set of organizing guidelines and verifiable checkpoints. This document is one in a family of Framework documents of the Quality Assurance (QA) Activity, which includes the other existing or in-progress specifications: Introduction; Operational Guidelines; and, Test Guidelines.
This QAWG version is substantially identical to the 2002-08-26 published version.
This section describes the status of this document at the time of its publication. Other documents may supersede this document. The latest status of this document series is maintained at the W3C.
This document is a W3C Working Draft (WD), made available by the W3C Quality Assurance (QA) Activity for discussion by W3C members and other interested parties. For more information about the QA Activity, please see the QA Activity statement.
This version is the second published WD, and supersedes all previous drafts including the first published WD. It is expected that updated WD versions of this document will be produced regularly, along with other members of the Framework documents family. Future progression of this document beyond Working Draft is possible, but has not yet been determined by W3C. The QA Working Group has a resolved issue, that this document should be progressed to a final stable status (either note or recommendation), and conformance of Working Group specifications to this document should eventually become mandatory, by a process yet to be decided.
This is a major reordering and revision of the first published WD version, 2002-05-15. The reordering and revision was done according to the email proposal -- which was discussed and endorsed at the Montreal face-to-face -- in response to issues raised on the discussion list about the first published WD. Note that significant comments about old Guideline 12 (now Guidelines 14 and 15) -- Theme 2, specification anatomy -- have not been addressed in this version. It is anticipated that this is a next major undertaking of the QA Working Group [QAWG].
Rather, this version has focused on a complete reorganization and restructuring of the guidelines of Theme 1, Conformance Content. Old Guidelines 1 - 11 are restructured into 13 new guidelines, organized around a central concept of specification "dimensions of variability" (DoV). The QAWG has done this with the specific goal of presenting it to the W3C, to solicit feedback on:
About #3, in this version the QAWG has included checkpoints that require explanation and justification of the use of DoVs within a specification, and has warned of the dangers of excessive variability, but has generally avoided specific value judgements. The QAWG has debated and has a currently active issue about whether it is possible to provide some sort of quantitative constraints on variability (e.g., maximum number of DoV, or a computed "variability index", etc.)
In addition to feedback about the overall direction and organization, there are specific issues about which the QAWG desires feedback. The most persistent issues involve the relationships amongst the ways of dividing or grouping the technology -- profiles, modules, levels -- and whether there are relationships that are better (should be encouraged) and ones that are worse (should be discouraged). Some specific issues are flagged in the text, and may also be found in the QAWG Issues List, specifically atomicity of modules, levels minimality, modules-levels, levels-profiles, legacy specifications. (Issues and editor notes are flagged in the text with "@@".)
Finally, this WD comes with these disclaimers. Work is still needed on the structuring of text within checkpoints, and in particular clarifying exactly what are the requirements (test assertions) within the checkpoints; and, the new "Definitions" chapter is still under construction, and is missing terms which ought to be and eventually will be in it.
A future version of this document will be accompanied by a QA Framework: Specification Examples & Techniques document, which will illustrate the guidelines and checkpoints with case studies, and explain how to satisfy the checkpoints.
There are no known patent or IPR constraints associated with this Working Draft of QA Framework: Specification Guidelines. The QA Working Group Patent Disclosure page contains details, in conformance with W3C policy requirements.
Please send comments to email@example.com, the publicly archived list of the QA Interest Group [QAIG]. Please note that any mail sent to this list will be publicly archived and available. Do not send information you would not want to see distributed, such as private data.
Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or made obsolete by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress".
A list of current W3C Recommendations and other technical documents can be found at http://www.w3.org/TR.
1.1 Motivation for this guidelines document
1.2 Understanding and using this document
1.3 Checkpoint priorities
1.5 Major Themes
1.6 Variability, complexity, and roadmap of guidelines
G 1. Define Use Cases.
G 2. Identify what needs to conform and how.
G 3. Address the use of profiles to divide the technology.
G 4. Address the use of modules to divide the technology.
G 5. Specify conformance policy.
G 6. Clarify the relation between deprecated features and conformance.
G 7. Address the use of functional levels to divide the technology.
G 8. Define discretionary items.
G 9. Allow extensions or NOT!
G 10. Provide a conformance clause.
G 11. Specify how to make conformance claims.
G 12. Publish an Implementation Conformance Statement proforma.
G 13. Support general document conformance conventions.
G 14. Use granular grammars to author the specification.
G 15. Include test assertions.
3.1 Test assertions
3.2 Conformance definition
3.3 Conformance disclaimer
6. References (non-normative)
7. Change history
An appendix to this document [SPEC-CHECKLIST] presents all checkpoints in a tabular form, for convenient reference.
The quality of the specification has direct impact on the quality of implementations. Quality encompasses utility which refers to the usefulness of the specification to the intended users and objectivity which focuses on the whether the specification is presented in an accurate, clear, complete, and unbiased manner.
This document defines a common framework for specifying conformance requirements and definitions, and for representing the structure of the document as schemata, both of which facilitate the generation of test materials. The primary goal is to assist W3C Working Groups (WGs) by providing guidelines and verifiable checkpoints for writing clearer, more implementable, and better testable specifications (technical reports). Good specifications lead to better implementations and foster the development of conformance test suites and tools. Conforming implementations lead to interoperability.
These guidelines were developed so that WGs can apply them in a common-sense and workable manner. This document identifies the conformance requirements and statements to be included or addressed in specifications as well as addressing the anatomy of the specification. Conformance requirements are the expressions that convey the criteria to be fulfilled in an implementation of a specification. The conformance requirements are stated in a conformance clause or statements within the specification. The anatomy of the specification pertains to the method for writing the specification using schemata. A specification represented by an XML grammar with sufficient granularity of its information facilitates the generation of test materials by providing the ability to point to, extract, query, manipulate and/or automatically generate test materials. Given the symbiotic connection between specification and test materials, this document also addresses the intermixed quality practices of specification authoring and test material production and maintenance within the W3C process.
This document is part of a family of QA Framework documents designed to improve the quality of W3C specifications as well as their implementations by solidifying and extending current quality practices within the W3C. The QA Framework documents are:
The process for developing testable technical reports and specifications is affected by QA activities beyond those that are explicitly provided in this document. Specifically, the QA Framework documents are interrelated and complement each other. For example, the anatomy of a specification is dependent on the type of test materials that will be built - hence there is interrelationship between this document and the Test Guidelines. Links between applicable guidelines in this document and the other Framework documents will be given.
The guidelines are intended for all Working Groups as well as developers of conformance materials for W3C specifications. Not only are the Working Groups the consumer of these guidelines they are also key contributors. The guidelines capture the experiences, good practices, activities, and lessons learned of the Working Groups and present them in a comprehensive, cohesive set of documents for all to use and benefit from. The objective is to reuse what works rather than reinvent and to foster consistency across the various Working Group quality activities and deliverables.
The ultimate goal of a specification is to be implemented. In order to be implemented, a specification needs to be written such that developers and users can comprehend what is expected of them and build correct, robust, interoperable software and content. Correct utilization of specifications leads to portability and interoperability. In particular, it is easier to implement clear specifications, where the possibility of misinterpretation has been eliminated, safeguarding the quality of implementations. In addition, if care is given to the specifications, interoperability between W3C technologies is easier to track and assert.
Given that the W3C invests time and resources into producing specifications that are eventually implemented, and especially given that some of those specifications are interrelated, it makes sense to enhance and streamline the specification authoring process. This should be done for three main reasons:
The W3C needs to ensure that the deliverables of its WGs are clear, implementable and sound technical specifications with testable requirements and that can be used to easily generate Quality Assurance materials (tests and suites). A specification is testable if there exists some finite cost-effective process with which a person or software can check that the requirement has been met.
The benefits of producing clear, testable specifications include:
The benefits derived by writing granular specifications that provide detailed control over the information in the specification, include
In particular, information from the specifications is easier to extract if specification authoring involves a standardized set of specification schemas where both technology and intended behaviour is specified. Tests can be generated directly from the specifications, where needed, to generate primers for test suites. In this way, a fairly broad coverage of test cases can be achieved in a straightforward way. An added bonus is that a graphical representation of the specification can be generated in parallel to the normative HTML version. Specification granularity allows for more concise reporting when running test suites by pointing directly to the relevant part of the specification. Specification granularity allows for generating tests that can be validated against the specification itself, thus making it easier in those cases where different interpretations of the normative text are given.
Use of a formal (or formal-like) specification language may enable 'validation' of the specification as well as auto generation of tests or test components.
Synchronizing writing specifications with building test materials provides a feedback loop. Testing can help find ambiguities, inconsistencies, and holes in a specification.
Interaction between WGs and other bodies in specification authoring is related to test suite production and maintenance. The reasons to develop specifications, test assertions, and test materials together are:
Since this document is a member of a family of documents, the reader is strongly encouraged to be familiar with the other documents in the family. There is an interrelationship between the Guidelines in this document and the Guidelines in the other Framework documents. In particular, Guidelines 1, 2, 3 and 5 in the QA Framework: Operational Guidelines [QAF-OPS ] are especially relevant and interrelated to the Guidelines in this document.
This document describes what goes into the specification with respect to conformance and conformance topics, followed by rules for specification anatomy that should facilitate test development as well as produce better, more testable specifications. It does not preclude the need to apply the W3C Manual of Style [STYLE-MAN] and to conform to the Publication Rules [PUBRULES ] (member-only).
This document employs the WAI (Web Accessibility Initiative) model for representing guidelines or general principles for the development of conformance materials. See, for example, Web Content Accessibility Guidelines 1.0 [WCAG10]. Each guideline includes:
The checkpoint definitions in each guideline define the processes and operations that need to be implemented in order to accomplish the guideline. Each checkpoint definition includes:
Each checkpoint is intended to be specific enough so that someone can implement the checkpoint as well as verify that the checkpoint has been satisfied. A checkpoint will contain at least one, and may contain multiple individual requirements. See the "Conformance" chapter for further discussion of requirements and test assertions.
Caveat. The set of checkpoints as a whole has been written in anticipation that the checkpoints are being applied to new, in-development specifications. It is not expected that existing specifications that pre-date these guidelines will be able to satisfy all checkpoints as they are stated. Many legacy specifications may indirectly comply with the spirit or intent of some checkpoints, without actually satisfying all requirements in those checkpoints. It has been a design goal of this guidelines document, to phrase checkpoint requirements and to set priorities so that well-written legacy specifications that meet the most critical requirements may qualify for conformance degree "A-conforming" (see "Conformance" chapter.)
A separate appendix to this document [SPEC-CHECKLIST] presents all checkpoints in a tabular form, for convenient reference.
Satisfaction of the checkpoints in this guidelines specification are key requirements to produce a high quality, testable standard that is a suitable basis for successfully interoperable products. Some checkpoints are more critical than others. Therefore each checkpoint is assigned a priority level based on the checkpoint's impact on the quality of the specifications produced by the Working Groups.
The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" will be used as defined in RFC 2119 [RFC2119].
Unusual terms in these framework documents are defined when first used. This document contains a "Definitions" chapter. Most generally useful QA-specific terms will eventually be in the QA Glossary [QA-GLOSSARY].
The guidelines address two general themes: specifying conformance requirements in the specification; and addressing the anatomy of the specification.
Theme 1, Conformance content: general requirements and definitions concerning conformance and related issues. It is intended to contribute towards mutual understanding amongst developers of specifications and conformance test suites and tools. It does not define specific conformance requirements for any specific specification - this is the responsibility of the WG chartered to develop the specification. For the developers of specifications, it helps them to develop conformance requirements and to create testable, unambiguous specifications. A conformance clause:
Theme 2, Anatomy: invest in writing granular specification with careful modeling of the information conveyed. This allows for:
There are 15 guidelines that are presented in a sequence that runs approximately from the general to the specific. The later (higher-numbered) guidelines are likely to depend upon decisions made in accordance with earlier guidelines.
The first four guide the establishment of strategic objectives and address the overall motivation for the creation of the specification. The next two bring the QA concerns into the plan. Guidelines 7 through 9 are about additional specification tools to adjust the boundaries between conformant and non-conformant implementations. The final six guidelines pertain to particular documentation features.
It is necessary to clearly distinguish between the specification -- the W3C Technical Report -- and the technology that it describes. For example, in Guideline 3, profiling is a method to partition the technology into groups, and profiles are technology groups. Profiling and profiles of the technology may or may not lead to division of the specification into multiple parts or units, concurrent or subsequent. This document will strive to keep the distinction clear, even though in common usage concepts such as profiles typically are used to refer to both. [@@Ed note. This is not implemented completely in text yet.]
In dealing with Theme 1, considerable complexity is introduced by a basic reality of diverse W3C specifications: multiple dimensions of variability (DoV) can be accommodated in a specification. The Working Group may determine that conformant products are allowed to vary in one or more ways. Guidelines 2 through 9 cover the "dimensions of variability" that a specification may permit. This document recognizes the following dimensions of variability:
The above are not necessarily all orthogonal to one another -- some of the possible associations and relationships are noted in the individual guidelines.
The dimensions of variability comprise a tool used in this guidelines document to help organize, classify and assess the conformance characteristics of W3C specifications. While variability complicates interoperability, its net effect is not necessarily negative in all cases, when compared to the alternatives -- e.g., profiles, compared to a large monolithic specification. Different sorts of variability have different negative and positive impacts. In any case, specification writers need to carefully consider and justify any conformance variability allowed, do so by reference to the project requirements and use cases, and explicitly document the choices made.
Guideline 1. Define Use Cases.
When writing specifications it is critical to understand their primary purpose and scope. Clearly defined scope helps to keep the specification content focused and unambiguous.
If the specification describes content requirements (for example, [WCAG10], [UAAG10]), understanding of their purpose is the key to defining the minimal sufficient set. If the specification describes a protocol or Application Programmer Interface (API) -- examples include XML Protocol, DOM -- there should be a clear understanding of the primary use cases. For the purposes of this document, a use case is a specification mechanism or technique that captures the ways a specification would be used, including the set of interactions between the user and the specification as well as the services, tasks, and functions the specification is required to perform. A complete set of use cases specifies all the different ways to use the system (specification).
Readers of the specification face similar difficulties to writers. To understand what the document says, the reader needs to know the context of the author, and the scenarios the author had in mind. In case of protocols and APIs specifications developers try to assess whether the specifications cover the use cases the product needs to cover. Having no use cases described in the specification invites misuse of the specification itself.
Checkpoint 1.1. Define the scope of the specification. [Priority 1]
This is very basic but powerful requirement. The specification must clearly define what scenarios are in scope and what are out of scope in order to be interpreted, implemented, and tested correctly.
Checkpoint 1.2. Include User Scenarios. [Priority 2]
A User Scenario is an instance of a use case, representing a single path through the use case. Thus, there may be a scenario for the main flow through the use case and other scenarios for each possible variation of flow through the use case (e.g., representing each option). Unless otherwise specified, when included in a specification a user scenario is considered to be informative. The specification should have an extensive list of the user scenarios that the authors have in mind. Priorities may be assigned to user scenarios, describing how important the particular scenario is for the specification. User scenarios in their turn may help to assess what features are missing and what features are superfluous in the specification.
Checkpoint 1.3. Provide examples. [Priority 1]
Illustrate behaviors or requirements in the specification by short and precise examples. The more numerous the examples, the better it is for the specification's comprehensibility. Specifications should illustrate every major feature by example, although the satisfaction of this checkpoint does not impose any specific relationship between examples, on the one hand, and features or requirements.
Checkpoint 1.4. Ensure that every test assertion is covered by an example. [Priority 3]
Whereas the previous checkpoint can be satisfied by a collection of examples with any relationship to the requirements and behaviors of the specification, this checkpoint requires that there be a mapping from each test assertion in the specification to an example. The relationship of test assertions to examples need not be one-to-one, however examples should be short and precise, for clarity and comprehensibility.
Checkpoint 1.5. Include an interpretation section. [Priority 2]
It is hard to understand the formal descriptions of content without informative interpretations. The recent complex specifications like XML Schema [XML-SCHEMA] and XML Protocol have shown an necessity to have a "Primer" part or section to illustrate how to use the specification. Such a section does not substitute for the numerous examples needed for each requirement or behavior described in the specification, but illustrates how the specification fulfills the User Scenarios.
Guideline 2. Identify what needs to conform and how.
Categorizing the specification provides a basis for classifying the software that may be affected by the specification -- and thus, provides the answer to "what needs to conform?". To answer this question, it helps to look at the nature of the specification and categorize it. Most specifications can be classified into one of the following categories:
The categories indicate what the specification describes. One specification could potentially fall into more than one category.
[@@Ed note. A survey of W3C recommendations according to this categorization scheme is being worked on within QAWG. It is planned that it will be linked from QA Framework: Specification Examples & Techniques [SPEC-EXTECH] when that document is published in the near future.]
From this categorization of specifications, the WG can identify the "class of products" that are affected by the specification. Classes of products can be generalized as either or producers or consumers, or as content itself. Identifying which are producers and consumers is clear for a protocol-type specification, the two parties to the dialog are the targets. For a processor-type specification, the processor is the consumer of an XML vocabulary defined in the specification. For content-type specifications, there may be one or more consumers that take the content and 'play' it in some way.
The following is a non-exhaustive list of classes of products for W3C specifications.
One specification could define more than one player. For example, MathML could address the behavior of display of math notation and also a consumer that parses the MathML as a formula and uses it for mathematical processing.
The conformance clause identifies the "class of products" (i.e., object of the claim) that is the target of the specification. In addition to identifying what conforms (i.e., class of products), it describes how conformance can be met. This would be a description of the conformance requirements, conditions and/or criteria for each class of product.
Checkpoint 2.1. Identify all classes of product. [Priority 1]
This checkpoint has several individual requirements:
Example. Within the SMIL 2.0 Language Profile ([SMIL20], chapter 13), there are 2 classes of products: documents and basic user agents. Within Mathematical Markup Language (MathML) 2.0 [MATHML20] there are output-compliant, authoring tools and input-compliant, rendering/reading tools.
Example. Schema Part 1 could be said to define an abstract notion of "schema validity" but this checkpoint can only be satisfied if the Recommendation says explicitly whether it is setting expectations of an XML parser or of a "schema validator" that could be stand-alone. Schema Part 2 defines data types, so it is a specification of type 2 above -- content/data -- and it is used as a foundation by other specifications (e.g., path 2.0) as well as being part of the schema validator and hence parser requirements. The specification could define the behavior of a data-input tool that rejects data not fitting the schema, but it probably would not because the tool would be expected to use a parser module to validate the data. To satisfy this checkpoint, the Schema Part 2 specification has to make clear whether it is to be taken as an independent specification of a parser (that produces data from arbitrary strings) or a foundation to be used by other specifications.
Specifications that are used as components of other specifications and rely on these other specifications to define conformance must also,
Example. XPath ( XML Path Language 1.0 , [XPATH10]) relies on specifications that use XPath -- such as XPointer ( XML Pointer Language , [XPOINTER]) and XSLT ( XSL Transformations 1.0 , [XSLT10]) -- to specify criteria for conformance of implementations of XPath.
If the specification defines and addresses just one class of product, see the disclaimer requirements of Checkpoint 10.5.
Checkpoint 2.2. For each class of product, define the conformance requirements. [Priority 1]
The conformance requirements indicate the conditions to be satisfied in order to claim conformance. In addition to specifying the requirements for claiming conformance, it may be appropriate to specify that which is not a requirement. It is likely that these conformance definitions will reference normative text within the specification or in other related specifications.
Checkpoint 2.3. For each class of product, indicate minimal support requirements. [Priority 3]
For each class of product, describe the minimum set of elements, attributes, etc., that are required to claim conformance. If there are no minimum support requirements -- i.e., the minimal support requirement includes all of the functionality of the specification -- explicitly state that. This is helpful in providing the reader a full understanding of the conformance requirements, especially when some of the classes of products have minimal requirements whereas others do not.
If there are distinct minimal requirements, then -- implicitly or de-facto, if not explicitly -- the specification has levels (minimal = level 1) or modules (minimal = core module or core module set). If the specification has profiles, then any specification-wide, cross-profile distinct minimal requirements would represent something that all the profiles have in common. This effectively is a "rule for profiles", i.e., a conformance constraint on profiles that are valid by some definition in the specification.
Checkpoint 2.4. Identify which of the categories of object are specified in the document as a whole. [Priority 3]
In the Introductory section of the specification, identify which of the types of object (see the enumeration of categories in the initial text of this guideline) are specified in the document as a whole. Doing so helps keep the scope of the specification in focus, both for the reader, and for the author(s) who must define the classes of product and their conformance criteria.
Some specifications define more than one of the enumerated object types. XForms is an example. It defines: Content, via an XML vocabulary; Content, via named datatypes; Syntax, in the form of a set of functions to supplement the XPath core set; behavior of a processor; behavior of a user agent; a set of events.
Guideline 3. Address the use of profiles to divide the technology.
Dividing the technology that a specification describes into functional groupings can facilitate its implementation by enabling a developer to build to portions of the technology rather than implementing it all. Some of the dimensions of variability, but not all, can be used to divide the technology: product classes, profiles, modules, and (to some extent) levels divide the technology, while extensibility separates specified behavior from other product features.
A profile is a subset of the technology that supports a particular functional objective. Profiles can also be used to group the technologies of a set of standards and define how they are required to operate together (e.g., XHTML plus MathML plus SVG). Profiles may be based on hardware considerations associated with target product classes -- for example, SVG Tiny is aimed at mobile phones -- or they may be driven by other functional requirements of their target constituencies -- for example, a graphical profile tailored for technical illustrations in aircraft maintenance manuals.
Modules are independent divisions or functional groupings of the technology, that can be implemented independently (e.g., audio vs. video module). See Guideline 4 for the full discussion of the topic. Within W3C specifications, a very common way to define profiles is to modularize the technology, and gather together several modules to define a profile.
Levels are used to group functionality into nested subsets. See Guideline 7 for a full discussion of the topic. A profile can require a particular level in its entirety. A profile can also be defined as a subset of a level or levels. Hierarchy is an essential attribute of levels. While it is not an essential attribute of profiles, nevertheless profiles can be hierarchical in the sense that Profile B can be defined as all of Profile A plus additional capabilities.
The use of profiles to divide the technology is described in the specification, and may or may not be reflected and paralleled by the structure and organization of the specification.
Specifications may define individual profiles, or may define rules for profiles, or both. An individual profile defines the requirements for classes of products that conform to that profile. Rules for profiles define validity criteria for profiles themselves -- i.e., if others (users, applications, or other standards) define their own profiles of the standard, then rules for profiles define the constraints that those profiles must satisfy in order to be considered valid profiles of the specification.
For example, XHTML Modularization ([XHTML-MOD], section 3) and Synchronized Multimedia Integration Language (SMIL 2.0), [SMIL20] specifications both define rules for profiles -- what constraints must a profile meet in order to be classified as a "Host Language Profile" or an "Integration Set Profile." SMIL further defines some specific profiles, using the modularization. Separate recommendations -- XHTML Basic [XHTML-BASIC] and XHTML 1.1 [XHTML11] -- define specific profiles based on the XHTML modularization.
The SMIL Recommendations (e.g., [SMIL20]) illustrate several of these concepts at once. Modularization is an approach in which markup functionality is specified as a set of modules that contain semantically-related XML elements, attributes and attribute values. Profiling is a method for defining subsets of technology by identifying the functionality, parameters, options, and/or implementation requirements necessary to satisfy the requirements of a particular community of users. In SMIL, it is the creation of an XML-based language through combining these modules that provides the functionality required by a particular application community. Based on a modularization framework, SMIL defines rules for profiles -- e.g., "SMIL Host Language Conformance" (section 2.4.1, [SMIL20]) -- as well as defining some specific profiles -- e.g., the SMIL Language Profile and the SMIL Basic Profile (sections 13 and 14, [SMIL20]).
It should noted that profiling is not applicable for all W3C Recommendations, for example Namespaces in XML [XML-NAMES].
Exercise caution - profiles represent one of the dimensions of variability. Excessive variability fragments and confuses the marketplace, which inhibits interoperability. Furthermore, dividing the conformance policy of a single specification in two or more ways (i.e., by two or more dimensions of variability) creates complexity -- complexity generally retards interoperability, while simplicity generally facilitates it.
Checkpoint 3.1. Choose whether or not to have profiles. [Priority 1]
It is possible for a specification to define and use one or more of profiles, modules, functional levels, or none of these. If profiles are defined and supported in the specification, explicitly say so. Furthermore, the specification must explain why profiles are necessary, by reference to use cases and/or project requirements.
If profiles are not used, see the disclaimer requirements of Checkpoint 10.5.
Checkpoint 3.2. Include a table of contents entry. [Priority 2]
The reader should be able to easily identify and locate information about profiles. A link from the table of contents (TOC) provides this ability. The form of TOC linkage can vary, depending on the style of the specification -- some have only a high-level TOC, some have a detailed TOC, and some have both. This checkpoint could be satisfied by TOC links to specific profile-related text from a detailed TOC. For a high-level TOC, it could be satisfied by a link to a conformance clause section that either contains or points to profile information. It could be satisfied by both of the two previous when both types of TOC are present.
Checkpoint 3.3. If profiles are chosen, indicate whether or not their use is mandatory for conformance. [Priority 1]
If profiles are defined, then
For example, must content conform to one of the profiles, or is there a concept of conformance of content independent of conformance to one of the profiles? Is a producer (of content) conforming if it generates content that is otherwise valid but does not conform to a profile?
Note. If there is a "full" profile defined -- for example, incorporating all of the defined functionality of the specification, including extensibility features -- then any valid content, as well as any correct producers and fully capable consumers, might seem to be automatically using that profile. However, profiles (e.g., of content) often include self-identification requirements, and these would have to be observed for conformance of valid content to even a "full" profile.
For example: if one and only one profile can be implemented at a time.
This checkpoint is not applicable if profiles are not chosen.
Checkpoint 3.4. If profiles are chosen, define any minimal requirements. [Priority 2]
A profile places requirements on each class of product that is affected by the profile's definition.
For example, for a user-agent product class, a profile must define the minimal language support requirements for that target's user agents. This may, for example, be done by requiring specific modules (in a modularized specification), levels (if the specification is level-structured), or specific lists elements and attributes.
To illustrate, SMIL 2.0 [SMIL20] has a SMIL 2.0 Language Profile for user agents but also provides a SMIL 2.0 Basic Profile for wireless and embedded devices. The SMIL 2.0 Language Profile requires that a user agent implement the BasicAnimation module but not the SplineAnimation Module. The SMIL 2.0 Basic Profile on the other hand does not require implementation of any of the Animation modules.
This checkpoint is not applicable if profiles are not chosen.
Checkpoint 3.5. If profiles are chosen, define their relationships and interaction with other dimensions of variability. [Priority 2]
Dependency or interrelationship between profiles and modules is common -- XHTML [XHTML-MOD], SMIL [SMIL20], SVG 1.1 [SVG11]. Less often, deprecated features, levels, discretionary choices, or extensions could depend on profiles.
This checkpoint is not applicable if profiles are not chosen.
Checkpoint 3.6. If profiles are chosen, address rules for profiles. [Priority 2]
If the specification anticipates and allows that groups may define new profiles in the future,
It is a further requirement of this checkpoint that,
Well-designed rules for profiles will facilitate that derived profiles are well-specified, and testability will enable an independent tester to verify conformance of a derived profile to the rules.
It is recommended that rules for derived profiles should impose at least these two rules on derived profiles: derived profiles should be specified in a way that meets all the pertinent checkpoints of this document (QA Framework: Specification Guidelines); and, derived profiles should not clash with pre-defined profiles, if there are any in the base specification. Checkpoints from Guideline 9 (extensions) can be adapted into rules for profiles.
This checkpoint is not applicable if profiles are not chosen.
Guideline 4. Address the use of modules to divide the technology.
Dividing the technology that a specification describes into functional groupings can facilitate its implementation by enabling a developer to build to portions of the technology rather than implementing it all. Some of the dimensions of variability, but not all, can be used to divide the technology: product classes, profiles, modules, and (to some extent) functional levels divide the technology, while extensibility separates specified behavior from other product features.
Modules are non-hierarchical, discrete divisions or functional groupings of the technology. For example, SMIL 2.0 [SMIL20] defines a module as follows:
A module is a collection of semantically-related XML elements, attributes, and attribute values that represents a unit of functionality. Modules are defined in coherent sets.
Modules generally can be implemented independently of one another -- e.g., audio vs. video module. That notwithstanding, it is possible for one module's definition (and therefore implementation) to have explicit dependency upon another module. It is not only possible, but common to implement multiple modules.
Modules and profiles. Within W3C, a common relationship between modules and profiles is to build profiles by assembling modules in different collections. This approach is used in both XHTML and SMIL 2.0, and in fact is the driving motivation behind their modularization. In both, it is required that modules be atomic when included in profile definitions -- profiles cannot subset modules. Conversely, there are examples where profiles subset (some) modules, and are defined by a collection of modules and pieces of modules (e.g., the Tiny and Basic profiles of the SVG Mobile [SVG-MOBILE] subdivide some of the modules of SVG 1.1 [SVG11]).
Atomicity of modules. Several W3C specifications with both modules and profiles have the requirement that modules must be atomic -- modules may not be subdivided in profiles. Such a constraint requires:
This is a recommended design constraint for specifications that modularize their technologies for the purpose of building profiles, because it limits the variability amongst derived profiles. Examples: The rules for profiles in XHTML and SMIL require atomicity of modules, in order that the resulting profiles qualify for classifications such as "Host Language Conformant" or "Integration Set Conformant". See Guideline 3 for full discussion of profiles.
Modules and levels. Levels are specifically designed to represent hierarchical steps, for example DOM level 1, DOM level 2, and DOM level 3. See Guideline 7 for full discussion of levels. Levels in W3C specifications most commonly reflect progressive historical development of the specification, rather than all-at-once design and standardization. In current W3C practice, nested profiles have often been used in cases of one-time definition of functional levels within a specification. The nested Tiny, Basic, Full profiles of SVG Mobile [SVG-MOBILE] are an example that could, in principle, have been defined as three levels.
The relationship of modules and levels is less obvious than the relationship of modules and profiles. In principle, at least, two relationships can be envisioned:
[@@Issue about goodness criteria, from 7/24 telecon. I.e., levels could divide modules or whole modules could be grouped into levels during progressive (historical) development of the standard. Is one good and the other bad? Is either one to be encouraged or discouraged?]
Exercise caution - modules represent one of the dimensions of variability. Excessive variability fragments and confuses the marketplace, which inhibits interoperability. Furthermore, dividing the conformance policy of a single specification in two or more ways creates complexity -- complexity generally retards interoperability, while simplicity generally facilitates it.
Checkpoint 4.1. Choose whether or not to have modules. [Priority 1]
It is possible for a specification to define and use one or more of profiles, modules, functional levels, or none of these. If modules are defined and supported in the specification, explicitly document it. Furthermore, the specification must explain why modules are necessary, by reference to use cases and/or project requirements.
If modules are not used, see the disclaimer requirements of Checkpoint 10.5.
Checkpoint 4.2. Include a table of contents entry. [Priority 2]
The reader should be able to easily identify and locate modules information. A link from the table of contents (TOC) provides this ability. The form of TOC linkage can vary, depending on the style of the specification -- some have only a high-level TOC, some have a detailed TOC, and some have both. This checkpoint could be satisfied by TOC links to specific modules-related text from a detailed TOC. For a high-level TOC, it could be satisfied by a link to a conformance clause section that either contains or points to modules information. It could be satisfied by both of the two previous when both types of TOC are present.
Checkpoint 4.3. If modules are chosen, indicate any mandatory conditions or constraints on their usage. [Priority 1]
If there are additional conditions or constraints associated with modules, they must also be described. Such conditions include,
The conditions or constraints normally will be tailored according to class of product.
This checkpoint is not applicable if modules are not chosen.
Checkpoint 4.4. If modules are chosen, define their relationships and interaction with other dimensions of variability. [Priority 2]
Often there is dependency or interrelationship among modules, on the one hand, and profiles or discretionary choices on the other. Modules may have levels (although no such examples are known amongst W3C standards), or deprecated features. Extensions could be defined based on modules. [@@Issue about goodness criteria, from 7/24 telecon. I.e., levels could divide modules or whole modules could be grouped into levels during progressive (historical) development of the standard. Is one good and the other bad? Is either one to be discouraged?]
This checkpoint is not applicable if modules are not chosen.
Guideline 5. Specify conformance policy.
A look at various W3C Technical Reports shows that the term "conformance" is often qualified, resulting in more than one type of conformance. It is important to convey an understanding of what is meant by conformance and where the products of a class are allowed to have more or less functionality than defined in the specification. If the specification defines behavior for more than one class of product, there may be a separate conformance policy for each class. Often, the specification will allow discretionary choices, such as the choice of one or more natural languages for verbal content and messages, but require a conforming product to make a choice only within the allowable range. (See Guideline 8 for more discussion.)
Sometimes a product developer can choose to implement certain modules. There may be per-module conformance requirements that apply if and only if the developer chooses to implement a particular module.
Where all products of a class must be substantially alike, it should be clear that a "strict conformance" policy is in effect for that product class. Strict conformance is defined as conformance of an implementation that employs only the requirements and/or functionality defined in the specification and no more (i.e., no extensions to the specification are implemented).
Sometimes a product developer can choose to implement extensions. There may be conformance requirements for non-interference of extensions. (See Guideline 9 for more discussion.)
Overall, the intent of the WG should be clear. In particular, a reader intending to implement a product in one of the product classes addressed by the specification should know what the WG wants for interoperability among products in the class. The developer should understand what forms, if any, of "product differentiation" are allowed among conformant products. The specification may need to explain how the rules apply and possibly provide examples.
Guideline 10, "conformance clause" is related to this guideline. This Guideline 5 focuses on the establishment and scope of definition of a conformance policy, while Guideline 10 focuses on where and how to document it. That is, the verification of these checkpoints will require looking at the Conformance Clause.
Exercise caution - the conformance policy is one of the dimensions of variability, and there are several opportunities for variability within it. Excessive variability fragments and confuses the marketplace, which inhibits interoperability. Furthermore, dividing the conformance policy of a single specification in two or more ways creates complexity -- complexity generally retards interoperability, while simplicity generally facilitates it.
Checkpoint 5.1. Make it clear where there are universal requirements for minimum functionality. [Priority 1]
The reader must be able to recognize any minimum that applies to all conforming products of each class. If levels are used (see Guideline 7), the lowest level represents the minimum set of requirements. If profiles are used there may be different minima for each profile. If modules are used, there will normally be different minima for each module. Furthermore, a module may itself be a minimum (i.e., required) for a particular class of product.
Checkpoint 5.2. Make it clear when conformance requirements are strict. [Priority 1]
The reader must be able to recognize when a policy of "strict conformance" applies. As defined above, this implies that all conformant products of a class behave essentially the same way. If profiles are used, each profile may have its own conformance boundaries. If modules are used, each module may have its own conformance boundaries.
Use the definition provided above (or in the QA Glossary [QA-GLOSSARY]). The definition may be modified to adjust its scope, for example when applying it to modules, profiles or levels. In such cases, the scope of "requirements [...] defined in the specification" is narrowed from specificationto the module, profile, or functional level that is the target of the statement.
Checkpoint 5.3. Make it clear where requirements stop and product-specific extra features begin. [Priority 1]
The reader must be able to recognize conformance requirements as distinct from allowable extra functionality. If profiles are used (see Guideline 3), make it clear whether extra capabilities of the platform may be exploited. If modules are used (see Guideline 4), there may be different provisions for extra features applying to each. If deprecation applies (see Guideline 6), make it clear whether support of the obsolete features is optional, part of a level, or required. If levels are used (see Guideline 7), make it clear whether the highest level may be exceeded with additional features or functionality. If discretionary choices are allowed (see Guideline 8), make it clear if more than one may be implemented, when it is technically possible to do so.
Checkpoint 5.4. If special conformance terms are used, include a definition in the specification. [Priority 1]
It is necessary to define terms that govern application of the conformance provisions. Ideally, all terms are from QA documents and other existing literature and need only be cited. If special terms are constructed, such as to combine modules and levels or modules and discretionary choices, these terms must be defined in the specification. The conformance clause must reference all locally-defined terms that affect conformance.
Guideline 6. Clarify the relation between deprecated features and conformance.
After the initial publication of a specification, specification developers may consider the deprecation of a feature (e.g., function argument, element or attribute) defined in the specification. A deprecated feature is a feature whose use is discouraged because it has been outdated by newer constructs or is no longer viable. Deprecated features may become obsolete and no longer defined in future versions of the specification. Deprecation of a feature may warn implementers that the feature was a bad idea and it may be withdrawn in the future. Specification developers need to consider the effect of deprecation on all the classes of products that implement the specification (e.g., authoring tools, user agents) as well as the conformance consequences on each class of product. For the purpose of backward compatibility, it may be necessary to specify different requirements for the support of deprecated features for each class of product. For example, authoring tools (producers) would not use the feature, but user agents (consumers) would continue to support it.
Checkpoint 6.1. Identify and clearly indicate each deprecated feature. [Priority 1]
Any editions or versions of a specification after the first edition or version must
If there are no deprecated features, see the disclaimer requirements of Checkpoint 10.5.
Checkpoint 6.2. For each class of product, specify the degree of support required for each deprecated feature and the conformance consequences of the deprecation. [Priority 1]
Define what it means for a feature to be deprecated and how this affects conformance. For example, a deprecated-features section of MathML 2.0 ([MATHML20], section 18.104.22.168) describes, about deprecated MathML 1.x features, that MathML-output-compliant authoring tools may not generate MathML markup containing deprecated features; whereas MathML-input-compliant rendering/reading tools must support deprecated features.
Checkpoint 6.3. Include an explanation for the deprecation. [Priority 3]
Providing the rationale for deprecating a feature helps implementers and users to understand the motivation for the deprecation, the impact and consequences on current and future implementations, and perhaps insight into its eventual disappearance from the specification.
Checkpoint 6.4. Include examples to illustrate how to avoid using deprecated features. [Priority 3]
Examples are helpful in providing alternatives or better ways to get the same results. By showing what can be done in place of the deprecated feature will help to get implementers to discontinue use of the deprecated feature.
Checkpoint 6.5. Include a table of contents entry. [Priority 2]
The reader must be able to easily identify and locate information about deprecated features. A link from the table of contents (TOC) provides this ability. The form of TOC linkage can vary, depending on the style of the specification -- some have only a high-level TOC, some have a detailed TOC, and some have both. This checkpoint could be satisfied by TOC links to specific deprecation-related text from a detailed TOC. For a high-level TOC, it could be satisfied by a link to a conformance clause section that either contains or points to deprecation information. It could be satisfied by both of the two previous when both types of TOC are present. In the case that there are numerous deprecated features, consider a link to a table or list, which links to the individual locations in the text.
Guideline 7. Address the use of functional levels to divide the technology.
Dividing the technology that a specification describes into functional groupings can facilitate its implementation by enabling a developer to build to portions of the technology rather than implementing it all. Some of the dimensions of variability, but not all, can be used to divide the technology: product classes, profiles, modules, and (to some extent) functional levels divide the technology, while extensibility separates specified behavior from other product features.
Functional levels -- or in common usage, simply "levels" -- are used to group functionality into nested subsets, ranging from minimal or core functionality to full or complete functionally. Level 1 is the minimum or core of the technology. Level 2 includes all of level 1 plus additional functionality. This nesting continues until level n, which consists of the entire technology.
[@@Issue: Scope of minimality of levels. Levels represent the minimal or core requirement within the levels DoV. Do they also represent a minimal or core requirement in the 3- DoV subspace spanned by the profiles-modules-levels DoVs?]
Levels may result from progressive historical development and enrichment of the technology in a series of specifications, as in the case of CSS and DOM. Levels may also be defined explicitly in a single edition of the specification, including in the first edition. No examples of this are found in W3C specifications. Rather, it is more common in current W3C practice to use profiles to accomplish this. For example, SVG Mobile [SVG-MOBILE] defines three nested profiles -- Tiny, Basic, Full -- which are each targeted at a specific graphics hardware community (mobile phone, hand-held computer, desktop computer).
See Guideline 3 for full discussion of profiles, including comments on possible profiles-levels relationships. See Guideline 4 for a full discussion of modules, including possible modules-levels relationships.
Exercise caution - levels represent one of the dimensions of variability. Excessive variability fragments and confuses the marketplace, which inhibits interoperability. Furthermore, dividing the conformance policy of a single specification in two or more ways creates complexity -- complexity generally retards interoperability, while simplicity generally facilitates it.
Checkpoint 7.1. Address whether the specification uses or will use functional levels. [Priority 1]
It is possible for a specification to define and use one or more of profiles, modules, functional levels, or none of these. If levels are defined and supported in the specification, explicitly say so. Furthermore, the specification must explain why levels are necessary, by reference to use cases and/or project requirements.
If levels are not used, see the disclaimer requirements of Checkpoint 10.5.
A specification may come to have levels in a number of ways:
In all cases, to satisfy this checkpoint, a specification must address the use of levels.
Checkpoint 7.2. Include a table of contents entry. [Priority 2]
The reader must be able to easily identify and locate levels information. A link from the table of contents (TOC) provides this ability. The form of TOC linkage can vary, depending on the style of the specification -- some have only a high-level TOC, some have a detailed TOC, and some have both. This checkpoint could be satisfied by TOC links to specific levels-related text from a detailed TOC. For a high-level TOC, it could be satisfied by a link to a conformance clause section that either contains or points to levels information. It could be satisfied by both of the two previous when both types of TOC are present.
Checkpoint 7.3. If levels are used, define their relationships and interaction with other dimensions of variability. [Priority 2]
Levels can be dependent on, or apply to, modules. Less often, there can be a relationship between levels, on the one hand, and profiles or deprecated features on the other.
This checkpoint is not applicable if levels are not used.
Guideline 8. Define discretionary items.
Discretionary items are often made available in specifications, to give implementers of the technology the opportunity to decide from alternatives when building applications and tools. Discretionary items are defined as deliberate and explicit grants of discretion by the specification, to the implementations, that describe or allow optionality of behavior, functionality, parameter values, error handling, etc. Discretionary items may be considered necessary because of environmental conditions (e.g., hardware limitations or software configuration, or external systems), locality (e.g., time zone or language), optional choices providing flexibility of implementation, dependence on other specifications, etc.
Discretionary items come in three basic variants:
One type of discretionary item is implementation dependent values. Implementation dependent values are used when it is not desired or not possible to define the behavior or values of a function. Implementation dependent means that an implementation may determine the effect, rather than using an effect mandated by the specification. Details in a specification may deliberately be omitted (i.e., not specified), so as to provide freedom to adapt implementations to different environments and different requirements.
Exercise caution - discretionary behaviors represent one of the dimensions of variability. Excessive variability fragments and confuses the marketplace, which inhibits interoperability. Furthermore, dividing the conformance policy of a single specification in two or more ways creates complexity -- complexity generally retards interoperability, while simplicity generally facilitates it. Implementation-dependent values is a form of discretionary behaviors which can have particularly harmful interoperability impacts -- it is not a recommended practice.
[@@Issue: add a checkpoint? -- "if impl dependent is allowed, document how it affects interoperability of implementations."]
Checkpoint 8.1. Explicitly state the cases and conditions where discretionary items are allowed. [Priority 1]
Although there may be many individual discretionary items sprinkled throughout the specification, they often arise from broader ideas about how the specified software should operate. To help readers, implementers and testers to find these discretionary items, it is recommended that where ever possible these items be collected and presented as a group. For example, in XSLT [XSLT10] most discretionary items offer two choices, representing design philosophies to escalate an error or to continue processing. Thus, XSLT could address discretion as a choice between two groups of items.
Furthermore, the specification must explain why discretionary choices are necessary, by reference to its use cases and/or project requirements.
If there are no discretionary items in the specification, see the disclaimer requirements of Checkpoint 10.5.
Checkpoint 8.2. Indicate implementation dependencies and address allowable differences between implementations, where applicable. [Priority 1]
Examples of allowable differences to be addressed include:
Checkpoint 8.3. Describe alternative approaches and the conditions under which an implementation is considered to be conforming. [Priority 1]
Specifications may describe several different ways to accomplish its operation (e.g., a choice of file formats, protocols, or encodings). In such a case, enumerate the approaches and specify if there are limitations on the number of approaches or combination of approaches that can be implemented. Some possible ways to define conformance when allowing alternative approaches include mandating that an implementation:
Note that if the specification does not include the different approaches, this becomes an implementation detail.
Checkpoint 8.4. Include a statement regarding consistent handling of a discretionary item within an implementation. [Priority 2]
The effect of each individual discretionary item should be consistent within a single implementation. For example, a browser's rendering of a XSL-FO (XSL Formatting Object) should be the same for every invocation regardless of the document instance.
Checkpoint 8.5. Include a table of contents entry. [Priority 2]
The reader must be able to easily identify and locate information about discretionary items. A link from the table of contents (TOC) provides this ability. The form of TOC linkage can vary, depending on the style of the specification -- some have only a high-level TOC, some have a detailed TOC, and some have both. This checkpoint could be satisfied by TOC links to specific discretionary items text from a detailed TOC. For a high-level TOC, it could be satisfied by a link to a conformance clause section that either contains or points to discretionary-items information. It could be satisfied by both of the two previous when both types of TOC are present. In the case that there are numerous discretionary features, consider a link to a table or list, which links to the individual locations in the text.
Guideline 9. Allow extensions or NOT!
An extension to a specification is a mechanism to incorporate functionality beyond what is defined in the specification. Allowing extensions affects how conformance is defined as well as what conformance claims can be made. Exercise caution in determining the extent to which extensions are allowed or not allowed. Since extensions can seriously compromise interoperability, specification writers should carefully consider whether extensions should be allowed.
Extensions may be private (often vendor specific) or may be public (a full description of the extension is public). Private extensions are usually truly private, i.e., valid for a specific implementation or are only known by prior agreement between implementations. Public extensions are extensions in which the syntax, semantics, identifiers, etc. are defined and published allowing anyone to implement the extended functionality.
Specifications allow extensions for various reasons. Extensions allow implementers to include features that they consider to be required by their customers. Also, extensions, often define new features that may migrate into future versions of the specifications. However, the use of extensions can have a severe negative impact on interoperability. Some methods for enabling extension have less impact on interoperability than other methods. For example, a specification that allows private extensions (e.g., proprietary) is highly likely to impede interoperability, whereas a specification than permits only registered extensions partially mitigates the negative impacts.
Checkpoint 9.1. If extensions are disallowed, explicitly state it. [Priority 1]
If extensions are not allowed, then make sure it is clear to the reader that not only are extensions not allowed, but the circumstances under which they are not allowed. The implementations of the specification precisely implement the specification. This is strict conformance. Strict conformance is often imposed on applications or content (e.g., a software program or document instance). This prohibition of extensions could be applied to a specific profile or module, rather than to the entire specification.
Checkpoint 9.2. If extensions are allowed, explicitly state it. [Priority 1]
State the conditions under which extensions are allowed, the applicability of the extensions, their effect on conformance claims, their anticipated effect on interoperability, and any limitations or restrictions on the use of the extension. Because of potentially serious negative impacts on interoperability, and to help product developers understand the motivation for their inclusion and their intended use, the specification must also explain the rationale for allowing extensions, by reference to the use cases and/or project requirements.
Checkpoint 9.3. If extensions are allowed, make it clear that the extensions do not negate support for required functionality. [Priority 1]
An extension does not change the fact that an implementation needs to support all required functionality in the specifications exactly as specified; nor does it cause the non-conformance of functionality defined in the specification. The specification can include statements such as:
Checkpoint 9.4. If extensions are allowed, use a standard mechanism to define the extension. [Priority 3]
One mechanism to allow extensions within a specification is to provide a standard way of defining the extension or a "standard way of being non-standard". This helps to ensure predictable handling of extensions, that is, its recognition as such and the appropriate actions (i.e., to ignore or to implement). The nature of the extension dictates the method for defining the extension. It may be possible to define a generic function or mechanism that indicates external functionality. This external function may take the form of an escape or control character or may be an identifier, the presence of indicates that an extension follows. Another method, especially when extending a list of numeric parameters is to use a scheme where positive values represent standardized values and negative values are reserved for private extensions.
Checkpoint 9.5. If extensions are allowed, register or publish them. [Priority 3]
Registration is a procedure that allows extensions to be acknowledged and made available to the public. Registration provides for a degree of rigor and technical review for any proposed extension. Typically the WG would be responsible for processing the registration of an extension, thus ensuring adequate quality of a proposed extension and a technical description sufficient to be uniformly implementable. Often registered extensions may migrate into a later version of the specification.
[@@Issue. This was raised as an issue in QAWG project review (member-only), but we never got to it on the agenda.]
Checkpoint 9.6. If extensions are allowed, require that implementations include a way to operate without the extension. [Priority 3]
If an implementation contains extensions, require it to have a mode under which the implementation can be directed to produce only conforming files (documents) or to operate in a strictly conforming manner.
Checkpoint 9.7. Include a table of contents entry. [Priority 2]
The reader must be able to easily identify and locate information about extensions and extensibility. A link from the table of contents (TOC) provides this ability. The form of TOC linkage can vary, depending on the style of the specification -- some have only a high-level TOC, some have a detailed TOC, and some have both. This checkpoint could be satisfied by TOC links to specific extensibility section(s) from a detailed TOC. For a high-level TOC, it could be satisfied by a link to a conformance clause section that either contains or points to extensibility information. It could be satisfied by both of the two previous when both types of TOC are present.
Guideline 10. Provide a conformance clause.
A conformance clause is a part or collection of parts of a specification that defines the requirements, criteria, or conditions to be satisfied by an implementation or application in order to claim conformance. Typically the conformance clause is a high-level description of what is required of implementations and applications.
Guideline 5, "conformance policy" is related to this guideline. Guideline 5 focuses on the establishment and scope of definition of a conformance policy, while this Guideline 10 focuses (among other topics) on how and where to document it.
Checkpoint 10.1. Include a conformance clause. [Priority 1]
Every specification must document its conformance policy and specific conformance requirements somewhere in the specification. As used in this checkpoint, "clause" does not necessarily imply a specific single document section or location (see next checkpoint).
Checkpoint 10.2. Create a separate conformance section. [Priority 2]
Having the conformance clause exist as a separate section within the specification makes it clearly identifiable, allowing a reader to find the overall conformance policy, as well as all specific conformance provisions from a single starting point.
Checkpoint 10.3. Include a conformance clause entry in the table of contents. [Priority 2]
It must be possible for the reader to start at the table of contents and find all aspects of the conformance requirements, including the overall policy.
Checkpoint 10.4. Make normative reference to specifications on which the current specification depends. [Priority 2]
Often a specification is dependent on other specifications or portions of specifications. For example, SVG 1.0 requires that the class of product called "user agent" be consistent with the XML 1.0 Recommendation [XML10 ] and (conditionally) support Cascading Style Sheets, level 2 [CSS2]. Dependence on other specifications affects the conformance boundaries of the current specification, and thereby affects the requirements on conformant products. To ensure clarity and understanding of these implications, describe the relationship between the specifications and any conformance implications. The linking parts of the Manual of Style ([STYLE-MAN], section 11.5.1) describe the recommended way of citing an external reference from the prose of a specification, as well as how to construct an entry in its References section.
Checkpoint 10.5. Identify all dimensions of variability that are not used. [Priority 1]
It must be possible for the reader to determine, from the conformance clause, if:
This requirement applies separately to each dimension. For each dimension where variability is permitted, see the checkpoint about making explicit statements and justifications, under each respective guideline 2-9.
This checkpoint is not applicable for specifications (drafts and Recommendations) that are issued before this document is final. Although such legacy documents are exempt from these requirements to make explicit statements of non-use of dimensions, the respective checkpoints still apply for dimensions that are used.
Guideline 11. Specify how to make conformance claims.
A specification may differentiate conformance claims by designating different degrees or types of conformance in order to apply and group requirements according to modules, profiles, levels, or priorities. When a conformance claim is linked to functionality, impact and/or incremental degrees of implementation, the term 'conformance level' is often used to indicate the varying degrees of conformance. The WG includes in the specification the way they want people to claim their conformance.
Checkpoint 11.1. Identify and define all conformance designations. [Priority 1]
In current W3C practice, a number of different naming conventions are used to label conformance, in cases where there is not a single, monolithic (strict) conformance definition. The naming convention used to label the conformance can provide useful information. Degrees, for example, implies incremental importance or difficulty. This Specification Guidelines document uses "degrees" for example, to refer to three successively more demanding degrees of conformance (A, AA, AAA).
Commonly used conformance designations include categories, degrees, and levels. Use of "conformance levels" is discouraged in new specifications, because of the potential for confusion with "functional levels".
Each specification must
Functional (technology) specifications commonly use groupings of functionality to distinguish amongst conformance designations. For example, SVG Mobile [SVG-MOBILE] defines three conformance categories as three nested profiles -- Tiny, Basic, and Full -- based on the modularization description of SVG 1.1 (Scalable Vector Graphics 1.1, [SVG11], section 1.1.2). Each of the profiles is equivalent to a functional level that subsumes the next lower level (if any), providing additional functionality and/or complexity.
The conformance designations of the guidelines documents of WAI and the QA Framework, by comparison, reflect the a target object's satisfaction of successively larger sets of prioritized checkpoints. A basic conformance designation (A) is associated with the Priority 1 checkpoints. The other designations, for AA (double-A) and AAA (triple-A), are associated with the addition respectively of all Priority 2 and all Priority 3 checkpoints. Examples: Web Content Accessibility Guidelines 1.0 [WCAG10]), in which the ; and, QA Framework: Specification Guidelines (this document, see Conformance section.)
Checkpoint 11.2. Provide specific wording of the claim. [Priority 3]
A well-formed conformance claim includes: date of the claim, specification name, date and version, URI of the specification, conformance level satisfied, and information about the subject (that which is claiming conformance). Information about the subject refers to information such as, the name of the software or software component, version information, and operating environment.
Checkpoint 11.3. Provide a conformance disclaimer. [Priority 3]
Although it is possible to prove with certainty when something does not conform, the reverse is not necessarily true. Especially for functional specifications, where a claim goes beyond syntax testing, a claim of conformance is not a guarantee that the claimant is 100% conforming with the specification. A disclaimer can help clarify the meaning of a conformance claim as well as its limitations. For example, this document contains a conformance disclaimer.
Checkpoint 11.4. Impose no restrictions about who can make a claim or where claims can be published. [Priority 1]
Claimants (or relevant assuring parties) are solely responsible for the validity of their claims, keeping claims up to date, and proper use of the conformance icons. As of the publication of this document, W3C does not act as an assuring party, but it may do so in the future, or it may establish recommendations for assuring parties. Claimants are expected to modify or retract a claim if it may be demonstrated that the claim is not valid.
Checkpoint 11.5. Include a table of contents entry. [Priority 2]
The reader should be able to easily identify and locate the information on how to make conformance claims. A link from the table of contents (TOC) provides this ability. The form of TOC linkage can vary, depending on the style of the specification -- some have only a high-level TOC, some have a detailed TOC, and some have both. This checkpoint could be satisfied by TOC links to specific conformance-claims text from a detailed TOC. For a high-level TOC, it could be satisfied by a link to a conformance clause section that either contains or points to conformance-claims information. It could be satisfied by both of the two previous when both types of TOC are present.
Guideline 12. Publish an Implementation Conformance Statement proforma.
An Implementation Conformance Statement (ICS) provides a mechanism whereby a supplier of an implementation of the specification provides information about the implementation in a standardized manner. It is used to indicate which capabilities and options have been implemented, as well as the limitations of the implementation. A ICS usually takes the form of a questionnaire where product implementors report on the conformance of their product to the named specification.
An ICS is useful in disclosing optional functionality and discretionary behavior and values. The results of the ICS can be used to identify the subset of test cases from a conformance test suite that are applicable to the implementation to be tested. This will allow the implementation to be tested for conformance against only the relevant requirements.
The basic and detailed information that an ICS provides can also be used to assess and deduce the interoperability potential of two or more products.
Checkpoint 12.1. Include an Implementation Conformance Statement proforma as part of the specification. [Priority 3]
If an ICS is included as part of the specification, indicate whether it is a normative or informative part of the specification.
Checkpoint 12.2. Require the ICS be completed as part of the conformance claim. [Priority 3]
An ICS provides specific information about the implementation and can be helpful in substantiating the conformance claim.
Guideline 13. Support general document conformance conventions.
There is a lot to be said for consistency and clarity within a document - it facilitates the understanding and comprehension of the document. Authors and editors of specifications should already be familiar with the W3C Manual of Style [STYLE-MAN] and Publication Rules (member-only) [PUBRULES ], which help to achieve this. With respect to conformance, it is important to provide clear and unambiguous statements, so that the reader knows what is required in order to claim conformance and what is optional. To achieve this objective, throughout the document, employ uniformity of structure and style, and consistency of terminology and phraseology.
Checkpoint 13.1. Use conformance key words. [Priority 1]
Use RFC 2119 key words to denote whether or not requirements are mandatory, optional, or suggested. Using these keywords helps to identify the testable statements in a specification.
Checkpoint 13.2. Distinguish normative and informative text. [Priority 2]
Normative statements are the prescriptive parts of the specification whereas informative statements are for informational purposes and assist in the understanding or use of the specification. It is important that the reader be able to distinguish between normative and informative statements in order to know what is required to claim conformance. SMIL 2.0 is an example, indicating within every subsection whether it is normative or informative, and even separately labelling pieces of subsections that contain both kinds of text.
Checkpoint 13.3. Follow Web Accessibility Initiative and Internationalization Guidelines. [Priority 1]
Applying the principles of the Web Content Accessibility Guidelines 1.0 [WCAG10] and Internationalization guidelines not only ensures accessibility and internationalization, but also contributes to the principles of conformance - clear, unambiguous, testable statements. For example, use simpler words to express your ideas, markup text with structural elements, add markup to distinguish common words from keywords.
Checkpoint 13.4. Use consistent terminology. [Priority 1]
Use identical wording to express identical provisions, and analogous wording to express analogous provisions.
Guideline 14. Use granular grammars to author the specification.
Using a W3C endorsed grammar language (DTD or XML Schema) provides control over the information conveyed in the specification. This allows for and facilitates automatic generation of test materials, more detailed reporting of test results as well as specification coverage.
Examples include the XML and DOM Working Groups, that both use the xmlspec DTD .
Using a granular grammar to author a specification draws further on separation of content and presentation and simplifies the task of generating readable versions of the specification, maintaining the possibility to generate test materials from the granular version of the specification. In addition, using a grammar to author the specification makes it easier to logically group the constituents of the specification, thus adding control over interpretation and implementation.
Using a grammar greatly simplifies controlling conformance with W3C publication rules and guidelines/checkpoints. Also, having written the specification using a grammar, automated validation with regard to these aspects is simplified.
Finally, using an advanced schema for specification authoring allows for greater control over coverage of the specification in the Test Suite.
Checkpoint 14.1. Use W3C endorsed grammar. [Priority 2]
One example is the DTD noted above, used by
a number of W3C specifications. There are other examples that use XHTML, with
the application of
class attributes to convey significant
information about structure or meaning.
Checkpoint 14.2. Identify intended behavior in the specification using markup. [Priority 2]
There are different types of technologies that are standardized by W3C specifications, and this checkpoint applies in different ways to those. For API specifications, such as DOM, the intended behavior is the algorithm output; the specification contains wording on what the expected result of applying a particular method on an interface is. On user-centric specifications, such as WAI guidelines specifications, the intended behavior is not so much a question of testing algorithms, but rather how user agents should behave given different input or how markup should be designed. In this case, intended behavior is more a question of parsing documents. On processor specifications, such as XML, the intended behavior is, for example, preserving or expanding entity references, and the intended behavior is a particular state after processing a document.
Checkpoint 14.3. Supply prose description of intended behavior together with each test assertion. [Priority 2]
Following the guideline below, group test assertions and statements of intended behavior in the specification, or group the pointer to the test assertions together with the statement of intended behavior in the specification (in case the test assertion is, for example, given in an appendix).
Guideline 15. Include test assertions.
Some specifications include test assertions as part of the specification, and more should. A test assertion is a statement of behavior, action or condition that can be measured or tested. It is derived from the specification's requirements and bridges the gap between the narrative of the specification and the test cases. Each test assertion is an independent, complete, testable statement for requirements in the specification. Each test assertion results in one or more test cases. Multiple test assertions can be combined to form a test case, in this case one tests multiple facets of a particular behaviour. Including test assertions as part of the specification facilitates and promotes the development of test materials. Tests can point directly to the test assertion in the specification. Specific benefits include:
Checkpoint 15.1. Supply test assertions in the specification.. [Priority 1]
In order to enable pointing to test assertions from tests as well as to give a map of the specification from the point of view of tests, use a mechanism for making explicit test assertions in the specification.
Checkpoint 15.2. Tag test assertions according to a granular grammar. [Priority 2]
Where applicable, use an existing markup vocabulary for making explicit the test assertions for the specification. If possible, use markup provided in QA Framework: Specification Examples & Techniques [SPEC-EXTECH]. Otherwise construct test assertion markup according to that document and/or QA Framework: Test Examples & Techniques [TEST-EXTECH].
This section defines conformance of Working Group specifications -- i.e., technical reports -- to the requirements of this QA Framework guidelines specification. The requirements of this guidelines specification are detailed in the checkpoints of the preceding "Guidelines" chapter, and apply to the technical reports produced by Working Groups.
The test assertions of this Specification Guidelines document are found in the prioritized checkpoints. A checkpoint will contain at least one, and may contain multiple individual requirements. These requirements are the test assertions of this specification. A checkpoint is satisfied by satisfying all of the individual requirements. Failing one individual requirement means that the checkpoint is not satisfied.
This section defines three degrees of conformance to this guidelines specification:
A specification conforms to the QA Framework: Specification Guidelines at degree X (A, AA, or AAA) if the Working Group meets at least all degree X conformance requirements.
To make an assertion about conformance to this document, specify:
This specification conforms to W3C's QA Framework: Specification Guidelines, available at http://www.w3.org/TR/qaframe-spec/, AA-Conforming.
The checkpoints of this guidelines specification present verifiable conformance requirements about the specifications (technical reports) that Working Groups produce. As with any verifiable test requirements, it is also true of these specification requirements that:
[@@Ed note. under construction.] This section contains terms used in this specification, with functional or contextual definitions appropriate for this specification. See also [QA-GLOSSARY]. Some terms in this section have been borrowed or adapted from other specifications.
The following QA Working Group and Interest Group participants have contributed significantly to the content of this document:
Significantly reorganized and revised the first published WD. This version produced as a series of editor's drafts. The changes below are reverse chronological (most recent first), so more recent ones may build on older ones.
First published WD.