Copyright ©2002 W3C ® (MIT, INRIA, Keio), All Rights Reserved. W3C liability, trademark, document use and software licensing rules apply.
Draft "Introduction" section for 2002-11-06 publication.
This is the editor version of the spec GL Introduction and has no real standing. Please see the QA WG document lists for the latest WG or TR versions.
An appendix to this document [SPEC-CHECKLIST] presents all checkpoints in a tabular form, for convenient reference.
[@@Ed note. There is some difficulty with writing the introduction, as GL14 has disappeared from the list, and the treatment of markup grammars, as they relate to test assertion topics, has been rolled into GL15. This was discussed at Tokyo. Although I agree with the decision to limit granular grammars (gg) treatment here to QA topics -- i.e., let's not get into gg benefits that fall in the realm of Style Manual or pubrules -- on the other hand, "test assertions" alone may not cover it. In particular, the 4th bullet below, facilitate auto-generation of TM, may have gotten lost in the process. Accordingly, I have moved most of the "Theme 2: granular grammar" stuff into an appendix for safekeeping, while we decide what to do. I do suspect that we need a lower-priority gg checkpoint back, in order to fully cover both the "TM generation" question as well as the TA requirements. "Theme 2, as an organizing concept per se, is gone.]
Goals. The principal goal of this document is to define a framework to assist the W3C Working Groups (WGs) in writing specifications that:
Scope. Within this Specifications Guidelines document, the term "specifications" is specifically limited to: W3C Technical Reports (@@W3C Recommendations? W3C Rec-track documents?@@)
Target Audience. The target audience of these specification guidelines is:
It is a design goal of these guidelines the WGs can apply them in a common-sense and workable manner.
[@@Ed note. in more detail...too much? remove all or part of this pgph?@@] This document identifies the conformance requirements and statements to be included or addressed in specifications as well as addressing structure and the anatomy of the specification. [@@Definition:] Conformance requirements are the expressions that convey the criteria to be fulfilled in an implementation of a specification. The anatomy of the specification pertains to the method for writing the specification using schemata.
More scope. This document describes what goes into the specification with respect to conformance and conformance topics, followed by rules for specification anatomy that should facilitate test development as well as produce better, more testable specifications.
Limitation of scope. The set of checkpoints as a whole has been written in anticipation that the checkpoints are being applied to new, in-development specifications. It is not expected that existing specifications that pre-date these guidelines will be able to satisfy all checkpoints as they are stated. Many legacy specifications may indirectly comply with the spirit or intent of some checkpoints, without actually satisfying all requirements in those checkpoints.
[@@Ed note. where to put following? delete it?] More audience. The Framework as a whole is intended for all Working Groups, as well as the specific target users (above). Not only are the Working Groups the consumers of these guidelines, they are also key contributors. The guidelines aim to capture the experiences, good practices, activities, and lessons learned of the Working Groups -- that is, to standarize the best of current practice -- and present them in a comprehensive, cohesive set of documents for WGs. The objective is that the WGs can reuse what works rather than reinvent. This will foster consistency across the various Working Group quality activities and deliverables. [@@Ed note. Maybe this goes before the 4-bullet list (if it's not deleted)?]
Good specifications lead to better implementations and foster the development of conformance test suites and tools. Conforming implementations lead to interoperability.
The quality of the specification has direct impact on the quality of implementations. Quality encompasses utility which refers to the usefulness of the specification to the intended users and objectivity which focuses on the whether the specification is presented in an accurate, clear, complete, and unbiased manner.
Providing requirements and definitions about conformance topics, as well as guidance in the structure and anatomy of specifications, will foster a mutual understanding amongst developers of specifications, implementations, and conformance test materials. Specifically, well-structured specifications with clear and comprehensive conformance requirements:
[@@Ed note. I'm out of time. There may be additional bits in Appendix A -- the entire previous "Motivations" section -- that belong in here.]
This document is part of a family of QA Framework documents designed to help the WGs improve all aspects of their quality practices. The QA Framework documents are:
The QA Framework documents are interrelated and complement each other. For example, the anatomy of a specification is related to the type of test materials that will be built, hence there is interrelationship between this document and the Test Guidelines. The reader is strongly encouraged to be familiar with the other documents in the family.
@@Links between applicable guidelines in this document and the other Framework documents will be given [@@Ed note. this has never been done -- delete this?].
The Framework as a whole is intended for all Working Groups, as well as developers of conformance materials for W3C specifications. [@@Ed note. where to put following? delete it? goals/target audience?] Not only are the Working Groups the consumers of these guidelines, they are also key contributors. The guidelines capture the experiences, good practices, activities, and lessons learned of the Working Groups and present them in a comprehensive, cohesive set of documents for all to use and benefit from. The objective is to reuse what works rather than reinvent and to foster consistency across the various Working Group quality activities and deliverables.
This document does not preclude the need to apply the W3C Manual of Style [STYLE-MAN] and to conform to the Publication Rules [PUBRULES ] (member-only). It is intended to complement those specifications.
This document employs the WAI (Web Accessibility Initiative) model of organizing guidelines and verifiable checkpoints. See, for example, Web Content Accessibility Guidelines 1.0 [WCAG10]. Each guideline includes:
The checkpoints in each guideline define the processes and operations needed to fulfull the purpose of the guideline. Each checkpoint includes:
Each checkpoint is intended to be specific enough so that someone can implement the checkpoint as well as verify that the checkpoint has been satisfied. A checkpoint will contain at least one, and may contain multiple individual normative requirements, that use RFC2119 normative keywords. See the "Conformance" chapter for further discussion of requirements and test assertions.
A separate appendix to this document [SPEC-CHECKLIST] presents all checkpoints in a tabular form, for convenient reference. This is an Implementation Conformance Statement (ICS) pro-forma for this specification. (See GL12.)
The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY ", and "OPTIONAL" will be used as defined in RFC 2119 [RFC2119]. When used with the normative RFC2119 meanings, they will be all upper case. Occurrences of these words in lower case comprise normal prose usage, with no normative implications.
Unusual terms in these framework documents are defined when first used. This document contains a "Definitions" chapter. Generally useful QA-specific terms will eventually be in the QA Glossary [QA-GLOSSARY]. If terms are in the QA Glossary, their definition herein will refer to that QA Glossary entry with a link. [@@Ed note. Following represents the current resolution of the issue, whose reversal has been requested with a "re-open" request.] The definitions herein may supplement or build on that generic definition with other information that is useful or helpful in the specification guidelines context. They will not contradict the generic definitions.
Some checkpoints are more critical than others for producing a high quality, testable standard that is a sound basis for successfully interoperable products. Therefore each checkpoint is assigned a priority level based on the checkpoint's impact on the quality of the specifications produced by the Working Groups.
There are 15 guidelines (@@14@@):
The guidelines are of two general types:
[@@Ed note. Don't know where this should go, if anywhere. Keep somewhere? Delete?] It is necessary to clearly distinguish between the specification -- the W3C Technical Report -- and the technology that it describes. For example, in Guideline 3, profiling is a method to partition the technology into groups, and profiles are technology groups. Profiling and profiles of the technology may or may not lead to division of the specification into multiple parts or units, concurrent or subsequent. This document will strive to keep the distinction clear, even though in common usage concepts such as profiles typically are used to refer to both.
Each of the latter set of eight guidelines GL2 through GL9 addresses a way in which the conformance policy of a specification might allow variation amongst conforming implementations. For example, a specification might allow implementations to choose between one of two well defined behaviors for a given functionality, thus two conforming implementations might vary on that aspect.
For this reason, these eight guidelines are collectively called the "dimensions of variability (DoV)". The eight dimensions of variability recognized by this document are:
The above are not necessarily all orthogonal to one another. There are many possible associations, dependencies, and interrelationships. As a general policy, these specification guidelines do not attempt to legislate correct or proper relationships amongst the DoV. Rather, they try to clarify the nature of each dimension, and require specification to make deliberate and well documented choices. Some discussion of possible interrelationships, including examples will be found in the Specification Examples & Techniques. [@@Ed note. I don't know how much more to say here. Simple concept. If we want to go into it any more deeply, a 'h2' "DoV Concepts" section is in order. Right now, we define each dimension initially within its GL.]
The dimensions of variability are one of the principal concepts in this guidelines document to help organize, classify and assess the conformance characteristics of W3C specifications. The eight DoV get special attention because, since they are at the core of the definition of a specification's conformance policy, there is significant potential for negative interoperability impacts if they are handled carelessly or without careful deliberation.
As a general principle, variability complicates interoperability. In theory, interoperability is best when there are numerous identical, complete, correct implementations. However, in practice, the net effect of conformance variability is not necessarily always negative, when compared to the alternatives. For example profiles -- subdivisions of the technology targeted at specific applications communities -- introduce variability amongst implementations. Some will implement Profile ABC, some will implement Profile XYZ, and the two might not intercommunicate well if ABC and XYZ are fairly different. However, if ABC and XYZ are subsets of a large monolithic specification -- too large for many implementors to tackle in toto -- and if they are well targeted at actual application sectors, then subdivision by profiles may actually enhance interoperability.
Different sorts of variability have different negative and positive impacts. The principal danger is "excessive" variability -- variability which goes beyond that needed for positive interoperability tradeoffs, and which unnecessarily complicates the conformance landscape. Specification writers need to carefully consider and justify any conformance variability allowed, do so by reference to the project requirements and use cases, and explicitly document the choices made.
[@@Ed note. That is enough for general introduction to DoV. I'm not sure where we go from here -- a detailed DoV concepts section? More stuff just in the GL (definitions) and ET? ...?]
[@@Ed note. Someone needs to draft some use cases.]
U
This section defines conformance of Working Group specifications -- i.e., technical reports -- to the requirements of this QA Framework guidelines specification. The requirements of this guidelines specification are detailed in the checkpoints of the preceding "Guidelines" chapter, and apply to the technical reports produced by Working Groups.
The test assertions of this Specification Guidelines document are found in the prioritized checkpoints. A checkpoint will contain at least one, and may contain multiple individual requirements. These requirements are the test assertions of this specification. A checkpoint is satisfied by satisfying all of the individual requirements. Failing one individual requirement means that the checkpoint is not satisfied.
This section defines three degrees of conformance to this guidelines specification:
A specification conforms to the QA Framework: Specification Guidelines at degree X (A, AA, or AAA) if the Working Group meets at least all degree X conformance requirements.
To make an assertion about conformance to this document, specify:
Example:
This specification conforms to W3C's QA Framework: Specification Guidelines, available at http://www.w3.org/TR/qaframe-spec/, AA-Conforming.
The checkpoints of this guidelines specification present verifiable conformance requirements about the specifications (technical reports) that Working Groups produce. As with any verifiable test requirements, it is also true of these specification requirements that:
[@@Ed note. The entire former "Motivations" section is here right now, including the extensive discussion of granular grammars.]
The ultimate goal of a specification is to be implemented. In order to be implemented, a specification needs to be written such that developers and users can comprehend what is expected of them and build correct, robust, interoperable software and content. Correct utilization of specifications leads to portability and interoperability. In particular, it is easier to implement clear specifications, where the possibility of misinterpretation has been eliminated, safeguarding the quality of implementations. In addition, if care is given to the specifications, interoperability between W3C technologies is easier to track and assert.
Given that the W3C invests time and resources into producing specifications that are eventually implemented, and especially given that some of those specifications are interrelated, it makes sense to enhance and streamline the specification authoring process. This should be done for three main reasons:
The W3C needs to ensure that the deliverables of its WGs are clear, implementable and sound technical specifications with testable requirements and that can be used to easily generate Quality Assurance materials (tests and suites). A specification is testable if there exists some finite cost-effective process with which a person or software can check that the requirement has been met.
The benefits of producing clear, testable specifications include:
The benefits derived by writing granular specifications that provide detailed control over the information in the specification, include
In particular, information from the specifications is easier to extract if specification authoring involves a standardized set of specification schemata where both technology and intended behavior is specified. Tests can be generated directly from the specifications, where needed, to generate primers for test suites. In this way, a fairly broad coverage of test cases can be achieved in a straightforward way. An added bonus is that a graphical representation of the specification can be generated in parallel to the normative HTML version. Specification granularity allows for more concise reporting when running test suites by pointing directly to the relevant part of the specification. Specification granularity allows for generating tests that can be validated against the specification itself, thus making it easier in those cases where different interpretations of the normative text are given.
Use of a formal (or formal-like) specification language may enable 'validation' of the specification as well as auto generation of tests or test components.
Synchronizing writing specifications with building test materials provides a feedback loop. Testing can help find ambiguities, inconsistencies, and holes in a specification.
In particular
Interaction between WGs and other bodies in specification authoring is related to test suite production and maintenance. The reasons to develop specifications, test assertions, and test materials together are:
achieve confidence in the resulting test materials as a measure of conformance.