QA Specification Guidelines draft

A lot of effort goes into writing a good specification. It takes more than knowledge of the technology to make a specification precise, implementable and testable. It takes planning, organization, and foresight about the technnology and how it will be implemented and used. The goal of this document is to help W3C editors write better specifications, by making a specification easier to interpret without ambiguity and clearer as to what is required in order to conform. It focuses on how to define and specify conformance for a specification. Additionally, it addresses how a specification might allow variation among conforming implementations. The document is presented as a set of guidelines or principles, supplemented with good practices, examples, and techniques.

Status of This Document

This is a QAWG Editor's development draft. It has no status or standing, and does not necessarily represent QAWG consensus. Following is the proposed SoTD for the first publication of this "SpecLite" revision of Spec Guidelines...

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

This document is a W3C Working Draft (WD), made available by the W3C Quality Assurance (QA) Activity for discussion by W3C members and other interested parties. For more information about the QA Activity, please see the QA Activity statement.

This is the first publihed version of this document after the decision to completely redesign the QA Framework (QAF), resolved by the QA Working Group (QAWG) at its 2004 Technical Plenary face-to-face. It is a lighter-weight, less authoritarian, more user-friendly and useful version of the previously published Candidate Recommendation version of the Specification Guidelines. Because of the extensive nature of the changes, it has been returned to Working Draft.

This draft accurately reflects the intended overall structure and content of the new, lightweight Specification Guidelines. However it is incomplete in some details (generally indicated by "@@" or "TBD"), especially in the examples -- lots more examples are intended. Also, some sections more resemble outlines than completed prose. Because this is a significant departure from the last-published Specification Guidelines, QAWG wants review and comment on the overall direction as soon as possible. Incomplete details will be fleshed out in the next publication.

This Specification Guidelines document will be coordinated with the other parts of the QA Framework, The QA Handbook and QA Framework: Test Guidelines. Synchronization at uniform level of maturity will occur no later than Last Call. The first publications of the various parts will occur somewhat more independently.

Publication as a Working Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

Comments on this document may be emailed to www-qa@w3.org, the publicly archived list of the QA Interest Group (QAIG). Commenters please note that your comments will be publicly archived and available, do not send information that should not be distributed, such as private data. If you are ab le to give your feedback by 12 June 2004, the QAWG can consider it at its next face-to-face meeting.

Introduction

Scope and Goals

This document is a guide for W3C editors and authors. It provides guidelines for writing better specifications. This document differs from other W3C process and publication related documents in that it looks at specifications from a conformance viewpoint. It address the most basic and critical topics with respect to conformance, including how to convey what is required in order to conform and how to allow variation among conforming impelmentations. The goal is to make specifications precise, easier to interpret without ambiguity or confusion, and clearer as to what is required in order to conform. Good specifications lead to better implementations and foster the develoment of test suites and tools. Conforming implemenations lead to interoperablity.

Motivation

Everyone benefits from having well-written specifications. Editors may have less rework and thus, fewer errata. Implementers can implement sooner and have a better chance to conform to the specification. Test developers are able to derive unambigious requirements. Users benefit by having interoperable solutions. W3C gains by having recommendations produced with higher quality and reduced maintenance.

Why SpecLite?

It is not an easy task to write accurate, clear, complete, unbiased specifications. It requires planning, organization, and foresight about the technology, how it will be implemented and used, and how technology decisions affect conformance. This document provides a collection of principles, good practices, examples, and techniques that leads the reader though the decisions that need to be made in order to write precise requirements and establish, define, and specify conformance for specifications. Realizing that editors and authors are busy, under pressure to get the specification published, and already have a reading list of W3C documents (a good place to start, Editor's home page), this document can be used as a checklist or reminder of things to consider, a how-to guide with examples and techniques which can be adapted, a reference for understanding conformance concepts and terminology, or all of these.

About this document

This document is a practical guide to writing a specification, presenting editors with topics to consider. Consequently, much of the document is non-normative recommendations labeled Good Practices. There are however, a limited number of normative requirements, labeledPrinciples, that must be implemented in order to conform to this document. The overall objective of these principles and good practices is to facilitate the creation of a conformance section in every specification. A conformance template (@@need to create the template and link to it) is provided to assist editors satisfy the requirements of this document and end up with a conformance section. Note that for some specifications (e.g., QAH, WebArch) where conformance is not an issues (e.g., no normative content), the conformance section may not be a unique section but rather a set of statements contained in another section.

The topics presented are inclusive (self-contained) providing information needed to understand and successfuly apply the principle, although related information and advanced topics may be referenced.

If in a hurry - start at 1st section, Specifying confomrance -- this is the expected outcome of adhering to this document. It serves as a roadmap to other parts of this document, which help you achieve Secifying Conformance.

Audience of this document

The primary audience of this document is editors and authors, however, it is applicable to a broader audience including:

This document makes no distinction between the terms editors and authors and will be refered them collectively as editors.

Structure of this document

Other QA Framework documents

This document can be used as a stand alone document or as part of a family of QA Framework documents designed to help the WGs improve all aspects of their quality practices. Relationship to QAH and TestLite.

Guidelines

A. Specifying Conformance

(@@ this section could actually be at the end of the document. I like it here, since it provides a roadmap of things to come. Also, if someone is in a hurry, this is what you want them to see and do)

It is important to convey an understanding of what is meant by conformance and how it applies.

A.1. Include a conformance section

The conformance section of a specification is a high-level description of what is required of implementers and application developers. It, in turn, refers to other parts of the standard. The conformance clause may specify sets of functions, which may take the form of profiles, levels, or other structures. The conformance clause may specify minimal requirements for certain functions and minimal requirements for implementation-dependent values. Additionally it may specify the permissibility of extensions, options, and alternative approaches and how they are to be handled

A.2 Specify how to make conformance claims

B. The Nature of the specification

B.1. Scope

The path to a quality specification begins with its scope. It is critical to convey what the specification is about by describing its intent and applicability. As the specification develops, its a good idea to revisit the scope to make sure it still reflects the intent of specification or if it needs to be modified.

Principle: Define the scope

Describe what the specification is about.

In addition to containing the subject of the specification, a more complete scope contains:

Note that it is not necessary to write a separate statement for each of these items, rather, they can be combined.

Why care? It helps to keep the specification content focused. It also will help readers know the limits or boundaries of the specification and whether it is of interest to them.

Good Practice: Be direct, providing simple, understandable statements of facts.

Don't specify requirements

Good Practice: Use examples or use cases to illustrate

Providing examples, use cases or both helps to clarify the scope for the reader.

B.2 What needs to conform

@@Here are different ways to say basically the same thing.

What is the target of the spec what will implement it
What will be the object of the conformance claim

B.3 Make a list of normative (and non-normative) references

C. Piece needed to Specify Conformance

C.1 Define your terms

Examples: valid, well-formed, foo-conformant, document conformance (CC/PP) consumer conformance (CC/PP)

Good Practice: being consistent and using the same term when you mean the same thing don't be a thesaurus.

C.2 Identify normative and informative parts

C.3 What is mandatory

Indicate what requirements are mandatory as opposed to optional, recommended, good ideas, etc.

Expressions that conveys the criteria to be fulfilled in an implementation of the specification

D. Managing Variability

D.1 Subdivide

@@ not sure what to do with these questions, they lead you to doing the right thing.

D.2 Optionality and options

Options in specifications provide implementations the freedom to make choices about

These choices also called dscretionary items, give implementers of the technology the opportunity to decide from alternatives when building their applications and tools.They describe or allow optionality of behavior, functionality, parameter values, error handling, etc. They may be considered necessary because of environmental conditions (e.g., hardware limitations or software configuration), locality differences (e.g., language or time zones), dependencies on other technologies, or the need for flexibility.

Although there are perceived benefits to providing options, there is also a downside: options increase the variations that can exist among implementations. The greatest way to undo the utility of a specification is with too many options. Bascially, different implementations may specify different combinations of options. This makes comparisons between implementations difficult as well as complicates conformance testing and increases the chance of non-interoperable implementations.

Good Practice: Determine the need for each option

Make sure there is a real need for the option.

Examine the reason for the option - is it to address a real, existing need? Should it really be an option or made a mandatory part of the specification? Be careful not to provide options in anticipation for something that sounds like a good idea but will probably never be realized - ask the implementers if they ever plan to need this. As the specification progresses, if the option is not implemented, consider removing it.

Technique: Develop a set of tests and analyze the results to see what is implemented and how it is implemented.

Example: Tests were used to determine if options were needed. As part of its exit criteria for Candidate Recommendation, a Working Group created a set of tests to 'test the specification'. The tests were able to show where there was need for optionality (e.g., diversity among implementations and flexibility justified) and where it was possible to narrow the choices (e.g., implementations used a much narrower set of values than what was permitted).

Good Practice: Indicate any limitations or constraints

Provide as much information as possible to narrow the allowable choices and to increase predictability.

For implementation dependent values or features, if possible, provide a range or set of permitted values rather than leaving it completely open.

For options, especially enumerated lists, make sure that the number of choices/options that can be implemented is known. Specifically, can zero, exactly one, or several of the allowable choices be implemented? Does this number depend on other parts of the specification or other chosen options?

Examples of constraints include mandating that an implemenation implement only one choice, implement every choice (allowing the user to decide), be allowed to implement none of the choices, or be required to implement a specified values and as many additional values as they wish from an open-ended list.

Good Practice: Address the impact of the option

Think about the implications of both implementing the option and of not implementing the option

Discuss the implications of either using the option or not. It may be helpful to provide rationale for choosing one option over another or for not using the option at all. Consider the unintended consequence of the option - on other options, other parts of the specification, on other specifications.

Good Practice: Make options easy to find

Use tags to label options.

Why care?: To make it easier for implementers, test creaters and users to discover the choices and variability allowed. For test creaters, having a name for each item and a link target for each one, allows systematic reference to individual items from test cases as well as elsewhere in the specificaton.

D.3 Extensibility and Extenstions

To accomodate changes in technology and information on the Web, a specification can be designed for extensibility. A specification is extensible when it provides a mechansim to allow any party to create extensions. Extensions incorporate additional features beyond what is defined in the specification. However, extensions can compromise interoperabiltiy if there are too many differences between implementations. The impact of extensions can be mitigated through features specifically designed to allow new functionality. These features provide a 'standard way to be non-standard' by including hooks, conformance rules, or other mechanisms by which new functionality may be added in a conforming way.

Principle: Deal with the likelihood of extensions

Consider whether some parts of the specification will benefit from extensibility. If so, define a mechanism to allow for the extension.

Good Practice: State the conditions under which extensions are prohibitted.

This doesn't have to be complex, it may be a simple statement (e.g., extensions are not allowed). The statement is often associated with the conformance clause.

Why care?Extensions increase variability between implementations. Defining a mechanism will help to ensure extensions are defined in a consistent manner leading to predictable handling of extensions, including the ability to take appropriate actions (e.g., do the extension, ignore, or take a fall-back behavior). By planning for extensions and designing a specification for extensibility, there is less chance that extensions will interfere with conformance to the base specification and a better chance at achieving interoperability. On the other hand, there may be areas in a specification that would not benefit from extensibility and extensions are strictly forbidden (e.g., permissible characters in XML 1.0 names for elements and attributes, most of the built-in datatypes in Schema Part 2). This is called strict conformance. Strict conformance is defined as conformance of an implementation that employs only the requirements and/or functionality defined in the specification and no more.

Technique: Technology and application needs will dictate the best strategy for enabling extensibility.

Many specifications include extensibility mechanism along with their technology (XSLT, WSDL, SOAP, SVG). Whereas other specifications are predominately about the extensibility mechanisms (e.g., Componete Extension (CX) API, HTTP Extension Framework) from which specifications are based (e.g., CC/PP Exchange Protocol).

When designing for extensibility, it can get complicated. Questions to consider that can impact design decisions include, but definately not limited to:

Example: Mechanism defined within the specification, extension indicator, error handling instructions

XSLT 1.0 provides mechanisms that allow an XSLT stylesheet to determine whether the XSLT processor by which it is being processed has implementations of particular extensions available, and to specify what should happen if those extensions are not available. It defines two boolean functions: function-available(QName) and element-available(QName) that must be present in every impelmentation. These functions signal that there is an extension, proivde for the XSLT processor to indicate if the extension is not available by returning a value of false, and if not available provides handling instructions (e.g., signal an error, perform fallback and not signal error).

WSDL 2.0 defines binding extension elements which are used to provide information specific to a particular binding. It is a two-part extensibility model based on namespace-qualified elements and attributes. It provides the syntax and semantics for signaling extensions. Extension elements can be marked as mandatory, meaning that they MUST be processed correctly by the WSDL processor - i.e., either agree to fully abide by all the rules and semantics signaled by the extention or immediate cease processing (fault). @@ someone should check this, does it make sense?

CC/PP Exchange protocol using HTTP Extension Framework defines a HTTP extension to exchange CC/PP descriptions effectively. The CC/PP exchange protocol conforms to HTTP/1.1 and is a generic extension mechamism for HTTP/1.1 designed to interoperate with existing HTTP applications. The CC/PP exchange protocol uses an extension declaration to indicate that an extension has been applied to a message and possibly to reserve a part of the header namespace. It provides rules for which of the HTTP Extension Framework extension declaration strengths and extension delcaration scopes to use and defines the syntax and semantics of the header fields.(@@ someone should check that I captured this correctly)

OWL is a vocabulary extension of RDF. OWL imposes additional semantic conditions on RDF called semantic extensions of RDF. These semantic extensions confom to the semantic conditions for simple interpretations described in the RDF Semantics Recommendation. The OWL semantics is consistent with RDF semantics, but OWL when understood as RDF is 'incomplete' versus the same OWL when understood as OWL. Thus, by understanding OWL more is learned and nothing that is learned contradicts what is learnt by RDF alone.

XHTML Modularization defines a set of conformance rules to allow for the extension of XHTML's layout and presentation capabiltiies. This method uses the extensibility of XML without breaking the XHTML standard. It defines the extension by using XML DTD or Schema which is uniquely identifiable, does not redefine behavior of the technology it extends, and the syntactic and semantic requirements are described and documented. Additionally, it cn have some organization and naming constraints defined by the technology it extends.

Principle: Prevent extensions from breaking conformance

Extensions must not contradict or negate conformance to the specification. If it conformed without the extension, conformance should hold true with the extension.

Example: in XSLT 1.0, extension attributes (from other namespaces) can be present on the official XSLT elements, but they are prohibited from changing the specified behavior within the detectability of conforming behavior. Thus, an extension attribute can cause the element to perform faster but cannot change the result.

Good Practice: Define error handling for unknown extensions (@@is this only applicable to agents?)

When defining extensibility, include error handling instructions for when an extension is not available or understood.

A good way to handle these two approaches is to have a way in the syntax to distinguish which behavior is expected, e.g., mustUnderstand attribute in Soap 1.2

Example: a hypothetical XHTML specification could suggest that user agents, when they come across an unknown extension, take some fall-back behavior, but recognizse the input as 'legal' rather than 'illigal'.

Applicability: Extensible specifications such as languages and protocols. Unlikely candidates include: content specifications (?) @@remore or keep Applicability?

D.4 Deprecation

The need for deprecation comes about when there are features (e.g., function argument, element or attribute) defined in the specification that are no longer viable or have become outdated by newer constructs. These deprecated features should be avoided and may become obsolete and no longer defined in future versions of the specification.

Principle 1: Identify each deprecated feature

If there are deprecated features from a previous specification version, let the reader know about them.

Why care? It provides the user and implementer a 'user beware' warning, letting them know that use of the feature is discouraged.

It helps the reader find the deprecated features when presented as a collection in one place rather than distributed throughout the document.

Technique: Provide a list of the features or a list of links to the features. Make this list easty to find. Make it part of the Conformance Section of the document or put it in an Appendix.

Example: HTML 4.01, Appendix A: Changes includes a section called "Deprecated elements" which provides a linked list to the elements.

Let the reader know that there are deprecated features in the specification. Don't blend the discussion of a deprecated feature into the specification so that it appears as just another feature, without a means to easily find it.

Example: Avoid doing. Section 2.2 .2 Use of IRIs as Namespace Names in Namespaces XML 1.1 discusses the deprecation of relative IRI references. Difficult to find, luckily this is a very short Rec.

Principle 2: Specify the degree of support required for each deprecated feature and its conformance consequence.

Consider the effect of deprecation on all classes of products that implement the specification (e.g., authoring tools, user agents).

Why care? Deprecating a feature has a direct affect on implementations - e.g., disallow use of the feature or continue to support it for backward compatibility reasons.

For the purposes of backward compatibility, it may be necessary to specify different requirements of support for each class of product - e.g., prohibit its use in authoring tools, but require continued support by user agents.

Technique: Define what it means for a feature to be deprecated. Do this for each class of product. Also clarify the relation between the deprecated feature and Foo-conformance.

Example: MathML2.0 section 7.2.1.2 describes deprecated MathML 1.x features in terms of MathML-output-conformant authoring tools, MathML-input-confomrant rendering/reading tools, and MathML-roundtrip-conrormant processors.

Good Practice: Indicate workarounds.

Explain and give examples of how to avoid the deprecated feature

Examples are helpful in providing alternatives or better ways to get the same results. Showing what can be done in place of the deprecated feature will help get implementers and users to discontinue use of the feature. Additional information to provide insight into why the deprecation is also helpful.

Example: Namespaces XML 1.1 deprecation of IRI references includes a link to the deprecation ballot results, providing background information on the proposal to deprecate, what this means with respect to conformance to XML 1.0 and Namespaces as well as the affect on other specifications (i.e., DOM, XPath).

Good Practice: Identify obsolete features

For historical purposes, a list of obsolete features is helpful. Providing an alternative is also helpful. This is usually done as an informative list.

Technique: List obsolete features in the Change section or separate appendix of a specification.

Example: HTML 4.01, Appendix A: Changes lists the elements that are obsolete and suggests an alternative element for use. "The following elements are obsolete: LISTING, PLAINTEXT, and XMP. For all of them, authors should use the PRE element instead."

D5. Error Handling?

E. Do Quality Control

As all editors know, the work is not finished when the writing is completed. In fact, various review and checks of all W3C documents are required prior to their publication and advancement. There are many things that can be done to improve the quality and clarity of the document, aside from the W3C process and publication rules and before Last Call reviews. Many of these things can be an integrated part of the specification development. Ensuring a quality document prior to external reviews can save time and energy in that the Working Group may get fewer comments and issues to resolve.[@@ this seems a bit pedantic?)

Example: One Working Group's document entered CR prior to a through quality review, resulting in a huge number of issues to resolve and the eventual retreat back to Working Draft for major revisions. Many of the comments could have been prevented, such as: inconsistency and incompleteness of the document (e.g., links to supplementary materials which did not exist or were not complete, overlapping and conflicting requirements), undefined terminology, and unimplementable requirements.

Example: When defining a language (such as XHTML) some requirements are defined in a natural language (e.g., English) while others are defined using a formal specification. Often there is overlap between the two approaches - especially with XML Languages where a DTD or an XML Schema can be used for two purposes: to define the syntactic requirements set by the language and/or to allow validating documents conforming to the said language. This overlap can lead to contradictions between the prose and the formal language, i.e., the prose says more than the formal language, formal language is more constraining than the prose, or the formal language says something different than the prose.

Good Practice: Check everything first.

Do a systematic and thorough review. Don't forget to examine related specifications and tools where correlation and consistency is essential.

Don't rush to publication or publish too soon. First make sure that everything is finished and ready to go. This is obvious, but often forgotten in the desire to meet deliverable dates.

Checking everything seems simple, espeically if you have a monolithic specification. But with the modularization of specifications as well as interactions between specifications, there is a need to think globally. As an illustration, consider this situation: If you submit a web page address to the W3C MarkUp Validator, it could happen that the result page tells you that your document is "Valid XHTML 1.0 Strict" and provides a link to the XHTML 1.0 Recommendation. If you follow this link you will encounter a problem: There is no definition of what it means for a document to be identified as "Valid XHTML 1.0 Strict".

Technique: Do a walk-through, where experts do a detailed review of the content. Perhaps, establish a team, where each person focuses on a different aspect of the specification's correctness. For example, each person checks a different part of the specification or do it by expertise - the grammarian checks for consistency, grammar, spelling, readability, the test builder checks requirements for precision and implementability, the conformant checks that conformance criteria exists.

Good Practice: Use validation techniques during specification development @@need a better word than validation.

Don't wait until the writing is done, start early by integrating techiques into the writing process.

Why? Catching problems as you go saves time and effort later on. It helps to mimize problems such as:

Technique: Write sample code or tests. A test driven specification can provide an understanding of a concept as well as help enhance clarity and usability as well as keep the focus. It provides an illustration of correct usage. It can be used to facilitate issue resolution as well as deployment. An added benefit is that it provides a jump-start on testing efforts, including conformance testing and providing tests to implementers as they build their software.

Example: WebOnt or RDF experience - writing tests to drive the specification. OWL Test Cases

Example: CSS editors are encouraged (required?) to provide tests with all new content.

Test assertions facilitate the development of consistent, complete specifications and promote the early development of conformance tests. This exercise helps to uncover incosistencies, ambiguities, gaps, and non-testable statements in the specification. It can provide early feedback to the editors regarding areas that need attention. An added benefit is that the assertions can be used as input to test development efforts.

Example: When trying to write test assertions for all its conformance requirements, one Working Group found that informative information was needed to complete the normative requirement. Consequently, the conformance requirements were rewritten.

Technique: use formal languages. @@ Leave here or as Advance Topic? Either way, we need prose, examples, experiences, etc. Or it needs to be removed.

It is possible using formal languages (e.g., XML Schema, UML) to check for correctness of the specification and that requirements are specified clearly and unambigiously. The checking can be done via automated tools.

EXamples? For formal language specifications - test the formal language. (wiki FormalLanguageVsProse)

Conformance to this document

@@ need to figure this out. Much of this document is non-normative - the goal is to get people to think about these aspects when writing a specification, in some cases there isn't anything else to do other than to be aware of the topic. For topics that don't apply are won't be part of the specification, it would be silly to require a statement in their specification that they don't do something - although there are some topics that you do want to do this. Thus, it would be difficult to have a meaningful test of SpecLite Principles - basically it is a checklist asking if they read the Principle and considered it. What is key, is that there is a conformance clause containing at least a minimal set of information (we need to determine this).

Do this by -- completing the conformance template (which needs to be developed) would signify conformance to this document

Having a conformance section in the specification that addresses all the Principles in this document, would signify conformance

QA Specification Guidelines

W3C Editor's Draft 17 May 2004

Abstract