Disposition of Comments for QA Framework: Specification Guidelines second Last Call Working Draft

As a separate example, any specification that defines the behavior of a class of product that creates XML should address creation of XML 1.1 and anticipate future XML versions. In XSLT, creation of XML is specified by <xsl:output method="xml" version="1.0" />, and each version of XSLT defines the allowable range of values for the version attribute. Another option is to reference the XML Infoset - for instance, XML Inclusions are compatible both with XML 1.0 and XML 1.1 since they reference normatively the XML Infoset, which is the same for the two versions of XML.

In its section Referencing the Unicode Standard and ISO/IEC 10646, the specification Character Model for the World Wide Web 1.0: Fundamentals [CHARMOD] gives a detailed indications for referencing Unicode Standard and ISO/IEC 10646. Specification editors are encouraged to follow these recommendations.

The QA Working Group agrees that for deprecated, obsolete, optional features and subdivisions, it was not clear how a specification could match the requirements if none of these were present in the specification. To clarify this, the Good Practices and Requirements have been amended. For instance, the Identify deprecated features now reads in What does it mean?:

If the specified technology has already been published in a previous version of the specification, indicate the features from the previous version now deprecated or state in the conformance section that no features were deprecated.

Similarly for subdivisions, optional features and obsolete features

The QA Working Group has accepted the comment and decided to create a new section which is not mandatory : Beyond Conformance.

This section contains previous Good Practices which were workflow related and that a WG might want to explore during the specification editing. Some of the good practices have been rewritten as well to remove their workflow orientation when it was not necessary.

See also the correspondence table

On [date of the publication], this specification [name of the
specification], edited by [name of the publishing entity], explains in
section [link to where] why it does not need a conformance clause and
is thus conformant to Specification Guidelines WD, November 22, 2004

The QA Working Group has accepted the comment and has used the example provided:

On [date of the publication], this specification [name of the specification], edited by [name of the publishing entity], explains in section [link to where] why it does not need a conformance clause and is thus conformant to Specification Guidelines WD, November 22, 2004 published at http://www.w3.org/TR/.../.

the QA Working Group has accepted the comment and has accepted the rewording for the legend of the illustration: Profile X defines an implementation of Level 2 (and thus, implicitly, Level 1 also) for a specific class of products for example.

The QA Working Group agrees that the wording was too limitative, and has now reworded Good Practice 19 as follows:

Good Practice 19: Warn extension creators to create extensions that do not interfere with conformance

and complemented by the following text in the "what does it mean":

Extensions can be created in different contexts: directly by implementers, in other specifications, etc

The QA Working Group agrees the former text was narrowing the scope of the good practice without good reason and made the new text broader, while illustrating it with the former text. Good practice 23 now reads:

What does it mean? : For each class of product affected by an error condition, address error handling. For instance: for a language, address what effect an error (be it syntactic or semantic) in the input has to a processor of this language; for a protocol, address how a party to this protocol should behave when a bogus message is received; for an A.P.I., indicate what exceptions are raised.

The QA Working Group agrees that being more specific about the story can only helps understand the point made and is now making an explicit reference to itself in the text:

In 2004, QA Working Group documents entered Candidate Recommendation prior to a thorough quality review, resulting in a huge number of issues to resolve and the eventual retreat back to Working Draft for major revisions.

(while the story doesn't exactly match the reality of what happened, the QA Working Group believes it can still be used as a fair illustration of the point made)

The QA Working Group has accepted the comment and has reworded as follows:

QA Specification Guidelines: For editing each Good Practice and Requirement of QA Specification Guidelines, a template and a method were developed to make the edition of each part easy to follow and discuss by the QA Working Group and easy to incorporate in the final document for the editor.

The QA Working Group has accepted the comment and has added verbiage to the new version of the ICS with:

• a comments column,
• absolute links to the spec,
• asking users to link back to the relevant spec parts.:

  The presentation is intended to be convenient for organizers and evaluators of QA projects in W3C Working Groups, to facilitate assessing specifications against the checkpoints. The table includes spaces for scoring each checkpoint, "yes" (satisfied), "no" (not satisfied), "n/a" (not applicable), and comments.

  Users of this ICS are encouraged to put links in the comments column back to the reviewed specification parts illustrating how it conforms to the Specification Guidelines.

The QA Working Group has completed a SpecGL ICS for SpecGL itself.

We believe SpecGL now conforms to itself, since it implements all the Requirements; it also implements all but one Good Practice. Follow belows details answers to the comments made in the footnotes of the ICS the TAG filled for the previous version of SpecGL.

1) No link to Class of Products statement

We believe linking to the section including the statement is sufficient

2) ICS required for claiming conformance?

We have clarified that "if all the Requirements are checked on the ICS as being satisfied, then conformance can be claimed as detailed below", linking to the conformance claim section, which requires linking to a completed ICS.

3) Class of Product vs Scope

The scope is what is covered by the specification; the class of products are part of defining the scope, but a more narrow (and in our opinion important) aspect of it that directly affects conformance.

4) "Using the defined term conformance model..."

Done

5) Checking whether references have been reviewed carefully

The good practice has been reworded to be less workflow oriented

6) Positive statements for no subdivision

SpecGL clarifies now that a positive statement is indeed needed, and indeed states that it is not subdivided.

7) No extensibility mechanism

We disagree that SpecGL doesn't define an extensibility mechansim. In section 4.3 (Extensibility), we say that one can extend SpecGL as long as it doesn't negate the requirements given in the specification. This is a basic extensibility mechanism, and we don't think at this time there is any need for a more complex one.

8) No deprecated features listed

SpecGL clarifies that making a positive statement about deprecated features is only needed when there was a previous version of the specification (which is not the case for SpecGL itself)

9) No obsolete features identified

Same as 8

10) & 11) Hard to demonstrate some GP

The QA WG has moved the given GP into a separate informative section of SpecGL, since they were indeed too workflow-oriented to fit with the other Good Practices.

As a response to the proposal, the QA Working Group decided to include in its Good Practice regarding extensibility mechanisms a technique on this topic:

When designing for extensibility, it can get complicated. Points to consider that can affect design decisions include, but are definitely not limited to, the following topics [...]

• Avoid "untested hooks": if an extensibility mechanism is defined, make sure it is well tested during the implementation phase; experience has shown that untested extensibility just doesn't work

Conformance is the fulfillment of a product, process, or service of specified requirements. These requirements are detailed in a specification as part of a conformance clause and in the body of the specification. A conformance clause is the section of a specification that identifies all the criteria that must be satisfied in order to claim conformance to the specification.

A clear presentation of conformance is crucial to successful interoperability of implementations. The conformance model and the language used for normative information determine the testability of a specification. They also influence the overall implementation landscape, ranging from a narrow conformance with few allowable variations in implementations to multiple conformance types, resulting in numerous conforming implementations. The model must be chosen carefully, to produce the intended implementation range.

The QA Working Group has accepted the comment. Rather than attempting to create our own definition of "specification" it has decided to use the definition provided by ISO Guide 2-4:

The term specification is used as defined in ISO Guide 2-4 as meaning a document that prescribes requirements to be fulfilled by a product, process or service.

The QA Working Group has accepted the comment. The notion of umbrella spec has been removed from SpecGL, moved to the Variability in Specifications document, and alluded to in the Scope and Goals section of the Specification Guidelines. The relevant section now reads:

The term specification is used as defined in ISO Guide 2-4 [ISO-GUIDE] as meaning a document that prescribes requirements to be fulfilled by a product, process or service. Specifications can be defined in one document or as a coherent set of several documents, and can import requirements of other specifications with normative references.

The QA Working Group agreed that the terminology was imprecise, and substituted the term "technical reports" to describe all the documents that go through the W3C process, normative or not.

The QA WG disagrees with the assessment that such an explanation is often unnecessary. We believe that based on past experience there is often a significant amount of confusion about whether or not a technical report is normative and consequently whether or not one could conform to it. We therefore feel that the simplest approach is always to require an explanation of whether or not conformance is an issue. We did agree with you that it is not necessary to require an actual conformance section in such documents.

Our revised text reads:

Note that for some technical reports (e.g., The QA Handbook [QA-HANDBOOK], Architecture of the World Wide Web, Volume One [WEB-ARCH]) where conformance is not an issue (e.g., no normative content), the conformance clause may be an explanation of why there is no 'conformance to this document' and may be presented in another section rather than in a separate conformance section.

Good Practice 04, Provide an Implementation Conformance Statement Pro Forma

What does it mean? An Implementation Conformance Statement (ICS) provides information about an implementation to a specification, by presenting in a uniform manner the implemented capabilities (e.g., functions, features) and options as well as limitations of the implementation. An ICS pro forma typically takes the form of a blank questionnaire or checklist for an implementation. It provides the implementer a way to indicate the features implemented. Think of it as an inventory of what has been implemented. Note that a completed ICS does not indicate conformance of the implementation. Hence, answering "yes" to indicate a capability is supported does not mean that the capability has been tested.

This Good Practice suggests that the specification itself include an ICS pro forma. Providing this pro forma makes it conducive to completing and helps to ensure consistency among completed ICS.

Why care? An ICS pro forma provides a concise summary of a specification, i.e., the capabilities and options defined in the specification as well as any defined subdivisions (e.g., profiles, modules) and conformance designations. The ICS provided with the specification is blank, waiting for the implementer to complete. This blank ICS provides implementers and users a quick overview of features defined in the specification. A completed ICS not only provides information on what has been implemented (mandatory and optional features), but can also be used to document the presence of extensions or any specializations that have been made. A completed ICS provides information useful to facilitate the selection of applicable tests for the particular implementation. However, that is not all. Although the ICS content is independent of testing, associating it with conformance tests makes it an essential piece in the reporting of conformance results (see techniques in Good Practice 05).

Changes were also made to related Good Practice 05

Good Practice 05: Require an ICS as part of valid conformance claims.

Why Care: Providing a completed ICS with the conformance claim might help customers and users to determine quickly the implemented capabilities as well as easily verify the level of support for individual requirements of the specifications. Combining the ICS with a conformance test suite, can strengthen the claim. Specifically, the ICS augmented with links to conformance tests, provides a very nice way to indicate not only what has been implemented, but also, what has been implemented correctly (i.e., conforms to the specification).

Technique:

Augment the ICS by providing links to the test suite, such that each feature has associated with it a test (or set of tests). Explain what it means to check Yes or No. Specifically, does Yes/No indicate that the implementation has the relevant feature and passes the applicable tests or does Yes/No only indicate that the feature is implemented. In the latter case, add an additional column, to indicate the result of executing the tests. To avoid confusion as to the role of ICS, we recommend adding an additional column.

The QA Working Group has accepted the comment and has reworded the affected section as recommended and it now reads

Requirement 02: Define Scope

Why Care: This is one of the first sections a reader reads, so it is important to capture their attention and make sure they understand what the specification is about. It helps to keep the specification content focused. It helps reviewers determine when the specification is over-stepping its mandate and then gives the possibility to revise the specification in development. It also helps readers know the limits or boundaries of the specification and whether it is of interest to them.

The QA Working Group has accepted the comment and has included the example you recommended and it now reads

Good Practice 06: Provide examples, use cases, and graphics.

HTML 4.01 [HTML401]: The HTML 4.01 specification, dsigned in a very educative way, has some very good examples.

The QA Working Group has accepted the comment and has reworded the affected section as recommended and it now reads:

Using RFC 2119 [RFC2119] Keywords (MUST, SHOULD, MAY, ...) makes it easy to spot conformance requirements. According to the RFC itself, they should be used only to establish interoperation [WIKI-RFC-KEYWORDS]. They are usually written with distinctive formatting, such as upper case or bold. It is a good idea to create a specific markup for them too. It will be easier to extract conformance requirements and better for accessibility (See The Manual of Style: RFC 2119 Key Words [MANUAL-STYLE]).

The QA Working Group has accepted your comment in principle but believe that the current wording discourages variability and that it isn't appropriate to get into technical details as to what is good or bad variability, but rather should be carefully evaluated by each Working Groups (as expressed by when warranted).

Good Practice 13: Create subdivisions of the technology when warranted.

What does it mean? It may make sense to subdivide the technology into related groups of functionality to target specific constituencies, address specific capabilities or hardware considerations, provide for incremental implementation, facilitate usability, etc.,..

The QA Working Group has accepted the comment and has reworded the affected section as recommended and it now reads

The CSS2.1 [CSS2.1] specification defines the notion of vendor-specific extension. An initial dash or underscore is guaranteed never to be used in a property or keyword by any current or future level of CSS. Thus typical CSS implementations may not recognize such properties and may ignore them according to the rules for handling parsing errors. However, because the initial dash or underscore is part of the grammar, CSS2.1 implementers should always be able to use a CSS-conforming parser, whether or not they support any vendor-specific extensions.

The QA Working Group has accepted the comment and has reworded the affected section as recommended and it now reads:

There are typically two approaches: (see section 4.2.3 Extensibility from Architecture of the World Wide Web, Volume One [WEB-ARCH])

• a so-called "must Understand" policy, as implemented in SOAP 1.2 by the mustUnderstand attribute; in this type of mechanism, a processor encountering a syntax token not defined in the specification is required to know how to process the said token or must fail for the whole unit where the token appears
• a so called "cmust Ignore" policy (as implemented in SOAP 1.2 by the mustIgnore attribute), where a processor not knowing how to process an unknown syntax token must skip part or the totality of the unit where the token appears

A good way to handle these two approaches is to have a way in the syntax to distinguish which behavior is expected (e.g., mustUnderstand/mustIgnore attributes in SOAP 1.2). Which policy to choose depends heavily on the importance of the processing of the data, the user experience of applications based on the said format, etc.

The QA Working Group has accepted the comment with modification. The comment is a good one, but a bit out of scope since it is about how to write good tests. However, we have reworded the affected section to clarify and it now reads:

What does it mean? For each feature, the Working Group might seek early implementation to demonstrate the feature. If early implementations are not available (e.g., due to commercial constraints, IPR, etc.), it is beneficial to write test cases to illustrate a concept or use case of the technology. This provides a way to to study the interactions between the different parts of the specification and reveal problems. Additionally, these test cases can be incorporated into a test suite.

The QA Working Group has accepted the comment and has reworded the affected section as recommended and it now reads:

What does it mean? If an existing formal language (e.g. DTD, Schemas, ...) is expressive enough to describe the technical requirements of the specification, use it and when the English prose and the formal language overlap, make it clear which one takes precedence in case of discrepancy.

Why care?When possible, there is an immediate benefit of using a formal language to describe conformance requirements. It minimizes ambiguities introduced by the interpretation of the prose. There is also the possibility of using existing tools for the given language to facilitate testing and validation.

However, prose remains necessary to allow implementers to understand the specification, as well as to express additional requirements the formal language cannot express; this means that there are possible overlaps between the prose and the formal language, in which case, it is important to define which one is the main point of reference in case of disjunction.

The QA Working Group disagrees with the comment. The Quality Assurance Working Group believes that such a good practice would be inappropriate for the following reasons:

• some of the qualities mentioned in the comment (WAI AAA conformance, conformance to QA Specification Guidelines, and more generally the "conformance to other specifications" aspect) can be legitimately claimed as long as the relevant specifications define a conformance that can apply to specifications or documents - as this is the case for WCAG 1.0 and the Specification Guidelines; a specification claims of conformance to other specifications can be deficient the same way a product can be deficient with regard to its conformance claim, and these deficiencies should be fixed either during the development process or through the errata process
• for the other qualities mentioned where objective evaluation criteria are not available (simple, easy, device-independent), it is unclear that this defects affects that many - if any - specifications that it would deserve a good practice in the current version of Specification Guideline;