Tools for improving W3C Specification Quality

Project acronym: QUESTION-HOW
Project Full Title:Quality Engineering Solutions via Tools, Information and Outreach for the New Highly-enriched Offerings from W3C: Evolving the Web in Europe
Project/Contract No. IST-2000-28767
Workpackage 1, Deliverable D1.5

Project Manager: Daniel Dardailler <danield@w3.org>
Author of this document: same

Created: 29 August 2002. Last updated: 29 August 2002.

Introduction

The goal of this project is to provide a framework of guidelines and tools used to analyse a W3C specification and show the dependencies between the different features and attributes of this specification.

The goal is to make the W3C specifications more amenable to writing test suites and interoperable products.

The W3C Specifications are, most of the time, based on a DTD or an XML schema language which define the syntactic structure of a language. The specification further explains in plain test the features and their uses.

For the point of view of a developer it may become difficult to work out the dependencies between the different features: nested elements, forbidden elements, discretionary behaviors, etc. or to find the list of values accepted by attributes.

A first document in a form of a W3C guidelines/checklist has been produced to describe what constitutes a testable specification.

Specification Guidelines

As with most of W3C work, this is not something done in solo by one person but rather the contribution of W3C staff to a specific Working Group.

In this case, this is the QA Working Group, chaired by a W3C staff funded by this project..

The results of the WG is a series of Guidelines for QA development and operational process, better specifications and QA tools.

This deliverable report on the Specification Guidelines only, edited by another W3C staff funded by QH.

The principal goal of this document is to help W3C Working Groups to write clearer, more implementable, and better testable technical reports. It both provides a common framework for specifying conformance requirements and definitions, and also addresses the representation of specifications (technical reports) as schemata, both of which facilitate the generation of test materials.

The material is presented as a set of organizing guidelines and verifiable checkpoints.

The QA WG work is still going on at the time of this report and their latest work is available online on the W3C site.

We're reproducing here for project record the list of checkpoints designed as part of this project.

Checklist of Checkpoints

This checklist includes all checkpoints from the Specification Guidelines presented in a tabular format. The checkpoint priorities are included, and the order of the specification guidelines document is maintained.

The presentation is intended to be convenient for organizers and evaluators of QA projects in W3C Working Groups, to facilitate assessing specifications against the checkpoints. The table includes spaces for scoring each checkpoint, "yes" (satisfied), "no" (not satisfied), "n/a" (not applicable).

For a description of the meaning of the priorities in the following table, please consult the conformance clause of the specification guidelines ([QAF-SPEC], chapter 3) .

Checklist table

Nbr	Checkpoint	Priority	Yes	No	N/A
Guideline 1. Define Use Cases.
1.1	Define the scope of the specification.	[Priority 1]
1.2	Include User Scenarios.	[Priority 2]
1.3	Provide examples.	[Priority 1]
1.4	Ensure that every test assertion is covered by an example.	[Priority 3]
1.5	Include an interpretation section.	[Priority 2]
Guideline 2. Identify what needs to conform and how.
Nbr	Checkpoint	Priority	Yes	No	N/A
2.1	Identify all classes of product.	[Priority 1]
2.2	For each class of product, define the conformance requirements.	[Priority 1]
2.3	For each class of product, indicate minimal support requirements.	[Priority 3]
2.4	Identify which of the categories of object are specified in the document as a whole.	[Priority 3]
Guideline 3. Address the use of profiles to divide the technology.
Nbr	Checkpoint	Priority	Yes	No	N/A
3.1	Choose whether or not to have profiles.	[Priority 1]
3.2	Include a table of contents entry.	[Priority 2]
3.3	If profiles are chosen, indicate whether or not their use is mandatory for conformance.	[Priority 1]
3.4	If profiles are chosen, define any minimal requirements.	[Priority 2]
3.5	If profiles are chosen, define their relationships and interaction with other dimensions of variability.	[Priority 2]
3.6	If profiles are chosen, address rules for profiles.	[Priority 2]
Guideline 4. Address the use of modules to divide the technology.
Nbr	Checkpoint	Priority	Yes	No	N/A
4.1	Choose whether or not to have modules.	[Priority 1]
4.2	Include a table of contents entry.	[Priority 2]
4.3	If modules are chosen, indicate any mandatory conditions or constraints on their usage.	[Priority 1]
4.4	If modules are chosen, define their relationships and interaction with other dimensions of variability.	[Priority 2]
Guideline 5. Specify conformance policy.
Nbr	Checkpoint	Priority	Yes	No	N/A
5.1	Make it clear where there are universal requirements for minimum functionality.	[Priority 1]
5.2	Make it clear when conformance requirements are strict.	[Priority 1]
5.3	Make it clear where requirements stop and product-specific extra features begin.	[Priority 1]
5.4	If special conformance terms are used, include a definition in the specification.	[Priority 1]
Guideline 6. Clarify the relation between deprecated features and conformance.
Nbr	Checkpoint	Priority	Yes	No	N/A
6.1	Identify and clearly indicate each deprecated feature.	[Priority 1]
6.2	For each class of product, specify the degree of support required for each deprecated feature and the conformance consequences of the deprecation.	[Priority 1]
6.3	Include an explanation for the deprecation.	[Priority 3]
6.4	Include examples to illustrate how to avoid using deprecated features.	[Priority 3]
6.5	Include a table of contents entry.	[Priority 2]
Guideline 7. Address the use of functional levels to divide the technology.
Nbr	Checkpoint	Priority	Yes	No	N/A
7.1	Address whether the specification uses or will use functional levels.	[Priority 1]
7.2	Include a table of contents entry.	[Priority 2]
7.3	If levels are used, define their relationships and interaction with other dimensions of variability.	[Priority 2]
Guideline 8. Define discretionary items.
Nbr	Checkpoint	Priority	Yes	No	N/A
8.1	Explicitly state the cases and conditions where discretionary items are allowed.	[Priority 1]
8.2	Indicate implementation dependencies and address allowable differences between implementations, where applicable.	[Priority 1]
8.3	Describe alternative approaches and the conditions under which an implementation is considered to be conforming.	[Priority 1]
8.4	Include a statement regarding consistent handling of a discretionary item within an implementation.	[Priority 2]
8.5	Include a table of contents entry.	[Priority 2]
Guideline 9. Allow extensions or NOT!
Nbr	Checkpoint	Priority	Yes	No	N/A
9.1	If extensions are disallowed, explicitly state it.	[Priority 1]
9.2	If extensions are allowed, explicitly state it.	[Priority 1]
9.3	If extensions are allowed, make it clear that the extensions do not negate support for required functionality.	[Priority 1]
9.4	If extensions are allowed, use a standard mechanism to define the extension.	[Priority 3]
9.5	If extensions are allowed, register or publish them.	[Priority 3]
9.6	If extensions are allowed, require that implementations include a way to operate without the extension.	[Priority 3]
9.7	Include a table of contents entry.	[Priority 2]
Guideline 10. Provide a conformance clause.
Nbr	Checkpoint	Priority	Yes	No	N/A
10.1	Include a conformance clause.	[Priority 1]
10.2	Create a separate conformance section.	[Priority 2]
10.3	Include a conformance clause entry in the table of contents.	[Priority 2]
10.1	Make normative reference to specifications on which the current specification depends.	[Priority 2]
10.5	Identify all dimensions of variability that are not used.	[Priority 1]
Guideline 11. Specify how to make conformance claims.
Nbr	Checkpoint	Priority	Yes	No	N/A
11.1	Identify and define all conformance designations.	[Priority 1]
11.2	Provide specific wording of the claim.	[Priority 3]
11.3	Provide a conformance disclaimer.	[Priority 3]
11.4	Impose no restrictions about who can make a claim or where claims can be published.	[Priority 1]
11.5	Include a table of contents entry.	[Priority 2]
Guideline 12. Publish an Implementation Conformance Statement proforma.
Nbr	Checkpoint	Priority	Yes	No	N/A
12.1	Include an Implementation Conformance Statement proforma as part of the specification.	[Priority 3]
12.2	Require the ICS be completed as part of the conformance claim.	[Priority 3]
Guideline 13. Support general document conformance conventions.
Nbr	Checkpoint	Priority	Yes	No	N/A
13.1	Use conformance key words.	[Priority 1]
13.2	Distinguish normative and informative text.	[Priority 2]
13.3	Follow Web Accessibility Initiative and Internationalization Guidelines.	[Priority 1]
13.4	Use consistent terminology.	[Priority 1]
Guideline 14. Use granular grammars to author the specification.
Nbr	Checkpoint	Priority	Yes	No	N/A
14.1	Use W3C endorsed grammar.	[Priority 2]
14.2	Identify intended behavior in the specification using markup.	[Priority 2]
14.3	Supply prose description of intended behavior together with each test assertion.	[Priority 2]
Guideline 15. Include test assertions.
Nbr	Checkpoint	Priority	Yes	No	N/A
15.1	Supply test assertions in the specification..	[Priority 1]
15.2	Tag test assertions according to a granular grammar.	[Priority 2]

Deviations from plan

None, even though the WG work on the Specification Guidelines is not completed, the QH funded part of the work is done.