W3C QA Framework: Operational Guidelines

W3C Working Draft 5 December 2001

This version:
Latest version:
Previous version:
Kirill Gavrylyuk (kirillg@microsoft.com)
Lofton Henderson (lofton@rockynet.com)


This document defines a common Operational Framework for building conformance test suites and tools for W3C specifications. It presents operational and procedural guidelines for groups undertaking conformance materials development. This document is one in a family of QA Framework documents, which includes the other existing or in-progress specifications: Overview & Process; Specification Guidelines; and, Technical Guidelines.

Status of this document

This document is a Note, made available by the W3C Quality Assurance Activity (QA) for discussion on the QA email list.

This version incorporates the discussions at the first QA face-to-face, and supersedes the first, incomplete draft. Open issues and other incompletions are flagged with "@@". Where applicable, the presentation style is Guidelines-plus-Checkpoints, similar to WAI standards. Where appropriate, some documents in this Framework family (including this one) have an associated and copiously hyperlinked "Examples & Techniques" document, in which details and especially discretionary choices are elaborated. In this document version, guidelines and checkpoints are not yet numbered or formatted [@@we need to do this before release].

Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress".

Please send comments to www-qa@w3.org.

Table of contents

1. Introduction
1.1 Terminology
2. Operational framework
2.1 W3C Process considerations
2.2 WG Charter considerations
2.2.1 Define commitment to QA
2.2.2 Where to produce test suite
2.2.3 Staffing commitments
2.3 Principal aims of TS development
2.3.1 Goals
2.3.2 Scope of W3C QA framework Certification Branding
2.4 Starting the test suite
2.4.1 Resources/staffing
2.4.2 Starting & Synchronizing with spec progression
2.4.3 methodology
2.4.4 Transferring a TS from outside W3C(to revise)
2.4.5 Test suite process document Tests Contribution IPR and copyright Communication Tests review process Test materials publication Usability of the test suite by external parties
2.4.6 Results publication
2.5 Test suite maintenance
2.5.1 Resources/staffing
2.5.2 Contributions/review
2.5.3 Support for spec versioning/errata
2.5.4 Appeal process
2.5.5 Maintenance of the test suite after WG disappears
2.6 WG relationship to QA Activity
2.7 Logos & branding.
2.8 other Ops&Proc topics
3. Conformance
4. Acknowledgments
5. References

(This is done by my rudimentary XSLT process--it numbers subsequent h2..h5 headers and pulls them forward, here, as a hyperlinked numbered TOC)

1. Introduction

This document defines a common Operational Framework for building conformance test suites and tools for W3C specifications. It presents operational and procedural guidelines for groups undertaking conformance materials development. Therefore it ames to provide step by step instructions and options on how to set up, aquire or develop and maintain the test suite. It also represents relationship between W3C process and QA and charter considerations for Working Groups that are about to start.

This document is one in a family of QA Framework documents described in the QA Framework Introduction. It references other documents in the family, such as:

  1. Introduction
  2. Specification guidelines
  3. Technical guidelines

It also refers to several examples of the succesful QA work within W3C and beyond.

The first 2 sections give short description of how QA is reflected in the current W3C Process document and what should be addressed by WG in the charter. Then we describe in details the scope of TS development in W3C WG, what are the goals of QA work within WG. The main 2 sections of the document describe in details the procedure for starting and maintenance of the test suite taking into account level of commitment that WG took in the charter. The "Relationship to QA Activity" section ties QA process within Working Group with resources available from QA Activity.

...we probably need to include at least the 2nd half of the following terminology section into the framework...

1.1 Terminology

One of the deliverables of the QA Activity is a Glossary [@@GLOSSARY]. Unusual terms in this framework specification are defined when first used, and most generally useful QA-specific terms will eventually be in the Glossary.

(@@Ed. note -- following is not uniformly implemented yet.)

Although the guidelines of this framework are nominally informative, the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" will be used as defined in RFC 2119 [RFC2119], and will be used as if the guidelines are normative.

2. Operational framework

2.1 W3C Process considerations

In the current W3C Process Document two QA related guidelines may be found:

  1. "[...]groups may produce technical reports, review the work of other groups, develop sample code or test suites, etc." (sec 3.)
  2. "W3C should make every effort to maintain its Recommendations (e.g., by tracking errata, providing test bed applications, helping to create test suites, etc.)" (sec 4.)

Nevertheless, there is a body of contemporary QA experience and activity amongst the working groups. The Matrix identifies more than a score of test suites and validators, in various states of development.

Guideline. Working Groups should consider completion and publication of significant test materials to be a criterion for CR-exit and PR-entrance.

Explanation & Rationale. A de facto convention of "CR-exit" seems to be a common process goal amongst a number the existing test suite efforts (e.g., SVG, UAAG,...). This makes sense, as it is natural for test suites and implementations to develop in parallel -- each is a help to the development of the other. And, the Process Document does address implementation requirement:

2.2 WG Charter considerations

Guideline. Working Group charters should address goals and plans for test suites and tools.

Explanation & Rationale. In approving and initiating the QA Activity, W3C has endorsed the principle that, in order for W3C Web standards to achieve full interoperability and access to all, the quality of implementation must be given as much attention as the standards' development. The principal factor for improving the quality of implementation is early availability of conformance test materials. Therefore, conformance test materials -- test suites and tools -- must be considered as WG deliverables the same as the standards themselves. Accordingly, they must be addressed in the WG charter.

2.2.1 Define commitment to QA


Explanation. Detailed planning and specification of test materials is inappropriate in the charter, as it binds the WG inflexibly to test materials implementation specifics for which flexibility may be needed. There are numerous possible levels of commitment that a WG could make, based on anticipated WG resources, existence of outside efforts, and the WG's current stage in the process [REC-TRACK]. A range of possible commitments has been identified. See "Examples & Techniques" [@@EX-TECH].

[...in EX-TECH, we would have:

The range of possibilities for WG commitment is:

This 11-point enumeration is derived from proposal to the QA mail list, after the 4/2001 QA Workshop.

The further down the scale (i.e., the higher the number) that the WG commits, the better it is for achieving W3C's ultimate goals of interoperability and accessibility.

/EX-TECH, end of "Examples & Techniques" content, back to qaFrame.]

2.2.2 Where to produce test suite

Checkpoint. The WG charter should address where and how conformance test materials will be produced.

Explanation. There are several possibilities for producing conformance test materials:

2.2.3 Staffing commitments

Checkpoint. The WG charter TS plans should address staffing commitments.

Explanation. There will be at least some staffing required from the WG, for any of the acceptable options presented so far. Depending upon the general intent and plan, and how the test suite will be built, the commitment can range from minimal to significant. See "Examples & Techniques" [@@EX-TECH].

[...in EX-TECH, we would have:

-- quick tabular summary of size-of-effort data, from section 2.3 of http://www.w3.org/Graphics/SVG/Test/svgTest-manual.htm.

-- quantify some examples, e.g., estimated XML uptake effort?

/EX-TECH, end of "Examples & Techniques" content, back to qaFrame.]

2.3 Principal aims of TS development

2.3.1 Goals

Common frameworks for W3C QA activities are obviously shaped by the high-level goals for development and deployment of test materials. Before further developing the framework guidelines, it is useful to state some of these high-level goals.

The over-arching goal is stated in the QA Activity Statement:

More specific goals of test materials development should include:

2.3.2 Scope of W3C QA framework

The goals of a common W3C QA framework do not encompass: Certification

While certification may have a positive quality assurance role in standards-based Web environments, W3C does not presently intend to initiate or offer any certification services. Nevertheless, test suites and materials development should not arbitrarily preclude:

Factors of rigor, objectivity, and defensibility are key determinants of usability for certification, and these same properties are positive assets of any test materials.

Note: although tests suites may be used for compliance verification, Working Group MAY make test suites users aware of the fact that Branding

Certification aside, some W3C activities support "branding" -- use of a conformance icon to indicate some level of standard compliance. Examples: HTML validation, CSS validation, WCAG compliance. While these content branding schemes are relatively non-controversial, there are nevertheless some serious issues which should be addressed -- particularly in the contexts of user-agent and API conformance -- before any branding-related goals are articulated.

2.4 Starting the test suite

This section will guide through setup of QA process in the Working Group. For each checkpoint in this group WG should take one of the options depending on decisions taken in the charter for test suite origin and level of commitment.

2.4.1 Resources/staffing

Guideline: WG should address resources/staffing for the test suite development/maintenance.

Checkpoint. Depending on the plan for origin of the test suite WG should appoint test QA moderator from either of:

2.4.2 Starting & Synchronizing with spec progression

- start: the sooner, the better

- early benefits of, e.g., TA analysis

- when to start TS work (WD, CR, ...)

- coordination with spec (REC) progression

2.4.3 methodology

Before starting test suite development, test suite implementer (WG or 3rd party) should look at Technical Guidelines and decide what approach to take for test materials organization, tests criteria, etc.

2.4.4 Transferring a TS from outside W3C(to revise)

In case if WG and some external party decided to move the test suite to W3C from outside the following items should be resolved

- parties must to sort out IPR

- charter amendment? yes.

- need to sort out new home (WG), staffing, and other issues at "Charter" level of detail.

(several guidelines and checkpoints can be distilled from XML transfer example)

- after the transfer, test suite process MUST be established for future contributions.

2.4.5 Test suite process document

Guideline: WG should publish Test process document.

Test suite process document is a central QA document describing how testing process is set up within WG. Examples of the test process document could be found at DOM TS process, XML Schema TS process, XML Protocol TS process, etc. It is required part of QA process in WG and should address the following questions: Tests Contribution

Checkpoint. WG Test suite process document should explain contribution process.

Process document must describe where to submit test materials and who should be notified (moderator) of submission. Contribution process MAY describe the format of contributed material, it may contain validation harness, utilities that facilitates tests creation, templates. IPR and copyright

Checkpoint. WG Test suite process document should provide licenses for submitted and published test materials. WG MAY choose either W3C Document or W3C Software license, but W3C Document license is recommended. Communication

Checkpoint. WG Test suite process document should provide means for communication regarding test suites.

WG MAY choose mailing list to send all comments and announcements to. Tests review process

Checkpoint. WG Test suite process document MAY set/describe review procedure for submitted test materials

[Example, need to formalize into checkpoints Test materials publication

Checkpoint. WG Test suite process document should point to the web page with published test materials.

WG MAY describe the way it publishes test materials and it is recommended to use one of practices described in Technical Guidelines Usability of the test suite by external parties

Checkpoint. WG Test suite process document should mention usability of test materials by external parties.

2.4.6 Results publication

WG should encourage vendors to publish results for their implementations including/describing test harness that would allow anyone to reproduce the results.

2.5 Test suite maintenance

Once test suite development initiated, its maintenance requires almost the same amount of resources

2.5.1 Resources/staffing

Checkpoint. Working Group SHOULD appoint task force or another entity to maintain the test suite

It is recommended errata task force be involved in test suite maintenance, since this will help to keep test suite and specification in sync. Maintenance group should establish tracking system to track issues/changes to the test suite.

2.5.2 Contributions/review

Contributions may arrive through out all test suite/standard's life cycle. WG should be able to maintain contribution and review procedures described earlier.

2.5.3 Support for spec versioning/errata

Test suite structure needs to support spec versioning/errata (to include example from XSLT). The following process MAY apply:

2.5.4 Appeal process

Test Suite process identifies communication channel for appeals. Once appeal for specific tests comes in the following process MAY apply:

2.5.5 Maintenance of the test suite after WG charter expires

[Active issue]If there is still active work on the test suite even if the WG is not re-chartered to handle the work, then it is up to the Director to determine how this work should be done. It MAY be assigned to a Coordination Group or a related WG or the Team could take care of minor items.

2.6 WG relationship to QA Activity

The QA Activity anticipates different relationships to the various Working Groups. On the one hand, the QA-specific needs of the Working Groups will vary depending upon their resources and experience, as well as their stage in the Recommendation track. On the other hand, the W3C processes may evolve so that QA-specific process requirements are tied to milestones in the Recommendation track.

Potential relationships between Working Groups and the QA Activity include:

Key determinants of the WG/QA relationship will include level of available QA Activity resources, and the evolution of W3C policy on QA requirements.

2.7 Logos & branding.

- WAI does this for content; CSS also and basic HTML validation.

- Controversial topic, as email thread and UAAG CR comments show.

- Taxonomy dependent: content (WCAG, CSS, etc) is less difficult/controversial; UAs will be contentious.

2.8 other Ops&Proc topics

- ...any?...

3. Conformance

[This document should have a conformance clause. Uncertain what it should say now, as it is unclear what is the normative force of this document. At least it should probably address what is involved in (possibly voluntary) conformance to this document. Precisely what it might say is an issue that prehaps we can address later?]

4. Acknowledgments

The following QA Working Group and Interest Group participants have contributed significantly to the content of this document:

In addition ideas expressed by [@@...there may be some others in the email archive, e.g., Andrew Thackrah, whose ideas have been excerpted and who should be credited...] in QA email and telephone discussions have contributed to the contents of this document.

5. References

[@@Ed. note: This has not yet been cut down from original all-in-one Framework document. It should be edited to only include those references which actually pertain to this document.]

QA activity email thread about third-party participation in test materials production.
W3C-wide conformance activity survey covering all the WGs, "The Matrix".
W3C Process Document, 19 July 2001.
QA Activity test taxonomy, a classification scheme for conformance test materials.
Quality Assurance Interest Group of the W3C QA Activity.
Quality Assurance Working Group of the W3C QA Activity.
Process document for DOM WG Test suite.
Stages and milestones in the W3C Recommendation Track, per the Process Document (section 5.2).
Key words for use in RFCs to Indicate Requirement Levels, March 1997.
SVG Working Group's test suite resource page.
Web Content Accessibility Guidelines, version 1.0, W3C Recommendation, 5 May 1999.
Email proposal by David Marston, on the QA public mail list, for range of WG commitment levels to conformance test materials production.
OASIS XML Conformance TC's XML test suite resource page.
OASIS XML Conformance TC's XSLT/Xpath test suite resource page.