QA Framework: Process & Operational Guidelines

W3C Working Draft 2 January 2002

This version:
Latest version:
Previous version:
Kirill Gavrylyuk (kirillg@microsoft.com)
Lofton Henderson (lofton@rockynet.com)
Dimitris Dimitriadis (dimitris@ontologicon.com)


This document defines a common Operational Framework for building conformance test suites and tools for W3C specifications. It presents operational and procedural guidelines for groups undertaking conformance materials development. This document is one in a family of QA Framework documents, which includes the other existing or in-progress specifications: Overview & Process; Specification Guidelines; and, Technical Guidelines.

Status of this document

This document is a Note, made available by the W3C Quality Assurance (QA) Activity for discussion on the QA email list.

This version incorporates the discussions at the first QA face-to-face, plus subsequent QA WG teleconferences, and supersedes all previous drafts. Open issues and other incompletions are flagged with "@@". The number of such are mostly minor editorial issues in this draft, except for the issue of the conformance clause. This version intended as the final WG review version before final editing for first public working draft (FPWD).

Where applicable, the presentation style is Guidelines-plus-Checkpoints, similar to WAI standards. Where appropriate, some documents in this Framework family (including this one) have an associated and copiously hyperlinked "Examples & Techniques" document, in which details and especially discretionary choices are elaborated. In this document version, guidelines and checkpoints are not yet numbered or formatted.

Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress".

Please send comments to www-qa@w3.org.

Table of contents

1. Introduction
1.1 Terminology
2. Operational framework
2.1 W3C Process considerations
2.2 WG Charter considerations
2.2.1 Define commitment to QA
2.2.2 Where to do TS development Where to produce test suite Test materials home
2.2.3 Staffing commitments
2.3 Principal aims of TS development
2.3.1 Goals
2.3.2 Scope of W3C QA framework Certification Branding
2.4 Starting the QA work
2.4.1 Resources/staffing
2.4.2 Starting & Synchronizing with spec progression
2.4.3 Testing Methodology
2.4.4 Transferring a TS from outside W3C
2.4.5 QA process document Tests Contribution IPR and copyright Communication Tests review process Test materials publication Usability of the test suite by external parties
2.4.6 Results publication
2.5 Test materials maintenance
2.5.1 Resources/staffing
2.5.2 Contributions/review
2.5.3 Support for spec versioning/errata
2.5.4 Appeals process
2.5.5 Maintenance of the test suite after WG charter expires
2.6 WG relationship to QA Activity
2.6.1 Liaison and consultation.
2.6.2 Active reviews.
2.6.3 Adjudicate entry and exit criteria.
2.6.4 QA resources supplement.
2.6.5 Resolution of external QA requests
3. Conformance
4. Acknowledgments
5. References

1. Introduction

This document defines a common Operational Framework for building conformance test suites and tools for W3C specifications. It presents operational and procedural guidelines for groups undertaking conformance materials development. Therefore it aims to provide step by step instructions and options on how to set up, acquire or develop and maintain the test suite. It also represents relationship between W3C process and QA and charter considerations for Working Groups that are about to start.

This document is one in a family of QA Framework documents described in the QA Framework Introduction. It references other documents in the family, such as:

It also refers to several examples of the successful QA work within W3C and beyond.

The first 2 sections give short description of how QA is reflected in the current W3C Process document and what should be addressed by a WG in its charter. Then we describe in detail the scope of TS development in W3C WG, what are the goals of QA work within WG. The main 2 sections of the document describe in detail the procedure for starting and maintenance of the test suite taking into account level of commitment that WG took in the charter. The "Relationship to QA Activity" section ties QA process within Working Group with resources available from QA Activity.

1.1 Terminology

One of the deliverables of the QA Activity is a Glossary [@@GLOSSARY]. Unusual terms in this framework specification are defined when first used, and most generally useful QA-specific terms will eventually be in the Glossary.

Although the guidelines of this framework are nominally informative, the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" will be used as defined in RFC 2119 [RFC2119], and will be used as if the guidelines are normative.

2. Operational framework

2.1 W3C Process considerations

In the current W3C Process Document two QA related guidelines may be found:

  1. "[...]groups may produce technical reports, review the work of other groups, develop sample code or test suites, etc." (sec 3.)
  2. "W3C should make every effort to maintain its Recommendations (e.g., by tracking errata, providing test bed applications, helping to create test suites, etc.)" (sec 4.)

Nevertheless, there is a body of contemporary QA experience and activity amongst the working groups. The Matrix identifies more than a score of test suites and validators, in various states of development.

Guideline. Working Groups should consider completion and publication of significant test materials to be a criterion for CR-exit and PR-entrance.

Explanation & Rationale. A de facto convention of "CR-exit" seems to be a common process goal amongst a number the existing test suite efforts (e.g., SVG, UAAG,...). This makes sense, as it is natural for test suites and implementations to develop in parallel -- each is a help to the development of the other. And, the Process Document does address implementation requirement:

2.2 WG Charter considerations

Guideline. Working Group charters should address goals and plans for test suites and tools.

Explanation & Rationale. In approving and initiating the QA Activity, W3C has endorsed the principle that, in order for W3C Web standards to achieve full interoperability and access to all, the quality of implementation must be given as much attention as the standards' development. The principal factor for improving the quality of implementation is early availability of conformance test materials. Therefore, conformance test materials -- test suites and tools -- must be considered as WG deliverables the same as the standards themselves. Accordingly, they must be addressed in the WG charter.

2.2.1 Define commitment to QA


Explanation. Detailed planning and specification of test materials is inappropriate in the charter, as it binds the WG inflexibly to test materials implementation specifics for which flexibility may be needed. There are numerous possible levels of commitment that a WG could make, based on anticipated WG resources, existence of outside efforts, and the WG's current stage in the process [REC-TRACK].

A range of possible commitments has been identified:

  1. WG promises to think about testability when writing their documents.
  2. WG include members with QA expertise, plans regular spec reviews for testability.
  3. Beyond a few normative examples, WG aims to have numerous normative cases in the body of the Rec.
  4. WG commits that a test suite, not necessarily complete and thorough, will exist somewhere before they go to Rec.
  5. WG commits that they review and fully approve a test suite, not necessarily complete and thorough, before they go to Rec.
  6. WG commits to reviewing test cases on a continuing basis, including after going to Rec.
  7. WG will attempt to generate a complete catalog of test cases before going to Rec.
  8. WG insists on a complete catalog and partial implementation of a test suite before going to Rec.
  9. WG insists on a test suite, not necessarily complete and thorough, and provides an test suite and/or tools, before specification in question goes to Recommendation.
  10. WG commits to writing test cases themselves and delivers a test harness and suite at some point.
  11. WG insists on a complete test suite before going to Rec.

This 11-point enumeration is derived from a proposal to the QA mail list, after the 4/2001 QA Workshop.

The further down the scale (i.e., the higher the number) that the WG commits, the better it is for achieving W3C's ultimate goals of interoperability and accessibility.

Guideline. Working Group commitment to QA SHOULD be not lower then level 4 - produce some test suite before specification goes to Recommendation.

2.2.2 Where to do TS development Where to produce test suite

Checkpoint. The WG charter should address where and how conformance test materials will be produced.

Explanation. There are several possibilities for producing conformance test materials: Test materials home

Guideline. Test suites and tools should ultimately reside in W3C.

Checkpoint. W3C should provide secure and reliable repository location for the test materials.

Checkpoint. W3C should have ultimate responsibility -- though not necessarily sole responsibility -- for the ongoing evolution of test suites and tools, after some milestone of operational maturity of the materials.

Explanation & Rationale. This does not necessarily mean that the WG must solely build the test suites and tools -- in fact there are good reasons why external parties should have a strong participation during the building of the materials (see @@below). But once the materials have reached some advanced milestone of maturity and development (e.g., operationally usable), they should reside in and be the responsibility of W3C, and in particular the Working Group responsible for the subject standard.


  1. At the point that test materials become operationally deployed, then challenges to the correctness of both the test materials and specifications (RECs) normally increase, and the WG is the best venue for initial processing and adjudication of such queries against both.
  2. Past experience suggests that the WG is more likely to have better longevity than external, ad-hoc (dedicated to the specific test materials tasks) entities.
  3. Further to #1, it is more likely that specification (REC) errata processing and test suite maintenance can be kept synchronized if both responsibilities reside in the same body, the WG.

If the WG responsible for a standard ceases to operate, then another home for the test materials would be found within W3C. Those responsible for ongoing interpretation and maintenance (errata) of the standard should be designated.

2.2.3 Staffing commitments

Guideline. The WG charter TS plans should address staffing commitments.

Explanation. There will be at least some staffing required from the WG, for any of the acceptable options presented so far. Depending upon the general intent and plan, and how the test suite will be built, the commitment can range from minimal to significant. See "Examples & Techniques" [@@EX-TECH].

2.3 Principal aims of TS development

2.3.1 Goals

Common frameworks for W3C QA activities are obviously shaped by the high-level goals for development and deployment of test materials. Before further developing the framework guidelines, it is useful to state some of these high-level goals.

The over-arching goal is stated in the QA Activity Statement:

More specific goals of test materials development should include:

2.3.2 Scope of W3C QA framework

The goals of a common W3C QA framework do not encompass: Certification

While certification may have a positive quality assurance role in standards-based Web environments, W3C does not presently intend to initiate or offer any certification services. Nevertheless, test suites and materials development should not arbitrarily preclude:

Factors of rigor, objectivity, and defensibility are key determinants of usability for certification, and these same properties are positive assets of any test materials.

Checkpoint. Working Group SHOULD ensure that test materials are documented, self contained and reusable by external parties

Guideline. Although tests suites may be used for compliance verification, Working Group SHALL make test suites users aware of the fact that

  1. passing all of the tests does not guarantee full compliance of implementation to specification
  2. failing test suite means failing tests for specific feature they target

The last item requires some elaboration. Depending on the nature of the test, it may actually be sufficient to judge non-compliance of the subject. Based on taxonomy the following is valid: Branding

Certification aside, some W3C activities support "branding" -- use of a conformance icon to indicate some level of standard compliance. Examples: HTML validation, CSS validation, WCAG compliance. While these content branding schemes are relatively non-controversial, there are nevertheless some serious issues which should be addressed -- particularly in the contexts of user-agent and API conformance -- before any branding-related goals are articulated.

2.4 Starting the QA work

This section will guide through setup of QA processes in the Working Group. For certain checkpoints in this group WG should take one of the options depending on decisions taken in the charter for test suite origin and level of commitment.

2.4.1 Resources/staffing

Guideline. WG should address resources/staffing for the test suite development/maintenance.

Checkpoint. Depending on the plan for origin of the test suite WG should appoint test QA moderator from either of:

When test suite is developed outside of Working Group, moderator usually already exists where test suite coming from. It is recommended to invite moderator to become WG member, since he(she) needs access to all WG materials and resources. See also section on transfering test suite from outside W3C.

2.4.2 Starting & Synchronizing with spec progression

Guideline. WG SHOULD start synchronizing specification writing and test materials development as early as possible at the Working Draft phase of the specification cycle

Explanation and Rationale. There are several strong benefits from starting test materials development on the early stages:

Checkpoint. WG SHOULD introduce intermediate milestones for test test materials completion, bounded to Working Drafts/Candidate Recommendation publications

Checkpoint. Latest version of the spec draft and latest version of the published test materials should be synchronized.

After entering Recommendation phase every update to the specification through Errata or new versions MUST be reflected in the published test materials as outlined in Test Materials Maintenance.

2.4.3 Testing Methodology

Before starting test suite development, test suite implementer (WG or 3rd party) should look at Technical Guidelines and decide what approach to take for test materials organization, tests criteria, etc.

Guideline. Review Technical Guidelines before starting QA work.

2.4.4 Transferring a TS from outside W3C

In case when WG and some external party decide to move the test materials (TM) to W3C from external entity/group (EG) the following procedure is recommended:

Before transfer WG with the help of QA WG:

During transfer

After transfer, initial process setup

First W3C public release of new TM

After first public release, TM enters maintenance phase which is described below.

2.4.5 QA process document

Guideline. WG shall publish a QA process document.

A Working Group QA process document is a central QA document describing how testing process is set up within the WG. Examples of the test process document can be found at DOM TS process, XML Schema TS process, XML Protocol TS process, etc. It is a required part of QA process in WG and should address the following questions: Tests Contribution

Checkpoint. The WG Test suite process document SHALL explain contribution process.

The process document must describe where to submit test materials and whom to notify (moderator) of submission. Contribution process SHOULD describe the format of contributed material, it may contain validation harness, utilities that facilitates tests creation, templates, etc. IPR and copyright

Checkpoint. WG Test suite process document shall define the licenses applicable to submitted and published test materials. WG MAY choose either the W3C Document or the W3C Software license, but the W3C Document license is recommended. Communication

Checkpoint. WG Test suite process document shall provide means for communication regarding test suites.

WG MAY choose mailing list to send all comments and announcements to. Tests review process

Checkpoint. WG Test suite process document SHOULD set/describe review procedures for submitted test materials

The following example may serve as a guideline for tests review process

  1. Each contribution should be reviewed by at least one vendor other than submitter for it's eligibility. Any vendor involved in creation of WG test collection other then submitter may review tests being contributed.
  2. Eligibility of tests should be judged by:
  3. After review results should be sent back to submitter for his feedback.
  4. In case of disagreement, request for clarification should go to WG, tests in question should be left off the test suite till W3C WG clarifies the issue.
  5. Only those tests that are agreed to be correct by reviewers and submitter or approved by W3C WG clarification should be published. Test materials publication

Checkpoint. WG Test suite process document should point to the web page with published test materials.

WG MAY describe the way it publishes test materials and it is recommended to use one of practices described in Technical Guidelines Usability of the test suite by external parties

Guideline. WG Test suite process document SHOULD address usability of test materials by external parties.

As stated before in Certification section WG should make sure that TM are self-contained and usable and

Checkpoint. Process document should contain disclaimers described in Certification section

2.4.6 Results publication

WG should encourage vendors to publish results for their implementations including/describing test harness that would allow anyone to reproduce the results.

2.5 Test materials maintenance

Once test suite development is initiated, its maintenance requires an equally significant amount of resources.

2.5.1 Resources/staffing

Guideline. Working Group SHOULD find/allocate resources for test suite maintenance

Checkpoint. Working Group SHOULD designate a task force or another entity to maintain the test suite, as well as maintenance procedures.

It is recommended that the errata task force be involved in test suite maintenance, since this will help to keep test suite and specification in sync. Maintenance group shall establish tracking system to track issues/changes to the test suite.

2.5.2 Contributions/review

Contributions may arrive through out all test suite/standard's life cycle. WG should be able to maintain contribution and review procedures described earlier.

2.5.3 Support for spec versioning/errata

Test suite structure SHOULD support spec versioning/errata (to include: example from XSLT). The following process MAY apply:

2.5.4 Appeals process

Test Suite process MUST identify the communication channel for appeals of tests validity. Once appeals for specific tests come in, the following process MAY apply:

2.5.5 Maintenance of the test suite after WG charter expires

[Active issue]If there is still active work on the test suite even if the WG is not re-chartered to handle the work, then it is up to the Director to determine how this work should be done. Some options include:

2.6 WG relationship to QA Activity

The QA Activity anticipates different relationships to the various Working Groups. On the one hand, the QA-specific needs of the Working Groups will vary depending upon their resources and experience, as well as their stage in the Recommendation track. On the other hand, the W3C processes may evolve so that QA-specific process requirements are tied to milestones in the Recommendation track.

Potential relationships between Working Groups and the QA Activity include:

Key determinants of the WG/QA relationship will include level of available QA Activity resources, and the evolution of W3C policy on QA requirements.

2.6.1 Liaison and consultation.

In an assistive role, the QA Activity provides consultation to Working Groups in planning and building or acquiring of test materials. This should be the baseline relationship, applicable to all Working Groups. Consultation may be provided in formal and informal ways. Informal questions MAY be sent to www-qa@w3.org. In order to make assistance quick and accurate the following process is introduced for formal requests:

2.6.2 Active reviews.

The QA Activity provides detailed review and assessment of Working Group QA work and test materials per request from Working Group. QA Working Group also performs detailed review of the charter and specifications of the Working Group per request from the latter. The following procedure SHOULD be used inside QA Working Group for review of materials presented for review by Working Group. QA Working Group MAY pro-actively initiate reviews, provided it has resources to do so.

2.6.3 Adjudicate entry and exit criteria.

Although there are currently no W3C process requirements for QA-specific entry and exit criteria at milestones on the Recommendation track, Working Groups could define such for themselves. The QA Working Group may help identify those.

2.6.4 QA resources supplement.

QA Working Group could in some cases, provide resources for the planning and building of test materials for some Working Groups.

2.6.5 Resolution of external QA requests

The QA Working Group may receive requests from external parties or persons to resolve QA related questions that involve one or several W3C standards. The Chair of QA WG SHALL assign task team of the QA WG members to coordinate and drive the question to resolution. Assigned task team SHALL

3. Conformance

[@@tbd...This document should have a conformance clause. Uncertain what it should say now, as it is unclear what is the normative force of this document. At least it should probably address what is involved in (possibly voluntary) conformance to this document. Precisely what it might say is an issue that must be addressed before FPWD.]

4. Acknowledgments

The following QA Working Group and Interest Group participants have contributed significantly to the content of this document:

5. References

[@@Ed. note: This has not yet been cut down from original all-in-one Framework document. It should be edited to only include those references which actually pertain to this document.]

QA activity email thread about third-party participation in test materials production.
W3C-wide conformance activity survey covering all the WGs, "The Matrix".
W3C Process Document, 19 July 2001.
QA Activity test taxonomy, a classification scheme for conformance test materials.
Quality Assurance Interest Group of the W3C QA Activity.
Quality Assurance Working Group of the W3C QA Activity.
Process document for DOM WG Test suite.
Stages and milestones in the W3C Recommendation Track, per the Process Document (section 5.2).
Key words for use in RFCs to Indicate Requirement Levels, March 1997.
SVG Working Group's test suite resource page.
Web Content Accessibility Guidelines, version 1.0, W3C Recommendation, 5 May 1999.
Email proposal by David Marston, on the QA public mail list, for range of WG commitment levels to conformance test materials production.
OASIS XML Conformance TC's XML test suite resource page.
OASIS XML Conformance TC's XSLT/Xpath test suite resource page.
QA Framework: Technical Guidelines(not written).
QA Framework: Specification Guidelines(not written).