W3C

QA Framework: Process & Operational Guidelines

W3C Working Draft 18 January 2002

This version:
http://www.w3.org/QA/WG/2002/framework-20020118/qaframe-ops
Latest version:
http://www.w3.org/QA/WG/qaframe-ops
Previous version:
http://www.w3.org/QA/WG/2002/framework-20020102/qaframe-ops
Editors:
Kirill Gavrylyuk (kirillg@microsoft.com)
Dimitris Dimitriadis (dimitris@ontologicon.com)
Lofton Henderson (lofton@rockynet.com)
Contributors:
See Acknowledgments.

Abstract

This document defines a common Operational Framework for building conformance test suites and tools for W3C specifications. It presents operational and procedural guidelines for groups undertaking conformance materials development. This document is one in a family of QA Framework documents, which includes the other existing or in-progress specifications: Overview & Process; Specification Guidelines; and, Technical Guidelines.

Status of this document

This document is a Working Draft, made available by the W3C Quality Assurance (QA) Activity for discussion on the QA email list.

This version incorporates the discussions at the first QA face-to-face, plus subsequent QA WG teleconferences, and supersedes all previous drafts. All review comments against the previous version have been accepted and implemented, except as indicated in an associated disposition of comments document. This version still has minor editorial issues, flagged by "@@", (almost) all of which are needed external links.

This version is intended as the last WG review version before final editing for first public working draft (FPWD).

This part of the Framework document family will have a companion part, "Process & Operational Techniques and Examples" (in progress). Some of the lengthier example and "how to" text of this current guidelines document version will be moved to the "Techniques and Examples" document, when the latter is written.

Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress".

Please send comments to www-qa@w3.org, the publicly archived list of the QA Interest Group. Please note that any mail sent to this list will be publicly archived and available, do not send information you wouldn't want to see distributed, such as private data.

Table of contents

1. Introduction
1.1 Motivation for this Guidelines document
1.2 Navigating through this document.
1.3 Terminology
2. Guidelines
G 1. In charters, describe QA goals, criteria, deliverables and resources
G 2. In charter, address where and how conformance test materials will be produced.
G 3. Ensure that goals and scope of QA development are clear
G 4. Allocate staff for QA process.
G 5. Synchronize specification and test materials developments
G 6. Produce QA process document.
G 7. Plan transfer of test materials to W3C if needed
G 8. Plan for test materials maintenance
3. WG relationship to QA Activity
4. Conformance
5. Acknowledgments
6. References


1. Introduction

This document describes the process and operational activities for building conformance test suites and tools for W3C specifications. The primary goal is to assist the W3C Working Groups by providing guidelines for the planning, development and deployment of conformance test materials. In this guideline, the term "conformance test materials" refers conformance test suites, validation tools, conformance checklists, and any other materials that are used to check or indicate conformance.

This document is part of a family of QA Framework documents designed to improve the quality of W3C specifications as well as their implementation by solidifying and extending current quality practices within the W3C. The QA Framework documents are:

The guidelines are intended for all Working Groups as well as developers of conformance materials for W3C specifications. Not only are the WGs the consumer of these guidelines, they are also key contributors. The guidelines capture the experiences, good practices, activities, and lessons learned of the Working Groups and present them in a comprehensive, cohesive set of documents for all to use and benefit from. The objective is to reuse what works rather than reinvent and to foster consistency across the various WG quality activities and deliverables.

1.1 Motivation for this Guidelines document

As the complexity of W3C specifications and their interdependencies increases, quality assurance becomes even more important to ensuring their acceptance and deployment in the market. There has been a growing awareness and interest in conformance and quality. In approving and initiating the QA Activity, W3C has endorsed the principle that, in order for W3C Web standards to achieve full interoperability and access to all, the quality of the implementation must be given as much attention as the standards' development. The principal factor for improving the quality of implementation is early availability of conformance test materials. (ed note: do we need to say more? Or will the business case and justification stuff be in the Intro document.)

Although not explicitly stated, the W3C Process Document supports the development of conformance test materials.

  1. "[.] groups may produce technical reports, review the work of other groups, develop sample code or test suites, etc." (sec. 3.)
  2. "W3C should make every effort to maintain its Recommendations (e.g., by tracking errata, providing test bed applications, helping to create test suites, etc.)" (sec4.)
  3. In an effort to meet these suggestions and address the implementation requirements of the Process Document, some Working Groups have included the development of conformance materials as part of their CR-exit and PR-entrance criteria (e.g., SVG, UAAG, XSL-FO). This makes sense, since it is natural for test suites and implementations to develop in parallel - each is a help to the development of the other.

There is already a body of contemporary QA experience and activity amongst the Working Groups. The Matrix identifies more than a score of test suites and validators, in various states of development. Moreover, many WGs have already established procedures, techniques and tools for developing test materials (e.g., DOM). It makes sense to capitalize on what has already been done and share that with those who are starting out and those who are already in the process of developing conformance materials.

1.2 Navigating through this document.

The Guidelines in the document are organized chronologically starting with those that apply at the formation of a WG (e.g., charter considerations) and through a progression of guidelines leads the reader through the various process and operational activities necessary in planning, developing, deploying and maintaining conformance materials. This document is applicable to all WGs, including those that are being rechartered or already exist. WGs may already be doing some of these activities and should review the document and in so far as possible incorporate principles and guidelines into their work.

This document includes guidelines or general principles for the development of conformance materials. Each guideline includes:

The checkpoint definitions in each guideline define the processes and operations that need to be implemented in order to accomplish the guideline. Each checkpoint definition includes:

Each checkpoint is intended to be specific enough so that someone can implement the checkpoint as well as verify that the checkpoint has been satisfied.

1.3 Terminology

The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" will be used as defined in RFC 2119 [RFC2119].

Unusual terms in these framework documents are defined when first used, and most generally useful QA-specific terms will eventually be in the QA Glossary [QA-GLOSSARY].

2. Guidelines

Guideline 1. In charters, describe QA goals, criteria, deliverables and resources

In approving and initiating the QA Activity, W3C has endorsed the principle that, in order for W3C Web standards to achieve full interoperability and access to all, the quality of implementation must be given as much attention as the standards' development. The principal factor for improving the quality of implementation is early availability of conformance test materials. Therefore, conformance test materials -- test suites and tools -- must be considered as WG deliverables the same as the standards themselves. Accordingly, they must be addressed in the WG charter.

Checkpoints:

Checkpoint 1.1.Priority 1.In charters include section for test suites/tools plans. Plan to have at least a commitment to "QA level three".

Detailed planning and specification of test materials is inappropriate in the charter, as it binds the WG inflexibly to test materials implementation specifics for which flexibility may be needed. There are numerous possible levels of commitment that a WG could make, based on anticipated WG resources, existence of outside efforts, and the WG's current stage in the process [REC-TRACK].

Based on level of testability of the spec and extent of test materials, a range of possible commitments has been identified.

Level testability of the spec extent of test materials
1. WG plans regular spec reviews for testability following Specification Guidelines
2. WG plans regular spec reviews for testability following Specification Guidelines.Beyond a few normative examples, WG aims to have numerous normative use cases in the body of the Recommendation.
3. WG plans regular spec reviews for testability following Specification Guidelines and have numerous normative use cases in the body of the Recommendation. WG commits that a test suite for the specification, not necessarily complete and thorough, will exist before specification becomes Recommendation.
4. WG plans regular spec reviews for testability following Specification Guidelines and have numerous normative use cases in the body of the Recommendation. WG commits that they review and fully approve a test suite, not necessarily complete and thorough, before specification becomes Recommendation.
5. WG plans regular spec reviews for testability following Specification Guidelines and have numerous normative use cases in the body of the Recommendation. WG commits to reviewing incoming test cases on a continuing basis if needed, including after corresponding specification becomes Recommendation.
6. WG plans regular spec reviews for testability following Specification Guidelines and have numerous normative use cases in the body of the Recommendation. WG commits to reviewing incoming test cases.WG will attempt to generate a complete catalog of test cases before corresponding specification becomes Recommendation.
7. WG plans regular spec reviews for testability following Specification Guidelines and have numerous normative use cases in the body of the Recommendation. WG commits to reviewing incoming test cases.WG insists on a complete catalog and partial implementation of a test suite before corresponding standard becomes Recommendation.
8. WG plans regular spec reviews for testability following Specification Guidelines and have numerous normative use cases in the body of the Recommendation. WG insists on a test suite, not necessarily complete and thorough, and provides an test suite and/or tools, before specification in question goes to Recommendation.
9. WG plans regular spec reviews for testability following Specification Guidelines and have numerous normative use cases in the body of the Recommendation. WG commits to writing test cases themselves and delivers a test harness and suite at some point.
10. WG plans regular spec reviews for testability following Specification Guidelines and have numerous normative use cases in the body of the Recommendation. WG insists on a complete test suite before corresponding standard becomes Recommendation.

This 10-point enumeration is derived from a proposal to the QA mail list, after the 4/2001 QA Workshop.

The further down the scale (i.e., the higher the number) that the WG commits, the better it is for achieving W3C's ultimate goals of interoperability and accessibility.

Checkpoint 1.2.Priority 2.In charter, specify completion and publication of test materials to be a criterion for CR-exit and PR-entrance.

A de facto convention of "CR-exit" seems to be a common process goal amongst a number the existing test suite efforts (e.g., SVG, UAAG,...). This makes sense, as it is natural for test suites and implementations to develop in parallel -- each is a help to the development of the other. And, the Process Document does address implementation requirement:

Checkpoint 1.3.Priority 1.In charter, address QA staffing commitments.

There will be at least some staffing required from the WG, for any of the acceptable options presented so far. Depending upon the general intent and plan, and how the test suite will be built, the commitment can range from minimal to significant. See "Examples & Techniques" [@@EX-TECH].

Guideline 2. In charter, address where and how conformance test materials will be produced.

There are several possibilities for producing conformance test materials:

Checkpoints:

Checkpoint 2.1.Priority 1.In Charter, specify level of involvement in the ongoing evolution of test suites and tools.

There are good reasons why external parties should have a strong participation during the building of the materials. But even if the test suite is being developed outside of W3C, Working Group needs to ensure it is completed according to the level Working Group committed to in the charter.

Checkpoint 2.2.Priority 2.Ensure secure and reliable repository location for future test materials.

This doesn't necessarily mean that test suite/tools should physically reside in the W3C, though it is highly recommended:

  1. At the point that test materials become operationally deployed, then challenges to the correctness of both the test materials and specifications (RECs) normally increase, and the WG is the best venue for initial processing and adjudication of such queries against both.
  2. Further to #1, it is more likely that specification (REC) errata processing and test suite maintenance can be kept synchronized if both responsibilities reside in the same body, the WG.
Once the materials have reached some advanced milestone of maturity and development (e.g., operationally usable), W3C needs to assure that:

If the WG ceases to operate, then other responsible for ongoing interpretation and maintenance (errata) of the standard should be designated to ensure continued availability of the test suite.

Guideline 3. Ensure that goals and scope of QA development are clear

Common frameworks for W3C QA activities are obviously shaped by the high-level goals for development and deployment of test materials. Before further developing the framework guidelines, it is useful to state some of these high-level goals.

The over-arching goal is stated in the QA Activity Statement:

Specific goals of test materials development include:

The goals of QA work within WG do not encompass:

Checkpoints:

Checkpoint 3.1.Priority 2.Ensure that test materials are documented, self contained and reusable by accredited third parties for testing purposes.

While certification may have a positive quality assurance role in standards-based Web environments, W3C does not presently intend to initiate or offer any certification services. Nevertheless, developed test materials can be used by external parties including certification services. Factors of rigor, objectivity, and defensibility are key determinants of usability for certification, and these same properties are positive assets of any test materials.

Checkpoint 3.2.Priority 3.Specify policy for branding materials if applicable.

Certification aside, some W3C activities support "branding" -- use of a conformance icon to indicate some level of standard compliance. Examples: HTML validation, CSS validation, WCAG compliance. While these content branding schemes are relatively non-controversial, there are nevertheless some serious issues which should be addressed -- particularly in the contexts of user-agent and API conformance -- before any branding-related goals are articulated.

Checkpoint 3.3.Priority 2.Provide disclaimer regarding use of the test materials for compliance verification.

Although tests suites may be used for compliance verification, users must be aware of the fact that

  1. passing all of the tests does not guarantee full compliance of implementation to specification
  2. failing test suite means failing tests for specific feature they target

The last item requires some elaboration. Depending on the nature of the test, it may actually be sufficient to judge non-compliance of the subject. Based on taxonomy the following is valid:

Guideline 4. Allocate staff for QA process.

Before starting QA development, Working Group has to appoint several key participants of QA process.

Checkpoints:

Checkpoint 4.1.Priority 2.Appoint test QA moderator.

Depending on the plan for origin of the test suite WG can appoint test moderator from

When test suite is developed outside of Working Group, a moderator usually already exists where test suite coming from. It is recommended to invite moderator to become WG member, since he/she needs access to all WG materials and resources. See also section on transferring test suite from outside W3C.

Checkpoint 4.2.Priority 2.Designate a task force or another entity to maintain the test suite, as well as maintenance procedures.

It is recommended that the errata task force be involved in test suite maintenance, since this will help to keep test suite and specification in sync. Maintenance group shall establish tracking system to track issues/changes to the test suite.

Guideline 5. Synchronize specification and test materials developments

There are several strong benefits from starting synchronizing of specification and test materials developments as early as possible:

Checkpoints:

Checkpoint 5.1.Priority 3.Introduce intermediate milestones for test test materials completion

These milestone can be naturally bound to Working Drafts/Candidate Recommendation publications.

Checkpoint 5.2.Priority 3.Synchronize latest version of the spec draft and latest version of the published test materials.

Checkpoint 5.3.Priority 2.Support spec versioning/errata in the test materials

Structure of the test materials should allow versioning/errata to enable verification of compliance with different versions of the specification.

Guideline 6. Produce QA process document.

A Working Group QA process document is a central QA document describing how testing process is set up within the WG. Examples of the test process document can be found at DOM TS process, XML Schema TS process, XML Protocol TS process, etc. It is a required part of QA process in a WG and should address the following questions:

Checkpoints:

Checkpoint 6.1.Priority 2.Review Technical Guidelines before writing QA Process document and starting test materials development.

Taking into account the nature of the future standard, before starting test suite development, test suite implementer (WG or 3rd party) should look at Technical Guidelines and decide what approach to take for test materials organization, tests criteria, etc.

Checkpoint 6.2.Priority 2.In QA Process document, explain contribution process.

The process document must describe where to submit test materials and whom to notify (moderator) of submission. Contribution process SHOULD describe the format of contributed material, it may contain validation harness, utilities that facilitates tests creation, templates, etc.

Checkpoint 6.3.Priority 2.In QA Process document, define the licenses applicable to submitted and published test materials.

Working Group can choose either the W3C Document or the W3C Software license. If applicable, W3C Document license is recommended.

Checkpoint 6.4.Priority 2.In QA Process document, describe means for communication regarding test suites.

WG needs to choose mailing list to send all comments and announcements to.

Checkpoint 6.5.Priority 2.In QA Process document, describe review procedures for submitted test materials

The following example may serve as a guideline for tests review process

  1. Each contribution should be reviewed by at least one vendor other than submitter for it's eligibility. Any vendor involved in creation of WG test collection other then submitter may review tests being contributed.
  2. Eligibility of tests should be judged by:
  3. After review results should be sent back to submitter for his feedback.
  4. In case of disagreement, request for clarification should go to WG, tests in question should be left off the test suite till W3C WG clarifies the issue.
  5. Only those tests that are agreed to be correct by reviewers and submitter or approved by W3C WG clarification should be published.

Checkpoint 6.6.Priority 2.In QA Process document, describe the way test materials will be published and point to the corresponding web page.

Checkpoint 6.7.Priority 3.In QA Process document, describe how vendors can publish test results for their products, if applicable.

It is recommended to use one of practices for publishing test materials, described in Technical Guidelines. WG can encourage vendors to publish results for their implementations including/describing test harness that would allow anyone to reproduce the results.

Guideline 7. Plan transfer of test materials to W3C if needed

As mentioned before in the notes to checkpoint for test materials repository, it is recommended that test suite resides in the W3C. If test development was held outside by some external entity/group (ET) and Working Group(WG) together with ET decided to move test materials (TM) to W3C, the following checkpoints outline necessary steps:

Checkpoints:

Checkpoint 7.1.Priority 2.If plan to transfer the test materials to W3C, perform assessment of their quality

Checkpoint 7.2.Priority 2.If plan to transfer the test materials to W3C, identify staff resources to meet requirements.

Checkpoint 7.3.Priority 1.If plan to transfer the test materials to W3C, resolve IPR questions and reach agreement with external party that produced test materials.

Checkpoint 7.4.Priority 2.If plan to transfer the test materials to W3C, update QA commitments in the charter if necessary

The following procedure is recommended:

Before transfer WG with the help of QA WG:

During transfer

After transfer, initial process setup

First W3C public release of new TM

After first public release, TM enters maintenance phase which is described below.

Guideline 8. Plan for test materials maintenance

Once test suite development is initiated, its maintenance requires an equally significant amount of resources. Resources allocation is needed as prescribed by checkpoint above.

Checkpoints:

Checkpoint 8.1.Priority 3.Maintain contribution and review procedures throughout entire test suite's and standard's life cycle.

Checkpoint 8.2.Priority 2.In QA Process, specify procedure for updates of the test materials with new erratas/specification versions

The following procedure may serve as an example:

Checkpoint 8.3.Priority 2.In QA Process, identify the communication channel and procedure for appeals of tests validity

The following procedure may serve as example:

[Active issue]If there is still active work on the test suite even if the WG is not re-chartered to handle the work, then it is up to the Director to determine how this work should be done. Some options include:

3. WG relationship to QA Activity

The QA Activity anticipates different relationships to the various Working Groups. On the one hand, the QA-specific needs of the Working Groups will vary depending upon their resources and experience, as well as their stage in the Recommendation track. On the other hand, the W3C processes may evolve so that QA-specific process requirements are tied to milestones in the Recommendation track.

Potential relationships between Working Groups and the QA Activity include:

Key determinants of the WG/QA relationship will include level of available QA Activity resources, and the evolution of W3C policy on QA requirements.

3.0.1 Liaison and consultation.

In an assistive role, the QA Activity provides consultation to Working Groups in planning and building or acquiring of test materials. This should be the baseline relationship, applicable to all Working Groups. Consultation may be provided in formal and informal ways. Informal questions MAY be sent to www-qa@w3.org. In order to make assistance quick and accurate the following process is introduced for formal requests:

3.0.2 Active reviews.

The QA Activity provides detailed review and assessment of Working Group QA work and test materials per request from Working Group. QA Working Group also performs detailed review of the charter and specifications of the Working Group per request from the latter. The following procedure SHOULD be used inside QA Working Group for review of materials presented for review by Working Group. QA Working Group MAY pro-actively initiate reviews, provided it has resources to do so.

3.0.3 Adjudicate entry and exit criteria.

Although there are currently no W3C process requirements for QA-specific entry and exit criteria at milestones on the Recommendation track, Working Groups could define such for themselves. The QA Working Group may help identify those.

3.0.4 QA resources supplement.

QA Working Group could in some cases, provide resources for the planning and building of test materials for some Working Groups.

3.0.5 Resolution of external QA requests

The QA Working Group may receive requests from external parties or persons to resolve QA related questions that involve one or several W3C standards. The Chair of QA WG SHALL assign task team of the QA WG members to coordinate and drive the question to resolution. Assigned task team SHALL

4. Conformance

This section defines conformance of Working Group processes and operations to the requirements of this specification. The requirements of this specification are detailed in the checkpoints of the preceding "Guidelines" chapter of this specification, and apply to the Working Group QA-related documents and deliverables required by this specification.

This section defines three levels of conformance to this specification:

A Working Group conforms to the "QA Framework: Process & Operational Guidelines" at level X if the Working Group meets at least all Conformance Level X requirements.

To make an assertion about conformance to this document, specify:

Example:

This QA processes and operations of this Working Group [???] conform to W3C's "QA Framework: Process & Operational Guidelines", available at [...tbd...], Level 2.

5. Acknowledgments

The following QA Working Group and Interest Group participants have contributed significantly to the content of this document:

@@Any additions? Any deletions?@@

6. References

[@@Ed. note: This has not yet been cut down from original all-in-one Framework document. It should be edited to only include those references which actually pertain to this document.]

EXTERN-TA
QA activity email thread about third-party participation in test materials production.
MATRIX
W3C-wide conformance activity survey covering all the WGs, "The Matrix".
PROCESS
W3C Process Document, 19 July 2001.
TAXONOMY
QA Activity test taxonomy, a classification scheme for conformance test materials.
QAIG
Quality Assurance Interest Group of the W3C QA Activity.
QAWG
Quality Assurance Working Group of the W3C QA Activity.
DOM WG TS
Process document for DOM WG Test suite.
REC-TRACK
Stages and milestones in the W3C Recommendation Track, per the Process Document (section 5.2).
RFC2119
Key words for use in RFCs to Indicate Requirement Levels, March 1997.
SVGTEST
SVG Working Group's test suite resource page.
WCAG10
Web Content Accessibility Guidelines, version 1.0, W3C Recommendation, 5 May 1999.
WG-QA-RANGE
Email proposal by David Marston, on the QA public mail list, for range of WG commitment levels to conformance test materials production.
XMLTEST
OASIS XML Conformance TC's XML test suite resource page.
XSLT-TEST
OASIS XML Conformance TC's XSLT/Xpath test suite resource page.
QA-TECH-GUIDE
QA Framework: Technical Guidelines(not written).
QA-SPEC-GUIDE
QA Framework: Specification Guidelines(not written).