W3C

QA Framework: Operational Guidelines

W3C Working Draft 07 May 2002

This version:
http://www.w3.org/QA/WG/2002/framework-20020507/qaframe-ops
Latest version:
http://www.w3.org/QA/WG/qaframe-ops
Previous version:
http://www.w3.org/QA/WG/2002/framework-20020405/qaframe-ops
Editors:
Kirill Gavrylyuk (kirillg@microsoft.com)
Dimitris Dimitriadis (dimitris@ontologicon.com)
Lofton Henderson (lofton@rockynet.com)
Lynne Rosenthal (lsr@nist.gov)
Contributors:
See Acknowledgments.

Abstract

This document defines a common Operational Framework for building conformance test materials for W3C specifications. It presents operational and procedural guidelines for groups undertaking conformance materials development. This document is one in a family of Framework documents of the Quality Assurance (QA) Activity, which includes the other existing or in-progress specifications: Introduction; Specification Guidelines; and, Test Guidelines.

Status of this document

This WG version is substantially identical to 20020515 publication version.

This section describes the status of this document at the time of its publication. Other documents may supersede this document. The latest version of this document series is maintained at the W3C.

This document is a W3C Working Draft (WD), made available by the W3C Quality Assurance (QA) Activity for discussion by W3C members and other interested parties. For more information about the QA Activity, please see the QA Activity statement.

This is the second public Working Draft, and is significantly changed since the Previous Version. It incorporates the discussions from several subsequent QA Working Group [QAWG] teleconferences, as well as the second QAWG face-to-face meeting. In particular the QAWG has now examined, debated, and agreed upon the individual checkpoints and their priorities. For details, please see Change History.

This version supersedes all previous drafts. It is expected that updated WD versions of this document will be produced regularly, along with other members of the Framework documents family. Future progression of this document beyond Working Draft is possible, but has not yet been determined.

This part of the Framework document family now has an accompanying "QA Framework: Operational Examples & Techniques" (presently at first public Working Draft). Some of the lengthier examples and "how to" text of this current guidelines document version will be moved to the "Examples & Techniques" document, in future versions.

Please send comments to www-qa@w3.org, the publicly archived list of the QA Interest Group [QAIG]. Please note that any mail sent to this list will be publicly archived and available, do not send information you wouldn't want to see distributed, such as private data.

Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress".

A list of current W3C Recommendations and other technical documents can be found at http://www.w3.org/TR/.

Table of contents

1. Introduction
    1.1 Motivation for this Guidelines document
    1.2 Goals and scope of QA development
    1.3 Navigating through this document.
    1.4 Priorities
    1.5 Terminology
2. Guidelines
         G 1. Integrate Quality Assurance into Working Group activities.
         G 2. Define resources for Working Group QA activities.
         G 3. Synchronize QA activities with the milestones for WG deliverables.
         G 4. Define the QA process.
         G 5. QA Process: Plan test materials development.
         G 6. QA Process: Plan test materials publication.
         G 7. Plan the transfer of test materials to W3C if needed.
         G 8. Plan for test materials maintenance.
3. WG relationship to QA Activity
    3.1 Liaison and Consultation
    3.2 Active Reviews
    3.3 Adjudicate entry and exit criteria
    3.4 QA resource supplement
    3.5 Resolution of external QA requests
4. Conformance
    4.1 Conformance definition
    4.2 Conformance disclaimer
5. Acknowledgments
6. References
7. Change History

An appendix to this document [OPS-CHECKLIST] presents all checkpoints in a tabular form, for convenient reference.


1. Introduction

This document describes the process and operational activities for building conformance test suites and tools for W3C specifications. The primary goal is to assist the W3C Working Groups (WGs) by providing guidelines for the planning, development and deployment of conformance test materials. In this guideline, the term "conformance test materials" refers conformance test suites, validation tools, conformance checklists, and any other materials that are used to check or indicate conformance.

This document is part of a family of QA Framework documents designed to improve the quality of W3C specifications as well as their implementation by solidifying and extending current quality practices within the W3C. The QA Framework documents are:

The guidelines are intended for all Working Groups as well as developers of conformance materials for W3C specifications. Not only are the Working Groups the consumers of these guidelines, they are also key contributors. The guidelines capture the experiences, good practices, activities, and lessons learned of the Working Groups and present them in a comprehensive, cohesive set of documents for all to use and benefit from. The objectives are to reuse what works rather than reinvent and to foster consistency across the various Working Group quality activities and deliverables.

1.1 Motivation for this Guidelines document

As the complexity of W3C specifications and their interdependencies increases, quality assurance becomes even more important to ensuring their acceptance and deployment in the market. There has been a growing awareness and interest in conformance and quality. In approving and initiating the QA Activity, W3C has endorsed the principle that in order for W3C Web standards to achieve full interoperability and access to all, the quality of the implementation must be given as much attention as the standards' development. The principal factor for improving the quality of implementation is early availability of conformance test materials.

Although not explicitly stated, the W3C Process Document supports the development of conformance test materials.

[...] groups may produce technical reports, review the work of other groups, develop sample code or test suites, etc." (see Process Document, section 3.)

W3C should make every effort to maintain its Recommendations (e.g., by tracking errata, providing test bed applications, helping to create test suites, etc.) (see Process Document, section 5.2.5, "Ongoing work".)

In an effort to meet these suggestions and address the implementation requirements of the Process Document, some Working Groups have included the development of conformance materials as part of their CR-exit and PR-entrance criteria. Examples include Scalable Vector Graphics (SVG), User Agent Accessibility Guidelines (UAAG), and Extensible Style Language - Formatting Objects (XSL-FO). This makes sense, since it is natural for test suites and implementations to develop in parallel - each is a help to the development of the other.

There is already a body of contemporary QA experience and activity amongst the Working Groups. The Matrix identifies more than a score of test suites and validators, in various states of development. Moreover, many Working Groups have already established procedures, techniques and tools for developing test materials (e.g., Document Object Model - DOM). It makes sense to capitalize on what has already been done and share that with those who are starting out and those who are already in the process of developing conformance materials.

1.2 Goals and scope of QA development

Common frameworks for W3C QA activities are obviously shaped by the high-level goals for development and deployment of test materials. Before further developing the operational framework guidelines, it is useful to state some of these high-level goals.

The over-arching goal is stated in the QA Activity Statement:

Specific goals of test materials development include:

The goals of QA work within a Working Group do not encompass:

While certification may have a positive quality assurance role in standards-based Web environments, W3C does not presently intend to initiate or offer any certification services.

1.3 Navigating through this document.

The Guidelines in the document are organized chronologically. The document starts with the guidelines that apply at the formation of a Working Group (e.g., charter considerations) and continues leading the reader through the various process and operational activities necessary in planning, developing, deploying and maintaining conformance materials. This document is applicable to all Working Groups, including those that are being rechartered or already exist. Working Groups may already be doing some of these activities and should review the document and in so far as possible incorporate principles and guidelines into their work.

This document employs the WAI (Web Accessibility Initiative) model for representing guidelines or general principles for the development of conformance materials. See, for example, Web Content Accessibility Guidelines. Each guideline includes:

The checkpoint definitions in each guideline define the processes and operations that need to be implemented in order to accomplish the guideline. Each checkpoint definition includes:

Each checkpoint is intended to be specific enough so that someone can implement the checkpoint as well as verify that the checkpoint has been satisfied.

A separate appendix to this document [OPS-CHECKLIST] presents all checkpoints in a tabular form, for convenient reference.

1.4 Priorities

High quality and timely production of test materials are the key requirements to produce a high quality interoperable standard. Therefore each checkpoint has a priority level assigned by the QA Working Group based on the checkpoint's impact on the quality and timing of the test materials produced by a Working Group.

[Priority 1]
A Working Group must satisfy this checkpoint. Otherwise the test materials will not be produced by the time they are required to ensure the quality of the standard, or they may not be usable. Satisfying this checkpoint is a basic requirement to ensure quality and interoperability of the standard.
[Priority 2]
A Working Group should satisfy this checkpoint. Otherwise it may be difficult to produce high quality test materials by the time they are required to ensure the quality of the standard. Satisfying this checkpoint will significantly improve the interoperability of the standard.
[Priority 3]
A Working Group may address this checkpoint. Satisfying this checkpoint should facilitate maintaining high quality of the test materials and ensuring the quality of the standard. Satisfying this checkpoint will improve the interoperability of the standard.

1.5 Terminology

The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" will be used as defined in RFC 2119 [RFC2119].

Unusual terms in these framework documents are defined when first used, and most generally useful QA-specific terms will eventually be in the QA Glossary [QA-GLOSSARY].

2. Guidelines

Guideline 1. Integrate Quality Assurance into Working Group activities.

In approving and initiating the QA Activity, W3C has endorsed the principle that in order for W3C Web standards to achieve full interoperability and access to all, the quality of implementation must be given as much attention as the standards' development. The principal factor for improving the quality of implementation is early availability of conformance test materials (TM). Therefore, conformance test materials -- test suites and tools -- must be considered as Working Group deliverables the same as the standards themselves. Accordingly, they must be addressed in the Working Group charter.

Note. The following checkpoints require actions "In the Charter." New Working Groups -- i.e., those that are writing and submitting their Charters for approval -- must satisfy the checkpoints by making the specified commitments in the charter. Existing Working Groups may satisfy the checkpoints by amendment to their charter, or in other ways, the latter to be specified in QA Framework: Examples & Techniques [OPS-EXTECH].

Checkpoints:

Checkpoint 1.1. In the Charter, specify the level of commitment to QA. Plan to have at least a commitment to "QA level three". [Priority 1]

Detailed planning and specification of test materials is inappropriate in the charter because it inflexibly binds the Working Group to test materials implementation specifics for which flexibility may be needed. There are numerous possible levels of commitment that a Working Group could make based on anticipated Working Group resources, existence of outside efforts, and the Working Group's current stage in the process [REC-TRACK].

Based on the level of testability of the specification and the extent of test materials, a range of possible commitments has been identified.

Level testability of specification extent of test materials
1. Working Group plans regular specification reviews for testability, following Specification Guidelines
2. In addition to the regular specifications reviews mentioned in the previous level, Working Group commits to develop a set of test assertions, not necessarily complete, before beginning development of a test suite. Working Group commits that test suite for the specification, not necessarily complete and thorough, will exist before specification becomes Recommendation.
3. In addition to regular specification reviews, a Working Group aims to have numerous normative use cases in the body of the Recommendation. In addition to the commitment of the previous level, a Working Group ensures that the test suite produced is reviewed by the Working Group or externally and the review process meets the criteria established in this document.
4. Same as above. Working Group commits to establish and maintain the test cases contribution and review process and will produce and review the test suite, not necessarily complete and thorough, before the specification becomes Recommendation.
5. In addition to the commitments from the previous level, Working Group will derive the list of testable assertions as an addendum to specification by the time it becomes Recommendation. Same as above
6. Same as above In addition to the commitment for the previous level, a Working Group insists on a complete test suite before corresponding standard becomes Recommendation.
7. Same as above In addition to the commitment for the previous level, Working Group plans to maintain the test suite even after Specification becomes recommendation. Working Group plans to support versioning and errata in the test suite and review appeals.

This seven-point enumeration is derived from the proposal to the QA mail list, after the 4/2001 QA Workshop.

The further down the scale (i.e., the higher the number) that the Working Group commits, the better it is for achieving W3C's ultimate goals of interoperability and accessibility.

Examples & Techniques for checkpoint.

Checkpoint 1.2. In the Charter, plan at least for the "QA level five". [Priority 2]

This checkpoint advances the level of QA commitment required by the previous checkpoint and therefore supersedes it. Satisfying this level of requirements for the specification and the test materials significantly increases confidence in the interoperability of the standard.

Examples & Techniques for checkpoint.

Checkpoint 1.3. In the Charter, plan for the "QA level seven". [Priority 3]

This checkpoint advances the level of QA commitment required by the previous checkpoint and therefore supersedes it. Satisfying this level of requirements for the specification and the test materials further increases confidence in the interoperability of the standard.

Examples & Techniques for checkpoint.

Checkpoint 1.4. In the Charter, enumerate QA deliverables and expected milestones. [Priority 1]

Among QA deliverables, a Working Group may produce a set of normative use cases, produce or acquire test suite, validators, harnesses.

The W3C Process document requires for Charters, that deliverables be identified with milestones. It is vital that the milestones for QA deliverables are synchronized and even serve as criteria for WG technical deliverables (specifications).

Examples & Techniques for checkpoint.

Checkpoint 1.5. In the Charter, associate QA criteria with WG milestones. [Priority 2]

A de facto convention for entrance into PR phase -- and by implication, completion of CR phase, if there was a CR phase -- seems to be a common process goal amongst a number of the existing test suite efforts. For example, SVG, UAAG, and others have defined this criterion. It is natural for test suites and implementations to develop in parallel -- each is a help to the development of the other. And, the Process Document does address implementation requirement:

"Advancement of a technical report to Candidate Recommendation is an explicit call for implementation experience to those outside of the related Working Groups or the W3C itself." (see Process Document, section 5.2, about "Candidate Recommendation".)
"Preferably, the Working Group should be able to demonstrate two interoperable implementations of each feature." (see Process Document, section 5.2.4, about "Entrance Criteria".)

Examples & Techniques for checkpoint.

Guideline 2. Define resources for Working Group QA activities.

It is imperative to address QA resource requirements from the very beginning of the Working Group formation. Starting from the Working Group Charter and later in the Call For Participation special attention is required to the staffing and other resource requirements for a successful QA work.

Note. The following checkpoints require actions "In the Charter." New Working Groups -- i.e., those that are writing and submitting their Charters for approval -- must satisfy the checkpoints by making the specified commitments in the charter. Existing Working Groups may satisfy the checkpoints by amendment to their charter, or in other ways, the latter to be specified in QA Framework: Examples & Techniques [ OPS-EXTECH ].

Checkpoint 2.1. In the Charter, address where and how conformance test materials will be produced. [Priority 1]

There are several possibilities for producing conformance test materials:

There are good reasons why external parties should have a strong participation during the building of the materials. But even if the test suite is being developed outside of W3C, the Working Group needs to ensure it is completed according to the QA level to which the Working Group committed in its charter.

Examples & Techniques for checkpoint.

Checkpoint 2.2. In the Charter, address QA staffing commitments. [Priority 1]

There will be at least some staffing required from the Working Group, for any of the acceptable options presented so far. Depending upon the general intent and plan, and how the test suite will be built, the commitment can range from minimal to significant.

Examples & Techniques for checkpoint.

Checkpoint 2.3. In the Call for Participation, request allocation of QA resources to the Working Group. [Priority 1]

Once the Charter is prepared, the Director sends a Call for Participation to the Advisory Committee. At this point AC Representatives are asked to provide information about amount and type of resources their organization plans to allocate for the particular Working Group. Explicit indication of resources dedicated for Quality Assurance is required to assess Working Group capabilities against deliverables and milestones declared in the Charter.

Examples & Techniques for checkpoint.

Guideline 3. Synchronize QA activities with the milestones for WG deliverables.

There are several strong benefits from starting synchronization of the specification and test materials development as early as possible:

Checkpoints:

Checkpoint 3.1. Synchronize the publication of QA deliverables and the specification's drafts. [Priority 2]

In the checkpoint above we discussed declaring a QA criteria for each Working Group milestone. The Working Group milestones are usually bound to the specification's drafts. A natural part of the QA criteria for such a milestone would be publication of the updated test materials or other QA deliverables. From another point of view, each Working Draft of the specification is a new version of the document that requires an update of all dependencies like test materials.

Examples & Techniques for checkpoint.

Checkpoint 3.2. Support specification versioning/errata in the QA deliverables. [Priority 3]

This checkpoint is an extension of the previous one to the specification milestones after it reaches Recommendation status. If a Working Group plans to maintain the test suite after the specification becomes recommendation (see the Table of WG Commitment Levels), it has to allow versioning/errata in the structure of the test materials. There will be products that support different versions or even different errata levels of the specification, users should be able to verify product's compliance with specific level of errata/version.

Examples & Techniques for checkpoint.

Guideline 4. Define the QA process.

A Working Group QA process encompasses how the quality processes are set up within the Working Group. Documented examples of the QA Process can be found at DOM TS process, XML Schema TS process, XML Protocol TS process, etc.

Checkpoints:

Checkpoint 4.1. Appoint a QA moderator. [Priority 1]

Before starting test materials development, a Working Group has to appoint several key participants for its QA process.

The role of test moderator is to manage various QA processes. Depending on the plan for the origin of the test suite a Working Group can

When a test suite is developed outside of the Working Group, a moderator often already exists in the external group that is the source of the test suite. It is recommended to invite the moderator to become a Working Group member, since he/she needs access to all Working Group materials and resources. See also section on transferring test suite from outside W3C.

Examples & Techniques for checkpoint.

Checkpoint 4.2. Appoint a QA task force. [Priority 2]

Dependent on the level of involvement of a Working Group in the test materials development, a Working Group may need to appoint QA task force in addition to the test moderator. Such a task force can take responsibility for a QA framework development, test materials development, review of contributions, maintenance. See below for details on each of these activities.

Examples & Techniques for checkpoint.

Checkpoint 4.3. Produce the QA Process document. [Priority 1]

QA Process document describes:

Examples & Techniques for checkpoint.

Checkpoint 4.4. In the QA Process document, define means for QA-related communication. [Priority 2]

Working Group needs to choose mailing list to send all comments and announcements to.

Examples & Techniques for checkpoint.

Checkpoint 4.5. In the QA Process document, specify the framework for test materials development. [Priority 1]

The QA framework describes how to develop, document and use the tests.

Examples & Techniques for checkpoint.

Checkpoint 4.6. Specify a policy for branding materials. [Priority 3]

Some W3C activities support "branding" -- use of a conformance icon to indicate some level of standard compliance. Examples: HTML validation, CSS validation, WCAG compliance. While these content branding schemes are relatively non-controversial, there are nevertheless some serious issues which should be addressed -- particularly in the contexts of user-agent and API conformance -- before any branding-related goals are articulated.

Examples & Techniques for checkpoint.

Guideline 5. QA Process: Plan test materials development.

As we described earlier, a Working Group may be involved in test materials development at different levels. Nevertheless at any level a Working Group needs to have clear understanding of what QA framework it will use and how to insure the quality and usability of the test materials themselves. It is recommended that before starting a test suite development, the test suite implementer (Working Group or 3rd party) takes a look at Test Materials Guidelines and decide what approach to take for test materials organization, test criteria, etc..

Checkpoints:

Checkpoint 5.1. In the test framework, ensure test materials are documented and usable for the intended purpose. [Priority 1]

While W3C does not have any plans to offer any certification services, developed test materials can be used by external parties including certification services. Factors of rigor, objectivity, and defensibility are key determinants of any test materials.

Examples & Techniques for checkpoint.

Checkpoint 5.2. In the QA Process document, define a contribution process. [Priority 2]

The process document must describe where to submit test materials and whom to notify (e.g., moderator) of a submission. The contribution process describes the format of contributed material, it may contain validation harness, utilities that facilitates tests creation, templates, etc.

Examples & Techniques for checkpoint.

Checkpoint 5.3. In the QA Process document, define the licenses applicable to submitted test materials. [Priority 1]

A Working Group needs to publish a license agreement under which vendors and other external parties may submit test materials. Examples of such license agreements may be found at [@@link.....]. Contributors may have to negotiate license agreement for their specific needs.

Examples & Techniques for checkpoint.

Checkpoint 5.4. In QA Process document, define review procedures for submitted test materials. [Priority 2]

Once the contribution process is defined, Working Group must establish a review procedure for the tests being contributed.

Examples & Techniques for checkpoint.

Checkpoint 5.5. Test materials review process includes but not limited to checking for accuracy, scope and clarity of the tests. [Priority 1]

The following example meets the review requirements:

  1. Each contribution should be reviewed for its eligibility by at least one vendor other than the submitter. Any vendor involved in the creation of the Working Group test collection other than the submitter may review contributed tests.
  2. Eligibility of tests should be judged by:
  3. After review, results should be sent back to the submitter for feedback.
  4. In case of disagreement, a request for clarification should go to Working Group. Tests in question should be left out of the test suite until the W3C Working Group clarifies the issue.
  5. Only those tests that are agreed to be correct by reviewers and the submitter, or that are approved by W3C Working Group clarification, should be published.

Examples & Techniques for checkpoint.

Guideline 6. QA Process: Plan test materials publication.

Once the test suite development is in progress, a Working Group needs to publish the test suite drafts.

Checkpoints

Checkpoint 6.1. Ensure a secure, reliable, freely accessible repository location for test materials. [Priority 1]

This doesn't necessarily mean that test suite/tools should physically reside in the W3C, although that is a recommended way of meeting the checkpoint, because:

  1. At the point that test materials become operationally deployed, then challenges to the correctness of both the test materials and specifications normally increase, and the Working Group is the best venue for initial processing and adjudication of such queries against both.
  2. Further to #1, it is more likely that technical report (Recommendation) errata processing and test suite maintenance can be kept synchronized if both responsibilities reside in the same body, the Working Group.
Once the materials have reached some advanced milestone of maturity and development (e.g., operationally usable), W3C needs to assure that:

If the Working Group ceases to operate, then successors responsible for ongoing interpretation and maintenance (errata) of the standard should be designated to ensure continued availability of the test suite.

Examples & Techniques for checkpoint.

Checkpoint 6.2. In the QA Process document, define the licenses applicable to published test materials. [Priority 1]

If test materials are produced within the W3C, the Working Group is recommended to use the W3C Document License. There are several reasons why the Document License is preferred:

If the Document License is not applicable and a user needs to modify the test materials in order to use them, the W3C Software License can be used. For publishing test materials acquired from an external group, other licenses may be applicable (see transferring test suite).

Examples & Techniques for checkpoint.

Checkpoint 6.3. In the QA Process document, describe how the test materials will be published and point to the corresponding web page. [Priority 2]

If the test materials are to be published on the W3C site, it is recommended to locate them within the corresponding activity domain. It is strongly recommended not to publish test materials in the TR space for the following reasons:

It is recommended to use one of the practices for publishing test materials, described in Test Materials Guidelines.

Examples & Techniques for checkpoint.

Checkpoint 6.4. Provide a disclaimer regarding the use of the test materials for compliance verification. [Priority 1]

Although tests suites may be used for compliance verification, the Working Group should make users aware of the fact:

  1. passing all of the tests does not guarantee full compliance of an implementation to the specification
  2. failing the test suite means failing tests for the specific feature they target

Examples & Techniques for checkpoint.

Checkpoint 6.5. In the QA Process document, describe how the test results for the products can be published. [Priority 2]

Publishing test results for the products implementing the technical specification is useful for users, vendors, specification authors and the test suite quality itself. Admittedly, taking responsibility to publish test results for other vendor's product can be a problem. In order to solve like difficulties, the Working Group may encourage vendors to publish results for their implementations themselves. Such publication should include or describe a test harness that would allow anyone to reproduce the results. The Working Group may provide the web space to publish collective results.

Examples & Techniques for checkpoint.

Guideline 7. Plan the transfer of test materials to W3C if needed.

As stated before in the notes to the checkpoint for test materials repository, it is recommended that test materials reside in the W3C. If test development was done outside by some external entity or group (EG), and the Working Group (WG) together with EG decided to move test materials (TM) to W3C, the following checkpoints define the requirements for the Working Group.

Checkpoints:

Checkpoint 7.1. If transfer of the test materials to W3C is planned, perform an assessment of their quality. [Priority 2]

This checkpoint is a reflection of the checkpoint for the test review process. During test materials review, the Working Group must follow the criteria set in this document.

Examples & Techniques for checkpoint.

Checkpoint 7.2. If transfer of the test materials to W3C is planned, identify sufficient staff resources to meet the requirements. [Priority 1]

This checkpoint is essentially a reflection of the checkpoint 2.2 on the specific case of transferring the test materials from the outside to the W3C.

Examples & Techniques for checkpoint.

Checkpoint 7.3. If transfer of the test materials to W3C is planned, resolve IPR questions and reach agreement with the external party that produced test materials. [Priority 1]

The following procedure is an example that meets these requirements.

Legend for the procedure:

EG
the external group or entity
QAWG
the QA Working Group
TM
the test materials
WG
the subject Working Group

Before the transfer, WG with the help of QAWG:

During transfer:

After transfer, initial process setup:

First W3C public release of the new TM:

After the first public release, the TM enter the maintenance phase which is described below.

Examples & Techniques for checkpoint.

Guideline 8. Plan for test materials maintenance.

Once test materials development is initiated, its maintenance requires an equally significant amount of resources.

Checkpoints:

Checkpoint 8.1. Maintain the contribution and review procedures throughout the entire life cycle of the test materials and the standard itself. [Priority 3]

Examples & Techniques for checkpoint.

Checkpoint 8.2. In the Working Group's QA process document, specify a procedure for updates of the test materials to track new errata/specification versions. [Priority 2]

The following procedure is an example that meets the requirements:

Examples & Techniques for checkpoint.

Checkpoint 8.3. In the Working Group's QA process document, identify the communication channel and procedure for appeals of tests validity. [Priority 2]

The following procedure is an example that meets the requirements:

If there is still active work on the test suite even if the Working Group is not re-chartered to handle the work, then it is up to the Director to determine how this work should be done. Some options include:

Examples & Techniques for checkpoint.

3. WG relationship to QA Activity

The QA Activity works closely with other W3C WGs providing assistance and expertise in helping them achieve their QA goals and deliverables. The QA Activity anticipates different relationships with the various WGs depending on the QA-specific needs of a particular Working Group. For example, the resources and experience of Working Group members as well as their stage in the Recommendation track may influence the type and level of collaboration between the Working Group and QA Activity. Potential relationships between Working Groups and the QA Activity include:

The QA Activity strives to make their expertise available to the Working Groups, responding to requests and providing assistance on an as-needed basis. The degree of assistance and participation of the QA Activity members may be determined on a case-by-case basis, but a consultancy role should almost always be possible. Key determinants of the WG/QA Activity relationship will include level of available QA Activity resources and the evolution of W3C policy on QA requirements.

3.1 Liaison and Consultation

In an assistive role, the QA Activity provides consultation to all Working Groups with respect to planning, building, and/or acquiring test materials. Consultation may be provided informally in response to questions sent to the QA Activities IG mail exploder, www-qa@w3.org or formally following the QA Activity's Request for Assistance Process. [ED NOTE: Just as other WGs should have a TS- Process document, the QA WG has agreed to have a Process document containing the process for formal consultation, active reviews.] In addition to describing how WGs can submit requests to the QA Activity, it describes the process within the QA Activity for handling these requests including review procedures and criteria. The Request for Assistance Process was created to ensure a fair and impartial, quick and accurate response to requests.

WGs may submit requests for assistance to the QA Activity Working Group for the following:

Upon receiving one of these requests, a QA Working Group task team will review the request, appoint reviewers/consultants, inform the WG of the request's status, establish a liaison with the WG, and working in concert with the WG, respond to the request.

Other requests will be considered by the QA Activity Working Group on a case-by-case basis in accordance with the Request for Assistance Process. Decisions will be based on the nature of the request, (e.g., within the scope of the QA Activity domain). The WG will be informed of the QA Activity's decision and provided an opportunity to appeal the decision.

3.2 Active Reviews

At the request of a WG or in response to a general call for participation (e.g., public working draft or email to W3C Chair to review something) or due to a W3C Process requirement, the QA Activity Working Group may provide detailed review and assessment of a WG's deliverable. If such a detailed review is conducted, it will be conducted with the knowledge of the WG and in concert with the WG. If appropriate, a QA Activity member will participate in the WG meetings.

The QA Activity Working Group will follow the procedures set forth in its Request for Assistance Process for establishing a review team, performing the review, and communicating with the WG. The QA Framework Guidelines will be used in the assessment of the WG deliverable, verifying that appropriate Framework checkpoints are met. Additionally, any relevant WG documents (e.g., Charter, QA Process) will also be used.

3.3 Adjudicate entry and exit criteria

WGs may wish to define QA-specific entry and exit criteria for milestones on the Recommendation track. The QA Activity Working Group can help to identify these criteria.

3.4 QA resource supplement

At the request of a WG, the QA Activity WG may be able to provide resources for the planning and building of test materials.

3.5 Resolution of external QA requests

QA-related requests received from organizations or individuals external to the W3C, will be categorized as to the nature of the request and handled accordingly. QA-related questions of general interest will be posted and discussed on the QA-IG mail exploder. QA-related questions on a specific W3C technology will be shared with the appropriate WGs and/or Horizontal Teams and a coordinated dialogue and response will be pursued. Similarly, test material inquiries and offers to donate test materials will be coordinated with the appropriate WGs, Horizontal Teams, and W3C Management Team. Details of how these requests are handled within the QA Activity are described in the Request for Assistance Process.

4. Conformance

This section defines conformance of Working Group processes and operations to the requirements of this specification. The requirements of this specification are detailed in the checkpoints of the preceding "Guidelines" chapter of this specification, and apply to the Working Group QA-related documents and deliverables required by this specification.

4.1 Conformance definition

This section defines three levels of conformance to this specification:

A Working Group conforms to the "QA Framework: Operational Guidelines" at Level X (A, AA, or AAA) if the Working Group meets at least all Conformance Level X requirements.

To make an assertion about conformance to this document, specify:

Example:

"This QA processes and operations of this Working Group conform to W3C's 'QA Framework: Operational Guidelines', available at http://www.w3.org/TR/qaframe-ops/, Level AA."

4.2 Conformance disclaimer

The checkpoints of this specification present verifiable conformance requirements about the operational aspects of Working Group quality processes. As with any verifiable test requirements, users should be aware that:

  1. Passing all of the requirements to achieve a given conformance level -- A, AA, or AAA -- does not guarantee that the subject operations and processes are well-suited to or will achieve their intended purposes, nor does it guarantee the quality or suitability of test materials produced under the processes.
  2. Failing to achieve level A conformance does not mean that the subject quality processes and/or associated test materials are necessarily deficient to their intended purposes. It means that the processes have failed one or more checkpoints that best-practice experience has shown to facilitate and enable successful quality processes, that in turn have a high correlation with timely and successful development and maintenance of the test materials.

5. Acknowledgments

The following QA Working Group and Interest Group participants have contributed significantly to the content of this document:

6. References

EXTERN-TA
QA activity email thread about third-party participation in test materials production, available at http://lists.w3.org/Archives/Public/www-qa/2001Oct/0060.html.
MATRIX
W3C-wide conformance activity survey covering all the Working Groups, "The Matrix", available at http://www.w3.org/QA/TheMatrix.
TAXONOMY
QA Activity test taxonomy, a classification scheme for conformance test materials, available at http://www.w3.org/QA/Taxonomy.
OPS-CHECKLIST
An appendix to this operational guidelines document presents all checkpoints in tabular form. Available at http://www.w3.org/TR/2002/WD-qaframe-ops-20020515/qaframe-ops-checklist
OPS-EXTECH
QA Framework: Operational Examples & Techniques , D. Dimitriadis, L. Henderson, L. Rosenthal, K. Gavrylyuk, Eds., W3C Working Draft, May 2002, companion version to this document, available at [...]
PROCESS
World Wide Web Consortium Process Document , I. Jacobs, Ed., 19 July 2001, available at http://www.w3.org/Consortium/Process-20010719/.
QA-GLOSSARY
A comprehensive glossary of QA terms, maintained by the QA Working Group. (Initial version under construction.)
QAF-INTRO
QA Framework: Introduction , L. Henderson, D. Dimitriadis, K. Gavrylyuk, L. Rosenthal, Eds., W3C Working Draft, May 2002, companion version to this document, available at [...].
QAF-SPEC
QA Framework: Specification Guidelines , L. Rosenthal, D. Dimitriadis, K. Gavrylyuk, L. Henderson, Eds., W3C Working Draft, May 2002, companion version to this document, available at [...].
QAF-TEST
QA Framework: Test Guidelines (not yet published).
QAIG
Quality Assurance Interest Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/IG/.
QAWG
Quality Assurance Working Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/WG/.
DOM Working Group TS
DOM TS Process Document, D. Dimitriadis, Ed., 15 January 2002, available at http://www.w3.org/2002/01/DOMConformanceTS-Process-20020115.
REC-TRACK
Stages and milestones in the W3C Recommendation Track, per the Process Document (Process Document is available at http://www.w3.org/Consortium/Process-20010719/, see section 5.2).
RFC2119
Key words for use in RFCs to Indicate Requirement Levels, March 1997, available at http://www.ietf.org/rfc/rfc2119.txt.
WCAG10
Web Content Accessibility Guidelines 1.0 , W. Chisholm, I. Jacobs, G. Vanderheiden, Eds., W3C Recommendation, 5 May 1999, available at http://www.w3.org/TR/WCAG10/.
WG-QA-RANGE
Email proposal by David Marston, on the QA public mail list, for range of Working Group commitment levels to conformance test materials production, available at http://lists.w3.org/Archives/Public/www-qa/2001Apr/0004.html.

7. Change History

05-02-2002

Checkpoints numbering is changed, changes list refers to the numbering from the previous WD [1].

[1] http://www.w3.org/QA/WG/2002/framework-20020311/qaframe-ops

03-11-2002

Checkpoints numbering is changed, changes list refers to the numbering from the previous WD [1].

[1] http://www.w3.org/QA/WG/2002/02/qaframe-ops-0225.html

02-28-2002
Major rewrite of the previous draft. Incorporated Lynne's comments about Gd1 and Gd4, split Gd regarding QA process, added bunch of new checkpoints.