QA Framework: Process & Operational Guidelines

W3C Working Draft 18 January 2002

This version:

http://www.w3.org/QA/WG/2002/framework-20020118/qaframe-ops

Latest version:

http://www.w3.org/QA/WG/qaframe-ops

Previous version:

http://www.w3.org/QA/WG/2002/framework-20020102/qaframe-ops

Editors:

Kirill Gavrylyuk (kirillg@microsoft.com)

Dimitris Dimitriadis (dimitris@ontologicon.com)

Lofton Henderson (lofton@rockynet.com)

Contributors:

See Acknowledgments.

Copyright ©2002 W3C^® (MIT, INRIA, Keio), All Rights Reserved. W3C liability, trademark, document use and software licensing rules apply.

---

Abstract

This document defines a common Operational Framework for building conformance test suites and tools for W3C specifications. It presents operational and procedural guidelines for groups undertaking conformance materials development. This document is one in a family of QA Framework documents, which includes the other existing or in-progress specifications: Overview & Process; Specification Guidelines; and, Technical Guidelines.

Status of this document

This document is a Working Draft, made available by the W3C Quality Assurance (QA) Activity for discussion on the QA email list.

This version incorporates the discussions at the first QA face-to-face, plus subsequent QA WG teleconferences, and supersedes all previous drafts. All review comments against the previous version have been accepted and implemented, except as indicated in an associated disposition of comments document. This version still has minor editorial issues, flagged by "@@", (almost) all of which are needed external links.

This version is intended as the last WG review version before final editing for first public working draft (FPWD).

This part of the Framework document family will have a companion part, "Process & Operational Techniques and Examples" (in progress). Some of the lengthier example and "how to" text of this current guidelines document version will be moved to the "Techniques and Examples" document, when the latter is written.

Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress".

Please send comments to www-qa@w3.org, the publicly archived list of the QA Interest Group. Please note that any mail sent to this list will be publicly archived and available, do not send information you wouldn't want to see distributed, such as private data.

Table of contents

1. Introduction
    1.1 Motivation for this Guidelines document
    1.2 Navigating through this document.
    1.3 Terminology
2. Guidelines
        G 1. In charters, describe QA goals, criteria, deliverables and resources
        G 2. In charter, address where and how conformance test materials will be produced.
        G 3. Ensure that goals and scope of QA development are clear
        G 4. Allocate staff for QA process.
        G 5. Synchronize specification and test materials developments
        G 6. Produce QA process document.
        G 7. Plan transfer of test materials to W3C if needed
        G 8. Plan for test materials maintenance
3. WG relationship to QA Activity
4. Conformance
5. Acknowledgments
6. References

1. Introduction

This document describes the process and operational activities for building conformance test suites and tools for W3C specifications. The primary goal is to assist the W3C Working Groups by providing guidelines for the planning, development and deployment of conformance test materials. In this guideline, the term "conformance test materials" refers conformance test suites, validation tools, conformance checklists, and any other materials that are used to check or indicate conformance.

This document is part of a family of QA Framework documents designed to improve the quality of W3C specifications as well as their implementation by solidifying and extending current quality practices within the W3C. The QA Framework documents are:

• Introduction
• Specification Guidelines
• Process and Operational Guidelines (this document)
• Technical Guidelines

The guidelines are intended for all Working Groups as well as developers of conformance materials for W3C specifications. Not only are the WGs the consumer of these guidelines, they are also key contributors. The guidelines capture the experiences, good practices, activities, and lessons learned of the Working Groups and present them in a comprehensive, cohesive set of documents for all to use and benefit from. The objective is to reuse what works rather than reinvent and to foster consistency across the various WG quality activities and deliverables.

1.1 Motivation for this Guidelines document

As the complexity of W3C specifications and their interdependencies increases, quality assurance becomes even more important to ensuring their acceptance and deployment in the market. There has been a growing awareness and interest in conformance and quality. In approving and initiating the QA Activity, W3C has endorsed the principle that, in order for W3C Web standards to achieve full interoperability and access to all, the quality of the implementation must be given as much attention as the standards' development. The principal factor for improving the quality of implementation is early availability of conformance test materials. (ed note: do we need to say more? Or will the business case and justification stuff be in the Intro document.)

Although not explicitly stated, the W3C Process Document supports the development of conformance test materials.

1. "[.] groups may produce technical reports, review the work of other groups, develop sample code or test suites, etc." (sec. 3.)
2. "W3C should make every effort to maintain its Recommendations (e.g., by tracking errata, providing test bed applications, helping to create test suites, etc.)" (sec4.)
3. In an effort to meet these suggestions and address the implementation requirements of the Process Document, some Working Groups have included the development of conformance materials as part of their CR-exit and PR-entrance criteria (e.g., SVG, UAAG, XSL-FO). This makes sense, since it is natural for test suites and implementations to develop in parallel - each is a help to the development of the other.

There is already a body of contemporary QA experience and activity amongst the Working Groups. The Matrix identifies more than a score of test suites and validators, in various states of development. Moreover, many WGs have already established procedures, techniques and tools for developing test materials (e.g., DOM). It makes sense to capitalize on what has already been done and share that with those who are starting out and those who are already in the process of developing conformance materials.

1.2 Navigating through this document.

The Guidelines in the document are organized chronologically starting with those that apply at the formation of a WG (e.g., charter considerations) and through a progression of guidelines leads the reader through the various process and operational activities necessary in planning, developing, deploying and maintaining conformance materials. This document is applicable to all WGs, including those that are being rechartered or already exist. WGs may already be doing some of these activities and should review the document and in so far as possible incorporate principles and guidelines into their work.

This document includes guidelines or general principles for the development of conformance materials. Each guideline includes:

• The guideline number
• The statement of the guideline
• The rationale behind the guideline
• A list of checkpoint definitions.

The checkpoint definitions in each guideline define the processes and operations that need to be implemented in order to accomplish the guideline. Each checkpoint definition includes:

• The checkpoint number
• The statement of the checkpoint
• The priority of the checkpoint. Priority 1 checkpoints are highlighted through the use of style sheets.
• Optional informative notes, clarifying examples, and cross references to related guidelines or checkpoints.

Each checkpoint is intended to be specific enough so that someone can implement the checkpoint as well as verify that the checkpoint has been satisfied.

1.3 Terminology

The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" will be used as defined in RFC 2119 [RFC2119].

Unusual terms in these framework documents are defined when first used, and most generally useful QA-specific terms will eventually be in the QA Glossary [QA-GLOSSARY].

2. Guidelines

Guideline 1. In charters, describe QA goals, criteria, deliverables and resources

In approving and initiating the QA Activity, W3C has endorsed the principle that, in order for W3C Web standards to achieve full interoperability and access to all, the quality of implementation must be given as much attention as the standards' development. The principal factor for improving the quality of implementation is early availability of conformance test materials. Therefore, conformance test materials -- test suites and tools -- must be considered as WG deliverables the same as the standards themselves. Accordingly, they must be addressed in the WG charter.

Checkpoints:

Checkpoint 1.1.Priority 1.In charters include section for test suites/tools plans. Plan to have at least a commitment to "QA level three".

Detailed planning and specification of test materials is inappropriate in the charter, as it binds the WG inflexibly to test materials implementation specifics for which flexibility may be needed. There are numerous possible levels of commitment that a WG could make, based on anticipated WG resources, existence of outside efforts, and the WG's current stage in the process [REC-TRACK].

Based on level of testability of the spec and extent of test materials, a range of possible commitments has been identified.

This 10-point enumeration is derived from a proposal to the QA mail list, after the 4/2001 QA Workshop.

The further down the scale (i.e., the higher the number) that the WG commits, the better it is for achieving W3C's ultimate goals of interoperability and accessibility.

Checkpoint 1.2.Priority 2.In charter, specify completion and publication of test materials to be a criterion for CR-exit and PR-entrance.

A de facto convention of "CR-exit" seems to be a common process goal amongst a number the existing test suite efforts (e.g., SVG, UAAG,...). This makes sense, as it is natural for test suites and implementations to develop in parallel -- each is a help to the development of the other. And, the Process Document does address implementation requirement:

• "Advancement of a technical report to Candidate Recommendation is an explicit call for implementation experience to those outside of the related Working Groups or the W3C itself." (sec 5.)
• "Preferably, the Working Group should be able to demonstrate two interoperable implementations of each feature." (sec 5.2.4, about PR-entrance criteria)

Checkpoint 1.3.Priority 1.In charter, address QA staffing commitments.

There will be at least some staffing required from the WG, for any of the acceptable options presented so far. Depending upon the general intent and plan, and how the test suite will be built, the commitment can range from minimal to significant. See "Examples & Techniques" [@@EX-TECH].

Guideline 2. In charter, address where and how conformance test materials will be produced.

There are several possibilities for producing conformance test materials:

• build them entirely within W3C (normally, the WG). Example: SVG test suite.
• build them externally and transfer to W3C. Example: XML test suite.
• build procedures for assembling contributions from multiple sources. Example: XSLT test suite (a work in progress, this is actually being done in OASIS XSLT/Xpath Conformance TC, but it could just as well work within a W3C WG).
• hybrids of the above.

Checkpoints:

Checkpoint 2.1.Priority 1.In Charter, specify level of involvement in the ongoing evolution of test suites and tools.

There are good reasons why external parties should have a strong participation during the building of the materials. But even if the test suite is being developed outside of W3C, Working Group needs to ensure it is completed according to the level Working Group committed to in the charter.

Checkpoint 2.2.Priority 2.Ensure secure and reliable repository location for future test materials.

This doesn't necessarily mean that test suite/tools should physically reside in the W3C, though it is highly recommended:

1. At the point that test materials become operationally deployed, then challenges to the correctness of both the test materials and specifications (RECs) normally increase, and the WG is the best venue for initial processing and adjudication of such queries against both.
2. Further to #1, it is more likely that specification (REC) errata processing and test suite maintenance can be kept synchronized if both responsibilities reside in the same body, the WG.

• the materials have proper maintenance and support
• tests materials are freely available for download. If license is applied, it does not preclude from test suite being used by all interested parties
• test suite quality and usability is appropriate

If the WG ceases to operate, then other responsible for ongoing interpretation and maintenance (errata) of the standard should be designated to ensure continued availability of the test suite.

Guideline 3. Ensure that goals and scope of QA development are clear

Common frameworks for W3C QA activities are obviously shaped by the high-level goals for development and deployment of test materials. Before further developing the framework guidelines, it is useful to state some of these high-level goals.

The over-arching goal is stated in the QA Activity Statement:

• to enable W3C Web standards to achieve full interoperability and access to all, by addressing the quality and correctness of implementations.

Specific goals of test materials development include:

• provide self-assessment tools and materials for implementation builders;
• help implementation builders to achieve interoperability of implementations;
• help users to verify compliance of implementations with the stated conformance requirements of their respective specifications;

The goals of QA work within WG do not encompass:

• measurement of performance, scalability of implementations, or their assessment against any other "goodness" criteria, except what is stated as a conformance requirement of their respective specification.
• certification of implementations

Checkpoints:

Checkpoint 3.1.Priority 2.Ensure that test materials are documented, self contained and reusable by accredited third parties for testing purposes.

While certification may have a positive quality assurance role in standards-based Web environments, W3C does not presently intend to initiate or offer any certification services. Nevertheless, developed test materials can be used by external parties including certification services. Factors of rigor, objectivity, and defensibility are key determinants of usability for certification, and these same properties are positive assets of any test materials.

Checkpoint 3.2.Priority 3.Specify policy for branding materials if applicable.

Certification aside, some W3C activities support "branding" -- use of a conformance icon to indicate some level of standard compliance. Examples: HTML validation, CSS validation, WCAG compliance. While these content branding schemes are relatively non-controversial, there are nevertheless some serious issues which should be addressed -- particularly in the contexts of user-agent and API conformance -- before any branding-related goals are articulated.

Checkpoint 3.3.Priority 2.Provide disclaimer regarding use of the test materials for compliance verification.

Although tests suites may be used for compliance verification, users must be aware of the fact that

1. passing all of the tests does not guarantee full compliance of implementation to specification
2. failing test suite means failing tests for specific feature they target

The last item requires some elaboration. Depending on the nature of the test, it may actually be sufficient to judge non-compliance of the subject. Based on taxonomy the following is valid:

• for conformance testing of the (web) content: statement 1 is false, but statement 2 is true
• for content/driven conformance testing of the tools/product: both statements are true
• for interfaces (API) conformance testing of the tools or libraries: both statements are true

Guideline 4. Allocate staff for QA process.

Before starting QA development, Working Group has to appoint several key participants of QA process.

Checkpoints:

Checkpoint 4.1.Priority 2.Appoint test QA moderator.

• WG members (and other W3C members)
• W3C QA staff

When test suite is developed outside of Working Group, a moderator usually already exists where test suite coming from. It is recommended to invite moderator to become WG member, since he/she needs access to all WG materials and resources. See also section on transferring test suite from outside W3C.

Checkpoint 4.2.Priority 2.Designate a task force or another entity to maintain the test suite, as well as maintenance procedures.

It is recommended that the errata task force be involved in test suite maintenance, since this will help to keep test suite and specification in sync. Maintenance group shall establish tracking system to track issues/changes to the test suite.

Guideline 5. Synchronize specification and test materials developments

There are several strong benefits from starting synchronizing of specification and test materials developments as early as possible:

• it gives extensive list of use cases
• identifies ambiguities in the spec at the early stages - saves cycles in the late phases
• provides clear set of testable assertions - skeleton of the spec - in its turn facilitate development of interoperable implementations, which is criteria for entering Candidate Recommendation phase

Checkpoints:

Checkpoint 5.1.Priority 3.Introduce intermediate milestones for test test materials completion

These milestone can be naturally bound to Working Drafts/Candidate Recommendation publications.

Checkpoint 5.2.Priority 3.Synchronize latest version of the spec draft and latest version of the published test materials.

Checkpoint 5.3.Priority 2.Support spec versioning/errata in the test materials

Structure of the test materials should allow versioning/errata to enable verification of compliance with different versions of the specification.

Guideline 6. Produce QA process document.

A Working Group QA process document is a central QA document describing how testing process is set up within the WG. Examples of the test process document can be found at DOM TS process, XML Schema TS process, XML Protocol TS process, etc. It is a required part of QA process in a WG and should address the following questions:

Checkpoints:

Checkpoint 6.1.Priority 2.Review Technical Guidelines before writing QA Process document and starting test materials development.

Taking into account the nature of the future standard, before starting test suite development, test suite implementer (WG or 3rd party) should look at Technical Guidelines and decide what approach to take for test materials organization, tests criteria, etc.

Checkpoint 6.2.Priority 2.In QA Process document, explain contribution process.

The process document must describe where to submit test materials and whom to notify (moderator) of submission. Contribution process SHOULD describe the format of contributed material, it may contain validation harness, utilities that facilitates tests creation, templates, etc.

Checkpoint 6.3.Priority 2.In QA Process document, define the licenses applicable to submitted and published test materials.

Working Group can choose either the W3C Document or the W3C Software license. If applicable, W3C Document license is recommended.

Checkpoint 6.4.Priority 2.In QA Process document, describe means for communication regarding test suites.

WG needs to choose mailing list to send all comments and announcements to.

Checkpoint 6.5.Priority 2.In QA Process document, describe review procedures for submitted test materials

The following example may serve as a guideline for tests review process

1. Each contribution should be reviewed by at least one vendor other than submitter for it's eligibility. Any vendor involved in creation of WG test collection other then submitter may review tests being contributed.
2. Eligibility of tests should be judged by:
   • The accuracy of the test. Accuracy of a test is determined by a judgment of the reviewer whether the test case actually tests what the submitter states the test case tests. The accuracy is the extent to which the test matches the cited parts of the Specification. If it does not match, or only partially matches, the test should be considered inaccurate.
   • The scope of the test. Test should target MUST provisions of the specification and not MAY/SHOULD provisions. The test should target only conformance requirements. Data values limits, memory or performance limits should not be targets to the test unless those are specified in minimum conformance requirements.
   • The clarity of the test. Clarity of a test is a determination of whether the aspect being tested is clearly described with the anticipated results acceptably explained.
   • The clarity of aspect of the Specification being tested. The Test Suite aims to test parts of the Specification and errata that aren't vague.
3. After review results should be sent back to submitter for his feedback.
4. In case of disagreement, request for clarification should go to WG, tests in question should be left off the test suite till W3C WG clarifies the issue.
5. Only those tests that are agreed to be correct by reviewers and submitter or approved by W3C WG clarification should be published.

Checkpoint 6.6.Priority 2.In QA Process document, describe the way test materials will be published and point to the corresponding web page.

Checkpoint 6.7.Priority 3.In QA Process document, describe how vendors can publish test results for their products, if applicable.

It is recommended to use one of practices for publishing test materials, described in Technical Guidelines. WG can encourage vendors to publish results for their implementations including/describing test harness that would allow anyone to reproduce the results.

Guideline 7. Plan transfer of test materials to W3C if needed

As mentioned before in the notes to checkpoint for test materials repository, it is recommended that test suite resides in the W3C. If test development was held outside by some external entity/group (ET) and Working Group(WG) together with ET decided to move test materials (TM) to W3C, the following checkpoints outline necessary steps:

Checkpoints:

Checkpoint 7.1.Priority 2.If plan to transfer the test materials to W3C, perform assessment of their quality

Checkpoint 7.2.Priority 2.If plan to transfer the test materials to W3C, identify staff resources to meet requirements.

Checkpoint 7.3.Priority 1.If plan to transfer the test materials to W3C, resolve IPR questions and reach agreement with external party that produced test materials.

Checkpoint 7.4.Priority 2.If plan to transfer the test materials to W3C, update QA commitments in the charter if necessary

The following procedure is recommended:

Before transfer WG with the help of QA WG:

• performs assessment of the quality of candidate TM (by WG, QA WG)
• identifies and commits to derive set of QA deliverables from the candidate TM. These can be: releases, appeals/challenges, maintenance, submissions, Web site, mail list, etc. (by WG)
• identifies staff resources to meet requirements, TM moderator. It is recommended to invite TM moderator from the EG to become WG member should transfer happen.(by WG)
• takes decision to seek/accept transfer. (by WG)
• initiate charter revisions for QA commitments (by WG, QA WG may consult)

During transfer

• EG and W3C reach agreement to transfer (by WG, QA WG, EG)
• WG and EG perform actual transfer of TM, WG creates initial repository (by WG, EG)

After transfer, initial process setup

• WG appoints TM moderator. TM moderator creates QA Process document for WG (by WG, TM moderator, QA WG may consult)
• Logistics: TM home page, TM issues mailing list (by WG, TM moderator)
• Decide appropriate W3C IPR license (by WG, QA WG)

First W3C public release of new TM

• Make necessary enhancements prior to first public release: fix known/reported errors, documentation.(by WG)
• Announce first public release of TM (by TM moderator)
• Joint W3C/EG press release (by WG, QA WG, Communications Group, EG)

After first public release, TM enters maintenance phase which is described below.

Guideline 8. Plan for test materials maintenance

Once test suite development is initiated, its maintenance requires an equally significant amount of resources. Resources allocation is needed as prescribed by checkpoint above.

Checkpoints:

Checkpoint 8.1.Priority 3.Maintain contribution and review procedures throughout entire test suite's and standard's life cycle.

Checkpoint 8.2.Priority 2.In QA Process, specify procedure for updates of the test materials with new erratas/specification versions

The following procedure may serve as an example:

• Initiating test suite change/update. As soon as errata are accepted or new version/edition of the Rec is published, errata task force (or test suite maintenance group) identifies affected test cases and initiates updates to the test suite by adding entries to the tracking system with status "ACTIVE"
• Providing change to the test suite for review. Test case submitter or initiator of the change provides updated version for the affected tests. If no changes provided test maintenance group finds the way to provide it. Once changes are ready, maintenance group changes corresponded status in the tracking system to "RESOLVED"
• Review/acceptance of the change. Tests change goes through the same tests review process presented earlier. If test changes are accepted, they are added to the test suite with indicator of errata/version. Old tests stay in the test suite marked with last errata/version level they comply with. Status changes to "CLOSED". If tests are rejected, they go back to provider of the change with resolution, status is reverted back to "ACTIVE"

Checkpoint 8.3.Priority 2.In QA Process, identify the communication channel and procedure for appeals of tests validity

The following procedure may serve as example:

• Initiating appeal
• Review of appeal
• Providing change to the test suite for review in case of appeal acceptance
• Review/acceptance of the change

[Active issue]If there is still active work on the test suite even if the WG is not re-chartered to handle the work, then it is up to the Director to determine how this work should be done. Some options include:

• it may assigned to a Coordination Group;
• it may be assigned to a related WG;
• the Team could take care of minor items.

3. WG relationship to QA Activity

The QA Activity anticipates different relationships to the various Working Groups. On the one hand, the QA-specific needs of the Working Groups will vary depending upon their resources and experience, as well as their stage in the Recommendation track. On the other hand, the W3C processes may evolve so that QA-specific process requirements are tied to milestones in the Recommendation track.

Potential relationships between Working Groups and the QA Activity include:

• Liaison and consultation.
• Active reviews.
• Adjudicate entry and exit criteria.
• QA resources supplement.
• Resolution of external QA requests.

Key determinants of the WG/QA relationship will include level of available QA Activity resources, and the evolution of W3C policy on QA requirements.

3.0.1 Liaison and consultation.

In an assistive role, the QA Activity provides consultation to Working Groups in planning and building or acquiring of test materials. This should be the baseline relationship, applicable to all Working Groups. Consultation may be provided in formal and informal ways. Informal questions MAY be sent to www-qa@w3.org. In order to make assistance quick and accurate the following process is introduced for formal requests:

• WG may submit to QA WG questions/requests of the following types:
  • request for clarification on documents published by QA WG
  • requests for review of WG charter
  • requests for review of WG specification
  • requests for review of WG QA process document
  • requests for consultation on specific QA techniques/test suite architecture
  • requests for help on coordination of joint QA related activity or materials transfer with external organization involved
  Other requests will be considered by the QA WG on a case-by-case basis. They may be accepted, or may be considered out of scope and rejected (subject to appeal) by unanimous vote of the QA WG members.
• Within 1 week QA WG chair should appoint question to task team of QA WG members and give an estimated time for resolution.
• Assigned QA WG task team should respond to the question (request) with appropriate resolution within term allocated by QA WG chair

3.0.2 Active reviews.

• Any member of QA WG may participate in the review.
• QA WG Chair should appoint at least 2 reviewers
• Reviewers should verify that:
  • QA Process document is in place and addresses all checkpoints from QA Process section of this document.
  • Available test materials are produced consistently with best practices described in the QA Framework: Technical guidelines
  • If reviewing specification, verify it conformance to QA Framework: Specification Guidelines
  • If reviewing the charter, verify that guidelines/checkpoints from the Charter considerations section are addressed.

3.0.3 Adjudicate entry and exit criteria.

3.0.4 QA resources supplement.

3.0.5 Resolution of external QA requests

• Escalate the request to corresponding Working Group(s) and coordinate resolution if the matter involves more than one Working Group. TAG may be involved as well.
• If no active Working Group exists for the standard in question, it SHOULD be escalated according to maintenance guidelines in case if WG charter is expired

4. Conformance

This section defines conformance of Working Group processes and operations to the requirements of this specification. The requirements of this specification are detailed in the checkpoints of the preceding "Guidelines" chapter of this specification, and apply to the Working Group QA-related documents and deliverables required by this specification.

This section defines three levels of conformance to this specification:

• Conformance Level "A": all Priority 1 checkpoints are satisfied;
• Conformance Level "AA": all Priority 2 checkpoints are satisfied;
• Conformance Level "AAA": all Priority 3 checkpoints are satisfied;

A Working Group conforms to the "QA Framework: Process & Operational Guidelines" at level X if the Working Group meets at least all Conformance Level X requirements.

To make an assertion about conformance to this document, specify:

• The guidelines title: "QA Framework: Process & Operational Guidelines"
• The guidelines URI: [...tbd...]
• The conformance level satisfied: "Level 1", "Level 2", or "Level 3".

Example:

This QA processes and operations of this Working Group [???] conform to W3C's "QA Framework: Process & Operational Guidelines", available at [...tbd...], Level 2.

5. Acknowledgments

The following QA Working Group and Interest Group participants have contributed significantly to the content of this document:

• Selim Aissi (Intel)
• Daniel Dardailler (W3C),
• Dimitris Dimitriadis (Ontologicon)
• Karl DuBost (W3C),
• Peter Fawcett (RealNetworks),
• Kirill Gavrylyuk (Microsoft),
• Dominique Hazael-Massieux (W3C)
• Lofton Henderson (CGM Open),
• Ian Jacobs (W3C)
• David Marston (Lotus Development Corp),
• Olivier Thereaux (W3C),
• Lynne Rosenthal (NIST).
• Mark Skall (NIST)
• Andrew Thackrah (Open Group)

@@Any additions? Any deletions?@@

6. References

[@@Ed. note: This has not yet been cut down from original all-in-one Framework document. It should be edited to only include those references which actually pertain to this document.]

EXTERN-TA

QA activity email thread about third-party participation in test materials production.

MATRIX

W3C-wide conformance activity survey covering all the WGs, "The Matrix".

PROCESS

W3C Process Document, 19 July 2001.

TAXONOMY

QA Activity test taxonomy, a classification scheme for conformance test materials.

QAIG

Quality Assurance Interest Group of the W3C QA Activity.

QAWG

Quality Assurance Working Group of the W3C QA Activity.

DOM WG TS

Process document for DOM WG Test suite.

REC-TRACK

Stages and milestones in the W3C Recommendation Track, per the Process Document (section 5.2).

RFC2119

Key words for use in RFCs to Indicate Requirement Levels, March 1997.

SVGTEST

SVG Working Group's test suite resource page.

WCAG10

Web Content Accessibility Guidelines, version 1.0, W3C Recommendation, 5 May 1999.

WG-QA-RANGE

Email proposal by David Marston, on the QA public mail list, for range of WG commitment levels to conformance test materials production.

XMLTEST

OASIS XML Conformance TC's XML test suite resource page.

XSLT-TEST

OASIS XML Conformance TC's XSLT/Xpath test suite resource page.

QA-TECH-GUIDE

QA Framework: Technical Guidelines(not written).

QA-SPEC-GUIDE

QA Framework: Specification Guidelines(not written).