W3C

QA Framework: Introduction

W3C Working Draft 2 January 2002

This version:
http://www.w3.org/QA/WG/2002/framework-20020102/qaframe-intro
Latest version:
http://www.w3.org/QA/WG/qaframe-intro
Previous version:
http://www.w3.org/QA/WG/2001/framework-20011205/qaframe-intro
Editors:
Lofton Henderson (lofton@rockynet.com)
Kirill Gavrylyuk (kirillg@microsoft.com)
Dimitris Dimitriadis (dimitris@ontologicon.com)

Copyright ©2001 W3C® (MIT, INRIA, Keio), All Rights Reserved. W3C liability, trademark, document use and software licensing rules apply.

Abstract

This document introduces a common framework for building conformance test suites and tools for W3C specifications. It presents introductory, roadmap, and orientation information for the Quality Assurance (QA) Activity, as well as process and organizational guidelines, for groups undertaking development of conformance materials. This is the first of a family of QA Framework documents, which includes the other existing or in-progress specifications: Process & Operational Guidelines; Specification Guidelines; and, Technical Guidelines.

Status of this document

This document is a Note, made available by the W3C Quality Assurance Activity (QAA) for discussion on the QA email discussion list.

This version incorporates the discussions at the first QA face-to-face, plus subsequent QA WG teleconferences, and supersedes all previous drafts. Open issues and other incompletions are flagged with "@@". The number of such are mostly minor editorial issues, including some needed intra- and inter-document links, except for the issue of the conformance clause. This version intended as the final WG review version before final editing for first public working draft (FPWD).

Where appropriate, some documents in this Framework family (but not this one) have an associated and copiously hyperlinked "Examples & Techniques" document, in which details and especially discretionary choices are elaborated. Where applicable -- but not for this Introduction document -- the presentation style is Guidelines-plus-Checkpoints, similar to WAI standards.

Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress".

Please send comments to www-qa@w3.org.

Table of contents

1. About this QA framework
    1.1 Introduction
    1.2 Why do we need a QA framework?
    1.3 Who the Framework is for
    1.4 The Framework goals
2. A survey of QA resources
    2.1 Mail list & archive
    2.2 QA Web pages
    2.3 Glossary
    2.4 Taxonomy
    2.5 The Matrix
    2.6 Notes
    2.7 QA Framework documents
    2.8 QA issues document
    2.9 Expertise & consultancy
    2.10 Technical assets
3. Structure & content of Framework documents
    3.1 Applicable domain
    3.2 Document orientation & structure
    3.3 Informative guidelines
    3.4 Terminology
    3.5 Documents roadmap
        3.5.1 Contents
        3.5.2 Introduction
        3.5.3 Process & Operational Guidelines
        3.5.4 Specification Guidelines
        3.5.5 Technical Guidelines
4. Guide to using Framework documents
    4.1 Target audience
        4.1.1 For the overall Framework
        4.1.2 For the Introduction
        4.1.3 For the guideline parts
    4.2 Framework primer and guide
        4.2.1 First step -- charter commitment
        4.2.2 Set up processes & logistics
        4.2.3 Planning and writing the specification
        4.2.4 Reviewing and progressing the specification
        4.2.5 Designing and building test materials
        4.2.6 Publication of test materials
        4.2.7 Specification publication and beyond
5. Conformance
6. Acknowledgments
7. References

1. About this QA framework

1.1 Introduction

This document introduces a common framework for building conformance test suites and tools for W3C specifications. It presents introductory, roadmap, and orientational information for the Quality Assurance (QA) Activity, and aims thereby to provide a first point-of-entry and introduction to the Activity itself.

It also presents some process and organizational guidelines, for groups undertaking development of conformance materials. Thereby it aims to introduce the first planning and action items for those who will be active in the QA projects of their Working Groups.

This document is one of a family of documents that together define a common framework for planning and building conformance test materials for W3C Recommendations. The document family covers: a QA roadmap and resource guide; process, organizational, and operational guidelines for groups undertaking development of conformance materials; specification guidelines, for writing better, more testable Recommendations; and finally, detailed technical guidelines for the test suites (TS) and tools themselves.

The (root) documents in this family are:

Some of these documents have an additional associated "Examples & Techniques" document, which contain detailed, technology-specific methods and examples for implementing the guidelines.

1.2 Why do we need a QA framework?

The ultimate mission of W3C is "Leading the Web to its full potential...", and W3C's many active working groups contribute towards this goal, in large part, by building a family of high-quality Web standards. These standards are the building blocks of a technical architecture to maximize the Web's potential.

Most would agree that the writing of the standard is not, in itself, the end purpose of the WGs. It is a means to achieve the real end: the success of the standardized technology in practice and practical application. One measure of the success that can be applied to most Web standards is a set of complete, correct, high-quality, interoperable implementations that actually realize the functionality of the standard.

In the past few years, several WGs have discovered that the early production of test materials (TM) has been a major contributor to the success of their standards. The benefits of test materials work are several-fold, but a couple of the big benefits are: early detection and correction of unintentionally vague, contradictory, or unimplementable language in the specifications (RECs); and, a solid basis for the development of the interoperable implementations demonstration required by the W3C Process Document.

The individual conformance test materials efforts of the Working Groups have in several cases been inspired and on target, in others have been perfunctory or rudimentary. Even amongst the best efforts, however, there is little uniformity in the quality-control principles, methodologies, or even such basics as terminology. And even those WGs which have resources sufficient to their QA goals have effectively had to re-invent their processes, operational framework, and technical bits from scratch. Just figuring out where to start -- researching the numerous existing TS activities -- can be a hurdle.

This defines the mission of our QA Activity. QA aims assist the WGs by assembling and organizing the best of a body of good practice, defining a framework, and providing a roadmap for the working groups. These best-practice guides should dramatically ease the job of planning and implementing QA projects within the Working Groups.

1.3 Who the Framework is for

The last underscores a key reality of improved QA efforts associated with W3C Recommendations. QA is not a comprehensive service proffered to the Working Groups. Nor is it a mandate imposed on the Working Groups from without, notwithstanding that higher expectations are evolving for the Working Group QA deliverables and for the quality of the products of the WGs in general.

QA is properly an intergral part of the efforts of each Working Group. Each WG will provide the majority of the resources needed to meet its QA commitments. The QA Activity provides this Framework, as well as other tools and assets, to aid and guide the WG efforts.

Therefore the QA Framework is for everyone who is involved with the work of W3C's Working Groups. Some part of these Framework documents should be useful and helpful to anyone who is an active participate in generating such deliverables -- specifications and conformance test materials alike -- as well as anyone who is involved in assessing or evaluating such deliverables.

1.4 The Framework goals

Foremost amongst the purposes of defining a common QA Framework is the principal goal of the QA Activity itself:

As an integral part of the working modes of the WGs, QA is ideally a development context which is applicable to a spec's development from its earliest stages through completion. This Framework provides a collection of best-practice principles and guidelines, which span the life of WG activities from the identification of QA deliverables in the WG charters, through post-REC and possibly even post-WG maintenance.

While some might perceive QA projects as a regrettable drain on WG resources, there is ample experience, both within W3C as well as other standards venues, that shows significant improvement to the products of the WGs. This equates to a sound business case for the early investment of resources -- the cost in resources is more than offset by the benefits of more implementable and better implemented Recommendations. In fact, in some cases there could be a net resource gain, by dodging problems that would be costly to fix if detected late in a specification's development process.

As one of the principal resources and deliverables of the QA Activity, ideally, this Framework document family should provide to those undertaking test suite projects,

More specific goals include:

2. A survey of QA resources

Here is a summary of the resources provided by the QA Activity.

Caveat. The information in this section may be somewhat volatile. Nevertheless it is considered useful to collect it in one place.

2.1 Mail list & archive

The QAA tries to conduct its business, as much as possible, on its public mail list,

The archive for the public mail list is found at:

All public working drafts of all Framework documents, as well as other notes and white papers, are discussed on this list. Working group meeting minutes are announced to the list. QA issues are raised and discussed here as well.

Some of the content of these Framework documents comes from discussions on this public list.

The QAA has another list for the QA Working Group, publicly archived but only writable by members:

This list is designated for QA WG logistical topics, dissemination of WG teleconference details, and discussions around the production of internal, interim document drafts. The archive of this list is found at:

2.2 QA Web pages

Virtually all of the resources provided by QAA are accessible from the QA Web pages.

2.3 Glossary

A key permanent standing document of the QAA is a common glossary of QA-related terms, [QA-GLOSSARY]. The need is obvious. In the public mail archive, for example, it is not difficult to find threads where different personal favorite terms are introduced for the same concepts. Even such seemingly universal terms as "interoperability" and "conformance" are not commonly understood.

These Framework documents will include QA-related definitions when first encountered, and will have a glossary appendix, but ultimately these terms will migrate to the comprehensive QA Glossary.

2.4 Taxonomy

The QA landscape within W3C is complex. Several kinds of entities are standardized by W3C Recommendations, including:

Within each category, there are different possible conformance tests and test regimens.

The QA taxonomy [TAXONOMY] provides a classification scheme for the features of this landscape. The kinds of conformance tests that apply to the different sorts of standardized entities are defined, and their details are outlined. Taxonomy dependencies pervade the guidelines and checkpoints of this Framework document family, and are a principal reason that the main root documents of the family are supported by associated "Techniques & Examples" documents.

2.5 The Matrix

The Matrix [MATRIX] is the definitive survey of existing QA practice related to W3C specifications. It starts tracking every WG Rec-track project at Last Call WD, or earlier for others that have significant QA activities at earlier stages.

The matrix contains, for each Rec-track project,

and for most of the projects it also contains,

The matrix also has entries for a few specifications which are not Rec-track projects, such as the RFCs for HTTP and URI.

2.6 Notes

As well as developing these comprehensive Framework specifications, the deliverables of the QA WG include, "Promoting existing notes on QA as W3C Notes and developing new ones." For example, "Common User Agent Problems" (CUAP) is a W3C Note that explains some common mistakes in user agents due to incorrect or incomplete implementation of specifications, and suggests remedies. The collection of such Notes is expected to steadily grow, and will be accessible through a QA bibliography [QA-BIBLIOGRAPHY].

In a similar vein, but not promoted to the status of W3C Notes, are the proceedings of the first QA Workshop [WORKSHOP]. Two days of papers and presentations are available, and these are also linked through the bibliography.

2.7 QA Framework documents

A principal resource and deliverable of the QA Activity is the QA Framework document family, for which this document is the introduction. This group of documents collectively defines a common framework and aims to provide to the W3C Working Groups the resources and tools for all phases and aspects of the quality and conformance activities,

The structure and content of the family of Framework documents is detailed in the next chapter, "The QA Framework documents."

2.8 QA issues document

The QA Working Group maintains an Issues List [ISSUES-LIST], which is publicly readable. It covers all aspects of the QAWG's work, including but not limited to such topics as:

The Issues List documents the source and origination of individual issues, descriptions and argumentation, classification, current status, and the resolution (when status finally reaches "Closed".)

2.9 Expertise & consultancy

The QA Working Group expects to have access to considerable expertise on a spectrum of QA topics, methods, and techniques. One key aim of the QA Activity is to make this expertise available to the Working Groups, at least on an as-needed (request) basis, as they plan and implement their conformance and quality projects. The degree of participation of QA experts in WG projects will likely be limited by QAWG staff resource constraints, but a consultancy role should almost always be possible.

The Framework document, "QA Framework: Process & Operational Guidelines", explores this topic in some detail, including:

2.10 Technical assets

While the initial resources that the QA Working Group offers to other WGs focuses heavily on the Framework documents and available consulting expertise, it is intended that ongoing development within QAA will result in a collection of commonly useful technical assets. Some anticipated offerings include:

3. Structure & content of Framework documents

3.1 Applicable domain

Conformance test materials are being developed in an organizational landscape within W3C (and without) which is characterized by:

These framework documents should be applicable and useful throughout this diverse context.

3.2 Document orientation & structure

The Framework documents will utilize the style of guidelines and verifiable checkpoints, similar to WAI standards such as WCAG [WCAG10]. The degree to which a simple cookbook guide is achievable is complicated by the factors enumerated in the previous section -- diversity of potential test suites and tools, as well as organizational diversity. This complexity and diversity will be handled in the Framework document family by:

The guideline documents are closely linked to their respective "Examples & Techniques" documents. The linked material explores, defines, and presents examples of taxonomy-dependent implementations of checkpoints. This model from the WAI standards has served well to manage diversity and complexity.

These checkpoint technique documents are not intended to be a comprehensive survey of existing W3C practice. That is the purpose of the QA matrix [MATRIX]. That notwithstanding, several of the conformance test efforts will be drawn upon for examples.

3.3 Informative guidelines

These Framework guidelines are considered to be "informative guidelines". This does not preclude that portions of these guidelines may end up with a status other than Note, either by incorporation into the W3C Process Document [PROCESS], or by processing in the W3C Recommendation track.

3.4 Terminology

One of the deliverables of the QA Activity is a Glossary [QA-GLOSSARY]. Unusual terms in these framework documents are defined when first used, and most generally useful QA-specific terms will eventually be in the QA Glossary.

Although the guidelines of this framework are nominally informative, the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" will be used as defined in RFC 2119 [RFC2119], and will be used as if the guidelines are normative.

3.5 Documents roadmap

3.5.1 Contents

The Framework document family includes:

These documents are explained in the following subsections.

3.5.2 Introduction

This document is for everyone interested in QA within the W3C. Especially, it serves as a first read for people with the following roles:

The introduction document serves as a starter for most QA-related activities, especially as it contains pointers to more specific information on particular steps of the QA process, both by examples and by linking to that information explicitly.

3.5.3 Process & Operational Guidelines

This document pair is mainly to be read by people explicitly involved in QA activities. This includes the roles mentioned above, as well as people from the outside in case the QA activity is jointly launched or maintained. It explicitly deals with the following:

This document family, which will be comprised of two documents, one with principals and guidelines, the other with examples and pointers to existing QA work, serves as a more detailed set of guidelines as far as principal and technical questions are concerned. It should be considered a normative read for people involved in launching, taking over, or maintaining QA-related work.

3.5.4 Specification Guidelines

A document pair with guidelines, plus techniques and examples, for better, more testable specifications.

Given the fact that many specifications are written and maintained in XML form, it seems a good idea to think ahead and allow for easier generation of TS schemata, tests, assertions about applying the specification in question etc. There are examples (in particular the DOM Test Suite) where the language used to describe the technology-specific tests are generated from the specification itself. This is expected to become more and more frequent. In order for Working Groups to provide for this, this document will deal with guidelines for writing schemata for the specifications. By its general nature, this is expected to be a central W3C activity.

In particular, these two documents conain information about:

3.5.5 Technical Guidelines

These two documents deal exclusively with questions related to

4. Guide to using Framework documents

This chapter provides an introduction to the use of the Framework document family. It addresses:

4.1 Target audience

4.1.1 For the overall Framework

QA is properly an intergral part of the efforts of each Working Group. The QA WG charter succinctly summarizes the point that has been made from the first W3C conformance papers (see @@tbl/dd paper in member space?@@):

"[...] QA is to be considered a 'natural overhead' of any WG [...]. QA will succeed only if every person inside W3C participates in it."

Some parts of these Framework documents should prove useful to everyone involved with the work of W3C's Working Groups -- not only those WG members specializing in conformance test materials, but all of a WG's participants, and even external parties reviewing the products of the WGs.

4.1.2 For the Introduction

This first part, "Introduction", should be read by everyone involved with the work of the WGs. In addition to reviewing the scope and goals of the QA Activity, and QA activities within the WGs, it also provides:

As presently scoped, there will be seven documents in the Framework (see @@sec3.5@@). After the "Introduction", those parts that will be useful to any individual WG participant depends on the participant's role in the WG's work.

4.1.3 For the guideline parts

From a conformance and quality perspective, several roles are significant out in a WG's activities. The following identification of roles incorporates one principle from the "Process & Operational Guidelines" -- within a Working Group, there will be a group (WG-TS) that is focused on conformance test suites and tools. WG-TS consists of a subset of the WG members, and possibly other W3C members from outside of the WG.

all WG members
For any (potential) WG member, the charter and QA-definition parts of "Process & Operational Guidelines" (@@#2a@@) should be helpful in understanding what the WG has committed to deliver. Familiarity with the "Specification Guidelines" (@@#3a@@) will be helpful to any member who participates in the advancement of the WG's functional specifications to Recommendation.
WG spec editors & authors
As for all WG members, @@#2a@@ is useful. A good working understanding of @@#3a@@ will be needed in order to satisfy the spec guidelines and checkpoints, and "Specification Guidelines Techniques and Examples" (@@#3b@@) should be a valuable resource in choosing document structure, formats, and techniques that will facilitate satisfying the requirements.
WG chair
As the person ultimately responsible for both the advancement of the WG's specifications and the WG's QA projects, a familiarity with @@#2a@@, @@#3a@@, and "Technical Guidelines" (@@#4a@@) will be useful.
WG-TS participant
Those who are active in building the conformance test materials of the WG will need a good working understanding of both @@#4a@@ and "Technical Guidelines Techniques and Examples" (@@#4b@@). Because of the close dependency of test materials on the functional specifications, a good familiarity with @@#3a@@ and @@#3b@@ will also be helpful.
WG-TS moderator
The person who manages the WG's QA projects should have good working understandings of @@#3a@@, @@#3b@@, @@#4a@@, @@#4b@@. In addition, knowledge of all of the process and logistical setup guidelines of @@#2a@@ and techniques of @@#2b@@ are needed.
non-WG spec reviewers
Whether from other WGs, or the public at large, @@#3a@@ will be helpful to those who review a WG's specifications.
non-WG TM reviewers
Whether from other WGs, or the public at large, reviewers of the test materials of a WG should be familiar with @@#4a@@, and familiarity with @@#4b@@ as well would facilitate a critical review.
Reviewers of activity proposals & charters
For those W3C members who will be reviewing Activity proposals and proposed WG charters, and helping to form their AC representatives positions, the charter- and deliverable-related requirements defined in @@#2a@@ are helpful.
QA Activity members
Members of the QA WG are an expert resource for the W3C Working Groups, and accordingly should be expert on all parts of the Framework; members of the QA IG @@tbd@@, and accordingly need thorough familiarity with all parts as well.

4.2 Framework primer and guide

This section examines use of the Framework documents from the perspective of significant milestones in a WGs activities, from writing the charter through publishing Recommendations, and building or acquiring test suites and tools.

4.2.1 First step -- charter commitment

Because QA is considered to be an integral part of the activities of each WG, each WG has to consider and commit to a set of QA deliverables. A spectrum of possible commitments, and associated guidelines, is found in "Process & Operational Guidelines", section "@@#2a-link-tbd@@".

If a WG is being newly formed, then the WG charter needs to document its QA deliverables, just as it does all other WG deliverables. See @@#2a-link-tbd@@. If a WG is being re-chartered, this case should be considered the same as a newly formed WG -- the charter should document the WG's committed set of conformance and test related deliverables.

For an already-chartered Working Group undertaking new QA projects, if these deliverables are not documented in the charter already, then the W3C Process Document [W3C-PROCESS] suggests that the charter should be amended to document the new deliverables. Again, see @@#2a-link-tbd@@.

4.2.2 Set up processes & logistics

Once WG commitments to QA deliverables are made and documented in the charter, the next step is to set up the processes and logistics that the WG will use for managing its QA activities. These include, among other things:

See the sections (@@#2a-link-tbd@@) in "Process and Operational Guidelines", about how to do this.

4.2.3 Planning and writing the specification

There is a very tight relationship between how the specification (Rec) is written on the one hand, and on the other hand its testability and its suitability as the foundation for interoperable implementations.

New specification. "Specification Guidelines" (@@link-tbd@@) should be applied from very beginning. Among the key topics that authors and editors should consult are:

Consider the guidelines and checkpoints of "Specification Guidelines" even at the stage of planning the structure and presentation style of the spec. Along with W3C "pubrules" (@@link@@) and "Style Manual" (@@link@@), document authors and editors should refer to the spec guidelines throughout, about testable language, clarity, conciseness.

New "Edition" of specification. A new "Edition" of the same functional level of a specification is typically used for incorporation of errata (e.g., XML 1.0 Second Edition). Normally, this should not be considered as a good time to bring a specification for "Spec Guidelines" conformance, as the latter could significantly disrupt and restructure the specification.

New "Version" of specification. A new "Version" of the specification refers to a significant functional change and enhancement. This presents a good opportunity to improve the testability and implementability of the specification, as just described for new specifications.

4.2.4 Reviewing and progressing the specification

This stage in the specifications life has two significant aspects:

When the specification is published in TR space (@@link-tbd@@), then non-WG W3C members and the general public begin to review and comment. Such reviewers should consult and understand the material in "Specification Guidelines" (@@#3a@@), in order to have an informed set of evaluation criteria for the conformance, testability, and interoperability aspects of the specification.

WG members and especially WG-TS members should refer to @@link-tbd@@ in the "Process & Operational Guidelines", regarding synchronization of the test materials work with the spec progression. The project enters "The Matrix" (@@link to above resource description@@) at Last Call WD (if not sooner). And a de-facto process convention is emerging, that there should be significant conformance test materials at the stage of CR-exit/PR-commencement. This is the same timing as the explicit process requirement of two interoperable implementations.

4.2.5 Designing and building test materials

There are several scenarios for how the WG "builds" its conformance test materials:

In-WG build. Before starting the technical development, the WG-TS members should be thoroughly familiar with the material in "Technical Guidelines" (@@#4a@@). There is useful information for both high-level planning -- e.g., breadth-first Basic Effectivity versus fully detailed suite? -- as well as specific technical detail for the building of individual test cases. Another aspect of building test materials is an acceptance procedure for the individual bits, as they are built. This is addressed in the section @@link-tbd@@ of "Process & Operational Guidelines".

Import completed test materials. Several high-quality test suites have been developed outside of the relevant W3C WG, and then transferred to the WG. WG's which are considering such a transfer should refer to to @@link-tbd@@ of "Process & Operational Guidelines". Clearly, the quality of the candidate test materials should be carefully assessed, and for this the "Technical Guidelines" (@@#4a@@) can provide useful assessment criteria.

Assemble contributions. Some test suites have been built by implementing processes to assemble significant chunks of material from outside (or internal member) contributions. The @@link-tbd@@ section of the "Process and Operational Guidelines" addresses the steps needed to complete such a transfer. In addition, there should be careful quality assessment of contributions, for which "Technical Guidelines" (@@#4a@@) can be useful. Finally, there must be procedures for submission, review, and integration of contributions, which are described in the @@link-tbd@@ section of "Process & Operational Guidelines".

4.2.6 Publication of test materials

Typically, a WG-TS group will want to publish releases of test materials, particularly as the specification advances through its final stages (e.g., PR) towards Recommendation. Test material publication is addressed in @@link-tbd@@ section of "Process & Operational Guidelines".

As soon as the test materials become public, then there is definite need for a procedure to process challenges to correctness, make determinations, and appeal decisions. This is addressed in @@link-tbd@@ of "Process & Operational Guidelines".

Publication of test materials often comprises an implicit (or explicit) invitation for contributions. The considerations in "Assemble Contributions" (@@link@@) are equally applicable here.

4.2.7 Specification publication and beyond

When the specification reaches Recommendation, there is typically a concurrent publication of the test materials. This might be considered a "final" publication, or ongoing development may still be planned according to one of the mechanisms discussed above. In any case, a maintenance procedure must be in place for the test materials. Firstly, there are tie-ins between approved spec errata and validity or applicability of particular tests -- mechanisms for this are discussed in "Technical Guidelines", and the general process overview is discussed in @@link-tbd@@ of "Process and Operational Guidelines". Secondly, there is the above-discussed need for both challenge/review/appeal processes. Finally, even if WG-TS active development of test materials ceases, it may be desired to continue to consider submissions, review and integrate them per the guidelines of @@link-tbd@@ of "Process & Operational Guidelines".

4.2.8 Life after WG

It is possible that the WG and WG-TS may disband after its charter expires. This situation, and what to do about test materials, is discussed in @@link-tbd@@ of "Process & Operational Guidelines".

5. Conformance

[This document should have a conformance clause. Uncertain what it should say now, as it is unclear what is the normative force of this document. At least it should probably address what is involved in (possibly voluntary) conformance to this document. Precisely what it might say is an issue that must be addressed before FPWD.]

6. Acknowledgments

The following QA Working Group and Interest Group participants have contributed significantly to the content of this document:

7. References

[@@Ed. note: This has not yet been cut down from the original all-in-one Framework document. It should be edited to only include those references which actually pertain to this document.]

ISSUES-LIST
QA Activity Issues List, maintained by the QA Working Group.
MATRIX
W3C-wide conformance activity survey covering all the WGs, "The Matrix".
PROCESS
W3C Process Document, 19 July 2001.
TAXONOMY
QA Activity test taxonomy, a classification scheme for conformance test materials.
QA-BIBLIOGRAPHY
A centralized QA bibliography, listing the various notes and documents of the QA Working Group.
QA-GLOSSARY
Comprehensive glossary of QA terminology. (Under construction.)
QAIG
Quality Assurance Interest Group of the W3C QA Activity.
QAWG
Quality Assurance Working Group of the W3C QA Activity.
REC-TRACK
Stages and milestones in the W3C Recommendation Track, per the Process Document (section 5.2).
RFC2119
Key words for use in RFCs to Indicate Requirement Levels, March 1997.
SVGTEST
SVG Working Group's test suite resource page.
WCAG10
Web Content Accessibility Guidelines, version 1.0, W3C Recommendation, 5 May 1999.
WORKSHOP
Proceedings of 1st QA Workshop, held at NIST, March 2001.
WWW-QA-ARCHIVE
Public email list archive for the QA Activity.
WWW-QA-WG-ARCHIVE
QA Working Group email archive, publicly archived.