Copyright ©2001 W3C® (MIT, INRIA, Keio), All Rights Reserved. W3C liability, trademark, document use and software licensing rules apply.
This document introduces a common framework for building conformance test suites and tools for W3C specifications. It presents introductory, roadmap, and orientational information for the Quality Assurance (QA) Activity, as well as process and organizational guidelines, for groups undertaking development of conformance materials. This is the first of a family of QA Framework documents, which includes the other existing or in-progress specifications: Process & Operational Guidelines; Specification Guidelines; and, Technical Guidelines.
This document is a Note, made available by the W3C Quality Assurance Activity (QAA) for discussion on the QA email discussion list.
This version incorporates the discussions at the first QA face-to-face, and supersedes the first, incomplete draft. Open issues and other incompletions are flagged with "@@". Where applicable, the presentation style is Guidelines-plus-Checkpoints, similar to WAI standards. Where appropriate, some documents in this Framework family (but not this one) have an associated and copiously hyperlinked "Examples & Techniques" document, in which details and especially discretionary choices are elaborated. In this document version, guidelines and checkpoints are not yet numbered or formatted [@@we need to do this before release].
Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress".
Please send comments to www-qa@w3.org.
1. About this QA framework
1.1 Introduction
1.2 Why do we need a QA framework?
1.3 Who the framework is for
1.4 The Framework goals
2. A survey of QA resources
2.1 Mail list & archive
2.2 QA Web pages
2.3 Glossary
2.4 Taxonomy
2.5 The Matrix
2.6 Notes
2.7 QA Framework
2.8 QA issues document(s)
2.9 Expertise & consultancy
2.10 Technical assets
3. The QA framework documents
3.1 Applicable domain
3.2 Document orientation & structure
3.3 Informative guidelines
3.4 Terminology
3.5 Documents roadmap
3.5.1 Contents
3.5.2 Introduction
3.5.3 Process & Operational
Guidelines
3.5.4 Specification Guidelines
3.5.5 Technical Guidelines
4. Using the framework documents
4.1 Getting started
4.2 Roles
4.3 First step -- charter
4.4 Set up processes & logistics
4.5 Planning and writing the spec
4.6 Designing and building test materials
5. Conformance
6. Acknowledgments
7. References
This document introduces a common framework for building conformance test suites and tools for W3C specifications. It presents introductory, roadmap, and orientational information for the Quality Assurance (QA) Activity, and aims thereby to provide a first point-of-entry and introduction to the Activity itself.
It also presents some process and organizational guidelines, for groups undertaking development of conformance materials. Thereby it aims to introduce the first planning and action items for those who will be active in the QA projects of their Working Groups.
This document is one of a family of documents that together define a common framework for building conformance test materials for W3C Recommendations. The document family covers: a QA roadmap and resource guide; process, organizational, and operational guidelines for groups undertaking development of conformance materials; and finally detailed technical guidelines for the test suites (TS) and tools themselves.
The (root) documents in this family are:
[@@hyperlinks on above]
Some of these documents lead to associated "Examples & Techniques" documents, which contain detailed, technology-specific methods and examples for implementing the guidelines.
The ultimate mission of W3C is "Leading the Web to its full potential...", and W3C's many active working groups contribute towards this goal, in large part, by building a family of high-quality Web standards. These standards are the building blocks of a technical architecture to maximize the Web's potential.
Most would agree that the writing of the standard is not, in itself, the end purpose of the WGs. It is a means to achieve the real end: the success of the standardized technology in practice and practical application. One measure of the success that can be applied to most Web standards is a set of complete, correct, high-quality, interoperable implementations that actually realize the functionality of the standard.
In the past few years, several WGs have discovered that the early production of test materials (TM) has been a major contributor to the success of their standards. The benefits of test materials work are several-fold, but a couple of the big benefits are: early detection and correction of unintentionally vague, contradictory, or unimplementable language in the specifications (RECs); and, a solid basis for the development of the interoperable implementations demonstration required by the W3C Process Document.
The individual conformance test materials efforts of the Working Groups have in several cases been inspired and on target, in others have been perfunctory or rudimentary. Even amongst the best efforts, however, there is little uniformity in the quality-control principles, methodologies, or even such basics as terminology. And even those WGs which have resources sufficient to their motivation have effectively had to re-invent their processes, operational framework, and technical bits from scratch. Just figuring out where to start -- researching the numerous existing TS activities -- can be a hurdle.
This defines our the mission of our QA Activity. QA aims assist the WGs by assembling and organizing the best of a body of good practice, defining a framework, and providing a roadmap for the working groups. These best-practice guides should dramatically ease the job of [...to be completed...]
The last underscores a key reality of improved QA efforts associated with W3C Recommendations. QA is not a comprehensive service proffered to the Working Groups. Nor is it a mandate imposed on the Working Groups from without, notwithstanding that higher expectations are evolving for the QA efforts and the quality of the products of the WGs in general.
QA is properly an intergral part of the efforts of each Working Group. Each WG will provide the majority of the resources needed to meet its QA commitments. The QA Activity provides this Framework, as well as other tools and assets, to facilitate and guide the WG efforts.
Foremost amongst the purposes of defining a common QA Framework is the principal goal of the QA Activity itself:
As an integral part of the working modes of the WGs, QA is ideally a development context which is applicable to a spec's development from its earliest stages through completion. This Framework provides a collection of best-practice principles and guidelines, which span the life of WG activities from the identification of QA deliverables in the WG charters, through post-REC and possibly even post-WG maintenance.
While some might resent a perceived drain on WG resources, there is ample experience, both within W3C as well as other standards venues, that shows significant improvement to the products of the WGs. This equates to a sound business case for the early investment of resources -- the cost in resources is more than offset by the benefits of more implementable and better implemented Recommendations. In fact, in some cases there could be a net resource gain, by dodging problems that would be costly to fix if detected late in a specifications process.
As one of the principal resources and deliverables of the QA Activity, ideally, this Framework document family should provide to those undertaking test suite projects,
More specific goals include:
Here's a summary of the resources provided by QAA.
Caveat. The information in this section may be somewhat volatile. Nevertheless it is considered useful to collect it in one place.
The QAA tries to conduct its business, as much as possible, on its public mail list,
The archive for the public mail list is found at:
All public working drafts of all Framework documents, as well as other notes and white papers, are discussed on this list. Working group meeting minutes are announced to the list. QA issues are raised and discussed here as well.
Some of the content of these Framework documents comes from discussions on this public list.
The QAA has another archived public list,
There is little activity yet (late-2001) on this list, and its best use is yet to be defined (@@tbd). The archive of this list is found at:
Virtually all of the resource provided by QAA are accessible from the QA Web pages.
A key permanent standing document of the QAA is a common glossary of QA-related terms, [GLOSSARY]. The need is obvious. In the public mail archive, for example, it is not difficult to find threads where different personal favorite terminologies are introduced for the same concepts. Even such seemingly universal terms as "interoperability" and "conformance" are not commonly understood.
These Framework documents will include QA-related definitions when first encountered (@@will they), and have a glossary appendix (@@will we?), but ultimately these terms will migrate to the Glossary.
(@@note. can't point to glossary yet -- doesn't exist)
The QA landscape is complex because of [...]. The QA taxonomy [TAXONOMY] is a classification scheme for the features of this landscape. [@@to be completed]
The Matrix [MATRIX]is the definitive survey of existing practice. It starts tracking every WG Rec-track project at Last Call WD, or earlier for others that have significant QA activities at earlier stages.
The matrix contains, for each Rec-track project, [@@to be completed--brief summary of the columns, i.e., the purpose of the columns, i.e., the information items that the matrix tracks for each project.]
White papers on selected topics. Examples: CUAP; other (look to Brussels minutes, action items for other Notes).
(@@We -- QA -- ought to set up a "Bibliography", which is a permanent resource that we can point to from here. It would list the volatile document set which includes things like CUAP, ConfReq submission, etc]
[@@to be completed]
-- this document family.
[@@to be completed]
TBD. (Is there one or two? Is it a public resource?)
[@@to be completed]
It's available. Ask. Limited (staff resource constraints), but advisory is almost always available.
[@@to be completed]
And here's some future anticipated offerings:
Conformance test materials are being developed in an organizational landscape within W3C (and without) which is characterized by:
These framework documents should be applicable and useful throughout this diverse context.
The Framework documents will utilize the style of guidelines and verifiable checkpoints, similar to WAI standards such as WCAG [WCAG10]. The degree to which a simple cookbook guide is achievable is complicated by the factors enumerated in the previous section -- diversity of potential test suites and tools, as well as organizational diversity. This complexity and diversity will be handled in the Framework document family by:
The guideline documents are closely linked to their respective "Examples & Techniques" documents. The linked material explores, defines, and presents examples of taxonomy-dependent implementations of checkpoints. This model from the WAI standards has served well to manage diversity and complexity.
These checkpoint technique documents are not intended to be a comprehensive survey of existing W3C practice. That is the purpose of the QA matrix [MATRIX]. That notwithstanding, several of the conformance test efforts will be drawn upon for examples.
These Framework guidelines are considered to be "informative guidelines". This does not preclude that portions of these guidelines may end up with a status other than Note, either by incorporation into the W3C Process Document [PROCESS], or by processing in the W3C Recommendation track.
One of the deliverables of the QA Activity is a Glossary [@@GLOSSARY]. Unusual terms in these framework documents are defined when first used, and most generally useful QA-specific terms will eventually be in the Glossary.
(@@Ed. note -- following is not uniformly implemented yet.)
Although the guidelines of this framework are nominally informative, the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" will be used as defined in RFC 2119 [RFC2119], and will be used as if the guidelines are normative.
The Framework document family includes:
These documents are explained in the following subsections.
[@@ed note: this is only roughly outlined; no detailed outline and not much descriptive prose yet]
(If you've read this far, you're started.)
Here is described how to get started with the QA stuff in your WG. By reference to the guidelines.
??? would it be useful to list the players ... director, AC, WG members, QAA people, document editors, WG's QA manager???
If you are a new WG, or if you are re-chartering, or even if you are in mid-life as WG ... you have some QA responsibilities and deliverables.
You need to make your commitment, as described in the Ops&Procs -- what you intend to do, internal versus external, identify resources. (@@would be nice to link into Ops&Procs. but it doesn't exist yet.)
After the charter, the next thing you attend to is your QA logistical setup and processes. Web site, discussion list, QA manager. See the section (@@tbd) in Procs & Ops about how to do this. Then, depending on what have committed in your charter, and where the TS is to be built, ...
Apply Spec Guidelines from very beginning. See "Spec Guidelines". Consider the guidelines and checkpoints even at the stage of planning the structure and presentation style of your spec.
(Along with W3C style manuals) refer to them throughout, about testable language, clarity, conciseness.
For: editors and authors especially; all WG in spec planning phase.
[@@this section ought to fork for: in-house build; import completed external TS; in-house assemble from significant external chunks.]
If you are building test materials in the WG, this is for you. Before starting your technical development, have a look at Technical Guidelines.
Portions of these are equally applicable to maintenance, as well as to review and acceptance of external contributions.
For: test builder; test contributors; test reviewers; TS editor.
[This document should have a conformance clause. Uncertain what it should say now, as it is unclear what is the normative force of this document. At least it should probably address what is involved in (possibly voluntary) conformance to this document. Precisely what it might say is an issue that prehaps we can address later?]
The following QA Working Group and Interest Group participants have contributed significantly to the content of this document:
In addition ideas expressed by [@@...there may be some others in the email archive, e.g., AT, whose ideas have been excerpted and who should be credited...] in QA email and telephone discussions have contributed to the contents of this document.
[@@Ed. note: This has not yet been cut down from the original all-in-one Framework document. It should be edited to only include those references which actually pertain to this document.]