This document introduces a common framework for enhancing the quality practices of the W3C Working Groups in the areas of specification editing, production of test materials, and coordination efforts with internal and external groups. It presents introductory, roadmap, and orientation information for the Framework document family of the Quality Assurance (QA) Activity. This is the first of the family of QA Framework documents, which includes the other existing or in-progress specifications: Process & Operational Guidelines; Specification Guidelines; and, Test Materials Guidelines.
This section describes the status of this document at the time of its publication. Other documents may supersede this document. The latest status of this document series is maintained at the W3C.
This document is a W3C Working Draft (WD), made available by the W3C Quality Assurance (QA) Activity for discussion by W3C members and other interested parties. For more information about the QA Activity, please see the QA Activity statement.
This version is the first public Working Draft, and incorporates the discussions at the first QA face-to-face, plus several subsequent QA Working Group [QAWG] teleconferences, and supersedes all previous drafts. It is expected that updated WD versions of this document will be produced regularly, along with other members of the Framework documents family. Future progression of this document beyond Working Draft is possible, but has not yet been determined.
Please send comments to email@example.com, the publicly archived list of the QA Interest Group [QAIG]. Please note that any mail sent to this list will be publicly archived and available, do not send information you wouldn't want to see distributed, such as private data.
Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress".
A list of current W3C Recommendations and other technical documents can be found at http://www.w3.org/TR.
1. About this QA framework
1.2 Why do we need a QA framework?
1.3 The Framework is for the Working Groups
1.4 The Framework goals
2. QA resources
2.1 Expertise & consultancy
2.2 Technical assets
3. Structure & content of Framework documents
3.1 Applicable domain
3.2 Document orientation & structure
3.3 Normative guidelines
3.5 Documents roadmap
4. Guide to using Framework documents
4.1 Target audience
4.2 Framework primer and guide
This document introduces a common framework for promoting and facilitating the quality practices of the W3C Working Groups. It presents introductory, roadmap, and orientation information for the Framework document family of the Quality Assurance (QA) Activity.
The documents in this family collectively aim to provide the W3C Working Groups with resources and tools for all phases and aspects of their quality and conformance activities,
The structure and content of the Framework documents family are detailed in the later chapter, "Structure & content of Framework documents", but the main (root) documents are:
Some of these documents have an additional associated "Examples & Techniques" document, which contain detailed, technology-specific methods and examples for implementing the guidelines.
The ultimate mission of W3C is "Leading the Web to its full potential...", and W3C's many active working groups contribute towards this goal, in large part, by building a family of high-quality Web standards. These standards are the building blocks of a architecture to maximize the Web's potential.
Most would agree that the writing of the standard is not, in itself, the end purpose of the Working Groups (WGs). It is a means to achieve the real end: the success of the standardized technology in practice and practical application. One measure of the success that can be applied to most Web standards is a set of complete, correct, high-quality, interoperable implementations that actually realize the functionality of the standard.
In the past few years, several WGs have discovered that the early production of test materials (TM) has been a major contributor to the success of their standards. The benefits of test materials work are several-fold, but a couple of the big benefits are: early detection and correction of unintentionally vague, contradictory, or unimplementable language in the technical reports (TRs) [W3C-TR]; and, a solid basis for the development of the interoperable implementations demonstration required by the W3C Process Document.
The individual conformance test materials efforts of the Working Groups have in several cases been inspired and on target, in others have been rudimentary or incomplete. Even amongst the best efforts, however, there is little uniformity in the quality-control principles, methodologies, or even such basics as terminology. Moreover, these efforts are distributed throughout W3C, making it difficult or at least time consuming for WGs pursuing their QA goals to find and take advantage of what has already been done. Each WG has started from scratch, researching the numerous existing TS activities and defining their own processes, operational framework and deliverables.
This defines the mission of our QA Activity. QA aims assist the WGs by assembling and organizing the best of a body of good practice, defining a framework, and providing a roadmap for the working groups. These best-practice guides should dramatically ease the job of planning and implementing QA projects within the Working Groups.
The last underscores a key reality of improved quality practices associated with W3C technical reports. QA is not a comprehensive service proffered to the Working Groups, nor is QA an aspect of WG operation that can be exported to an external entity. Similarly, quality practices are not mandates imposed on the Working Groups from outside the WG, despite a trend towards higher expectations for Working Groups' QA deliverables and for the quality of their products in general.
QA is properly an intergral part of the efforts of each Working Group. Each WG will provide the majority of the resources needed to meet its QA commitments. The QA Activity provides this Framework, as well as other tools and assets, to aid and guide the WG efforts.
Therefore the QA Framework is for everyone who is involved with the work of W3C's Working Groups. Some part of these Framework documents should be useful and helpful to anyone who is an active participate in generating such deliverables -- specifications and conformance test materials alike -- as well as anyone who is involved in assessing or evaluating such deliverables.
Foremost amongst the purposes of defining a common QA Framework is the principal goal of the QA Activity itself:
As an integral part of the working modes of the WGs, QA is ideally a development context which is applicable to a spec's development from its earliest stages through completion. This Framework provides a collection of best-practice principles and guidelines, which span the life of WG activities from the identification of QA deliverables in the WG charters, through post-REC and possibly even post-WG maintenance.
While some might perceive QA projects as a regrettable drain on WG resources, there is ample experience, both within W3C as well as other standards venues, that shows significant improvement to the products of the WGs. This equates to a sound business case for the early investment of resources -- the cost in resources is more than offset by the benefits of more implementable and better implemented Recommendations. In fact, in some cases there could be a net resource gain, by dodging problems that would be costly to fix if detected late in a specification's development process.
This Framework document family should provide to those undertaking quality assurance projects:
More specific goals include:
A variety of resources are provided by the QA Activity, including:
A comprehensive survey of the resources of QA Activity is beyond the scope of this document, and may be located on the QA Working Group [QAWG] and QA Interest Group [QAIG] Web pages. However the last two items above are pertinent to other parts of these Framework documents.
The QA Working Group expects to have access to considerable expertise on a spectrum of QA topics, methods, and techniques. One key aim of the QA Activity is to make this expertise available to the Working Groups, at least on an as-needed (request) basis, as they plan and implement their conformance and quality projects. The degree of participation of QA experts in WG projects will likely be limited by QAWG staff resource constraints, but a consultancy role should almost always be possible.
The Framework document, "QA Framework: Process & Operational Guidelines", explores this topic in some detail, including:
While the initial resources that the QA Working Group offers to other WGs focuses heavily on the Framework documents and available consulting expertise, it is intended that ongoing development within QA Activity will result in a collection of commonly useful assets. Some anticipated offerings include:
Most of these will be discussed in "Test Materials Examples and Techniques" [TEST-EXTECH], to illustrate how to meet the "Test Materials Guidelines" [QAF-TEST] with common tools.
Conformance test materials are being developed in an organizational landscape within W3C (and without) which is characterized by:
These framework documents should be applicable and useful throughout this diverse context.
The Framework documents will utilize the style of structuring guidelines and verifiable checkpoints, similar to WAI standards such as WCAG [WCAG10]. The degree to which a simple cookbook guide is achievable is complicated by the factors enumerated previously -- diversity of potential test materials, as well as organizational diversity. Complexity and diversity will be managed in the Framework document family by:
The guideline documents are closely linked to their respective "Examples & Techniques" documents. The linked material explores, defines, and presents examples of taxonomy-dependent implementations of checkpoints. This model from the WAI standards has served well to manage diversity and complexity.
These checkpoint technique documents are not intended to be a comprehensive survey of existing W3C practice. That is the purpose of the QA matrix [MATRIX]. That notwithstanding, several of the conformance test efforts will be drawn upon for examples.
The Framework guidelines -- or more precisely, the checkpoints associated with the guidelines -- comprise the individual conformance requirements of these Framework documents. Each guidelines document contains a conformance chapter which unambiguously defines conformance to the guidelines and checkpoints.
The relationship of the Framework documents to the W3C Process is undefined at the time of publication. It is not precluded that all or parts of these guidelines may ultimately be incorporated (by reference) into the W3C Process Document [PROCESS].
The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" will be used as defined in RFC 2119 [RFC2119].
Unusual terms in these framework documents are defined when first used, and most generally useful QA-specific terms will eventually be in the QA Glossary [QA-GLOSSARY].
The Framework document family includes:
These documents are explained in the following subsections.
This document is for everyone interested in QA within the W3C. Especially, it serves as a first read for people with the following roles:
The introduction document serves as a starter for most QA-related activities, especially as it contains pointers to more specific information on particular steps of the QA process, both by examples and by linking to that information explicitly.
This document is primarily targeted to people explicitly involved in QA activities. This includes both those within the W3C Working Groups (as mentioned in the previous section) as well as others from organizations external to the W3C involved in developing test materials. The goal of this document is to present procedural and operational guidelines for groups undertaking conformance materials development. The document contains information about:
Accompanying the Process and Operational Guidelines is a companion document, Examples and Techniques. The Process and Operational Examples and Techniques document provides examples and pointers to existing QA work, illustrating the principles and guidelines set forth in the Process and Operational Guidelines.
This document is primarily targeted to people who are involved with either writing or reviewing the specifications of a WG, as well as those explicitly involved in QA activities. The goal of this document is to present guidelines for the writing of clearer, more implementatable, and better testable technical reports. The document presents traditional best-practice guidelines for:
It also addresses advanced and emerging methods for easier generation of TS schemata, tests, and assertions:
Accompanying the Specification Guidelines is a companion document, Examples and Techniques. The Specification Examples and Techniques document provides examples and pointers to existing QA work, illustrating the principles and guidelines set forth in the Specification Guidelines.
This document is primarily targeted to people who are actively involved with either building or assessing and acquiring conformance test materials. Accompanying the Test Materials Guidelines is a companion document, Examples and Techniques. Together, the two documents cover:
This chapter provides an introduction to the use of the Framework document family. It addresses:
QA is properly an intergral part of the efforts of each Working Group. The QA WG charter succinctly summarizes the point that has been made from the earliest W3C conformance papers:
"[...] QA is to be considered a 'natural overhead' of any WG [...]. QA will succeed only if every person inside W3C participates in it."
Some parts of these Framework documents should prove useful to everyone involved with the work of W3C's Working Groups -- not only those WG members specializing in conformance test materials, but all of a WG's participants, and even external parties reviewing the products of the WGs.
This first part, "Introduction", should be read by everyone involved with the work of the WGs. In addition to reviewing the scope and goals of the QA Activity, and QA activities within the WGs, it also provides a detailed roadmap and guide to the Framework documents family.
As presently scoped, there will be seven documents in the Framework (see "Documents Roadmap"). After the "Introduction", those parts that will be useful to any individual WG participant depends on the participant's role in the WG's work.
From the perspective of conformance and quality practices, several roles are significant in a WG's activities. In the following it is assumed that, associated with each WG, there will be a "Test" group (WG-TS), that is focused on conformance test suites and tools. WG-TS consists of a subset of the WG members, and possibly other W3C members from outside of the WG. (The WG-TS requirement is detailed in the process and operational guidelines.)
This section examines use of the Framework documents from the perspective of significant milestones in a WGs activities, from writing the charter through publishing Recommendations, and building or acquiring test suites and tools.
Because QA is considered to be an integral part of the activities of each WG, each WG has to consider and commit to a set of QA deliverables. A spectrum of possible commitments, and associated guidelines, is found in the charter-commitment checkpoint(s) of the process and operational guidelines ([QAF-OPS], Checkpoint 1.1).
If a WG is being newly formed, then the WG charter needs to document its QA deliverables, just as it does all other WG deliverables. Again, see the charter-commitment checkpoint(s) of the process and operational guidelines ([QAF-OPS], Checkpoint 1.1). If a WG is being re-chartered, this case should be considered the same as a newly formed WG -- the charter should document the WG's committed set of conformance and test related deliverables.
For an already-chartered Working Group undertaking new QA projects, if these deliverables are not documented in the charter already, then the W3C Process Document [PROCESS] suggests that the charter should be amended to document the new deliverables. Again, see the charter-commitment checkpoint(s) of the process and operational guidelines ([QAF-OPS], Checkpoint 1.1).
Once WG commitments to QA deliverables are made and documented in the charter, the next step is to set up the processes and logistics that the WG will use for managing its QA activities. These include, among other things:
See the sections about staffing WG QA efforts ([QAF-OPS], Guideline 4) and producing the WG's QA process document ([QAF-OPS], Guideline 6) in the process and operational guidelines, about how to do this.
There is a very tight relationship between how the specification (Rec) is written on the one hand, and on the other hand its testability and its suitability as the foundation for interoperable implementations.
New specification. "Specification Guidelines" [QAF-SPEC] should be applied from very beginning. Among the key topics that authors and editors should consult are:
Consider the guidelines and checkpoints of "Specification Guidelines" [QAF-SPEC] even at the stage of planning the structure and presentation style of the spec. Along with W3C "pubrules" (the member-only rules document for W3C editors who are publishing technical reports) and "Style Manual" [STYLE-MAN], document authors and editors should refer to the spec guidelines throughout, about testable language, clarity, conciseness.
New "Edition" of specification. A new "Edition" of the same functional level of a specification is typically used for incorporation of errata (e.g., XML 1.0 Second Edition). Normally, this should not be considered as a good time to bring a specification for "Specification Guidelines" conformance, as the latter could significantly disrupt and restructure the specification.
New "Version" of specification. A new "Version" of the specification refers to a significant functional change and enhancement. This presents a good opportunity to improve the testability and implementability of the specification, as just described for new specifications.
This stage in the specifications life has two significant aspects:
When the specification is published in TR space, then non-WG W3C members and the general public begin to review and comment. Such reviewers should consult and understand the material in "Specification Guidelines" [QAF-SPEC], in order to have an informed set of evaluation criteria for the conformance, testability, and interoperability aspects of the specification.
WG members and especially WG-TS members should refer to the sections (checkpoints) of the process and operational guidelines", regarding specification exit criteria ([QAF-OPS], checkpoint 1.2) and synchronization of the test materials with specification progression ([QAF-OPS], Guideline 5). The project enters "The Matrix" [MATRIX] at Last Call WD (if not sooner). Additionally, a de-facto process convention is emerging, that there should be significant conformance test materials at the stage of CR-exit/PR-commencement. This is the same timing as the explicit process requirement of two interoperable implementations.
There are several scenarios for how the WG "builds" its conformance test materials:
In-WG build. Before starting the development, the WG-TS members should be thoroughly familiar with the material in "Test Materials Guidelines" [QAF-TEST]. There is useful information for both high-level planning -- e.g., breadth-first Basic Effectivity versus fully detailed suite? -- as well as specific detail for the building of individual test cases. Another aspect of building test materials is an acceptance procedure for the individual bits, as they are built. This is addressed in the review-procedures requirements ([QAF-OPS], checkpoint 6.5) of the process and operational guidelines.
Import completed test materials. Several high-quality test suites have been developed outside of the relevant W3C WG, and then transferred to the WG. WG's which are considering such a transfer should refer to the test materials transfer guideline ([QAF-OPS], Guideline 7) of the process and operational guidelines. Clearly, the quality of the candidate test materials should be carefully assessed, and for this the "Test Materials Guidelines" [QAF-TEST] can provide useful assessment criteria.
Assemble contributions. Some test suites have been built by implementing processes to assemble significant chunks of material from outside (or internal member) contributions. The "Process and Operational Guidelines" [QAF-OPS] addresses the steps needed to complete such a transfer -- the are the same as the preceding paragraph about transferring completed test materials. In addition, there should be careful quality assessment of contributions, for which "Test Materials Guidelines" [QAF-TEST] can be useful. Finally, there must be procedures for submission, review, and integration of contributions ([QAF-OPS], checkpoints 6.2, 6.4, 6.5), which are described in several related checkpoints of the process and operational guidelines.
Typically, a WG-TS group will want to publish releases of test materials, particularly as the specification advances through its final stages (e.g., PR) towards Recommendation. Test material publication is addressed in TM publication checkpoint ([QAF-OPS], checkpoint 6.6) of the process and operational guidelines.
As soon as the test materials become public, then there is definite need for a procedure to process challenges to correctness, make determinations, and appeal decisions. This is addressed in a test appeal checkpoint ([QAF-OPS], checkpoint 8.3) of the process and operational guidelines.
Publication of test materials often comprises an implicit (or explicit) invitation for contributions. The considerations described in "Assemble Contributions" are equally applicable here.
When the specification reaches Recommendation, there is typically a concurrent publication of the test materials. This might be considered a "final" publication, or ongoing development may still be planned according to one of the mechanisms discussed above. In any case, a maintenance procedure must be in place for the test materials. Firstly, there are tie-ins between approved specification errata and validity or applicability of particular tests -- mechanisms for this are discussed in "Test Materials Guidelines" [QAF-TEST], and the general process overview is discussed in the errata update checkpoint ([QAF-OPS], checkpoint 8.2) of the process and operational guidelines. Secondly, there is the above-discussed need for both challenge/review/appeal processes. Finally, even if WG-TS active development of test materials ceases, it may be desired to continue to consider submissions, review and integrate them per the requirements of the test-contribution checkpoint ([QAF-OPS], checkpoint 8.1) of the process and operational guidelines.
It is possible that the WG and WG-TS may disband after its charter expires. This situation, and what to do about test materials, is discussed in a "secure repository" checkpoint ([QAF-OPS], checkpoint 2.2) of the process and operational guidelines.
This introductory document to the Framework documents family is entirely informative. It contains no conformance requirements. Therefore the concept of conformance to this document is not applicable.
The guidelines documents of this Framework document family do contain conformance requirements, and each such document contains a conformance chapter with an unambiguous definition of conformance to the documents guidelines and checkpoints.
The following QA Working Group and Interest Group participants have contributed significantly to the content of this document: