QA Framework Primer

Lofton Henderson, CGM Open
Lynne Rosenthal, NIST
Mark Skall, NIST
Last Modified:
24 August 2005


QA Framework Primer provides a general orientation to the QA Framework. The QA Framework is a set of related documents (a Recommendation, WG Notes, FAQ, Matrix, and Wiki) whose goal is to improve the quality of specifications and the implementation of these specifications. This Primer provides a roadmap to these documents. It introduces and orients the reader by presenting the QA Framework from three different prespectives - a document view, a role-based view, and a Working Group milestones view.

Status of this document

This document was previously published as a part of The QA Handbook — see section 6 of first published working draft of that document.

QA Framework Primer is closely linked with and will continue to be coordinated with the other parts of the QA Framework: The QA Handbook; QA Framework: Specification Guidelines; Matrix, Test FAQ.; and some more informal materials about testing in the QA Activity Wiki.

Comments on this document may be emailed to, the publicly archived list of the QA Working Group. Commenters please note that your comments will be publicly archived and available, do not send information that should not be distributed, such as private data.

Table of contents

  1. Introduction
  2. Document View of the QA Framework
  3. Role-based View of the QA Framework
  4. Milestones View of the QA Framework
  5. Acknowledgments
  6. Change History

1. Introduction

QA Framework Primer provides a general orientation to the QA Framework family of documents. The QA Framework consists of a set of related documents (a Recommendation, WG Notes, FAQs, Matrix, and Wiki) whose goal is to improve the quality of specifications and the implementation of these specifications. This Primer provides a roadmap of these documents. It introduces and orients the reader by presenting the QA Framework from three different prespectives - a document view, a role-based view, and a Working Group milestones view.

By presenting three views into the QA Framework, the reader is able to read and focus on areas of interest. The reader is encouraged to jump into any section of the document, reading only those sections relevant to the reader or reading the Primer in its entirety.

The target audience for this Primer is anyone and everyone with an interest in finding out about the QA Framework documents - to get a general understanding of the documents, to identify which documents are of interest, and/or to know when to use which document.

The QA Framework consists of these existing documents:

In addition to these documents, the QAWorking Group also developed several templates to help implement good practices described in these documents. These templates are available (via links) from the QA Framework document to which it applies. In particular, there is a Charter template available from the QA Handbook and How to write a Conformance Clause template available from the Specification Guidelines document.

2. Document View of the QA Framework

The following provides an overview of each document in the The QA Framework.

QA Handbook (QAH)

This document is a non-normative handbook about the process and operational aspects of certain quality assurance practices of W3C's Working Groups, with particular focus on testability and test topics. It is a a practical guide about applying good practices and quality assurance techniques to WG activities, especially developing Recommendations and test materials. It flags avoidable pitfalls, and provides techniques, tools, and templates that will facilitate and accelerate the work of the groups. Guidance in QAH is supported by real-world stories and examples. QAH will help Working Groups avoid known pitfalls and benefit from experiences gathered from the W3C Working Groups.

Specification Guidelines (Spec GL)

This document is a guide for editors of W3C specifications. It helps W3C editors write better specifications, by making a specification easier to interpret without ambiguity and clearer as to what is required in order to conform. It focuses on how to define and specify conformance. It addresses the most basic and critical topics with respect to conformance, including how to convey what is required for an implementation in order to conform. It also addresses how a specification might allow variation among conforming implementations. The document presents guidelines or requirements, supplemented with good practices, examples and techniques. As a result of using SpecG L, Working Groups will be able to produce specifications that are precise, easier to interpret without ambiguity or confusion, and clearer as to what is required in order to conform.

Variability in Specifications

This document details some of the most important conformance-related concepts evoked in the QA Specification Guidelines, by developing some of the analysis that need to be considered while designing a specification and providing advanced techniques, particularly for dealing with conformance variability and complexity. This document analyzes how design decisions of a specification's conformance model may affect its implementability and the interoperability of its implementations. To do so, it introduces the concept of variability - how much implementations conforming to a given specification may vary among themselves - and presents a set of well-known dimensions of variability. Its goal is to raise awareness of the potential cost that some benign-looking decisions may have on interoperability and to provide guidance on how to avoid these pitfalls by better understanding the mechanisms affected by variability.

Test FAQ

Test FAQ provides introductory information about the purpose of testing, how to get started, and what the testing process involves. This FAQ primarily documents what is already considered good testing practice or the norm, but it also includes a number of advanced testing goals that have not yet been fully achieved by any Working Group. The Test FAQ document is addressed to those who develop tests or organize testing efforts. It should also be useful to those who develop specifications or who run tests. It stresses early planning for testing, defines what makes a good test and examines test reporting, publishing and packaging of tests.

Matrix of W3C Specifications

The Matrix is a table of W3C Specifications that are in at least the Last Call stage (i.e., at Last Call, Candidate Recommendation, Proposed Recommendation or Recommendation). For each specification entry, a symbol indicates if the specification has a conformance clause and if there are conformance tools or test suites associated with the specification.

QA Wiki

The Wiki is an open forum that allows anyone to contribute, share and develop QA ideas, tools and methods at W3C. Anyone is encouraged to add individual opinions, comments, and solutions to problems directly on the pages. Topics focus on writing good specifications and building test suites.

3. Role-based View of the QA Framework

This section identifies the parts of the The QA Framework that might interest those who fill the various roles within a Working Group or that interact with a Working Group. While each part of framework is targeted itself at a specific principal audience, the various parts might have somewhat broader interest and applicability.

all participants
For any (potential) Working Group participant, the early planning and commitment parts of The QA Handbook might provide helpful context for understanding what the group has committed to deliver. Familiarity with the Specification Guidelines will be helpful to any participant who actively participates in the advancement of the specifications to Recommendation.
spec editors and authors
As for all participants, The QA Handbook might be interesting, for shedding some light on the context in which the group is operating. A good working understanding the Principles and Good Practices of Specification Guidelines, together with its collected examples, tools, and templates, should be a valuable resource in choosing document structure, formats, techniques and conformance-related designs that will facilitate production of a high-quality specification.
As the person ultimately responsible for all aspects of the group's work, a familiarity with the guidance for operations and process of The QA Handbook should be helpful — Chairs and Staff Contacts are the principal intended audience. Some familiarity with the advice and guidance of Specification Guidelines should be helpful as well, as the Chairs ultimately oversees the advancement of the specifications.
Test Task Force participant
Those who are active in building the test materials of the Working Group should benefit from reading the Test FAQ as well as the articles regarding test materials in the QA Wiki, and from their associated examples, techniques, and tools. Because of the close dependency of test materials on the functional specifications, a familiarity with the Specification Guidelines could be useful as well.
Test Task Force lead
The person who manages the Working Group's test suite projects should have working understandings of guidance and techniques for specifications of Specification Guidelines , as well as the main points of the Test FAQ.
Non-Working Group spec reviewers
Whether from other Working Groups, or the public at large, the Specification Guidelines will be helpful to those who review a Working Group's specifications, by providing some objective metrics by which to measure the specifications.
Reviewers of Activity proposals & charters
For those W3C Members who will be reviewing Activity proposals and proposed Working Group charters, and helping to form their Advisory Committee Representative's positions, the early planning and commitment parts of The QA Handbook might be helpful in evaluating whether or not the group's attention to test and quality deliverables is appropriate and consistent with the Working Group's overall program of work.

4. Milestone View of the QA Framework

This section identifies which parts (sections and documents) of the QA Framework are useful during the life of a Working Group. It progresses through significant milestones in a Working Group's life, from writing a Charter through publishing Recommendations, and looks at associated test suite and other quality assurance activities.

First Step — QA Commitment

Because QA is properly an integral part of the activities of each Working Group, each Working Group has to consider and commit to a set of QA deliverables appropriate to its work. A spectrum of possibilities are discussed and illustrated in the early planning and commitment module of The QA Handbook.

If a Working Group is being newly formed, and if the it is able to anticipate and agree at Charter time on deliverables like test suites, then it should consider documenting those QA deliverables in its Charter, just as it does all other deliverables. Again, see the early planning and commitment section of The QA Handbook. A Working Group being re-chartered is a similar case to a newly formed group, although the scope and direction of its work might be clearer.

For an already-chartered Working Group undertaking new test and other QA-related activities, if these deliverables are not documented in the Charter already, then there are a couple of options. The W3C Process Document describes how to amend a Charter to accommodate significant new deliverables, if it wishes to take this route.

Set Up Processes and Logistics

Once the Working Group is off and running, and assuming that it has planned on some test- or other quality-related deliverables, the next step is to chose and document the processes and logistics that it will use for its QA activities. These include such typical details as:

The sections within the Day-to-Day operations module of The QA Handbook give good-practice advice about how to do this, plus examples and a handy template for writing a QA Process Document.

Planning and Writing the Specification

There is a tight bond between how a specification is written on the one hand, and on the other hand its testability and its suitability as a basis for interoperable implementations.

New specification

QA Framework: Specification Guidelines should be applied from very beginning. Among the key topics that it addresses are:

Consider the advice of QA Framework: Specification Guidelines even at the stage of planning the structure and presentation style of the spec. Along with W3C "pubrules" and W3C Manual of Style, spec editors should refer to the spec guidelines throughout their work, on topics like testable language, clarity, conciseness.

New Edition of specification

A new Edition of the same functional level of a specification is typically used for incorporation of errata (e.g., XML 1.0 Second Edition). Normally, this should not be considered a good time to align a specification to QA Framework: Specification Guidelines — the changes associated with such alignment could significantly disrupt and restructure the specification.

New Version of specification

A new Version of the specification refers to a significant functional change and enhancement. This presents a good opportunity to improve the testability and implementability of the specification, as just described for new specifications.

Reviewing and Progressing the Specification

This stage in the specification's life has two significant aspects:

When the specification is published in TR space, then W3C Members not participating to the Working Group and the general public begin to review and comment. It would be valuable that such reviewers consult and understand the material in QA Framework: Specification Guidelines — it gives and informed set of evaluation criteria about the conformance, testability, and interoperability aspects of the specification.

Working Group participants and especially Test Task Force participants should refer to the good-practice pieces about advancement criteria and synchronization (between specs and test materials) of The QA Handbook. Projects enter The Matrix at Last Call Working Draft (if not sooner). A de-facto process convention is emerging, that there should be significant conformance test materials before finishing Candidate Recommendation phase. This timing coordinates with the explicit process requirement of two interoperable implementations.

Designing and Building Test Materials

There are several scenarios for how the Working Group "builds" its conformance test materials:

All these topics are addressed in more details in the Test FAQ.

Intra-group build

Before starting the development, the Test Task Force participants would benefit from a familiarity with the material in the Test FAQ and optionally in the articles of the QA Wiki regarding test suites. There is useful information for both high-level planning — e.g., does the Working Group want breadth-first Basic Effectivity or a fully detailed suite? — as well as specific details for building the individual test cases.

Another aspect of building test materials is an acceptance procedure for the individual bits, as they are built. This is addressed in the Test FAQ. with examples of review guidelines..

Imported test materials

Several high-quality test suites have been developed outside of the relevant W3C Working Group, and then transferred to the group, or have been the result of assembling pieces from various contributions. Groups which are considering such a transfer should refer to test materials acquisition module of The QA Handbook. Clearly, the quality of the candidate test materials should be carefully assessed, but also, the licensing issues (as discussed in the QA Handbook needs careful planning.

Publication of Test Materials

Typically, a Working Group Test Task Force will want to publish releases of test materials, particularly as the specification advances through its final maturity levels (e.g., Proposed Recommendation) towards Recommendation.

One hurdle on the way to publication is legal — deciding and agreeing on suitable publication licenses. Advice on navigating this potential quagmire is presented in the licensing module of The QA Handbook.

As soon as the test materials become public, then there is definite need for a procedure to process challenges to correctness, make determinations, and appeal decisions.

Publication of test materials often comprises an implicit (or explicit) invitation for contributions. The considerations described in "Imported test materials" are equally applicable here.

Recommendation stage and Beyond

When the specification reaches Recommendation, there is typically a concurrent publication of the test materials. This might be considered a "final" publication, or ongoing development may still be planned. In any case, a maintenance procedure must be in place for the test materials. Firstly, there are tie-ins between approved specification errata and applicability of particular tests. Secondly, there is the above-discussed need for both challenge/review/appeal processes. Finally, even if the Working Group ceases active development of test materials, it may want to continue to consider submissions, and review and integrate them.

Life after Working Group

It is possible that the Working Group and its Test Task Force may disband after its charter expires. The various aspects of this situation are introduced in The QA Handbook.

5. Acknowledgments

The following QA Working Group and Interest Group participants have contributed to this document:

6. Change history

  • revised Introduction to reflect 3 views
  • added document overview section (view 1)
  • renamed sections for views 2 and 3
  • updated entire document to read better and to reflect changes to Framework documents
  • removed TestGL references, since its development has stopped
  • referred to Wiki pages when relevant
  • updated with ref to Variability in Spec
  • Initial version created by separation from QAH.
Valid XHTML 1.0!

Last modified $Date: 2005/09/12 17:45:53 $ by $Author dom $

Copyright © 2000-2004 W3C® (MIT, ERCIM, Keio), All Rights Reserved. W3C liability, trademark, document use and software licensing rules apply. Your interactions with this site are in accordance with our public and Member privacy statements.