W3C

QA Framework: Operational Examples & Techniques

W3C Working Draft 5 April 2002

This version:
http://www.w3.org/QA/WG/2002/framework-20020405/qaframe-ops-extech
Latest version:
http://www.w3.org/QA/WG/qaframe-ops-extech
Previous version:
http://www.w3.org/QA/WG/2002/framework-20020311/qaframe-ops-extech
Editors:
Dimitris Dimitriadis (dimitris@ontologicon.com)
Kirill Gavrylyuk (kirillg@microsoft.com)
Lofton Henderson (lofton@rockynet.com)
Contributors:
See Acknowledgments.

Abstract

This document is part of the of the Quality Assurance (QA) Activity. It contains examples and techniques of QA related activities within the W3C as well as guidance as to how this work should be carried out. The document complements "QA Framework: Operational Guidelines" [QAF-OPS], by specifying or illustrating how to meet the operational and process-related checkpoints of that document.

Status of this document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. The latest status of this document series is maintained at the W3C.

This version is a WG-only draft.

Notice! This document's structure (in Chapter 4) corresponds to the 2002-03-11 WG-only version of the operational guidelines, not the present (same-date) version. I.e., it lags one version behind. The next versions of the two documents will be synchronized.

This document is a W3C Working Draft (WD), made available by the W3C Quality Assurance (QA) Activity for discussion by WG members, as well as other W3C members and other interested parties. For more information about the QA Activity, please see the QA Activity statement.

This version is the first preliminary draft -- basically a completed outline and structure, with some of the intended content sketched in. The previous draft was a course outline of intent. It is expected that updated WD versions of this document will be produced regularly, along with other members of the Framework documents family. Future progression of this document beyond Working Draft is possible, but has not yet been determined.

Please send comments to www-qa@w3.org, the publicly archived list of the QA Interest Group [QAIG]. Please note that any mail sent to this list will be publicly archived and available, do not send information you wouldn't want to see distributed, such as private data.

Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress".

A list of current W3C Recommendations and other technical documents can be found at http://www.w3.org/TR.

Table of contents

1. Introduction
2. Anatomy of this document
3. Conformance test suites
    3.1 DOM
    3.2 SVG
    3.3 XSLT
    3.4 XML
    3.5 Others? [tbd]
4. Operational guidelines in action
         G 1. Integrate Quality Assurance into Working Group activities.
         G 2. Define resources for Working Group QA activities.
         G 3. Synchronize QA activities with the milestones for WG deliverables.
         G 4. Define the QA process.
         G 5. QA Process: Plan test materials development.
         G 6. QA Process: Plan test materials publication.
         G 7. Plan the transfer of test materials to W3C if needed.
         G 8. Plan for test materials maintenance.
5. General remarks
6. Conformance
7. Acknowledgments
8. References


1. Introduction

This document contains examples and techniques of QA-related activities within the W3C as well as guidance as to how the quality processes of the Working Groups (WGs) should be carried out. It complements "QA Framework: Operational Guidelines" [QAF-OPS], by specifying or illustrating how to meet the operational and process-related checkpoints of that document.

The operational guidelines document should be considered a prerequisite to this one. In addition to containing all of the checkpoints to which the content of this document pertain, it explains principles underlying the conduct of WG quality processes.

2. Anatomy of this document

The structure of this document parallels the structure of the operational guidelines [QAF-OPS] document, particularly its Chapter 3, "Guidelines". It should be realized that some of the examples are W3C QA-related activities that were started before the QA activity, for example DOM and SVG. Common features in these example frameworks will be pointed out, and they will be compared against the statements of the checkpoints of the operational guidelines. Instead of pointing to each of the example QA-related frameworks, a short description of each such will be given together with links. The subsequent parts of the document will elaborate on the checkpoints and how they have been realized in various efforts.

3. Conformance test suites

Herein is a short description of each instance of a subset of W3C QA-related activities, together with some external to the W3C.

Each subsection covers the following topics:

3.1 DOM

3.1.1 General Outline

The DOM TS was jointly launched by the W3C and NIST and is produced in public. It uses coordination from the W3C DOM WG, and is produced using communication through a W3C-hosted mailing list as well as W3C-supplied storage for files, documents and so forth. The TS was to be developed in public, and the WG would endorse two bindings, Java and ECMA, the same as the official DOM bindings. Other bindings were welcomed but would not form part of the official TS.

3.1.2 Internal/External/hybrid and staffing requirements

The DOM TS was launched by the W3C DOM WG and NIST in an effort to invite the public to participate in the roduction of quality testing material for implementations of the DOM specifications. Staffing requirements were limited, so an expert was invited to the DOM WG in order to produce and edit the DOM TS Process Document. The DOM TS was in its initial phase very similar to a previously launched DOM Test Suite launched by NIST, which was used as a primer for the forthcoming work.[@@include links to relevant emails]

3.1.3 Charter considerations

Since the DOM charter did not include sections on producing test materials, this was included in teh DOM TS process, which outlined the DOM TS deliverables in detail.

3.1.4 Point in time for production of TS; is it during or after specification phase?

The DOM TS was produced after specification for the first two levels (DOM Level 1 & 2) and is produced in parallel with specifying DOM Level 3. DOM Level 1 was released in late 1998, so it was not possible to synchronise the specification of the firsl level of the ODM with the Test Suite.

Starting with the current level of the DOM, the TS is produced in parallel with the specification. Tests are being produced to allow for implementors to check their implenentations of DOM level 3 while contributing to relevant parts of the specification. Also, members of the DOM WG have commited to produce tests for those modules of the ODM that are particularly relevant to their implementations.

3.1.5 Pointer to process documents, if applicable

There is a separate DOM TS Process Document, and also an area where you can find more information and pointers to primers, FAQ's and tutorials, maintained by the DOM WG representative to the DOM TS.

3.1.6 Test material storage, maintenance, tracking

The DOM TS uses W3C machinery to store test files, using CVS for easy tracking, and an issue tracking system at Sourceforge, in order to simplify issues and maintenance.

3.2 SVG

[@@ to be done]

3.3 XSLT

[@@ to be done]

3.4 XML

[@@ to be done]

3.5 Others? [tbd]

[@@ any others beyond these four? @@]

4. Operational guidelines in action

This section looks at the guidelines and checkpoints in "Operational Guidelines", and how they have been used in the various existing frameworks. Where appropriate, pointers to relevant documents and materials will be provided. [@@ will be provided in a later version].

The structure of this section exactly parallels the structure of the its Chapter 3, "Guidelines", of the operational guidelines [QAF-OPS] document.

Each guideline links back to the corresponding guideline of "QA Framework: Operational Guidelines" [QAF-OPS], and each checkpoint similarly links back to corresponding checkpoint. [@@ these links are not implemented in this version.]

Under each checkpoint, there are multiple examples of how other QA or TS projects relate to the checkpoint, including at least one example where the checkpoint was satisfied. Finally, there will be a summary enumeration of ways in which the each checkpoint may be satisfactorily satisfied [@@this synthesis will be done in a future version].

Guideline 1. Integrate Quality Assurance into Working Group activities.

DOM discussion

In the DOM TS, the QA activities were integrated into Working Group activities by inviting a field expert to do the coordination between WG and TS Group. Coordination was necessary as the DOM WG QA activity was launched in coordination with an external party, and, furthermore, was to be produced externally.

SVG discussion

[@@ to be done]

Checkpoints

Checkpoint 1.1. In the Charter, specify the level of commitment to QA. Plan to have at least a commitment to "QA level three". [Priority 1]

Checkpoint 1.2. In the Charter, enumerate QA deliverables and expected milestones. [Priority 1]

Checkpoint 1.3. In the Charter, associate QA criteria with WG milestones. [Priority 2]

Guideline 2. Define resources for Working Group QA activities.

DOM discussion

The DOM WG appointed an invited expert to monitor and coordinate the work of the DOM TS. Furthermore, the members of the DOM TS have looked into producing tests fot that part of the specification which is of particular interest to their implementations.

SVG discussion

[@@ to be done]

Checkpoints

Checkpoint 2.1. In the Charter, address where and how conformance test materials will be produced. [Priority 1]

Checkpoint 2.2. In the Charter, address QA staffing commitments. [Priority 1]

Checkpoint 2.3. In the Call For Participation, request allocation of QA resources to the Working Group. [Priority 1]

Guideline 3. Synchronize QA activities with the milestones for WG deliverables.

DOM discussion

The milestones were given in the DOM TS Process document and were monitored by the DOM WG in order to assure the satisfactory continuation of the DOM TS group work. In those cases where the TS Group needed more time to achieve its goals, the DOM WG representative reported to the WG in order to appeal for an extension.

SVG discussion

[@@ to be done]

Checkpoints

Checkpoint 3.1. Synchronize the publication of QA deliverables and the specification's drafts. [Priority 2]

Checkpoint 3.2. Support specification versioning/errata in the QA deliverables. [Priority 3]

Guideline 4. Define the QA process.

DOM discussion

DOM TS Process document

SVG discussion

[@@ to be done]

Checkpoints

Checkpoint 4.1. Appoint a QA moderator and a QA task force. [Priority 2]

Checkpoint 4.2. Produce the QA Process document. [Priority 1]

Checkpoint 4.3. In the QA Process document, define means for QA-related communication. [Priority 2]

Checkpoint 4.4. In the QA Process document, specify the framework for test materials development. [Priority 2]

Checkpoint 4.5. Specify a policy for branding materials if applicable. [Priority 3]

Guideline 5. QA Process: Plan test materials development.

DOM discussion

As the DOM TS drew on experiences made by an external party (NIST), the work initialy drew heavily on those experiences made there. Where the subsequent work indicated digression, the future version of the DOM TS was branched off and formed the official W3C DOM TS. The test material planning was made by the DOM WG representative to the DOM TS in coordination with NIST and the public, after the DOM TS had been officially launched.

Several design issues were discussed by the DOM TS Ggroup itself, then brought to the DOM WG for acceptance. Such issues include the automatic generationof the schema language for the DOM TS, using the same test descritpion file to generate the official bindings (and the unofficial ones using a different transform) as well as using particular build tools to build working and official releases of the DOM TS.

SVG discussion

[@@ to be done]

Checkpoints

Checkpoint 5.1. In the test framework, ensure test materials are documented and reusable by accredited third parties for testing purposes. [Priority 2]

Checkpoint 5.2. In the QA Process document, define a contribution process. [Priority 2]

Checkpoint 5.3. In the QA Process document, define the licenses applicable to submitted test materials. [Priority 2]

Checkpoint 5.4. In QA Process document, define review procedures for submitted test materials. [Priority 2]

Guideline 6. QA Process: Plan test materials publication.

DOM discussion

As above, those issues were discusses internally in the DOM TS group, then brought to the DOM WG for acceptance.

SVG discussion

[@@ to be done]

Checkpoints

Checkpoint 6.1. Ensure a secure and reliable repository location for future test materials. [Priority 2]

Checkpoint 6.2. In the QA Process document, define the licenses applicable to published test materials. [Priority 2]

Checkpoint 6.3. In the QA Process document, describe how the test materials will be published and point to the corresponding webpage. [Priority 2]

Checkpoint 6.4. Provide a disclaimer regarding the use of the test materials for compliance verification. [Priority 2]

Checkpoint 6.5. In the QA Process document, describe how vendors can publish test results for their products, if applicable. [Priority 3]

Guideline 7. Plan the transfer of test materials to W3C if needed.

DOM discussion

This issue was addressed early on in the specification phase of the DOM TS work, as a choice had to be made between integrating an existing TS or just using the existing one as a primer for its succesor. The latter was preferred.

SVG discussion

[@@ to be done]

Checkpoints

Checkpoint 7.1. If transfer of the test materials to W3C is planned, perform an assessment of their quality. [Priority 2]

Checkpoint 7.2. If transfer of the test materials to W3C is planned, identify sufficient staff resources to meet the requirements. [Priority 2]

Checkpoint 7.3. If transfer of the test materials to W3C is planned, resolve IPR questions and reach agreement with the external party that produced test materials. [Priority 1]

Checkpoint 7.4. If transfer of the test materials to W3C is planned, update the QA commitments in the Working Group charter if necessary. [Priority 2]

Guideline 8. Plan for test materials maintenance.

DOM discussion

No clear plan for test material maintenance has been made, other than indicating in the process document that these issues will be adressed by the DOM TS group.

SVG discussion

[@@ to be done]

Checkpoints

Checkpoint 8.1. Maintain contribution and review procedures throughout test materials' and standard's entire life cycles. [Priority 3]

Checkpoint 8.2. In the Working Group's QA process document, specify a procedure for updates of the test materials to track new errata/specification versions. [Priority 2]

5. General remarks

Experiences drawn from previous efforts, obvious pitfalls, good examples, open issues, and so forth.

DOM discussion

There are a series of issues with the DOM TS:

SVG discussion

[@@ to be done]

6. Conformance

[@@tbd -- probably irrelevant for extech parts @@]


7. Acknowledgments

The following QA Working Group and Interest Group participants have contributed significantly to the content of this document:

8. References

ISSUES-LIST
QA Activity Issues List, maintained by the QA Working Group, available at http://www.w3.org/QA/WG/#issues.
MATRIX
W3C-wide conformance activity survey covering all the WGs, "The Matrix", available at http://www.w3.org/QA/TheMatrix.
PROCESS
W3C Process Document, 19 July 2001, available at http://www.w3.org/Consortium/Process-20010719/.
QA-GLOSSARY
Comprehensive glossary of QA terminology. (Under construction.)
QAF-OPS
"QA Framework: Operational Guidelines", Working Draft companion version to this document, available at [...].
QAF-SPEC
"QA Framework: Specification Guidelines", Working Draft companion version to this document, available at [...].
QAF-TEST
"QA Framework: Test Materials Guidelines", not yet published.
QAIG
Quality Assurance Interest Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/IG/.
QAWG
Quality Assurance Working Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/WG/.
RFC2119
Key words for use in RFCs to Indicate Requirement Levels, March 1997, available at http://www.ietf.org/rfc/rfc2119.txt.
SPEC-EXTECH
"QA Framework: Specification Examples and Techniques", not yet published.
STYLE-MAN
"W3C Manual of Style", summarizing the style and publication rules for W3C technical reports, available at http://www.w3.org/2001/06/manual/.
TAXONOMY
QA Activity test taxonomy, a classification scheme for conformance test materials, available at http://www.w3.org/QA/Taxonomy.
TEST-EXTECH
"QA Framework: Test Materials Examples and Techniques", not yet published.
W3C-TR
Location of all published W3C technical reports, see http://www.w3.org/TR/.
WCAG10
Web Content Accessibility Guidelines, version 1.0, W3C Recommendation, 5 May 1999, available at http://www.w3.org/TR/WCAG10/.