W3C

QA Framework: Operational Examples & Techniques

W3C Note 08 November 2002

This version:
http://www.w3.org/QA/WG/2002/11/qaframe-ops-extech-20021108
Latest version:
http://www.w3.org/QA/WG/qaframe-ops-extech
Previous version:
http://www.w3.org/TR/2002/WD-qaframe-ops-extech-20020515/
Editors:
Lofton Henderson (lofton@rockynet.com)
Lynne Rosenthal (lsr@nist.gov)
Dimitris Dimitriadis (dimitris@ontologicon.com)
Kirill Gavrylyuk (kirillg@microsoft.com)
Contributors:
See Acknowledgments.

Copyright ©2002 W3C ¨ ( MIT , INRIA , Keio), All Rights Reserved. W3C liability, trademark, document use and software licensing rules apply.

Abstract

This document is part of the of the Quality Assurance (QA) Activity. It presents examples and describes the techniques of operational aspects of quality practices within the W3C's Working Groups. It complements QA Framework: Operational Guidelines [QAF-OPS], by specifying or illustrating how Working Groups might meet the operational and process-related checkpoints of that document.

Status of this document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. The latest version of this document series is maintained at the W3C.

This document is a W3C Note, made available by the W3C Quality Assurance (QA) Activity for discussion by W3C members and other interested parties. For more information about the QA Activity, please see the QA Activity statement.

This draft is minimal revised from the Previous Version (20020515 publication), to track changes in checkpoint wording of the 20021108 publication of QA Framework: Operational Guidelines [QAF-OPS]. The document status has changed from Working Draft to Note, pursuant to resolution of Issue #68. The principal changes from the Previous Version are listed in the "Change history" section.

In this version, three case studies are presented in some detail -- how the QA projects of three existing Working Groups relate to the checkpoints of the operational guidelines [QAF-OPS]. This version lacks a systematic enumeration of techniques -- ways in which Working Groups might satisfy the operational and process-related checkpoints. This is a significant planned addition in a future version.

It is anticipated that this document will be updated more frequently than its companion (20021108) Operational Guidelines document. The per-checkpoint linkage between that document and future versions of this one will be maintained as the examples and techniques are improved.

Please send comments to www-qa@w3.org, the publicly archived list of the QA Interest Group [QAIG]. Please note that any mail sent to this list will be publicly archived and available, do not send information you wouldn't want to see distributed, such as private data.

Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than work in progress.

A list of current W3C Recommendations and other technical documents can be found at http://www.w3.org/TR.

Table of contents

1. Introduction
    1.1 Scope
    1.2 Overview of contents
    1.3 Anatomy of this document
2. Conformance test suite case studies
    2.1 DOM
    2.2 SVG
    2.3 XML
3. Operational guidelines in action
         G 1. Integrate Quality Assurance into Working Group activities.
         G 2. Define resources for Working Group QA activities.
         G 3. Synchronize QA activities with the milestones for WG deliverables.
         G 4. Define the QA process.
         G 5. Plan test materials development.
         G 6. Plan test materials publication.
         G 7. Plan the transfer of test materials to W3C if needed.
         G 8. Plan for test materials maintenance.
4. Conformance
5. Acknowledgments
6. References
7. Change history

1. Introduction

1.1 Scope

This document contains examples and techniques of QA-related activities within the W3C as well as guidance as to how the quality processes of the Working Groups (WGs) should be carried out. It is an informative complement to QA Framework: Operational Guidelines [QAF-OPS], specifying or illustrating how to meet the operational and process-related checkpoints of that document.

The operational guidelines document should be considered a prerequisite to this one. In addition to containing all of the checkpoints to which the contents of this document pertain, explains principles underlying the conduct of WG quality processes.

The operational guidelines document also contains all of the normative requirements for the operational aspects of quality practices within the W3C's Working Groups. This informative document illustrates how Working Groups might satify the operational and process-related checkpoints of that document.

1.2 Overview of contents

Three test suite (TS) projects related to W3C Recommendations are used as examples. Some of them were written in-house (within the responsible WG), some externally (outside of W3C), and some are hybrid (external plus in-house effort). All of them have succeeded in producing significant and valuable TS deliverables.

In all cases, the associated WGs have existed and were well advanced on their relevant Recommendations before the operational guidelines [QAF-OPS] were first published. Because these TS projects preceded the guidelines, it makes no sense to score them against checkpoints -- it is unreasonable to expect them to meet all of the checkpoints. Instead, the various aspects of the different projects are used to illustrate the meaning and intent of the checkpoints. In a sense, the checkpoints are the union of the best practice and techniques from all of projects (these and others).

This points to a timing-based reality of conformance to the operational guidelines. The conformance techniques (to be more fully developed in the future versions), will be different for new Working Groups versus existing Working Groups.

1.3 Anatomy of this document

The structure of this document parallels the structure of the operational guidelines [QAF-OPS] document, particularly its Chapter 3, "Guidelines". It should be realized that all of the given examples of W3C Working Groups' QA-related activities were started before the applicable parts of the W3C QA Activity. A useful milestone for the latter is the first publication of the Operational Guidelines, February 2002.

Relevant common features in these example frameworks will be pointed out, and they will be compared against the statements of the checkpoints of the operational guidelines. For each of the given example QA-related frameworks, a short description of each will be presented. The subsequent parts of the document will elaborate on the checkpoints -- including links to relevant process-related documents and resources -- and how they have been realized in various efforts.

This document is heavily weighted towards case studies. It has relatively little specification of techniques for meeting checkpoints. In the few cases where examples have been moved here from the operational guidelines document, the case studies and the examples are distinguished by headings, "Case studies:" and "Examples & techniques", respectively.

2. Conformance test suite case studies

Herein is a summary description of each of a small set of case studies of W3C QA-related activities, as well as some external to the W3C.

Each subsection covers the following topics:

2.1 DOM

2.1.1 General overview

The DOM TS was jointly launched by the W3C and NIST and is produced in public. It uses coordination from the W3C DOM WG, and is produced using communication through a W3C-hosted mailing list as well as W3C-supplied storage for files, documents and so forth. The TS was to be developed in public, and the WG would endorse two bindings, Java and ECMAscript, the same as the official DOM bindings. Other bindings were welcomed but would not form part of the official TS.

2.1.2 Internal/external/hybrid and staffing requirements

The DOM TS was launched by the W3C DOM WG and NIST, and featured a public call for participation in the production of quality testing material for implementations of the DOM specifications. Staffing requirements were limited, so an expert was invited to the DOM WG in order to produce and edit the DOM TS Process Document. The DOM TS was in its initial phase very similar to a previous DOM Test Suite launched by NIST, which was used as a primer for the forthcoming work. However, the subsequentially released and forthcoming versions differ greatly given the group's work.

2.1.3 Charter considerations

Since the DOM charter did not include sections on producing test materials, this was included in the DOM TS Process Document [DOM-TS-PROC], which outlined the DOM TS deliverables in detail.

2.1.4 When TS is produced -- during or after specification phase?

The DOM TS was produced after specification for the first two levels (DOM Level 1 & 2) and is produced in parallel with specifying DOM Level 3. DOM Level 1 was released in late 1998, so it was not possible to synchronize the specification of the first level of the DOM with the Test Suite.

Starting with the current level of the DOM, the TS is produced in parallel with the specification. Tests are being produced to allow for implementors to check their implementations of DOM level 3 while contributing to relevant parts of the specification. Tests for the various modules of the DOM Level 3 specification are being written in parallel with the specification process.

2.1.5 Pointer to process documents

There is a separate DOM TS Process Document [DOM-TS-PROC], and also a Test Web page where you can find more information and pointers to primers, FAQ's and tutorials, maintained by the DOM WG representative to the DOM TS.

2.1.6 Test material storage, maintenance, tracking

The DOM TS uses W3C machinery to store test files, using CVS for easy tracking, and an issue tracking system at Sourceforge, in order to simplify issues and maintenance.

2.2 SVG

2.2.1 General overview

As used in this case study, "SVG" refers collectively to two consecutively chartered Working Groups. SVG1 refers to the originally chartered group (first meeting, Sept 1998) that produced SVG 1.0 (Recommendation, Sept 2001). SVG2 is the successor to SVG1, (re-)chartered in 2001, and charged with developing SVG 1.1 (modularization and profiles), and SVG 2.0. SVG 1.1 is presently at the stage of Candidate Recommendation (CR).

SVG 1.0 was released with a comprehensive Basic Effectivity (BE) test suite, that was developed entirely within the working group. The test suite (TS) comprises 127 test cases that span the functionality of SVG, exercising each major functional feature of SVG 1.0 at least lightly, but not probing any features in detail. SVG2 is adapting and subsetting the tests of the SVG1 TS for smaller and less capable mobile devices that are the target constituency of the Tiny and Basic profiles.

2.2.2 Internal/external/hybrid and staffing requirements

The SVG1 test suite was developed entirely by the working group. A pilot study sponsored by one of the members became the basis for design. A "TS Editor" (moderator) managed the project from inception to completion. All WG members are expected to participate in the writing and review of test cases. The SVG2 approach is identical -- a WG member is the TS moderator, and all WG members contribute.

2.2.3 Charter considerations

Both SVG1 & SVG2 charters specify a comprehensive test suite as a deliverable, and they coordinate the dates of first TS delivery (first publication) and the specification CR milestone. There is no further detailed breakdown of milestones or deliverables. This was worked out (for SVG1) as the specification development progressed.

2.2.4 When TS is produced -- during or after specification phase?

The test suites were (are) produced during the specification-writing phase. The target is substantial completion at the Candidate Recommendation (CR) milestone, so that the TS could be used to facilitate the interoperable-implementations W3C Process requirements of CR ([PROCESS], section 5.2.3)

2.2.5 Pointer to process documents

SVG has no QA process document per se, but much of the content that these operational checkpoints anticipate and require may be found in the extensive reference document, SVG Conformance Test Suite -- Test Builder's Manual [SVG-MAN]. Unless otherwise specified in that document, the normal WG processes of the SVG Working Group are used to conduct the TS work.

2.2.6 Test material storage, maintenance, tracking

The SVG WG keeps the SVG TS on a CVS server of one of the WG members. Revision tracking is therefore handled by CVS. Maintenance procedures and issue tracking for SVG TS are not separately defined, but rather SVG TS uses the normal SVG WG processes for these.

2.3 XML

2.3.1 General overview

Note. Following its transfer from outside W3C to the XML WG, when this case study was written, the XML TS was in preparation for first publication by the WG. Therefore a number of the referenced documents were not yet publicly visible.

The OASIS/ NIST XML Test Suite (TS) forms the basis for the W3C XML TS. As a result of a multi-way collaboration between

it was transferred to W3C where it is being augmented with new functionality, corrected to align with the XML 1.0 Second Edition Recommendation and its errata, and maintained.

Prior to accepting the transfer of the OASIS/NIST Test Suite, the XML Core WG assessed the level of effort necessary for TS development and maintenance and ensured that staff resources were interested and available to work on the TS. Additionally, the WG enlisted the assistance of NIST to lead the effort. The TS is being developed as a deliverable of the WG, following the WG's Process. There is a W3C hosted XML TS mailing list, Issues list, and W3C supplied storage (CVS) for the test-suite related files.

The XML Conformance Test Suite, v1.0, Second Edition contains over 2000 test files and an associated test report. The test report contains background information on conformance testing for XML as well as test descriptions for each of the test files included in the TS.

2.3.2 Internal/external/hybrid and staffing requirements

A commitment from NIST to take editor-lead responsibility for the TS was obtained prior to the commencement of the work. Consequently, NIST joined the WG. Members of the XML Core WG contribute to the development and quality assessment of the TS. When the XML TS is released by the WG, the general public will be invited to review the tests as well as participate in contributing tests.

2.3.3 Charter considerations

The W3C XML Core WG charter does not include sections on producing test materials. A new charter provides for the possibility of working on the OASIS/NIST XML Test Suite. The WG does not have a separate QA Process, but follows the WG Process for the development, assessment and maintenance of the XML TS. This is not explicitly documented in a separate QA process document, but the individual decisions are on record in WG mail archives and minutes as part of the normal WG process.

2.3.4 When TS is produced -- during or after specification phase?

The work on the OASIS/NIST XML Test Suite was conducted after the XML 1.0 Recommendation was released. A revised edition of the OASIS/NIST Test Suite 1.0 (Second Edition) Test Suite was released to complement the W3C XML 1.0 Second Edition Recommendation (October 6, 2000). Within the XML Core WG, the XML TS is being updated to reflect the current Recommendation and all applicable Errata.

2.3.5 Pointer to process document

There is no separate QA Process document. The XML Core WG charter uses its established WG Process to handle test development/assessment activities, communication means and test materials related functions.

2.3.6 Test material storage, maintenance, tracking

The XML WG uses the W3C CVS repository to maintain the XML TS. The XML TS issues are maintained utilizing an internal assessment process that consists of a running log of test issues, XML TS mailing lists, XML Core WG discussions and the XML Test Suite Issues document that keeps track of the running log of issues.

3. Operational guidelines in action

This section looks at the guidelines and checkpoints in the operational guidelines [QAF-OPS ], and how their requirements are reflected in the various existing frameworks. Where appropriate, pointers to relevant documents and materials are provided.

The structure of this section exactly parallels the structure of the its Chapter 3, "Guidelines", of the operational guidelines [QAF-OPS] document -- each checkpoint in this section links back to the corresponding checkpoint of QA Framework: Operational Guidelines [QAF-OPS].

Under each checkpoint, there is discussion of how each of the example QA or TS projects relate to the checkpoint. Finally, there will be a summary enumeration of ways in which the each checkpoint may be satisfactorily satisfied. Note. This synthesis will be done in a future version of this document.

Guideline 1. Integrate Quality Assurance into Working Group activities.

DOM discussion

In the DOM TS, the QA activities were integrated into Working Group activities by inviting a field expert to do the coordination between WG and TS Group. Coordination was necessary as the DOM WG QA activity was launched in coordination with an external party, and, furthermore, was to be produced externally.

SVG discussion

QA was integrated into the SVG WG activities by a Charter commitment to produce test suites, and subsequent involvement of most WG members in the writing, reviewing, and maintenance of the test materials. The charters committed, in the cases of both SVG1 and SVG2, that a basic test suite (i.e., Basic Effectivity) would exist at or before Candidate Recommendation. The details of deliverables, milestones, and publication were generally worked out by the WG as the specifications developed.

XML discussion

The QA activities are integrated into WG activities by using the WG Process. Although the activities are not reflected in the WG Charter, deliverables are identified, expected release dates determined, staff allocated, and there is a clear commitment to developing and maintaining the TS.

Checkpoints

Checkpoint 1.1. Commit to at least "QA level three". [Priority 1]

Guidelines document description of checkpoint.

Checkpoint 1.2. Commit to at least "QA level five". [Priority 2]

Guidelines document description of checkpoint.

Checkpoint 1.3. Commit to at least "QA level seven". [Priority 3]

Guidelines document description of checkpoint.

Checkpoint 1.4. Enumerate QA deliverables and expected milestones. [Priority 1]

Guidelines document description of checkpoint.

Checkpoint 1.5. Associate QA criteria with WG milestones. [Priority 2]

Guidelines document description of checkpoint.

Guideline 2. Define resources for Working Group QA activities.

DOM discussion

The DOM WG appointed an invited expert to monitor and coordinate the work of the DOM TS. Furthermore, the members of the DOM WG have looked into producing tests fot that part of the specification which is of particular interest to their implementations.

SVG discussion

The SVG1 and SVG2 charters explicitly charge the WG with producing test materials, but do not further address specific staffing. In fact, the expectation has evolved that all WG members should contribute to the building of the TS. In both cases, a "TS Editor" has been assigned to lead and coordinate the TS production.

XML discussion

The WG appointed NIST to lead the TS effort and coordinate the work of the XML TS. Furthermore, members of the WG have reviewed and assessed the current set of tests, corrected tests, and contributed tests.

Checkpoints

Checkpoint 2.1. Address where and how conformance test materials will be produced. [Priority 1]

Case studies:

Examples & techniques:

There are several possibilities for producing conformance test materials:

There are good reasons why external parties should have a strong participation during the building of the materials. But even if the test suite is being developed outside of W3C, the Working Group needs to ensure it is completed according to the QA level to which the Working Group is committed in its charter.

Guidelines document description of checkpoint.

Checkpoint 2.2. Address QA staffing commitments. [Priority 1]

Guidelines document description of checkpoint.

Checkpoint 2.3. Request allocation of QA resources to the Working Group. [Priority 1]

Guidelines document description of checkpoint.

Guideline 3. Synchronize QA activities with the milestones for WG deliverables.

DOM discussion

The milestones were given in the DOM TS Process document and were monitored by the DOM WG in order to assure the satisfactory continuation of the DOM TS group work. In those cases where the TS Group needed more time to achieve its goals, the DOM WG representative reported to the WG in order to appeal for an extension.

Applicable in so far as the TS releases are (for levels currently being specified) synchronized with the relevant modules of the DOM specifications.

SVG discussion

The SVG1 and SVG2 charters establish the basic synchronization, which is that the first publications of basic test suites (TS) should be at or before the beginning of CR stage. Beyond that, there are no explicit (i.e., written down) synchronizations. This happens in fact as part of the WG process and the TS building. TS publications are synchronized with selected late-stage document versions (CR, PR, Rec). There is no explicit scheme for labelling/linking test cases according to errata, although this would be reconstructable from the CVS repository for the TS.

XML discussion

The TS work is being conducted in close coordination with the WG's other deliverables. The TS is being updated and augmented to reflect the current work of the WG, including resolved issues related to the Recommendation and published Errata. Tests for new functional areas are also being explored as the deliverables for those Recommendations progress (i.e., Xinclude and Namespace).

Checkpoints

Checkpoint 3.1. Synchronize the publication of QA deliverables and the specification's drafts. [Priority 2]

Guidelines document description of checkpoint.

Checkpoint 3.2. Support specification versioning/errata in the QA deliverables. [Priority 3]

Guidelines document description of checkpoint.

Guideline 4. Define the QA process.

DOM discussion

DOM TS Process document

SVG discussion

SVG supports the process requirements of a QA moderator (manager of TS development) and a task force (it is the whole WG, by explicit choice). TS-related communication is handled internally on the whole-WG list (member-only), and externally via a public TS comment list. SVG does not have a Process document, per se, that spells out these details. It does have a TS manual that defines the TS framework in some detail.

XML discussion

The WG Process is used.

Checkpoints

Checkpoint 4.1. Appoint a QA moderator. [Priority 2]

Guidelines document description of checkpoint.

Checkpoint 4.2. Appoint a QA task force. [Priority 2]

Guidelines document description of checkpoint.

Checkpoint 4.3. Produce the QA Process Document. [Priority 1]

Guidelines document description of checkpoint.

Checkpoint 4.4. Specify means for QA-related communication. [Priority 2]

Guidelines document description of checkpoint.

Checkpoint 4.5. Define the QA framework for test materials development. [Priority 2]

Guidelines document description of checkpoint.

Checkpoint 4.6. Specify a policy for branding materials. [Priority 3]

Guidelines document description of checkpoint.

Guideline 5. Plan test materials development.

DOM discussion

As the DOM TS drew on experiences made by an external party (NIST), the work initially drew heavily on those experiences. Where the subsequent work indicated digression, the future version of the DOM TS was branched off and formed the official W3C DOM TS. The test material planning was made by the DOM WG representative to the DOM TS in coordination with NIST and the public, after the DOM TS had been officially launched.

Several design issues were discussed by the DOM TS Group itself, then brought to the DOM WG for acceptance. Such issues include the automatic generation of the schema language for the DOM TS, using the same test description file to generate the official bindings (and the unofficial ones using a different transform) as well as using particular build tools to build working and official releases of the DOM TS.

Most discussion was done on the DOM Group mailing list and coordinated with the DOM WG. More information can be found in the DOM TS page, which serves as a complete reference for DOM TS related information.

SVG discussion

The planning for SVG's test materials development started with a study and prototype design, sponsored by one of the SVG WG members. This work was brought into the WG as the starting point of the TS development, and was further developed, refined, and applied by the moderator ("TS Editor"). Both user documentation (for TS releases) and contributors documentation were developed. The latter described the TS framework in some detail, and gave both technical criteria for the contributed test cases plus review/acceptance criteria. The actually processes and procedures for contribution and review were not explicitly defined, as all materials came from WG members (and therefore followed WG process).

XML discussion

The XML TS drew on the experiences and expertise of NIST, the original developer of the test suite framework. The original work was revised to included test requirements with a mapping to the test cases and to the specification. The planning of improvements and augmentation to the original work was discussed and decided by the WG. Techniques and methods were borrowed from other test suite development efforts, such as DOM and XSL-FO.

Checkpoints

Checkpoint 5.1. Ensure test materials are documented and usable for their intended purposes. [Priority 2]

Guidelines document description of checkpoint.

Checkpoint 5.2. Define a contribution process. [Priority 2]

Guidelines document description of checkpoint.

Checkpoint 5.3. Define the licenses applicable to submitted test materials. [Priority 2]

Guidelines document description of checkpoint.

Checkpoint 5.4. Define review procedures for submitted test materials. [Priority 2]

Case studies:

Examples & techniques:

The following example meets the review requirements:

  1. Each contribution should be reviewed for its eligibility by at least one vendor other than the submitter. Any vendor involved in the creation of the Working Group test collection other than the submitter may review contributed tests.
  2. Eligibility of tests should be judged by:
  3. After review, results should be sent back to the submitter for feedback.
  4. In case of disagreement, a request for clarification should go to Working Group. Tests in question should be left out of the test suite until the W3C Working Group clarifies the issue.

Only those tests that are agreed to be correct by reviewers and the submitter, or that are approved by W3C Working Group clarification, should be published.

Guidelines document description of checkpoint.

Guideline 6. Plan test materials publication.

DOM discussion

Plans for test material publication are discussed internally in the DOM TS group, and then decided by the DOM TS Group in coordination with the DOM WG.

SVG discussion

The SVG TS project has explicitly discussed and planned the key aspects of TS publication. CVS repository is used for the ongoing development of the TS, and at agreed milestones, a snapshot is taken and published under the W3C Document License. The TS home page links all of the materials and documentation, and also points to an implementation matrix that publishes the TS results of a half-dozen or so implementations.

XML discussion

Publication of the TS is being discussed by the WG. The TS will reside on the W3C web space and will be freely and publicly available. In general, the publication details of the XML TS are relatively invisible outside of the WG, because there has not been any publication yet.

Checkpoints

Checkpoint 6.1. Ensure a suitable repository location for test materials. [Priority 2]

Guidelines document description of checkpoint.

Checkpoint 6.2. Define the licenses applicable to published test materials. [Priority 2]

Guidelines document description of checkpoint.

Checkpoint 6.3. Describe how and where the test materials will be published. [Priority 2]

Guidelines document description of checkpoint.

Checkpoint 6.4. Provide a conformance verification disclaimer with the test materials. [Priority 2]

Guidelines document description of checkpoint.

Checkpoint 6.5. Address the publication of test results for products. [Priority 3]

Guidelines document description of checkpoint.

Guideline 7. Plan the transfer of test materials to W3C if needed.

DOM discussion

This issue was addressed early on in the specification phase of the DOM TS work, as a choice had to be made between integrating an existing TS or just using the existing one as a primer for its succesor. The latter was preferred.

This is addressed in the DOM TS Process document.

SVG discussion

This guideline is not applicable to the SVG TS -- all TS development so far has occurred within the SVG WG.

XML discussion

The issue was addressed prior to the transfer of the TS to XML WG, and prior to the start of TS development and maintenance there. Since that phase is finished, it is no longer applicable. Included in the discussions were the quality of the OASIS/NIST XML Test Suite, amount of work needed to correct and improve it, as well as IPR questions.

Checkpoints

Checkpoint 7.1. Perform a quality assessment of any test materials that are candidates for transfer. [Priority 2]

Guidelines document description of checkpoint.

Checkpoint 7.2. Identify sufficient staff resources to meet the needs of any transferred test materials. [Priority 2]

Guidelines document description of checkpoint.

Checkpoint 7.3. For any transferred test materials, resolve all IPR issues with the external party that produced the test materials. [Priority 1]

Case studies:

Examples & techniques:

The following procedure is an example that meets all of the transfer requirements.

Legend for the procedure:

EG
the external group or entity
QAWG
the QA Working Group
TM
the test materials
WG
the subject Working Group

Before the transfer, WG with the help of QAWG:

During transfer:

After transfer, initial process setup:

First W3C public release of the new TM:

After the first public release, the TM enter the maintenance phase which is described below.

Guidelines document description of checkpoint.

Guideline 8. Plan for test materials maintenance.

DOM discussion

No clear plan for test material maintenance has been made, other than indicating in the process document that these issues will be adressed by the DOM TS group.

Not applicable

SVG discussion

SVG has no explicit plan for TS maintenance. Because the TS process is basically the WG process, maintenance happens happens as a natural part of the WG's specification development and ongoing TS development.

XML discussion

No clear plan for test material maintenance has been made yet. The need to address this has been acknowledged by the WG, with plans to address it in the near future.

Checkpoints

Checkpoint 8.1. Maintain the contribution and review procedures throughout the entire life cycle of the test materials and the standard itself. [Priority 3]

Guidelines document description of checkpoint.

Checkpoint 8.2. Specify a test materials update procedure to track new specification versions/errata. [Priority 2]

Case studies:

Examples & techniques:

The following procedure is an example that meets the requirements:

Reviewing/accepting the change. Test changes go through the same tests review process as presented earlier. If the test changes are accepted, they are added to the test suite with an indicator of errata/version. Old tests stay in the test suite marked with the last errata/version level they comply with. The status changes to "CLOSED". If tests are rejected, they go back to the provider of the change with that determination, and the status is reverted back to "ACTIVE"

Guidelines document description of checkpoint.

Checkpoint 8.3. Identify a procedure for test validity appeals. [Priority 2]

Case studies:

Examples & techniques:

The following topical outline is an example that meets the requirements, i.e., if these topics were addressed in the process document::

If there is still active work on the test suite even if the Working Group is not re-chartered to handle the work, then it is up to the Director to determine how this work should be done. Some options include:

Guidelines document description of checkpoint.

4. Conformance

There is no concept of conformance to this document. Rather, conformance is measured relative to the checkpoints of the companion document, QA Framework: Operational Guidelines [QAF-OPS]. This document relates real-world examples to the checkpoints' requirements, and also presents techniques by which the checkpoints may be satisfied. In that sense, it defines conformance criteria for the conformance requirements (checkpoints) of the operational guidelines document.

5. Acknowledgments

The following QA Working Group and Interest Group participants have contributed significantly to the content of this document:

6. References

DOM-TS-PROC
DOM TS Process Document , D. Dimitriadis, Ed., 15 January 2002, available at http://www.w3.org/DOM/DOMTS-Process.
PROCESS
World Wide Web Consortium Process Document , I. Jacobs, Ed., 19 July 2001, available at http://www.w3.org/Consortium/Process-20010719/.
QAF-OPS
QA Framework: Operational Guidelines , L. Henderson, D. Hazaël-Massieux, K. Gavrylyuk, D. Dimitriadis, L. Rosenthal, Eds., W3C Working Draft, 08 November 2002, companion version to this document, available at http://www.w3.org/TR/2002/WD-qaframe-ops-20021108/.
SPEC
QA Framework: Specification Guidelines , D. Hazaël-Massieux, L.Henderson, L. Rosenthal, D. Dimitriadis, K. Gavrylyuk, Eds., W3C Working Draft, 08 November 2002, Working Draft companion version to this document, available at http://www.w3.org/TR/2002/WD-qaframe-spec-20021108/.
SVG-MAN
SVG Conformance Test Suite -- Test Builder's Manual , L. Henderson, 1 October 2001, available at http://www.w3.org/Graphics/SVG/Test/svgTest-manual.htm.
QAIG
Quality Assurance Interest Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/IG/.
QAWG
Quality Assurance Working Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/WG/.
W3C-TR
Location of all published W3C technical reports, see http://www.w3.org/TR/.

7. Change history

2002-11-08, (initial) companion to 20021108 Operational Guidelines (OpsGL) publication

Revised to track changes in corresponding OpsGL draft. See "Change history" of that corresponding Operational Guidelines draft.

2002-05-15, first published Working Draft