W3C

QA Framework: Operational Examples & Techniques

W3C Note 04 August 2003

This version:
http://www.w3.org/QA/WG/2003/08/qaframe-ops-extech-20030804
Latest version:
http://www.w3.org/QA/WG/qaframe-ops-extech
Previous version:
http://www.w3.org/QA/WG/2003/02/qaframe-ops-extech-20030217
Editors:
Lofton Henderson (lofton@rockynet.com)
Lynne Rosenthal (lsr@nist.gov)
Dimitris Dimitriadis (dimitris@ontologicon.com)
Kirill Gavrylyuk (kirillg@microsoft.com)
Contributors:
See Acknowledgments.

Abstract

This document is part of the of the Quality Assurance (QA) Activity. It presents examples and describes the techniques of operational aspects of quality practices within the W3C's Working Groups. It complements QA Framework: Operational Guidelines [QAF-OPS], by specifying or illustrating how Working Groups might meet the operational and process-related checkpoints of that document.

Status of this document

This version is a QA Working Group (QAWG) editor's draft, to accompany the 20030804 QAWG editor's draft of Operational Guidelines (OpsGL). Editorial alignment with revised OpsGL is still incomplete...

This section describes the status of this document at the time of its publication. Other documents may supersede this document. The latest version of this document series is maintained at the W3C.

This document is a W3C Note, made available by the W3C Quality Assurance (QA) Activity for discussion by W3C members and other interested parties. The document status is being maintained as a W3C Note, pursuant to resolution of Issue #68. For more information about the QA Activity, please see the QA Activity statement.

This draft differs from the Previous Version, by incorporating changes to wording, order, and statements of guidelines and checkpoints that have been made to OpsGL as a result of Last Call issue processing. Revision of the corresponding Examples&Techniques content itself is still in progress, especially for the revised Guidelines 1 & 2 of OpsGL. The principal changes from the Previous Version are listed in the "Change history" section.

In this version, as in all previous versions, three case studies are presented in some detail -- how the QA projects of three existing Working Groups relate to the checkpoints of the operational guidelines [QAF-OPS]. There is also some systematic enumeration of techniques -- ways in which Working Groups might satisfy the operational and process-related checkpoints.

It is anticipated that this document will be updated more frequently than its companion OpsGL specification. The per-checkpoint linkage between that document and future versions of this one will be maintained as the examples and techniques are improved.

Please send comments to www-qa@w3.org, the publicly archived list of the QA Interest Group [QAIG]. Please note that any mail sent to this list will be publicly archived and available, do not send information you wouldn't want to see distributed, such as private data.

Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than work in progress.

A list of current W3C Recommendations and other technical documents can be found at http://www.w3.org/TR.

Table of contents

  1. Introduction
  2. Conformance test suite case studies
  3. Operational guidelines in action
  4. Conformance
  5. Acknowledgments
  6. References
  7. Change history

Two separate appendices to this document [CHART-TEMPL] and [QAPD-TEMPL] contain respectively a Working Group Charter template, for demonstrating how to add QA-related information to the Charter, and a Working Group QA Process Document template (QAPD), demonstrating how to satisfy all of the QAPD-related checkpoints of the operational guidelines. Both of these are fill-in-the-blank templates that can be used by the WGs.


1. Introduction

1.1. Scope

This is an informative complement to QA Framework: Operational Guidelines [QAF-OPS]. It provides:

OpsGL should be considered a prerequisite to this document. Within its checkpoints, it contains all of the normative requirements for the operational aspects of quality practices within the W3C's Working Groups.

1.2. Overview of contents

Case studies

Three test suite (TS) projects related to W3C Recommendations are used as case-study examples. Some of them were written in-house (within the responsible WG), some externally (outside of W3C), and some are hybrid (external plus in-house effort). All of them have succeeded in producing significant and valuable TS deliverables.

In all cases, the associated WGs have existed and were well advanced on their relevant Recommendations before the operational guidelines [QAF-OPS] were first published. Because these TS projects preceded the guidelines, it makes no sense to measure them against checkpoints -- it is unreasonable to expect that they would meet all of the specific details of the checkpoints. Instead, the various aspects of the different projects are used to illustrate the meaning and intent of the checkpoints. In a sense, the checkpoints are the union of the best practice and techniques from all of projects (these and others).

This points to a timing-based reality of conformance to the operational guidelines. The conformance techniques (to be more fully developed in the future versions), will be different for new Working Groups than for existing Working Groups.

Note. The case study information was current as of approximately June, 2002. More recent developments may not be reflected.

Examples & techniques

Many of the checkpoints, have an examples & techniques section that specifically illustrates ways to satisfy the checkpoint.

1.3. Understanding and using this document

The structure of this document parallels the structure of the operational guidelines [QAF-OPS] document, particularly its Chapter 3, "Guidelines". It should be realized that all of the case-study examples from the W3C Working Groups pre-date the applicable parts of the W3C QA Activity. A useful milestone for the latter is the first publication of the Operational Guidelines, February 2002.

Relevant common features in these case-study frameworks will be pointed out, and they will be compared against the statements of the checkpoints of the operational guidelines. For each of the QA-related frameworks, a short descriptive summary will be presented. The subsequent parts of the document will elaborate on the checkpoints -- including links to relevant process-related documents and resources -- and how they have been realized in various efforts.

Within each checkpoint, In addition to these case studies, this document also specifies techniques that might be used to satisfy the checkpoint.

2. Conformance test suite case studies

Herein is a summary description of each of a small set of case studies of W3C QA-related activities, as well as some external to the W3C.

Each subsection covers the following topics:

2.1. DOM

General overview

The DOM TS was jointly launched by the W3C and NIST and is produced in public. It uses coordination from the W3C DOM WG, and is produced using communication through a W3C-hosted mailing list as well as W3C-supplied storage for files, documents and so forth. The TS was to be developed in public, and the WG would endorse two bindings, Java and ECMAscript, the same as the official DOM bindings. Other bindings were welcomed but would not form part of the official TS.

Internal/external/hybrid and staffing requirements

The DOM TS was launched by the W3C DOM WG and NIST, and featured a public call for participation in the production of quality testing material for implementations of the DOM specifications. Staffing requirements were limited, so an expert was invited to the DOM WG in order to produce and edit the DOM TS Process Document. The DOM TS was in its initial phase very similar to a previous DOM Test Suite launched by NIST, which was used as a primer for the forthcoming work. However, the subsequentially released and forthcoming versions differ greatly given the group's work.

Charter considerations

Since the DOM charter did not include sections on producing test materials, this was included in the DOM TS Process Document [DOM-TS-PROC], which outlined the DOM TS deliverables in detail.

When TS is produced -- during or after specification phase?

The DOM TS was produced after specification for the first two levels (DOM Level 1 & 2) and is produced in parallel with specifying DOM Level 3. DOM Level 1 was released in late 1998, so it was not possible to synchronize the specification of the first level of the DOM with the Test Suite.

Starting with the current level of the DOM, the TS is produced in parallel with the specification. Tests are being produced to allow for implementors to check their implementations of DOM Level 2 HTML and DOM Level 3 while contributing to relevant parts of the specification. Tests for the various modules of the DOM Level 2 HTML and DOM Level 3 specification are being written in parallel with the specification process.

Pointer to process documents

There is a separate DOM TS Process Document [DOM-TS-PROC], and also a Test Web page where you can find more information and pointers to primers, FAQ's and tutorials, maintained by the DOM WG representative to the DOM TS.

Test material storage, maintenance, tracking

The DOM TS uses W3C machinery to store test files, using CVS for easy tracking, and an issue tracking system at Sourceforge, in order to simplify issues and maintenance.

2.2. SVG

General overview

As used in this case study, "SVG" refers collectively to two consecutively chartered Working Groups. SVG1 refers to the originally chartered group (first meeting, Sept 1998) that produced SVG 1.0 (Recommendation, Sept 2001). SVG2 is the successor to SVG1, (re-)chartered in 2001, and charged with developing SVG 1.1 (modularization and profiles), and SVG 2.0. SVG 1.1 and the companion SVG Mobile Profiles are now Recommendations. Work continues on SVG 1.2, a functional enhancement to SVG 1.1 of more limited and immediate scope than SVG 2.0.

SVG 1.0 was released with a comprehensive Basic Effectivity (BE) test suite, that was developed entirely within the working group. The test suite (TS) comprises 127 test cases that span the functionality of SVG, exercising each major functional feature of SVG 1.0 at least lightly, but not probing any features in detail. SVG2 is adapting and subsetting the tests of the SVG1 TS for smaller and less capable mobile devices that are the target constituency of the Tiny and Basic profiles.

Internal/external/hybrid and staffing requirements

The SVG1 test suite was developed entirely by the working group. A pilot study sponsored by one of the members became the basis for design. A "TS Editor" (moderator) managed the project from inception to completion. All WG members are expected to participate in the writing and review of test cases. The SVG2 approach is identical -- a WG member is the TS moderator, and all WG members contribute.

Charter considerations

Both SVG1 & SVG2 charters specify a comprehensive test suite as a deliverable, and they coordinate the dates of first TS delivery (first publication) and the specification CR milestone. There is no further detailed breakdown of milestones or deliverables. This was worked out (for SVG1) as the specification development progressed.

When TS is produced -- during or after specification phase?

The test suites were (are) produced during the specification-writing phase. The target is substantial completion at the Candidate Recommendation (CR) milestone, so that the TS could be used to facilitate the interoperable-implementations W3C Process requirements of CR ([PROCESS], section 5.2.3)

Pointer to process documents

SVG has no QA process document per se, but much of the content that these operational checkpoints anticipate and require may be found in the extensive reference document, SVG Conformance Test Suite -- Test Builder's Manual [SVG-MAN]. Unless otherwise specified in that document, the normal WG processes of the SVG Working Group are used to conduct the TS work.

Test material storage, maintenance, tracking

The SVG WG keeps the SVG TS on a CVS server of one of the WG members. Revision tracking is therefore handled by CVS. Maintenance procedures and issue tracking for SVG TS are not separately defined, but rather SVG TS uses the normal SVG WG processes for these.

2.3. XML

General overview

Note. Following its transfer from outside W3C to the XML WG, when this case study was written, the XML TS was in preparation for first publication by the WG. Therefore a number of the referenced documents were not yet publicly visible.

The OASIS/ NIST XML Test Suite (TS) forms the basis for the W3C XML TS. As a result of a multi-way collaboration between

it was transferred to W3C where it is being augmented with new functionality, corrected to align with the XML 1.0 Second Edition Recommendation and its errata, and maintained.

Prior to accepting the transfer of the OASIS/NIST Test Suite, the XML Core WG assessed the level of effort necessary for TS development and maintenance and ensured that staff resources were interested and available to work on the TS. Additionally, the WG enlisted the assistance of NIST to lead the effort. The TS is being developed as a deliverable of the WG, following the WG's Process. There is a W3C hosted XML TS mailing list, Issues list, and W3C supplied storage (CVS) for the test-suite related files.

The XML Conformance Test Suite, v1.0, Second Edition contains over 2000 test files and an associated test report. The test report contains background information on conformance testing for XML as well as test descriptions for each of the test files included in the TS.

Internal/external/hybrid and staffing requirements

A commitment from NIST to take editor-lead responsibility for the TS was obtained prior to the commencement of the work. Consequently, NIST joined the WG. Members of the XML Core WG contribute to the development and quality assessment of the TS. When the XML TS is released by the WG, the general public will be invited to review the tests as well as participate in contributing tests.

Charter considerations

The W3C XML Core WG charter does not include sections on producing test materials. A new charter provides for the possibility of working on the OASIS/NIST XML Test Suite. The WG does not have a separate QA Process, but follows the WG Process for the development, assessment and maintenance of the XML TS. This is not explicitly documented in a separate QA process document, but the individual decisions are on record in WG mail archives and minutes as part of the normal WG process.

When TS is produced -- during or after specification phase?

The work on the OASIS/NIST XML Test Suite was conducted after the XML 1.0 Recommendation was released. A revised edition of the OASIS/NIST Test Suite 1.0 (Second Edition) Test Suite was released to complement the W3C XML 1.0 Second Edition Recommendation (October 6, 2000). Within the XML Core WG, the XML TS is being updated to reflect the current Recommendation and all applicable Errata.

Pointer to process document

There is no separate QA Process document. The XML Core WG charter uses its established WG Process to handle test development/assessment activities, communication means and test materials related functions.

Update. XML now does have a separate process document [XML-TS-PROC].

Test material storage, maintenance, tracking

The XML WG uses the W3C CVS repository to maintain the XML TS. The XML TS issues are maintained utilizing an internal assessment process that consists of a running log of test issues, XML TS mailing lists, XML Core WG discussions and the XML Test Suite Issues document that keeps track of the running log of issues.

3. Operational guidelines in action

This section looks at each of the guidelines and checkpoints in the operational guidelines [QAF-OPS]. For each checkpoint, there are two sections of content:

The Case Studies sections look at how the checkpoints' requirements are reflected in three existing operational frameworks (SVG, DOM, and XML). Where appropriate, pointers to relevant documents and materials are provided.

The Examples & Techniques sections demonstrate how the checkpoints might be satisified. The Examples & Techniques sections are supported by two separate appendices to this document:

These two appendices are fill-in-the-blank templates that can be used directly by the WGs, to satisfy a number of the operational checkpoints.

The structure of this chapter exactly parallels the structure of the its Chapter 3, "Guidelines", of the operational guidelines [QAF-OPS] document -- each checkpoint in this section links back to the corresponding checkpoint of QA Framework: Operational Guidelines [QAF-OPS].

Guideline 1. Commit to Quality Assurance in Working Group activities.

[@@Ed note. The Case studies and the E&T content of checkpoints (CP) 1.1, 1.2, 1.3 is yet to be rewritten, according to new formulations of CP1.1-1.3.]

Case studies summaries

DOM discussion. In the DOM TS, the QA activities were integrated into Working Group activities by inviting a field expert to do the coordination between WG and TS Group. Coordination was necessary as the DOM WG QA activity was launched in coordination with an external party, and, furthermore, was to be produced externally.

SVG discussion. QA was integrated into the SVG WG activities by a Charter commitment to produce test suites, and by subsequent involvement of most WG members in the writing, reviewing, and maintenance of the test materials. The charters committed, in the cases of both SVG1 and SVG2, that a basic test suite (i.e., Basic Effectivity) would exist at or before Candidate Recommendation. The details of deliverables, milestones, and publication were generally worked out by the WG as the specifications developed.

XML discussion. The QA activities are integrated into WG activities by using the WG Process. Although the activities are not reflected in the WG Charter, deliverables are identified, expected release dates determined, staff allocated, and there is a clear commitment to developing and maintaining the TS.

Checkpoints

Checkpoint 1.1. Commit to QA Level-3.Define QA commitment levels for operations, specifications, and test materials. [Priority 1]

[@@Ed note. Review/rewrite text according to new formulations of CP1.1-1.3.]

Case studies
Examples & techniques

Techniques for charter-related QA requirements in this checkpoint are demonstrated in the charter-template appendix [CHART-TEMPL].

Guidelines document description of checkpoint.

Checkpoint 1.2. Commit to QA Level-5.Commit to test materials. [Priority 2]

[@@Ed note. Review/rewrite text according to new formulations of CP1.1-1.3.]

Case studies
Examples & techniques

Techniques for charter-related QA requirements in this checkpoint are demonstrated in the charter-template appendix [CHART-TEMPL].

See the techniques for QA level three. The techniques for QA level five will be the same except for the specific extent of the activities to which the WG commits.

Guidelines document description of checkpoint.

Checkpoint 1.3. Commit to QA Level-7.Commit to complete test materials. [Priority 3]

[@@Ed note. Review/rewrite text according to new formulations of CP1.1-1.3.]

Case studies
Examples & techniques

Techniques for charter-related QA requirements in this checkpoint are demonstrated in the charter-template appendix [CHART-TEMPL].

See the techniques for QA level three and for QA level five. The techniques for QA level seven will be the same except for the specific extent of the activities to which the WG commits.

Guidelines document description of checkpoint.

Checkpoint 1.4. Enumerate QA deliverables and expected milestones. [Priority 1]
Case studies
Examples & techniques

Techniques for charter-related QA requirements in this checkpoint are demonstrated in the charter-template appendix [CHART-TEMPL].

The satisfaction of this checkpoint is demonstrated in the charter-template appendix, specifically the sections about WG deliverables and deliverables milestones.

Guidelines document description of checkpoint.

Checkpoint 1.5. Associate QA criteria with WG milestones.Define QA criteria for Recommendation-track advancement. [Priority 2]
Case studies
Examples & techniques

Techniques for charter-related QA requirements in this checkpoint are demonstrated in the charter-template appendix [CHART-TEMPL].

The satisfaction of this checkpoint is demonstrated in the charter-template appendix, in the milestones-criteria section.

Guidelines document description of checkpoint.

Guideline 2. Commit to resource level for Working Group QA activities.

Case studies summaries

DOM discussion. The DOM WG appointed an invited expert to monitor and coordinate the work of the DOM TS. Furthermore, the members of the DOM WG have looked into producing tests fot that part of the specification which is of particular interest to their implementations.

SVG discussion. The SVG1 and SVG2 charters explicitly charge the WG with producing test materials, but do not further address specific staffing. In fact, the expectation has evolved that all WG members should contribute to the building of the TS. In both cases, a "TS Editor" has been assigned to lead and coordinate the TS production.

XML discussion. The WG appointed NIST to lead the TS effort and coordinate the work of the XML TS. Furthermore, members of the WG have reviewed and assessed the current set of tests, corrected tests, and contributed tests.

Checkpoints

Checkpoint 2.1. Address where and how conformance test materials will be produced. [Priority 1]
Case studies
Examples & techniques

Techniques for charter-related QA requirements in this checkpoint are demonstrated in the charter-template appendix [CHART-TEMPL].

There are several possibilities for producing conformance test materials:

There are good reasons why it is useful for external parties to have a strong participation during the building of test materials. But even if a test suite is being developed outside of W3C, the Working Group needs to ensure it is completed according to the QA level to which the Working Group is committed in its charter.

The satisfaction of this checkpoint is demonstrated in the charter-template appendix, in the participation section.

Guidelines document description of checkpoint.

Checkpoint 2.2. Address QA staffing commitments. [Priority 1]
Case studies
Examples & techniques

Techniques for charter-related QA requirements in this checkpoint are demonstrated in the charter-template appendix [CHART-TEMPL].

By now, the WG (or the writers of its Charter) have already addressed the overall QA commitment level (QA level three, five, seven, or other), and the basic scenario for production of test materials and other QA deliverables (intra-WG, external, or hybrid). These will determine the total amount of WG staff resources that will be needed for the commitment and plan. To satisfy this checkpoint, the Working Group can:

The satisfaction of this checkpoint is demonstrated in the charter-template appendix, in the participation section.

Guidelines document description of checkpoint.

Checkpoint 2.3. Request allocation of QA resources to the Working Group. [Priority 1]
Case studies
Examples & techniques

(Under construction)

It is not unusual that responders to a WG Call for Participation have interest limited to the "technical invention" side of the WG's work -- designing the specific technical details of the standard. By emphasizing in its participation calls that QA specialists are needed and welcome, the WG improves its chances of getting interested and competent people to work on the QA commitments and plans.

A related topic has arisen in email discussions: whether, for QA-specific work, member organizations can add more participants to a WG's roster than the WG's rules normally allow. I.e., some WGs restrict the number of participants from a company to, for example, two. One proposed technique for staffing a WG's QA plans and commitments would be to allow additional members, but without additional voting rights.

(@@TBD -- prototype language that can be inserted into a CfP).

Guidelines document description of checkpoint.

Guideline 3. Synchronize QA activities with the specification milestones.

Case studies summaries

DOM discussion. The milestones were given in the DOM TS Process document and were monitored by the DOM WG in order to assure the satisfactory continuation of the DOM TS group work. In those cases where the TS Group needed more time to achieve its goals, the DOM WG representative reported to the WG in order to appeal for an extension.

Applicable in so far as the TS releases are (for levels currently being specified) synchronized with the relevant modules of the DOM specifications.

SVG discussion. The SVG1 and SVG2 charters establish the basic synchronization, which is that the first publications of basic test suites (TS) should be at or before the beginning of CR stage. Beyond that, there are no explicit (i.e., written down) synchronizations. This happens in fact as part of the WG process and the TS building. TS publications are synchronized with selected late-stage document versions (CR, PR, Rec). There is no explicit scheme for labelling/linking test cases according to errata, although this would be reconstructable from the CVS repository for the TS.

XML discussion. The TS work is being conducted in close coordination with the WG's other deliverables. The TS is being updated and augmented to reflect the current work of the WG, including resolved issues related to the Recommendation and published Errata. Tests for new functional areas are also being explored as the deliverables for those Recommendations progress (i.e., Xinclude and Namespace).

Checkpoints

Checkpoint 3.1. Synchronize the publication of QA deliverables and the specification's drafts. [Priority 2]
Case studies
Examples & techniques

(Under construction)

Guidelines document description of checkpoint.

Checkpoint 3.2. Support specification versioning/errata in QA deliverables. [Priority 3]
Case studies
Examples & techniques

(Under construction)

Guidelines document description of checkpoint.

Guideline 4. Define the QA process.

Case studies summaries

DOM discussion. DOM TS Process document

SVG discussion. SVG supports the process requirements of a QA moderator (manager of TS development) and a task force (it is the whole WG, by explicit choice). TS-related communication is handled internally on the whole-WG list (member-only), and externally via a public TS comment list. SVG does not have a Process document, per se, that spells out these details. It does have a TS manual that defines the TS framework in some detail.

XML discussion. The WG Process is used.

Checkpoints

Checkpoint 4.1. Appoint a QA moderator. [Priority 2]
Case studies
Examples & techniques

Depending on the plan for the origin of the test suite a Working Group can:

When a test suite is developed outside of the Working Group, a moderator often already exists in the external group that is the source of the test suite. It is recommended to invite the moderator to become a Working Group member, since he/she needs access to all Working Group materials and resources. See also section on transferring test suite from outside W3C.

Guidelines document description of checkpoint.

Checkpoint 4.2. Appoint a QA task force. [Priority 2]
Case studies
Examples & techniques

(under construction)

Guidelines document description of checkpoint.

Checkpoint 4.3. Produce the QA Process Document. [Priority 1]
Case studies
Examples & techniques

Techniques to satisfy this checkpoint are demonstrated in the QAPD-template appendix[QAPD-TEMPL], which can be used directly by Working Groups to produce a QA Process Document (QAPD).

Guidelines document description of checkpoint.

Checkpoint 4.4. Specify means for QA-related communication. [Priority 2]
Case studies
Examples & techniques

Techniques to satisfy this checkpoint are demonstrated in the QAPD-template appendix[QAPD-TEMPL], which can be used directly by Working Groups to produce a QA Process Document (QAPD). This checkpoint, specifically, is demonstrated in the "QA-related Communication" section.

Guidelines document description of checkpoint.

Checkpoint 4.5. Define branding policy details. [Priority 3]
Case studies
Examples & techniques

Techniques to satisfy this checkpoint are demonstrated in the QAPD-template appendix[QAPD-TEMPL], which can be used directly by Working Groups to produce a QA Process Document (QAPD). This checkpoint, specifically, is demonstrated in the Conformance section.

Guidelines document description of checkpoint.

Guideline 5. Plan test materials development.

Case studies summaries

DOM discussion. As the DOM TS drew on experiences made by an external party (NIST), the work initially drew heavily on those experiences. Where the subsequent work indicated digression, the future version of the DOM TS was branched off and formed the official W3C DOM TS. The test material planning was made by the DOM WG representative to the DOM TS in coordination with NIST and the public, after the DOM TS had been officially launched.

Several design issues were discussed by the DOM TS Group itself, then brought to the DOM WG for acceptance. Such issues include the automatic generation of the schema language for the DOM TS, using the same test description file to generate the official bindings (and the unofficial ones using a different transform) as well as using particular build tools to build working and official releases of the DOM TS.

Most discussion was done on the DOM Group mailing list and coordinated with the DOM WG. More information can be found in the DOM TS page, which serves as a complete reference for DOM TS related information.

SVG discussion. The planning for SVG's test materials development started with a study and prototype design, sponsored by one of the SVG WG members. This work was brought into the WG as the starting point of the TS development, and was further developed, refined, and applied by the moderator ("TS Editor"). Both user documentation (for TS releases) and contributors documentation were developed. The latter described the TS framework in some detail, and gave both technical criteria for the contributed test cases plus review/acceptance criteria. The actually processes and procedures for contribution and review were not explicitly defined, as all materials came from WG members (and therefore followed WG process).

XML discussion. The XML TS drew on the experiences and expertise of NIST, the original developer of the test suite framework. The original work was revised to included test requirements with a mapping to the test cases and to the specification. The planning of improvements and augmentation to the original work was discussed and decided by the WG. Techniques and methods were borrowed from other test suite development efforts, such as DOM and XSL-FO.

Checkpoints

Checkpoint 5.1. Define a framework for test materials development. [Priority 2]
Case studies
Examples & techniques

Techniques to satisfy this checkpoint are demonstrated in the QAPD-template appendix[QAPD-TEMPL], which can be used directly by Working Groups to produce a QA Process Document (QAPD). This checkpoint, specifically, is demonstrated in the ""Development Framework" section.

Guidelines document description of checkpoint.

Checkpoint 5.2. Ensure test materials are documented and usable for their intended purposes. [Priority 2]
Case studies
Examples & techniques

(under construction)

Guidelines document description of checkpoint.

Checkpoint 5.3. Define a contribution process. [Priority 2]
Case studies
Examples & techniques

Techniques to satisfy this checkpoint are demonstrated in the QAPD-template appendix[QAPD-TEMPL], which can be used directly by Working Groups to produce a QA Process Document (QAPD). This checkpoint, specifically, is demonstrated in the "Test materials contribution" sub-section.

Guidelines document description of checkpoint.

Checkpoint 5.4. Address license terms for submitted test materials. [Priority 2]
Case studies
Examples & techniques

Techniques to satisfy this checkpoint are demonstrated in the QAPD-template appendix [QAPD-TEMPL], which can be used directly by Working Groups to produce a QA Process Document (QAPD). This checkpoint, specifically, is demonstrated in the Licenses section.

[@@Ed note. Add more examples to Extech per KG email of 20021218.]

Guidelines document description of checkpoint.

Checkpoint 5.5. Define review procedures for submitted test materials. [Priority 2]
Case studies
Examples & techniques

The following example meets the review requirements:

  1. Each contribution should be reviewed for its eligibility by at least one vendor other than the submitter. Any vendor involved in the creation of the Working Group test collection other than the submitter may review contributed tests.
  2. Eligibility of tests should be judged by:
    • The accuracy of the test. The accuracy of a test is determined by a judgment of the reviewer whether the test case actually tests what the submitter states that the test case tests. The accuracy is the extent to which the test matches the cited parts of the specification. If it does not match, or only partially matches, the test should be considered inaccurate.
    • The scope of the test. Test should target MUST provisions of the specification and not MAY/SHOULD provisions. The test should target only conformance requirements. Data values limits, memory or performance limits should not be targets to the test unless those are specified in the conformance requirements.
    • The clarity of the test. The clarity of a test is a determination of whether the aspect being tested is clearly described with the anticipated results acceptably explained.
    • The clarity of aspect of the Specification being tested. The test suite aims to test parts of the Specification and errata that are not vague.
  3. After review, results should be sent back to the submitter for feedback.
  4. In case of disagreement, a request for clarification should go to Working Group. Tests in question should be left out of the test suite until the W3C Working Group clarifies the issue.

Only those tests that are agreed to be correct by reviewers and the submitter, or that are approved by W3C Working Group clarification, should be published.

Techniques to satisfy this checkpoint are also demonstrated in the QAPD-template appendix [QAPD-TEMPL], which can be used directly by Working Groups to produce a QA Process Document (QAPD). This checkpoint, specifically, is demonstrated in the contribution receipt and review section.

Guidelines document description of checkpoint.

Guideline 6. Plan test materials publication.

Case studies summaries

DOM discussion. Plans for test material publication are discussed internally in the DOM TS group, and then decided by the DOM TS Group in coordination with the DOM WG.

SVG discussion. The SVG TS project has explicitly discussed and planned the key aspects of TS publication. CVS repository is used for the ongoing development of the TS, and at agreed milestones, a snapshot is taken and published under the W3C Document License. The TS home page links all of the materials and documentation, and also points to an implementation matrix that publishes the TS results of a half-dozen or so implementations.

XML discussion. Publication of the TS is being discussed by the WG. The TS will reside on the W3C web space and will be freely and publicly available. In general, the publication details of the XML TS are relatively invisible outside of the WG, because there has not been any publication yet.

Checkpoints

Checkpoint 6.1. Ensure a suitable repository location for test materials. [Priority 2]
Case studies
Examples & techniques

(Under construction)

Guidelines document description of checkpoint.

Checkpoint 6.2. Define the licenses applicable to published test materials. [Priority 2]
Case studies
Examples & techniques

Techniques to satisfy this checkpoint are demonstrated in the QAPD-template appendix[QAPD-TEMPL], which can be used directly by Working Groups to produce a QA Process Document (QAPD). This checkpoint, specifically, is demonstrated in the Licenses section.

Guidelines document description of checkpoint.

Checkpoint 6.3. Describe how and where the test materials will be published. [Priority 2]
Case studies
Examples & techniques

Techniques to satisfy this checkpoint are demonstrated in the QAPD-template appendix[QAPD-TEMPL], which can be used directly by Working Groups to produce a QA Process Document (QAPD). This checkpoint, specifically, is demonstrated in the XXX section.

Guidelines document description of checkpoint.

Checkpoint 6.4. Provide a conformance verification disclaimer with the test materials. [Priority 2]
Case studies
Examples & techniques

Techniques to satisfy this checkpoint are demonstrated in the QAPD-template appendix [QAPD-TEMPL], which can be used directly by Working Groups to produce a QA Process Document (QAPD). This checkpoint, specifically, is demonstrated in the Conformance section.

[@@examples of disclaimers, where & how, and "prominent"@@]

Guidelines document description of checkpoint.

Checkpoint 6.5. Promote testing and the publication of test results. [Priority 3]
Case studies
Examples & techniques

Techniques to satisfy this checkpoint are demonstrated in the QAPD-template appendix[QAPD-TEMPL], which can be used directly by Working Groups to produce a QA Process Document (QAPD). This checkpoint, specifically, is demonstrated in the "Publication of test results" sub-section.

Guidelines document description of checkpoint.

Guideline 7. Plan the transfer of test materials to W3C if needed.

Case studies summaries

DOM discussion. This issue was addressed early on in the specification phase of the DOM TS work, as a choice had to be made between integrating an existing TS or just using the existing one as a primer for its succesor. The latter was preferred.

This is addressed in the DOM TS Process document.

SVG discussion. This guideline is not applicable to the SVG TS -- all TS development so far has occurred within the SVG WG.

XML discussion. The issue was addressed prior to the transfer of the TS to XML WG, and prior to the start of TS development and maintenance there. Since that phase is finished, it is no longer applicable. Included in the discussions were the quality of the OASIS/NIST XML Test Suite, amount of work needed to correct and improve it, as well as IPR questions.

Checkpoints

Checkpoint 7.1. Perform a quality assessment of any test materials that are candidates for transfer. [Priority 2]
Case studies
Examples & techniques

(Under construction)

[@@for OPS-EXTECH, a more comprehensive list or even checklist/template, such as...] "correctness, traceability, atomicity, user documentation, maintainer documentation, declaration of scope, completeness (vis a' vis declared scope), harnesses or interfaces for application of the TM, configurability, results assessment, results recording & reporting, automation features, versioning/errata support, declaration of publication licenses, integrated submission procedures, etc."

Guidelines document description of checkpoint.

Checkpoint 7.2. Identify sufficient staff resources to meet the needs of any transferred test materials. [Priority 2]
Case studies
Examples & techniques

(Under construction)

Guidelines document description of checkpoint.

Checkpoint 7.3. For any transferred test materials, resolve all IPR issues with the external party that produced the test materials. [Priority 1]
Case studies
Examples & techniques

The following procedure is an example that meets all of the transfer requirements.

Legend for the procedure:

EG
the external group or entity
QAWG
the QA Working Group
TM
the test materials
WG
the subject Working Group

Before the transfer, WG with the help of QAWG:

During transfer:

After transfer, initial process setup:

First W3C public release of the new TM:

After the first public release, the TM enter the maintenance phase which is described below.

Guidelines document description of checkpoint.

Guideline 8. Plan for test materials maintenance.

Case studies summaries

DOM discussion. No clear plan for test material maintenance has been made, other than indicating in the process document that these issues will be adressed by the DOM TS group.

Not applicable

SVG discussion. SVG has no explicit plan for TS maintenance. Because the TS process is basically the WG process, maintenance happens happens as a natural part of the WG's specification development and ongoing TS development.

XML discussion. No clear plan for test material maintenance has been made yet. The need to address this has been acknowledged by the WG, with plans to address it in the near future.

Checkpoints

Checkpoint 8.1. Provide for the long-term maintenance of the contribution and review procedures. [Priority 3]
Case studies
Examples & techniques

(Under construction)

Guidelines document description of checkpoint.

Checkpoint 8.2. Specify a test materials update procedure to track new specification versions/errata. [Priority 2]
Case studies
Examples & techniques

The following procedure is an example that meets the requirements:

Reviewing/accepting the change. Test changes go through the same tests review process as presented earlier. If the test changes are accepted, they are added to the test suite with an indicator of errata/version. Old tests stay in the test suite marked with the last errata/version level they comply with. The status changes to "CLOSED". If tests are rejected, they go back to the provider of the change with that determination, and the status is reverted back to "ACTIVE"

Techniques to satisfy this checkpoint are also demonstrated in the QAPD-template appendix[QAPD-TEMPL], which can be used directly by Working Groups to produce a QA Process Document (QAPD). This checkpoint, specifically, is demonstrated in the test materials maintenance section.

Guidelines document description of checkpoint.

Checkpoint 8.3. Identify a procedure for test validity appeals. [Priority 2]
Case studies
Examples & techniques

The following topical outline is an example that meets the requirements, i.e., if these topics were addressed in the process document::

If there is still active work on the test suite even if the Working Group is not re-chartered to handle the work, then it is up to the Director to determine how this work should be done. Some options include:

Techniques to satisfy this checkpoint are also demonstrated in the QAPD-template appendix[QAPD-TEMPL], which can be used directly by Working Groups to produce a QA Process Document (QAPD). This checkpoint, specifically, is demonstrated in the test-validity review and the test materials maintenance sections.

Guidelines document description of checkpoint.

4. Conformance

There is no concept of conformance to this document. Rather, conformance is measured relative to the checkpoints of the companion document, QA Framework: Operational Guidelines [QAF-OPS]. This document relates real-world examples to the checkpoints' requirements, and also presents techniques that would suffice to satisfy the checkpoints.


5. Acknowledgments

The following QA Working Group and Interest Group participants have contributed significantly to the content of this document:

6. References

DOM-TS-PROC
DOM TS Process Document, D. Dimitriadis, Ed., 15 January 2002, available at http://www.w3.org/DOM/DOMTS-Process.
PROCESS
World Wide Web Consortium Process Document, I. Jacobs, Ed., 19 July 2001, available at http://www.w3.org/Consortium/Process-20010719/.
QAF-OPS
QA Framework: Operational Guidelines, L. Henderson, D. HazaŽl-Massieux, K. Gavrylyuk, D. Dimitriadis, L. Rosenthal, Eds., W3C Working Draft, 08 November 2002, companion version to this document, available at http://www.w3.org/TR/2003/WD-qaframe-ops-20030210/.
SPEC
QA Framework: Specification Guidelines, D. HazaŽl-Massieux, L.Henderson, L. Rosenthal, D. Dimitriadis, K. Gavrylyuk, Eds., W3C Working Draft, 10 February 2003, Working Draft companion version to this document, available at http://www.w3.org/TR/2003/WD-qaframe-spec-20030210/.
SVG-MAN
SVG Conformance Test Suite -- Test Builder's Manual, L. Henderson, 1 October 2001, available at http://www.w3.org/Graphics/SVG/Test/svgTest-manual.htm.
QAIG
Quality Assurance Interest Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/IG/.
QAWG
Quality Assurance Working Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/WG/.
W3C-TR
Location of all published W3C technical reports, see http://www.w3.org/TR/.
XML-TS-PROC
XML TS Process Document, S. Martinez, 14 May 2002, available at http://www.w3.org/XML/Test/XMLConformanceTS-Process-20020514.html.

7. Change history

2003-08-04, (initial) companion to 20030804 Editor's draft of Operational Guidelines (OpsGL), which version differs from previous OpsGL version by incorporation of Last Call comments, and which version is anticipated to be substantially the Candidate Recommendation version.

2003-02-17, (initial) companion to 20030210 Operational Guidelines (OpsGL) Last Call WD publication.

2002-11-21, (initial) companion to 20021121 Operational Guidelines (OpsGL) publication

Revised to track changes in corresponding OpsGL draft. See "Change history" of that corresponding Operational Guidelines draft.

2002-11-08, (initial) companion to 20021108 Operational Guidelines (OpsGL) publication

Revised to track changes in corresponding OpsGL draft. See "Change history" of that corresponding Operational Guidelines draft.

2002-05-15, first published Working Draft