QA Framework: Operational Examples & Techniques

This document is part of the of the Quality Assurance (QA) Activity. It presents examples and describes the techniques of operational aspects of quality practices within the W3C's Working Groups. It complements QA Framework: Operational Guidelines [QAF-OPS], by specifying or illustrating how Working Groups might meet the operational and process-related checkpoints of that document.

Status of this document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. The latest version of this document series is maintained at the W3C.

This document is a W3C Note, made available by the W3C Quality Assurance (QA) Activity for discussion by W3C members and other interested parties. For more information about the QA Activity, please see the QA Activity statement.

This draft is signficantly revised from the Previous Version, to begin adding per-checkpoint Techniques, to the case studies that have already been included. The document status has changed from Working Draft to Note, pursuant to resolution of Issue #68. The principal changes from the Previous Version are listed in the "Change history" section.

In this version, as in previous versions, three case studies are presented in some detail -- how the QA projects of three existing Working Groups relate to the checkpoints of the operational guidelines [QAF-OPS]. This version includes, for the first time, some systematic enumeration of techniques -- ways in which Working Groups might satisfy the operational and process-related checkpoints.

It is anticipated that this document will be updated more frequently than its companion version (20030210, Last Call WD) of the Operational Guidelines. The per-checkpoint linkage between that document and future versions of this one will be maintained as the examples and techniques are improved.

Please send comments to www-qa@w3.org, the publicly archived list of the QA Interest Group [QAIG]. Please note that any mail sent to this list will be publicly archived and available, do not send information you wouldn't want to see distributed, such as private data.

Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than work in progress.

A list of current W3C Recommendations and other technical documents can be found at http://www.w3.org/TR.

Two separate appendices to this document [CHART-TEMPL] and [QAPD-TEMPL] contain respectively a Working Group Charter template, for demonstrating how to add QA-related information to the Charter, and a Working Group QA Process Document template (QAPD), demonstrating how to satisfy all of the QAPD-related checkpoints of the operational guidelines. Both of these are fill-in-the-blank templates that can be used by the WGs.

1. Introduction

1.1. Scope

This document contains examples and techniques of QA-related activities within the W3C as well as guidance as to how the quality processes of the Working Groups (WGs) might be carried out. It is an informative complement to QA Framework: Operational Guidelines [QAF-OPS]. Not only does it specifying or illustrating how WGs might satisfy the operational and process-related checkpoints of that document, it also aims to further illustrate and clarify the meaning of the principles in those checkpoints.

The operational guidelines document should be considered a prerequisite to this one. In addition to containing all of the checkpoints to which the contents of this document pertain, it explains principles underlying the conduct of WG quality processes.

The operational guidelines document also contains all of the normative requirements for the operational aspects of quality practices within the W3C's Working Groups. This document is completely information -- it contains no conformance requirements.

1.2. Overview of contents

Case studies

Three test suite (TS) projects related to W3C Recommendations are used as case-study examples. Some of them were written in-house (within the responsible WG), some externally (outside of W3C), and some are hybrid (external plus in-house effort). All of them have succeeded in producing significant and valuable TS deliverables.

In all cases, the associated WGs have existed and were well advanced on their relevant Recommendations before the operational guidelines [QAF-OPS] were first published. Because these TS projects preceded the guidelines, it makes no sense to measure them against checkpoints -- it is unreasonable to expect that they would meet all of the specific details of the checkpoints. Instead, the various aspects of the different projects are used to illustrate the meaning and intent of the checkpoints. In a sense, the checkpoints are the union of the best practice and techniques from all of projects (these and others).

This points to a timing-based reality of conformance to the operational guidelines. The conformance techniques (to be more fully developed in the future versions), will be different for new Working Groups than for existing Working Groups.

Note. The case study information was current as of approximately June, 2002. More recent developments may not be reflected.

Examples & techniques

Many of the checkpoints, have an examples & techniques section that specifically illustrates ways to satisfy the checkpoint.

1.3. Understanding and using this document

The structure of this document parallels the structure of the operational guidelines [QAF-OPS] document, particularly its Chapter 3, "Guidelines". It should be realized that all of the case-study examples from the W3C Working Groups pre-date the applicable parts of the W3C QA Activity. A useful milestone for the latter is the first publication of the Operational Guidelines, February 2002.

Relevant common features in these case-study frameworks will be pointed out, and they will be compared against the statements of the checkpoints of the operational guidelines. For each of the QA-related frameworks, a short descriptive summary will be presented. The subsequent parts of the document will elaborate on the checkpoints -- including links to relevant process-related documents and resources -- and how they have been realized in various efforts.

Within each checkpoint, In addition to these case studies, this document also specifies techniques that might be used to satisfy the checkpoint.

2. Conformance test suite case studies

Herein is a summary description of each of a small set of case studies of W3C QA-related activities, as well as some external to the W3C.

2.1. DOM

General overview

The DOM TS was jointly launched by the W3C and NIST and is produced in public. It uses coordination from the W3C DOM WG, and is produced using communication through a W3C-hosted mailing list as well as W3C-supplied storage for files, documents and so forth. The TS was to be developed in public, and the WG would endorse two bindings, Java and ECMAscript, the same as the official DOM bindings. Other bindings were welcomed but would not form part of the official TS.

Internal/external/hybrid and staffing requirements

The DOM TS was launched by the W3C DOM WG and NIST, and featured a public call for participation in the production of quality testing material for implementations of the DOM specifications. Staffing requirements were limited, so an expert was invited to the DOM WG in order to produce and edit the DOM TS Process Document. The DOM TS was in its initial phase very similar to a previous DOM Test Suite launched by NIST, which was used as a primer for the forthcoming work. However, the subsequentially released and forthcoming versions differ greatly given the group's work.

Charter considerations

Since the DOM charter did not include sections on producing test materials, this was included in the DOM TS Process Document [DOM-TS-PROC], which outlined the DOM TS deliverables in detail.

When TS is produced -- during or after specification phase?

The DOM TS was produced after specification for the first two levels (DOM Level 1 & 2) and is produced in parallel with specifying DOM Level 3. DOM Level 1 was released in late 1998, so it was not possible to synchronize the specification of the first level of the DOM with the Test Suite.

Starting with the current level of the DOM, the TS is produced in parallel with the specification. Tests are being produced to allow for implementors to check their implementations of DOM level 3 while contributing to relevant parts of the specification. Tests for the various modules of the DOM Level 3 specification are being written in parallel with the specification process.

Pointer to process documents

There is a separate DOM TS Process Document [DOM-TS-PROC], and also a Test Web page where you can find more information and pointers to primers, FAQ's and tutorials, maintained by the DOM WG representative to the DOM TS.

Test material storage, maintenance, tracking

The DOM TS uses W3C machinery to store test files, using CVS for easy tracking, and an issue tracking system at Sourceforge, in order to simplify issues and maintenance.

2.2. SVG

General overview

As used in this case study, "SVG" refers collectively to two consecutively chartered Working Groups. SVG1 refers to the originally chartered group (first meeting, Sept 1998) that produced SVG 1.0 (Recommendation, Sept 2001). SVG2 is the successor to SVG1, (re-)chartered in 2001, and charged with developing SVG 1.1 (modularization and profiles), and SVG 2.0. SVG 1.1 is presently at the stage of Candidate Recommendation (CR).

SVG 1.0 was released with a comprehensive Basic Effectivity (BE) test suite, that was developed entirely within the working group. The test suite (TS) comprises 127 test cases that span the functionality of SVG, exercising each major functional feature of SVG 1.0 at least lightly, but not probing any features in detail. SVG2 is adapting and subsetting the tests of the SVG1 TS for smaller and less capable mobile devices that are the target constituency of the Tiny and Basic profiles.

Internal/external/hybrid and staffing requirements

The SVG1 test suite was developed entirely by the working group. A pilot study sponsored by one of the members became the basis for design. A "TS Editor" (moderator) managed the project from inception to completion. All WG members are expected to participate in the writing and review of test cases. The SVG2 approach is identical -- a WG member is the TS moderator, and all WG members contribute.

Charter considerations

Both SVG1 & SVG2 charters specify a comprehensive test suite as a deliverable, and they coordinate the dates of first TS delivery (first publication) and the specification CR milestone. There is no further detailed breakdown of milestones or deliverables. This was worked out (for SVG1) as the specification development progressed.

When TS is produced -- during or after specification phase?

The test suites were (are) produced during the specification-writing phase. The target is substantial completion at the Candidate Recommendation (CR) milestone, so that the TS could be used to facilitate the interoperable-implementations W3C Process requirements of CR ([PROCESS], section 5.2.3)

Pointer to process documents

SVG has no QA process document per se, but much of the content that these operational checkpoints anticipate and require may be found in the extensive reference document, SVG Conformance Test Suite -- Test Builder's Manual [SVG-MAN]. Unless otherwise specified in that document, the normal WG processes of the SVG Working Group are used to conduct the TS work.

Test material storage, maintenance, tracking

The SVG WG keeps the SVG TS on a CVS server of one of the WG members. Revision tracking is therefore handled by CVS. Maintenance procedures and issue tracking for SVG TS are not separately defined, but rather SVG TS uses the normal SVG WG processes for these.

2.3. XML

General overview

Note. Following its transfer from outside W3C to the XML WG, when this case study was written, the XML TS was in preparation for first publication by the WG. Therefore a number of the referenced documents were not yet publicly visible.

The OASIS/ NIST XML Test Suite (TS) forms the basis for the W3C XML TS. As a result of a multi-way collaboration between

it was transferred to W3C where it is being augmented with new functionality, corrected to align with the XML 1.0 Second Edition Recommendation and its errata, and maintained.

Prior to accepting the transfer of the OASIS/NIST Test Suite, the XML Core WG assessed the level of effort necessary for TS development and maintenance and ensured that staff resources were interested and available to work on the TS. Additionally, the WG enlisted the assistance of NIST to lead the effort. The TS is being developed as a deliverable of the WG, following the WG's Process. There is a W3C hosted XML TS mailing list, Issues list, and W3C supplied storage (CVS) for the test-suite related files.

The XML Conformance Test Suite, v1.0, Second Edition contains over 2000 test files and an associated test report. The test report contains background information on conformance testing for XML as well as test descriptions for each of the test files included in the TS.

Internal/external/hybrid and staffing requirements

A commitment from NIST to take editor-lead responsibility for the TS was obtained prior to the commencement of the work. Consequently, NIST joined the WG. Members of the XML Core WG contribute to the development and quality assessment of the TS. When the XML TS is released by the WG, the general public will be invited to review the tests as well as participate in contributing tests.

Charter considerations

The W3C XML Core WG charter does not include sections on producing test materials. A new charter provides for the possibility of working on the OASIS/NIST XML Test Suite. The WG does not have a separate QA Process, but follows the WG Process for the development, assessment and maintenance of the XML TS. This is not explicitly documented in a separate QA process document, but the individual decisions are on record in WG mail archives and minutes as part of the normal WG process.

When TS is produced -- during or after specification phase?

The work on the OASIS/NIST XML Test Suite was conducted after the XML 1.0 Recommendation was released. A revised edition of the OASIS/NIST Test Suite 1.0 (Second Edition) Test Suite was released to complement the W3C XML 1.0 Second Edition Recommendation (October 6, 2000). Within the XML Core WG, the XML TS is being updated to reflect the current Recommendation and all applicable Errata.

Pointer to process document

There is no separate QA Process document. The XML Core WG charter uses its established WG Process to handle test development/assessment activities, communication means and test materials related functions.

Test material storage, maintenance, tracking

The XML WG uses the W3C CVS repository to maintain the XML TS. The XML TS issues are maintained utilizing an internal assessment process that consists of a running log of test issues, XML TS mailing lists, XML Core WG discussions and the XML Test Suite Issues document that keeps track of the running log of issues.

3. Operational guidelines in action

This section looks at each of the guidelines and checkpoints in the operational guidelines [QAF-OPS ]. For each checkpoint, there are two sections of content:

The Case Studies sections look at how the checkpoints' requirements are reflected in three existing operational frameworks (SVG, DOM, and XML). Where appropriate, pointers to relevant documents and materials are provided.

The Examples & Techniques sections demonstrate how the checkpoints might be satisified. The Examples & Techniques sections are supported by two separate appendices to this document:

These two appendices are fill-in-the-blank templates that can be used directly by the WGs, to satisfy a number of the operational checkpoints.

Guideline 1. Integrate Quality Assurance into Working Group activities.

Case studies summaries

DOM discussion. In the DOM TS, the QA activities were integrated into Working Group activities by inviting a field expert to do the coordination between WG and TS Group. Coordination was necessary as the DOM WG QA activity was launched in coordination with an external party, and, furthermore, was to be produced externally.

SVG discussion. QA was integrated into the SVG WG activities by a Charter commitment to produce test suites, and subsequent involvement of most WG members in the writing, reviewing, and maintenance of the test materials. The charters committed, in the cases of both SVG1 and SVG2, that a basic test suite (i.e., Basic Effectivity) would exist at or before Candidate Recommendation. The details of deliverables, milestones, and publication were generally worked out by the WG as the specifications developed.

XML discussion. The QA activities are integrated into WG activities by using the WG Process. Although the activities are not reflected in the WG Charter, deliverables are identified, expected release dates determined, staff allocated, and there is a clear commitment to developing and maintaining the TS.

Checkpoints

Checkpoint 1.1. Commit to at least "QA level three". [Priority 1]

Case studies

SVG. The SVG1 charter (member-only) addressed the topic with Publish a comprehensive test suite to aid developers in achieving SVG conformance. Together with the milestones schedule, it would satisfy the TS portion of at least level 3, although the details of this commitment were not spelled out in the charter (or any other single location). The SVG2 charter (member-only) makes a similar commitment.
DOM. As the W3C QA framework did not exist when the DOM TS Process was written, several of the checkpoints below do not apply. In particular, no explicit commitments to QA related deliverables are made in the DOM charter. Any such commitments have been made in the DOM TS Process document or in email on the DOM TS mailing list (archives).
XML. The initial work on the test suite was done by OASIS/NIST, and it was transferred to the XML Core WG. Level 3 was already built into the TS by the OASIS/NIST creators. Because the (original) XML Charter far pre-dated these events, no commitment exists in the Charter. However the transfer was discussed -- and commitments to transfer and support the TS are on record -- on the archived XML mail list (member-only) and a teleconference, between W3C (XML WG and QAWG), NIST, and OASIS.

Examples & techniques

Techniques for charter-related QA requirements in this checkpoint are demonstrated in the charter-template appendix [CHART-TEMPL].

New Charter overview.
In the Charter, include a section or sections on QA and explicitly define the QA activities that the Working Group commits to perform. It is helpful to the reader to also include a high-level summary statement of quality commitment, in the Mission statement. Use the specification and test materials commitments of the QA commitment table as a guide, when choosing and expressing your commitment. In the charter-template appendix, see the Mission, Work Items, Success Criteria, and Deliverables sections.
Existing group overview.
There are subcases to consider:
- First, the WG has not produced any QA deliverables so far. This subcase is similar to the "new charter" case -- the WG needs to articulate and document a commitment -- except that the way in which the QA commitment is expressed differs. Some possible techniques (non-exhaustive) include:
  - amend the Charter;
  - discuss in a meeting or teleconference, and record consensus of commitment in the minutes;
  - produce a separate standalone document, that asserts WG consensus and describes the particular agreed activities;
  - publish on the WG's Web page.
  From the perspective of record-of-commitment, the first two are probably better than the last two.
- Second, the WG has already produced all of the required specification and test materials deliverables. In this case, documenting a commitment is moot. The WG could claim satisfaction of this checkpoint ("Yes") when completing its Operational Guidelines ICS, and include a note to the effect that all of the deliverables are already done.
- Third, the WG has already produce some, but not all of the required deliverables. In this case, apply the previous bullet to the done deliverables, and the first bullet to those that are not done.
In all cases, use the specification and test materials commitments of the QA commitment table as a guide, for those QA activities that must either be committed or already done.
New Charter specifics.
In the deliverables section of the Charter, state a commitment to QA level three, with a very brief summary of its particulars. The detailed content of the level-three commitment is:
- each of the specification milestone deliverables LC, CR, and PR, will be assessed against the QA Framework: Specification Guidelines ICS
- a set of test assertions to accompany each completed (Rec) specification, and a commitment (deliverable) for ample use cases and examples within each specification.
- test materials (e.g., test suite) to be produced for the Recommendation, WG commits that any test materials will be subjected to review procedures at least as rigorous as required by OpsGL checkpoints CP7.1, CP5.4, and CP8.1.
This is more detail than is appropriate for the Charter, but a brief summary is helpful. See the prototype level-3 commitment in the deliverables section of the Charter template.
Existing group specifics.
The specific details that an existing group addresses are the same as the specifics for the new-charter case. The difference is in how the specifics are addressed, as discussed above in the existing-group overview.

Checkpoint 1.2. Commit to at least "QA level five". [Priority 2]

Case studies

SVG. Both SVG1 and SVG2 could be considered to conform in fact to level 5 in the TS column, although the details of this commitment were not spelled out in the charter (or any other single location). The position in the first (spec testability) column is uncertain. It depends on the definition of "normative use cases" -- the SVG specifications have thorough coverage with examples.
DOM. No such plans were made for the DOM TS.
XML. As with the previous checkpoint -- level 5 was built into the TS prior to transfer, and the agreements and/or commitments are on record in mail archives.

Examples & techniques

Techniques for charter-related QA requirements in this checkpoint are demonstrated in the charter-template appendix [CHART-TEMPL].

See the techniques for QA level three. The techniques for QA level five will be the same except for the specific extent of the activities to which the WG commits.

New Charter overview.
See overview for level three, and observe those principles. In the charter-template appendix, see the Mission, Work Items, Success Criteria, and Deliverables sections.
Existing group overview.
See overview for level three -- the principles are the same at this level five.
New Charter specifics.
In the deliverables section of the Charter, state a commitment to QA level five, with a very brief summary of its particulars. The detailed content of the level-five commitment includes all level-three items plus:
- a test materials contribution and review procedure, to be implemented no later than Recommendation stage;
- a commitment to the effect of, "As an appendix or addendum to the specification Recommendation, a set of associated test assertions, conformant with the checkpoints of Guideline 14 of QA Framework: Specification Guidelines."
This is probably more detail than is appropriate for the Charter, but a brief summary is helpful. See the prototype level-5 commitment in the deliverables section of the Charter template.
Existing group specifics.
See specific details for level three. There will be no difference for level five, for existing groups, except for the enhanced deliverables list to be addressed, as indicated in the preceding item.

Checkpoint 1.3. Commit to at least "QA level seven". [Priority 3]

Case studies

SVG. Both SVG1 and SVG2 might be considered to conform in fact to level 7 in the TS column, although the details of this commitment are not spelled out in the charter (or any other single location). This is dependent on the definition of "complete". The SVG test suite is comprehensive in the sense that it has complete breadth-first lightweight coverage -- it is a Basic Effectivity test suite) -- but it lacks complete detailed or drill-down coverage.
DOM. No such plans were made for the DOM TS.
XML. An upgrade of the TS materials to level 7 is being done now. Per comments in preceding checkpoints, this is not documented in the charter.

Examples & techniques

Techniques for charter-related QA requirements in this checkpoint are demonstrated in the charter-template appendix [CHART-TEMPL].

See the techniques for QA level three and for QA level five. The techniques for QA level seven will be the same except for the specific extent of the activities to which the WG commits.

New Charter overview.
See overview for level three, and observe those principles. In the charter-template appendix, see the Mission, Work Items, Success Criteria, and Deliverables sections.
Existing group overview.
See overview for level three -- the principles are the same at this level seven.
New Charter specifics.
In the deliverables section of the Charter, state a commitment to QA level seven, with a very brief summary of its particulars. The detailed content of the level-seven commitment includes all level-three items plus all level-five items plus:
- a complete set of test materials no later than Recommendation ("complete" is defined in OpsGL);
- maintenance of test materials, including at least version/errata support and handling of reports about tests' correctness.
This is probably more detail than is appropriate for the Charter, but a brief summary is helpful. See the prototype level-7 commitment in the deliverables section of the Charter template.
Existing group specifics.
See specific details for level three and for level five. There will be no difference for level seven, for existing groups, except for the enhanced deliverables list to be addressed, as indicated in the preceding items.

Checkpoint 1.4. Enumerate QA deliverables and expected milestones. [Priority 1]

Case studies

SVG. Both the SVG1 and SVG2 charters specify delivery of a comprehensive test suite, and relate delivery to document milestones. There is no further detailed breakdown of milestones or deliverables. This was worked out (for SVG1) as the specification development progressed.
DOM. See the DOM TS Process document, where a timeframe is indicated.
XML. Not quite. A new XML WG Charter (member-only), in the approval process in April 2002, says ..may do work in related supporting areas such as: overseeing coordination with the OASIS/NIST XML Conformance Testing test suites. Deliverables, tentative schedules, commitments, and milestones were discussed and agreed in (member-only) meetings and mail lists.

Examples & techniques

Techniques for charter-related QA requirements in this checkpoint are demonstrated in the charter-template appendix [CHART-TEMPL].

New charter techniques.
If the WG has followed the techniques of the preceding three checkpoints, then this checkpoint is partially done already. In the deliverables section of the charter, there will either be a listing of or a pointer to the minimal deliverables necessary to meet the WG's chosen QA commitment level. The WG could finish the satisfaction of this checkpoint by adding:
- any other QA deliverables and/or intermediate milestones that are not part of the already-enumerated set;
- and, associate delivery dates with each deliverable and milestone.
It is a fine point, and of no substantive importance, whether some items -- e.g., TM requirements documents, TM-specification coverage maps, etc -- are listed as deliverables in their own right or milestones associated with other deliverables.
Existing group techniques.
A technique that would satisfy this checkpoint for existing WGs would be: document on the WG web page the QA deliverables with milestones and status. Any of the techniques listed under the above discussion of QA commitment level for existing WGs would also be suitable.

The satisfaction of this checkpoint is demonstrated in the charter-template appendix, specifically the sections about WG deliverables and deliverables milestones.

Checkpoint 1.5. Associate QA criteria with WG milestones. [Priority 2]

Case studies

SVG. The SVG1 charter (member-only) scheduled a BE TS and Candidate Recommendation (CR) for the same date. The SVG2 charter (member-only) commits to a BE TS date that is earlier than the date for CR.
DOM. See the DOM TS Process document, where a connection to WG deliverables is indicated. Please note that the QA deliverables statement is valid only for DOM Level 3, as the previous two levels had already been specified (in part) before the DOM WG decided to provide a DOM TS.
XML. Not quite, see comments in previous checkpoint.

Examples & techniques

Techniques for charter-related QA requirements in this checkpoint are demonstrated in the charter-template appendix [CHART-TEMPL].

New Charter techniques. In the charter, minimally state that:
- Completed test materials will be used as a criterion for PR entrance of the WG's specification(s).
- Completed test materials will be used as the measure to show two correct interoperable implementations during CR phase.
The charter can optionally also link other QA deliverables. For example, it could link a test materials design document to the first published WD of the WG's specification(s).
Existing group techniques. The dependencies that existing working groups would commit to are the same as the new Charter case, but the technique for documenting them would differ. Any of the techniques listed under the above discussion of QA commitment level for existing WGs would be suitable.

The satisfaction of this checkpoint is demonstrated in the charter-template appendix, in the milestones-criteria section.

Guideline 2. Define resources for Working Group QA activities.

Case studies summaries

DOM discussion. The DOM WG appointed an invited expert to monitor and coordinate the work of the DOM TS. Furthermore, the members of the DOM WG have looked into producing tests fot that part of the specification which is of particular interest to their implementations.

SVG discussion. The SVG1 and SVG2 charters explicitly charge the WG with producing test materials, but do not further address specific staffing. In fact, the expectation has evolved that all WG members should contribute to the building of the TS. In both cases, a "TS Editor" has been assigned to lead and coordinate the TS production.

XML discussion. The WG appointed NIST to lead the TS effort and coordinate the work of the XML TS. Furthermore, members of the WG have reviewed and assessed the current set of tests, corrected tests, and contributed tests.

Checkpoints

Checkpoint 2.1. Address where and how conformance test materials will be produced. [Priority 1]

Case studies

SVG. In both SVG1 and SVG2, the test suites were defined as WG deliverables -- it was explicit that the WG was to produce the TS, and explicit that completion of a basic test suite was to occur at or before CR.
DOM. This information is given in the DOM TS Process document and not in the DOM WG charter. Please see relevant parts of the DOM TS Process.
XML. The new XML Charter (member-only, in process) indicates that the WG scope may include, ..overseeing coordination with the OASIS/NIST XML Conformance Testing test suites. The details of the "where and how" are in meeting minutes and mail archives, as described in an earlier checkpoint.

Examples & techniques

Techniques for charter-related QA requirements in this checkpoint are demonstrated in the charter-template appendix [CHART-TEMPL].

There are several possibilities for producing conformance test materials:

build them entirely within W3C (normally, the Working Group). Example: SVG test suite.
build them externally and transfer to W3C. Example: XML test suite.
build procedures for assembling contributions from multiple sources. Example: XSLT test suite (a work in progress, this is actually being done in OASIS (Organization for Structured Information Standards) XSLT/Xpath Conformance Technical Committee, but it could just as well work within a W3C Working Group).
hybrids of the above.

There are good reasons why it is useful for external parties to have a strong participation during the building of test materials. But even if a test suite is being developed outside of W3C, the Working Group needs to ensure it is completed according to the QA level to which the Working Group is committed in its charter.

The satisfaction of this checkpoint is demonstrated in the charter-template appendix, in the participation section.

Checkpoint 2.2. Address QA staffing commitments. [Priority 1]

Case studies

SVG. While the charters do not explicitly address staffing commitments (nor do any other standing documents), they do charge the WG with producing the TS. In fact, all WG members are expected to participate in the TS work. This was explicitly documented in the minutes of a SVG2 face-to-face (member-only).
DOM. There are no QA staffing commitments made, but the way the TS is to be produced and coordinated is outlined in the DOM TS Process.
XML. QA staffing commitments are not addressed in the Charter, and no specific staffing levels are documented or committed to elsewhere. Staffing resource was addressed and agreed in multi-way discussions between W3C (XML and ), NIST, and OASIS prior to the TS transfer to W3C, and accordingly should be on record in the XML mail archives. Several WG members do most of the TS work, but the whole WG participates in the sense that TS topics are discussed as a regular part of WG meetings and teleconferences.

Examples & techniques

Techniques for charter-related QA requirements in this checkpoint are demonstrated in the charter-template appendix [CHART-TEMPL].

By now, the WG (or the writers of its Charter) have already addressed the overall QA commitment level (QA level three, five, seven, or other), and the basic scenario for production of test materials and other QA deliverables (intra-WG, external, or hybrid). These will determine the total amount of WG staff resources that will be needed for the commitment and plan. To satisfy this checkpoint, the Working Group can:

produce and document an estimate of the total staff resources to meet the commitment and plan;
address the question of whether all WG members are expected to participate in WG work, or whether instead the resources will come from a dedicated task team.

The satisfaction of this checkpoint is demonstrated in the charter-template appendix, in the participation section.

Checkpoint 2.3. Request allocation of QA resources to the Working Group. [Priority 1]

Case studies

SVG. QA resources were not specifically mentioned in the Activity Proposal/Call for Participation.
DOM. Not applicable.
XML. No Call for Participation was done. The participation happened when the TS was transferred from OASIS to W3C.

Examples & techniques

(Under construction)

It is not unusual that responders to a WG Call for Participation have interest limited to the "technical invention" side of the WG's work -- designing the specific technical details of the standard. By emphasizing in its participation calls that QA specialists are needed and welcome, the WG improves its chances of getting interested and competent people to work on the QA commitments and plans.

A related topic has arisen in email discussions: whether, for QA-specific work, member organizations can add more participants to a WG's roster than the WG's rules normally allow. I.e., some WGs restrict the number of participants from a company to, for example, two. One proposed technique for staffing a WG's QA plans and commitments would be to allow additional members, but without additional voting rights.

(@@TBD -- prototype language that can be inserted into a CfP).

Guideline 3. Synchronize QA activities with the specification milestones.

Case studies summaries

DOM discussion. The milestones were given in the DOM TS Process document and were monitored by the DOM WG in order to assure the satisfactory continuation of the DOM TS group work. In those cases where the TS Group needed more time to achieve its goals, the DOM WG representative reported to the WG in order to appeal for an extension.

Applicable in so far as the TS releases are (for levels currently being specified) synchronized with the relevant modules of the DOM specifications.

SVG discussion. The SVG1 and SVG2 charters establish the basic synchronization, which is that the first publications of basic test suites (TS) should be at or before the beginning of CR stage. Beyond that, there are no explicit (i.e., written down) synchronizations. This happens in fact as part of the WG process and the TS building. TS publications are synchronized with selected late-stage document versions (CR, PR, Rec). There is no explicit scheme for labelling/linking test cases according to errata, although this would be reconstructable from the CVS repository for the TS.

XML discussion. The TS work is being conducted in close coordination with the WG's other deliverables. The TS is being updated and augmented to reflect the current work of the WG, including resolved issues related to the Recommendation and published Errata. Tests for new functional areas are also being explored as the deliverables for those Recommendations progress (i.e., Xinclude and Namespace).

Checkpoints

Checkpoint 3.2. Support specification versioning/errata in the QA deliverables. [Priority 3]

Case studies

SVG. Although the TS is de facto maintained to keep current with the specification, there is no explicit linkage mechanism between test cases and errata. CVS tagging enables maintaining linkage of specification versions with appropriate TCs, but there is no association or declaration mechanism in the TS data itself. There is implicit linkage with specification versions (1.0, 1.1, ...) via the SVG DOCTYPE declaration and external DTD reference in each test case (TC).
DOM. The DOM TS keeps its source files in W3C CVS, therefore versioning of source files is simplified. Also, there are guidelines on versioning of TS in the DOM TS Process document.
XML. The Test Suite has versions, and individual test cases are versioned by virtue of residing in CVS. However there is no explicit TS design feature for associating particular test case versions or instances with specification (XML Recommendation) versions or errata.

Examples & techniques

(Under construction)

Guideline 4. Define the QA process.

Case studies summaries

SVG discussion. SVG supports the process requirements of a QA moderator (manager of TS development) and a task force (it is the whole WG, by explicit choice). TS-related communication is handled internally on the whole-WG list (member-only), and externally via a public TS comment list. SVG does not have a Process document, per se, that spells out these details. It does have a TS manual that defines the TS framework in some detail.

Checkpoints

Checkpoint 4.1. Appoint a QA moderator. [Priority 2]

Case studies

SVG. Both SVG1 and SVG2 TS projects have a moderator (referred to as "TS editor" in SVG1).
DOM. An invited expert to the DOM WG was appointed contact between the DOM WG and the DOM TS group. He also edited the DOM TS Process document. Relevant announcements were made on the public DOM mailing list by the DOM WG chair.
XML. A moderator from NIST was appointed as part of the TS transfer agreements and process.

Examples & techniques

Depending on the plan for the origin of the test suite a Working Group can:

appoint a test moderator from the Working Group members (and other W3C members);
W3C QA staff;
invite moderator from external organization.

When a test suite is developed outside of the Working Group, a moderator often already exists in the external group that is the source of the test suite. It is recommended to invite the moderator to become a Working Group member, since he/she needs access to all Working Group materials and resources. See also section on transferring test suite from outside W3C.

Checkpoint 4.4. Specify means for QA-related communication. [Priority 2]

Case studies

SVG. There is no special plan or mechanism -- the whole-WG mail list is used (w3c-svg-wg@w3.org) for TS topics, as for all WG business. There is an external, public email list for accepting comments on the TS. There is a TS home page.
DOM. Intra-WG communication in connection to the DOM TS is done using the member-only DOM WG mailing list. Public communication is defined in the DOM TS Process, and is done using the DOM TS mailing list (archives)
XML. The XML TS has a publicly archived mailing list, public-xml-testsuite@w3.org. The XML Core WG page (member-only) contains some status and pointers for the TS work.

Examples & techniques

Techniques to satisfy this checkpoint are demonstrated in the QAPD-template appendix[QAPD-TEMPL], which can be used directly by Working Groups to produce a QA Process Document (QAPD). This checkpoint, specifically, is demonstrated in the "QA-related Communication" section.

Checkpoint 4.5. Define the QA framework for test materials development. [Priority 2]

Case studies

SVG. The SVG TS Manual is intended to define the framework in sufficient detail that new contributors can understand, build, and submit test cases.
DOM. Most of the TS material primer development had taken place in a DOM TS that NIST had previously launched; any additions to this were discussed and decided on in the DOM TS Group mailing list (archives). The framework was radically altered, any decisions taken were coordinated with the DOM WG.
XML. The test documentation (in CVS, but not yet published) contains description of the framework and test development.

Examples & techniques

Guideline 5. Plan test materials development.

Case studies summaries

DOM discussion. As the DOM TS drew on experiences made by an external party (NIST), the work initially drew heavily on those experiences. Where the subsequent work indicated digression, the future version of the DOM TS was branched off and formed the official W3C DOM TS. The test material planning was made by the DOM WG representative to the DOM TS in coordination with NIST and the public, after the DOM TS had been officially launched.

Several design issues were discussed by the DOM TS Group itself, then brought to the DOM WG for acceptance. Such issues include the automatic generation of the schema language for the DOM TS, using the same test description file to generate the official bindings (and the unofficial ones using a different transform) as well as using particular build tools to build working and official releases of the DOM TS.

Most discussion was done on the DOM Group mailing list and coordinated with the DOM WG. More information can be found in the DOM TS page, which serves as a complete reference for DOM TS related information.

SVG discussion. The planning for SVG's test materials development started with a study and prototype design, sponsored by one of the SVG WG members. This work was brought into the WG as the starting point of the TS development, and was further developed, refined, and applied by the moderator ("TS Editor"). Both user documentation (for TS releases) and contributors documentation were developed. The latter described the TS framework in some detail, and gave both technical criteria for the contributed test cases plus review/acceptance criteria. The actually processes and procedures for contribution and review were not explicitly defined, as all materials came from WG members (and therefore followed WG process).

XML discussion. The XML TS drew on the experiences and expertise of NIST, the original developer of the test suite framework. The original work was revised to included test requirements with a mapping to the test cases and to the specification. The planning of improvements and augmentation to the original work was discussed and decided by the WG. Techniques and methods were borrowed from other test suite development efforts, such as DOM and XSL-FO.

Checkpoints

Checkpoint 5.1. Ensure test materials are documented and usable for their intended purposes. [Priority 2]

Case studies

SVG. SVG1 published good release notes with TS releases, as well as the Manual. The TS home page describes and points to all the materials.
DOM. All of the tests used in the DOM TS are validated using a DOM TS ML schema generated directly form the relevant DOM specification using XLST transforms that are part of the DOM TS framework. In addition, each test is required to contain information in prose on intended testing purpose and part of specification being tested. For more information, see the DOM TS ML document.
XML. The "TS Report" document (in CVS, not yet published) contains documentation including a matrix of test requirements and a description for each test (point to the CVS). Each test is required to contain information in prose on the intended testing purpose and part of the specification being tested

Examples & techniques

(under construction)

Checkpoint 5.3. Address license terms for submitted test materials. [Priority 2]

Case studies

SVG. This has not been directly addressed by the SVG TS project. However, because all materials have so far come from WG members, in fact it is somewhat covered by the WG patent and IP policies.
DOM. Defined in the DOM TS Process document, with a pointer to the relevant grant of license wording. This wording is also sent to any party submitting tests using the required mailing mechanism for submitting tests, as outlined in the DOM TS Process document.
XML. There is no QA Process document per se. It is unclear whether the topic has been explicitly dealt with.

Examples & techniques

Techniques to satisfy this checkpoint are demonstrated in the QAPD-template appendix [QAPD-TEMPL], which can be used directly by Working Groups to produce a QA Process Document (QAPD). This checkpoint, specifically, is demonstrated in the Licenses section.

[@@Ed note. Add more examples to Extech per KG email of 20021218.]

Checkpoint 5.4. Define review procedures for submitted test materials. [Priority 2]

Case studies

SVG. Chapter 6 of the Manual gives review criteria for all tests. But there are no procedures, just criteria. The criteria address accuracy, scope, correctness, clarity, and other aspects. Additionally, a checklist has been generated, for use in reviewing test cases.
DOM. Defined in the DOM TS Process document.
XML. There is no QA Process document per se. All tests are reviewed by the WG members, who run the tests. The review process and criteria are not defined in any TS documents, but the review criteria are, de facto, what is applied during test review.

Examples & techniques

The following example meets the review requirements:

Each contribution should be reviewed for its eligibility by at least one vendor other than the submitter. Any vendor involved in the creation of the Working Group test collection other than the submitter may review contributed tests.
Eligibility of tests should be judged by:
- The accuracy of the test. The accuracy of a test is determined by a judgment of the reviewer whether the test case actually tests what the submitter states that the test case tests. The accuracy is the extent to which the test matches the cited parts of the specification. If it does not match, or only partially matches, the test should be considered inaccurate.
- The scope of the test. Test should target MUST provisions of the specification and not MAY/SHOULD provisions. The test should target only conformance requirements. Data values limits, memory or performance limits should not be targets to the test unless those are specified in the conformance requirements.
- The clarity of the test. The clarity of a test is a determination of whether the aspect being tested is clearly described with the anticipated results acceptably explained.
- The clarity of aspect of the Specification being tested. The test suite aims to test parts of the Specification and errata that are not vague.
After review, results should be sent back to the submitter for feedback.
In case of disagreement, a request for clarification should go to Working Group. Tests in question should be left out of the test suite until the W3C Working Group clarifies the issue.

Only those tests that are agreed to be correct by reviewers and the submitter, or that are approved by W3C Working Group clarification, should be published.

Techniques to satisfy this checkpoint are also demonstrated in the QAPD-template appendix [QAPD-TEMPL], which can be used directly by Working Groups to produce a QA Process Document (QAPD). This checkpoint, specifically, is demonstrated in the contribution receipt and review section.

Guideline 6. Plan test materials publication.

Case studies summaries

DOM discussion. Plans for test material publication are discussed internally in the DOM TS group, and then decided by the DOM TS Group in coordination with the DOM WG.

SVG discussion. The SVG TS project has explicitly discussed and planned the key aspects of TS publication. CVS repository is used for the ongoing development of the TS, and at agreed milestones, a snapshot is taken and published under the W3C Document License. The TS home page links all of the materials and documentation, and also points to an implementation matrix that publishes the TS results of a half-dozen or so implementations.

XML discussion. Publication of the TS is being discussed by the WG. The TS will reside on the W3C web space and will be freely and publicly available. In general, the publication details of the XML TS are relatively invisible outside of the WG, because there has not been any publication yet.

Checkpoints

Guideline 7. Plan the transfer of test materials to W3C if needed.

Case studies summaries

DOM discussion. This issue was addressed early on in the specification phase of the DOM TS work, as a choice had to be made between integrating an existing TS or just using the existing one as a primer for its succesor. The latter was preferred.

SVG discussion. This guideline is not applicable to the SVG TS -- all TS development so far has occurred within the SVG WG.

XML discussion. The issue was addressed prior to the transfer of the TS to XML WG, and prior to the start of TS development and maintenance there. Since that phase is finished, it is no longer applicable. Included in the discussions were the quality of the OASIS/NIST XML Test Suite, amount of work needed to correct and improve it, as well as IPR questions.

Checkpoints

Checkpoint 7.3. For any transferred test materials, resolve all IPR issues with the external party that produced the test materials. [Priority 1]

Case studies

SVG. This and the other checkpoints are not applicable -- all TS development took place within the SVG WG.
DOM. Not applicable, the DOM TS was launched in coordination between the W3C and NIST, NIST donated the first bulk of tests.
XML. This was done in discussions involving OASIS, NIST, W3C, and is accordingly on record in the XML WG mail archive.

Examples & techniques

The following procedure is an example that meets all of the transfer requirements.

Legend for the procedure:

EG: the external group or entity
QAWG: the QA Working Group
TM: the test materials
WG: the subject Working Group

Before the transfer, WG with the help of QAWG:

performs an assessment of the quality of candidate TM (by WG, QAWG)
identifies and commits to derive a set of QA deliverables from the candidate TM. These can be: releases, appeals/challenges, maintenance, submissions, Web site, mail list, etc. (by WG)
identifies staff resources to meet the requirements, including at least a TM moderator. It is recommended to invite the TM moderator from the EG (if one exists) to become a WG member should the transfer happen. (by WG)
takes the decision to seek/accept the transfer. (by WG)
initiates charter revisions for QA commitments (by WG, QAWG may consult)

During transfer:

EG and W3C reach agreement to transfer the TM (by WG, QAWG, EG)
WG and EG perform the actual transfer of the TM, WG creates an initial repository (by WG, EG)

After transfer, initial process setup:

WG appoints a TM moderator. The TM moderator creates a QA Process Document for WG (by WG, TM moderator, QAWG may consult)
setup the TM home page, a TM issues mailing list (by WG, TM moderator)
determine the appropriate W3C IPR license (by WG, QAWG)

First W3C public release of the new TM:

make any necessary enhancements prior to the first public release: fix known/reported errors, produce documentation (by WG)
announce the first public release of TM (by TM moderator, Communications Group)
joint W3C/EG press release (by WG, QAWG, Communications Group, EG)

After the first public release, the TM enter the maintenance phase which is described below.

Guideline 8. Plan for test materials maintenance.

Case studies summaries

DOM discussion. No clear plan for test material maintenance has been made, other than indicating in the process document that these issues will be adressed by the DOM TS group.

SVG discussion. SVG has no explicit plan for TS maintenance. Because the TS process is basically the WG process, maintenance happens happens as a natural part of the WG's specification development and ongoing TS development.

XML discussion. No clear plan for test material maintenance has been made yet. The need to address this has been acknowledged by the WG, with plans to address it in the near future.

Checkpoints

Checkpoint 8.2. Specify a test materials update procedure to track new specification versions/errata. [Priority 2]

Case studies

SVG. SVG has no QA Process document per se, nor is this explicitly described in any of the WG or TS standing documents. It is informally handled as part of normal WG process concurrent with the evolution of specification versions and the processing of specification errata.
DOM. Relevant wording in the DOM TS Process document.
XML. XML has no QA Process document per se, nor is this documented in WG standing documents or archives. It will de facto be handled as part of the WG process.

Examples & techniques

The following procedure is an example that meets the requirements:

Initiating test suite change/update. As soon as errata are accepted or a new version/edition of the Recommendation is published, the errata task force (or test suite maintenance group) identifies affected test cases and initiates updates to the test suite by adding entries to the tracking system with status "ACTIVE"
Providing test suite changes for review. Preferably, the test case submitter or initiator of the change provides updated versions for the affected tests. If no changes are thus provided, then the test maintenance group finds a way to provide it. Once changes are ready, maintenance group changes the corresponding status in the tracking system to "RESOLVED"

Reviewing/accepting the change. Test changes go through the same tests review process as presented earlier. If the test changes are accepted, they are added to the test suite with an indicator of errata/version. Old tests stay in the test suite marked with the last errata/version level they comply with. The status changes to "CLOSED". If tests are rejected, they go back to the provider of the change with that determination, and the status is reverted back to "ACTIVE"

Techniques to satisfy this checkpoint are also demonstrated in the QAPD-template appendix[QAPD-TEMPL], which can be used directly by Working Groups to produce a QA Process Document (QAPD). This checkpoint, specifically, is demonstrated in the test materials maintenance section.

Checkpoint 8.3. Identify a procedure for test validity appeals. [Priority 2]

Case studies

SVG. SVG has no QA Process document per se, but disputes of a tests validity are handled as part of the normal WG consensus process. There is a publicly archived mail list, svg-testsuite-comments@w3.org, for public comment on the tests.
DOM. Relevant wording in the DOM TS Process document.
XML. XML has no QA Process document per se, nor is this documented in WG standing documents or archives. Any disputes, disagreements are brought to the attention of the WG and put on the TS Issues List (member-only) for discussion.

Examples & techniques

The following topical outline is an example that meets the requirements, i.e., if these topics were addressed in the process document::

initiating an appeal
review of the appeal
providing change to the test suite for review, in case of appeal acceptance
review and acceptance of the change

If there is still active work on the test suite even if the Working Group is not re-chartered to handle the work, then it is up to the Director to determine how this work should be done. Some options include:

it may be assigned to a Coordination Group;
it may be assigned to a related Working Group;
the Team could take care of minor items.

Techniques to satisfy this checkpoint are also demonstrated in the QAPD-template appendix[QAPD-TEMPL], which can be used directly by Working Groups to produce a QA Process Document (QAPD). This checkpoint, specifically, is demonstrated in the test-validity review and the test materials maintenance sections.

4. Conformance

There is no concept of conformance to this document. Rather, conformance is measured relative to the checkpoints of the companion document, QA Framework: Operational Guidelines [QAF-OPS]. This document relates real-world examples to the checkpoints' requirements, and also presents techniques by which the checkpoints may be satisfied. In that sense, it defines conformance criteria for the conformance requirements (checkpoints) of the operational guidelines document.

5. Acknowledgments

The following QA Working Group and Interest Group participants have contributed significantly to the content of this document:

6. References

7. Change history

2002-11-21, (initial) companion to 20021121 Operational Guidelines (OpsGL) publication

Revised to track changes in corresponding OpsGL draft. See "Change history" of that corresponding Operational Guidelines draft.

QA Framework: Operational Examples & Techniques

W3C Note 17 February 2003

Abstract

Status of this document

Table of contents

1. Introduction

1.1. Scope

1.2. Overview of contents

Case studies

Examples & techniques

1.3. Understanding and using this document

2. Conformance test suite case studies

2.1. DOM

General overview

Internal/external/hybrid and staffing requirements

Charter considerations

When TS is produced -- during or after specification phase?

Pointer to process documents

Test material storage, maintenance, tracking

2.2. SVG

General overview

Internal/external/hybrid and staffing requirements

Charter considerations

When TS is produced -- during or after specification phase?

Pointer to process documents

Test material storage, maintenance, tracking

2.3. XML

General overview

Internal/external/hybrid and staffing requirements

Charter considerations

When TS is produced -- during or after specification phase?

Pointer to process document

Test material storage, maintenance, tracking

3. Operational guidelines in action

Guideline 1. Integrate Quality Assurance into Working Group activities.

Case studies summaries

Checkpoints

Checkpoint 1.1. Commit to at least "QA level three". [Priority 1]

Case studies

Examples & techniques

Checkpoint 1.2. Commit to at least "QA level five". [Priority 2]

Case studies

Examples & techniques

Checkpoint 1.3. Commit to at least "QA level seven". [Priority 3]

Case studies

Examples & techniques

Checkpoint 1.4. Enumerate QA deliverables and expected milestones. [Priority 1]

Case studies

Examples & techniques

Checkpoint 1.5. Associate QA criteria with WG milestones. [Priority 2]

Case studies

Examples & techniques

Guideline 2. Define resources for Working Group QA activities.

Case studies summaries

Checkpoints

Checkpoint 2.1. Address where and how conformance test materials will be produced. [Priority 1]

Case studies

Examples & techniques

Checkpoint 2.2. Address QA staffing commitments. [Priority 1]

Case studies

Examples & techniques

Checkpoint 2.3. Request allocation of QA resources to the Working Group. [Priority 1]

Case studies

Examples & techniques

Guideline 3. Synchronize QA activities with the specification milestones.

Case studies summaries

Checkpoints

Checkpoint 3.1. Synchronize the publication of QA deliverables and the specification's drafts. [Priority 2]

Case studies

Examples & techniques

Checkpoint 3.2. Support specification versioning/errata in the QA deliverables. [Priority 3]

Case studies

Examples & techniques

Guideline 4. Define the QA process.

Case studies summaries

Checkpoints

Checkpoint 4.1. Appoint a QA moderator. [Priority 2]

Case studies

Examples & techniques

Checkpoint 4.2. Appoint a QA task force. [Priority 2]