W3C

QA Process Document template for "QA Framework: Operational Guidelines"

This version:
http://www.w3.org/QA/WG/2003/02/OpsET-qapd
This document is an appendix to:
http://www.w3.org/QA/WG/2003/02/qaframe-ops-extech-20030217
Latest version of "QA Framework: Operational Guidelines":
http://www.w3.org/TR/qaframe-ops/
Latest version of "QA Framework: Operational Examples & Techniques":
http://www.w3.org/QA/WG/qaframe-ops-extech
Editors:
Lynne Rosenthal (lsr@nist.gov)
Lofton Henderson (lofton@rockynet.com)
Contributors:
See Acknowledgments in QA Framework: Operational Guidelines.

About this QA Process Document (QAPD) template

This is an Appendix to QA Framework: Operational Examples & Techniques. It contains a template for a Working Group QA Process Document (QAPD). It is complete in outline form -- all necessary sections and topics, as required by the QA Framework: Operational Guidelines (OpsGL), are represented.

This template should help implementers of QA Framework: Operational Guidelines (OpsGL) to satisfy OpsGL's checkpoints that require a QAPD or place requirements on its content -- start with this template and fill in the designated QA-related information. This will produce a complete QAPD that satisfies the OpsGL requirements.

This template contains prototype statements and content as required of a QAPD. The prototype statements contain fill-in-the-blank sections, which appear like this, "[specification name]". A WG would supply the appropriate information here to complete the prototype statement and make it an actual Charter statement.

Each subsection relates to a particular Checkpoint of OpsGL. The appropriate back reference and link immediately follows the subsection title, and looks like this:

[For OpsGL checkpoint CPx.y]

These may be deleted from the final QAPD text that the WG produces..

The following are some fill-in-the-blank items that are commonly used or repeated in this template::

The QA Process Document (QAPD) template follows this line.


[WG Name] QA Process Document

DATE

This version:
[http://...]
Latest version:
[http://...]
Previous version:
[http://...]
Editor:
[enter name(s)]

Status

This is the [DATE] QA Test Material Process Document.

Comments on this document are invited and are to be sent to the [name of mail list] mailing list [address of list]. An archive is available at [address of archive].

This document has been produced as part of the W3C [WG Name]. The authors of this document are the [WG Name] WG members.

Table of contents

Introduction

This document describes the QA process for ensuring that a comprehensive conformance [test materials - choice of: test suite | validator tools | tests material | other(specify)] is available for the[Name-of-specification) | W3C-[WG Name]-specifications)]. It provides information about the WG's quality-related logistical and communications setup, contribution mechanism, licensing policies, publication and maintenance plans, as well as any other critical processes and operational details needed to produce and maintain the [test material type]. It supplements the W3C Process document.

The [test materials type] produced by the W3C [WG name] will be known as [name and acronym of test materials].

Requirements

  1. The [test material type] should be developed in a public framework. The development of this framework itself is a central area of interest, in addition to the [test material name] in general. The [WG name] welcomes the participation of interested parties in developing the [test materials type].
  2. A standard approach to [test material type] IPR which meets the needs of both contributors and users should be established.
  3. The [test material name] is intended to be used as a tool to aid implementors in developing [specification name] implementations. Validation and certification of these implementations are outside the scope of this work. The [test materials type] will be provided for information and help only. However, we intend to produce as comprehensive, functional, and general tests as possible, and this should be the overall goal in designing and implementing the [test material name].
  4. The [test material type] should be available in a common environment so they can be accessed publicly and shared among developers and users.
  5. The [test material type] will be hosted on the W3C site once developed. It may also after development be mirrored at various sites in order to simplify access and enhance availability to the community.

Parties involved

The [test material name] will be produced in a public framework with contributions from developers and companies in the community. The [test material name] will be coordinated and supervised by the [WG Name and/or other names]. The representatives will use resources from their organisations as well as invite individuals and companies (through representatives) to allow for a larger number of people to get involved.

QA-related Communication

[For OpsGL checkpoint CP4.4]

[Choose one: a) if you use the WGs established list or b) if you set up a dedicated list]

a)The main channel of communication for the [test material name] will be the existing [WG name] mailing list [address]. An archive is available at [address]

b)The main channel of communication for the [test material name] will be a dedicated mailing list, called [test material name] mailing list [address]. An archive is available at [address].

The public web page for QA information is named /Test/ and is available at: [http://www.w3.org/WGnameTest/]. This page includes links to this process document, the [test material name]and information about the test suite including:

DevelopmentFramework

[For OpsGL checkpoint CP4.5]

The [test material type]; will be produced in a public framework with contributions from members of the [WG name], W3C, or the community at large. The framework will provide a stable mechanism for submission of test materials, saving information about those tests, version handling, use of the tests, and so forth.

Test material contributions

[For OpsGL checkpoint CP5.2]

Choose one:

a)Tests or series of tests should be submitted to the [test material name]via the [WG name or test material name @@ this is a or b from above] mailing list: [address].

b)Tests or series of tests should be submitted to the [test material name] test submission mailing list: name-testsuite-submit@w3.org. An archive is available at: [address]

In order to simplify for individuals and companies the production of the test material and, if they so wish, contribute with tests of their own, we propose the following process for submitting tests:

  1. The test or series of tests, get submitted to the framework. Submitter should also send a notification to the mailing list that will be set up for the [test material name] to indicate that he/she has submitted a test to the [test material name] framework.
  2. It is acknowledged (presumably by the moderators/reviewers of the test materials) that the test has been received. This should be done via email directly to the submitter, and a copy should be sent to the mailing list. The test receives the "Received"; status.
  3. The test together with the necessary documentation is saved for future reference.

Receipt and review of test contributions

[For OpsGL checkpoint CP5.4]

The test suite's moderators evaluate the test materials included in a contribution according tot he following criteria.

  1. Relevance for the particular implementation of [technology name]. It can here be decided that the particular test is irrelevant to the [test material name] since it tests a feature that is not in the specification.
  2. Part of the specification being tested. If there are other, previously submitted tests that are functionally equivalent and test the same part of the specification, this test should either be kept for archive purposes but not published or should replace the previously submitted test, in which case the previously submitted test would be archived.
  3. Quality of code and documentation. The tests must be well-formed XML; the documentation must be in accordance with [a) the XML file in which specification, functionality and possible scenarios are documented or b) the schema for test submissions, avialable at the [test material name] website, which outlines such things as the part of the specification under test, functionality of test, location of test, and so on].
  4. Scenarios that underly the particular test layout and its intended scope.
  5. Development guide for the [test material name], once developed. The existence of these guidelines will be communicated through the [test material name] mailing list.

If a given test judged suitable for inclusin in the [test material type] according to these criteria, the test is added to the [test material type]/span> with status "Accepted".

Tests that are judged to be inappropriate for publication arereurned to the contributor. Such tests are archived and not included in the [test material type].

Test status and review procedures

[For OpsGL checkpoint CP8.3 (?)]

Once a test is included into the [test material name], it becomes subject to the dispute and review process outlined below.

At any time, any member of the community may dispute the validity of any test in the [test material name]. This is dome by raising the issue on the [test material name] mailing list. If the dispute is felt to be valid either by the community and/or the [test material name] Task Force, the status of the test is changed to reflect the fact that its validity has been disputed, and the test is passed on to the WG for detailed review, the results of which are outlined below.

  1. Stable. The test and its accompanying documentation have undergone review by the WG and have been found to be free of errors and relevant to the specification.
  2. Disputed: test error. The validity of the test has been questions by one or more members of the community. A consensus exists in the community or the [test material name] Task Force agrees (or both) that the test is in error. The [test material name] Task Force has submitted the test and the associated dispute to the WG for detailed review, along with a suggested resolution.
  3. Disputed: spec error. As a result of running the test, one or more members of the community has questioned the correctness, consistency, clarity, etc. of the part(s) of the specification addressed by the test. A consensus exists in the community, or the [test material name] Task Force aggress (or both) that a problem exists with the specification. The [test material name] Task Force has submitted the test and the associated dispute to the WG's erratum process.

Documentation

Each contribution to the [test material name] must be fully documented, [@@ if there is a schema (indicated in subsection:Recept and Review of Test Contributions #3) add:] and conform to the schema for test submissions, which is available at the [test material name] website. The documentation provides such information as:

Status of the test suite according to the above

  1. The [test material name] should consist only of submitted, reviewed and stable tests.
  2. The [test suite name] for all levels will also consist of documentation on the tests it contains, scenarios that have served as a base for them, commented code, and possibly comments from the editors.

It is proposed that the W3C [WG name] WG representative act as moderator and controller for incoming tests. The [test material name] moderator is chosen by the [WG name] WG. All tests should be kept for archive purposes, whether they get published or not.

Conformance

[For OpsGL checkpoints CP4.6, CP6.4]

The [test material name] aims, as explained above, at helping implementers to write applications that support the [technology] specification. In no way are these conformance tests in the sense of providing companies or institutions with certification or branding of [technology] support. The only claim that could be made is that a particular implementation is conformant to a particular version of the [test material name]. There are two cases of results of running the [test material type]:

  1. The implementation fails to pass the test suite. In this case it can be asserted that the implementation fails to meet the relevant [technology] specification.
  2. The implementation passes the test suite. In this case all that can be asserted is that the implementation is conformant to that particular version of the [test material name] test suite.

Licenses

[For OpsGL checkpoints CP5.3, CP6.2]

Grant of License for Contributions under the W3C Software License

The Contributor hereby makes certain materials, as described below (the "Materials";), available for use in supporting the World Wide Web Consortium (W3C). The Contributor hereby grants to MIT, ERCIM, and Keio (the "W3C Hosts";) a perpetual, nonexclusive, royalty-free, world-wide right and license to use, copy, modify, create derivatives of and distribute the Materials, in whole or in part, solely in connection with the W3C, and to allow others to do the same. W3C Hosts will publish and distribute the Materials, or any modifications or derivatives thereof, pursuant to the W3C Software License (20021231), and/or the W3C Document License (20021231) as modified from time to time.

The Contributor represents that she/he has all rights necessary to contribute the Materials, and that use of the Materials as contemplated herein by W3C Hosts does not violate any copyright, patent, trademark, or contractual obligations.

The Contributor agrees that any derivatives of the Materials created in connection with the W3C shall be owned by the W3C Hosts. Any publication or distribution of the Materials or any derivative thereof, will retain attribution of authorship to the Contributor. Whenever modifications are made to the Materials, this fact, and the nature of the modifications, will be clearly identified in the distributed version thereof. The W3C Hosts make no commitment to support or distribute the Materials.

Test material license

The [test material name] produced by the [WG name] Working Group will be distributed under the W3C Document License.

Publication

Publication of test material

The [test material name] will be published on the W3C [WG name] website, available from: [http://www.w3.org/[WGname]/Test/].

The [test material name] will be published after thorough review by the WG and the community, and with the consensus of the WG.

The [test material name] will contain the conformance disclaimer as stated above.

Publication of test results

[For OpsGL checkpoint CP6.5]

The [WG name] WG encourages the publication of test results. To facilitate publication of results, the WG will make available on its webspace, a place to publish test results. Only test results that have been supplied by the developer of the tested implementation will be considered for publication.

The WG will also consider a 'members only' webspace to publish developer submitted test results. This restricted publication of results may be used for situations in which the specification and/or tests are being debugged. For example, when the specification is in CR and a set of tests are being used to assess the features of the specification by determining what is being implemented.

Test Material Maintenance

[For OpsGL checkpoints CP8.2, CP8.3]

The WG is committed to maintaining the quality and relevance of the [test material name]. This will be done by ensuring that the test materials are consistent with any erratum and are updated as needed to reflect the current versions and status of the specification. The procedures describe above will be used to accept new contributions as well as to handle tests that are in dispute.