This is the 10 December 2003 XML Conformance Test Suites Process Document.
Comments on this document are invited and are to be sent to the XML TS mailing list firstname.lastname@example.org. An archive is available at http://lists.w3.org/Archives/Public/public-xml-testsuite/.
This document has been produced as part of the W3C XML Activity. The authors of this document are the XML WG members.
A joined effort between W3C and NIST has been established to produce a highly comprehensive conformance test suite (XML TS) for the XML specifications. The XML 1.0 Second Edition Test Suite is being augmented to reflect the current work of the W3C XML Core Working Group. W3C will be responsible for assessing issues with the TS. Also, the test suites will be jointly developed by these two parties and will take the form of a public framework as explained below.
For more information on the current status of the XML TS, please visit the XML TS page, where you will find additional information as well as pointers to relevant resources.
The XML TS will be produced in a public framework with contributions from developers and companies in the community. The XML TS will be coordinated and supervised by the XML Core WG. The representatives will use resources from their organisations as well as invite individuals and companies (through representatives) to allow for a larger number of people to get involved.
Since this will be a public framework, there are a series of procedural issues that need to be resolved. There is a need to have a stable mechanism for submitting tests to the test suite. It's been proposed that we look into a multi-level process (with the possibility of tests passing between the different levels for reasons explained below). The points below assume that there is a technically stable mechanism for submitting tests, saving information about those tests, version handling and so forth.
The XML TS editor will maintain a list of tests needed that will be available at the XML TS development web site. It is presupposed that developers or organizations submitting tests have done a sanity check with regard to the functionality of the tests by referring to the relevant specification.
The main channel of communication for the XML TS will be the XML TS mailing list: email@example.com.
The main channel of communication for submitting tests to the XML TS will be the XML TS mailing list: firstname.lastname@example.org. An archive is available at http://lists.w3.org/Archives/Public/public-xml-testsuite/.
In order to simplify for individuals and companies the production of the test suite and, if they so wish, contribute with tests of their own, we propose the following process for submitting tests:
The test suite's moderators investigate the test given the following criteria. At each point is also indicated possible reasons for not accepting to publish the particular test.
If the test is decided to be stable, which means that it is relevant, that it indeed tests a particular identifiable part of the specification, and that it is not wrongly written, it becomes "Reviewed and Stable" and receives this status.
Tests that at this or some other stage are judged not to be appropriate for publishing receive the "Inappropriate" status. These tests should be kept for archive purposes.
In cases where tests have received the "Reviewed and Stable" status but are found to be erroneous, the following procedure is proposed:
If it is found that a test that has been called "Reviewed and Stable" actually is not stable (if the tests are not correctly written or for any other reason), this particular test gets submitted to the XML WG for further consideration. Possible outcomes of this stage is that the test, after consideration, is judged to be
We propose that each test or whole suite comes fully documented with regard to the following (there will be an XML wrapper for this kind of information available from the TS website):
It is proposed that the W3C XML WG representative act as moderator and controller for incoming tests. The XML TS moderator is choeon by the XML WG. All tests should be kept for archive purposes, whether they get published or not.
The XML TS aims, as explained above, at helping implementors to write applications that support the XML specifications. In no way are these conformance tests in the sense of providing companies or institutions with certification of XML support. The only claim that could be made is that a particular implementation is conformant to a particular version of the XML TS. There are two cases of results of running the test suite:
The XML TS will use the following grant of license:
The Contributor hereby grants to the W3C, a perpetual, nonexclusive, royalty-free, world-wide right and license under any Contributor copyrights in this contribution to copy, publish and distribute the contribution under the W3C Software License (19980720), as well as a right and license of the same scope to any derivative works prepared by the W3C and based on, or incorporating all or part of the contribution.
The Contributor vouches that she/he has all rights necessary to contribute the Materials in a way that does not violate copyright, patent, and trademark rights; contractual obligations, or libel regulations.
Contributor further agrees that any derivative works of this contribution prepared by the W3C shalll be solely owned by the W3C. The Contributor agrees that all contributed Materials when published or otherwise distributed by the W3C will be governed by the W3C Software License (19980720).
W3C will retain attribution of authorship to the Contributor. Whenever modifications are made to the Materials, this fact, and the nature of the modifications, will be clearly signalled in the distributed version thereof. The W3C makes no a-priori commitment to support or distribute contributions.