This is the 16 March 2001 DOM Conformance Test Suites Process Document.
Comments on this document are invited and are to be sent to the DOM TS mailing list email@example.com. An archive is available at http://lists.w3.org/Archives/Public/www-dom-ts/.
This document has been produced as part of the W3C DOM Activity. The authors of this document are the DOM WG members.
There is an agreed upon need to produce a highly comprehensive conformance test suite (DOM TS) for each of the DOM Level specifications so far specified and also for DOM Level 3. In order to accomplish this, it has been agreed between W3C and NIST that these test suites be jointly produced. W3C will be responsible for coordinating these issues. NIST has agreed to allocate resources to update their current test suite to full Level 1 specification support, taking into account the alterations needed as indicated below. The test suites will be jointly developed by these two parties and will take the form of a public framework as explained below.
A test suite for DOM Level 2 will at a later stage be produced, building on the test suite for Level 1 and drawing on the experiences made while producing it, as well as experience drawn from related work in this area. It is understood that the Level 2 specification is quite large and covers too big an area for one single person to be able to write the entire test suite. For this reason, among others, but also aiming at wide-spread acceptance as well as contributions both from individuals and companies, it has been decided that the test suite be publicly developed, and the DOM WG suggests the design of both this test suite and the one for Level 1 be as indicated below.
As far as the test suite for Level 3 is concerned, it has been decided that it initially takes on the form of a small feature test suite, not complete, and advances to more comprehensive and public status similar to the test suites on level 1 and 2 once the DOM Level 3 WD reaches CR status.
This document does not deal with technical issues, as these have not yet been finally decided upon. There will be a document that in detail explains the technical design of the test suite framework; however it is appreciated that these questions will be of interested to the community and in order to allow for input, it has been decided to let this issue take top priority once the interested parties have gathered to start working with the test suite.
The DOM TS for all levels of the DOM will be produced in a public framework with contributions from developers and companies in the community. The DOM TS will be coordinated and supervised by the DOM WG and NIST. The representatives will use resources from their organisations as well as invite individuals and companies (through representatives) to allow for a larger number of people to get involved.
Since this will be a public framework, there are a series of procedural issues that need to be resolved. There is a need to have a stable mechanism for submitting tests to the test suite. It's been proposed that we look into a multi-level process (with the possibility of tests passing between the different levels for reasons explained below). The points below assume that there is a technically stable mechanism for submitting tests, saving information about those tests, version handling and so forth.
The DOM TS editor will maintain a list of tests needed that will be available at the DOM TS development web site. It is presupposed that developers or organizations submitting tests have done a sanity check with regard to the functionality of the tests by referring to the relevant specification.
The main channel of communication for the DOM TS will be the DOM TS mailing list firstname.lastname@example.org.
In order to simplify for individuals and companies the production of the test suite and, if they so wish, contribute with tests of their own, we propose the following process for submitting tests:
The test suite's moderators investigate the test given the following criteria. At each point is also indicated possible reasons for not accepting to publish the particular test.
If the test is decided to be stable, which means that it is relevant, that it indeed tests a particular identifiable part of the specification, and that it is not wrongly written, it becomes "Reviewed and Stable" and receives this status.
Tests that at this or some other stage are judged not to be appropriate for publishing receive the "Inappropriate" status. These tests should be kept for archive purposes.
In cases where tests have received the "Reviewed and Stable" status but are found to be erroneous, the following procedure is proposed:
If it is found that a test that has been called "Reviewed and Stable" actually is not stable (if the tests are not correctly written or for any other reason), this particular test gets submitted to the DOM WG for further consideration. Possible outcomes of this stage is that the test, after consideration, is judged to be
We propose that each test or whole suite comes fully documented with regard to the following (there will be an XML wrapper for this kind of information available from the TS website):
Nodeinterface in DOM Level 1, this should be suitably indicated, either by giving a link to the relevant part of the specification (in this case http://www.w3.org/TR/1998/REC-DOM-Level-1-19981001/level-one-core.html#ID-1950641247), or by indicating, in text, that it is this part of the specification that is being tested.
It is proposed that the W3C DOM WG representative act as moderator and controller for incoming tests. The DOM TS moderator is choeon by the DOM WG. All tests should be kept for archive purposes, whether they get published or not.
The DOM TS aims, as explained above, at helping implementors to write applications that support the DOM specifications. In no way are these conformance tests in the sense of providing companies or institutions with certification of DOM support. The only claim that could be made is that a particular implementation is conformant to a particualr version of the DOM TS. There are two cases of results of running the test suite:
It is understood that no submitting party, individual or company, can claim ownership of submitted tests. All copyright notes and similar text must be removed from the code prior to submitting the test (or tests) to the DOM TS. It is also understood that no submitter can try to withdraw submitted tests, but must ask the TS moderator that the test be withdrawn. Reasons for this should be given, for reasons of future reference. In this case, the moderator should send a notification to the DOM TS mailing list in which is stated that the contributor for a particular test wishes to withdraw the submitted test. Only after that and after approval from the DOM WG can the test be withdrawn from the DOM TS.
The W3C Document Notice and License will apply for the DOM Test suites.