About XML Schema · XML Schema Test Suite FAQ · Members only resource: XML Schema Working Group

XML Schema Test Suite Process Document

14 February 2003

This version:

http://www.w3.org/XML/2004/xml-schema-test-suite/XMLSchemaTS-Process.html

Latest version:

http://www.w3.org/XML/2004/xml-schema-test-suite/XMLSchemaTS-Process.html

Editor:

Henry S. Thompson <ht@w3.org>

Status

This is the 14 February, 2003, XML Schema Test Suite Process Document, amended 21 November 2006.

Comments on this document are invited and should be sent to the XML Schema TS mailing list: public-xml-schema-testsuite@w3.org. An archive is available at http://lists.w3.org/Archives/Public/public-xml-schema-testsuite/.

This document has been produced as part of the W3C XML Schema Activity. The authors of this document are the XML Schema WG members.

Table of contents

• 1. Introduction
• 2. Requirements
• 3. Parties involved
• 4. Procedural issues
• 5. Cautionary note
• 6. Ownership
• 7. Timeframe

1. Introduction

A joint effort between W3C and The National Institute of Standards and Technology (NIST) has been established to produce a comprehensive test suite for the XML Schema Recommendation. The test suite will be known as the XML Schema Test Suite (XML Schema TS, TS). The TS is being augmented to reflect the current work of the W3C XML Schema Working Group (WG). W3C will be responsible for assessing issues with the test suite. The test suite will be jointly administered by W3C and NIST, and will take the form of a public framework as explained below.

For more information on the current status of the TS, please visit the XML Schema TS page, which contains additional information and pointers to relevant resources.

2. Requirements

1. The TS should be developed in a public framework. The development of this framework itself is a central area of interest, in addition to the TS in general. The WG welcomes the participation of interested parties in developing the test suite.
2. The TS is intended to be used as a tool to aid implementors in developing software that supports W3C XML Schema. Validation and certification of such software are outside the scope of this work. The tests and test suite will be provided for information and assistance only. However, we intend to produce as comprehensive, functional and general a test suite as possible, and this should be the overall goal in designing and implementing the XML Schema TS.
3. The TS will be hosted on the W3C site once developed. Where the test suite should reside during development is under discussion. The test suite might also be mirrored at various sites in order to simplify access and enhance availability to the community.

3. Parties involved

The XML Schema TS will be produced in a public framework, and will consist of contributions from vendors and other developers in the community. The XML Schema TS will be coordinated and supervised by the XML Schema WG and NIST. The WG representatives will use resources from their organizations as well as inviting contributions from individuals and vendors (through representatives) to allow for maximum community involvement.

4. Procedural issues

Since this will be a public framework, certain procedural issues need to be resolved:

• A mechanism for administration of the TS;
• A channel of communication for issues regarding the test suite (announcements, discussion, adjudications, and so on);
• A mechanism for test submission;
• Criteria for acceptance of tests for publication;
• A mechanism for review and reevaluation of tests;
• Appropriate standards for documentation of tests and test results;
• A mechanism for versioning of the test suite, and
• A mechanism for reporting test results.

The points below address these issues given that there is a technically stable mechanism for submitting tests, saving information about those tests, version handling and so forth.

When submitting tests, developers or organizations are assumed to have done a sanity check with regard to the functionality of the tests by referring to the relevant Recommendation(s).

Administration

The WG will establish a task force (XML Schema TS Task Force, TS Task Force) to oversee the administration of the TS. The responsibilities of the Task Force in administering the TS will include:

• Maintenance of the TS itself. The Task Force will be responsible for the content of the TS: addition of new tests, removal of invalid tests, changing the status of tests in response to disputes and dispute resolution, and versioning of the TS.
• Initial evaluation and preliminary review of all contributed tests.
• Receipt and maintenance of test result reports.
• Referral of disputed tests to the WG for detailed review.
• Monitoring of and participation in TS-related discussion.
• Maintenance of some measure of TS coverage of the Recommendation and, from this, maintenance of a "wishlist" of areas of the Recommendation for which tests are required. The Task Force will publish this list on the XML Schema TS website.
• Maintenance of the XML Schema TS website and TS-related mailing lists. The Task Force will ensure currency of the content of the site and associated documentation and links.
• Modification of tests in the TS when necessary or desirable.

Communication

The main channel of communication for the XML Schema TS will be the XML Schema TS mailing list: public-xml-schema-testsuite@w3.org. This list will be closely and routinely monitored by the TS Task Force.

Test submission

Tests should be submitted to the XML Schema TS framework at  xml-schema-testsuite-submit@w3.org. An archive is available at http://lists.w3.org/Archives/Public/xml-schema-testsuite-submit/.

In order to simplify the procedure for test submission for individuals and organizations, the following mechanism for test submission will be established:

1. The test, or series of tests, is submitted to the framework. The submitter should also send a notification to the XML Schema TS mailing list to indicate that they have submitted a contribution to the TS framework. Submitting parties are also encouraged to check the tests they are submitting against the "wishlist" of desired tests.
2. Receipt of the contribution is acknowledged by the TS Task Force. This should be done via email directly to the submitter, with a copy to the XML Schema TS mailing list.

Receipt and review of test contributions

The TS Task Force evaluates the tests included in a contribution according to the following criteria:

1. Relevance of a test to the W3C XML Schema Recommendation. It can here be decided that the particular test is not suitable for inclusion in the TS since, for example, it tests a feature that is not in the Recommendation. If there are other, previously submitted tests that are functionally equivalent and test the same part of the Recommendation, this test shall either be kept for archive purposes but not included in the TS, or substituted for the previously submitted test, which in turn shall be withdrawn from the TS.
2. Quality of tests and documentation. The tests and the accompanying documentation must be well-formed XML; the documentation must conform to the schema for test submissions, available at the XML Schema TS website, which outlines such things as the part of Recommendation under test, functionality of test, location of test, and so on. 

If a given test is judged suitable for inclusion in the TS according to these criteria, the test is added to the TS with status Accepted (see below).

All submitted tests are kept for archive purposes, whether published or not.

Documentation

Each contribution to the TS must be fully documented, and the documentation must conform to the schema for test submissions, which is available at the TS website. The documentation provides such information as:

• Name of submitter, date of creation, and, if appropriate, organization;
• Suggested ID for each test (this may be changed by the TS Task Force to comply with the identification mechanism of the technical platform);
• Part(s) of the Recommendation under test. The area under test should be suitably indicated by a link to the relevant part of the Recommendation and, optionally, by a text annotation;
• The expected validity of the test;
• If appropriate, a textual annotiation containing remarks on the test in human readable format, for example, any scenario which prompted the writing of the test.

Tests that are judged to be inappropriate for publication are returned to the contributor. Such tests are archived and not included in the TS.

Test status and review procedure

Once a test is included into the TS, it becomes subject to the dispute and review process outlined here.

When a test is added to the TS, it is automatically queued for preliminary review by the TS Task Force. Once it has been reviewed, and if it has been found to be free of errors, its status is changed to Stable (see below). Otherwise it is withdrawn from the TS and returned to its originator.

At any time, any member of the community may dispute the validity of any test in the TS. This is done by raising the issue for discussion via Bugzilla, which automatically copies the bug report to the TS mailing list: community discussion of disputes is the first stage in dispute resolution. If, after discussion, the dispute is felt to be valid either by the community and/or the TS Task Force, the TS Task Force will change the status of the test to reflect the fact that its validity has been disputed, and forward the test to the WG for detailed review, the results of which are outlined below.

The status of each test in the TS, as well as the date on which that status was attained, are contained in the documentation describing the test, and may change over time as a result of disputes and reviews.

The status of a test is intended to indicate the test's stability, relevance and reliability as a true test of functionality specified by the XML Schema Recommendation. The status of a test may take one of four values, with the following meanings:

• Accepted. The test has been received by the TS Task Force and contains no obvious errors, either in the test or the accompanying documentation.
• Stable. The test and its accompanying documentation have undergone preliminary review by the TS Task Force and have been found to be free of errors and relevant to the Recommendation or the test has undergone detailed review by the WG and been found to be free of errors and relevant to the Recommendation.
• Disputed: test error. The validity of the test has been questioned by one or more members of the community. A consensus exists in the community, or the TS Task Force agrees (or both) that the test is in error. The TS Task Force has submitted the test and the associated dispute to the WG for detailed review, along with a suggested resolution.
• Disputed: spec error. As a result of running the test, one or more members of the community has questioned the correctness, consistency, clarity, etc. of the part(s) of the Recommendation addressed by the test. A consensus exists in the community, or the TS Task Force agrees (or both) that a problem exists with the Recommendation. The TS Task Force has submitted the test and the associated dispute to the WG's erratum process.

1. The WG agrees that the test is in error. The test is withdrawn from the TS and returned to the submitter.
2. The WG finds that the test is not in error. The status of the test is set to Stable.
3. The WG finds that the apparent error in the test is due to some problem with the part(s) of the Recommendation addressed by the test. The status of the test is set to Disputed: spec error, and the review proceeds as described for this status.

1. The WG determines that the test is in error and that the Recommendation is not. The test is withdrawn from the TS and returned to the submitter.
2. The WG determines that the test is not in error, but that a problem exists with the Recommendation. That is, the test exercises some functionality which should be clearly specified in the Recommendation, but for some reason is not. The status of the test remains unchanged until such time as an official erratum is issued, when the status of the test is set to Stable
3. The WG determines that neither the test is nor the Recommendation is in error. The status of the test is set to Stable.

Versioning

The TS Task Force will be responsible for the publication, naming and archiving of versions of the TS. The Task Force will confer on a regular basis on the status of the TS. Broadly, the purpose of each conference will be to review events which have occurred since the previous conference, and which might affect the status of the TS, including:

• Submission of new test contributions. The Task Force will decide whether some or all new contributions should be included, in whole or in part, into the TS (see above).
• Results of preliminary review of tests by the TS Task Force. Previously accepted tests may either have their status changed or may be removed from the TS (see above).
• Disputes of existing tests. The Task Force will review new disputes and disputes still under discussion in the community at the time of the previous conference and decide whether sufficient grounds exist to forward one or more of these disputes to the WG. If a dispute is forwarded to the WG, the Task Force changes the status of the affected test(s) (see above).
• Feedback from the WG on previously referred disputes. As a consequence of such feedback, the Task Force may remove one or more tests from the TS and/or change the status of one or more tests (see above).

If, as a result of the conference, the status of the TS is changed (i.e. one or more tests are added to or removed from the TS, and/or one or more tests undergo a change in status) the TS Task Force will publish a new version of the TS incorporating any changes. However, in order to avoid overly frequent revisions of the TS and to simplify the task of administration, the Task Force has the discretion to decide the schedule on which changes to the TS will be incorporated and a new version released.

The TS Task Force will announce new versions of the TS on the XML Schema TS mailing list.

When a new version of the TS is published, the previous version will be frozen and archived.

Reporting test results

The Working Group strongly encourages all members of the community to report the results obtained when exercising processors against the TS or any part thereof. Such feedback gives the WG valuable insight into such issues as the relevance to the community of various parts of the Recommendation, which parts are open to conflicting interpretation, and so on.

The WG reserves the right to publish reported results. Access to results is determined by the vendor or developer responsible for the creation of the processor for which results are being reported. Two levels of access are offered:

1. W3C Members. Information on the origin of the test results will be available only to members of the WorldWide Web Consortium. Outside the W3C, the results may be published only in such a way that the identity of the processor which produced the results, and of the vendor or developer which produced the processor, are not divulged, e.g. as part of aggregated statistical information.
2. Public. The reported results will be publicly available at the TS website.

The level of access to the results is stipulated in the result report.

Note: If any developer or vendor feels that this level of access to results does not offer sufficient confidentiality to permit them to report results for their own processor, they are encouraged to contact the TS Task Force through the TS website.

All reports of test results must conform to the schema for test result reporting, available at the TS website.

Test results should be reported to the XML Schema TS Task Force at TS Task Force Contact.

5. Cautionary note

As explained above, the purpose of the XML Schema TS is to aid developers and vendors in the production of software that supports W3C XML Schema. The tests included in the TS are, strictly speaking, conformance tests: each test examines whether a processor behaves in the manner prescribed by the Recommendation in response to a given situation. However, the TS is not, and should not be interpreted as, a means to provide developers, vendors or institutions with any kind of certification of XML Schema support.

Each test in the TS has a status, which indicates the level of confidence a tester can place in the validity of the test as an accurate measure of some functionality specified in the Recommendation. A status of stable indicates the highest level of confidence. Thus, the WG recommends the following approach to the interpretation of TS results:

1. If a processor produces results which differ from the documented expected results for at least one test with status stable, there is a high degree of confidence that the processor does not conform to the Recommendation in those areas where the results differed;
2. If a processor produces results which agree with the documented expected results for all tests with status stable, there is a high level of confidence that the processor correctly implements those features of the Recommendation covered by those tests in the version of the TS used for testing.

6. Ownership

Contributions to the XML Schema TS will be made under the following grant of license:

The Contributor hereby makes certain Materials available to the World Wide Web Consortium (W3C) for use in supporting its activities. The Contributor hereby grants to the W3C a perpetual, nonexclusive, royalty-free, world-wide right and license under any owned copyrights in the Materials to copy, publish and distribute the Materials in whole or in part under the W3C DOCUMENT NOTICE AND LICENSE as well as a right and license of the same scope to any derivative works prepared by the W3C and based on, or incorporating all or part of the Materials.

The Contributor vouches that she/he has all rights necessary to contribute the Materials in a way that does not violate copyright, patent, and trademark rights, contractual obligations, or libel regulations.

The Contributor agrees that any derivative works of the Materials prepared by the W3C shall be solely owned by the W3C. The Contributor agrees that all contributed Materials when published or otherwise distributed by the W3C will be governed by the W3C DOCUMENT NOTICE AND LICENSE.

W3C will retain attribution of authorship to the Contributor. Whenever modifications are made to the Materials, this fact, and the nature of the modifications, will be clearly signalled in the distributed version thereof. The W3C makes no a-priori commitment to support or distribute the Materials.

7. Timeframe

Preliminary dates:

• XML Schema TS: March 2003

Tony Cincotta, NIST David Ezell, National Association of Convenience Stores, chair C.M. Sperberg-McQueen, W3C, staff contact