About XML Schema · XML Schema Test Suite FAQ · Members only resource: XML Schema Working Group

XML Schema Test Suite Framework

Contents

1. Introduction
2. Structure of the Test Suite
3. Documentation/Metadata
4. Using the Test Suite
5. Reporting test results
6. Contributing tests to the Test Suite

Introduction

The XML Schema Test Suite (XML Schema TS, TS) is a joint effort between W3C and The National Institute of Standards and Technology (NIST) to produce a comprehensive test suite for the W3C XML Schema Recommendation). The TS is developed and updated in a public framework as described here.

The XML Schema TS consists of tests contributed by vendors and other developers in the community. The TS is coordinated and supervised by the XML Schema WG (WG) and NIST.

The TS is intended to be used as a tool to aid implementers in developing software that supports W3C XML Schema. Validation and certification of such software are outside the scope of the TS. The tests and test suite are provided for information and assistance only. However, the intent is to publish as comprehensive, functional and general a test suite as possible.

For more information on the current status of the TS, please visit the XML Schema TS home page, which contains additional information and pointers to relevant resources.

Structure of the Test Suite

The TS is subdivided into test collections, one for each version of the Recommendation for which tests are available (thus, if tests were available for two versions of the Recommendation, the TS would contain two test collections).

Each test collection consists of one or more sets of tests, which may be grouped, for example, according to the originator of the test set, the subject area of the test set, and/or the date upon which the test set was incorporated into the TS.

Finally, the  tests in each set consist of at least one schema document, accompanied by zero or more instance documents to be validated against it.

The basis for evaluation of a processor under test is the comparison between the validation outcome it returns for each of the individual schema and instance documents in the TS and the expected validation outcome for that document, as expressed in the TS documentation/metadata.

Documentation/Metadata

The functions of the metadata file include:

• Definition of each test: the one or more schema and zero or more instance documents that constitute the test, plus a unique identifier for the test
• Expected validation outcome of each test document
• Status and status history of each test document within the TS (accepted, stable or disputed - see the TS Process Document for details)
• Traceability of tests (via links to the relevant section(s) of the Recommendation/errata)
• Description/other documentation of the tests and test documents

Using the Test Suite

1. Download the desired version(s) of the TS from the TS main page
2. Uncompress the TS into a directory/folder of your choice
3. Check the README file at the top level of the TS directory structure for the most uptodate information on the release
4. Create a test harness to navigate the metadata document, directing the processor under evaluation to each desired test (alternatively, direct the processor under evaluation via a browser using the browsable test report)
5. (Optional but preferred) produce a test result report to be forwarded to the TS Task Force (see below)

NOTE: The schema for test submissions and result reporting is intended to be sufficiently documented to enable both the construction of a test harness and the compilation of a result report with the minimum effort.

Reporting test results

The Working Group strongly encourages all members of the community to report the results obtained when exercising processors against the TS or any part thereof. Such feedback gives the WG valuable insight into such issues as the relevance to the community of various parts of the Recommendation, which parts are open to conflicting interpretation, and so on.

All reports of test results must conform to the TS schema.

Test results should be reported to the XML Schema TS Task Force at TS Task Force Contact.

The WG reserves the right to publish reported results. Access to results is determined by the vendor or developer responsible for the creation of the processor for which results are being reported. Two levels of access are offered:

1. W3C Members. Information on the origin of the test results will be available only to members of the WorldWide Web Consortium. Outside the W3C, the results may be published only in such a way that the identity of the processor which produced the results, and of the vendor or developer which produced the processor, are not divulged, e.g. as part of aggregated statistical information.
2. Public. The reported results will be publicly available at the TS website.

The level of access permitted to the results is stipulated by the submitter in the result report.

Note: If any developer or vendor feels that this level of access to results does not offer sufficient confidentiality to permit them to report results for their own processor, they are encouraged to contact the TS Task Force through the TS website.

Contributing tests to the Test Suite

The TS consists entirely of voluntary contributions of tests from interested parties. We encourage all members of the community to consider contributing any tests they may have developed to the TS.

Tests should be submitted to the XML Schema TS framework at  public-xml-schema-testsuite-submit@w3.org . An archive is available at http://lists.w3.org/Archives/Public/public-xml-schema-testsuite-submit/. Each submission must be fully documented (see below).

The mechanism for test submission is as follows:

1. The test, or series of tests, is submitted to the framework. The submitter should also send notification to the XML Schema TS mailing list to indicate that they have submitted a contribution to the TS framework. Submitting parties are also encouraged to check the tests they are submitting against the "wishlist" of desired tests.
2. Receipt of the contribution is acknowledged by the TS Task Force. This is done via email directly to the submitter, with a copy to the XML Schema TS mailing list.

Tony Cincotta, NIST David Ezell, National Association of Convenience Stores, chair C.M. Sperberg-McQueen, W3C, staff contact