W3C

Guidelines for Running the XML Query Test Suite

This document provides information to implementers who wish to run the XQuery Test Suite on their implementation. It includes guidelines how test cases can be customized in order to run on an implementation, and describes the process of evaluating the results. The documentation of the XML Query Test Suite, which defines the structure of the test cases and the catalog, can be found in [1]. Guidelines for submitting results to the XML Query Working Group can be found in [3].

Obtaining a Test Harness

Implementers are expected to write their own test harness that implements the following tasks:

Read test cases from the catalog, apply customization if applicable (see below)
Execute tests, using source files specified in the catalog
Use appropriate comparator to match result
Produce categorization of test result (pass, fail etc., see below)

Ideally, the test harness produces an XML file containing all test results in the format shown below, that can be sent to the working group.

Test Suite Customization

In order to run the test suite on an XQuery implementation, implementers may customize the test suite and make a number of well-defined changes to the test cases. All changes made to the original test suite must be documented in free-text form as part of the result submission. Changes beyond the ones listed below must be highlighted.

Setting the Default Context Item

XQuery supports a number of different ways to refer to source data as query context. Among these are implicit context, external variables, the fn:doc() function, the default collection function, implementation-defined functions, or parameter passing through host-language binding.

Test cases that do not refer to any input document (i.e., the catalog does not contain any “input-file” for the “test-case”) or test a specific context setting (i.e., a “context” attribute in the “input-file” element defines a specific context setting) may not be customized this way.

The following example is a customizable test case (because the input-file element does not define a context attribute)

(: Name: Axes001 :)
(: Description: Child Element :)

(: insert-start :)
declare variable $input-context external;
(: insert-end :)

$input-context/child

and the corresponding catalog entry:

<test-case name="Axes001" FilePath="Axes" ... >
...
<query name="Axes001.xq" date="2003-02-25">
<description>Child Element</description>
</query>
<input-file role="principal-data"
variable="input-context">TreeCompass</input-file>
<output-file role="principal" compare="XML">Axes001.xml</output-file>
</test-case>

The context setting for this test case may be customized in one of the following six ways. Note that option 3 and 5 (default) are only applicable for one source document.

Unchanged: use external variables as indicated in the original query
Implicit variable declaration: Remove variable declarations between insert-start and insert-end comments. The implementation binds the input context to the variable $input-context.

(: Name: Axes001 :)
(: Description: Child Element :)

$input-context/child

Implicit context: Remove variable declarations between insert-start and insert-end comments, and the variable references, and refer to the implicit context (.).

(: Name: Axes001 :)
(: Description: Child Element :)

./child

Use the fn:doc function: Remove variable declarations between insert-start and insert-end comments, and replace variable references with the fn:doc() function, including the input file specified in the catalog as argument

(: Name: Axes001 :)
(: Description: Child Element :)

fn:doc("TreeCompass.xml")/child

Use the (default) collection: Remove variable declarations between insert-start and insert-end comments, and replace variable references with the fn:collection() function (including optional URI).

(: Name: Axes001 :)
(: Description: Child Element :)

fn:collection()/child

Implementation-defined function: Remove variable declarations between insert-start and insert-end comments, and replace variable references with an implementation-defined function resolving to the input context.

(: Name: Axes001 :)
(: Description: Child Element :)

impl-fn:impl-udf("TreeCompass")/child

Setting a Specific Context Item

When explicitly testing fn:doc and fn:collection (including URI argument), test cases must customize the URI used to resolve the input data specified in the catalog. The same mechanism declaring an external variable (referring to the URI value) as introduced in the previous section is used. For example, a test case explicitly testing the fn:doc() function looks like:

(: Name: FnDoc001 :)
(: Description: Test fn:doc() function :)

(: insert-start :)
declare variable $input-context external;
(: insert-end :)

$input-context/child

Implementers may bind a URI value to the external variable, or remove the variable declaration. References to the external variable in the test case must be replaced with the fn:doc or fn:collection function as specified in the test case. The argument of the function must be either the external variable, or a constant URI value.

Test cases explicitly testing the implicit context, external variables or default collection must not contain an external variable declaration, and must not be customized.

Customizing Namespaces

Test cases can refer to validated documents (which requires namespace declaration) or imported schemas. Such test cases contain schema import declarations located within the same comments as the context item described above.

(: input-start :)
import schema namespace example="http://www.example.com";
(: input-end :)

This declaration must refer to a source schema document defined in the test suite catalog having the same name and URI.

This declaration can be customized in one of the following three ways:

Unchanged: use schema import as indicated in the original query
Remove the schema import declaration from the query, and add namespace declaration using same name and URI to statically known namespaces before the query is executed.
Replace schema import with namespace declaration using same name and URI.

declare namespace example="http://www.example.com";

Host Language Binding

Test cases can be embedded in a host language, for example using the xmlquery function in SQL. This may require escaping certain characters like quotes.

select xmlquery('$input-context/child'
passing xmlcol as "input-context")
from TreeCompass

Comparing Results

In order to check correctness of running a test case, the result of the implementation must be compared to the result provided in the test suite. The implementations result of the test case must be serialized and compared to the expected file(s) provided in the test suite. The catalog defines for each test case, which of the following five comparators has to be applied:

XML: The test harness must canonicalize both, the actual result and the expected result according to the “Canonical XML” recommendation [2], which refers to a number of open-source implementations. Byte-comparison can then be applied to the resulting XML documents. If the test harness does this process in a different manner, it must be documented.
XML fragment: For XML fragments, the same root node must be created for both, implementation result and test suite result. The resulting XML can be compared using XML comparison.
Text: text is compared using byte-comparison.
Ignore: no comparison needs to be applied; the result is always true if the implementation successfully executes the test case.
Inspect: A human is required to make the call about correctness of the result according to the description in the test case.
Error: The expected result of the test case is and error, identified as an eight-character error code (e.g., XPST0003). The result of a test is true, if the implementation raises an error. However, raising an error because an implementation does not support the feature is not considered a correct result.

It is possible that a test case provides multiple expected results. In this case, successfully comparing the actual result to one of the provided expected results is a "pass".

References

[1] XQuery Test Suite Documentation

[2] Canonical XML Version 1.0, W3C Recommendation 15 March 2001
(http://www.w3.org/TR/xml-c14n)

[3] Guidelines for Submitting XQTS Results

Webmaster · Last modified: $Date: 2005/08/10 17:31:59 $ by $Author: liam $

Copyright © 1994-2005 W3C^® (MIT, ERCIM, Keio), All Rights Reserved. W3C liability, trademark, document use and software licensing rules apply. Your interactions with this site are in accordance with our public and Member privacy statements.