We are currently evaluating the feasibility of using the FOTS test suite format. This format is the format used by the actual FOTS suite, the official test suite for the F&O spec, maintained jointly by the XSL and XML Query Working Groups.
We still have to check the IP and licensing terms of FOTS (Florent is double-checking it with the XSL and XML Query WGs), and to check its support in all major implementations (see below).
- Does not seem to cope properly with heterogeneous sequences in the result (think for instance of the HTTP Client's http:send-request()...) A general assertion mechanism (getting any XPath expression against the result and returning a boolean) would solve that problem.
Here is a simple example of an FOTS test suite. It is very simple, but gives a first idea of the overall structure of a test suite in this format. It is also used to double-check support in some processors. Let's say this directory is /fots-example, so wherever you see that path, you have to substitute the real path on your machine.
A test suite has a dedicated directory. Let's call it the test suite directory:
fots-example/ catalog.xml set.xml sample.xml results/ saxon/ exceptions.xml
The test suite directory must contain a test suite catalog, a file named
catalog.xml. The catalog contains references to test set
catalogs (breaking down the whole suite in several pieces), as well as
global environments definitions (an environment defines more or
less an expression context, like the set of namespace bindings, the
context node by refereing to an XML file, the in-scope schemas, etc.):
<catalog xmlns="http://www.w3.org/2010/09/qt-fots-catalog" test-suite="try" version="0.1.0"> <!-- an empty context (the context node must be undefined) --> <environment name="empty"/> <!-- set prefix "s" and refer to sample.xml as context item --> <environment name="sample"> <namespace prefix="s" uri="http://example.org/sample"/> <source role="." file="sample.xml" uri="http://example.org/sample.xml"> <description>Some sample input tree.</description> <created by="Florent Georges" on="2012-05-14"/> </source> </environment> <test-set name="set" file="set.xml"/> </catalog>
The test set files contain the actual test cases (as well as local, more specific environments, optoinally):
<test-set xmlns="http://www.w3.org/2010/09/qt-fots-catalog" name="set"> <description>Some test set...</description> <!-- test that "hello" is a string, equal to 'hello' --> <test-case name="sample-001"> <description>Some test case...</description> <created by="Florent Georges" on="2012-05-14"/> <environment ref="empty"/> <test>"hello"</test> <result> <all-of> <assert-eq>'hello'</assert-eq> <assert-type>xs:string</assert-type> </all-of> </result> </test-case> <!-- test sample.xml contains a child element named 'sample' --> <test-case name="sample-002"> <description>Other test case...</description> <created by="Florent Georges" on="2012-05-14"/> <environment ref="sample"/> <test>sample</test> <result> <assert-type>element(sample)</assert-type> </result> </test-case> </test-set>
The results should go in a separate sub-directory in
results/. For instance Saxon uses:
results/saxon/. The content of that directory is
The content of the sample.xml file is only:
In addition to the below state of the FOTS support in various processors, let's keep an eye on this project by Leo Woerteler. This is an FOTS harness written in XQuery (XQuery 3.0 + Map Facility it seems...) Could be a good alternative for XQuery implementations not providing access to their own harness.
The BaseX harness for FOTS (as for other WG's test suite formats) is in the open-source project BaseXdb/basex-tests on GitHub. It is Maven-based. In order to use it, you have to clone the repository on your machine, compile it using Maven (we assume Maven is install and available in the PATH as "mvn"), then run the main class from the command-line, in the test suite directory.
The driver must be started from the test suite directory:
BASEX=~/.m2/repository/org/basex/basex/7.2.2-SNAPSHOT/basex-7.2.2-20120514.173900-14.jar BXTST=.../basex-tests/target/classes/ java -cp $BXTST:$BASEX org.basex.tests.w3c.QT3TS
BaseX creates a log file with the overall results and more infos for each failing test
in the file
qt3ts.log, at the root of the test suite directory.
The FOTS driver for Saxon still have to be adapted to Saxon HE, but that should not be a problem. Florent is looking at it. Thanks to Mike to make the code available (and for his work on FOTS)!
the test cases to ignore for Saxon (because it does not support
some optional features or because the implementation is not
complete yet). It must be present, but can be empty:
Saxon puts its results (and look for the exception file) in
Matthias tols us that Zorba should have support for running FOTS test suits any soon.