Tests

We are currently evaluating the feasibility of using the FOTS test suite format. This format is the format used by the actual FOTS suite, the official test suite for the F&O spec, maintained jointly by the XSL and XML Query Working Groups.

We still have to check the IP and licensing terms of FOTS (Florent is double-checking it with the XSL and XML Query WGs), and to check its support in all major implementations (see below).

Drawbacks

• Does not seem to cope properly with heterogeneous sequences in the result (think for instance of the HTTP Client's http:send-request()...) A general assertion mechanism (getting any XPath expression against the result and returning a boolean) would solve that problem.

Example

Here is a simple example of an FOTS test suite. It is very simple, but gives a first idea of the overall structure of a test suite in this format. It is also used to double-check support in some processors. Let's say this directory is /fots-example, so wherever you see that path, you have to substitute the real path on your machine.

A test suite has a dedicated directory. Let's call it the test suite directory:

fots-example/
   catalog.xml
   set.xml
   sample.xml
   results/
      saxon/
         exceptions.xml

The test suite directory must contain a test suite catalog, a file named catalog.xml. The catalog contains references to test set catalogs (breaking down the whole suite in several pieces), as well as global environments definitions (an environment defines more or less an expression context, like the set of namespace bindings, the context node by refereing to an XML file, the in-scope schemas, etc.):

<catalog xmlns="http://www.w3.org/2010/09/qt-fots-catalog" test-suite="try" version="0.1.0">

   <!-- an empty context (the context node must be undefined) -->
   <environment name="empty"/>

   <!-- set prefix "s" and refer to sample.xml as context item -->
   <environment name="sample">
      <namespace prefix="s" uri="http://example.org/sample"/> 
      <source role="." file="sample.xml" uri="http://example.org/sample.xml">
         <description>Some sample input tree.</description>
         <created by="Florent Georges" on="2012-05-14"/>
      </source>
   </environment>

   <test-set name="set" file="set.xml"/>

</catalog>

The test set files contain the actual test cases (as well as local, more specific environments, optoinally):

<test-set xmlns="http://www.w3.org/2010/09/qt-fots-catalog" name="set">

   <description>Some test set...</description>

   <!-- test that "hello" is a string, equal to 'hello' -->
   <test-case name="sample-001">
      <description>Some test case...</description>
      <created by="Florent Georges" on="2012-05-14"/>
      <environment ref="empty"/>
      <test>"hello"</test>
      <result>
         <all-of>
            <assert-eq>'hello'</assert-eq>
            <assert-type>xs:string</assert-type>
         </all-of>
      </result>
   </test-case>

   <!-- test sample.xml contains a child element named 'sample' -->
   <test-case name="sample-002">
      <description>Other test case...</description>
      <created by="Florent Georges" on="2012-05-14"/>
      <environment ref="sample"/>
      <test>sample</test>
      <result>
         <assert-type>element(sample)</assert-type>
      </result>
   </test-case>

</test-set>

The results should go in a separate sub-directory in results/. For instance Saxon uses: results/saxon/. The content of that directory is implementation-defined.

The content of the sample.xml file is only:

<sample/>

Support

In addition to the below state of the FOTS support in various processors, let's keep an eye on this project by Leo Woerteler. This is an FOTS harness written in XQuery (XQuery 3.0 + Map Facility it seems...) Could be a good alternative for XQuery implementations not providing access to their own harness.

BaseX

The BaseX harness for FOTS (as for other WG's test suite formats) is in the open-source project BaseXdb/basex-tests on GitHub. It is Maven-based. In order to use it, you have to clone the repository on your machine, compile it using Maven (we assume Maven is install and available in the PATH as "mvn"), then run the main class from the command-line, in the test suite directory.

The driver must be started from the test suite directory:

BASEX=~/.m2/repository/org/basex/basex/7.2.2-SNAPSHOT/basex-7.2.2-20120514.173900-14.jar
BXTST=.../basex-tests/target/classes/

java -cp $BXTST:$BASEX org.basex.tests.w3c.QT3TS

BaseX creates a log file with the overall results and more infos for each failing test in the file qt3ts.log, at the root of the test suite directory.

Saxon

The FOTS driver for Saxon still have to be adapted to Saxon HE, but that should not be a problem. Florent is looking at it. Thanks to Mike to make the code available (and for his work on FOTS)!

The file results/saxon/exceptions.xml contains the test cases to ignore for Saxon (because it does not support some optional features or because the implementation is not complete yet). It must be present, but can be empty:

<exceptions/>

Saxon puts its results (and look for the exception file) in results/saxon/.

Zorba

Matthias tols us that Zorba should have support for running FOTS test suits any soon.