DAWG: Test case structure

Document Editor
Steve Harris – IAM Research Group, Southampton
Jeen Broekstra – Information Systems Group, Eindhoven University of Technology
Lee Feigenbaum – IBM
$Revision: 1.20 $


This document describes the testing process used by the RDF Data Access Working Group.

Status of This Document

Working Document.

The RDF Data Access Working Group (DAWG) uses a test-driven process.  The test area is a collection of the current test cases of the working group.

Tests are divided into collections (corresponding to directories) for manageability.  Each collection of tests has a manifest file within its directory (usually named manifest.ttl, but sometimes manifest.n3). There is also a pair of overall manifests containing entries pointing to the individual test collection manifests: One is a manifest of syntax-only tests (positive and negative tests); the other is a manifest of query-evaluation tests.


The Working Group recently completed reorganizing the test suite. The output of this reorganization process can be found in the data-r2 directory. The tests here are restructured copies of tests from the previous test suite along with new tests aimed at covering as much of the SPARQL Query Language as possible. The purpose of this restructuring is to enhance usability, clear away obsolete tests and provide an up-to-date, consistent, and easy-to-use suite of test cases that SPARQL query language implementors can use to evaluate and report on their implementation.

The Working Group decided on 21 Aug 2007 that the tests as-is constitute a test suite that the group will use to generate an implementation report for the SPARQL Query Language for RDF.

Manifest Vocabularies

The DAWG's test manifest files define three vocabularies to express tests:

  1. manifest vocabulary (prefixed with mf: below)
  2. query-evaluation test vocabulary (prefixed with qt: below)
  3. DAWG test approval vocabulary (prefixed with dawgt: below)

All examples below use these prefix bindings (specified in turtle):

@prefix rdf:     <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs:    <http://www.w3.org/2000/01/rdf-schema#> .
@prefix mf:      <http://www.w3.org/2001/sw/DataAccess/tests/test-manifest#> .
@prefix dawgt:   <http://www.w3.org/2001/sw/DataAccess/tests/test-dawg#> .
@prefix qt:      <http://www.w3.org/2001/sw/DataAccess/tests/test-query#> .

Manifest Structure

A manifest is a list (RDF Collection) of tests. Every test has a name (mf:name); many tests also have a comment (rdfs:comment) explaining the purpose of the test. The dawgt:approval predicate relates a test to its official Working Group status (e.g. dawgt:Approved). Tests are grouped (via their rdf:type) as syntax tests (an rdf:type of either mf:PositiveSyntaxTest or mf:NegativeSyntaxTest) or query-evaluation test (an rdf:type of mf:QueryEvaluationTest).

Syntax Tests

Each syntax test has an mf:action, the object of which is a resource identifying a (possible) query string. An example definition of a syntax test is:

<#syntax-basic-01>  mf:name  "syntax-basic-01.rq" ;
     rdf:type   mf:PositiveSyntaxTest ;
     mf:action  <syntax-basic-01.rq> ;
     dawgt:approvedBy <http://lists.w3.org/Archives/Public/public-rdf-dawg/2007JanMar/0047> ;
     dawgt:approval dawgt:Approved .

A SPARQL implementation passes a mf:PositiveSyntaxTest if it parses the query string without error. A SPARQL implementation passes a mf:NegativeSyntaxTest if it raises an error while attempting to parse the query string.

Query-Evaluation Tests

Each query-evaluation test has an mf:action and an mf:result. The object of mf:action is a resource with properties taken from the query-evaluation test vocabulary. At a minimum, a test's action includes a qt:data relation and a qt:query relation. The qt:data predicate points to a URI that can be dereferenced to yield the default graph for the test. The qt:query prediate points to a URI that can be dereferenced to yield the query string for the test. Query-evaluation tests may also use the qt:graphData predicate to indicate the named graph components of the test's RDF dataset.

Query evaluation tests also contain an mf:result which points to a URI that can be dereferenced to yield the expected results of the test query. These results are expressed in one of several possible ways:

A SPARQL implementation passes a query-evaluation test if the graph produced by evaluating the query against the RDF dataset (and encoding in the DAWG result set vocabulary, if necessary) is href="http://www.w3.org/TR/rdf-concepts/#section-graph-equality">equivalent [RDF-CONCEPTS] to the graph named in the result (after encoding in the DAWG result set vocabulary, if necessary). Equivalence can be tested by checking that the graphs are isomorphic and have identical IRI and literal nodes. Note that testing whether two result sets are isomorphic is simpler than full graph isomorphism. Iterating over rows in one set, finding a match with the other set, removing this pair, then making sure all rows are accounted for, achieves the same effect.

Query evaluation tests that involve the REDUCED keyword have slightly different passing criteria. These tests are indicated in the manifest files with the mf:resultCardinality predicate with an object of mf:LaxCardinality. To pass such a test, the result set produced by a SPARQL implementation must contain each solution in the expected result set at least once and no more than the number of times that the solution occurs in the expected result set. (That is, the expected result set contains the solutions with cardinalities as they would be if the query did not contain REDUCED; to pass the test, an implementation must produce the correct results with cardinalities between one and the cardainlity in the expected result set.)

An example definition of a query-evaluation test is:

<#dawg-regex-002> a mf:QueryEvaluationTest ;
      mf:name    "regex-query-002" ;
      dawgt:approval dawgt:Approved ;
      dawgt:approvedBy <http://lists.w3.org/Archives/Public/public-rdf-dawg/2007AprJun/0029.html> ;
          "Case insensitive unanchored match test" ;
          [ qt:query  <regex-query-002.rq> ;
            qt:data   <regex-data-01.n3> ] ;
      mf:result  <regex-result-002.n3> .

Test annotations


A number of tests in the open-world directory illustrate features of SPARQL by depending on how a SPARQL query processor can extend the set of core types and operations as defined by the operator table [http://www.w3.org/TR/rdf-sparql-query/#OperatorMapping].

These tests are marked by property mf:requires and an object value from one of the URIs described below.

Requires the processor to understand comparisons of literal of type xsd:date. Without proivding operations on the xsd:date datatype, a processor would raise an error on the operations of "=" and "!=" etc. With an understanding of xsd:date, a processor can perform value-based operations and provide the operations described in "XQuery 1.0 and XPath 2.0 Functions and Operators" (e.g. date-equals date-less-than)
This indicates that the test uses the fact that plain literals, without language tags test are the same value as an xsd;string with the same lexicial form. This is covered by rules "xsd 1a" and "xsd 1b" from RDF Semantics [http://www.w3.org/TR/rdf-mt/#DtypeRules].
This indicates that a processor extends the SPARQL operator model by using the fact that values of literals can be in disjoint value spaces and hence can not be equal by value. For example, an xsd:integer can not be the same value as an xsd:boolean because these two datatypes define disjoint value spaces.
This indicates that the test assumes the SPARQL query processor has support for plain literals with language tags. The minimum set of operators in the SPARQL operator table does not include language tag handling, only plain literals without language tag (simple literals) and certain XSD datatypes.

This annotation indicates a feature of SPARQL that implementers might note:

The test involves handling of ill-formed literals.