I don't think there's an explicit statement in the test guidelines to this effect, but I find the following rule works: if the <source> element in the test-sources section of the catalog has a "schema" attribute, then validate the source document against that schema before running the query. An import-schema declaration is needed in the query only if the query contains explicit references to components declared in the schema.

The rule would obviously fix the tests in discussion (and in that case should be added to the Testsuite instructions), but my question still remains: shouldn't they be moved to the XML Schema optional features?

As it stands now, it is impossible for a Minimal Conformance XQuery processor that does not implement the Schema Validation optional feature to pass 100% of the Minimal Conformance tests.

This is a bug in the testing methodology.

This should be stated explicitly, not made as an hidden rule that some
implementations do even though it is not stated anywhere.

Best
Dana


(In reply to comment #1)
> I don't think there's an explicit statement in the test guidelines to this
> effect, but I find the following rule works: if the <source> element in the
> test-sources section of the catalog has a "schema" attribute, then validate the
> source document against that schema before running the query. An import-schema
> declaration is needed in the query only if the query contains explicit
> references to components declared in the schema.

*** Bug 11824 has been marked as a duplicate of this bug. ***

The original design of the test suite seems to have been that if a test depends on an optional feature, then it should go in a test group (in the catalog) whose name implies a dependency on that feature. However, there doesn't seem to be a section of the catalog for tests that depend on the input document being pre-validated outside the query ("construction from a PSVI" in spec-terminology.) So it's not obvious where one would move these tests to indicate such a dependency.

Some other tests such as Constr-cont-constrmod-2 simply give two alternative results, one for validated input and one for unvalidated. Since there is no indication of the relationship of test results to processor features, this is a very unsatisfactory resolution.

In the XQuery Update tests we have moved in the direction of having the metadata for a test group describe its dependencies on optional features. For example:

            <GroupInfo>
               <title>Revalidation Declaration = strict</title>
               <description/>
               <depends-on>
                  <feature supported="true">revalidation:strict</feature>
                  <description>These tests are designed to be run by a product  that supports
                     revalidation mode strict</description>
                  <spec-citation spec="XQUPDATE" section-number="2.2.1"
                     section-title="Revalidation Declaration"
                     section-pointer="id-revalidation-declaration"/>
               </depends-on>
            </GroupInfo>

I think this is the right strategic direction: we should put these tests in a group and label it

<depends-on>
  <feature supported="true">input-construction:PSVI</feature>
</depends-on>

and if appropriate, include another copy of the test with

<depends-on>
  <feature supported="true">input-construction:infoset</feature>
</depends-on>

and a different expected result.

The source document "atomic.xml" is used by the following tests:


fn-union-node-args-015 to 022
fn-intersect-node-args-015 to 022
fn-except-node-args-015 to 021

Constr-cont-constrmod-1 to 8
Constr-cont-nsmode-5 to 10
Constr-compelem-compname-8
Constr-compattr-compname-8
Constr-docnode-constrmod-1 to 4
Constr-comppi-compname-8 to 9
Constr-compelem-constrmod-1, 2, 5, 6

casthcds1 to 42
schema-import-4 to 31

The first three groups don't seem to care whether the input is typed or untyped, the result will be the same either way.

The last two groups are positioned within the SchemaImport part of the test group hierarchy.

So the problems are confined to the (appallingly named) Constr- groups.

Some of these appear to have alternate results. In some cases the alternate results appear to be tagged 'input-document="typed"' or 'input-document="untyped"' In fact, the only tests that use this attribute are

Constr-cont-constrmod-4
Constr-cont-constrmod-8
Constr-cont-nsmode-5
Constr-cont-nsmode-6
Constr-cont-nsmode-7
Constr-cont-nsmode-8
Constr-compelem-constrmod-8

which is (almost) a subset of the tests listed above.

The following query retrieves all tests whose source documents have a schema:

declare namespace c='http://www.w3.org/2005/02/query-test-XQTSCatalog';
//c:test-case[c:input-file=/*/c:sources/c:source[@schema]/@ID]/<t group='{string-join(ancestor::c:test-group/c:GroupInfo/c:title, '/')}' file='{string(c:input-file)}'>{@name}</t>

Most of these, but by no means all, are in the "Optional Features" part of the catalog. Among those in the Minimal Conformance section are ForExprTypeNNN and orderByNNN (source document orderData), LocalNameFromQNameFuncNNN and namespaceURIFromQNameFuncNNN (source document QNameSource), and the three CatalogNNN tests.

The ForExprTypeNNN tests have been split into two groups: where the results depend on schema validation they have been moved into the "Schema Import" group (even if they do not import a schema); in other cases they have been left in the "Minimal Conformance" group.

For the NamespaceURIFromQName and LocalNameFromQName tests, a different strategy has been employed again: although these tests were clearly originally written to use typed input, they have been modified to cast the typed input to a string so that they no longer depend on schema validation (and, as a side-effect, no longer test what they were designed to test).

Proposal: follow the crowd. In future, tests that depend on schema-validating the input should go in the "Schema Import" group. Tests that currently have two results for the two cases should be split into two tests. Scrap the almost-unused @input-document='typed|untyped' attribute which appears to have been an earlier attempt to solve this problem.

I am marking this bug as resolved as it is not relevant in the QT3 as the classification is done in a  different way.