The same is true of doc, see bug#3444.

It is (or will be) documented that if the catalog specifies a context attribute
of doc or collection that the query has to be edited to use that.
Although as I say in comment 6 in bug #3444, I don't understand the logic in
distributing a test file that has the wrong query, together with instructions
on how to fix the test.

However I implemented it in my test harness, so I allowed bug #3444  to close....

see also bug #3455

Thanks David for the reference. That's a weird design. Fortunately my test harness is written in XSLT 2.0 so parsing and modifying the queries shouldn't be too difficult, but it does seem a crazy way of doing things.

(Am I allowed to cheat by calling the collection() function from my XSLT test driver and passing the result, to save the effort of modifying the query? I'm waiting to see the published rules...)

> Fortunately my test harness is written in XSLT 2.0 
doesn't everyone do that?:-)

> Am I allowed to cheat by calling the collection() function from my XSLT test
In my own case,
a) before running the test I make a directory for each collection specified in the catalog, and copy in the specified files.
b) edit the "query" to give the $input-context variable a definition that is collection(the uri to the collection directory) and then this essentially tests your directory handling collection support from saxon. I put "query" in quotes here as I don't (ever, for any tests) edit the query text but rather the XML version of it which allows me to easily make transforms from within my test harness (that's written in xslt2).

The lines between textually editing the query text, transforming some representation of that query before executing it, and not transforming it at all, but simply doing all the work of calling the function in the test harness seem to be rather blurred and probably not really specifiable given there is no defined API to the query. I still think it would be better if the doc tests used doc($some-var) and the collection tests used collection($var) then you wouldn't have to edit the query at all, and would just need to pass in
a (system-specific) string $var as the URI that represented the document or collection.

I was going to look at the current guidelines in cvs but it isn't letting me download any files

I can see
http://dev.w3.org/cvsweb/2006/xquery-test-suite/TestSuiteStagingArea/TestSuiteDocumentation/Guidelines%20for%20Running%20the%20XML%20Query%20Test%20Suite.html

but trying to see the file (or it seems, any other file) produces
Error: Unexpected output from cvs co: cvs.bin checkout: cannot find module `2006/xquery-test-suite/TestSuiteStagingArea/

David

Trying to guess what the test driver is supposed to do, in the absence of any
documentation. Looking at the intent of fn-collection-10, the query reads like
this:

(: Name: fn-collection-10 :)
(: Description: Test that two uses of fn:collection are stable.:)

(: insert-start :)
declare variable $input-context external;
(: insert-end :)

let $c1 := $input-context
let $c2 := $input-context 
for $c at $p in $c1
return $c is $c2[$p]

My first thought was to modify it to say:


declare variable $input-context := collection('collection2');

But the intent of the test seems to be that collection() is called twice, so
instead it's the references to $input-context that must be changed:

let $c1 := collection('collection2')
let $c2 := collection('collection2')

Is this the correct interpretation?

Couldn't the text that needs to be replaced in the query be flagged by some
more distinctive markup, rather than making it look like an XQuery variable
reference? For example

let $c1 := ##input-context##

or even

let $c1 := collection('##input-context##')

Will the name of the variable always be "input-context"?

? 

> declare variable $input-context := collection('collection2');
That's essentially what I do.
Although actually what I do is edit the (in memory) xsl version of this so that the external parameter has that definition.
As you observe, because I just define the variable in terms of collection, rather than replace every occurrence of the variable by an explict call to collation, it probably defeats the purpose of the test. (I hadn't noticed this before now)

However I think that expecting the testers to make changes outside of the
(: insert-start :) .. (:insert-end :_ area isn't really reasonable.
Although some comments on that below.


> let $c1 := ##input-context##
eek I'd rather the xquery test file always used xquery syntax.

> let $c1 := collection('##input-context##')
That would be alright although I don't see why it couldn't be
collection($collection-uri)
with $collection-uri being an external string valued variable that the catalog could specify as needing to be set to some system-specific string.

> Will the name of the variable always be "input-context"?
It's specified in the catalog(ue)
<input-file role="principal-data" variable="input-context" context="collection">foo
means edit $input-context to refer to (or be) collection("some-uri-denoting-the-collection-foo")

$input-context

(In reply to comment #5)

I'm sorry that you are finding this to be such a bother. We've tried to document this in our Guideline for Running the XML Query Test Suite (and in the catalog schema), but perhaps we weren't clear enough.

We started with the desire to allow input documents to be provided either by external variables or my invocations of the fn:doc function. In most test cases we allow the writer of the test harness to make this decision. Test cases in which only fn:doc is allowed are identifed with context="doc". The external variable name (or names) used in the query is specified in the test-case element. It is usually $input-context, but not always. When the fn:doc transformation is done, it is the test harness that chooses the URI.

When we added tests for fn:collection, we decided to use the same mechanism, specifying context='collection' or context='default collection'. Again, this allows the test harness to choose the URI.

Carmelo, please modify our Guidelines for Running the XML Query Test Suite in the following ways:

- make clearer that the query must be transformed based on the setting of the context attribute. If the attribute is not present, then the test harness can choose any one of the transformations (including the context="external" identity transformation).

- for each transformation, show the appropriate context='...' attribute.

- Add the transformation for context="collection", which will turn $var in a query into fn:collection('some_URI"). 

OK, I've now made the changes (but in an ad-hoc way that will require constant maintenance as the test suite changes, for example it can only handle two variables). But I still don't understand why this is necessary! Why can't the test simply say

collection($input-context1)

and have the URI be passed as a parameter? Then the driver would only have to fiddle with URIs, rather than fiddling with the source of the query.

It's a big step from saying the driver is allowed to change the query source to saying that it must do so.

> When we added tests for fn:collection, we decided to use the same mechanism,

The problem is that it's not the same mechanism, it just looks similar, and it's being  used for an entirely different purpose.

In the rest of the test suite the implementer is _allowed_ to edit the query if necessary (it is very unlikely to be necessary as support for external variables is mandatory, so just running the query as is, while supplying the variable values should work in all cases)

In the case of the doc and collection functions the implementer is _forced_ to edit the query as the query, as distributed, does not test the specified feature.

As pointed out in comment #5 the edits required to make this work in the case of
fn-collection-10 are _far_ more invasive than edits required in other cases. the edits are not resticted to just editing a variable definition between the marker comments to supply a value, every occurrence of the variable in the whole test has to be replaced by a call to collection()

In my test harnes I (effectively) edit the query to say

declare variable $input-context := collection('zzzz')


let $c1 := $input-context
let $c2 := $input-context 
....

But then as Michael points out, collection() is only being called once so the test is not testing its stability. Instead the test must be edited to say



let $c1 := collection('zzzz')
let $c2 := collection('zzzz')

with the edits happening within the expression body I don't think it's reasonable to expect the test harness to do this.

Why can't the doc and collection tests have the much simpler structure so that they look like

(: Name: fn-collection-10 :)
(: Description: Test that two uses of fn:collection are stable.:)

(: insert-start :)
declare variable $input-uri external;
(: insert-end :)

let $c1 := collection($input-uri)
let $c2 := collection($input-uri)
for $c at $p in $c1
return $c is $c2[$p]


so that, as in all other cases, it is not _necessary_ to edit the query text, and it is sufficient just to pass in a value for the external variable (which the catalog would specify needs to be set to a system specific uri denoting the collection)

David

David, Michael, 

I hear what you are saying, but must decline to change the test cases and documentation in the way that you suggest.

I believe this approach to testing fn:doc and fn:collection, while ineligant in your eyes, is workable.

We will improve our documentation in this area and I will call people's attention to it when I announce XQTS 1.0.

Please mark this closed if you agree with this resolution. If you would like to bring this to the WGs for consideration, then please reopen it.
 

I'm not entirely happy with this resolution, so I reopen.

I agree that it is very messy that testers must do these extensive edits, and I think it's too much to ask. I neither understand why the solution preferred by several testers, to pass in the URI via a variable, can be the resolution.

I'll continue the discussion member only, on member-query-test@w3.org.

If anyone else(David, Michael) is unpleased with the resolution it might be an idea to state so, since I might re-close if I find the resolution ok.


Frans

> If anyone else(David, Michael) is unpleased with the resolution it might be an
> idea to state so, since I might re-close if I find the resolution ok.

I think the resolution is completely inexplicable, and the problem should be fixed not just classified as "wontfix" without any indication of why not.

I reopened the similar report on the doc tests, see
http://www.w3.org/Bugs/Public/show_bug.cgi?id=3444#c10

David

On the (hopefully false) assumption that the tests will stay as they are to XQTS 1.0, I had a look at the new documentation, below are some comments (cvs 1.21).


> Setting a Specific Context Item
This section (which is misnamed) is a mixture of the old and new version of the doc tests and is almost completely misleading. For example it says
> "implementers may bind a URI value to the external variable"
Oh it would be nice if that were still true!
Given that the mechanisms for doc and collection are the same I suggest this section is deleted and merged with the following one.


> Customizing fn:collection() Tests
This special processing applies equally to the doc tests and so the section title (and some of the examples) need to reflect that. I don't think any tester would have a chance of inferring from this documentation, that as confirmed in bug #3444 c9 that transformations are required for the doc tests, unless they have been following the bug database.


> based on the value of the "context" attribute of the <inputFile> element. 

It's <input-file>  not <inputFile> 

>  &ltinput-document>reviews</input-document>
< not &lt;

In addition to the transformations described, you need to document that the external variable declaration must be deleted, as otherwise you'll just get an error that it is not set when you run the transformation.


> Then the references to external variables become fn:collection(...) and should > be customized as follows:


the actual suggested "customisation" is a syntax error

> fn:collection("bibs.xml", "reviews.xml")

collection can only take 1 argument not 2, which must be set to a system-specific URI which denotes the collection.


> Then the references to external variables become fn:collection() and should be customized as follows:
It would be helpful to say explictly here that the default collection for the system must be set to a URI representing the specified documents.
Incidentally a conforming XQuery application is _not_ forced to have any way of changing the default collection from its intial value of "none".
so you may want to allow [err:FODC0002].

I didn't jump in earlier, but I have to agree with the commentators.
Having tests for a function where that function doesn't appear in
the test is just bizarre.  It is much easier to modify a test driver
to pass URIs in some cases, properly marked up, than to have to edit
certain test cases in a not-easily-automatable fashion.

(In reply to comment #14)

I note the documentation's changed again this morning. I think all the issues in comment #14 (and several others) are addressed. Of course I still think the current mechanism places an undue burden on the test implementor but at least it is clearly and accurately documented now, so thanks for that!
The new documentation highlights several other context= types that I certainly haven't implemented (or implemented correctly) in my harness, I will need to check. What's the context attribute for type 7 (implementation defined function)?

David

The XSL and XQuery WGs discussed this on Tuesday and asked me to change our tests for fn:doc and fn:collection in the way that you have been suggesting.

I have made these changes (reflected in XQTS_current.zip v1.28). Rather than explain the changes here, I suggest that you look at the new "Test Cases with fn:doc and fn:collection" section in our Guideline for Running the XML Query Test Suite.

Please mark this bug report as closed if you agree with our resolution.

Thanks for the changes which are a great improvement.

One comment, however: the tests that use a default collection continue to use "input-file" rather than "input-URI", and it's unclear how the test driver is supposed to distinguish between

<input-file role="principal-data">collection2</input-file>

used in (say) fn-collection-10d, from

<input-file role="principal-data">works-mod</input-file>

used in externalcontextitem-22

I think the first case should be input-URI rather than input-file.

It would also be useful to distinguish between 

<input-URI role="principal-data" variable="input-context">collection2</input-URI>

used in fn-collection-10, and

<input-URI role="principal-data" variable="input-context">works-mod</input-URI>

used in fn-doc-5; the test driver may need to treat document URIs and collection URIs differently. The obvious way to achieve this would be to write role="collection" in the case where a collection URI is used.

More changes in response to your feedback and David's feedback.

We've moved from using a missing variable attribute to explicit elements:

<contextItem role="principal-data">works-mod</contextItem>

and

<defaultCollection role="principal-data">collection1</defaultCollection>

Out documentation has been updated to provide further details. These changes are reflected in XQTS_current.zip (revision 1.29).

I'll mark this as resolved for the moment. Please close this report if you agree with this resolution.

Thanks. All working fine now.