Bugzilla – Bug 14955
[QT3TS] test-set 'prod-VersionDecl' is also a mess
Last modified: 2012-10-02 15:55:32 UTC
QT_TS3's test-set 'prod-VersionDecl' derives from XQTS's test-group
'VersionProlog'. So prod-VersionDecl inherits the mess described in
Bug 13445 comment #0. In addition, the XQTS->QT_TS3 conversion process
caused the test queries to lose their comments (as Michael Kay pointed
out in Bug 13445 comment #1), some of which were (at least at one time)
essential to the point of the test.
So I'll echo the question posed in the earlier bug...
Since, in XQuery 3.0, the result of comment-before-versiondecl is
implementation-dependent, is there any point having test cases with
comment-before-versiondecl? (Do the results tell us anything?)
Hi Michael, do think you could propose a way forward for these tests
In Bug 13445 comment 0, I suggested two ways to clean up the mess: undo most of the changes to all the version_declaration-nnn test cases, or just remove them.
To move forward to resolve this bug I have made changes to the tests in question.
Action taken: Put back in place the comments surrounding the version declaration to the following tests. Therefore, we expect only an error (i.e. <error code="*"/>), but of course the result is "implementation-dependent":
Split the test into two versions:
version_declaration-006 - Expected error with dependency XQ10
version_declaration-006a - created test case which should work with XQ30+ dependency
Please can you Michael confirm that changes are acceptable for the test-set prod-VersionDecl.
Why do tests
only expect an error? Why is ignoring the comments no longer permitted?
Similarly for K-VersionProlog-1.
If comments are ignored, the remaining queries
do have errors (use of undefined $input-context).
The expected outcome for the following tests should accept the results of ignoring the comments also:
I will make the change to the FOTS, Is this acceptable?
Agreed. And K-VersionProlog-1 too please. Or do you disagree about this one?
Thanks. Bug fixes applied to the tests:
Confirmed fixed. Thanks.
(I finally have some time + inclination to look at this again.)
> To move forward to resolve this bug I have made changes to
> the tests in question.
I would have preferred that the WG first make a decision on the basic question: Should the test-suite include test-cases where the results are implementation-dependent?
> Please can you Michael confirm that changes are acceptable
> for the test-set prod-VersionDecl.
I'd say the test-set is still a mess, though not as bad.
("vd" = "version_declaration" ...)
vd-001 & vd-002:
For each, the expectation is now any-of a specified value or <error code="*"/>. However, the spec says that the result is implementation-dependent, so in fact, an implementation that returned some value other than the 'expected' one would be conformant but wouldn't pass the test.
(This is also true of K-VersionProlog-1.)
Perhaps, if we're going to keep these test-cases, we need an element <implementation-dependent/> to properly express their expectation. (The documentation for it could also explain the point of including 'unfailable' tests in the test suite. E.g., see bug 13445 comment 2.)
vd-003 & vd-004:
The fact that $input-context is undefined (which Tim Mills pointed out in comment 5) is presumably an error introduced when migrating test-cases from XQTS into TS3. The variable certainly wasn't undefined in the corresponding XQTS cases, because $input-context was part of the XQTS-wide conventions for supplying context to test queries. Thus, the proper response to comment 5 would have been to repair the queries, not change their expectations.
vd-005 to vd-009:
These cases are (now) basically the same as prolog-version-8 to -12 respectively. There's no point having both. Generally, I'd recommend dropping the latter, but someone should first check whether the (long-standing) differences between vd-005 and pv-8 signify anything.
Require WG discussion
(1) Should the test-suite include test-cases where the results are
(2) If so, should the test suite introduce <implementation-dependent/>
to properly express the expectation for such test-cases?
(3) Does anyone think that the differences between version_declaration-005
and prolog-version-8 are significant? (I.e., any objection to deleting
At the face-to-face meeting today,
(1) There was some support and no objection to keeping tests where the results are
(2) Michael Kay indicated that the test-suite already has assertions that would
allow a test to accept any result.
(something like: "type is item()* or error is *")
However, both of those turned out to be moot with respect to the current bug, because the WG decided to modify the spec to say that the handling of comment-before-versiondecl is not entirely implementation-dependent, but rather an implementation-dependent choice between raising a static error or ignoring the comment.
Just as a point of information, during the conversion of XQuery tests to the new QT3 format we found many tests that were obviously redundant, but we did not delete them because we thought there might be a need for traceability: every test in the XQuery test suite has a corresponding test with the same name in QT3. I'm not sure whether that reasoning still has any merit (or whether it ever did), but we would be departing from that principle.
I cleaned up the test-set in accordance with the decision in comment #13.