Bug 14955 - [QT3TS] test-set 'prod-VersionDecl' is also a mess
[QT3TS] test-set 'prod-VersionDecl' is also a mess
Status: RESOLVED FIXED
Product: XPath / XQuery / XSLT
Classification: Unclassified
Component: XQuery 3 & XPath 3 Test Suite
Working drafts
All All
: P2 normal
: ---
Assigned To: Michael Dyck
Mailing list for public feedback on specs from XSL and XML Query WGs
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2011-11-28 21:29 UTC by Michael Dyck
Modified: 2012-10-02 15:55 UTC (History)
4 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Michael Dyck 2011-11-28 21:29:49 UTC
QT_TS3's test-set 'prod-VersionDecl' derives from XQTS's test-group
'VersionProlog'. So prod-VersionDecl inherits the mess described in
Bug 13445 comment #0. In addition, the XQTS->QT_TS3 conversion process
caused the test queries to lose their comments (as Michael Kay pointed
out in Bug 13445 comment #1), some of which were (at least at one time)
essential to the point of the test.

So I'll echo the question posed in the earlier bug...

Since, in XQuery 3.0, the result of comment-before-versiondecl is
implementation-dependent, is there any point having test cases with
comment-before-versiondecl? (Do the results tell us anything?)
Comment 1 O'Neil Delpratt 2012-05-18 09:45:57 UTC
Hi Michael, do think you could propose a way forward for these tests
Comment 2 Michael Dyck 2012-05-22 06:36:39 UTC
In Bug 13445 comment 0, I suggested two ways to clean up the mess: undo most of the changes to all the version_declaration-nnn test cases, or just remove them.
Comment 3 O'Neil Delpratt 2012-05-23 11:48:28 UTC
To move forward to resolve this bug I have made changes to the tests in question.

Action taken: Put back in place the comments surrounding the version declaration to the following tests. Therefore, we expect only an error (i.e. <error code="*"/>), but of course the result is "implementation-dependent":

version_declaration-001
version_declaration-002
version_declaration-003
version_declaration-004 
K-VersionProlog-1

Split the test into two versions:
version_declaration-006  - Expected error with dependency XQ10
version_declaration-006a - created test case which should work with XQ30+ dependency

Please can you Michael confirm that changes are acceptable for the test-set prod-VersionDecl.
Comment 4 Tim Mills 2012-05-25 10:53:50 UTC
Why do tests

version_declaration-001
version_declaration-002

only expect an error?  Why is ignoring the comments no longer permitted?
Comment 5 Tim Mills 2012-05-25 10:58:45 UTC
Similarly for K-VersionProlog-1.

If comments are ignored, the remaining queries

version_declaration-003
version_declaration-004 

do have errors (use of undefined $input-context).
Comment 6 O'Neil Delpratt 2012-05-25 11:18:06 UTC
The expected outcome for the following tests should accept the results of ignoring the comments also:

version_declaration-001
version_declaration-002
version_declaration-003
version_declaration-004

I will make the change to the FOTS, Is this acceptable?
Comment 7 Tim Mills 2012-05-25 11:23:15 UTC
Agreed.  And K-VersionProlog-1 too please.  Or do you disagree about this one?
Comment 8 O'Neil Delpratt 2012-05-25 13:44:05 UTC
Thanks. Bug fixes applied to the tests:
version_declaration-001
version_declaration-002
K-VersionProlog-1
Comment 9 Tim Mills 2012-05-25 13:52:24 UTC
Confirmed fixed.  Thanks.
Comment 10 Michael Dyck 2012-07-17 02:25:25 UTC
(I finally have some time + inclination to look at this again.)

> To move forward to resolve this bug I have made changes to
> the tests in question.

I would have preferred that the WG first make a decision on the basic question: Should the test-suite include test-cases where the results are implementation-dependent?

> Please can you Michael confirm that changes are acceptable
> for the test-set prod-VersionDecl.

I'd say the test-set is still a mess, though not as bad.

("vd" = "version_declaration" ...)

---------------
vd-001 & vd-002:
For each, the expectation is now any-of a specified value or <error code="*"/>. However, the spec says that the result is implementation-dependent, so in fact, an implementation that returned some value other than the 'expected' one would be conformant but wouldn't pass the test.

(This is also true of K-VersionProlog-1.)

Perhaps, if we're going to keep these test-cases, we need an element <implementation-dependent/> to properly express their expectation. (The documentation for it could also explain the point of including 'unfailable' tests in the test suite. E.g., see bug 13445 comment 2.)

---------------
vd-003 & vd-004:
The fact that $input-context is undefined (which Tim Mills pointed out in comment 5) is presumably an error introduced when migrating test-cases from XQTS into TS3. The variable certainly wasn't undefined in the corresponding XQTS cases, because $input-context was part of the XQTS-wide conventions for supplying context to test queries. Thus, the proper response to comment 5 would have been to repair the queries, not change their expectations.

----------------
vd-005 to vd-009:
These cases are (now) basically the same as prolog-version-8 to -12 respectively. There's no point having both. Generally, I'd recommend dropping the latter, but someone should first check whether the (long-standing) differences between vd-005 and pv-8 signify anything.
Comment 11 O'Neil Delpratt 2012-07-17 09:04:33 UTC
Require WG discussion
Comment 12 Michael Dyck 2012-07-23 17:50:29 UTC
Specifically:

(1) Should the test-suite include test-cases where the results are
    implementation-dependent?

(2) If so, should the test suite introduce <implementation-dependent/>
    to properly express the expectation for such test-cases?

Also,

(3) Does anyone think that the differences between version_declaration-005
    and prolog-version-8 are significant? (I.e., any objection to deleting
    the latter?)
Comment 13 Michael Dyck 2012-07-23 18:29:55 UTC
At the face-to-face meeting today,

(1) There was some support and no objection to keeping tests where the results are
    implementation-dependent.

(2) Michael Kay indicated that the test-suite already has assertions that would
    allow a test to accept any result.
    (something like: "type is item()* or error is *")

However, both of those turned out to be moot with respect to the current bug, because the WG decided to modify the spec to say that the handling of comment-before-versiondecl is not entirely implementation-dependent, but rather an implementation-dependent choice between raising a static error or ignoring the comment.
Comment 14 Michael Kay 2012-07-23 20:56:18 UTC
Just as a point of information, during the conversion of XQuery tests to the new QT3 format we found many tests that were obviously redundant, but we did not delete them because we thought there might be a need for traceability: every test in the XQuery test suite has a corresponding test with the same name in QT3. I'm not sure whether that reasoning still has any merit (or whether it ever did), but we would be departing from that principle.
Comment 15 Michael Dyck 2012-10-02 15:55:32 UTC
I cleaned up the test-set in accordance with the decision in comment #13.