This is an archived snapshot of W3C's public bugzilla bug tracker, decommissioned in April 2019. Please see the home page for more details.

Bug 13445 - test-group VersionProlog is a mess
Summary: test-group VersionProlog is a mess
Status: RESOLVED MOVED
Alias: None
Product: XML Query Test Suite
Classification: Unclassified
Component: XML Query Test Suite (show other bugs)
Version: 1.0.3
Hardware: All Windows 3.1
: P2 normal
Target Milestone: ---
Assignee: Benjamin Nguyen
QA Contact: Mailing list for public feedback on specs from XSL and XML Query WGs
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-07-29 06:46 UTC by Michael Dyck
Modified: 2016-04-12 13:49 UTC (History)
3 users (show)

See Also:


Attachments

Description Michael Dyck 2011-07-29 06:46:45 UTC
Checking the CVS history of XQTSCatalog.xml and of the files
in Queries/XQuery/Expressions/PrologExpr/VersionProlog/,
I find an alarming amount of churn, all revolving around
the following paragraph in 4.1 [Version Declaration]:

    If a version declaration is present, no Comment may occur
    before the end of the version declaration. If such a Comment
    is present, the result is implementation-dependent.

(I refer to this as "the comment-before-versiondecl clause".)

I've appended the whole history below. The upshot is:

(1) The test cases that originally (or nearly originally) were intended to
    address this clause no longer do so.

(2) For test cases
        version_declaration-001
        version_declaration-002
        version_declaration-003
        version_declaration-004
        K-VersionProlog-1
    the catalog says
        <expected-error>*</expected-error>
    which is inconsistent with the query.
   
So:

Is there any point having test cases with comment-before-versiondecl,
when the resulting behaviour is implementation-dependent? (Do the
results tell us anything?)

If yes, then we should undo most of the changes to all the
version_declaration-nnn test cases.  If no, we should probably
just remove them.


-----------------------------------------------------------------

Abbreviations:
    vd      = version_declaration
    pv      = prolog-version
    KVP     = K-VersionProlog
    error=* = <expected-error>*</expected-error>
    TTF     = Testing Task Force


2005-10-05:
Ravi Chennoju (rchennoj) adds test cases vd-001 through vd-009
(XQTSCatalog.xml rev 1.349):
    001-005 expect no error
    006-008 expect XPST0031
    009    expects XPST0003
All have a comment preceding the VersionDecl, and so the behaviour
is implementation-defined according to the XQuery spec, but Ravi is
presumably not aware of this.

2005-11-01:
Ravi adds XQST0038 to vd-005's expectations.
(XQTSCatalog.xml rev 1.389)

2006-02-06:
Carmelo Montanez (cmontane) adds vd-010, expects XQST0087.

2006-02-15/17:
Apparently, Carmelo becomes aware of the comment-before-versiondecl
clause, because he adds pv-1 thru pv-12, all of which have
comment-AFTER-versiondecl. In fact, pv-4 through pv-12 are simply
copies of vd-001 through vd-009 with the versiondecl moved up.
At about the same time, he adds 
    <expected-error>*</expected-error>
to vd-001 through vd-010 (XQTSCatalog.xml rev 1.522), presumably to
reflect their implementation-dependent behaviour.

2006-05-11:
In Bug 3275, Jinghao Liu thinks vd-1 shouldn't expect error=*.  Andrew
and Ravi agree that it's there because of the comment-before-versiondecl
clause.

[So, the intent of the tests seems fairly clear: test cases vd-*
exercise the imp-dep behaviour of comment-before-versiondecl, and
pv-* exercise the non-imp-dep behaviour that vd-* were originally
intended to cover. Call this the Revised Intent. ]

---

2006-05-22:
On the TTF mailing list, Franz Englich (fenglich) rightly points out
that vd-005 thru vd-010 expect both a specific error code and error=*,
which is redundant. In each case, one or the other should be removed.
http://lists.w3.org/Archives/Member/member-query-test/2006May/0045.html

2006-05-22:
cmontane removes error=* from vd-009 (XQTSCatalog.xml rev 1.603)

2006-06-08:
TTF meeting 64 decides that in each case, error=* should be removed:
> ACTION TTF-064-03: Carmello to fix version_declaration-010 test and
> remove '*' for expected error and use specific error
> ACTION TTF-064-04: Ravi to fix version_declaration-00[5,6,7,8] test
> and remove '*' for expected error and use specific error
http://lists.w3.org/Archives/Member/member-query-test/2006Jun/0054.html

(That is, it seems the TTF forgot that error=* was there to cover the
imp-dep behaviour of comment-before-versiondecl. As far as I can see,
they should have decided to remove the specific error.)

2006-06-08: cmontane removes error=* from vd-010 (rev 1.618)
2006-06-09: rchennoj removes error=* from vd-005 thru 008 (rev 1.619)

(This leaves vd-001 thru 004 expecting a value or error=*, still in
line with the Revised Intent, but vd-005 thru 010 expecting specific
errors, which is incorrect, and will bite us later.)

---

(2006-07-19: cmontane moves vd-005)

(2006-09-15:
In Bug 3731, Per Bothner questions error=* for vd-001 thru 005, but he's
just misreading the metadata.)

2007-01-26
In Bug 4281, Michael Kay points out the comment-before-versiondecl in
vd-010, which is inconsistent with its specific error message.

2007-07-23
In vd-010, Andrew moves up the versiondecl, making it consistent with
the error message, though inconsistent with the Revised Intent.
(2007-08-09: In the Bug, Andrew reports making this change.)

2007-11-22
fenglich adds KVP-1 through KVP-5, all with comment-before-versiondecl.
The comment for KVP-1 clearly indicates that this is intentional, but oddly,
only KVP-1 has error=*; the others expect a specific error or a value.

2009-05-20
In Bug 6934, Josh Spiegel lists many cases of comment-before-versiondecl,
including vd-* and KVP-*.  His implementation, which (conformantly) raises
an error in such cases, only passes vd-001 thru 0004.

2009-10-14
To "fix" bug 6934, fenglich moves up the versiondecl in vd-001 thru 008, and
KVP-1 through 5. However:

    vd-001 thru 004 and KVP-1 still expect error=*,
    which is inconsistent with their (changed) queries.

    None of vd-* adhere to the Revised Intent any more.
    That is, there are no longer any tests that exercise the
    comment-before-versiondecl clause. The vd-* queries
    are basically duplicates of pv-* queries. 
    (But missing a couple changes that some pv-queries received.)

xxx
Comment 1 Michael Kay 2011-08-01 12:57:04 UTC
Just to add to this, I'm sure the conversion to QT3 format hasn't helped matters. Generally, the automatic conversion moved comments out of the query into the metadata, and where comments in the query were part of the essence of the test, manual intervention was needed. If running the query produced the correct result, the need for this intervention might well have gone unnoticed.

(These tests are by no means the only ones that don't do anything useful, of course; I didn't regard that as something to worry about during the conversion).
Comment 2 Andrew Eisenberg 2012-02-10 19:45:34 UTC
> Is there any point having test cases with comment-before-versiondecl,
> when the resulting behaviour is implementation-dependent? (Do the
> results tell us anything?)

I agree that these test cases won't tell us anything with regards to our specification. I would still leave them in the test suite. Every so often there will be an implementation that hasn't prepared for this situation and ends abnormally. Leaving these test cases in the test suite will help catch such omissions.
Comment 3 O'Neil Delpratt 2016-04-12 13:49:50 UTC
This bug issue was fixed in the bug #14955 for the QT3 test suite therefore I am marking this resolved - moved