Please let's stop torturing ourselves in this way.

Let's be clear: with static errors, there is never one absolutely correct error code. A static error means that your input text is not in the space of legal XQuery input texts. The art of finding a good error message is to identify the nearest legal XQuery in the space of all possible inputs, and to tell the user how the actual input differs from the legal XQuery that you think he intended. That will never be a precise art, and we can never legislate that all products will search for the nearest legal XQuery in the same way.

That's why the test suite rules clearly say that producing the right error code is not necessary to claim a pass. This is particularly true for static errors. (For dynamic errors, the situation is a little different, and with try/catch there is a good incentive to try to achieve interoperability. But for static errors there is none.)

So please let's stop beating our heads against this brick wall. 

If we add XPST0003 as an expected result for this test then we should add it as an expected result for every single test that produces a static error, and that would be crazy.

Thanks for the quick reply, here are my thoughts:

I agree with the point you are making, and I am not trying to "beat our heads against a brick wall", I am trying to make the XQTS better by reporting some discrepancies that I found.
I think you agree that feedback is a good thing, right?

So,the 
http://dev.w3.org/2006/xquery-test-suite/PublicPagesStagingArea/Guidelines%20for%20Running%20the%20XML%20Query%20Test%20Suite.html
section "Comparing Results" state that:
"Error: The expected result of the test case is an error, identified as an eight-character error code (e.g., XPST0003). The result of a test is true if the implementation raises an error. However, raising an error because an implementation does not support the feature is not considered a correct result."

This can be improved because:
1- "The result of a test is true if the implementation raises an error."
This is vague and because of this too permissive IMHO.

2- the part "However, raising an error because an implementation does not support the feature is not considered a correct result."
is wrong:
XQTS also has tests for XQuery 1.1 features. These tests accept raising an error if the implementation does not support the feature when the implementation state that it supports XQuery 1.0, correct?

Instead, let's add the suggestion you made:
"If the expected result of a test is a static error, a test harness may report the test as passing no matter the static error code it returns. 
In the case of dynamic errors, the return error code must match one of the expected error codes of the test."

Does this makes sense?

As regards tests that have dependencies (e.g. on Static typing or XQuery 1.1), we really need to try to move to a situation where the catalog describes what results are expected under what scenario, rather than just listing all possible results for all scenarios.

As regards static vs dynamic errors, I agree with what you are saying: for dynamic errors we should strive to achieve interoperability at the level of error codes, but with static errors this is neither feasible nor useful.

I agree: what should we do with this reported issue in the meantime:

- leave it opened for now
- should we assign it to someone
- close it because this is a known issue?

[Personal response]
You don't actually say *why* you think XPST0003 should be added as an alternative result for this test-case.

The description of XPST0003 says:
"It is a static error if an expression is not a valid instance of the grammar defined in A.1 EBNF."

The query is basically:
    "�"
which is an instance of the StringLiteral production, containing an instance of the CharRef production. (That's under XQuery; it's different for XPath.)

The production for CharRef is a cross-reference to the CharRef production in XML 1.0 or 1.1 (imp-def choice), which in either case is:
    CharRef ::= '&#' [0-9]+ ';' | '&#x' [0-9a-fA-F]+ ';'
And
    �
is a syntactically valid instance of that production.

Now, the CharRef production in the XML specs is also annotated with a well-formedness constraint:
    Characters referred to using character references
    MUST match the production for Char.
And the character referred to by
    �
does not match that production. Is *that* why you think the query is not an instance of the XQuery grammar?

If so, I would point out that well-formedness constraints are constraints on *XML documents*, not XQuery expressions/modules. That is, I believe that when XQuery's grammar cross-reference another spec's production, it only "imports" the EBNF of that production, not any annotations it might have. (That's why the XQuery spec needs to separately define the conditions for an error such as XQST0090.)

More formally, XQuery's A.1.1 defines the cross-reference notation as follows:
    [http://www.w3.org/TR/REC-example/#NT-Example]
    matches any string matched by the production defined in
    the external specification as per the provided reference.
And I believe it's correct to say that
    �
matches the CharRef production in the XML spec. It doesn't satisfy the associated well-formedness constraint (or more precisely, a textual object containing such an instance would not meet that constraint, and so would not be a well-formed XML document), but that's a separate matter.

Dear Sorin, 

I suggest closing this bug with no changes to the TS, unless you strongly feel otherwise. I can try to suggest a discussion on the general issue of static error messages to the WG, if you think the matter is important.

Michael,

I totally agree with your comment #3.

However, this is not what is being done today. Hence, let's try to achieve that.

I think we should not close the bug, and bring this issue up for discussion.

Best regards
Dana


(In reply to comment #3)
> As regards tests that have dependencies (e.g. on Static typing or XQuery 1.1),
> we really need to try to move to a situation where the catalog describes what
> results are expected under what scenario, rather than just listing all possible
> results for all scenarios.
> 
> As regards static vs dynamic errors, I agree with what you are saying: for
> dynamic errors we should strive to achieve interoperability at the level of
> error codes, but with static errors this is neither feasible nor useful.

Based on my analysis in comment #5, I maintain that adding XPST0003 to the expected-errors for K2-Literals-1 would be incorrect, and thus that this bug should be resolved as INVALID.

I too agree with Michael Kay's comment #3 (advocating widespread changes to the XQTS catalog), but he raised it in response to comment #2 (discussing wording of the XQTS documentation), which is (given the above) irrelevant to the original point of this issue (the expected-errors for a particular test-case).

That is, while both comment #2 and comment #3 have merit, they should be raised/discussed separately from this issue, and not keep it from being closed.

I disagree with closing this bug until we discuss the status of the static error codes, as stated in comment #3. Depending on the result of that discussion there might be different possible resolutions
 for this bug.

(In reply to comment #1)
> Please let's stop torturing ourselves in this way.
> 
> Let's be clear: with static errors, there is never one absolutely correct error
> code. 

Mike,

what you say in the sentence above might be true.

Could  you please point us in the legal text that says that all static errors are
interchangeable  ?

Sorry, we are not very sophisticated (we are simple nerds). 
We only implement and test what's written.

For us, if it's not written, it doesn't exist and is not true.

Thanks
Dana

In response to comment #10, the specification states

If more than one error is present, or if an error condition comes within the scope of more than one error defined in this specification, then any non-empty subset of these errors may be reported.

Suppose the user wrote the query

return(a, b, c) + "

Intuitively, we might say there are two errors here: use of an unknown/undeclared function "return", and there is an unclosed string literal. A processor is allowed to report any subset of these two errors. Another processor might decide that the real error is the use of a FLWOR expression with no for or let clause; a third processor might decide that the user intended to write "return (a, b, c) + " and that their error was to omit the opening quotation mark. So the decision as to "what errors are present" amounts to deciding how the query as written differs from some legal query that is sufficiently similar, and processors may differ as to how they do this.

So the total set of errors present in the query is itself a processor judgement, and the specification explicitly allows the processor then to report only a subset of these errors.

Some people might read the specification of the processing model in 2.2.3.1 as if it requires "parsing" to complete before variable and function names are resolved, meaning that XPST0003 takes precedence over any other error. That's not my reading, for a number of reasons:

(a) The text doesn't say the steps are sequential. It says "The query is parsed. Variable and function names are bound." It doesn't say that one step completes before the other starts. 

(b) it's perfectly normal for names to be bound during parsing where possible, and if the WG had intended to disallow this, it would have said so.

(c) a literal reading of the processing model leads to all kind of absurdities, for example it says "Before a query can be processed, its input data must be represented as an XDM instance."

(d) the statement that any subset of static errors can be reported explicitly contradicts any assertion that some static errors take precedence over others.

I think one can defend any static error message that has the property "if the user fixed this, and also made some other unspecified changes, then the query would be correct". And that means one can defend any static error message whatsoever.

The thing is that in the spec it's stated that the production from the XML spec is imported, but it's not specifically stated that the restrictions are not imported.

Anyway, I have marked this bug as RESOLVED INVALID.