In practice, people are extending the language. 

We originally offered option declarations (http://www.w3.org/TR/xquery/#dt-option-declaration) and extension expressions (http://www.w3.org/TR/xquery/#id-extension-expressions) for this purpose. My memory is that we decided, at the time, not to add language to the spec allowing people to extend the syntax in other ways, but other people seem to remember differently, and I have not done the archaeology needed to be sure.

Obviously, these extensions themselves are not part of XQuery. We can't test for syntactic extensions in our test suites without knowing what the extensions are, which is just impractical. 

I suspect the right answer may be to add language in the conformance section requiring vendors to specify any syntactic extensions, and to very explicitly warn vendors that the XQuery Working Group regards any such vendor extensions as outside the scope of XQuery, and that we might well add to the XQuery languages in ways that conflict with any extensions they make to the syntax.

We might also require a mode that does not accept any extensions.

The specification is not completely silent on this matter if you look at the text of error XPST0003:

<quote>
err:XPST0003
It is a static error if an expression is not a valid instance of the grammar defined in A.1 EBNF.
</quote>

This is the error that is raised in case of a parse error:

<quote>
A parse error is raised as a static error [err:XPST0003].
</quote>

I think the right way forward is probably along these lines:

1. The XQuery specification governs only what we define. Any extensions to the BNF are not part of the XQuery language.

2. Users can easily determine whether a query uses grammar not found in our EBNF by using the XQuery applets. Thus, there is no need to ask vendors to provide such a facility.

3. As a Working Group, we control the XQuery BNF. If a vendor extends the BNF in their product, that has no bearing on extensions we make in the future, even if they are incompatible with vendor extensions.

I will make a more formal proposal along these lines unless someone convinces me to take a different approach.

I propose that we add this section to Conformance:

<add>
5.2 BNF Extensions

The XQuery language is defined in this specification. Any extensions to the
BNF are not part of the XQuery language. Users can easily determine whether a query uses grammar not found in our
EBNF by using the XQuery applets [APPLETS]. 

As a Working Group, we control the XQuery BNF. If a vendor extends the BNF
in their product, that has no bearing on extensions we make in the future, even
if they are incompatible with existing vendor extensions.
</add>

In addition, we should consider whether to soften this error text:

<quote>
err:XPST0003
It is a static error if an expression is not a valid instance of the grammar
defined in A.1 EBNF.
</quote>

(In reply to comment #3)
> I propose that we add this section to Conformance:
> 
> <add>
> 5.2 BNF Extensions
> 
> The XQuery language is defined in this specification. Any extensions to the
> BNF are not part of the XQuery language. Users can easily determine whether a
> query uses grammar not found in our
> EBNF by using the XQuery applets [APPLETS]. 
> 
> As a Working Group, we control the XQuery BNF. If a vendor extends the BNF
> in their product, that has no bearing on extensions we make in the future, even
> if they are incompatible with existing vendor extensions.
> </add>

Ooops - change that paragraph to:

The XQuery BNF is found in [Appendix A: XQuery BNF]. If a vendor extends this BNF in their product, that has no bearing on the grammar of future versions of this specification, which may contain grammatical productions that conflict with existing vendor extensions.


> In addition, we should consider whether to soften this error text:
> 
> <quote>
> err:XPST0003
> It is a static error if an expression is not a valid instance of the grammar
> defined in A.1 EBNF.
> </quote>
> 

Jim Melton and I attempted to resolve issues raised at the last meeting with the following language:

-------------

5.2 EBNF Extensions 

The XQuery language is defined in this specification. The XQuery EBNF is defined in [Appendix A: XQuery EBNF]. Any extensions to the EBNF are not part of the XQuery language. Users can determine whether a query uses grammar not found in the EBNF by using the XQuery applets [APPLETS].  

An implementation that otherwise conforms to this specification may also recognize and evaluate syntax that is not part of the XQuery EBNF.  Such an implementation remains conformant to this specification, provided such extensions do not otherwise conflict with the syntax and rules of this specification.

Future versions of XQuery may extend the EBNF in ways that conflict with extensions provided by implementations of this specification, thus causing such implementations to become non-conforming. 

----------------

<quote>
err:XPST0003
It is a static error if an expression is not syntactically valid, using the syntax defined in Appendix A.1 EBNF plus any implementation-defined EBNF extensions.
</quote>
</quote>

[personal response:]

I think the phrases "EBNF extensions" and "extensions to the EBNF" aren't
quite right. EBNF is a system of notation, so, properly speaking, "EBNF
extensions" would be extensions to that notation (in the same sense that
EBNF is itself an extension of BNF), which is not what we're concerned with
here. Instead, I suggest changing (in the text of Comment #5) all
occurrences of "EBNF" to "grammar", and the one occurrence of "grammar" to
"syntax".

Also, in the blurb for XPST0003, I suggest changing "Appendix A.1 EBNF" to
"Appendix A XQuery Grammar", since it takes more than just A.1 to define
the grammar. (For example, an implementation could extend the syntax for
IntegerLiteral (which appears in A.2) to include hexadecimal literals.
Presumably we would want such an extension to be covered by XPST0003's
"plus" clause.)

In the database community, there seems to be a high level of user tolerance for vendor extensions to standard languages. Users say they want portability, but when it comes down to it, they put other requirements higher on their list. So this proposal seems reasonable for XQuery. 

But in the world of programming languages, especially for use on the web, and especially in the XML community, I get the impression that users regard vendor extensions as poison, unless the extensions are implemented within a framework that allows applications to remain portable - which for syntax extensions is not the case.

For XPath there is the additional complication that every host language (including those defined by W3C working groups) seems to want to define its own subset and/or superset. I believe it is very much in users' interests that XPath should be the same language wherever it is used. I would like to see a much stronger statement that syntax extensions to XPath are banned. Of course, we can't stop people defining XPath-like languages that have a different syntax, but we should come down very firmly against anyone describing such a language as XPath. 

Another way to resolve this issue would be to add the following paragraph to section 2.2.3, Expression Processing:

"Any syntactic extensions to the [XQuery: XQuery] [XPath: XPath] language are implementation-defined. The effect of syntactic extensions, including their error behavior, is implementation-defined. Syntactic extensions may be used without restriction to modify the semantics of the [XQuery: query] [ XPath: XPath expression]."

I think the text in comment #8 is too tolerant. If the language allows syntax that we don't permit, then the language being implemented is not XQuery|XPath. The effect is not implementation-defined because the language processor is not an "implementation". (Saying it is implementation-defined says that a conformant implementation must do certain things, and this acknowledges the possibility that implementations of such languages might be conformant to our specification.)

I would prefer something like:

"It is possible to define languages that are extensions of XQuery|XPath, in the sense that the user can choose to use a subset of the language that follows the syntax and semantics of XQuery|XPath as defined in this specification. Such languages are not interoperable or future-proof; their behavior is outside the scope of this specification; and neither the language nor its implementation can claim conformance with this specification. However, the specification of such a language may describe it as an extension of XQuery|XPath provided that every query|expression that conforms to the XQuery|XPath grammar behaves as described in this specification."

On 2009-Jan-20, the Working Groups decided to adopt comment #8 for XPath, comment #9 for XQuery.

(In reply to comment #10)
> On 2009-Jan-20, the Working Groups decided to adopt comment #8 for XPath,
> comment #9 for XQuery.
> 

Oops, I got that backwards ;->

Jonathan