This case is covered by 5.b.i

There terminology of the spec is very unfortunate here, but the phrase "the implementation of the function is implementation-dependent" basically means "the function is a built-in (as opposed to a user-written) function".

Presumably, the Function Call terminology is meant to parallel the definition of functions in the Data Model, where the implementation of a function is either "a host language expression" or "an implementation-dependent function implementation".

And anyway, the Function Call text does immediately say "(e.g., it is a built-in function ...". It seems like that should have been enough to answer the original question.

> In addition, this section does not seem to clearly differentiate where the
> DC comes from.

That's correct. You'll find the same thing for every other kind of expression: its writeup doesn't say where its dynamic context comes from.

In general, an expression E1's dynamic context "comes from" whatever expression E0 caused E1 to be evaluated, and it's the same dynamic context unless there's something about *E0* that makes it different. So it's the writeup for E0's kind of expression that would talk about that difference.

E.g., with
    let $x := 1 return f($x)
it's the section for Let Expressions, not the one for Function Calls, that tells you how to construct the dynamic context for f($x).

Re Michael and Michael,

> "(e.g., it is a built-in function ...". It seems like that should have been 
> enough to answer the original question.

I must have read it a dozen times, but it didn't click for me. What is a built-in function in light of XPath? XPath does not have built-in functions. I truly didn't get it and thought it was an omission. Does it make sense to specifically *define* what an "implementation defined function" is?

In fact, I would even prefer to distinguish the two and specifically mention functions defined by specifications and functions defined by implementations.

> an expression E1's dynamic context "comes from" whatever expression E0 caused 
> E1 to be evaluated

Makes sense to my common understanding. Still, an example I gave in bug 4378, comment 30, which I think follows this reasoning, is contested by Michael Kay. To stick to XPath:

let $f := name#0
return a/b/c/$f()

1) Using my understanding of Michael Dyck's reasoning:
Here c sets the context of $f and it should return the name of "c".

2) Using Michael Kay's comment from bug 4378, comment 30
The context is "captured" on assignment, which means that the result of the assignment is not focus-dependent anymore and will return the name of whatever is in the context (probably the parent of "a").

I can understand the merits of either, but I can only find in the spec the reasoning of Michael Dyck.

The definition of the phrase "functions with an implementation-defined implementation" is in XDM §2.8.1:

*implementation*: This enables the function, when it's called, to map instances of its parameter types into an instance of its result type. The implementation is either:
a host language expression, which is always associated with a static context, or
an implementation-dependent function implementation, which is optionally associated with both a static and a dynamic context.

Now, as I say, it's very unfortunate choice of terminology, because there's nothing implementation-dependent about a function like "name", except of course for the internal implementation; and whoever chose this phrase had some kind of twisted logic that says "the implementation of name#1 is different for each XPath implementation, therefore we will refer to it as a function with an implementation-dependent implementation". But it's a classic case where 95% of readers, if they get this far, will misunderstand what we are saying because of a poor choice of language.

On the second point:

self::a / 
  let $f := name#0
  return b/c/d/$f()

the value returned by $f() is "a", not "d".

The explanation is in XP31 §3.1.6

<quote>
Furthermore, if the function referenced by a NamedFunctionRef has an implementation-dependent implementation, then the implementation of the function returned by the NamedFunctionRef is associated with the static context of this NamedFunctionRef expression and to the dynamic context in which it is currently being evaluated.
</quote>

Again, a horrible sentence (especially the use of "associated to"). The antecedent of the last "it" is unclear: it is referring to the NamedFunctionRef being evaluated, not to the function being evaluated. We spent many happy Tuesday afternoons deciding these matters and the spec is sometimes disappointing in the way it reflects the outcome.

I would like to propose that the following sentence in XP31 §3.1.6

<quote>
Furthermore, if the function referenced by a NamedFunctionRef has an implementation-dependent implementation, then the implementation of the function returned by the NamedFunctionRef is associated with the static context of this NamedFunctionRef expression and to the dynamic context in which it is currently being evaluated.
</quote>

be rewritten, to make its meaning clear:

in XP31 §3.1.6

<quote>
In the case of built-in functions (more precisely, functions having an implementation-dependent implementation) that depend on the static or dynamic context, the static context that applies is the static context of the NamedFunctionRef, and the dynamic context is the dynamic context in which the NamedFunctionRef is evaluated. 

Note: The result of any dynamic function call to the function returned by the NamedFunctionRef does not depend in any way on the static or dynamic context of that dynamic function call.

For example, if $a is bound to the element <a><b><c/></b></a>, then the expression

let $f := $a/name#0 return $a/b/c/$f()

returns "a" (and not "c").
</quote>

We also then need to adjust the wording for function-lookup, which currently says that it "deliberately uses the same wording" - which is not actually true, since it corrects some of the grammatical problems in the original.

> Again, a horrible sentence (especially the use of "associated to"). The 
> antecedent of the last "it" is unclear: it is referring to the 
> NamedFunctionRef being evaluated, not to the function being evaluated. 

I have to agree, because I did read that sentence until "implementation dependent", after which I thought "oh, this does not apply". 

<quote>
...then the implementation of the function returned by the NamedFunctionRef is associated with the static context of this NamedFunctionRef expression and to the dynamic context in which it is currently being evaluated.
</quote>

Thanks for clarifying this, I would've thought, and still think, "it" applies to the subject in that sentence. Simplified:

"then the X returned by Y is associated with SC and to the DC in which it is evaluated."

In my English-to-Dutch translation in my head, there is no way that "it" can refer to something other than X, i.e. the function, not the NamedFunctionRef.

I agree to your suggestions for improving the text. Though I still think it is probably a good idea to termdef the "built-in functions" and to add a Note that this applies to all externally defined functions incl. FO31, XQ31, XSLT30 and that standards built on XPath are not allowed to deviate from this (are they?).

Another suggestion is to include the word *closure* in the text, as that is a commonly well understood term. Something like "the dc and sc of the NamedFunctionRef expression become part of the closure of the function implementation".

Final note on function types, because I still feel like we are missing something. 

<quote>
If F's implementation is implementation-dependent (e.g., it is a built-in function or external function or host-language-dependent function, or a partial application of such a function):
</quote>

Let me try:
* fn:name is a built-in function
* fn:current is a host-language-dependent function
* ex:network-address (extension function in processor) is an external function
* f:filter-nodes (stylesheet function) is ???
* function f() { 'test' } is a FunctionBody function

In your new text you suggest "built-in" functions, but I'm uncertain whether that means all of the above, or part of it. If it means all of it (the last one has no name, does not apply), then the "in the case of" can be removed as the subset is already the whole set.

In other words is the set of functions referred to by §3.1.5.1 item 5.b.i the same as the set of functions referred to by §3.1.6?

(In reply to Abel Braaksma from comment #3)
>
> What is a built-in function in light of XPath? XPath does not have
> built-in functions.

The XPath spec has defined "built-in functions" since 2.0. (One can debate the precision of the definition, but it's clear that XPath *has* built-in functions.)

(In reply to Abel Braaksma from comment #3)
> 
> To stick to XPath:
> 
> let $f := name#0
> return a/b/c/$f()
> 
> 1) Using my understanding of Michael Dyck's reasoning:
> Here c sets the context of $f and it should return the name of "c".

All you can infer from what I said is that the evaluation of the StepExpr $f() gets its dynamic context from its parent RelativePathExpr. Whether/how that evaluation *uses* that dynamic context is up to its semantics. (Which I believe is consistent with what Michael Kay is saying.)

(In reply to Michael Kay from comment #5)
> 
> Note: The result of any dynamic function call to the function returned by
> the NamedFunctionRef does not depend in any way on the static or dynamic
> context of that dynamic function call.

No, that's not true. E.g. consider:
    let $f := abs#1, $i := -1 return $f($i)
Here, the result of the DFC $f($i) definitely does depend on the dynamic context of that DFC (via the dynamic context of the VarRef).

Other than that, I believe the suggested change is an improvement.

(In reply to Abel Braaksma from comment #6)
> 
> <quote>
> ...then the implementation of the function returned by the NamedFunctionRef
> is associated with the static context of this NamedFunctionRef expression
> and to the dynamic context in which it is currently being evaluated.
> </quote>
> 
> Thanks for clarifying this, I would've thought, and still think, "it"
> applies to the subject in that sentence. Simplified:
> 
> "then the X returned by Y is associated with SC and to the DC in which it is
> evaluated."
> 
> In my English-to-Dutch translation in my head, there is no way that "it" can
> refer to something other than X, i.e. the function, not the NamedFunctionRef.

(In English, one can say:
   The house is close to the city's center and to its train station.
where "its" refers to the city and not the house. Which is not to say that quoted sentence is clear and unambiguous, just that Michael Kay is supplying a grammatically plausible way to interpret it [which is, I believe, the way it was intended].)


> Another suggestion is to include the word *closure* in the text, as that is
> a commonly well understood term. Something like "the dc and sc of the
> NamedFunctionRef expression become part of the closure of the function
> implementation".

I think that would raise more problems than it solves. (Does every function implementation have a closure? What is it? Is it different from the nonlocal variable bindings?)

 
> Final note on function types, because I still feel like we are missing
> something. 
> 
> <quote>
> If F's implementation is implementation-dependent (e.g., it is a built-in
> function or external function or host-language-dependent function, or a
> partial application of such a function):
> </quote>
> 
> Let me try:
> * fn:name is a built-in function
> * fn:current is a host-language-dependent function
> * ex:network-address (extension function in processor) is an external
> function
> * f:filter-nodes (stylesheet function) is ???

(I believe the intent was that a stylesheet function would also be classified as a host-language-dependent function. From the point of view of the XPath spec, there's no distinction between an external function and a host-lang-dependent function; that distinction only has meaning from the point of view of the host language (if any).)

> * function f() { 'test' } is a FunctionBody function
> 
> In your new text you suggest "built-in" functions, but I'm uncertain whether
> that means all of the above, or part of it.

Certainly not *all*, as that would include "FunctionBody functions".

Strictly speaking, "built-in function" means "functions defined in the F&O spec", which would only include fn:name from your list above. However, Michael Kay's parenthetical goes on to more precisely define the set of functions addressed by the paragraph.

> In other words is the set of functions referred to by §3.1.5.1 item 5.b.i
> the same as the set of functions referred to by §3.1.6?

I think it's clear that 3.1.5.1 and both the existing and suggested wordings for 3.1.6 are all referring to the set of functions with imp-dep implementations. (The suggested wording for 3.1.6 then adds the subordinate clause "that depend on the static or dynamic context" and thus addresses a subset.)

(In reply to Michael Kay from comment #5)
> 
> We also then need to adjust the wording for function-lookup, which currently
> says that it "deliberately uses the same wording" - which is not actually
> true, since it corrects some of the grammatical problems in the original.

Specifically, fn:function-lookup's note says:
    The above rule deliberately uses the same wording as the corresponding rule
    for Named Function References.

I think you're assuming that 'Named Function References' refers to the section by that name in the XPath/XQuery specs. That's probably correct (given the capitalization), but:

(a) It's odd that the F&O text isn't a link to that section.

(b) There's also text in F&O 1.7.4 "Properties of functions" about named function references that could conceivably be meant, and in any case should perhaps be aligned/linked.

(In reply to Michael Dyck from comment #10)
> 
> > Another suggestion is to include the word *closure* in the text, as that is
> > a commonly well understood term. Something like "the dc and sc of the
> > NamedFunctionRef expression become part of the closure of the function
> > implementation".
> 
> I think that would raise more problems than it solves. (Does every function
> implementation have a closure? What is it? Is it different from the nonlocal
> variable bindings?)

Fair enough. 

Though there is a section in XP31 that describes how functions are rewritten using a wrapper function with the same signature (for function coercion, actually), which I think is what most people mean with closure: a set of vars or contextual objects that are "closed in" in the function body.


> > Let me try:
> > * fn:name is a built-in function
> > * fn:current is a host-language-dependent function
> > * ex:network-address (extension function in processor) is an external
> > function
> > * f:filter-nodes (stylesheet function) is ???
> 
> (I believe the intent was that a stylesheet function would also be
> classified as a host-language-dependent function. From the point of view of
> the XPath spec, there's no distinction between an external function and a
> host-lang-dependent function; that distinction only has meaning from the
> point of view of the host language (if any).)

I now think that the main problem lies with using the term "the implementation of F is implementation-dependent". Colloquially, this refers to anything else that the spec calls "implementation-dependent", but here it is specifically meant to mean "any function that does not have a function body".

Which is still confusing, as both in XQuery and XSLT you can create a function that has a function body that is not a FunctionBody, that is considered here as "implementation-dependent" but that actually is very much "user-specified/dependent".

If we "click around" we may land on the DC definition of "named functions" which uses "implementation-dependent implementation" differently: 

<quote>
Named functions can include functions with implementation-dependent implementations; these functions do not have a static context or a dynamic context of their own.
</quote>

This suggests that "IDI" functions are something special with no DC/SC, where apparently it typically means "any function, except FunctionBody functions".
 
> Strictly speaking, "built-in function" means "functions defined in the F&O
> spec", which would only include fn:name from your list above. However,
> Michael Kay's parenthetical goes on to more precisely define the set of
> functions addressed by the paragraph.

In the discussed section, "built-in functions" is not linked. But it has indeed a TERMDEF. Following the definition leads (alas) to more questions:

<quote>
[Definition: The built-in functions are the functions defined in FO31 in the <snip>FO31-namespaces</snip>] The set of built-in functions is specified by the host language. Additional functions may be provided in the static context. XPath per se does not provide a way to declare named functions, but a host language may provide such a mechanism.
</quote>

This definition starts by saying exactly what it is: any function in FO31. Then a host-language defined limitation "set of ..." is applied*, which can be expanded by an undefined means and by "a way to declare named functions".

*) The sentence "The set of built-in functions is specified by the host language" seems to contradict the previous statement, which says that the built-in functions *are* the FO31 functions. I think what is meant is "A host language may define a limitation of the whole set of built-in functions that is available in the static context."

---------------------------------------------

I also noticed the following (related, but not mentioned here before)

1)
<quote>
Errors raised by external functions are implementation-defined (see 2.2.4 Consistency Constraints).
</quote>

--> There is no mention of "external functions" in 2.2.4. There is a mention in D.14, again pointing to 2.2.4.

--> There is no definition of "external functions"

2)
<quote>
Errors raised by host-language-dependent functions are implementation-defined.
</quote>

--> here, and in the para before it, "host-language-dependent function" is used as term, but has no explanation, nor does it appear elsewhere in the spec. While the term may seem clear, it is ambiguous: FO31 is a host language of XP31, but this is not meant. XSLT30 is a host language and I think it means fn:current and the like, but not stylesheet functions?

3)
The term built-in functions is used 12x, of which 5x it is not linked to the TERMDEF, of which at least 3x ought to be linked (at least in the section discussed here).

And may I suggest an alternate wording? Because I do not think the term "implementation-dependent implementation" has any benefit being used here, the sole difference seems to be:

a) Functions that have a FunctionBody
b) Functions that do not have a FunctionBody

Isn't that much simpler? Together they are clearly the total set, something that is not easily achieved using different terminology for the two sets.

Would it not suffice to reorder 3.1.5.1 point 5(b) to read

i) If F's implementation is a FunctionBody:
ii) Otherwise, ...

One of the problems here is making the rules host-language independent. For a user-defined function in XSLT, the implementation is not (syntactically) a FunctionBody, but it's equivalent to a FunctionBody as far as the XPath engine is concerned.

I would personally prefer to base this on the "context-dependent" property defined in F+O:

[Definition] A function may have the property of being context-dependent: the result of such a function depends on the values of properties in the static and dynamic evaluation context as well as on the actual supplied arguments (if any).

* some functions in F+O are described as context-dependent

* user-written functions in XQuery, XSLT, and XPath are not context-dependent

* a partial application of F is context-depedendent if F is context-dependent (if I remember correctly...!)

* constructor functions are not context-dependent

* a function returned by a named function reference is NOT context dependent, even if the named function is, because the context is bound at the point where the reference is evaluated.

And then rule 5b becomes:

* (i) if F is context-dependent then...

* (ii) otherwise....

(In reply to Michael Kay from comment #14)
> I would personally prefer to base this on the "context-dependent" property
> defined in F+O:
> 
> [Definition] A function may have the property of being context-dependent:
> the result of such a function depends on the values of properties in the
> static and dynamic evaluation context as well as on the actual supplied
> arguments (if any).
This makes sense to me. Note that this deviates quite strongly from the current definition (which only takes care of FunctionBody-functions and the rest, as comment 13 and bottom of comment 12 point out). 

But I think it is not the whole story:

> user-written functions in XQuery, XSLT, and XPath are not context-dependent
I assume that you mean by user-written function:
a) xsl:function (XSLT), 
b) FunctionDecl (XQuery) or
c) FunctionBody-function (XP). 

But there is a difference here. XP and XQ "capture" the in-scope local variables (but not the context), XSLT functions can only access global variables. In other words, XP and XQ create a closure at definition, xsl:function does not.

> The focus (context item, context position, and context size) is absentDM31.
This quote is from the current rule 5.b.ii.A. Your proposal mentions context-dependent vs. non-context-dependent functions, but perhaps a better split would be between focus-dependent and focus-independent functions, since it is only the focus that is absent with the "FunctionBody-functions" (user-defined functions). The rest of the context is available.

> a partial application of F is context-depedendent if F is context-dependent 
> (if I remember correctly...!)
We say about this:

<quote>
implementation: The implementation of F, associated with the same contexts as in F. If these contexts are absent in F, it is associated with SC and DC.
</quote>

Which is, upon re-reading, more ambiguous than I thought. Does "contexts" refer to DC and SC, or is the plural meant to refer to parts of each context? Esp. the second line reads as if either the whole context is there or not, then associate it (associate what, the absent context?).

Can we define this more clearly? (I tried some text proposals, but couldn't figure it out...)

(In reply to Abel Braaksma from comment #12)
> There is no mention of "external functions" in 2.2.4
(btw: I realize now that "external function" as mentioned in XPath is what is defined in XQuery as "external function" and in XSLT as "extension function").

(In reply to Michael Kay from comment #14)
> One of the problems here is making the rules host-language independent. For
> a user-defined function in XSLT, the implementation is not (syntactically) a
> FunctionBody, but it's equivalent to a FunctionBody as far as the XPath
> engine is concerned.

XPath and XQuery define specific languages, and do not define a concept like "equivalent to a FunctionBody as far as the XPath engine is concerned".

I think it is better to distinguish:

1. external functions
2. host language functions
3. implementations of functions in the F&O library
4. functions written in the language of the XQuery or XPath specification

From a language perspective, 1-3 are treated the same way in the current text of the XPath specification, under file:///home/jonathan/project/WWW/XML/Group/qtspecs/specifications/xquery-31/html/shared-31.html#id-eval-function-call  5.b.i., and functions written in XQuery or XPath are treated under 5.b.ii.:

<quote>
b. If FC is not a partial function application:

i. If F's implementation is implementation-dependent (e.g., it is a built-in function or external function or host-language-dependent function, or a partial application of such a function):

!!! SNIP !!!

ii. If F's implementation is a FunctionBody:
The FunctionBody is evaluated. The dynamic context for this evaluation is obtained by taking the dynamic context of the module InlineFunctionExpr that contains the FunctionBody, and making the following changes:

!!! SNIP !!!
</quote>

Built-in is a defined term:

<quote>
[Definition: The *built-in* functions supported by XQuery 3.1 are defined in [XQuery and XPath Functions and Operators 3.1].] 
</quote>

This seems correct to me. What exactly needs fixing?

In last week's call, we agreed that the following would probably fix the problem.

   1. The processor "makes the information available" as opposed to
      invoking using SC and DC
   2. The Note (which is non-normative).

   Replace the note with following normative text:

     An API used to invoke external functions must state whether the
     static and dynamic contexts are available to a function that is
     invoked. The F&O specification states how the static and dynamic
     contexts are available to a function that is invoked. A host
     language must state whether and how the static and dynamic
     contexts are available to a function that is invoked.

Abel, do you agree?

(In reply to Jonathan Robie from comment #17)
> In last week's call, we agreed that the following would probably fix the
> problem.
> 
>    ...
>    2. The Note (which is non-normative).

(This is referring to the Note that I suggested near the end of https://lists.w3.org/Archives/Public/public-xsl-query/2016Jan/0005.html)

(Also, in last week's call, I specifically said that my suggestions would not resolve the whole bug, merely one aspect of it.)

I have checked in changes that probably resolve this issue.  Please check and see if you agree.

Some notes on the current text of section 3.1.5.1:

1) under 5.a, 4th bullet point
"If this is not an XPath 3.1 expression then the new function's implementation is associated with a static context and a dynamic context in one of two ways: if F's implementation is already associated with contexts, then those are used; otherwise, SC and DC are used. "

This sentence is of the the form "If X then (if Y then A else B)". The first "if" is not closed, what if it *is* an XPath 3.1 expression?

2) general
Text is still very hard to digest, I would prefer an example that shows the differences in SC/DC treatment and the PFA/non-PFA behavior.

3) 5.b.i
5.b: (ii) and (iii) talk about "implementation", (i) does not.

4) 5.b.iii
Text says " A host language must state how the static and dynamic contexts are used in functions that it provides. "

The word "provides" suggest predefined functions, not necessarily functions created with host language syntax, such as xsl:function

5) 5.b.iii (in XP31 only)
Under 5.b.iii, 3rd bullet mentions "external functions". This is XQ31 specific, we decided earlier to remove it, I believe, otherwise we need to define it.

6) general
Does the new text unambiguously resolve an expression like:

let $f := name#0
return a/b/c/$f()

7) 5.b under Note
Change diff doesn't fully work, i.e., this first Note is not diff-marked, but it is new. I am not certain if this applies to more text, leading potentially to missing improvements or reintroduced bugs.

We discussed comment #20 today, and decided these two need attention:
 
> 2) general
> Text is still very hard to digest, I would prefer an example that shows the
> differences in SC/DC treatment and the PFA/non-PFA behavior.
 
> 5) 5.b.iii (in XP31 only)
> Under 5.b.iii, 3rd bullet mentions "external functions". This is XQ31
> specific, we decided earlier to remove it, I believe, otherwise we need to
> define it.

At the meeting on 2016-04-26, the WG agree that this bug is now resolved.  Action A-641-04 was raised to track the remaining editorial changes needed.