The under-specification is to some extent deliberate: Like the doc() and collection() functions, these functions access information from the external environment (or context), and the specification is therefore deliberately written in abstract terms, since different environments will inevitably differ. We do explain in a non-normative note that the intent is that they should correspond to operating system environment variables if that's a meaningful concept. There's nothing to stop implementors deciding to map the concept to something other than operating system variables if they choose; for example, they could simply access command line parameters, or properties from some product-specific configuration file.

You're right to point out that the F+O specification assumed that "environment variables" would be added to the description of the dynamic context, and that this proposed addition has not been implemented. We should fix that.

You're also right that the description could be clearer about whether the set of environment variables is expected to be stable for the duration of a query/transformation. The way the notes are written suggests that the WG did not intend to require stability, but this doesn't seem to be stated normatively.

The note that says external functions may modify the values of variables is referring of course to the possibility that external functions may be non-deterministic, and should perhaps say so.

I disagree with the answer in #2. 

All the mentioned problems in my original bug report are more a justification to eliminate 
such functions from the XQuery 3.0 standard, rather then to complicate the specification
unnecessarily even more in order to "fix" those problems.

I think those functions should be eliminated from the standard for the following reasons:
- they are not part of any user requirements
- the are 100% implementation dependent (no standard semantics of any kind)
- they are untestable
- they make no technical sense on environments that use virtualized infrastructure (aka cloud)
- if implemented, they create a security breach

For all the above reasons, obliging all XQuery processors to implement such functions makes
no sense. They should not be part of "core" XQuery.

However, this looks like a great submission for particular modules, to be standardized by
EXPath for example. That's the entire purpose of such organization.

>All the mentioned problems in my original bug report are more a justification
to eliminate such functions from the XQuery 3.0 standard.

Do you have any new information that was not available to the WG when it approved the inclusion of these functions? As I recall it, these points were all considered at the time, and you have not provided any justification to reopen the debate.

Yes, Michael, I do have more information. We are trying to implement them,
and those functions make no sense, technical or architectural.

The WG is sometimes talking decisions that have to be reverted. Mistakes happen.

Thanks.

Dana, you will have a much better chance of influencing the content of the 3.0 specs if you come to the teleconferences (and F2Fs) regularly and participate in the discussion than waiting until the WG has reached a consensus and then complaining about it.  And please don't respond by telling us how busy you are. We're all busy, and we have to struggle to balance competing demands on our time and other resources.

Dana, you will have a much better chance of influencing the content of the 3.0 specs if you come to the teleconferences (and F2Fs) regularly and participate in the discussion rather than waiting until the WG has reached a consensus and then complaining about it.  And please don't respond by telling us how busy you are. We're all busy, and we have to struggle to balance competing demands on our time and other resources. 

More specifically, when you flatly state "those functions make no sense, technical or architectural", you are not speaking for everybody.  You might be speaking for yourself, or for a group of people with whom you work.  But, clearly, enough members of the WG concluded that they DO make sense that they adopted a proposal to include them in 3.0. 

You may very well have good reason why the functions should be removed from 3.0, and sound technical arguments are appropriate. But bland, flat statements without better justification for changing our minds aren't nearly as likely to succeed.  Merely saying something does not make it true. 

Of course you're right that the WG makes mistakes and that decisions are sometimes overturned.  But the WG has to be persuaded that they did make a mistake and that the right solution is to reverse themselves.

| So what are environment variables ? Operating system-related variables ?
| Process-related  ? Implementation -defined completely ?

The section to which you link gives an explanation, and a reference to POSIX.  I think it's pretty clear.  It refers to the operating system concept of environment variables.

| 2. The text implies that the result of such a function call depends of the
| current status of the external environment where the program is executed. As a
| consequence such a function is nondeterministic, right ?

It's deterministic - stable. But the same query run on two different days might give two different answers, just as with functions to get the current date.

| 3. The text also implies that the invocation of XQuery external functions could
| have side effects on the external environment. Shouldn't this better be left to
| XQuery scripting ? Such a note can only add confusion.

No. Consider the external function "putenv" to set a system environment variable.

| 4. In general, I am not convinced that such a function should be standardized
| and imposed to all XQuery processors.

When the Working Group discussed it (in at least 3 calls and an email thread) enough people felt it was useful, and/or had customers wanting it, or in some cases already using a comparable facility, that it was added.

In C or C++ on any POSIX-complaint system you could use the getenv() function to implement this feature.  That includes everything from MS-DOS through Unix and Linux and OS X to Windows, as well as mobile platforms. Java systems would probably map this to system properties.

In response to comment #5.

Jim, you make it sound like my interest is to make myself listened.
Honestly, I don't care about that. I care about XQuery not talking decisions
that make no technical sense.

And in 2011 taking a decision that ignores the fact that a single query could
be executed in distributed fashion, in parallel on a large set of machines, in a 
cloud or virtualized infrastructure or such, is-- how can I say it politely ? -- terribly
 naive. In such environments such functions make no sense.

No single reply to my bug actually answered my concerns listed in comment #2.

The WG can take any decision it wants about it.

I can even close the bug if the decision of the group is such.

(But this bug, and its answers, will remain logged for posterity :-)

Best regards
Dana

(In reply to comment #2)
> I think those functions should be eliminated from the standard for the
> following reasons:
> - they are not part of any user requirements

Clearly not the case.

> - the are 100% implementation dependent (no standard semantics of any kind)

Also untrue. There are systems (like the cloud) where they make less sense than others, but that is not a reason to suggest that the many other XQuery implementations where they do make sense should lose this facility.

> - they are untestable

Not a reason to remove functionality.

> - they make no technical sense on environments that use virtualized
> infrastructure (aka cloud)

See above.

> - if implemented, they create a security breach

Can you please explain this further? It's not obvious what you're referring to here.

(In reply to comment #9)
> (In reply to comment #2)
> > I think those functions should be eliminated from the standard for the
> > following reasons:
> > - they are not part of any user requirements
> 
> Clearly not the case.

Dear John,

the W3C has a very specific was to describe requirements. It is called the Requirement document.

Here is the link to it:
http://www.w3.org/TR/xquery-30-requirements/

I do not see anything related to this functionality there.

Why do we spend time approving requirement documents when later on we are allowed to 
bypass them any time we feel like  !?


> 
> > - the are 100% implementation dependent (no standard semantics of any kind)
> 
> Also untrue. There are systems (like the cloud) where they make less sense than
> others, but that is not a reason to suggest that the many other XQuery
> implementations where they do make sense should lose this facility.

Excuse me, by the text we will come up with (if we accept those) will be required to say: the semantics
of such functions is implementation dependent.

Otherwise, I will oppose it very strongly. Such functions make no sense in many environments-- cloud is only one of them, but what about data integration, distributed queries, etc !?

Please do not forget that data integration for example is still one of the major use cases for XQuery.

How should a query processor return for such functions when the query is split into smaller queries executed in a variety of other systems !?

It makes perfect NO sense.


> 
> > - they are untestable
> 
> Not a reason to remove functionality.

Yes, it is. We clutter the specification with lot of stuff -- for example we need to extend the dynamic
context -- and it does not improve portability.

The cluttering of the specification is not to be ignored. XQuery is complicated enough.

> 
> > - they make no technical sense on environments that use virtualized
> > infrastructure (aka cloud)
> 
> See above.
> 
> > - if implemented, they create a security breach
> 
> Can you please explain this further? It's not obvious what you're referring to
> here.

John, I am a little but surprised by that comment. 

As soon as MarkLogic implements and ships those functions, please let me know, and 
I'll try to come up with a nice demo for you.

It seems that demos are better then a thousand words.

Best regards
Dana

Liam,

before we go ahead and so enthusiastically add those functions, could you
please look at which programming languages have such functions IN THE
CORE of the language and not in an additional library ?

That's what my complaint is.

We do not need those functions in the core of the language.

That's why we have extensibility points, and modules for.

That's what nicely designed programming languages like Java do.


Best
Dana









(In reply to comment #9)
> (In reply to comment #2)
> > I think those functions should be eliminated from the standard for the
> > following reasons:
> > - they are not part of any user requirements
> 
> Clearly not the case.

Dear John,

the W3C has a very specific was to describe requirements. It is called the Requirement document.

Here is the link to it:
http://www.w3.org/TR/xquery-30-requirements/

I do not see anything related to this functionality there.

Why do we spend time approving requirement documents when later on we are allowed to 
bypass them any time we feel like  !?


> 
> > - the are 100% implementation dependent (no standard semantics of any kind)
> 
> Also untrue. There are systems (like the cloud) where they make less sense than
> others, but that is not a reason to suggest that the many other XQuery
> implementations where they do make sense should lose this facility.

Excuse me, by the text we will come up with (if we accept those) will be required to say: the semantics
of such functions is implementation dependent.

Otherwise, I will oppose it very strongly. Such functions make no sense in many environments-- cloud is only one of them, but what about data integration, distributed queries, etc !?

Please do not forget that data integration for example is still one of the major use cases for XQuery.

How should a query processor return for such functions when the query is split into smaller queries executed in a variety of other systems !?

It makes perfect NO sense.


> 
> > - they are untestable
> 
> Not a reason to remove functionality.

Yes, it is. We clutter the specification with lot of stuff -- for example we need to extend the dynamic
context -- and it does not improve portability.

The cluttering of the specification is not to be ignored. XQuery is complicated enough.

> 
> > - they make no technical sense on environments that use virtualized
> > infrastructure (aka cloud)
> 
> See above.
> 
> > - if implemented, they create a security breach
> 
> Can you please explain this further? It's not obvious what you're referring to
> here.

John, I am a little but surprised by that comment. 

As soon as MarkLogic implements and ships those functions, please let me know, and 
I'll try to come up with a nice demo for you.

It seems that demos are better then a thousand words.

Best regards
Dana



(In reply to comment #7)
> | So what are environment variables ? Operating system-related variables ?
> | Process-related  ? Implementation -defined completely ?
> 
> The section to which you link gives an explanation, and a reference to POSIX. 
> I think it's pretty clear.  It refers to the operating system concept of
> environment variables.
> 
> | 2. The text implies that the result of such a function call depends of the
> | current status of the external environment where the program is executed. As
> a
> | consequence such a function is nondeterministic, right ?
> 
> It's deterministic - stable. But the same query run on two different days might
> give two different answers, just as with functions to get the current date.
> 
> | 3. The text also implies that the invocation of XQuery external functions
> could
> | have side effects on the external environment. Shouldn't this better be left
> to
> | XQuery scripting ? Such a note can only add confusion.
> 
> No. Consider the external function "putenv" to set a system environment
> variable.
> 
> | 4. In general, I am not convinced that such a function should be standardized
> | and imposed to all XQuery processors.
> 
> When the Working Group discussed it (in at least 3 calls and an email thread)
> enough people felt it was useful, and/or had customers wanting it, or in some
> cases already using a comparable facility, that it was added.
> 
> In C or C++ on any POSIX-complaint system you could use the getenv() function
> to implement this feature.  That includes everything from MS-DOS through Unix
> and Linux and OS X to Windows, as well as mobile platforms. Java systems would
> probably map this to system properties.

>before we go ahead and so enthusiastically add those functions, could you
please look at which programming languages have such functions IN THE
CORE of the language and not in an additional library ?

Two points:

(a) "before"? We've already added them. It's not an irrevocable decision, but it's a decision.

(b) we're not adding things to the core of the language, we are adding things to the library of functions which all products must implement. There are many languages, I think, that have a getenv function in the standard function library.

Dana, I don't really understand the amount of fuss this has generated.  We did not put it in the language book. It's in the function library.

However... languages with access to environment variables include...

C (part of the core, required, standard libraru for ANSI and ISO C)
C++ (same)

Scheme (get-environment-variable / getenv didn't make it into R6RS but appears to have been approved for addition to their next version)

Perl (via a built-in map; it was felt too important to be delegated to the POSIX module even though that's part of the standard distribution)

Python (via os.environ, a core required module)

Java, via System.getenv()

Scala (not surprising, as it runs on a JVM)

SQL implementations do appear to provide GETENV() even if it's not in any standard.

There are lots more, of course, but I don't see why I should spend time demonstrating something to you that you could answer yourself easily with a Web search.

(In reply to comment #13)
> Dana, I don't really understand the amount of fuss this has generated. 


Liam,

I wouldn't have said much if defining those functions would not require adding 
even more complexity to an already overly complex language (changing the
dynamic context to add such operating system variables is not without consequences
on complexity....).

And this for some functions that assume that the execution of an XQuery 
program is done on a SINGLE physical machine, which in 2011 is not only absurd,
and without long term architectural vision, but even against the core philosophy 
that created XQuery in the first place.

XQuery has been envisioned in 1998 as a data processing language FOR THE WEB.

That's very unfortunate that in 2011 we have to go back to a single machine.

But I think we did the tour of the problem at this point. We need to move on.

Best
Dana



















 We did
> not put it in the language book. It's in the function library.
> 
> However... languages with access to environment variables include...
> 
> C (part of the core, required, standard libraru for ANSI and ISO C)
> C++ (same)
> 
> Scheme (get-environment-variable / getenv didn't make it into R6RS but appears
> to have been approved for addition to their next version)
> 
> Perl (via a built-in map; it was felt too important to be delegated to the
> POSIX module even though that's part of the standard distribution)
> 
> Python (via os.environ, a core required module)
> 
> Java, via System.getenv()
> 
> Scala (not surprising, as it runs on a JVM)
> 
> SQL implementations do appear to provide GETENV() even if it's not in any
> standard.
> 
> There are lots more, of course, but I don't see why I should spend time
> demonstrating something to you that you could answer yourself easily with a Web
> search.

(In reply to comment #10)
> (In reply to comment #9)
> > (In reply to comment #2)
> > > I think those functions should be eliminated from the standard for the
> > > following reasons:
> > > - they are not part of any user requirements
> > 
> > Clearly not the case.
> 
> Dear John,
> 
> the W3C has a very specific was to describe requirements. It is called the
> Requirement document.
> 
> Here is the link to it:
> http://www.w3.org/TR/xquery-30-requirements/
> 
> I do not see anything related to this functionality there.
> 
> Why do we spend time approving requirement documents when later on we are
> allowed to 
> bypass them any time we feel like  !?

You know quite well why the working group might want to add functionality that isn't in the requirements document - it's not like it's the first time we've ever done that. And you know as well as I do the inherent pitfalls of a waterfall development process - the working group is responding to changes in external requirements for the language.

> > > - the are 100% implementation dependent (no standard semantics of any kind)
> > 
> > Also untrue. There are systems (like the cloud) where they make less sense than
> > others, but that is not a reason to suggest that the many other XQuery
> > implementations where they do make sense should lose this facility.
> 
> Excuse me, by the text we will come up with (if we accept those) will be
> required to say: the semantics
> of such functions is implementation dependent.
> 
> Otherwise, I will oppose it very strongly. Such functions make no sense in many
> environments-- cloud is only one of them, but what about data integration,
> distributed queries, etc !?
> 
> Please do not forget that data integration for example is still one of the
> major use cases for XQuery.
> 
> How should a query processor return for such functions when the query is split
> into smaller queries executed in a variety of other systems !?
> 
> It makes perfect NO sense.

You're not saying anything new here. I accept that there are situations where the only sensible response to asking for an environment variable is to say it doesn't exist. This is not a reason to remove this functionality.

> > > - they make no technical sense on environments that use virtualized
> > > infrastructure (aka cloud)
> > 
> > See above.
> > 
> > > - if implemented, they create a security breach
> > 
> > Can you please explain this further? It's not obvious what you're referring to
> > here.
> 
> John, I am a little but surprised by that comment. 
> 
> As soon as MarkLogic implements and ships those functions, please let me know,
> and 
> I'll try to come up with a nice demo for you.
> 
> It seems that demos are better then a thousand words.

This was a serious question, and I'd prefer no response to a glib dismissal.

In response to action A-461-02 to firm up the specification of these functions:

(a) Add a new component to the definition of the dynamic context in the language book:

environment-variables: a set of (name, value) pairs. Both the name and the value are strings, and the names are unique within the set. The set of environment variables is implementation-defined and MAY be empty. A possible implementation is to provide the set of POSIX environment variables (or similar concepts on other operating system platforms) appropriate to the process in which the query|expression is initiated. The set of environment variables, and their values, are _stable_.

(b) Change the specification of fn:available-environment-variables and fn:environment-variable in F+) so that

(i) the function is defined as an accessor for this component of the dynamic context

(ii) it is made clear that the function is deterministic.

Retain the notes about possible mappings to POSIX and operating systems; remove the notes that talk about side-effects.

The specification of these functions has been improved in the areas indicated. The WG decided that the functions should be deterministic.