XQuery 1.0 and XPath 2.0 Data Model

Consider the following schema document and instance:

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
 <xs:element name="root">
  <xs:simpleType>
   <xs:restriction base="xs:decimal">
    <xs:annotation>
     <xs:documentation>Always two signif. digits</xs:documentation>
    </xs:annotation>
    <xs:pattern value=".*\..."/>
   </xs:restriction>
  </xs:simpleType>
 </xs:element>
</xs:schema>

<root>3.00</root>

The instance is valid per the schema corresponding to the schema document.

A query which attempted to construct an element including this one
would however fail, as I read the spec., because the serialization
would include <root>3.0</root>, which is invalid per the type.

Decided at the Aug 2004 f2f: closed with no action.

Here are the comments from the DOM Working Group regarding the
XQuery/XPath Data Model document [1].

While no other specification, including the specification of XML syntax,
will perfectly match the W3C XML Information Set (Infoset) in all
aspects, the Infoset seems essential as a yardstick to see where
specifications may fail to interoperate properly when exchanging XML
information. In the latest version of DOM Level 3 Core, the DOM working
group has included a mapping between the Infoset and the information in
a DOM tree to be sure there is no confusion, as Infoset continues to be
the standard way of describing various types of standardized
representations of XML content. Note that this mapping between the DOM
model and the Infoset is done in a separate appendix in the
specification, and did not necessitate a complete rewriting of the DOM
specifications. The DOM WG has supported the development of the Infoset
to provides a set of definitions for use in W3C specifications that need
to refer to the information in an XML document.

Some DOM information items have been excluded from the Infoset, with the
understanding that Infoset definitions are most useful for XML
information that is common to multiple specifications and the DOM-only
information has not seemed harmful to compatibility between
specifications that did not require that particular information. DOM,
for example, is perhaps the only standard representation of XML (besides
the XML syntax itself) that cares to distinguish between CDATA sections
and regular text, but the ability to distinguish does not hurt the close
correspondance between infoset and DOM. The Infoset has also been
augmented to include a number of additional items used by other
specifications, for which DOM has included support in later versions of
the specification. The Infoset must be a work in progress adding
newly-identified information that needs common definition so that
specifications relying on infoset definitions may be
interoperable. There is no doubt that there are significant new domains
of XML data that require sharing between specifications that would
benefit from Infoset standardization.

DOM will commonly be used to make use of most other specifications at
W3C including XPath, Web Services, XML Schema, HTML, XML, CSS, XML
Signatures, and so on.  It is highly desirable for specifications to be
defined in terms of the Infoset so that the common XML information can
be exchanged as the currency of XML representation, and where
information is lost or mangled by a specification, that is known as
well. The mismatch between the DOM and XPath specifications in the past
has created issues that could have easily been better aligned in a
common model.

The DOM Working Group is very disapointed by the removal of the Infoset
mapping, included in the previous version of the Data Model, despite our
encouragements and praise of the mapping [2]. The intent of our previous
comments was to help the XSL and XML Query groups have an accurate
mapping between their specifications and the Infoset, and certainly not
meant to discourage the XSL and XML Query groups to have such
mapping. Therefore, we would like to object to the absence of _direct_
mapping between the XQuery/XPath Data Model and the Infoset, and
believes the approach taken by the latest specification of the Data
Model is harmful for the future of the XML Architecture. We are still
concerned about our previous comments [2] regarding the discrepancy
between the XQuery/XPath Data Model and the Infoset, and believe those
remain to be addressed.

[1] http://www.w3.org/TR/2003/WD-xpath-datamodel-20031112/
[2] http://lists.w3.org/Archives/Public/public-qt-comments/2003Jul/att-0003/XPDM2-DOM.html
[3] http://lists.w3.org/Archives/Member/w3c-ac-forum/2000JulSep/0215.html

We’ll add the mapping.

Here are the comments from the DOM Working Group regarding the
XQuery/XPath Data Model document [1].
[…]
Finally, the DOM Working Group reiterates his previous comment (DOM8)
regarding the name conflict between the accessor dm:node-name and the
DOM attribute Node.nodeName. Again, the notion of node names between DOM
and the XQuery/XPath Data Model is significantly different, even if the
XQuery/XPath specification does not call for physical implementation of
the abstract model. A different name (dm:universal-name, or
dm:node-uname, etc.) should be used to avoid conflict and confusion,
such as the ones reported in September 2000 [3].

[1] http://www.w3.org/TR/2003/WD-xpath-datamodel-20031112/
[2] http://lists.w3.org/Archives/Public/public-qt-comments/2003Jul/att-0003/XPDM2-DOM.html
[3] http://lists.w3.org/Archives/Member/w3c-ac-forum/2000JulSep/0215.html

Decided at joint telcon 183: reject.

The WG has discussed this issue and decided that no change is
necessary. We note, in particular, that our node-name function is in a
different namespace which is an accepted method for providing globally
unique names. There was no support in the WGs for renaming our
accessor.

We also point out that the node-name function exists in F&O and that
the node-name accessor provides the abstract functionality required for
this user-visible function.

We have observed a welcome trajectory over drafts of the Data Model to
reduce the distance between its core and the W3C Infoset
Recommendation.  We actually see no remaining impediment to completing
this and actually identifying the Data Model explicitly as an
extension of the Infoset.  This would probably need to be done on two
levels: the first, in the spirit of the PSVI, adding new properties
and possibly new information items, the second adding accessors.

Although in one way this might seem to be a purely editorial request,
or at best a rhetorical one, we actually think participating in the
use of the Infoset abstraction in this way will actually make the
Query specs stronger and simpler.

A major advantage of adopting this suggestion, connected to our
previous comment XSCH-QL-018 [1] about 'validate' expressions, is that
the need for serialization in the definition of validation goes away
-- infosets are by definition validatable as such.

Henry Thompson
By and on behalf of the W3C XML Schema WG

We’ll add the mapping.

Data model should support cdata sections. Preserving of cdata in input documents can be
controlled by flag in static context (and declaration in xslt/query prolog).
Without this it would be impossible to preserve structure of document durning transformations.
(it's especially big problem for nearly-identity transformations than changes only small parts of original document).
There are also other uses, such as xpath based search & replace option in XML editors
(ability to find and modify cdata sections) and finer control in XSLT over which content is serialized as cdata.

This will need cdata constructors in XSLT and XQuery, and cdata() SequenceType.
Maybe this feature can wait to V3.0, but will be needed at some point.

Decided at BEA face-to-face: reject.

Thank you for your comment.

You asked that the Data model should support cdata sections.
The XML Query and XSL Working Groups considered this request.

Since CDATA sections are an artifact of input encoding,
they are not reflected in the Data Model, and the comment
was therefore rejected.

SECTION 2: Basics

The language should allow for nested sequences. Nested sequences (a.k.a sequences of sequences) will help in returning a set of related values easily. The current mechanism requires one to put those values inside an XML element which removes node identities, typing etc. 

 Allowing nested sequence in later versions might be quite hard because of implicit rules governing atomization, normalization and concatenation of sequences.

Decided at the Aug 2004 f2f: closed with no action.

[My apologies that these comments are coming in after the end of the Last 
Call comment period.]

Section 6.2.4

The description of how the children property of the element node is 
constructed from PSVI indicates that if the [schema normalized value] 
property exists, and the processor chooses to use that means of setting 
the property, the order of nodes in the sequence is implementation 
defined.

This should be phrased to indicate that only the text node will be in some 
implementation-dependent position relative to the PI and comment children, 
but the order of the PI's and comments themselves is the same as in the 
[children] property.

Thanks,

Henry
[Speaking on behalf of reviewers from IBM, not just personally.]

Decided at BEA face-to-face: order will be clarified.

[My apologies that these comments are coming in after the end of the Last 
Call comment period.]

Section 6.2.2 and 6.3.2

The penultimate bullet in the description of dm:string-value indicates 
that "If the element type is xs:dateTime, xs:date, or xs:time, returns the 
original lexical representation of the typed value, recovered as follows:"

The process described actually sounds as if it is operating on the typed 
value, and the conversion back to a lexical value might not yield the 
original lexical representation - for instance, the three strings 
"2004-01-22T24:00:00+05:00", "2004-01-23T00:00:00+05:00" and 
"2004-01-23T00:00:00.000+05:00" all represent the same xs:dateTime value. 
If this process must yield the original lexical representation, the 
description of the procedure needs to be corrected; otherwise, it should 
make it explicit that the original lexical representation might not 
result.

Thanks,

Henry
[Speaking on behalf of reviewers from IBM, not just personally.]

Decided at BEA face-to-face: fixed.

[My apologies that these comments are coming in after the end of the Last 
Call comment period.]

Section 3

The second paragraph states that "The data model supports well-formed XML 
documents conforming to [Namespaces in XML]."

The Infoset recommendation errata supports documents conforming to XML 1.1 
and Namespaces in XML 1.1.  If Data Model refers to Infoset, this section 
needs to either include Namespaces 1.1 or be worded more generally. This 
comment applies throughout this section.

Thanks,

Henry
[Speaking on behalf of reviewers from IBM, not just personally.]

Decided at BEA face-to-face: fixed

[My apologies that these comments are coming in after the end of the Last 
Call comment period.]

Section 2.5

In the second paragraph, it is unclear how long the implementation-defined 
anonymous type name must persist.  For instance, if the same query or 
transformation is evaluated twice, with the same static context each time, 
must the names used to identify the same anonymous types in each 
evaluation be consistent.

Thanks,

Henry
[Speaking on behalf of reviewers from IBM.]

Decided at joint telcon 183: added wording to clarify.

SECTION 6.1.3: Construction from an Infoset (of a document node)

A document node has a document-uri property, but these rules do not
tell how to set it. If the document-uri property does not exist in the
Infoset, we should say so, and we should say how it gets into the data
model.

Decided at BEA face-to-face: agreed to clarify.

SECTION 6.5.3: Processing instruction information items

Under "target" it says that the value of the target property in 
the data model is the same as the value of the [target] property
in the Infoset.  However, the Infoset allows an arbitrary Name
for this value (see XML 1.0 section 2.6 rule [17]) which means
that a target can have arbitrarily many colons.  The Namespaces
recommendation did not limit this.  On the other hand, the 
XQuery data model limits targets to NCName (no colons).  What 
happens when a processing instruction containing one or more
colons is loaded into the data model?  This should be specified.
(See also comment re: 6.5.1)

Decided at BEA face-to-face: it’s an error to map a PI whose name is not an NCName.

SECTION 6.5.1: Overview (of processing instructions)

It says that the target property must be an NCName.
XML 1.0 only requires the target to be a Name, ie, permits any
number of colons.  Oddly, Namespaces 1.0 has nothing to say about
the target of processing instructions.  It seems like the target
of a processing instruction should be allowed to be a Name, since
that is the only restriction in XML 1.0 + Namespaces 1.0; or
at least a QName, which would be a restriction in the spirit if
not the letter of Namespaces 1.0.
(see also comment re: 6.5.3)

Decided at BEA face-to-face: it’s an error to map a PI whose name is not an NCName.

SECTION 6.6.3: Comment information item

It says "Although the data model is able to represent comments, 
it may be unnecessary or even onerous for some applications to 
do so. Applications should construct nodes in the data model to 
represent comments. The decision whether or not to represent
comments is considered outside the scope of the data model, 
consequently the data model makes no attempt to control or
identify if any or all comments are ignored."

This paragraph is confusing.  Either "application" is referring
to a program that invokes an XQuery or XPath processor, or
"application" means the XQuery or XPath processor itself.  

In the first interpretation,
it is hard to interpret the sentence "Applications
should construct nodes in the data model to represent comments"
for two reasons: a) it is not the user program's responsibility
to construct nodes at all; nodes are a formalism of the data
model and not necessarily things that the user program knows 
how to construct, aside from invoking the XQuery or XPath 
processor (where is the C or Java library to create a node?)
b) It is not the business of the data model to tell
users how to write their programs anyway.

The second interpretation seems to give the XQuery 
or XPath processor the license to discard comments from the user's
XML data.  Granted the user may want them discarded, but there
are ways he can express that desire, for example, using XPath
expressions that filter out comments.  In addition, if the 
user has explicitly coded a comment constructor in his XQuery
expression, is it really right to disregard the user's stated 
desire to create a comment?  If you really intend that XQuery
support for comments is optional, then it ought to be called out
in the list of optional features (Xquery section 2.6 "Optional
features").

Decided at BEA face-to-face: comments are no longer optional.

SECTION 6.2.2: Accessors (in 6.2 Element nodes)

dm:string-value(): the note about "recovering" the original 
lexical representation of xs:dateTime, xs:date and xs:time 
types hints that the model is that the original lexical
representation can be lost (probably as a result of forming the 
PSVI).  If that is the case, then don't we need to talk about some 
other types as well?  For example, xs:duration.  There are
many equivalent ways to represent a zero duration, such as
P0Y = P0M = P0D = PT0H = PT0M = PT0S.  Given an element of 
simple type xs:duration whose value is the zero duration, 
which of these (or many other candidates) is the string value?
In general many types have more than one lexical representation
for the same value.  Since the model is that the implementaiton
is not required to save the true original lexical representation,
we need rules telling which lexical representation to 
regenerate as the string value.

The spec has been significantly redrafted in this area and we believe your concern has now been addressed.

Data Model: specification too fuzzy

Under which circumstances do the implicit operations
performed during the schema validation like adding default
element and attributes values happen ?

For example, if one element has validation property valid, yet one
of his ancestors is not valid, section 6.2.2 states that the element 
will have
type annotation xdt:untyped.

Are the default attributes and values added to the element or not in 
this case?

The spec has been significantly redrafted in this area and we believe your concern has now been addressed.

Data Model: inconsistency, major

The Data model takes an inconsistent position with respect of elements
resp. attributes that are labeled with type annotation xdt:untyped,
resp. xdt:untypedAtomic.

In section 6.2.4 it is stated that an element AND all it's ancestors
have to have a validation property "valid" for the element to be
labeled with a Qname different then xdt:untyped.

This would mean that, as a side effect of this definition the
following constraint should hold: if an element has type annotation
xdt:untyped, then all it's descendent elements have also a type
annotation xdt:untyped.

Does this constraint indeed hold ? was this the intent of the
specification ? If yes, please state this constraint explicitly.

On the other hand, other portions of the specification do NOT require
the ancestors to be valid. Section 3.3.1 is one example. Section 6.3.2
that deals with type annotations of attributes seems to be
inconsistent with it since it doesn't look at the validity of the
ancestors. Moreover, the computation of the nillability property also
ignores the validity of ancestors, potentially creating inconsistent
data model instances.

What is the intent here ?

The spec has been significantly redrafted in this area and we believe your concern has now been addressed.

Data Model: request for simplification

What is the rationale for supporting functionality beyond
the Infoset: e.g. documents with empty content, with multiple
element children, etc ?

Is this rationale serious enough to overcome all the serialization and
application communication difficulties that derive from there ?

If no, please simplify.

/ Michael Rys <mrys@microsoft.com> was heard to say:
| There are outside requirements to support XQuery on a document node that
| may contain more than one element besides XSLT. The SQL-2003 XML
| Datatype follows that model. 
|
| Restricting a document node now would not satisfy their requirements
| either.

In summary:

- XML 1.0 defines external parsed entities which have multiple root elements.
- XSLT 1.0 backwards compatibility requires them.
- SQL-2003 requires them.

I believe this represents compelling evidence in favor of keeping the
existing functionality.

The spec has been significantly redrafted in this area and we believe your concern has now been addressed.

Data Model: inconsistency, request for simplification

Section 2.5. states that anonymous types can have an identifier that
is not a Qname, while this seem to be inconsistent with the rest of the 
document.
(the signature of the type() accessor is always an optional Qname).

We should simplify the data model and request the name of a type (even
anonymous) to be a legal Qname.

The spec has been significantly redrafted in this area and we believe your concern has now been addressed.

Data Model: request for simplification

We would like to simplify the definition of the typed value
as follows.

The typed value of a document should raise an error.
The typed value of a element with complex type with
mixed content should raise an error.
The typed error on a namespace node should raise an error.
The typed value of a processing instruction should raise an error.
The typed value of comment should raise an error.

I do not see the rationale for supporting the cases above. They do not 
bring
any additional expressive power.

The typed value only makes sense for attributes, for elements with
simple content or complex type with simple content and for text nodes.

In general, as a design philosophy, it is preferable to add this kind 
of limitations
in V1. We can always convert an error into a real value in the next 
versions,
but not a wrong kind of value into another value or error later on.

Decided at BEA face-to-face: typed-value is xs:string

Data Model:  specification too fuzzy

The Data Model misses certain consistency constraints that seem
to be required to preserve a data model instance internally consistent.

Examples are:
(a) is it true that if a node has a base uri propery, then all the 
descendants
have the same base-uri property?
(b) is it true that if a node has the nillability property being true, 
then it has
no text  and elements nodes as children, and it has an attribute 
xsi:nil="true"?
(c) if the type annotation of an element is a certain Qname different 
from xdt:untyped
and the element has  an attribute xsi:type, then are the value of the 
attribute and the type
annotation required to be the same ?
(d) etc

If the data model is generated via Infoset or PSVI it is relatively 
clear that those extra constraints
are not violated. The problem is that Data Model instances can be 
generated
directly by applications from other input structures (e.g SQL records, 
COBOL structures) etc.

The concern is that the application generated data model instances 
(other then Infoset and
PSVI) do not have a set of rules and constraints clearly spelled out 
that they will have to satisfy
to create correct data model instances.

Dana's action is completed:
http://lists.w3.org/Archives/Member/w3c-xsl-wg/2004May/0037.html

Norm: I'm inclined to incorporate these

Dana: the most interesting constraints aren't in here yet, the ones
about typed values and types and strings, etc. because there's too
much fluctuation.

Norm asks what to do.

Dana suggest leaving it open.

ACTION: Norm to add a note to the issue pointing out that additional
constraints may be desirable and leave the issue open.

ACTION: Incorporate the constraints in the message that completed
Dana's action.

Accepted.

Functions and operators: request for functionality

XQuery's function and operators builtin library should
have enough functions to introspect all the data model
accessors for the nodes. e.g. we should be able to ask for
the nillable property and for the type annotation.

The spec has been significantly redrafted in this area and we believe your concern has now been addressed.

If an element has a complex type with complex content, the
typed-value() accessor raises a type error when applied to the
element. So if the Cities element has a known type, the following
raises a type error:

  <Cities>
    <City>
        <CityId>01</CityId>
        <Longitude>100</Longitude>
        <Latitude>32</Latitude>
    </City>
    <City>
        <CityId>02</CityId>
        <Longitude>54</Longitude>
        <Latitude>24</Latitude>
    </City>
  </Cities> * 2

If you the same element is placed in a document node, the
typed-value() accessor returns its string value, as an instance
of xdt:untypedAny. Consider the following query.

document {
  <Cities>
    <City>
        <CityId>01</CityId>
        <Longitude>100</Longitude>
        <Latitude>32</Latitude>
    </City>
    <City>
        <CityId>02</CityId>
        <Longitude>54</Longitude>
        <Latitude>24</Latitude>
    </City>
  </Cities>
} * 2

This query succeeds. Either both queries should succeed,
or both should fail.

The spec has been significantly redrafted in this area and we believe your concern has now been addressed.

Section 3.3.1:

The list of cases for determining the type is still not quite right:
it does not cover the case where [type definition] or [member type
definition] is present but does not have a {name} property.  It should
be something like:

  If [member type definition] exists:

     If its {name} property is present: the {target namespace}
     and {name} properties of the [member type definition].

     Otherwise, the namespace and local name of the appropriate
     anonymous type name.

and likewise for [type definition].

The spec has been significantly redrafted in this area and we believe your concern has now been addressed.

Sections 6.2.1, 6.2.2 and 6.3.2:

There is a strange asymmetry between elements and attributes:
attributes have a string-value property, and elements don't, though
they both have dm:string-value accessors.  Furthermore, instead of the
definition of the dm:string-value accessor for attributes saying that
it returns the string-value property, it has a long description of the
construction of the value.

If the dm:string-value accessor returns the string-value property, it
should say so, and the description of the value's construction should
be in the natural places (6.3.3 and 6.3.4).  If it does *not* return
the string-value property, this needs to be explained and justified.
And what is the use of the string-value property if no accessor
returns it?

The descriptions of the dm:string-value accessors are also
unsatisfactory in themselves.  They take the form "if the type is
<type>, then <something about that value>".  What value?  No value has
been mentioned, only a type.  It must be the value whose type is
<type> - but is that the same as what dm:typed-value returns?

The spec has been significantly redrafted in this area and we believe your concern has now been addressed.

Section 6.2.2 and elsewhere:

The types xs:anyURI, xs:QName, xs:dateTime, xs:date and xs:time are
described specially.  Presumably these descriptions also apply to types
derived from these types.

The spec has been significantly redrafted in this area and we believe your concern has now been addressed.

Section 6.2.4:

In the first sentence, "effected" should be "affected".

Under [children], the creation of a text child for the [schema
normalized value] and the removal of the original text nodes seems
unnecessarily complicated.  It would be simpler and clearer to leave
the children alone, and say that the element has a string-value
property whose value is the [schema normalized value].

The spec has been significantly redrafted in this area and we believe your concern has now been addressed.

Section 6.3.2 under "dm:string-value":

Types other than xdt:untypedAtomic, xs:anyURI, xs:QName, the time/date
types, and types derived from xs:string, are not mentioned.  Numbers
for example.

Accepted editorial suggestions.

Section 6.7.1:

"A text node must not contain the empty string as its content".  But
according to 6.2.4, text nodes may arise from the [schema normalized
value] property of an element, in which case they may be empty strings
(because the [schema normalized value] may be an empty string).  The
natural solution would be to say that no text node is added in 6.2.4
if it would be empty, but see also our comments on that section above.

The spec has been significantly redrafted in this area and we believe your concern has now been addressed.

Section 6.7.4:

This does not discuss text nodes constructed from [schema normalized
value] property of an element as described in 6.2.4.

Accepted editorial suggestions.

The rules for dm:typed-value() are confusing and inconsistent for
elements and document nodes. Here is an example that illustrates one
set of problems: what is the result of the following query?

  <Cities>
    <City>
        <CityId>01</CityId>
        <Longitude>100</Longitude>
        <Latitude>32</Latitude>
    </City>
    <City>
        <CityId>02</CityId>
        <Longitude>54</Longitude>
        <Latitude>24</Latitude>
    </City>
  </Cities> * 2

In the current specification, this question can not be answered unless
we know the schema information associated with the Cities
element. Here is the current set of rules:

1. If the element has a known complex type, and it does not allow
mixed content, then it raises a type error.

2. If this element's type is xdt:untypedAny, the typed value of the
element is its string value, as an instance of xdt:untypedAtomic, so
this query evaluates to (2 * 0110032025424). This is surprising, and
it is not useful for users.

3. If this element's type is known, and it allows mixed content, the
typed value of the element is its string value, as an instance of
xdt:untypedAtomic, so this query evaluates to (2 * 0110032025424).

It would be much simpler and cleaner to say that dm:typed-value()
returns an error for any node with child elements. This is
incompatible with the behavior of XPath 1.0, but so is the current
behavior for elements of known complex type.

The spec has been significantly redrafted in this area and we believe your concern has now been addressed.

(IBM-DM-031) The Data Model document lists namespace nodes as one of the 
seven kinds of node. But a node is needed only to preserve identity and to 
represent a position on an axis. The identity of a namespace node is never 
used, and the namespace axis is deprecated in XPath 2.0 and not supported 
in XQuery. Neither XPath nor XQuery provides a way to construct a 
namespace node independently of an element