XSLT and XQuery Serialization 3.0

1 Introduction

This document defines serialization of the W3C XQuery and XPath Data Model 3.0 (XDM), which is the data model of at least [XML Path Language (XPath) 3.0], [XSL Transformations (XSLT) Version 3.0], and [XQuery 3.0: An XML Query Language], and any other specifications that reference it.

In this document, examples and material labeled as "Note" are provided for explanatory purposes and are not normative.

Serialization is the process of converting an instance of the [XQuery and XPath Data Model (XDM) 3.0] into a sequence of octets. Serialization is well-defined for most data model instances.

2 Sequence Normalization

An instance of the data model that is input to the serialization process is a sequence. Prior to serializing a sequence using any of the output methods whose behavior is specified by this document (3 Serialization Parameters), the serializer MUST first compute a normalized sequence for serialization; it is the normalized sequence that is actually serialized. [Definition: The purpose of sequence normalization is to create a sequence that can be serialized as a well-formed XML document or external general parsed entity, that also reflects the content of the input sequence to the extent possible.] [Definition: The result of the sequence normalization process is a result tree.]

The normalized sequence for serialization is constructed by applying all of the following rules in order, with the initial sequence being input to the first step, and the sequence that results from any step being used as input to the subsequent step. For any implementation-defined output method, it is implementation-defined whether this sequence normalization process takes place.

Where the process of converting the input sequence to a normalized sequence indicates that a value MUST be cast to xs:string, that operation is defined in Section 18.1.2 Casting to xs:string and xs:untypedAtomic ^FO30 of [XQuery and XPath Functions and Operators 3.0]. Where a step in the sequence normalization process indicates that a node should be copied, the copy is performed in the same way as an XSLT xsl:copy-of instruction that has a validation attribute whose value is preserve and has a select attribute whose effective value is the node, as described in Section 11.9.2 Deep Copy^XT30 of [XSL Transformations (XSLT) Version 3.0], or equivalently in the same way as an XQuery content expression as described in Step 1e of Section 3.8.1.3 Content ^XQ30 of [XQuery 3.0: An XML Query Language], where the construction mode is preserve. The steps in computing the normalized sequence are:

1. If the sequence that is input to serialization is empty, create a sequence S₁ that consists of a zero-length string. Otherwise, copy each item in the sequence that is input to serialization to create the new sequence S₁.

2. For each item in S₁, if the item is atomic, obtain the lexical representation of the item by casting it to an xs:string and copy the string representation to the new sequence; otherwise, copy the item, which will be a node, to the new sequence. The new sequence is S₂.

3. For each subsequence of adjacent strings in S₂, copy a single string to the new sequence equal to the values of the strings in the subsequence concatenated in order, each separated by a single space. Copy all other items to the new sequence. The new sequence is S₃.

4. For each item in S₃, if the item is a string, create a text node in the new sequence whose string value is equal to the string; otherwise, copy the item to the new sequence. The new sequence is S₄.

5. For each item in S₄, if the item is a document node, copy its children to the new sequence; otherwise, copy the item to the new sequence. The new sequence is S₅.

6. For each subsequence of adjacent text nodes in S₅, copy a single text node to the new sequence equal to the values of the text nodes in the subsequence concatenated in order. Any text nodes with values of zero length are dropped. Copy all other items to the new sequence. The new sequence is S₆.

7. It is a serialization error [err:SENR0001] if an item in S₆ is an attribute node, a namespace node or a function item. Otherwise, construct a new sequence, S₇, that consists of a single document node and copy all the items in the sequence, which are all nodes, as children of that document node.

S₇ is the normalized sequence.

The result tree rooted at the document node that is created by the final step of this sequence normalization process is the instance of the data model to which the rules of the appropriate output method are applied. If the sequence normalization process results in a serialization error, the serializer MUST signal the error.

<xsl:document>
  <xsl:copy-of select="$seq" validation="preserve"/>
</xsl:document>

declare construction preserve;

document {
  for $s in $seq return
    if ($s instance of document-node())
    then $s/child::node()
    else $s
}

3 Serialization Parameters

There are a number of parameters that influence how serialization is performed. Host languages MAY allow users to specify any or all of these parameters, but they are not REQUIRED to be able to do so. However, the host language specification MUST specify how the value of all applicable parameters is to be determined.

It is a serialization error [err:SEPM0016] if a parameter value is invalid for the given parameter. It is the responsibility of the host language to specify how invalid values should be handled at the level of that language.

The following serialization parameters are defined:

The value of the method parameter is an expanded QName. If the value has a null namespace URI, then the local name identifies a method specified in this document and MUST be one of xml, html, xhtml, or text; in this case, the output method specified MUST be used for serializing. If the namespace URI is non-null, then it identifies an implementation-defined output method; the behavior in this case is not specified by this document.

In those cases where they have no important effect on the content of the serialized result, details of the output methods defined by this specification are left unspecified and are regarded as implementation-dependent. Whether a serializer uses apostrophes or quotation marks to delimit attribute values in the XML output method is an example of such a detail.

The detailed semantics of each parameter will be described separately for each output method for which it is applicable. If the semantics of a parameter are not described for an output method, then it is not applicable to that output method.

Implementations MAY define additional serialization parameters, and MAY allow users to do so. For this purpose, the name of a serialization parameter is considered to be a QName; the parameters listed above are QNames in no namespace, while any additional serialization parameters must have names that are namespace-qualified. If the serialization method is one of the four methods xml, html, xhtml, or text, then the additional serialization parameters MAY affect the output of the serializer to the extent (but only to the extent) that this specification leaves the output implementation-defined or implementation-dependent. For example, such parameters might control whether namespace declarations on an element are written before or after the attributes of the element, or they might define the number of space or tab characters to be inserted when the indent parameter is set to yes; but they could not instruct the serializer to suppress the error that occurs when the HTML output method encounters characters that are not permitted (see error [err:SERE0014]).

(validate lax { document { . } })
   /output:serialization-parameters
   /output:*[local-name() eq $param-name]/data(@value)

(validate lax { document { . } })
  /output:serialization-parameters/output:use-character-maps
  /output:character-map[@character eq $char]/string(@map-string)

(document { . })/output:serialization-parameters
   /(count(distinct-values(*/node-name(.))) eq (count(*)))

<output:serialization-parameters
       xmlns:output="http://www.w3.org/2010/xslt-xquery-serialization">
  <output:method value="xml"/>
  <output:version value="1.0"/>
  <output:indent value="yes"/>
</output:serialization-parameters>

<output:serialization-parameters
       xmlns:output="http://www.w3.org/2010/xslt-xquery-serialization"
       xmlns:book="http://example.org/book"
       xmlns="http://example.org/book/chapter">
  <output:cdata-section-elements value="heading book:footnote"/>
</output:serialization-parameters>

<output:serialization-parameters
       xmlns:output="http://www.w3.org/2010/xslt-xquery-serialization"
       xmlns="http://example.org/ext">
  <output:method value="html"/>
</output:serialization-parameters>

<output:serialization-parameters
       xmlns:output="http://www.w3.org/2010/xslt-xquery-serialization"
       xmlns:ext="http://example.org/ext">
  <output:method value="ext:jsp"/>
  <output:use-character-maps>
    <output:character-map character="&#xAB;" map-string="&lt;%"/>
    <output:character-map character="&#xBB;" map-string="%&gt;"/>
  </output:use-character-maps>
</output:serialization-parameters>

4 Phases of Serialization

Serialization comprises five phases of processing (preceded optionally by the sequence normalization process described in 2 Sequence Normalization).

For an implementation-defined output method, any of these phases MAY be skipped or MAY be performed in a different order than is specified here. For the output methods defined in this specification, these phases are carried out sequentially as follows:

1. A meta element is added to the normalized sequence along with discarding an existing meta element, as controlled by the include-content-type parameter for the XHTML and HTML output methods.

2. Markup generation produces the character representation of those parts of the serialized result that describe the structure of the normalized sequence. In the cases of the XML, HTML and XHTML output methods, this phase produces the character representations of the following:

   • the document type declaration;

   • start tags and end tags (except for attribute values, whose representation is produced by the character expansion phase);

   • processing instructions; and

   • comments.

   In the cases of the XML and XHTML output methods, this phase also produces the following:

   • the XML or text declaration; and

   • empty element tags (except for the attribute values);

   In the case of the text output method, this phase replaces the single document node produced by sequence normalization with a new document node that has exactly one child, which is a text node. The string value of the new text node is the string value of the document node that was produced by sequence normalization.

3. Character expansion is concerned with the representation of characters appearing in text and attribute nodes in the normalized sequence. For each text and attribute node, the following rules are applied in sequence.

   1. If the node is an attribute that is a URI attribute value and the escape-uri-attributes parameter is set to require escaping of URI attributes, apply URI escaping as defined below, and skip rules b-e. Otherwise, continue with rule b.

      [Definition: URI escaping consists of the following three steps applied in sequence to the content of URI attribute values:]

      1. normalize to NFC using the method defined in Section 5.4.6 fn:normalize-unicode ^FO30

      2. percent-encode any special characters in the URI using the method defined in Section 6.4 fn:escape-html-uri ^FO30

      3. escape according to the rules of the XML or HTML output method, whichever is applicable, any characters that require escaping, and any characters that cannot be represented in the selected encoding. For example, replace < with <. (See also section 7.3 Writing Character Data)

      [Definition: The values of attributes listed in D List of URI Attributes are URI attribute values. Attributes are not considered to be URI attributes simply because they are namespace declaration attributes or have the type annotation xs:anyURI.]

   2. If the node is a text node whose parent element is selected by the rules of the cdata-section-elements parameter for the applicable output method, create CDATA sections as described below, and skip rules c-e. Otherwise, continue with rule c.

      Apply the following two processes in sequence to create CDATA sections

      1. Unicode Normalization if requested by the normalization-form parameter.

      2. apply changes as detailed in the description of the cdata-section-elements parameter for the applicable output method.

   3. Apply character mapping as determined by the use-character-maps parameter for the applicable output method. For characters that were substituted by this process, skip rules d and e. For the remaining characters that were not modified by character mapping, continue with rule d.

   4. Apply Unicode Normalization if requested by the normalization-form parameter.

      [Definition: Unicode Normalization is the process of removing alternate representations of equivalent sequences from textual data, to convert the data into a form that can be binary-compared for equivalence, as specified in [UAX #15: Unicode Normalization Forms]. For specific recommendations for character normalization on the World Wide Web, see [Character Model for the World Wide Web 1.0: Normalization].]

      The meanings associated with the possible values of the normalization-form parameter are defined in section 5.1.8 XML Output Method: the normalization-form Parameter.

      Continue with step e.

   5. Escape according to the rules of the XML or HTML output method, whichever is applicable, any characters (such as < and &) where XML or HTML requires escaping, and any characters that cannot be represented in the selected encoding. For example, replace < with <. (See also section 7.3 Writing Character Data). For characters such as > where XML defines a built-in entity but does not require its use in all circumstances, it is implementation-dependent whether the character is escaped.

4. Indentation, as controlled by the indent parameter and the suppress-indentation parameter, MAY add or remove whitespace according to the rules defined by the applicable output method.

5. Encoding, as controlled by the encoding parameter, converts the character stream produced by the previous phases into an octet stream.

   Note:

   Serialization is only defined in terms of encoding the result as a stream of octets. However, a serializer may provide an option that allows the encoding phase to be skipped, so that the result of serialization is a stream of Unicode characters. The effect of any such option is implementation-defined, and a serializer is not required to support such an option.

5 XML Output Method

The XML output method serializes the normalized sequence as an XML entity that MUST satisfy the rules for either a well-formed XML document entity or a well-formed XML external general parsed entity, or both. A serialization error [err:SERE0003] results if the serializer is unable to satisfy those rules, except for content modified by the character expansion phase of serialization, as described in 4 Phases of Serialization. The effects of the character expansion phase could result in the serialized output being not well-formed, but will not result in a serialization error. If a serialization error results, the serializer MUST signal the error.

If the document node of the normalized sequence has a single element node child and no text node children, then the serialized output is a well-formed XML document entity, and the serialized output MUST conform to the appropriate version of the XML Namespaces Recommendation [XML Names] or [XML Names 1.1]. If the normalized sequence does not take this form, then the serialized output is a well-formed XML external general parsed entity, which, when referenced within a trivial XML document wrapper like this:

<?xml version="version"?>
<!DOCTYPE doc [
<!ENTITY e SYSTEM "entity-URI">
]>
<doc>&e;</doc>

where entity-URI is a URI for the entity, and the value of the version pseudo-attribute is the value of the version parameter, produces a document which MUST itself be a well-formed XML document conforming to the corresponding version of the XML Namespaces Recommendation [XML Names] or [XML Names 1.1].

[Definition: A reconstructed tree may be constructed by parsing the XML document and converting it into an instance of the data model as specified in [XQuery and XPath Data Model (XDM) 3.0].] The result of serialization MUST be such that the reconstructed tree is the same as the result tree except for the following permitted differences:

• If the document was produced by adding a document wrapper, as described above, then it will contain an extra doc element as the document element.

• The order of attribute and namespace nodes in the two trees MAY be different.

• The following properties of corresponding nodes in the two trees MAY be different:

  • the base-uri property of document nodes and element nodes;

  • the document-uri and unparsed-entities properties of document nodes;

  • the type-name and typed-value properties of element and attribute nodes;

  • the nilled property of element nodes;

  • the content property of text nodes, due to the effect of the indent and use-character-maps parameters.

• The reconstructed tree MAY contain additional attributes and text nodes resulting from the expansion of default and fixed values in its DTD or schema; also, in the presence of a DTD, non-CDATA attributes may lose whitespace characters as a result of attribute value normalization.

• The type annotations of the nodes in the two trees MAY be different. Type annotations in a result tree are discarded when the tree is serialized. Any new type annotations obtained by parsing the document will depend on whether the serialized XML document is assessed against a schema, and this MAY result in type annotations that are different from those in the original result tree.

  Note:

  In order to influence the type annotations in the instance of the data model that would result from processing a serialized XML document, the author of the XSLT stylesheet, XQuery expression or other process might wish to create the instance of the data model that is input to the serialization process so that it makes use of mechanisms provided by [XML Schema], such as xsi:type and xsi:schemaLocation attributes. The serialization process will not automatically create such attributes in the serialized document if those attributes were not part of the result tree that is to be serialized.

  Similarly, it is possible that an element node in the instance of the data model that is to be serialized has the nilled property with the value true, but no xsi:nil attribute. The serialization process will not create such an attribute in the serialized document simply to reflect the value of the property. The value of the nilled property has no direct effect on the serialized result.

• Additional namespace nodes MAY be present in the reconstructed tree if the serialization process did not undeclare one or more namespaces, as described in 5.1.7 XML Output Method: the undeclare-prefixes Parameter, and the starting instance of the data model contained an element node with a namespace node that declared some prefix, but a child element of that node did not have any namespace node that declared the same prefix.

  The result tree MAY contain namespace nodes that are not present in the reconstructed tree, as the process of creating an instance of the data model MAY ignore namespace declarations in some circumstances. See Section 6.2.3 Construction from an Infoset ^DM30 and Section 6.2.4 Construction from a PSVI ^DM30 of [XQuery and XPath Data Model (XDM) 3.0] for additional information.

• If the indent parameter has the value yes,

  • additional text nodes consisting of whitespace characters MAY be present in the reconstructed tree; and

  • text nodes in the result tree that contained only whitespace characters MAY correspond to text nodes in the reconstructed tree that contain additional whitespace characters that were not present in the result tree

  See 5.1.3 XML Output Method: the indent and suppress-indentation Parameters for more information on the indent parameter.

• Additional nodes MAY be present in the reconstructed tree due to the effect of character mapping in the character expansion phase, and the values of attribute nodes and text nodes in the reconstructed tree MAY be different from those in the result tree, due to the effects of URI expansion, character mapping and Unicode Normalization in the character expansion phase of serialization.

  Note:

  The use-character-maps parameter can cause arbitrary characters to be inserted into the serialized XML document in an unescaped form, including characters that would be considered to be part of XML markup. Such characters could result in arbitrary new element nodes, attribute nodes, and so on, in the reconstructed tree that results from processing the serialized XML document.

A consequence of this rule is that certain characters MUST be output as character references, to ensure that they survive the round trip through serialization and parsing. Specifically, CR, NEL and LINE SEPARATOR characters in text nodes MUST be output respectively as "
", "…", and "
", or their equivalents; while CR, NL, TAB, NEL and LINE SEPARATOR characters in attribute nodes MUST be output respectively as "
", "
", "	", "…", and "
", or their equivalents. In addition, the non-whitespace control characters #x1 through #x1F and #x7F through #x9F in text nodes and attribute nodes MUST be output as character references.

For example, an attribute with the value "x" followed by "y" separated by a newline will result in the output "x
y" (or with any equivalent character reference). The XML output cannot be "x" followed by a literal newline followed by a "y" because after parsing, the attribute value would be "x y" as a consequence of the XML attribute normalization rules.

It is a serialization error [err:SEPM0004] to specify the doctype-system parameter, or to specify the standalone parameter with a value other than omit, if the instance of the data model contains text nodes or multiple element nodes as children of the root node. The serializer MUST either signal the error, or recover by ignoring the request to output a document type declaration or standalone parameter.

<x:foo xmlns:x="http://example.org/x"
       xmlns:y="http://example.org/y"
       xmlns:z="http://example.org/z">
       
       <x:bar xmlns:z="">...</x:bar>
       
</x:foo>

6 XHTML Output Method

The XHTML output method serializes the instance of the data model as XML, using the HTML compatibility guidelines defined in the XHTML specification.

It is entirely the responsibility of the person or process that creates the instance of the data model to ensure that the instance of the data model conforms to the [XHTML 1.0] or [XHTML 1.1] specification. It is not an error if the instance of the data model is invalid XHTML. Equally, it is entirely under the control of the person or process that creates the instance of the data model whether the output conforms to XHTML 1.0 Strict, XHTML 1.0 Transitional, or any other specific definition of XHTML.

The serialization of the instance of the data model follows the same rules as for the XML output method, with the general exceptions noted below and parameter-specific exceptions in 6.1 The Influence of Serialization Parameters upon the XHTML Output Method. These differences are based on the HTML compatibility guidelines published in Appendix C of [XHTML 1.0], which are designed to ensure that as far as possible, XHTML is rendered correctly on user agents designed originally to handle HTML.

• [Definition: The following XHTML elements have an EMPTY content model: area, base, br, col, hr, img, input, link, meta, basefont, frame, isindex, and param.] Given an empty instance of an XHTML element whose content model is not EMPTY (for example, an empty title or paragraph) the serializer MUST NOT use the minimized form. That is, it MUST output <p></p> and not <p />.

• If an element that has no children is an XHTML element with an EMPTY content model, the serializer MUST use the minimized tag syntax, for example <br />, as the alternative syntax <br></br> allowed by XML gives uncertain results in many existing user agents. The serializer MUST include a space before the trailing />, e.g. <br />, <hr /> and <img src="karen.jpg" alt="Karen" />.

• The serializer MUST NOT use the entity reference ' which, although defined in XML and therefore in XHTML, is not defined in HTML and is not recognized by all HTML user agents.

• The serializer SHOULD output namespace declarations in a way that is consistent with the requirements of the XHTML DTD if this is possible. The XHTML 1.0 DTDs require the declaration xmlns="http://www.w3.org/1999/xhtml" to appear on the html element, and only on the html element. The serializer MUST output namespace declarations that are consistent with the namespace nodes present in the result tree, but it MUST avoid outputting redundant namespace declarations on elements where the DTD would make them invalid.

  Note:

  If the html element is generated by an XSLT literal result element of the form <html xmlns="http://www.w3.org/1999/xhtml"> ... </html>, or by an XQuery direct element constructor of the same form, then the html element in the result document will have a node name whose prefix is "", which will satisfy the requirements of the DTD. In other cases the prefix assigned to the element is implementation-dependent.

<head>
<meta http-equiv="Content-Type" content="text/html; charset=EUC-JP" />
...

<meta http-equiv="Content-Type" content="text/html;version='3.0'" />

<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />

7 HTML Output Method

The HTML output method serializes the instance of the data model as HTML.

<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="html" version="4.0"/>
<xsl:template match="/">
  <html>
    <xsl:apply-templates/>
  </html>
</xsl:template>
...
</xsl:stylesheet>

In the example, the version attribute of the xsl:output element indicates the version of the HTML Recommendation [HTML] to which the serialized result is to conform.

It is entirely the responsibility of the person or process that creates the instance of the data model to ensure that the instance of the data model conforms to the HTML Recommendation [HTML]. It is not an error if the instance of the data model is invalid HTML. Equally, it is entirely under the control of the person or process that creates the instance of the data model whether the output conforms to HTML. If the result tree is valid HTML, the serializer MUST serialize the result in a way that conforms with the version of HTML specified by the version serialization parameter.

<script>if (a &lt; b) foo()</script>

<script><![CDATA[if (a < b) foo()]]></script>

<script>if (a < b) foo()</script>

<script type="application/ecmascript">
      document.write ("<em>This won't work</em>")
</script>

<script type="application/ecmascript">
      document.write ("<em>This won't work</em>")
</script>

<script type="application/ecmascript">
      document.write ("&lt;em&gt;This won't work&lt;/em&gt;")
</script>

<script type="application/ecmascript">
      document.write ("&lt;em&gt;This will work&lt;\/em&gt;")
</script>

<OPTION selected="selected">

<OPTION selected>

<BODY bgcolor='&amp;{{randomrbg}};'>

<BODY bgcolor='&{randomrbg};'>

<HEAD>
<META http-equiv="Content-Type" content="text/html; charset=EUC-JP">
...

<meta http-equiv="Content-Type" content="text/html;version='3.0'"/>

<meta http-equiv="Content-Type" content="text/html;charset=utf-8"/>

8 Text Output Method

The Text output method serializes the instance of the data model by outputting the string value of the document node created by the markup generation step of the phases of serialization without any escaping.

A newline character in the instance of the data model MAY be output using any character sequence that is conventionally used to represent a line ending in the chosen system environment.

9 Character Maps

The use-character-maps parameter is a list of characters and corresponding string substitutions.

Character maps allow a specific character appearing in a text or attribute node in the instance of the data model to be replaced with a specified string of characters during serialization. The string that is substituted is output "as is," and the serializer performs no checks that the resulting document is well-formed. This mechanism can therefore be used to introduce arbitrary markup in the serialized output. See Section 23.1 Character Maps^XT30 of [XSL Transformations (XSLT) Version 3.0] for examples of using character mapping in XSLT.

Character mapping is applied to the characters that actually appear in a text or attribute node in the instance of the data model, before any other serialization operations such as escaping or Unicode Normalization are applied. If a character is mapped, then it is not subjected to XML or HTML escaping, nor to Unicode Normalization. The string that is substituted for a character is not validated or processed in any way by the serializer, except for translation into the target encoding. In particular, it is not subjected to XML or HTML escaping, it is not subjected to Unicode Normalization, and it is not subjected to further character mapping.

Character mapping is not applied to characters in text nodes whose parent elements are listed in the cdata-section-elements parameter, nor to characters for which output escaping has been disabled (disabling output escaping is an [XSL Transformations (XSLT) Version 3.0] feature), nor to characters in attribute values that are subject to URI escaping defined for the HTML and XHTML output methods, unless URI escaping has been disabled using the escape-uri-attributes parameter in the output definition.

On serialization, occurrences of a character specified in the use-character-maps in text nodes and attribute values are replaced by the corresponding string from the use-character-maps parameter.

If a character is mapped, then it is not subjected to XML or HTML escaping.

A serialization error [err:SERE0008] occurs if character mapping causes the output of a string containing a character that cannot be represented in the encoding that the serializer is using for output. The serializer MUST signal the error.

10 Conformance

[Definition: Serialization is intended primarily as a component of a host language such as [XSL Transformations (XSLT) Version 3.0] or [XQuery 3.0: An XML Query Language].] Therefore, this document relies on specifications that use it to specify conformance criteria for Serialization in their respective environments. Specifications that set conformance criteria for their use of Serialization MUST NOT change the semantic definitions of Serialization as given in this specification, except by subsetting and/or compatible extensions. It is the responsibility of the host language to specify how serialization errors should be handled.

Certain facilities in this specification are described as producing implementation-defined results. A claim that asserts conformance with this specification MUST be accompanied by documentation stating the effect of each implementation-defined feature. For convenience, a non-normative checklist of implementation-defined features is provided at E Checklist of Implementation-Defined Features.