I have reclassified this as an enhancement request, so that it gets into the right part of the Working Group agenda.

On another related enhancement request, the WG decided that we should wait until HTML5 has reached a more advanced stage of development before considering in detail what XSLT support was required.

I have to say I am a little confused by this request. In a DOM-to-DOM transformation, no serialization is taking place, therefore I would expect that the following rule in section 2.6 applies: "When serialization is not being performed, either because the implementation does not support the serialization option, or because the user is executing the transformation in a way that does not invoke serialization, then the content of the xsl:output  and xsl:character-map  declarations has no effect." In particular, I would not expect the result of DOM-to-DOM transformation to depend in any way on the fact that xsl:output specifies method="html".

I would personally expect to handle this requirement not by a serialization option, but by a function such as normalize-html() which converts a tree by applying your rules (1) and (2). Whether the requirement for that function merits including it in the standard library is something the WG will have to debate.

When an XML document with an xml-stylesheet PI specifying an XSLT transformations is received by Internet Explorer, it performs the transform, serializes an then parsers the serialization. As a consequence, there are XSLT transformations out there that depend on no-namespace output nodes getting treated as HTML nodes after their serialization has been parsed as HTML.

Firefox, on the other hand, doesn't perform the serialization step. It parses the incoming XML document into a DOM tree, transforms it into another DOM tree and renders this second DOM tree. In order to be compatible with XSLT programs written for IE (and, by now, for previous versions of Gecko), Gecko already needs to perform the local name lower-casing in such a DOM-to-DOM transform by observing parameters meant for serialization even though no serialization happens. 

(Note that transformations that disable output escaping and target IE are not compatible with the Gecko approach, but that level of incompatibility with IE has proven acceptable. Complete incompatibility with IE and previous versions of Gecko would not be similarly acceptable.)

When HTML 5-compliance is implemented for the DOM node namespaces as well, the processing needs to be adjusted as described in this bug report in order to retain compatibility with existing content.

A function such as normalize-html() would not solve the problem, since existing content isn't calling the function. Furthermore, the behavior of outputting no-namespace nodes for no-namespace nodes specified in the XSLT program is useless in a UA that doesn't treat no-namespace nodes as HTML nodes, so there's no value in not implementing the change described in this bug report in HTML 5-compliant UAs. (Making both no-namespace and XHTML-namespace nodes behave as HTML nodes in the resulting DOM is not an acceptable solution, since that complication is exactly what is being removed here.)

(personal response) It seems to me that the kind of thing you are describing is entirely outside the scope of the XSLT and XPath specifications. These specifications are described in terms of an abstract data model (XDM) which you can map to concrete data structures (such as a Gecko implementation of an HTML5 DOM) in any way you choose. If you choose to map a node called "{http://www.w3.org/xhtml}P" in the XDM to a node called "p" in your concrete DOM, or vice versa, that's entirely up to you. Please take advantage of the abstractions built in to the specifications, which are there precisely so that XSLT and XPath don't need to change every time someone has this kind of requirement.

>Gecko already needs to perform the local name lower-casing in such a DOM-to-DOM transform by observing parameters meant for serialization even though no serialization happens

If you want this to be regarded as conformant behaviour, I think it's best to present it not as an XSLT transformation proper, but as an optimized implementation of a pipeline that consists of XSLT transformation, HTML serialization, and HTML parsing of the result into an HTML DOM.

Michael Kay
http://www.saxonica.com/

(In reply to comment #3)
> (personal response) It seems to me that the kind of thing you are describing is
> entirely outside the scope of the XSLT and XPath specifications. These
> specifications are described in terms of an abstract data model (XDM) which you
> can map to concrete data structures (such as a Gecko implementation of an HTML5
> DOM) in any way you choose. If you choose to map a node called
> "{http://www.w3.org/xhtml}P" in the XDM to a node called "p" in your concrete
> DOM, or vice versa, that's entirely up to you. Please take advantage of the
> abstractions built in to the specifications, which are there precisely so that
> XSLT and XPath don't need to change every time someone has this kind of
> requirement.

As I pointed out in bug 6777, defining an XDM mapping for HTML5 that differed from the HTML5 -> DOM -> Infoset -> XDM mapping would defeat the point of harmonizing HTML5 and XHTML5 DOMs in the first place. In practical terms, Gecko uses the same DOM to XDM mapping for all kinds of DOMs, except XDM to result DOM when the output method is 'html'. I suppose we could spec-wise define a special XDM to DOM mapping when the output method is 'html'. In any case, this needs to be triggered on the output method even though it is a DOM-to-DOM transform.

I observe that browsers implement XSLT 1.0 rather than XSLT 2.0. Where do you propose this output mapping be specified?

> >Gecko already needs to perform the local name lower-casing in such a DOM-to-DOM transform by observing parameters meant for serialization even though no serialization happens
> 
> If you want this to be regarded as conformant behaviour, I think it's best to
> present it not as an XSLT transformation proper, but as an optimized
> implementation of a pipeline that consists of XSLT transformation, HTML
> serialization, and HTML parsing of the result into an HTML DOM.

The results are not equivalent to the pipeline you describe. It is a DOM-to-DOM transform with special munging for local names currently and for namespace in the near future.

My interest here isn't getting the behavior regarded as conforming. I'm interested in having the behavior specified to avoid a situation where creating a new independent implementation of the Web the platform involved discovering unspecified information.

>I observe that browsers implement XSLT 1.0 rather than XSLT 2.0. 

Well, clearly, any solution that involves changing XSLT 1.0 is a non-starter. Apart from anything else, most XSLT 1.0 processors are no longer actively developed.

>In any case, this needs to be triggered on the output method even though it is a DOM-to-DOM transform.

Well, I don't think there's anything to stop you defining a process called say H5Transform which takes an XSLT stylesheet, processes the source document as defined in the XSLT specification, and then does some further processing on the result tree based on what you find in the xsl:output definition. That wouldn't require any changes to XSLT, and you could do it without waiting for a new XSLT release. You could even define it using XSLT 1.0 rather than 2.0 as the baseline.

>I think this the XSLT spec should specify this behavior.

I don't. XSLT is a pure XML-to-XML transformation language, there's no room for special-casing it for particular XML vocabularies. We've even moved the serialization stuff into a separate spec to keep it separate.

(In reply to comment #5)
> >In any case, this needs to be triggered on the output method even though it is a DOM-to-DOM transform.
> 
> Well, I don't think there's anything to stop you defining a process called say
> H5Transform which takes an XSLT stylesheet, processes the source document as
> defined in the XSLT specification, and then does some further processing on the
> result tree based on what you find in the xsl:output definition. That wouldn't
> require any changes to XSLT, and you could do it without waiting for a new XSLT
> release. You could even define it using XSLT 1.0 rather than 2.0 as the
> baseline.

Would it be OK to do this in a deliverable of the HTML WG or the Web Apps WG? As noted in previous comments, regardless of which namespace HTML nodes are assigned, DOM-to-DOM transform in a browser with method='html' needs to deviate from the XSLT 1.0 spec by ASCII-lowercasing the local names in order to be compatible with existing content, so at least that behavior needs a specification home.

> >I think this the XSLT spec should specify this behavior.
> 
> I don't.

OK.

Just for the record, Firefox today uses a DOM-to-DOM transformation, i.e. we don't do any serialization. Once we have the result DOM, that is what we display.

So if we didn't pay attention to the output method (as specified explicitly in  xml:output, or using the implicit default mechanism), then all XSLT out there that generates HTML would in firefox be generating plain XML, which isn't what anyone wants.

The alternative would be for us to output and then parse the output. But this seems like a expensive operation just to add HTML behavior to the result.


I'm a little unclear on what the result of an XSLT transform is expected to yield. It doesn't seem entirely true that XSLT is an XML-to-XML transform since output (which is what generates the result XML serialization) is an optional (in XSLT 1) or at least external (in XSLT 2) step.

And at least XSLT 1.0 talks about the result "tree", not the result "XML".

I always thought of it as an infoset-to-infoset transform, in which case I'd have thought that the result infoset would contain the semantic information that we're dealing with HTML nodes.

However, I haven't payed much attention to XSLT since XSLT 1.0, so I'm definitely not expressing a strong opinion here.


And I totally agree that amending XSLT 1.0 is most likely a non-starter.

>I'm a little unclear on what the result of an XSLT transform is expected to
yield. It doesn't seem entirely true that XSLT is an XML-to-XML transform since
output (which is what generates the result XML serialization) is an optional
(in XSLT 1) or at least external (in XSLT 2) step.

Yes, I wasn't being 100% accurate when I said XML-to-XML. The XSLT stylesheet describes two processes, a transformation process followed by an optional serialization process. The transformation process is actually XDM-to-XDM (assuming that if you're talking about about 1.0, "XDM" means the tree data model described in the XPath 1.0 specification); and the serialization process is XDM to XML, HTML, or text. If you want something else as output, such as a DOM (in either XML or HTML flavour), or indeed visible pixels on the browser screen, then you are doing post-processing on the XDM output which is outside the scope of the XSLT specification.

It's unclear to me where this bug report is leading. Is a change needed to the XSLT specification (or related specifications), and if so, what is that change?

I think what we're looking for is a specification for not just serialization output, but also DOM output. I.e. just like there is a specification for "XDM to serialized XML/HTML", that there was a specification for "XDM to DOM".

This seems critical to getting interoperability between implementations of client side XSLT.

(In reply to comment #8)

> It's unclear to me where this bug report is leading. Is a change needed to the
> XSLT specification (or related specifications), and if so, what is that change?
> 



I suspect that one thing that would help here is some standardised way of passing
the serialisation parameters through into (or along with) the XDM output tree. It's not clear what form that should take though as all the "obvious" ways using attributes or process instructions are likely to (by default) to produce nodes in the result rather than just affect the serialisation.


Much as I've complained elsewhere about the PSVI's bad status as a non-xml set of properties that have to be passed by some unspecified API along with the validated XML, perhaps something like that is needed here as well.

Currently the model is that if XSLT doesn't do the serialisation xsl:output is essentially ignored. 

As well as the client side use in the initial comment here this often bites people using say msxml where it's very easy to end up getting an msxml dom and using the dom serialiser.


The XSLT spec could say that if the result of the transformation is passed on as an XDM tree and not serialised then the system _MAY_ pass on the serialisation parameters specified in xsl:output (or defaulted, eg because the document element is html in no-namespace) as properties specified in the serialisation spec, which a later process MAY use for whatever reason.

So in the case in comment #7, the browser's internal xslt transform could produce a DOM tree along with an "html output method" property which it wouldn't actually use for serialisation, but could use to do whaetever html-specific stuff it needs to do.

I have a question, perhaps a painfully ignorant one (my apologies to
the sensitive among those who read this comment), which perhaps
Henri Sivonen or one of the others interested here can answer.
 
The original issue description says

    It follows that an HTML 5-compliant UA treats elements in the
    http://www.w3.org/1999/xhtml namespace as HTML elements 
    but doesn't treat no-namespace elements as HTML elements.

I am taking "treat ... elements as HTML elements" to mean supplying
the default styling and behavior specified in the various HTML
specifications.  

My question is: why?  Why not make HTML 5 specify that elements
in the http://www.w3.org/1999/xhtml namespace are treated as
HTML elements, and elements for which no namespace is specified
(i.e. elements with unqualified names, called in the description above
"no-namespace elements") are also treated as HTML if their local
name matches an element name in the HTML vocabulary (if necessary,
by performing an ad-hoc transformation on the DOM to supply an
explicit namespace name where appropriate)?  That 
would seem to match (large parts of) the existing Web better than 
the rule enunciated in the bug description.

Two subsidiary points may be worth mentioning.  First, my understanding
of the Namespaces recommendation is that its awkward ways of 
describing unqualified names (namespace-name + local name pairs
in which the namespace name is null) stem precisely from a desire
to ensure that unqualified names can in appropriate cases (e.g. 
where there is external information) be recognized as identical in
denotation (at least), or short-hand for, qualified names.  

It is not unusual for people and specifications to treat unqualified
names as if they were in a namespace distinct from any other 
namespace, and to refer to 'the anonymous namespace' or 'the
no-namespace namespace'.  I have even heard otherwise well informed
people say that unqualified names are "not in any namespace".  But
these formulations are all, strictly speaking, different from what the
Namespaces spec says.  An unqualified name is a name for which
the Namespaces spec does not identify a namespace name; it is
not a name for which the Namespaces spec licenses the inference
that it is not in any namespace.  

That is:  I don't believe the Namespaces rec requires anyone or any
spec to treat {}p and {http://www.w3.org/1999/xhtml}p as 
denoting different things. 

There are sometimes good reasons for specs (e.g. XSD and the QT)
specs to treat unqualified names as if they were known not to be 
in any namespace, but that treatment is not imposed on them by 
the Namespaces spec, and there can be cases where a spec will do 
better to say "under such and such circumstances, we claim the 
unqualified name 'xyz' for the  'zyxxyz' namespace:  conforming 
zyxxyz processors, that is, will treat {}xyz and {zyxxyz}xyz as the 
same thing."

It seems to me at first glance that HTML 5 might do well to use
the special nature of unqualified names and to say that in an
HTML processor, unqualified names which happen to be known 
to be in the HTML vocabulary should be treated as being in the
appropriate namespace.

What am I missing here?

(In reply to comment #11)
> The original issue description says
> 
>     It follows that an HTML 5-compliant UA treats elements in the
>     http://www.w3.org/1999/xhtml namespace as HTML elements 
>     but doesn't treat no-namespace elements as HTML elements.
> 
> I am taking "treat ... elements as HTML elements" to mean supplying
> the default styling and behavior specified in the various HTML
> specifications.

Yes. Specifically, it means that the element nodes that get constructed are instances of the classes that implement HTML behavior and HTML DOM interfaces.

> My question is: why?

To hide the HTML vs. XML origin differences in implementations on higher layers than the parser.

> Why not make HTML 5 specify that elements
> in the http://www.w3.org/1999/xhtml namespace are treated as
> HTML elements, and elements for which no namespace is specified
> (i.e. elements with unqualified names, called in the description above
> "no-namespace elements") are also treated as HTML if their local
> name matches an element name in the HTML vocabulary (if necessary,
> by performing an ad-hoc transformation on the DOM to supply an
> explicit namespace name where appropriate)?

Mainly because it's more complicated to have to test for two namespaces plus owner document (for no-namespace nodes in XML DOMs) all over an implementation codebase than checking for one namespace.

> Two subsidiary points may be worth mentioning.  First, my understanding
> of the Namespaces recommendation is that its awkward ways of 
> describing unqualified names (namespace-name + local name pairs
> in which the namespace name is null) stem precisely from a desire
> to ensure that unqualified names can in appropriate cases (e.g. 
> where there is external information) be recognized as identical in
> denotation (at least), or short-hand for, qualified names.  

Sure, but XHTML foiled this by assigning the elements with HTML semantics to a namespace.

> That is:  I don't believe the Namespaces rec requires anyone or any
> spec to treat {}p and {http://www.w3.org/1999/xhtml}p as 
> denoting different things. 

Maybe not, but treating them as same requires more C++, more Java, more complex XPath, etc. Treating them the same in XPath got objection over in bug 6777.

> It seems to me at first glance that HTML 5 might do well to use
> the special nature of unqualified names and to say that in an
> HTML processor, unqualified names which happen to be known 
> to be in the HTML vocabulary should be treated as being in the
> appropriate namespace.
> 
> What am I missing here?

That it's more complex to allow different namespaces in the data model and write code to treat them similarly than to unify things in the data model and write less conditions in the surrounding code.

I wonder if a serialization option to move no-namespace element nodes into a user-specified namespace would be generally useful?  (Another use case would be in XQuery, which has considerable problems where the default namespace of the input document is different from the default namespace of the output.)

Are you proposing that such a default would have to be added to the <xsl:output> element? If so, that doesn't really help us with coming up with a standard that is compatible with existing content.

Of course, if the existing content isn't compatible with xslt2.0 anyway, this might be a lost cause given that the only solution would be to modify xslt1.0, which is unlikely to happen.

XSLT already allows you to supplement the serialization parameters specified to xsl:output when invoking the transformation. Since you run the transformation, you are in total control of what you do with its result tree.

> I wonder if a serialization option to move no-namespace element nodes into a
user-specified namespace would be generally useful? 

Probably xsl:namespace-alias can be extended to support this useful behaviour. But this is only XSLT, not XQuery feature.

But more to the original problem. I think that once HTML5 is more finished spec, serialization spec can say that if xsl:output/@method=("html", "xhtml") and xsl:output/@version/xs:decimal(.)>5 then elements in no namespace or XHTML namespace are serialized in the same way using syntax defined in HTML5. The latest spec clarifications allow implementations to behave like this now.

But Henri's scenario when result of transformation is not serialized but stored directly into DOM, which is only materialization of XDM, is IMHO out of scope of XSLT spec (as Mike already pointed out). If I understand it correctly, this problem is mainly when <?xml-stylesheet?> is used. Then the right place to fix this problem is probably in the <?xml-stylesheet?> spec by adding note explaining this normalization. I think it is good time to make this change as XML Core WG is preparing new edition of stylesheet association spec.

Another possibility would be to define JS API for XPath and XSLT in HTML5 spec and describe this process there. 

(In reply to comment #16)
> But Henri's scenario when result of transformation is not serialized but stored
> directly into DOM, which is only materialization of XDM, is IMHO out of scope
> of XSLT spec (as Mike already pointed out).

Out of curiosity, when you (and Mike) say out of scope for the XSLT spec, does this mean also out of scope for the XSL Working Group. I would have thought that DOM output is as much in scope as serialization output which the working group did produce a spec for.

> If I understand it correctly, this
> problem is mainly when <?xml-stylesheet?> is used. Then the right place to fix
> this problem is probably in the <?xml-stylesheet?> spec by adding note
> explaining this normalization. I think it is good time to make this change as
> XML Core WG is preparing new edition of stylesheet association spec.

Agreed, the stylesheet association spec is entirely too under defined. For example such a simple question of if the XDM tree produced by a XSLT transformation is output as a serialization or a DOM is totally unanswered. Resulting in different browsers doing different things.

And if <?xml-stylesheet?> applies to documents loaded for other purposes than rendering (such as when loaded using XMLHttpRequest).

It'd also be nice to officially define the parameter-passing processing instructions that firefox has implemented.

https://developer.mozilla.org/En/XSLT/PI_Parameters

>Probably xsl:namespace-alias can be extended to support this useful behaviour.

Presumably by means of an option that makes it apply to all constructed/copied elements, not only to literal result elements?

(In reply to comment #15)
> XSLT already allows you to supplement the serialization parameters specified to
> xsl:output when invoking the transformation. Since you run the transformation,
> you are in total control of what you do with its result tree.

This issue is about doing a DOM-to-DOM transformation for *existing* (without yet-to-be-defined supplementary parameters) content, so there's no serialization involved but the serialization method 'html' (without any new parameters) needs to affect the DOM-to-DOM transformation.

(In reply to comment #16)
> But more to the original problem. I think that once HTML5 is more finished
> spec, serialization spec can say that if xsl:output/@method=("html", "xhtml")
> and xsl:output/@version/xs:decimal(.)>5 then elements in no namespace or XHTML
> namespace are serialized in the same way using syntax defined in HTML5. The
> latest spec clarifications allow implementations to behave like this now.

It would be useful to have an HTML5 serializer in non-browser XSLT processors, but that's separate from this issue. This issue is about making existing XSLT transforms already out there work with DOM5.

> But Henri's scenario when result of transformation is not serialized but stored
> directly into DOM, which is only materialization of XDM, is IMHO out of scope
> of XSLT spec (as Mike already pointed out).

Considering the previous statements about this being out of scope for XSLT and considering that what needs to be specced is XSLT 1.0 behavior with DOM5--not XSLT 2.x behavior--I'm now pursuing putting the delta spec (delta from XSLT 1.0) into the HTML5 spec (bug 6776).

I'm OK with resolving this as WONTFIX for XSLT specs themselves if the specifiers of XSLT are OK with putting the delta spec from bug 6776 in the HTML 5 spec for the benefit of future browser implementors of XSLT.

>I would have thought that DOM output is as much in scope as serialization output

There are two parts to that:

(a) I'm sure we could get XDM-to-DOM mappings added to our charter if we wanted
to, though it's not there at the moment

(b) I'm not sure we would want to - because (a) we're pretty overloaded on the
current work programme already, and (b) it's hard to summon up much enthusiasm
for a rather tedious piece of work which will inevitably have to deal with a
lot of ugly corner cases, DOM being what it is.

The HTML 5 spec now addresses this:
http://www.whatwg.org/specs/web-apps/current-work/#dom-based-xslt-1.0-processors

XSL WG resolution:

We are closing this as we don't believe it is an XSLT bug. We also believe it is possible for HTML5 to achieve its objectives by taking advantage of the flexibility within the XPath 1.0 specification to define the evaluation context and the mappings of data models, and we propose to submit comments on the HTML5 specification to that effect.