Added section 7.2, Language Attributes, to the polyglot spec. It reads:

]]
7.2 Language Attributes

Polyglot markup should use the language attributes in the html element to set the default language for the document.

When using language attributes, polyglot markup must use both the lang and xml:lang attributes. Neither attribute is to be used without the other.
[[

Sent mail to group to confirm and to ask clarification on one point:

I am confused about one point. Bug 10152 [1] says:

]]
mention that the meta Content-Language element has no usefulness at all in XML for setting the language of content.
[[

But Leif makes this point

> 	Are there reasons to say anything special about http-
> equiv="Content-language" with regard to Polyglot Markup? Based on my
> previous testing, then all the dominating browser engines behave exactly the
> same way with regard to both http-equiv="Content-Language"
> and lang/xml:lang, regardless of XHTML or HTML parsing is used. So I think
> not.
> --
> leif halvard silli

Richard or Leif,

Can either of you  (or anyone else) point me to a resource that might decide this?

Thanks,

I like the wording in section 7.2.  I think that that is sufficient. 

Wrt the http-equiv, I think that just not mentioning it in your text, as you do, is fine. No XML parser will pick up on the http-equiv, regardless of what browsers do, so you need to use xml:lang, but if you use xml:lang you must use lang too, and therefore you don't need to use the http-equiv. I don't think it's necessary to actually say all that though, especially as we are still waiting to hear the fate of the http-equiv for Content-Language, and the spec currently says it's obsolete.

I wondered whether it might be better to reverse the two paragraphs. (Since the second refers to 'the' attributes which are defined in the second para.) Just a thought.

The other thing you could say if you want to be thorough is that the values of both lang and xml:lang must be the same, eg. "Neither attribute is to be used without the other, and the values must be the same."

Thanks for the suggestions. The section now reads as follows:

[]
7.2 Language Attributes

When using language attributes, polyglot markup must use both the lang and xml:lang attributes. Neither attribute is to be used without the other, and the values for both lang and xml:lang must be the same.

Polyglot markup should use the language attributes in the html element to set the default language for the document.
[]


I believe this satisfies the issues in this bug and I have resolved the bug as FIXED.

Thanks for your help and patience.

Eliot

(In reply to comment #2)

> No XML parser will pick up on the http-equiv, regardless of what
> browsers do, 

SUMMARY: If it can be argued that it is incorrect of 'application/xhtml+xml' browsers/parser to pick up the language from http-equiv, then I insist that we require authors to use xml:lang/lang everytime http-equiv actually have an effect. (See below.) But if it works the same in both XHTML and HTML, then the polyglot specification can remain as it is.

PROPOSAL: If it is correct that there is a definite difference in XML and HTML's handling of the HTTP-EQUIV content-language element, then the spec should regulate the use of meta@content-language in polyglot documents. Otherwise we would risk that HTML5-parsers would see language(s) that an 'application/xhtml+xml' parsers would not see. We need to establsih to which extent you are correct. 

The simplest thing would be to say that polyglot documents are REQUIRED to have the lang/xml:lang attributes _on the root element_ whenever the document has a http-equiv content-language element that actually affects the language of the document - unless there also is an HTTP header that specifies the same language as HTTP-EQUIV specifies.

(This is what HTML5 says:  the http-equiv content-language element affects the document language everytime  it contains a single language tag. Thus if it contains zero or more than one language tag, then there is no document language effect. The issue on what syntax that is legal in HTML5, has not be decided yet, see http://lists.w3.org/Archives/Public/public-html/2010Jun/0569.html . However, the rules how the language gets decided are not disputed. )

DISCUSSION:

(1)  as Eliot cited in comment  #1, what you say is is not true. Because, when a Web browser is parsing a document in "application/xhtml+xml" mode, then that Web browser *is* an XML parser, or what? 

(2) It is true that XML 1.0 does not mention the http-equiv content-language element:
     But can one from this conclude that it is incorrect of a XHTML+XML parser to pick up the language from the meta Content-Langauge tag? XML 1.0 says that the XML parser may pick up the language also from HTTP and from MIME. I do therefore not understand why it can't pick up the language from HTTP-EQUIV as well. In particular, a compliant 'application/xhtml+xml' parser must be expected to know what http-equiv means, no?
     I hope someone who is expert on both XML and HTTP can answer.

(3) Perhaps you are comparing content-language with Content-Type and HTML5's meta@charset? I agree that that is a useful comparsion. However:
      The reason why META@charset and META@http-equiv content-type  literaly doesn't make any sense in XML documents, is because, at that point when the META@charset element is being processed, the encoding has already been decided. Either via external protocol, or by reading the <?xml ?> declaration. Whereas for language, that's a per-element issue.
      Thus, the reason why meta@charset doesn't work in XML is, to my mind, different from why Content-Langauge eventaully shoudln't work.

Added the following note to section 7.2: 

If polyglot markup uses the http-equiv attribute on the <meta> element, polyglot markup MUST use both the lang and xml:lang attributes. 

I believe that this fulfills the requirements of this bug and so I have resolved it.

Thanks for your help,

Eliot

I am close to satisfied. It is a great step in right direction. But:

FIRSTLY: 

A problem in your proposed text is the word "both". I feel that the requirement to use both @lang and xml:@lang was specified in the first paragraph of section '7.2 Language Attributes'. Therefore, to repeat in this note that _both_ has to be used, feels unecessary - and also makes it seem as if the requirement to use both is linked ot the meta element (instead of being a general rule) . I would suggest the following two-step solution to these wording problems:

  1) The very first paragraph of section 7.2 currently 
        BEGINS:  "When using language attributes".
        Let it instead begin like this: "When SPECIFIYNG THE language MAPPING OF AN ELEMENT, [then ...]"

  2) W.r.t. your new note, change the last bit, after the comma:
     FROM: "polyglot markup must use both the lang and xml:lang attributes."
     TO:  "then it is REQUIRED to specify the language mapping of the root element."

SECONDLY: 

  Before the comma, your new note currently says: 

   "If polyglot markup uses the http-equiv attribute on the <meta> element,"

   Comments/Changes:

      1) please work/insert 'Content-Language' into that statement. 
          Otherwise the statement is meaningless.

      2) please rewrite the entire part before the comma like so:
           "Whenever the http-equiv Content-Language  <meta> element 
            specifies the language of the root element, [then it is REQUIRED
            {see proposed text above}]."

      3) After the full stop, add this:
         "The meta content-language element according to HTML5 specifices the language of the root element whenever its @content attribute contains no more and no less than exactly one language tag."

(NOTE: Currently HTML5 only permits no more an no less than a single language tage inside content-language. But depending on ISSUE-88, it can be permitted with both less and more. http://www.w3.org/html/wg/tracker/issues/88 )

The 2 March Editor's Draft has the following for Section 7.2, Language Attributes:

]]
7.2 Language Attributes

When specifying the language mapping of an element, polyglot markup uses both the lang and xml:lang attributes. Neither attribute is to be used without the other, and polyglot markup maintains identical values for both lang and xml:lang.

Polyglot markup uses the language attributes in the html element to set the default language for the document. 

Note
 Whenever the http-equiv="content-language" attribute on the <meta> element specifies the language of the root element, then it is required to specify the language mapping of the root element. According to Content language state in [HTML5], the http-equiv="content-language" attribute on the <meta> element specifices the language of the root element whenever its content attribute contains no more and no less than exactly one language tag. 
[[

I believe this covers all of your points in comment #6. Thanks so much for keeping at this and following this through to the end. I appreciate the help.

Eliot

Super. The only thing that can destroy the consens we now have is if ISSUE-88 is resolved in a way which for instance makes the entire <meta http-equiv="content-language" ... > forbidden in HTML5 ... :-) 

But I take the chance and close this bug. 

But we must remember to look at this issue again, when ISSUE-88 is resolved.

Hold on. As far as I'm aware XML processors have no way of detecting the language of a document from a meta element.  The prescribed way according to the XML spec is to use the xml:lang attribute, and this is what XML processors such as XSLT expect to find for their functions. 

Since a polyglot document represents the subset of markup that works as both HTML and XML, then this note should NOT be included.  Language should be set using the xml:lang + lang attributes only to set language of the root element for polyglot documents. Even if the current HTML spec doesn't change, the use of meta for establishing the language of the root element is therefore moot.

Proposed action: remove the note.

(In reply to comment #9)
> Hold on. As far as I'm aware XML processors have no way of detecting the
> language of a document from a meta element.  The prescribed way according to
> the XML spec is to use the xml:lang attribute, and this is what XML processors
> such as XSLT expect to find for their functions. 
> 
> Since a polyglot document represents the subset of markup that works as both
> HTML and XML, then this note should NOT be included.  Language should be set
> using the xml:lang + lang attributes only to set language of the root element
> for polyglot documents. Even if the current HTML spec doesn't change, the use
> of meta for establishing the language of the root element is therefore moot.
> 
> Proposed action: remove the note.

Polyglot allows <meta charset="UTF-8"/> despite that it doesn't work in XML. We can say that it, in principle, allows <meta charset="<anyvalue>"/>.

*WHEN* <meta charset="<anyvalue>"/> is used, the encoding must simultaneously be declared/made known for XML parsers. And, because <?xml encoding="<anyvalue>"/> is forbidden, the  the  document, as well as the <meta charset="*"/> MUST - THUS -  be UTF-8. 

This is very similar to the issue at hand: *IF* <meta http-equiv="Content-Language" content="*" />
is present, then there is a REQUIREMENT to also delcare the language. Thus, both xml:lang="*" and lang="*" must, in that case bse used. (Otherwise it is completely up to the author whether he wants to declare a language or not.)

I can see - I agree - that the wording

]] Whenever the http-equiv="content-language" attribute on the <meta> element
specifies the language of the root element, [[

could be made clearer.  My proposal to Eliot is that the word 'specifies' is replacd with 'affects'.

The point was to say that when the <meta> content-language causes HTML-parsers to  _perceive_ a language, _then_ one MUST declare the language of the document.

Richard, if you think that http-equiv="content-language" should be forbidden to use in polyglot markup, then it would be consistent to require that Polyglot Markup says that polyglot markup does not contain <meta> http-equiv="content-language".

However, the purpose of <meta> http-equiv="content-language" is not to specify the language of the document, whether in HTML5 or in XML. The purpose is to specify who the language audience is  - et cetera et cetera.

If HTML5 had considered the <meta> http-equiv="content-language" as a method for specifying the language of the document, then we could say that it is forbidden in polyglot markup, as it does not specify the language in XML.

But HTML5, in my reading, only acknowledges that <meta> http-equiv="content-language" *affects* the language under certain conditions, and requires UAs to let it affect the language in a uniform way.

The spec already says:
"Polyglot markup uses the language attributes in the html element to set the default language for the document."

I understand this (because of the nature of the polyglot document) to be prescriptive, ie. it could be written: "If you want to set the default language of the document you must use language attributes in the html element."

If you use language attributes in the html element, the meta element has no role to play in defining the default language of a polyglot document.

This doesn't disallow the use of the meta element (eg. when used as metadata), but it makes irrelevant the mechanism associated with it for defining the default language of the document.

(In reply to comment #12)

> This doesn't disallow the use of the meta element (eg. when used as metadata),
> but it makes irrelevant the mechanism associated with it for defining the
> default language of the document.


But don't you agree, that authors needs to be made aware that they are, in effect (like it or not) declaring a language on the HTML side if they do use an element such as this:

<meta http-equiv="Content-Language" content="ru"/>

?

In other words: If an author does use the above element, then, in the HTML DOM, this will be perceived as the author had used <html lang="ru">. As a result, we get a difference between HTML DOM and XML DOM, which is the worst sin, in Polyglot Markup.

I don't want to forbid anything. I just want that when an author does use an element such as the above, then Polyglot Markup REQUIRES that the <html> tag contains a language declaration, like so:

<html lang="ru" xml:lang="ru" xmlns="http://www.w3.org/1999/xhtml" >

Or, like so for that matter (the language does not need to reflect the <meta> element):

<html lang="en" xml:lang="en" xmlns="http://www.w3.org/1999/xhtml" >

Authors must be made aware of all pitfalls within what is permitted to do, that can create different DOMs. 

Currently, HTML5 only permits those variants of the <meta http-equiv="content-language" content="*"/> that DO affect the DOM. So, unless HTML5 opens up an allows also those variants which have no effect on the document language, then we can simplify the note to say that 

<html lang="*" xml:lang="*" xmlns="http://www.w3.org/1999/xhtml" >

is required EVERYTIME the document has a 

<meta http-equiv="content-language" content="*"/>

declaration.

But as HTML5 is currently defined, I cannot accept that Polyglot Markup permits  <meta http-equiv="content-language" content="*"/> to be used without a REQUIREMENT  that authors to use xml:lang and lang on the root element.

(In reply to comment #12)

> If you use language attributes in the html element, the meta element has no
> role to play in defining the default language of a polyglot document.

Right. But we also have to consider that not every polyglot document will declare a language. And if a document without a language declaration nevertheless uses a http-equiv=content-language declaration, then what? 

> This doesn't disallow the use of the meta element (eg. when used as metadata),
> but it makes irrelevant the mechanism associated with it for defining the
> default language of the document.

The condition which permits that one can use the meta element for meta purposes is that there is a lang/xml:lang declaration.

COMPROMISE PROPOSAL: 

Perhaps we should, as you say, remove the note. And instead have another *section* which explains when and how to use <meta http-equiv="content-language" content="*"/>?

The new note could - as HTML5 currently stands - say that <meta http-equiv="content-language" content="*"/> can be used, provided that the document has a language declaration in the HTML element.

(In reply to comment #14)
> (In reply to comment #12)

> The new note

Please read that as "The new section". Thanks.

The polyglot document is fairly laconic, so Elliot may think that it is enough to say 'when defining the default language for the document use the language attributes' and leave it to others to explain the ins and outs.  If not, then as HTML5 currently stands, I think it may be better to have a note that says something like:

"If you do not use language attributes on the html element, you should not have a <meta type=content-language content=*> element with a single language value for the content attribute. This is because HTML5 interprets this as setting the default language for the root element, but XML does not."

or 

"If you have a <meta type=content-language content=*> element with a single language value for the content attribute, you must use language attributes on the html element. Otherwise HTML5 interprets this as setting the default language for the root element, but XML does not."

However, I actually think that we need to push for resolution of ISSUE-88 before worrying about the content-language meta in the polyglot document, since the current behaviour of content-language meta is currently under doubt.

(In reply to comment #16)

> However, I actually think that we need to push for resolution of ISSUE-88
> before worrying about the content-language meta in the polyglot document, since
> the current behaviour of content-language meta is currently under doubt.

Agreed.

However, I wanted to ask 2 questions  regarding comment #9:

> The prescribed way according to
> the XML spec is to use the xml:lang attribute, and this is what XML processors
> such as XSLT expect to find for their functions. 

QUESTION 1: Are XML processors, such as XSLT, required to react to HTTP Content-Language (coming from the web server), at all? 

I'll answer myself: they are not! This is pretty obvious because Content-Language is not meant to affect the language of the document .... Hence we must conclude that it isn't only http-equiv="Content-Language" that is dangerous when it comes to Polyglot Markup, but even Content-Language coming from the HTTP server. (And in my Change Proposal for ISSUE-88, I did therefore also speak about the HTTP server.)

CONCLUSION: Thus, we must realize that, on the HTML side, the language can be affected even by HTTP. (HTML5 does not attempt to regulate HTTP ...)

QUESTION 2: Thus, perhaps we can ignore ISSUE-88 and simply make it REQUIRED to use xml:lang/lang in polyglot markup?

Because, that is the only practical way to become immune against the HTTP effect.  Keep in mind, then that the attribute could be empty:

<html xml:lang="" lang="" xmlns="http://www.w3.org/1999/xhtml" >

(In reply to comment #17)

> QUESTION 1: Are XML processors, such as XSLT, required to react to HTTP
> Content-Language (coming from the web server), at all? 
> 
> I'll answer myself: they are not! This is pretty obvious because
> Content-Language is not meant to affect the language of the document .... 

Though XML 1.0 does say:

]] Language information may also be provided by external transport protocols (e.g. HTTP or MIME). When available, this information may be used by XML applications, but the more local information provided by xml:lang should be considered to override it.[[

Is it not likely that XML 1.0 speaks about Content-Language ?

The only difference seems to be that HTML5 _requires_ user agents to use the external information, whereas XML 1.0 only says that applications "might" do so. 

The net effect, though, is that XML and HTML are not guaranteed treat Content-Language the same way.

(In reply to comment #16)

> I think it may be better to have a note that says something like:
> 
> "If you do not use language attributes on the html element, you should not have
> a <meta type=content-language content=*> element 
   <ins> 
                 or a HTTP Content-Language: header
    </ins>
>   with a single language value
> for the content attribute. This is because HTML5 interprets this as setting the
> default language for the root element, but XML does not."
> 
> or 
> 
> "If you have a <meta type=content-language content=*> element 
   <ins> 
                 or a HTTP Content-Language: header
    </ins>
>  with a single
> language value for the content attribute, you must use language attributes on
> the html element. Otherwise HTML5 interprets this as setting the default
> language for the root element, but XML does not."

Those are fine, both of them. Except that they forget to mention the possibility of a HTTP Content-Language: header from the server. If that were added, then I could accept it.  I pasted in what I mean above.

Meanhile, I have filed Bug 12278 'Make lang and xml:lang required on the root element'.

In the editpr's draft of 18 March, I rewrote the note in section 7.2 so that it now reads:

]]
Whenever either the http-equiv="content-language" attribute on the <meta> element or an HTTP Content-Language: header specifies the language of the root element, then polyglot markup is required to specify the language mapping of the root element. According to Content language state in [HTML5], the http-equiv="content-language" attribute on the <meta> element specifices the language of the root element whenever its content attribute contains no more and no less than exactly one language tag. Therefore, not specifying the language mapping of the root element would mean that HTML5 would interpret this as setting the default language for the root element, while XML did not.
[[

I believe that takes care of this, though I wasn't totally certain about the request here. Let me know if I've missed something. 

Thanks,

Eliot

(In reply to comment #20)
Issues:
* Sentence 2 = incorrect: doesn't list HTTP Content-Language
* The HTML5 Reference should be the language determination rules
   http://www.w3.org/TR/html5/elements#language 
   and not the Content Languag state:
   http://dev.w3.org/html5/spec/semantics.html#attr-meta-http-equiv-content-language
* Second last sentence is incorrect: doesn't mention the crucial 
   condition that the language attributes of the root element are
   lacking ...
* A problem in the paragraph (the 'pretext') *preceding* the NOTE:
   # 'default' is synonymous with 'fallback'. 'fallback/default' is not the correct word when the language value stems from the language attributes. Only when it stems from HTML5's fallback mechanism, should we use default/fallback.



        New text, for the 'pretext' paragraph and for the NOTE:


]]
    Polyglot markup avoids that the language of the root element is set by HTML5's fallback language mechanism as this mechanism is not required to work in XML.

    [NOTE:] HTML5's fallback language mechanism activates whenever the root element is lacking language attributes. But for the mechanism to actually set a fallback language, it has to locate a http-equiv="Content-Language" meta element or a HTTP Content-Language: header (anyone of them, but meta element is considered first) whose content value is no more and no less than exactly a one language tag, see the <a href="http://www.w3.org/TR/html5/elements#language" >language determination rules</a> of [HTML5].
[[



PS: I said "not required to work in XML", because when I read the language determination rules of HTML5, then it *does* seem to work in "normal" XHTML browsers.
PPS: It is fully legal to uppercase the content value of @http-equiv. Hence I wrote http-equiv="Content-Language", to get it symmetrical with the typical way to type the HTTP header syntax.

(In reply to comment #21)
> (In reply to comment #20)

The HTMLwg just decided that http-equiv="Content-Language" should be illegal.

http://www.w3.org/mid/4D84B9B4.3040809@intertwingly.net

As a result, here is an updated proposal (I only added this string: "(which is non-conforming HTML5)":


]]
    Polyglot markup avoids that the language of the root element is set by
HTML5's fallback language mechanism as this mechanism is not required to work
in XML.

    [NOTE:] HTML5's fallback language mechanism activates whenever the root element is lacking language attributes. But for the mechanism to actually set a fallback language, it has to locate an http-equiv="Content-Language" meta element (which is non-conforming HTML5) or a HTTP Content-Language: header  (anyone of them, but meta element is considered first) whose content value is no more and no less than exactly a one language tag, see the <a href="http://www.w3.org/TR/html5/elements#language">language determination rules</a> of [HTML5].
[[

If the meta content-language element is obsoleted by the spec, will the fallback rules remain? It doesn't seem logical to forbid use of a construct then go on to describe how that construct affects behaviour in the browser. Why don't we wait a little to see how things settle before rewriting this text over and over.

> If the meta content-language element is obsoleted by the spec, will
> the fallback rules remain?

Yes, processing <meta http-equiv=content-language> in this way is
required, due to legacy content.

> It doesn't seem logical to forbid use of a construct then go on to
> describe how that construct affects behaviour in the browser.

It may not be logical, but as the #whatwg topic reminds us, we need to
leave our sense of logic at the door when it comes to the processing
required by legacy content. There are many, many features which are
considered nonconforming from an authoring standpoint but are
nevertheless specced in (sometimes excruciating) detail.

(In reply to comment #23)
> If the meta content-language element is obsoleted by the spec, will the
> fallback rules remain?

Yes. (If by "rules" you refer to how HTML5 user agents are required to treat it.)

The HTML5 Change Proposal that the chairs adopted
http://lists.w3.org/Archives/Public/public-html/2010Apr/0308
does not affect how user agents, according HTML5, are going to treat it.

See my explantion here: 
http://www.w3.org/mid/20110319171903849246.ab7bc210@xn--mlform-iua.no

> It doesn't seem logical to forbid use of a construct
> then go on to describe how that construct affects behaviour in the browser.

HTML5 forbids. But Polyglot Markup does not really forbid anything. Instead, the text in equestion  *prescribes* the use of @lang/@xml:lang to avoid the trouble that HTTP/http-equiv lang causes. It makes sense, to me, to describe why Polyglot Markup has this extra rule. That's similar to how Polyglot Markup gives a reason for a few other necessities.

Meanwhile the text speaks about both HTTP (which is not forbidden) and http-equiv. The problem with not mentioning http-equiv is that it gives the author a false security - he/she may think she/he is safe due to the omission of http-equiv. Whereas reality is that anyhting EXEPT the invalid http-equiv, with a *single*, *valid* language tag inside @content, will cause the UA to look inside the HTTP Content-Language: header.

>  Why
> don't we wait a little to see how things settle before rewriting this text over
> and over.

The "over and over" (_excluding the very last change_ proposed in Comment #22) has nothing to do with an instable HTML5 spec. Rather it has to do with difficulty in explaining the HTML5 rules. 

Comment #22 proposes some small changes to the proposal madein #21. These small chanvges are a result of the HTMLwg decision in ISSUE-88.

It looks to me as if ISSUE-88 has to be revisited: http://www.w3.org/mid/4D84E757.7070408@intertwingly.net
So the editor could of course wait and see. However, I think it is very good if the text tries to express the current status in HTML5 - that could help us in eventually bring ISSUE-88 up again.

Per the comments on this bug, I've amended Section 7.2 to now read:

]]
7.2 Language Attributes

When specifying the language mapping of an element, polyglot markup uses both the lang and xml:lang attributes. Neither attribute is to be used without the other, and polyglot markup maintains identical values for both lang and xml:lang.

Polyglot markup uses the language attributes in the html element to set the default language for the document overtly. Although HTML5 sets the language of the root element via a fallback language mechanism, this mechanism is not required to work in XML.

HTML5 activates the fallback language mechanism whenever the root element lacks language attributes. For the mechanism to actually set a fallback language, however, it has to locate either an http-equiv="Content-Language" declaration on the meta element or an HTTP Content-Language: header, either of whose content value is no more and no less than exactly one language tag. Note that although the mechanism can locate either the meta element or the header, the meta element is considered first. For more information about determining language in HTML5, see the language determination rules. [HTML5]. 
[[

I believe that this satisfies all of the comments in this bug, so I am once more going to resolve it as FIXED.

Thank you all for your continued vigilance and support on this and the rest of the spec. I truly appreciate your help.

Eliot

mass-move component to LC1

mass-move component to LC1