12242 2011-03-05 02:39:03 +0000 Make UTF-16 an invalid encoding in Polyglot Markup (i18n-issue-17) 2011-08-04 05:07:22 +0000 1 1 1 Unclassified HTML WG LC1 HTML/XHTML Compatibility Authoring Guide (ed: Eliot Graff) unspecified PC All CLOSED FIXED http://dev.w3.org/html5/html-xhtml-author-guide/html-xhtml-authoring-guide.html#character-encoding P2 normal --- 1 xn--mlform-iua eliotgra addison cowan eliotgra ishida mike public-html-admin public-html-wg-issue-tracking shadow2531 www-international xn--mlform-iua public-html-bugzilla oldest_to_newest 46388 0 xn--mlform-iua 2011-03-05 02:39:03 +0000 * According to HTML5, HTML-parsers must as minimum support UTF-8 and Windows-1252. http://dev.w3.org/html5/spec/parsing.html#character-encodings-0 * While according to XML, XML-parsers must as mininum support UTF-8 and UTF-16. * Polyglot Markup, though, "prefers" UTF-8 (based on HTML5's UTF-8 preference, one should think), but else follows the XML approach and permits both UTF-8 or UTF-16. AS A RESULT, it becomes possible to author "polyglot markup" that works fine in XML-parsers, but which isn't required to work in all and any HTML-parser. We should not declare mark-up that isn't required to work in a HTML-parser as "polyglot markup". Hence we should conclude that UTF-16 should not be a recommended encoding for Polyglot Markup. Discussion: * It was suggested early on, e.g. by e.g. Sam Ruby, that UTF-8 should be the only recommended encoding for polyglot markup. And this can be a very useful suggestion. For instance, it would become a very useful way to "force" many HTML editing programs to default to UTF-8, one should think. It also meets HTML5 which says that new documents SHOULD default to UTF-8. * However, the problem is to justify *exclusion* of UTF-16 by inference from the specs. Because, the use of UTF-16 does not seem to break with the principles behind Polyglot Markup, as laid out in its introduction: http://dev.w3.org/html5/html-xhtml-author-guide/html-xhtml-authoring-guide.html#introduction * Permission to use UTF-16 in polyglot markup is logical, for instance because - UTF-16 can be reliably detected via the BOM, in both XMLand HTML5 - though HTML5 says that, quote: "Using non-UTF-8 encodings can have unexpected results on form submission and URL encodings, which use the document's character encoding by default", the use of non-UTF-8 probably creates form problems in XML-on-the-web too. Thus XML and HTML are probaly in same boat here - and hence it does not seem logical to use against UTF-16 that some form submission problems could occur. * That said, the problems with non-UTF-8 *should* carry *some* weight: e.g. those form submission problems could cause greater problems in XML and it is a small irriation that it is not permitted/ possible to use an explicit character declaration in UTF-16 encoded documents. However, the fact that HTML-parsers aren't required to support UTF-16, is a more fundamental nail in the coffin. Can it have any real-world effect? Not so much when it comes to "big" browsers - they support multiple encodings. But for "simpllistic" parsers of differnent kinds, it could probably have an effect. 46636 1 eliotgra 2011-03-11 18:22:31 +0000 I've removed all references to UTF-16 within the polyglot spec in the 11 March Editor's Draft. If you log a bug against the HTML5 spec to _require_ UTF-16, please notify me. Thanks, again, for your continued help. Eliot 46642 2 xn--mlform-iua 2011-03-12 17:45:08 +0000 (In reply to comment #1) > I've removed all references to UTF-16 within the polyglot spec in the 11 March > Editor's Draft. Super. > If you log a bug against the HTML5 spec to _require_ UTF-16, please notify me. ;-) 46822 3 cowan 2011-03-18 18:59:28 +0000 Mark Davis mentioned to me that UTF-16 documents constitute less than 0.1% of the Web, so it's a fairly marginal encoding. According to http://googleblog.blogspot.com/2010/01/unicode-nearing-50-of-web.html (which is more than a year old), the web is 70% ASCII and UTF-8, 20% 8859-1/Win1252/8859-15, and 10% for all other encodings. Scrubbing UTF-16 is clearly the Right Thing. 47006 4 addison 2011-03-28 15:32:30 +0000 The Internationalization Core WG has discussed this issue (as I18N-ISSUE-17) and has no objection to making invalid the UTF-16 character encoding for Polyglot documents. See also: http://lists.w3.org/Archives/Public/public-i18n-core/2011JanMar/0132.html http://www.w3.org/2011/03/23-i18n-minutes.html#item06 53131 5 mike 2011-08-04 05:06:58 +0000 mass-move component to LC1 53179 6 mike 2011-08-04 05:07:22 +0000 mass-move component to LC1