9985 2010-06-22 21:37:57 +0000 [parser] How to parse </foo </bar> 2010-10-04 14:57:02 +0000 1 1 1 Unclassified HTML WG pre-LC1 HTML5 spec (editor: Ian Hickson) unspecified All All RESOLVED WONTFIX http://www.macruby.org/ P1 critical --- 1 w3c ian ap bzbarsky hsivonen ian james mike public-html-admin public-html-wg-issue-tracking public-html-bugzilla oldest_to_newest 36358 0 w3c 2010-06-22 21:37:57 +0000 WebKit received a bug report [1] about a layout problem on http://www.macruby.org/ due to the HTML5 parsing algorithm. (You can visit the site in a Firefox or WebKit nightly build to see the issue.) The trouble boils down to this reduction: Should say PASS: <div> <div style="visibility:hidden"> <p></p </div> PASS </div> Essentially, the missing ">" on the close tag of the <p> element causes the tokenizer to consume the </div> characters as well, resulting in the wrong DOM. According to my tests, both the legacy WebKit parser and the legacy Firefox parser terminate a tag token upon encountering a "<" character. The HTML5 spec recognizes that case as a parse error, but has different error recovery. (This issue is on our "top five" list of behavioral differences likely to cause compatibility problems.) Is there a particular reason why we don't terminate start and end tag tokens upon encountering a "<" character? [1] https://bugs.webkit.org/show_bug.cgi?id=40961 36359 1 w3c 2010-06-22 21:52:15 +0000 [2:38pm] Hixie: abarth: the </p</div> thing was for compat with IE [2:39pm] abarth: Hixie: i see. some folks in the webkit community are worried about whether that will cause problems for mobile sites that are used to a webkit monoculture [2:39pm] Hixie: yup, it probably will. [2:40pm] zcorpan_: abarth: opera does the same thing as IE for < in tags. we've found that there are some pages that break if we follow mozilla, and other pages break if we follow ie [2:40pm] zcorpan_: abarth: however we've followed ie here for a long time and it doesn't come up very often [2:48pm] abarth: zcorpan_: i see [2:48pm] abarth: zcorpan_: maybe its a crapshoot either way [2:48pm] abarth: i'm all for aligning behavior so we don't have these problems in the future [2:49pm] abarth: i'm not sure how to estimate the risks in the mobile space [2:49pm] abarth: but there are also likely counterbalancing risks in the intranet that IE folks are worried about 36394 2 w3c 2010-06-24 00:42:18 +0000 Same problem on this site: http://swlab.mt.haw-hamburg.de/~s1991802/cornick/kontakt.php 36856 3 ian 2010-07-14 18:30:59 +0000 EDITOR'S RESPONSE: This is an Editor's Response to your comment. If you are satisfied with this response, please change the state of this bug to CLOSED. If you have additional information and would like the editor to reconsider, please reopen this bug. If you would like to escalate the issue to the full HTML Working Group, please add the TrackerRequest keyword to this bug, and suggest title and text for the tracker issue; or you may create a tracker issue yourself, if you are able to do so. For more details, see this document: http://dev.w3.org/html5/decision-policy/decision-policy.html Status: Rejected Change Description: no spec change Rationale: Since the spec is compatible with IE, I think it's not worth changing at this point. Either way we'll have compatibility problems. 39050 4 hsivonen 2010-09-15 11:36:04 +0000 Hixie, do you have quantitative data showing that doing this the IE way is better than doing this the old WebKit way (or the old Gecko way, which was slightly different)? Now it appears that Apple is adding app-specific hacks to the system WebKit because of this. It's really sad if any vendor has to add app-specific hacks because of this, but at least Microsoft already has chosen to bear the engine versioning burden independently of this issue. Interesting bits from IRC (see http://krijnhoetmer.nl/irc-logs/whatwg/20100915 ) # [11:27] <hsivonen> othermaciej: what HTML5 parsing difference from old WebKit is breaking mail apps with system WebKit? # [11:27] <othermaciej> hsivonen: <foo<foo> # [11:28] <hsivonen> othermaciej: how does Outlook deal? # [11:28] <hsivonen> does Outlook use the Word engine these days? does Word parse differently from Trident? # [11:28] <othermaciej> I believe Outlook uses the Word engine # [11:30] <hsivonen> othermaciej: it would good to know if the emails are generated by an email app in the wild or if they are hand-crafted advertisements # [11:30] <othermaciej> I believe at least some of them were produced by an automated reporting system of some kind # [11:33] <hsivonen> annevk: wouldn't it then make sense to pick the solution that sucks less for editors? # [11:34] <hsivonen> now we've picked the solution that makes the tokenizer code simpler # [11:50] <zcorpan_> http://html5.org/tools/web-apps-tracker?from=899&to=902 even # [11:56] <zcorpan_> http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2007-June/011804.html # [11:57] <zcorpan_> iirc, it was a web compat requirement to not close script for </script<div> # [11:58] <othermaciej> annevk: it's by far the top source of breakage for us (other than just plain implementation bugs, which are mostly now fixed) # [12:06] <zcorpan_> http://www.gearthblog.com/blog/archives/2006/06/more_detail_on.html has </ul <div> (and looks broken with html5 parser) # [12:07] <Philip`> http://philip.html5.org/data/gt-in-tag.txt has <foo<foo>s in case anyone is looking for those # [12:07] <Philip`> (Ignore the filename, it lies) # [12:20] <annevk> e.g. for http://pageranking.cbgw-lensahn-slh.de/ HTML5 is better # [12:24] <othermaciej> some bad content had this in it: <style type='text/css'td{width='60%' cellpadding='20%'}</style> # [12:25] <othermaciej> which ate the rest of the page instead of making an empty style element with some bogus attributes # [12:38] <jgraham> You need to weight by badness of the problem of course # [12:39] <jgraham> Like eating the whole page on a few pages is worse than slight issues on more pages # [12:46] <zcorpan_> if we change this, we need to investigate carefully what to change to. old webkit and gecko don't agree in all cases (iirc) and they don't make </script<div> close the script, iirc # [12:46] <hsivonen> what did Opera do in 2006? # [12:47] <hsivonen> it would suck to use circular reasoning to make HTML5 do something, because of Opera if Opera changed to match HTML5 # [12:50] <zcorpan_> oh, previously we parsed <p<div> as <p <div=""> i.e. with an attribute "<div" # [12:52] <zcorpan_> so we were still closer to ie than gecko and webkit for both <p<div> and <p <div> # [12:54] <zcorpan_> we fixed that in 2008 to match ie and html5 # [12:59] <hsivonen> https://bugzilla.mozilla.org/show_bug.cgi?id=507498 # [13:00] <hsivonen> https://bugzilla.mozilla.org/show_bug.cgi?id=510252 # [13:00] <hsivonen> https://bugzilla.mozilla.org/show_bug.cgi?id=523516 # [13:00] <hsivonen> https://bugzilla.mozilla.org/show_bug.cgi?id=543652 # [13:01] <hsivonen> https://bugzilla.mozilla.org/show_bug.cgi?id=590416 # [13:11] <zcorpan_> also see http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2007-June/011891.html 39051 5 hsivonen 2010-09-15 11:46:35 +0000 Reopening to make sure Hixie sees the new comment. 39056 6 ian 2010-09-15 14:40:29 +0000 I'm not sure how to measure whether one way is quantitatively better or worse than the other. In general the parser was designed to match IE when browsers differed and the IE behaviour was not unreasonable. I'm happy to change the spec here is there is evidence that the spec isn't compatible with the Web. However, I'm very curious to hear how IE handles these cases. Is the behaviour specified here not actually a complete description of what IE does? 39062 7 w3c 2010-09-15 19:25:07 +0000 My thinking on this topic has evolved a bit since I filed this bug report. Here are my current thoughts: == New data == 1) This authoring error is somewhat common (at least based on data from greping the web), but different pages expect browsers to tokenize these cases different based on their audience. The majority of pages on the public web (and like in most intranets) expect browsers to tokenize these cases in the IE way. 2) The cases we've seen where pages expect us to tokenize these cases in the WebKit way have almost exclusively been content authored only for WebKit. For example, I filed this report in response to a bug on macruby.org, which is a web site hosted by Apple for folks with Mac computers, which are more likely to run WebKit than to run IE. 3) The biggest problems with the spec's current tokenization have been in Mac applications that use WebKit internally to render internally generated content. Most notably, AIM on Mac appears to rely on WebKit's legacy tokenization behavior to function. Apple also has problems on its internal web sites because those web sites are almost exclusively viewed using WebKit. 4) In the rare cases we've seen on the public web where the spec's tokenization cases problems, it's been easy to evangalize. I attribute this to two reasons: A) The busted markup looks dumb. In one case the author even apologized for making that mistake. B) The pages are busted in the same way in IE. By and large, folks want their site to work in IE. 5) The area I'm most concerned about is the mobile web, by which I mean web sites designed and optimized for mobile browsers. Until recently, mobile browsing was a pretty serious WebKit monoculture, which means these pages are likely to expect the legacy WebKit tokenization. == Takeaways == A) It seems likely the spec's current tokenization in this case is more compatible with both the public web and with private intranets than the legacy WebKit-behavior. B) It seems likely the spec's current tokenization in this case is less compatibile with the mobile web than the legacy WebKit-behavior. C) In order to not break legacy Mac applications that use WebKit internally, Apple will need to ship the legacy WebKit tokenization to these applications. (My understanding is that it will be limited to the specific applications and versions that are problematic.) == Conclusions == This is going to cause pain either way. We should pick a path that minimizes long-term pain here, even if it means more pain in the short term. I believe the best road to less long-term pain is to match IE's tokenization in this case, in no small part because I think it's unlikely that IE will change their tokenization for the foreseeable future. 39067 8 913 Ms2ger 2010-09-16 09:19:17 +0000 Created attachment 913 http://www.freetype.org/patents.html: contains </em</a> 39068 9 w3c 2010-09-16 09:45:07 +0000 > http://www.freetype.org/patents.html: contains </em</a> There are lots of examples of these pages. I have a list of thousands of them somewhere I could dig up. In this case, the page looks fine under either tokenization. It took me a while to spot the difference, but the legacy WebKit behavior is slightly better. 39119 10 ian 2010-09-20 20:30:44 +0000 EDITOR'S RESPONSE: This is an Editor's Response to your comment. If you are satisfied with this response, please change the state of this bug to CLOSED. If you have additional information and would like the editor to reconsider, please reopen this bug. If you would like to escalate the issue to the full HTML Working Group, please add the TrackerRequest keyword to this bug, and suggest title and text for the tracker issue; or you may create a tracker issue yourself, if you are able to do so. For more details, see this document: http://dev.w3.org/html5/decision-policy/decision-policy.html Status: Rejected Change Description: no spec change Rationale: Closing per comment 7. The reasoning there is pretty much the reasoning I used to prefer matching IE over other browsers when there was a difference between browsers; it is unfortunately the case that these differences will cause pain either way, as Adam noted. 913 2010-09-16 09:19:17 +0000 2010-09-16 09:19:17 +0000 http://www.freetype.org/patents.html: contains </em</a> freetype-patents-20100719.html text/html 3517 Ms2ger PCFET0NUWVBFIEhUTUwgUFVCTElDCiAgICAiLS8vVzNDLy9EVEQgSFRNTCA0LjAgVHJhbnNpdGlv bmFsLy9FTiIKICAgICJodHRwOi8vd3d3LnczLm9yZy9UUi9SRUMtaHRtbDQwL2xvb3NlLmR0ZCI+ CjxodG1sPgoKPGhlYWQ+CiAgPG1ldGEgaHR0cC1lcXVpdj0iQ29udGVudC1UeXBlIgogICAgICAg IGNvbnRlbnQ9InRleHQvaHRtbDsgY2hhcnNldD1VVEYtOCI+CiAgPG1ldGEgbmFtZT0iZGVzY3Jp cHRpb24iCiAgICAgICAgY29udGVudD0iRnJlZVR5cGUgYW5kIFBhdGVudHMiPgogIDxtZXRhIG5h bWU9ImtleXdvcmRzIgogICAgICAgIGNvbnRlbnQ9IkZyZWVUeXBlIHBhdGVudHMgVHJ1ZVR5cGUg QXBwbGUgbGVnYWwgaW5mcmluZ2luZyBpbnRlcnByZXRlciI+CiAgPGxpbmsgcmVsPSJzdHlsZXNo ZWV0IgogICAgICAgIGhyZWY9ImZyZWV0eXBlLmNzcyI+CiAgPHRpdGxlPkZyZWVUeXBlIGFuZCBQ YXRlbnRzPC90aXRsZT4KPC9oZWFkPgoKPGJvZHkgdGV4dD0iIzAwMDAwMCIKICAgICAgYmdjb2xv cj0iI0ZGRkZGRiIKICAgICAgbGluaz0iIzAwMDBFRiIKICAgICAgdmxpbms9IiM1MTE4OEUiCiAg ICAgIGFsaW5rPSIjRkYwMDAwIj4KCjx0YWJsZSB3aWR0aD0iMTAwJSIKICAgICAgIGJvcmRlcj0i MCIKICAgICAgIGNlbGxzcGFjaW5nPSIwIgogICAgICAgY2VsbHBhZGRpbmc9IjUiPgogIDx0cj4K ICAgIDx0ZCBiZ2NvbG9yPSIjMDY0MjVGIj4KICAgICAgPGEgaHJlZj0iaW5kZXgyLmh0bWwiPgog ICAgICAgIDxpbWcgc3JjPSJpbWFnZS9mb25kMy5qcGciCiAgICAgICAgICAgICBhbGlnbj0icmln aHQiCiAgICAgICAgICAgICBib3JkZXI9IjAiCiAgICAgICAgICAgICBoc3BhY2U9IjAiCiAgICAg ICAgICAgICB2c3BhY2U9IjAiCiAgICAgICAgICAgICBhbHQ9IkZyZWVUeXBlIEhvbWVwYWdlIj4K ICAgICAgPC9hPgogICAgPC90ZD4KICA8L3RyPgoKICA8dHI+CiAgICA8dGQgYmdjb2xvcj0iIzA2 NDI1RiI+CiAgICAgIDxoMSBjbGFzcz0iaW50cm8iPgogICAgICAgIEZSRUVUWVBFICYgUEFURU5U UwogICAgICA8L2gxPgogICAgPC90ZD4KICA8L3RyPgoKICA8dHI+CiAgICA8dGQgYmdjb2xvcj0i IzY2OTk5OSI+CiAgICAgICZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOzxhIGhy ZWY9ImluZGV4Mi5odG1sIgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICBjbGFzcz0iaW5kZXgiPkhvbWVwYWdlPC9hPgogICAgPC90ZD4KICA8L3RyPgo8L3RhYmxl PgoKPHRhYmxlIGFsaWduPSJjZW50ZXIiCiAgICAgICB3aWR0aD0iODAlIj4KICA8dHI+CiAgICA8 dGQ+CiAgICAgIDxoMSBjbGFzcz0ic2VjdGlvbiI+CiAgICAgICAgVGhlIFRydWVUeXBlIGJ5dGVj b2RlIHBhdGVudHMgaGF2ZSBleHBpcmVkIQogICAgICA8L2gxPgoKICAgICAgPHA+U2luY2UgTWF5 IDIwMTAsIGFsbCBwYXRlbnRzIHJlbGF0ZWQgdG8gYnl0ZWNvZGUgaGludGluZyBoYXZlCiAgICAg IGV4cGlyZWQgd29ybGR3aWRlLiAgSXQgaXQgdGh1cyBubyBsb25nZXIgbmVjZXNzYXJ5IHRvIGRp c2FibGUgdGhlCiAgICAgIGJ5dGVjb2RlIGludGVycHJldGVyLCBhbmQgc3RhcnRpbmcgd2l0aCBG cmVlVHlwZSB2ZXJzaW9uIDIuNCwgaXQgaXMKICAgICAgZW5hYmxlZCBieSBkZWZhdWx0LjwvcD4K CiAgICAgIFRoZSBhZmZlY3RlZCBwYXRlbnRzIGFyZQogICAgICAKICAgICAgPHVsPgogICAgICAg IDxsaT4KICAgICAgICAgIDxwPjxhCiAgICAgICAgICBocmVmPSJodHRwOi8vd3d3LmRlbHBoaW9u LmNvbS9kZXRhaWxzP3BuPVVTMDUxNTU4MDVfXyI+UGF0ZW50Jm5ic3A7IzE6CiAgICAgICAgICBV UzUxNTU4MDU6IDxlbT5NZXRob2QgYW5kIGFwcGFyYXR1cyBmb3IgbW92aW5nIGNvbnRyb2wgcG9p bnRzIGluCiAgICAgICAgICBkaXNwbGF5aW5nIGRpZ2l0YWwgdHlwZWZhY2Ugb24gcmFzdGVyIG91 dHB1dCBkZXZpY2VzPC9lbTwvYT48L3A+CiAgICAgICAgPC9saT4KICAgICAgICA8bGk+CiAgICAg ICAgICA8cD48YQogICAgICAgICAgaHJlZj0iaHR0cDovL3d3dy5kZWxwaGlvbi5jb20vZGV0YWls cz9wbj1VUzA1MTU5NjY4X18iPlBhdGVudCZuYnNwOyMyOgogICAgICAgICAgVVM1MTU5NjY4OiA8 ZW0+TWV0aG9kIGFuZCBhcHBhcmF0dXMgZm9yIG1hbmlwdWxhdGluZyBvdXRsaW5lcyBpbgogICAg ICAgICAgaW1wcm92aW5nIGRpZ2l0YWwgdHlwZWZhY2Ugb24gcmFzdGVyIG91dHB1dCBkZXZpY2Vz PC9lbT48L2E+PC9wPgogICAgICAgIDwvbGk+CiAgICAgICAgPGxpPgogICAgICAgICAgPHA+PGEK ICAgICAgICAgIGhyZWY9Imh0dHA6Ly93d3cuZGVscGhpb24uY29tL2RldGFpbHM/cG49VVMwNTMy NTQ3OV9fIj5QYXRlbnQmbmJzcDsjMzoKICAgICAgICAgIFVTNTMyNTQ3OTogPGVtPk1ldGhvZCBh bmQgYXBwYXJhdHVzIGZvciBtb3ZpbmcgY29udHJvbCBwb2ludHMgaW4KICAgICAgICAgIGRpc3Bs YXlpbmcgZGlnaXRhbCB0eXBlZmFjZSBvbiByYXN0ZXIgb3V0cHV0IGRldmljZXM8L2VtPjwvYT48 L3A+CiAgICAgICAgPC9saT4KICAgICAgPC91bD4KCiAgICAgIDxoMSBjbGFzcz0ic2VjdGlvbiI+ CiAgICAgICAgT3RoZXIgcGF0ZW50IGlzc3VlcwogICAgICA8L2gxPgoKICAgICAgPHA+VGhlIGNv bG91ciBmaWx0ZXJpbmcgYWxnb3JpdGhtIG9mIE1pY3Jvc29mdCdzIENsZWFyVHlwZSB0ZWNobm9s b2d5CiAgICAgIGZvciBzdWJwaXhlbCByZW5kZXJpbmcgaXMgY292ZXJlZCBieSBwYXRlbnRzLiAg Tm90ZSB0aGF0IHN1YnBpeGVsCiAgICAgIHJlbmRlcmluZyBwZXIgc2UgaXMgcHJpb3IgYXJ0OyB1 c2luZyBhIGRpZmZlcmVudCBjb2xvdXIgZmlsdGVyIHRodXMKICAgICAgY2lyY3VtdmVudHMgTWlj cm9zb2Z0J3MgcGF0ZW50IGNsYWltcy48L3A+CgogICAgPC90ZD4KICA8L3RyPgo8L3RhYmxlPgoK PGZvbnQgc2l6ZT0tMz5MYXN0IHVwZGF0ZTogMDQtSnVsLTIwMTA8L2ZvbnQ+Cgo8dGFibGUgd2lk dGg9IjEwMCUiCiAgICAgICBib3JkZXI9IjAiCiAgICAgICBjZWxsc3BhY2luZz0iMCIKICAgICAg IGNlbGxwYWRkaW5nPSI1Ij4KICA8dHI+CiAgICA8dGQgYmdjb2xvcj0iIzY2OTk5OSI+CiAgICAg ICZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOzxhIGhyZWY9ImluZGV4Mi5odG1s IgogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBjbGFzcz0iaW5k ZXgiPkhvbWVwYWdlPC9hPgogICAgPC90ZD4KICA8L3RyPgoKICA8dHI+CiAgICA8dGQgYmdjb2xv cj0iIzA2NDI1RiI+CiAgICAgIDxhIGhyZWY9ImluZGV4Lmh0bWwiPgogICAgICAgIDxpbWcgc3Jj PSJpbWFnZS9mb25kMy5qcGciCiAgICAgICAgICAgICBhbGlnbj0icmlnaHQiCiAgICAgICAgICAg ICBib3JkZXI9IjAiCiAgICAgICAgICAgICBoc3BhY2U9IjAiCiAgICAgICAgICAgICB2c3BhY2U9 IjAiCiAgICAgICAgICAgICBhbHQ9IkZyZWVUeXBlIEhvbWVwYWdlIj4KICAgICAgPC9hPgogICAg PC90ZD4KICA8L3RyPgo8L3RhYmxlPgoKPC9ib2R5Pgo8L2h0bWw+Cg==