10806 2010-09-29 11:45:56 +0000 ignoring escapes is not needed for compatibility with existing content 2011-03-24 12:38:22 +0000 1 1 1 Unclassified HTML WG pre-LC1 HTML5 spec (editor: Ian Hickson) unspecified PC Windows NT CLOSED WONTFIX http://dev.w3.org/html5/spec/Overview.html#content-type-sniffing NE, WGDecision P2 normal --- 10804 1 julian.reschke ian annevk hsivonen ian mike mjs public-html-admin public-html-wg-issue-tracking rubys public-html-bugzilla oldest_to_newest 39929 0 julian.reschke 2010-09-29 11:45:56 +0000 <http://dev.w3.org/html5/spec/Overview.html#content-type-sniffing>: "If it is a U+0022 QUOTATION MARK ('"') and there is a later U+0022 QUOTATION MARK ('"') in s If it is a U+0027 APOSTROPHE ("'") and there is a later U+0027 APOSTROPHE ("'") in s Return the encoding corresponding to the string between this character and the next earliest occurrence of this character." This is indeed a violation of the Content-Type syntax defined in RFC 2616, in not handling backslash-escapes inside quoted-string properly. The spec claims that this is required for "backwards compatibility with legacy content". I'm attaching a test case that shows that the following browsers *do* handle escapes despite what the spec says: Opera, Safari, Konqueror 4.4 Please remove the requirement to violate the base syntax. 39930 1 917 julian.reschke 2010-09-29 11:46:35 +0000 Created attachment 917 test case for backslash-escapes in quoted-string 39931 2 annevk 2010-09-29 11:51:29 +0000 It seems better for Opera to match Chrome / Gecko / IE here. We may very well hit problems because of this. 39932 3 julian.reschke 2010-09-29 12:03:21 +0000 (In reply to comment #2) > It seems better for Opera to match Chrome / Gecko / IE here. We may very well > hit problems because of this. Please elaborate. The backslash character isn't even allowed in charset names. I believe that requiring an incompatibility with a base spec needs much stronger justification. 39933 4 annevk 2010-09-29 12:07:06 +0000 (In reply to comment #3) > The backslash character isn't even allowed in charset names. Right, so we may need to ignore the rule rather than handle it. 39937 5 julian.reschke 2010-09-29 12:22:31 +0000 (In reply to comment #4) > Right, so we may need to ignore the rule rather than handle it. Nope, unless you can prove that there is existing content that "breaks" because it uses "\x" although it wants to say "x". In absence of that proof, the right thing to do is to handle escape sequences as specified. 40045 6 ian 2010-09-30 03:54:05 +0000 EDITOR'S RESPONSE: This is an Editor's Response to your comment. If you are satisfied with this response, please change the state of this bug to CLOSED. If you have additional information and would like the editor to reconsider, please reopen this bug. If you would like to escalate the issue to the full HTML Working Group, please add the TrackerRequest keyword to this bug, and suggest title and text for the tracker issue; or you may create a tracker issue yourself, if you are able to do so. For more details, see this document: http://dev.w3.org/html5/decision-policy/decision-policy.html Status: Rejected Change Description: no spec change Rationale: WebKit trunk doesn't support this. Are you sure you haven't set your default encoding to UTF-8 in Safari? What version are you testing? Opera doesn't support this either. It just ignores all punctuation. For instance, see: http://www.hixie.ch/tests/adhoc/html/parsing/encoding/121.html This is known to be incompatible with legacy content, however (the spec used to do this too, but had to change for compat reasons). 40069 7 julian.reschke 2010-09-30 07:41:24 +0000 1) I'm running Safari 5.0.2 on Windows, and "Text Encoding" is set to "default". 2) The question of *why* Opera is handling the escapes properly isn't really relevant; what counts is the observable behavior. You have claimed that this violation is needed for "compatibility", yet the shipping versions of Opera and Safari on Windows, nor Konqueror do this. I think this is proof that the claim is incorrect. Please provide more data. 40072 8 ian 2010-09-30 07:52:33 +0000 What does Safari get for you on this test (the expected and actual encodings, not the pass/fail state)?: http://www.hixie.ch/tests/adhoc/html/parsing/encoding/113.html 40083 9 julian.reschke 2010-09-30 08:14:14 +0000 (In reply to comment #8) > What does Safari get for you on this test (the expected and actual encodings, > not the pass/fail state)?: > http://www.hixie.ch/tests/adhoc/html/parsing/encoding/113.html Expected result: Windows-1252 Encoding used by browser is: Windows-1254 40084 10 ian 2010-09-30 08:23:12 +0000 Upon further investigation, it turns out WebKit has changed behaviour. It used to ignore all punctuation (it didn't support backslash escapes, just ignored punctuation like Opera). Newer trunk builds however now don't ignore punctuation. (In reply to comment #7) > 2) The question of *why* Opera is handling the escapes properly isn't really > relevant; what counts is the observable behavior. The point is that Opera isn't handling the escapes at all. For example, if you put a backslash before the final quote, it doesn't continue the string. It still ends at the quote. For example, consider: http://www.hixie.ch/tests/adhoc/html/parsing/encoding/122.html http://www.hixie.ch/tests/adhoc/html/parsing/encoding/123.html EDITOR'S RESPONSE: This is an Editor's Response to your comment. If you are satisfied with this response, please change the state of this bug to CLOSED. If you have additional information and would like the editor to reconsider, please reopen this bug. If you would like to escalate the issue to the full HTML Working Group, please add the TrackerRequest keyword to this bug, and suggest title and text for the tracker issue; or you may create a tracker issue yourself, if you are able to do so. For more details, see this document: http://dev.w3.org/html5/decision-policy/decision-policy.html Status: Rejected Change Description: no spec change Rationale: As far as I can tell, no browsers with more than 1% market share do anything with backslashes at all, and the browsers that ignored backslashes are actively moving towards what the spec says. 40107 11 julian.reschke 2010-09-30 08:58:53 +0000 You still haven't provided evidence that this is needed for compatibility. Will raise a tracker issue. 40129 12 hsivonen 2010-09-30 09:51:52 +0000 (In reply to comment #11) > You still haven't provided evidence that this is needed for compatibility. > > Will raise a tracker issue. Are you suggesting that backlash-processing complexity be introduced into implementations without compatibility with existing content requiring it? 40130 13 julian.reschke 2010-09-30 10:02:29 +0000 What I'm suggesting is that implementations should treat quoted-strings in HTTP parameters uniformly, so special-casing Content-Type is *adding* complexity. 40131 14 annevk 2010-09-30 10:06:48 +0000 Opera will very soon match Safari nightlies and everyone else here. Us "supporting" this was just a side effect of supporting the way too liberal UTS22 matching rules. 40133 15 julian.reschke 2010-09-30 10:17:55 +0000 (In reply to comment #14) > Opera will very soon match Safari nightlies and everyone else here. Us > "supporting" this was just a side effect of supporting the way too liberal > UTS22 matching rules. A, self-fulfilling prophecy. Anyway, I have shown that the claim that this is "required for compatibility" is wrong; thus will escalate the issue. 40140 16 julian.reschke 2010-09-30 11:54:52 +0000 Raised as http://www.w3.org/html/wg/tracker/issues/126 917 2010-09-29 11:46:35 +0000 2010-09-29 11:46:35 +0000 test case for backslash-escapes in quoted-string enctest2.html text/html 2473 julian.reschke PCFET0NUWVBFIEhUTUw+DQo8bWV0YSBodHRwLWVxdWl2PSJDb250ZW50LVR5cGUiIGNvbnRlbnQ9 J3RleHQvaHRtbDsgY2hhcnNldD0iXFVcVEYtOCInPg0KPHA+VGVzdDoNCjxwcmU+Jmx0OyFET0NU WVBFIEhUTUw+DQombHQ7bWV0YSBodHRwLWVxdWl2PSJDb250ZW50LVR5cGUiIGNvbnRlbnQ9J3Rl eHQvaHRtbDsgY2hhcnNldD0iXFVcVEYtOCInPjwvcHJlPg0KPHA+RXhwZWN0ZWQgcmVzdWx0OiA8 c3BhbiBpZD0iZXhwZWN0ZWQiPlVURi04PC9zcGFuPg0KPGRpdj4NCiA8c3R5bGUgc2NvcGVkPg0K ICAucGFzcyB7IGJhY2tncm91bmQ6IGdyZWVuOyBjb2xvcjogd2hpdGU7IHBhZGRpbmc6IDAuNWVt OyBmb250LXdlaWdodDogYm9sZDsgfQ0KICAuZmFpbCB7IGJhY2tncm91bmQ6IHJlZDsgY29sb3I6 IHllbGxvdzsgcGFkZGluZzogMC41ZW07IGZvbnQtd2VpZ2h0OiBib2xkOyB9DQogPC9zdHlsZT4N CiA8cD5FbmNvZGluZyB1c2VkIGJ5IGJyb3dzZXIgaXM6IDxzcGFuIGlkPSJlbmNvZGluZyI+U2Ny aXB0IGRpZCBub3QgcnVuLjwvc3Bhbj4NCiA8cD5SZXN1bHQ6IDxzcGFuIGlkPSJyZXN1bHQiPlNj cmlwdCBkaWQgbm90IHJ1bi48L3NwYW4+DQogPHNjcmlwdD4NCiAgIHZhciBlbmNvZGluZyA9ICd1 bmtub3duJzsNCiAgIGlmICgn4CcgPT0gJ1x1MDBFMCcpIHsgLy8gMHhFMA0KICAgICBpZiAoJ5kn ID09ICdcdTIxMjInKSB7IC8vIDB4OTkNCiAgICAgICBpZiAoJ/4nID09ICdcdTAwRkUnKSAvLyAw eEZFDQogICAgICAgICBlbmNvZGluZyA9ICdXaW5kb3dzLTEyNTInOw0KICAgICAgIGVsc2UgaWYg KCf+JyA9PSAnXHUwMTVGJykNCiAgICAgICAgIGVuY29kaW5nID0gJ1dpbmRvd3MtMTI1NCc7DQog ICAgICAgZWxzZSBpZiAoJ/4nID09ICdcdTIwMEYnKQ0KICAgICAgICAgZW5jb2RpbmcgPSAnV2lu ZG93cy0xMjU2JzsNCiAgICAgICBlbHNlIGlmICgn/icgPT0gJ1x1MjBBQicpDQogICAgICAgICBl bmNvZGluZyA9ICdXaW5kb3dzLTEyNTgnOw0KICAgICAgIGVsc2UNCiAgICAgICAgIGVuY29kaW5n ID0gJ3Vua25vd24gd2l0aCAweEUwID0gVSswMEUwIGFuZCAweDk5ID0gVSsyMTIyJzsNCiAgICAg fSBlbHNlIGlmICgn/ycgPT0gJ1x1MDJEOScpIC8vIDB4RkYNCiAgICAgICBlbmNvZGluZyA9ICdJ U08tODg1OS0zJzsNCiAgICAgZWxzZSBpZiAoJ/4nID09ICdcdTAxNzcnKSAvLyAweEZFDQogICAg ICAgZW5jb2RpbmcgPSAnSVNPLTg4NTktMTQnOw0KICAgICBlbHNlIGlmICgn/icgPT0gJ1x1MDIx QicpDQogICAgICAgZW5jb2RpbmcgPSAnSVNPLTg4NTktMTYnOw0KICAgICBlbHNlIGlmICgn/icg PT0gJ1x1MDE1RicpDQogICAgICAgZW5jb2RpbmcgPSAnSVNPLTg4NTktOSc7DQogICAgIGVsc2Ug aWYgKCe+JyA9PSAnXHUwMEJFJykgLy8gMHhCRQ0KICAgICAgIGVuY29kaW5nID0gJ0lTTy04ODU5 LTEnOw0KICAgICBlbHNlIGlmICgnvicgPT0gJ1x1MDE3OCcpDQogICAgICAgZW5jb2RpbmcgPSAn SVNPLTg4NTktMTUnOw0KICAgICBlbHNlDQogICAgICAgZW5jb2RpbmcgPSAndW5rbm93biB3aXRo IDB4RTAgPSBVKzAwRTAnOw0KICAgfSBlbHNlIGlmICgn4CcgPT0gJ1x1MDEwMScpIC8vIDB4RTAN CiAgICAgZW5jb2RpbmcgPSAnSVNPLTg4NTktMTAnOw0KICAgZWxzZSBpZiAoJ+AnID09ICdcdTBF NDAnKSAvLyAweEUwDQogICAgIGVuY29kaW5nID0gJ0lTTy04ODU5LTExJzsNCiAgIGVsc2UgaWYg KCfgJyA9PSAnXHVGRkZEJykgeyAvLyAweEUwDQogICAgIGlmICgn4pi6JyA9PSAnXHUyNjNBJykg Ly8gMHhFMiAweDk4IDB4QkENCiAgICAgICBlbmNvZGluZyA9ICdVVEYtOCc7DQogICAgIGVsc2UN CiAgICAgICBlbmNvZGluZyA9ICd1bmtub3duIChidXQgQVNDSUktY29tcGF0aWJsZSknOw0KICAg fQ0KICAgZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQoJ2VuY29kaW5nJykuZmlyc3RDaGlsZC5kYXRh ID0gZW5jb2Rpbmc7DQogICB2YXIgZXhwZWN0ZWQgPSBkb2N1bWVudC5nZXRFbGVtZW50QnlJZCgn ZXhwZWN0ZWQnKS5maXJzdENoaWxkLmRhdGE7DQogICBpZiAoZXhwZWN0ZWQgPT0gJyhkZW1vIC0g bm8gZXhwZWN0ZWQgcmVzdWx0KScpIHsNCiAgICAgZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQoJ3Jl c3VsdCcpLmZpcnN0Q2hpbGQuZGF0YSA9ICdkZW1vJzsNCiAgIH0gZWxzZSBpZiAoZW5jb2Rpbmcg PT0gZXhwZWN0ZWQpIHsNCiAgICAgZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQoJ3Jlc3VsdCcpLmZp cnN0Q2hpbGQuZGF0YSA9ICdQQVNTJzsNCiAgICAgZG9jdW1lbnQuZ2V0RWxlbWVudEJ5SWQoJ3Jl c3VsdCcpLmNsYXNzTmFtZSA9ICdwYXNzJzsNCiAgIH0gZWxzZSB7DQogICAgIGRvY3VtZW50Lmdl dEVsZW1lbnRCeUlkKCdyZXN1bHQnKS5maXJzdENoaWxkLmRhdGEgPSAnRkFJTCc7DQogICAgIGRv Y3VtZW50LmdldEVsZW1lbnRCeUlkKCdyZXN1bHQnKS5jbGFzc05hbWUgPSAnZmFpbCc7DQogICB9 DQogPC9zY3JpcHQ+DQo8L2Rpdj4NCg==