9207 2010-03-08 01:25:39 +0000 Anything else: This part of the spec is problematic, for example, a query string variable &lang_id=1 in as part of an attribute of say an img tag, will get converted into an character token when it shouldn't be. Why is the set of characters a-z, A-Z, 0- 2010-10-04 13:57:28 +0000 1 1 1 Unclassified HTML WG pre-LC1 HTML5 spec (editor: Ian Hickson) unspecified Other other RESOLVED FIXED http://www.whatwg.org/specs/web-apps/current-work/#tokenizing-character-references P3 normal LC 9352 1 contributor ian ayg ian julian.reschke mike mirthy mjs Ms2ger public-html-admin public-html-wg-issue-tracking public-html-bugzilla oldest_to_newest 32840 0 contributor 2010-03-08 01:25:39 +0000 Section: http://www.whatwg.org/specs/web-apps/current-work/#tokenizing-character-references Comment: Anything else: This part of the spec is problematic, for example, a query string variable &lang_id=1 in as part of an attribute of say an img tag, will get converted into an character token when it shouldn't be. Why is the set of characters a-z, A-Z, 0-9? This poses a unique problem for any entities that aren't closed properly. Posted from: 146.115.114.89 32841 1 mirthy 2010-03-08 01:28:58 +0000 Discovered as part of a webkit bug that uses an HTML5 spec tokenizer: https://bugs.webkit.org/show_bug.cgi?id=35831 Examples where the tokenizer will mangle the URL: <img src="http://www.webkit.org/getImage.aspx?id=12345&lang_id=1"/> &amp_energy=100 &lt-now=10 (In reply to comment #0) > Section: > http://www.whatwg.org/specs/web-apps/current-work/#tokenizing-character-references > > Comment: > Anything else: This part of the spec is problematic, for example, a query > string variable &lang_id=1 in as part of an attribute of say an img tag, will > get converted into an character token when it shouldn't be. Why is the set of > characters a-z, A-Z, 0-9? This poses a unique problem for any entities that > aren't closed properly. > > Posted from: 146.115.114.89 > 32842 2 zcorpan 2010-03-08 08:35:31 +0000 It's for IE compat. Personally I don't see much problem with extending the list to include underscore, equals sign, and other characters that would improve or at least not hurt Web compat, based on research. (I research the equals sign and concluded that it would be reasonably safe to change, but Hixie rejected the proposal. http://lists.w3.org/Archives/Public/public-html/2009Jul/0421.html ) 33615 3 zcorpan 2010-03-16 13:00:18 +0000 Some raw data at: http://philip.html5.org/data/entities-in-attribute-with-no-semicolon-or-alphanumeric.txt # [13:54] <Philip`> zcorpan: I recently remembered that I forgot to do http://philip.html5.org/data/entities-in-attribute-with-no-semicolon-or-alphanumeric.txt # [13:55] <Philip`> I changed your regexp a bit to exclude > from attribute 'values', because otherwise it seemed to pick up a lot of entities that were in content after an element with attributes # [13:56] <Philip`> zcorpan: Also I think the file is probably truncated, because it seemed to get stuck at some point (I guess a page hit a worst-complexity case in the pattern) so I killed it after it had stopped for a few minutes -- http://krijnhoetmer.nl/irc-logs/whatwg/20100316#l-254 33909 4 zcorpan 2010-03-25 15:13:58 +0000 http://simon.html5.org/dump/entities-in-attribute-with-no-semicolon-or-alphanumeric.xml counts the occurrences for each character appearing after the entity: / 7 , 29 " 546 - 16 185 & 108 % 499 > 24 ' 37 _ 10 : 62 = 36 Â 1 å 4 . 30 Ð 4 Å 2 ç 2 # 1 < 1 The " and ' items are probably mostly the attribute value's end quote. 34075 5 zcorpan 2010-03-29 08:48:13 +0000 I've analyzed the individual cases: / 7 should be replaced , 29 should be replaced (sometimes , is a typoed ; but not always) " 546 should be replaced (end of attribute value) - 16 should most often be replaced. exception: <a href="http://www.promedia-med.com/index.php?shp=3&cl=details&cnid=0bd466d6983607786.88175745&anid=0bd466d69855b9c84.32939037&micro-Pipettierhelfer&" class="extrabold link1">micro-Pipettierhelfer</a> 185 should be replaced & 108 should be replaced % 499 should be replaced and following %3b should be consumed also? <a href="/View/Dealer/Boat-City/TA3803.aspx?Ne=23&N=29&amp%3bsid=1184817ADB85&amp%3bN=0" target="_blank"> > 24 should be replaced (end of unquoted attribute value) ' 37 should be replaced (end of single quoted attribute value or nested js string) _ 10 should not, or should. <a href="http://del.icio.us/post?tittle=&url=http://mujer.terra.es/muj/articulo/articulo.cfm?id=mu214379&not_estatica=1&gen=1"> vs <a href="http://www.libridvd.it/prezzo_libro-autodaf&egrave_leuropa_gli_ebrei_e_lantisemitismo-9788871806068.html"><img src="http://www.libridvd.it/immagini/scheda.gif" alt="Scheda libro Autodaf&egrave;. L&#39;Europa, gli ebrei e l&#39;antisemitismo" border="0"></a> : 62 should be replaced and the : should be consumed also (it is typoed ; ) = 36 should mostly not be replaced, but some cases unclear e.g. &amp= Â 1 n/a (encoding error) å 4 n/a (encoding error) . 30 should be replaced Ð 4 n/a (encoding error) Å 2 n/a (encoding error?) ç 2 n/a (encoding error) # 1 n/a (double escaped NCR + encoding error) < 1 n/a? 34076 6 zcorpan 2010-03-29 09:06:24 +0000 If anyone wants to look at the individual cases, I modified the script in http://simon.html5.org/dump/entities-in-attribute-with-no-semicolon-or-alphanumeric.xml as follows to show only '/': <script><![CDATA[ onload=function(){ var pre = document.getElementsByTagName('pre')[1]; var lines = pre.textContent.split('\n'); var data = []; var chars = {}; var tmp; var tmp2; for (var i = 0; i < lines.length; ++i) { tmp = /^([^\t]+)\t(.+)$/.exec(lines[i]); tmp2 = /&(AElig|AMP|Aacute|Acirc|Agrave|Aring|Atilde|Auml|COPY|Ccedil|ETH|Eacute|Ecirc|Egrave|Euml|GT|Iacute|Icirc|Igrave|Iuml|LT|Ntilde|Oacute|Ocirc|Ograve|Oslash|Otilde|Ouml|QUOT|REG|THORN|Uacute|Ucirc|Ugrave|Uuml|Yacute|aacute|acirc|acute|aelig|agrave|amp|aring|atilde|auml|brvbar|ccedil|cedil|cent|copy|curren|deg|divide|eacute|ecirc|egrave|eth|euml|frac12|frac14|frac34|gt|iacute|iuml|laquo|lt|macr|micro|middot|nbsp|not|ntilde|oacute|ocirc|ograve|ordf|ordm|oslash|otilde|ouml|para|plusmn|pound|quot|raquo|reg|sect|shy|sup1|sup2|sup3|szlig|thorn|times|uacute|ucirc|ugrave|uml|uuml|yacute|yen|yuml)([^;a-zA-Z0-9])/.exec(tmp[2]); for (var j = 2; j < tmp2.length; ++j) { // CHANGE HERE: if (tmp2[j] == '/') data.push([tmp[1], tmp[2]]); } } for (i = 0; i < data.length; ++i) { document.getElementsByTagName('pre')[0].textContent += data[i][1] + '\n\n'; } } ]]></script> 34440 7 ian 2010-04-02 21:44:14 +0000 see also bug 9351 34443 8 ian 2010-04-02 22:39:23 +0000 EDITOR'S RESPONSE: This is an Editor's Response to your comment. If you are satisfied with this response, please change the state of this bug to CLOSED. If you have additional information and would like the editor to reconsider, please reopen this bug. If you would like to escalate the issue to the full HTML Working Group, please add the TrackerRequest keyword to this bug, and suggest title and text for the tracker issue; or you may create a tracker issue yourself, if you are able to do so. For more details, see this document: http://dev.w3.org/html5/decision-policy/decision-policy.html Status: Partially Accepted Change Description: see diff given below Rationale: Based on the data, I've only changed this for '='. 34445 9 ian 2010-04-02 22:39:54 +0000 *** Bug 9351 has been marked as a duplicate of this bug. *** 34447 10 ian 2010-04-02 22:41:03 +0000 http://html5.org/tools/web-apps-tracker?from=4958&to=4959