13676 2011-08-05 01:44:05 +0000 Clarify what the code-point length of a string with isolated surrogate is. 2011-10-06 23:50:17 +0000 1 1 1 Unclassified HTML WG HTML5 spec unspecified Other other RESOLVED FIXED http://www.whatwg.org/specs/web-apps/current-work/#common-parser-idioms P3 normal --- 1 contributor ian annevk ayg cam hsivonen ian kennyluck mike public-html-admin public-html-wg-issue-tracking public-html-bugzilla oldest_to_newest 54208 0 contributor 2011-08-05 01:44:05 +0000 Specification: http://www.whatwg.org/specs/web-apps/current-work/multipage/common-microsyntaxes.html Multipage: http://www.whatwg.org/C#common-parser-idioms Complete: http://www.whatwg.org/c#common-parser-idioms Comment: Clarify what the code-point length of a string with isolated surrogate is. Posted from: 114.43.115.245 by kennyluck@csail.mit.edu User agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; ja-jp) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4 54209 1 kennyluck 2011-08-05 02:09:01 +0000 Assuming the intension of the current text is to count "\ud840\udc87" // as code-point length = 1 "\ud840+\udc87" // as code-point length = 3 (which isn't very clear as far as I can tell), I would suggest the spec to include a sentence like "Unpaired surrogates count as one code point each." (wording from [1]) Alternatively, it might be clearer to replace the sentence # The code-point length of a string is the number of Unicode code points in that string. by | The code-point length of a string is the number of Unicode characters after the string is converted to a sequence of Unicode characters[2]. This will then work for both a string of Unicode characters(theory) and a DOMString(reality), before the internal representation of the value of an input element[3] is made clear. Having said that, I am not convinced that defining @maxlength this way is the best, I tried to analyze other possibilities[4] but wasn't confident enough to file a bug (my preference is to count 16 bits) [1] http://download.oracle.com/javase/1,5.0/docs/api/java/lang/String.html (definition of String.codePointCount) [2] http://dev.w3.org/2006/webapi/WebIDL/#dfn-obtain-unicode [3] http://www.whatwg.org/specs/web-apps/current-work/multipage/association-of-controls-and-forms.html#concept-fe-value [4] http://lists.w3.org/Archives/Public/www-international/2011AprJun/0105 54298 2 ian 2011-08-06 06:12:57 +0000 Yeah this is fallout from the decision to stop having the spec be in terms of Unicode but instead have the spec in terms of UTF-16 code points. Sigh. I'll need to fix this definition somehow. 55218 3 annevk 2011-08-16 09:17:58 +0000 Web IDL will probably introduce the term "code units" for 16-bit code units. I think all our specifications should use that and we should just accept they are not exactly ideal I think. 55238 4 ayg 2011-08-16 15:06:31 +0000 I use the term "element", following ES: http://es5.github.com/#x8.4 But that's admittedly quite confusing, even though I xref it, so someplace defining "code unit" would be nice. 55462 5 cam 2011-08-19 03:33:53 +0000 I just added http://dev.w3.org/2006/webapi/WebIDL/#dfn-code-unit, if that is helpful. 55507 6 ayg 2011-08-19 20:00:39 +0000 Thanks, I switched over to using that: http://aryeh.name/gitweb.cgi?p=editing;a=commitdiff;h=73d8d864 57851 7 contributor 2011-10-05 17:31:47 +0000 Checked in as WHATWG revision r6633. Check-in comment: Redefine code-point length in terms of UTF-16 16bit code units. http://html5.org/tools/web-apps-tracker?from=6632&to=6633 57852 8 ian 2011-10-05 17:35:21 +0000 This affects all manner of algorithms. I'm not even sure what I want for some of them. I'm pretty sure I don't want everything to work with code units rather than characters... I still have to add an xref for code unit; I also have to figure out what to do for a lot of these cases. I guess I need to define that isolated surrogates in a Unicode string are considered as being characters too, or something (with their code point being the value of the surrogate). 57921 9 contributor 2011-10-06 23:27:58 +0000 Checked in as WHATWG revision r6648. Check-in comment: Try to tidy up some more of the Unicode/code unit mess with a probably over-reaching definition (there's over 2000 uses of the word 'character' in the text, so I didn't check that all of them use this new definition... hopefully it works out; otherwise, we'll just have to try something else again). http://html5.org/tools/web-apps-tracker?from=6647&to=6648 57922 10 contributor 2011-10-06 23:33:32 +0000 Checked in as WHATWG revision r6649. Check-in comment: Define 'code unit'. http://html5.org/tools/web-apps-tracker?from=6648&to=6649 57923 11 contributor 2011-10-06 23:38:08 +0000 Checked in as WHATWG revision r6650. Check-in comment: Define 'Unicode code point'. http://html5.org/tools/web-apps-tracker?from=6649&to=6650 57924 12 contributor 2011-10-06 23:44:41 +0000 Checked in as WHATWG revision r6651. Check-in comment: Reorder the definitions and fix them so that they aren't cyclic. http://html5.org/tools/web-apps-tracker?from=6650&to=6651 57926 13 ian 2011-10-06 23:50:17 +0000 EDITOR'S RESPONSE: This is an Editor's Response to your comment. If you are satisfied with this response, please change the state of this bug to CLOSED. If you have additional information and would like the editor to reconsider, please reopen this bug. If you would like to escalate the issue to the full HTML Working Group, please add the TrackerRequest keyword to this bug, and suggest title and text for the tracker issue; or you may create a tracker issue yourself, if you are able to do so. For more details, see this document: http://dev.w3.org/html5/decision-policy/decision-policy.html Status: Accepted Change Description: see diff given below Rationale: Ok I think I'm done here.