14526 2011-10-20 15:27:03 +0000 WF2: When adding filenames to the data set, should there be normalization of decomposed forms? 2012-07-20 04:31:50 +0000 1 1 1 Unclassified WHATWG HTML unspecified Other other RESOLVED FIXED http://www.whatwg.org/specs/web-apps/current-work/#constructing-the-form-data-set P3 normal Unsorted 1 contributor ian bzbarsky ej ian me mike mjs naruse VYV03354 contributor oldest_to_newest 58546 0 contributor 2011-10-20 15:27:03 +0000 Specification: http://www.whatwg.org/specs/web-apps/current-work/multipage/association-of-controls-and-forms.html Multipage: http://www.whatwg.org/C#constructing-the-form-data-set Complete: http://www.whatwg.org/c#constructing-the-form-data-set Comment: When adding filenames to the data set, should there be normalization of decomposed forms? Posted from: 71.184.125.56 User agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:10.0a1) Gecko/20111017 Firefox/10.0a1 58547 1 bzbarsky 2011-10-20 15:27:27 +0000 Apparently at least some sites make assumptions about precomposed vs decomposed forms; see https://bugzilla.mozilla.org/show_bug.cgi?id=695995 58564 2 ian 2011-10-20 20:16:03 +0000 Why is https://bugzilla.mozilla.org/show_bug.cgi?id=695995 a problem? The bug doesn't say why it matters what the uploaded filename is in that case. Are there servers doing comparisons or something? 58567 3 bzbarsky 2011-10-20 20:19:44 +0000 I'm still trying to get that information. Note that the bug also cites https://support.mozilla.com/fi/questions/874246 which seems to suggest that servers are doing _something_ dumb with it. 58701 4 bzbarsky 2011-10-22 05:39:36 +0000 Yes, the server involved in the cited Mozilla bug is doing comparisons without normalizing. 58871 5 ian 2011-10-25 05:07:43 +0000 Hmm. I'm not sure whether that's truly a problem. I mean, what if the uploaded filename is in uppercase vs lowercase? Or has one space or two somewhere in the filename? Surely basing anything on the file name of the uploaded file is rife with problems, canonicalisation being the least of them. 58874 6 bzbarsky 2011-10-25 05:12:41 +0000 Dunno. I'm passing on what info I have so far. If I get more, I'll pass on more! But the fact remains that there appears to be (mostly) browser interop here on an observable behavior that's broken at least some servers... A good question is how strong that interop actually is. 58877 7 ian 2011-10-25 05:28:48 +0000 Yeah, I guess I'll have to test it. I don't suppose there's a convenient test I can start from, by any chance? Otherwise I'll just build one. 58883 8 bzbarsky 2011-10-25 06:00:47 +0000 I don't have a test offhand, sorry. 59116 9 ian 2011-10-28 00:53:42 +0000 Ok I built a test: http://damowmow.com/playground/demos/filename-upload/ Results: Firefox/10.0a1 on Mac and Opera/9.80 on Mac send the filename decomposed. Everyone else I tested (IE/9, Firefox/5 on Windows, Safari/5 on Mac and Windows, Chrome/16 on Mac and Windows) send the filename composed. No difference between GET and POST. I guess I'll update the spec to say to send the filename composed. Any particular guess as to what kind of normalisation I should be applying here? NFC? I'll test to see what browsers do using the example in the third row of figure 6 of http://unicode.org/reports/tr15/ unless someone gets there before me. 59117 10 bzbarsky 2011-10-28 00:59:38 +0000 No idea on choice of normalization. Not something I know well enough to comment on intelligently... 59143 11 ian 2011-10-28 19:39:40 +0000 I created a test that would distinguish normalisation forms: http://damowmow.com/playground/demos/filename-upload/002.html When I create the given file name on Mac, I get a file whose name's bytes are displayed to the console by ls(1) piped through hexdump as: c5 bf cc a3 cc 87 e2 84 ab This isn't what I expected. In particular, it means that it is not normalising singletons, but is doing NFD for composition. As far as I can tell. Uploading this file results in the following (recent builds or latest shipping copy in all cases, only testing POST): Mac Firefox: same as file system (c5 bf cc a3 cc 87 e2 84 ab) Mac Opera: same as file system (c5 bf cc a3 cc 87 e2 84 ab) Mac Safari: NFC (e1 ba 9b cc a3 c3 85) Mac Chrome: NFC (e1 ba 9b cc a3 c3 85) On Windows I had more trouble creating the file. I copied and pasted the string from the page in IE to a command shell to create the file. According to dir, the file had three characters, which it displayed as "??Å.txt". No idea what kind of "Å" that is, unfortunately. Then I tried uploading it (sorry about the old software versions): IE9: original bytes (e1 ba 9b cc a3 e2 84 ab) Win Firefox 5: original bytes (e1 ba 9b cc a3 e2 84 ab) Win Safari 5: NFC (e1 ba 9b cc a3 c3 85) Win Chrome: NFC (e1 ba 9b cc a3 c3 85) So basically as far as I can tell, all browsers except WebKit-based browsers do no normalisation, they just trust the file system. On Mac this is slightly problematic only because Mac's file system does its own normalisation. WebKit always does NFC normalisation on the file name before submission. 59427 12 ian 2011-11-01 17:30:15 +0000 I went with requiring NFC, since that seems like the only option that will lead to any kind of interop. 59428 13 contributor 2011-11-01 17:31:17 +0000 Checked in as WHATWG revision r6810. Check-in comment: Require NFC for file names from <input type=file>. http://html5.org/tools/web-apps-tracker?from=6809&to=6810 59466 14 bzbarsky 2011-11-02 02:59:44 +0000 We may not be out of the woods here. See https://bugzilla.mozilla.org/show_bug.cgi?id=695995#c18 I asked the commenter to comment here directly as needed... 59522 15 VYV03354 2011-11-02 20:43:18 +0000 Mac OS uses a special variant of NFD to avoid normalizing CJK Compatibility Ideographs because some of Compatibility Ideographs is important (even required) in Japan. Roughly speaking, It excludes a specific ranges of code points from Normalization. I found a proposal document from Apple (but rejected by UTC). http://www.unicode.org/review/resolved-pri.html#pri7 http://www.unicode.org/review/pr-7b.html Note that this proposal is a bit different from what Mac OS is actually using. Mac OS also excludes code points from U+2000 to U+2FFF. I think we should define "willful violation of UAX #15" or "Web Normalization" or something other than NFC. 59552 16 ian 2011-11-03 15:31:19 +0000 So how exactly should it be defined? "File names must be exposed in a normalized form, whether in the DOM (e.g. in File objects) or in form submission, regardless of the conventions of the user agent's platform's file system. The normalization form used must be Unicode normalization Form C (NFC), except that input characters in the range U+2000 to U+2FFF, U+F900 to U+FA6A, and U+2F800 to U+2FA1D must be left unchanged in the output."? This isn't what any browser does as far as I can tell. Are we sure that what WebKit does is broken for CJK? 59577 17 naruse 2011-11-03 18:33:00 +0000 (In reply to comment #15) > I found a proposal document from Apple (but rejected by UTC). > http://www.unicode.org/review/resolved-pri.html#pri7 > http://www.unicode.org/review/pr-7b.html > Note that this proposal is a bit different from what Mac OS is actually using. > Mac OS also excludes code points from U+2000 to U+2FFF. > > I think we should define "willful violation of UAX #15" or "Web Normalization" > or something other than NFC. Recent document says exactly the same what you say: "U+2000 through U+2FFF, U+F900 through U+FAFF, and U+2F800 through U+2FAFF are not decomposed" http://developer.apple.com/library/mac/#qa/qa1173/_index.html 59611 18 naruse 2011-11-04 06:25:31 +0000 (In reply to comment #16) > So how exactly should it be defined? "File names must be exposed in a > normalized form, whether in the DOM (e.g. in File objects) or in form > submission, regardless of the conventions of the user agent's platform's file > system. The normalization form used must be Unicode normalization Form C (NFC), > except that input characters in the range U+2000 to U+2FFF, U+F900 to U+FA6A, > and U+2F800 to U+2FA1D must be left unchanged in the output."? I think so. But whether such behavior should be portable (should be applied other than Mac OS X) is debatable. Imagine following situation, a directory has two file, U+795E.txt and U+FA19.txt. And the user want to upload them. As you can notice, DOM and uploaded server can't distinguish them. Normalization considered harmful. It is not harmful only where the file's filesystem uses normalization, and the filesystem and the browser uses exactly the same algorithm. Idealy normalization over filenames should be done only for files on the normalized filesystems such as HFS Plus. (but an assumption filenames on Mac OS X are normalized can be acceptable) > This isn't what any browser does as far as I can tell. Are we sure that what > WebKit does is broken for CJK? Yes, current WebKit normalizes those Kanjis, and it is considered breakage. You can see the breakage by uploading U+FA19.txt. After uploading, it become U+795E.txt and you can find the left part of the Kanji is changed. These kanjis have the same meaning "god", and specified as compatibility character thorough some political reason, but people don't want to normalize them other than the true normalization situation. 61100 19 ian 2011-12-07 18:32:04 +0000 That argues for not doing any kind of normalisation. bz: What do you think? Looks like NFC is out, and modified NFC would cause problems on Windows. Suggestions? I'm leaning back towards "trust the filesystem". 61114 20 bzbarsky 2011-12-07 20:18:51 +0000 I guess I can live with that if UAs actually converge on it.... 61200 21 ian 2011-12-08 00:02:52 +0000 I guess we should file a bug on WebKit and see if they're willing to change? 61344 22 ian 2011-12-09 22:23:17 +0000 I'm going to remove the normalisation stuff and, if nobody else gets there before me, file a bug on WebKit to remove the normalisation. 70773 23 contributor 2012-07-18 17:27:56 +0000 This bug was cloned to create bug 18153 as part of operation convergence. 71240 24 ian 2012-07-20 04:31:17 +0000 Filed https://bugs.webkit.org/show_bug.cgi?id=91817 and reverted spec. 71241 25 contributor 2012-07-20 04:31:50 +0000 Checked in as WHATWG revision r7195. Check-in comment: Revert r6810 since it doesn't work. http://html5.org/tools/web-apps-tracker?from=7194&to=7195