This is an archived snapshot of W3C's public bugzilla bug tracker, decommissioned in April 2019. Please see the home page for more details.

Bug 9663 - Should sequences of bytes be replaced by a single U+FFFD, or one U+FFFD per input byte?
Summary: Should sequences of bytes be replaced by a single U+FFFD, or one U+FFFD per i...
Status: RESOLVED FIXED
Alias: None
Product: HTML WG
Classification: Unclassified
Component: pre-LC1 HTML5 spec (editor: Ian Hickson) (show other bugs)
Version: unspecified
Hardware: Other other
: P3 normal
Target Milestone: LC
Assignee: Ian 'Hixie' Hickson
QA Contact: HTML WG Bugzilla archive list
URL: http://www.whatwg.org/specs/web-apps/...
Whiteboard:
Keywords: NE
Depends on:
Blocks:
 
Reported: 2010-05-06 06:13 UTC by contributor
Modified: 2010-10-04 14:29 UTC (History)
6 users (show)

See Also:


Attachments

Description contributor 2010-05-06 06:13:36 UTC
Section: http://www.whatwg.org/specs/web-apps/current-work/#parsing-0

Comment:
Should sequences of bytes be replaced by a single U+FFFD, or one U+FFFD per
input byte?

Posted from: 213.236.208.46
Comment 1 Adam Barth 2010-08-04 23:26:43 UTC
The WebKit HTML5 parser replaces each byte individually.
Comment 2 Anne 2010-08-08 07:09:51 UTC
This depends on the octet-to-character conversion layer. And more specifically, the error handling can presumably differ depending on the encoding. It should be fixed at the encoding layer I think.
Comment 3 Adam Barth 2010-08-08 07:12:19 UTC
I misunderstood which part of the spec you were referring to.  You seem to be liking to WebSRT, but I thought you were linking to the "preprocessing the input stream" for HTML parsing.  :)
Comment 4 Ian 'Hixie' Hickson 2010-09-10 22:44:39 UTC
EDITOR'S RESPONSE: This is an Editor's Response to your comment. If you are satisfied with this response, please change the state of this bug to CLOSED. If you have additional information and would like the editor to reconsider, please reopen this bug. If you would like to escalate the issue to the full HTML Working Group, please add the TrackerRequest keyword to this bug, and suggest title and text for the tracker issue; or you may create a tracker issue yourself, if you are able to do so. For more details, see this document:
   http://dev.w3.org/html5/decision-policy/decision-policy.html

Status: Did Not Understand Request
Change Description: no spec change
Rationale: This phrase appears all over the place. What should it be replaced with?
Comment 5 Simon Pieters 2010-09-13 18:42:23 UTC
First try, probably isn't quite right:

Numbers are bytes in hex. "Anything but ..." includes EOF.

Stray 80-BF:
FE-FF:
replace with one U+FFFD.

C0-C1 followed by 80-BF:
replace the 2-byte sequence with one U+FFFD.

C0-FD followed by anything but 80-BF:
replace the first byte with one U+FFFD and reprocess the second byte.

E0-FD followed by 80-BF followed by anything but 80-BF:
replace the first two bytes with one U+FFFD and reprocess the third byte.

F0-FD followed by two 80-BF followed by anything but 80-BF:
replace the first three bytes with one U+FFFD and reprocess the forth byte.

F0-F4 followed by three 80-BF that represent a code point above U+10FFFF:
replace all four bytes with one U+FFFD.

F5-FD followed by three 80-BF followed by anything but 80-BF:
replace the first four bytes with one U+FFFD and reprocess the fifth byte.

FC-FD followed by four 80-BF followed by anything but 80-BF:
replace the first five bytes with one U+FFFD and reprocess the sixth byte.

Overlong forms (e.g. F0 80 80 A0):
replace the whole byte sequence with one U+FFFD.
Comment 6 Ian 'Hixie' Hickson 2010-09-28 19:16:05 UTC
EDITOR'S RESPONSE: This is an Editor's Response to your comment. If you are satisfied with this response, please change the state of this bug to CLOSED. If you have additional information and would like the editor to reconsider, please reopen this bug. If you would like to escalate the issue to the full HTML Working Group, please add the TrackerRequest keyword to this bug, and suggest title and text for the tracker issue; or you may create a tracker issue yourself, if you are able to do so. For more details, see this document:
   http://dev.w3.org/html5/decision-policy/decision-policy.html

Status: Partially Accepted
Change Description: see diff given below
Rationale: Ok, I tried. Let me know what I screwed up! :-)
Comment 7 contributor 2010-09-28 19:16:37 UTC
Checked in as WHATWG revision r5530.
Check-in comment: Tighten up UTF-8 error handling definitions
http://html5.org/tools/web-apps-tracker?from=5529&to=5530