Abstract

The utf-8 encoding is the most appropriate encoding for interchange of Unicode, the universal coded character set. Therefore for new protocols and formats, as well as existing formats deployed in new contexts, this specification requires (and defines) the utf-8 encoding.

The other (legacy) encodings have been defined to some extent in the past. However, user agents have not always implemented them in the same way, have not always used the same labels, and often differ in dealing with undefined and former proprietary areas of encodings. This specification addresses those gaps so that new user agents do not have to reverse engineer encoding implementations and existing user agents can converge.

Status of This Document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

This is a snapshot of the WHATWG document, as of 29 September 2015, published after discussion with the WHATWG editors. No changes have been made in the body of this document other than to align with W3C house styles. The primary reason that W3C is publishing this document is so that HTML5 and other specifications may normatively refer to a stable W3C Recommendation.

This second publication of the Candidate Recommendation takes into account changes made to the editor's draft since its initial publication as CR. These changes are largely due to issues discovered during implementation. This document will remain a Candidate Recommendation at least until 24 November 2015 in order to ensure the opportunity for wide review. See the disposition of comments for the CR period.

Note

Sending comments on this document

If you wish to make comments regarding this document, please raise them as github issues against the latest editor's draft. Only send comments by email if you are unable to raise issues on github (see links below). All comments are welcome.

To make it easier to track comments, please raise separate issues or emails for each comment, and point to the section you are commenting on  using a URL for the dated version of the document.

This document was published by the Internationalization Working Group as a Candidate Recommendation. This document is intended to become a W3C Recommendation. If you wish to make comments regarding this document, please send them to www-international@w3.org (subscribe, archives). W3C publishes a Candidate Recommendation to indicate that the document is believed to be stable and to encourage implementation by the developer community. This Candidate Recommendation is expected to advance to Proposed Recommendation no earlier than 23 December 2015. All comments are welcome.

Publication as a Candidate Recommendation does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

This document was produced by a group operating under the 5 February 2004 W3C Patent Policy. W3C maintains a public list of any patent disclosures made in connection with the deliverables of the group; that page also includes instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy.

This document is governed by the 1 September 2015 W3C Process Document.

Table of Contents

1. Preface

The utf-8 encoding is the most appropriate encoding for interchange of Unicode, the universal coded character set. Therefore for new protocols and formats, as well as existing formats deployed in new contexts, this specification requires (and defines) the utf-8 encoding.

The other (legacy) encodings have been defined to some extent in the past. However, user agents have not always implemented them in the same way, have not always used the same labels, and often differ in dealing with undefined and former proprietary areas of encodings. This specification addresses those gaps so that new user agents do not have to reverse engineer encoding implementations and existing user agents can converge.

In particular, this specification defines all those encodings, their algorithms to go from bytes to scalar values and back, and their canonical names and identifying labels. This specification also defines an API to expose part of the encoding algorithms to JavaScript.

User agents have also significantly deviated from the labels listed in the IANA Character Sets registry. To stop spreading legacy encodings further, this specification is exhaustive about the aforementioned details and therefore has no need for the registry. In particular, this specification does not provide a mechanism for extending any aspect of encodings.

2. Security background

There is a set of encoding security issues when the producer and consumer do not agree on the encoding in use, or on the way a given encoding is to be implemented. For instance, an attack was reported in 2011 where a shift_jis lead byte 0x82 was used to “mask” a 0x22 trail byte in a JSON resource of which an attacker could control some field. The producer did not see the problem even though this is an illegal byte combination. The consumer decoded it as a single U+FFFD and therefore changed the overall interpretation as U+0022 is an important delimiter. Decoders of encodings that use multiple bytes for scalar values now require that in case of an illegal byte combination, a scalar value in the range U+0000 to U+007F, inclusive, cannot be “masked”. For the aforementioned sequence the output would be U+FFFD U+0022.

This is a larger issue for encodings that map anything in the range 0x00 to 0x7F, inclusive, to something other than U+0000 to U+007F, inclusive, when there is no lead byte present. These are “ASCII-incompatible” encodings and other than iso-2022-jp, utf-16be, and utf-16le, which are unfortunately required due to deployed content, they are not supported. (Investigation is ongoing whether more labels of other such encodings can be mapped to the replacement encoding, rather than the unknown encoding fallback.) An example attack is injecting carefully crafted content into a resource and then encouraging the user to override the encoding, resulting in e.g. script execution.

Encoders used by URLs found in HTML and HTML's form feature can also result in slight information loss when an encoding is used that cannot represent all scalar values. E.g. when a resource uses the windows-1252 encoding a server will not be able to distinguish between an end user entering “💩” and “💩” into a form.

The problems outlined here go away when exclusively using utf-8, which is one of the many reasons that is now the mandatory encoding for all things.

Note

See also the Browser UI chapter.

3. Conformance

All diagrams, examples, and notes in this specification are non-normative, as are all sections explicitly marked non-normative. Everything else in this specification is normative.

The key words "MUST", "MUST NOT", "REQUIRED", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in the normative parts of this document are to be interpreted as described in RFC2119. For readability, these words do not appear in all uppercase letters in this specification. [RFC2119]

Conformance requirements phrased as algorithms or specific steps may be implemented in any manner, so long as the end result is equivalent. (In particular, the algorithms defined in this specification are intended to be easy to follow, and not intended to be performant.)

User agents may impose implementation-specific limits on otherwise unconstrained inputs, e.g. to prevent denial of service attacks, to guard against running out of memory, or to work around platform-specific limitations.

4. Terminology

Hexadecimal numbers are prefixed with "0x".

In equations, all numbers are integers, addition is represented by "+", subtraction by "−", multiplication by "×", division by "/", calculating the remainder of a division (also known as modulo) by "%", logical left shifts by "<<", logical right shifts by ">>", bitwise AND by "&", and bitwise OR by "|".

For logical right shifts operands must have at least twenty-one bits precision.

A byte is a sequence of eight bits, represented as a double-digit hexadecimal number in the range 0x00 to 0xFF, inclusive.

An ASCII byte is a byte in the range 0x00 to 0x7F, inclusive.

A code point is a Unicode code point and is represented as a four-to-six digit hexadecimal number, typically prefixed with "U+". In equations and indexes code points are prefixed with "0x". [UNICODE]

A scalar value is a code point that is not in the range U+D800 to U+DFFF, inclusive.

An ASCII code point is a code point in the range U+0000 to U+007F, inclusive.

The ASCII whitespace are code points U+0009, U+000A, U+000C, U+000D, and U+0020.

The ASCII digits are code points in the range U+0030 to U+0039, inclusive.

A string is a sequence of code points.

Comparing two strings in an ASCII case-insensitive manner means comparing them exactly, code point for code point, except that the characters in the range U+0041 to U+005A (i.e. LATIN CAPITAL LETTER A to LATIN CAPITAL LETTER Z), inclusive, and the corresponding characters in the range U+0061 to U+007A (i.e. LATIN SMALL LETTER A to LATIN SMALL LETTER Z), inclusive, are considered to also match.

A token is a piece of data, such as a byte or code point.

A stream represents an ordered sequence of tokens. End-of-stream is a special token that signifies no more tokens are in the stream.

When a token is read from a stream, the first token in the stream must be returned and subsequently removed, and end-of-stream must be returned otherwise.

When one or more tokens are prepended to a stream, those tokens must be inserted, in given order, before the first token in the stream.

Inserting the sequence of tokens &#128169; in a stream " hello world", results in a stream "&#128169; hello world". The next token to be read would be &.

When one or more tokens are pushed to a stream, those tokens must be inserted, in given order, after the last token in the stream.

5. Encodings

An encoding defines a mapping from a scalar value sequence to a byte sequence (and vice versa). Each encoding has a name, and one or more labels.

5.1 Encoders and decoders

Each encoding has an associated decoder and encoder. Each decoder and encoder have a handler algorithm. A handler algorithm takes an input stream and a token, and returns finished, one or more tokens, error optionally with a code point, or continue.

An error mode as used below is either replacement (default) or fatal for a decoder and one of fatal (default) or HTML for an encoder.

Note

An XML processor would set error mode to fatal. [XML]

Note

HTML exists as error mode due to URLs and HTML forms requiring a non-terminating legacy encoder. The HTML error mode causes a sequence to be emitted that cannot be distinguished from legitimate input and can therefore lead to silent data loss. Developers are strongly encouraged to use the utf-8 encoding to prevent this from happening. [URL] [HTML]

To run an encoding's decoder or encoder encoderDecoder with input stream input, output stream output, and optional error mode mode, run these steps:

  1. If mode is not given, set it to replacement, if encoderDecoder is a decoder, and fatal otherwise.

  2. Let encoderDecoderInstance be a new encoderDecoder.

  3. While true:

    1. Let result be the result of processing the result of reading from input for encoderDecoderInstance, input, output, and mode.

    2. If result is not continue, return result.

    3. Otherwise, do nothing.

To process a token token for an encoding's encoder or decoder instance encoderDecoderInstance, stream input, output stream output, and optional error mode mode, run these steps:

  1. If mode is not given, set it to replacement, if encoderDecoderInstance is a decoder instance, and fatal otherwise.

  2. Let result be the result of running encoderDecoderInstance's handler on input and token.

  3. If result is continue or finished, return result.

  4. Otherwise, if result is one or more tokens, push result to output.

  5. Otherwise, if result is error, switch on mode and run the associated steps:

    replacement
    Push U+FFFD to output.
    HTML
    Prepend U+0026, U+0023, followed by the shortest sequence of ASCII digits representing result's code point in base ten, followed by U+003B to input.
    fatal
    Return error.
  6. Return continue.

5.2 Names and labels

The table below lists all encodings and their labels user agents must support. User agents must not support any other encodings or labels.

Authors must use the utf-8 encoding and must use the ASCII case-insensitive "utf-8" label to identify it.

New protocols and formats, as well as existing formats deployed in new contexts, must use the utf-8 encoding exclusively. If these protocols and formats need to expose the encoding's name or label, they must expose it as "utf-8".

To get an encoding from a string label, run these steps:

  1. Remove any leading and trailing ASCII whitespace from label.

  2. If label is an ASCII case-insensitive match for any of the labels listed in the table below, return the corresponding encoding, and failure otherwise.

Note

This is a much simpler and more restrictive algorithm of mapping labels to encodings than section 1.4 of Unicode Technical Standard #22 prescribes, as that is found to be necessary to be compatible with deployed content.

Name Labels
The Encoding
utf-8 "unicode-1-1-utf-8"
"utf-8"
"utf8"
Legacy single-byte encodings
ibm866 "866"
"cp866"
"csibm866"
"ibm866"
iso-8859-2 "csisolatin2"
"iso-8859-2"
"iso-ir-101"
"iso8859-2"
"iso88592"
"iso_8859-2"
"iso_8859-2:1987"
"l2"
"latin2"
iso-8859-3 "csisolatin3"
"iso-8859-3"
"iso-ir-109"
"iso8859-3"
"iso88593"
"iso_8859-3"
"iso_8859-3:1988"
"l3"
"latin3"
iso-8859-4 "csisolatin4"
"iso-8859-4"
"iso-ir-110"
"iso8859-4"
"iso88594"
"iso_8859-4"
"iso_8859-4:1988"
"l4"
"latin4"
iso-8859-5 "csisolatincyrillic"
"cyrillic"
"iso-8859-5"
"iso-ir-144"
"iso8859-5"
"iso88595"
"iso_8859-5"
"iso_8859-5:1988"
iso-8859-6 "arabic"
"asmo-708"
"csiso88596e"
"csiso88596i"
"csisolatinarabic"
"ecma-114"
"iso-8859-6"
"iso-8859-6-e"
"iso-8859-6-i"
"iso-ir-127"
"iso8859-6"
"iso88596"
"iso_8859-6"
"iso_8859-6:1987"
iso-8859-7 "csisolatingreek"
"ecma-118"
"elot_928"
"greek"
"greek8"
"iso-8859-7"
"iso-ir-126"
"iso8859-7"
"iso88597"
"iso_8859-7"
"iso_8859-7:1987"
"sun_eu_greek"
iso-8859-8 "csiso88598e"
"csisolatinhebrew"
"hebrew"
"iso-8859-8"
"iso-8859-8-e"
"iso-ir-138"
"iso8859-8"
"iso88598"
"iso_8859-8"
"iso_8859-8:1988"
"visual"
iso-8859-8-i "csiso88598i"
"iso-8859-8-i"
"logical"
iso-8859-10 "csisolatin6"
"iso-8859-10"
"iso-ir-157"
"iso8859-10"
"iso885910"
"l6"
"latin6"
iso-8859-13 "iso-8859-13"
"iso8859-13"
"iso885913"
iso-8859-14 "iso-8859-14"
"iso8859-14"
"iso885914"
iso-8859-15 "csisolatin9"
"iso-8859-15"
"iso8859-15"
"iso885915"
"iso_8859-15"
"l9"
iso-8859-16 "iso-8859-16"
koi8-r "cskoi8r"
"koi"
"koi8"
"koi8-r"
"koi8_r"
koi8-u "koi8-ru"
"koi8-u"
macintosh "csmacintosh"
"mac"
"macintosh"
"x-mac-roman"
windows-874 "dos-874"
"iso-8859-11"
"iso8859-11"
"iso885911"
"tis-620"
"windows-874"
windows-1250 "cp1250"
"windows-1250"
"x-cp1250"
windows-1251 "cp1251"
"windows-1251"
"x-cp1251"
windows-1252 "ansi_x3.4-1968"
"ascii"
"cp1252"
"cp819"
"csisolatin1"
"ibm819"
"iso-8859-1"
"iso-ir-100"
"iso8859-1"
"iso88591"
"iso_8859-1"
"iso_8859-1:1987"
"l1"
"latin1"
"us-ascii"
"windows-1252"
"x-cp1252"
windows-1253 "cp1253"
"windows-1253"
"x-cp1253"
windows-1254 "cp1254"
"csisolatin5"
"iso-8859-9"
"iso-ir-148"
"iso8859-9"
"iso88599"
"iso_8859-9"
"iso_8859-9:1989"
"l5"
"latin5"
"windows-1254"
"x-cp1254"
windows-1255 "cp1255"
"windows-1255"
"x-cp1255"
windows-1256 "cp1256"
"windows-1256"
"x-cp1256"
windows-1257 "cp1257"
"windows-1257"
"x-cp1257"
windows-1258 "cp1258"
"windows-1258"
"x-cp1258"
x-mac-cyrillic "x-mac-cyrillic"
"x-mac-ukrainian"
Legacy multi-byte Chinese (simplified) encodings
gbk "chinese"
"csgb2312"
"csiso58gb231280"
"gb2312"
"gb_2312"
"gb_2312-80"
"gbk"
"iso-ir-58"
"x-gbk"
gb18030 "gb18030"
Legacy multi-byte Chinese (traditional) encodings
big5 "big5"
"big5-hkscs"
"cn-big5"
"csbig5"
"x-x-big5"
Legacy multi-byte Japanese encodings
euc-jp "cseucpkdfmtjapanese"
"euc-jp"
"x-euc-jp"
iso-2022-jp "csiso2022jp"
"iso-2022-jp"
shift_jis "csshiftjis"
"ms932"
"ms_kanji"
"shift-jis"
"shift_jis"
"sjis"
"windows-31j"
"x-sjis"
Legacy multi-byte Korean encodings
euc-kr "cseuckr"
"csksc56011987"
"euc-kr"
"iso-ir-149"
"korean"
"ks_c_5601-1987"
"ks_c_5601-1989"
"ksc5601"
"ksc_5601"
"windows-949"
Legacy miscellaneous encodings
replacement "csiso2022kr"
"hz-gb-2312"
"iso-2022-cn"
"iso-2022-cn-ext"
"iso-2022-kr"
utf-16be "utf-16be"
utf-16le "utf-16"
"utf-16le"
x-user-defined "x-user-defined"
Note

All encodings and their labels are also available as non-normative encodings.json resource.

6. Indexes

Most legacy encodings make use of an index. An index is an ordered list of pointers and corresponding code points. Within an index pointers are unique and code points can be duplicated.

Note

An efficient implementation likely has two indexes per encoding. One optimized for its decoder and one for its encoder.

To find the pointers and their corresponding code points in an index, let lines be the result of splitting the resource's contents on U+000A. Then remove each item in lines that is the empty string or starts with U+0023. Then the pointers and their corresponding code points are found by splitting each item in lines on U+0009. The first subitem is the pointer (as a decimal number) and the second is the corresponding code point (as a hexadecimal number). Other subitems are not relevant.

Note

To signify changes an index includes an Identifier and a Date. If an Identifier has changed, so has the index.

The index code point for pointer in index is the code point corresponding to pointer in index, or null if pointer is not in index.

The index pointer for code point in index is the first pointer corresponding to code point in index, or null if code point is not in index.

These are the indexes defined by this specification, excluding index single-byte, which have their own table:

Index Notes
index big5 index-big5.txt This matches the Big5 standard in combination with the Hong Kong Supplementary Character Set and other common extensions.
index euc-kr index-euc-kr.txt This matches the KS X 1001 standard and the Unified Hangul Code, more commonly known together as Windows Codepage 949.
index gb18030 index-gb18030.txt This matches the GB18030-2000 standard for code points encoded as two bytes, except for 0xA3 0xA0 which maps to U+3000 to be compatible with deployed content.
index gb18030 ranges index-gb18030-ranges.txt This index works different from all others. Listing all code points would result in over a million items whereas they can be represented neatly in 207 ranges combined with trivial limit checks. It therefore only superficially matches the GB18030 standard for code points encoded as four bytes. See also index gb18030 ranges code point and index gb18030 ranges pointer below.
index jis0208 index-jis0208.txt This is the JIS X 0208 standard including formerly proprietary extensions from IBM and NEC.
index jis0212 index-jis0212.txt This is the JIS X 0212 standard. It is only used by the euc-jp decoder due to lack of widespread support elsewhere.

The index gb18030 ranges code point for pointer is the return value of these steps:

  1. If pointer is greater than 39419 and less than 189000, or pointer is greater than 1237575, return null.

  2. Let offset be the last pointer in index gb18030 ranges that is equal to or less than pointer and let code point offset be its corresponding code point.

  3. Return a code point whose value is code point offset + pointeroffset.

The index gb18030 ranges pointer for code point is the return value of these steps:

  1. Let offset be the last code point in index gb18030 ranges that is equal to or less than code point and let pointer offset be its corresponding pointer.

  2. Return a pointer whose value is pointer offset + code pointoffset.

The index shift_jis pointer for code point is the return value of these steps:

  1. Let index be index jis0208 excluding all pointers in the range 8272 to 8835, inclusive.

  2. Return the index pointer for code point in index.

The index big5 pointer for code point is the return value of these steps:

  1. Let index be index big5 excluding all pointers less than (0xA1 - 0x81) × 157.

    Note

    Avoid returning Hong Kong Supplementary Character Set extensions literally.

  2. If code point is U+2550, U+255E, U+2561, U+256A, U+5341, or U+5345, return the last pointer corresponding to code point in index.

    Note

    There are other duplicate code points, but for those the first pointer is to be used.

  3. Return the index pointer for code point in index.

Note

All indexes are also available as non-normative indexes.json resource. (index gb18030 ranges has a slightly different format here, to be able to represent ranges.)

7. Specification hooks

Note

The algorithms decode, utf-8 decode, utf-8 decode without BOM, encode, and utf-8 encode are intended for usage by other specifications. utf-8 decode is to be used by new formats. The get an encoding algorithm can be used first to turn a label into an encoding.

To decode a byte stream stream using fallback encoding encoding, run these steps:

  1. Let buffer be an empty byte sequence.

  2. Let BOM seen flag be unset.

  3. Read bytes from stream into buffer until either buffer contains three bytes or read returns end-of-stream.

  4. For each of the rows in the table below, starting with the first one and going down, if the first bytes of buffer match all the bytes given in the first column, then set encoding to the encoding given in the cell in the second column of that row and set BOM seen flag.

    Byte order mark Encoding
    0xEF 0xBB 0xBF utf-8
    0xFE 0xFF utf-16be
    0xFF 0xFE utf-16le
    Note

    For compatibility with deployed content, the byte order mark (also known as BOM) is considered more authoritative than anything else.

  5. If BOM seen flag is unset prepend buffer to stream.

  6. Otherwise, if BOM seen flag is set, encoding is not utf-8, and buffer contains three bytes, prepend the last byte of buffer to stream.

  7. Let output be a code point stream.

  8. Run encoding's decoder with stream and output.

  9. Return output.

To utf-8 decode a byte stream stream, run these steps:

  1. Let buffer be an empty byte sequence.

  2. Read three bytes from stream into buffer.

  3. If buffer does not match 0xEF 0xBB 0xBF, prepend buffer to stream.

  4. Let output be a code point stream.

  5. Run utf-8's decoder with stream and output.

  6. Return output.

To utf-8 decode without BOM a byte stream stream, run these steps:

  1. Let output be a code point stream.

  2. Run utf-8's decoder with stream and output.

  3. Return output.

To encode a code point stream stream using encoding encoding, run these steps:

  1. Let output be a byte stream.

  2. Run encoding's encoder with stream, output, and HTML.

  3. Return output.

Note

This is mostly a legacy hook for URLs and HTML forms. Layering utf-8 encode on top is safe as it never triggers errors. [URL] [HTML]

To utf-8 encode a code point stream stream, return the result of encoding stream using encoding utf-8.

8. API

This section uses terminology from the DOM and Web IDL. Non-browser user agents are not required to support this API. [DOM] [WEBIDL]

The following example uses the TextEncoder object to encode an array of strings into an ArrayBuffer. The result is a Uint8Array containing the number of strings (as a Uint32Array), followed by the length of the first string (as a Uint32Array), the utf-8 encoded string data, the length of the second string (as a Uint32Array), the string data, and so on.

function encodeArrayOfStrings(strings, encoding) {
  var encoder, encoded, len, bytes, view, offset;

  encoder = new TextEncoder(encoding);
  encoded = [];

  len = Uint32Array.BYTES_PER_ELEMENT;
  for (var i = 0; i < strings.length; i++) {
    len += Uint32Array.BYTES_PER_ELEMENT;
    encoded[i] = encoder.encode(strings[i]);
    len += encoded[i].byteLength;
  }

  bytes = new Uint8Array(len);
  view = new DataView(bytes.buffer);
  offset = 0;

  view.setUint32(offset, strings.length);
  offset += Uint32Array.BYTES_PER_ELEMENT;
  for (var i = 0; i < encoded.length; i += 1) {
    len = encoded[i].byteLength;
    view.setUint32(offset, len);
    offset += Uint32Array.BYTES_PER_ELEMENT;
    bytes.set(encoded[i], offset);
    offset += len;
  }
  return bytes.buffer;
}

The following example decodes an ArrayBuffer containing data encoded in the format produced by the previous example back into an array of strings.

function decodeArrayOfStrings(buffer, encoding) {
  var decoder, view, offset, num_strings, strings, len;

  decoder = new TextDecoder(encoding);
  view = new DataView(buffer);
  offset = 0;
  strings = [];

  num_strings = view.getUint32(offset);
  offset += Uint32Array.BYTES_PER_ELEMENT;
  for (var i = 0; i < num_strings; i++) {
    len = view.getUint32(offset);
    offset += Uint32Array.BYTES_PER_ELEMENT;
    strings[i] = decoder.decode(
      new DataView(view.buffer, offset, len));
    offset += len;
  }
  return strings;
}

Interface TextDecoder

dictionary TextDecoderOptions {
  boolean fatal = false;
  boolean ignoreBOM = false;
};

dictionary TextDecodeOptions {
  boolean stream = false;
};

[Constructor(optional DOMString label = "utf-8", optional TextDecoderOptions options),
 Exposed=Window,Worker]
interface TextDecoder {
  readonly attribute DOMString encoding;
  readonly attribute boolean fatal;
  readonly attribute boolean ignoreBOM;
  USVString decode(optional BufferSource input, optional TextDecodeOptions options);
};

A TextDecoder object has an associated encoding, decoder, stream, ignore BOM flag (initially unset), BOM seen flag (initially unset), error mode (initially replacement), and streaming flag (initially unset).

A TextDecoder object also has an associated serialize stream algorithm, that given a stream stream, runs these steps:

  1. Let output be the empty string.

  2. While true:

    1. Let token be the result of reading from stream.

    2. If encoding is one of utf-8, utf-16be, and utf-16le, and ignore BOM flag and BOM seen flag are unset, run these subsubsteps:

      1. If token is U+FEFF, set BOM seen flag.

      2. Otherwise, if token is not end-of-stream, set BOM seen flag and append token to output.

      3. Otherwise, return output.

    3. Otherwise, if token is not end-of-stream, append token to output.

    4. Otherwise, return output.

This algorithm is intentionally different with respect to BOM handling from the decode algorithm used by the rest of the platform to give API users more control.

decoder = new TextDecoder([label = "utf-8" [, options]])

Returns a new TextDecoder object.

If label is either not a label or is a label for replacement, throws a RangeError.

decoder . encoding

Returns encoding's name.

decoder . fatal

Returns true if error mode is fatal, and false otherwise.

decoder . ignoreBOM

Returns true if ignore BOM flag is set, and false otherwise.

decoder . decode([input [, options]])

Returns the result of running encoding's decoder. If options's stream is true, the method can be invoked multiple times to process a fragmented stream.

If the error mode is fatal and encoding's decoder returns error, throws a TypeError.

The TextDecoder(label, options) constructor, when invoked, must run these steps:

  1. Let encoding be the result of getting an encoding from label.

  2. If encoding is failure or replacement, throw a RangeError.

  3. Let dec be a new TextDecoder object.

  4. Set dec's encoding to encoding.

  5. If options's fatal member is true, set dec's error mode to fatal.

  6. If options's ignoreBOM member is true, set dec's ignore BOM flag.

  7. Return dec.

The encoding attribute's getter must return encoding's name.

The fatal attribute's getter must return true if error mode is fatal, and false otherwise.

The ignoreBOM attribute's getter must return true if ignore BOM flag is set, and false otherwise.

The decode(input, options) method, when invoked, must run these steps:

  1. If the streaming flag is unset, set decoder to a new encoding's decoder, set stream to a new stream, and unset the BOM seen flag.

  2. If options's stream is true, set the streaming flag, and unset the streaming flag otherwise.

  3. If input is given, push a copy of input to stream.

  4. Let output be a new stream.

  5. While true:

    1. Let token be the result of reading from stream.

    2. If token is end-of-stream and the streaming flag is set, return output, serialized.

    3. Otherwise, run these subsubsteps:

      1. Let result be the result of processing token for decoder, stream, output, and error mode.

      2. If result is finished, return output, serialized.

      3. Otherwise, if result is error, throw a TypeError.

      4. Otherwise, do nothing.

Interface TextEncoder

[Constructor(optional DOMString utfLabel = "utf-8"),
 Exposed=Window,Worker]
interface TextEncoder {
  readonly attribute DOMString encoding;
  [NewObject] Uint8Array encode(optional USVString input = "");
};

A TextEncoder object has an associated encoding and encoder.

A TextEncoder object offers no stream option as no encoder requires buffering of scalar values.

encoder = new TextEncoder([utfLabel = "utf-8"])

Returns a new TextEncoder object.

If utfLabel is not a label for utf-8, utf-16be, or utf-16le, throws a RangeError.

encoder . encoding

Returns encoding's name.

encoder . encode([input = ""])

Returns the result of running encoding's encoder.

The TextEncoder(utfLabel) constructor, when invoked, must run these steps:

  1. Let encoding be the result of getting an encoding from utfLabel.

  2. If encoding is failure, or is none of utf-8, utf-16be, and utf-16le, throw a RangeError.

  3. Let enc be a new TextEncoder object.

  4. Set enc's encoding to encoding.

  5. Set enc's encoder to a new enc's encoding's encoder.

  6. Return enc.

The encoding attribute's getter must return encoding's name.

The encode(input) method, when invoked, must run these steps:

  1. Convert input to a stream.

  2. Let output be a new stream.

  3. While true, run these substeps:

    1. Let token be the result of reading from input.

    2. Let result be the result of processing token for encoder, input, output.

    3. If result is finished, convert output into a byte sequence, and then return a Uint8Array object wrapping an ArrayBuffer containing output.

      These encodings cannot return error.

9. The encoding

9.1 utf-8

9.1.1 utf-8 decoder

utf-8's decoder's has an associated utf-8 code point, utf-8 bytes seen, and utf-8 bytes needed (all initially 0), a utf-8 lower boundary (initially 0x80), and a utf-8 upper boundary (initially 0xBF).

utf-8's decoder's handler, given a stream and byte, runs these steps:

  1. If byte is end-of-stream and utf-8 bytes needed is not 0, set utf-8 bytes needed to 0 and return error.

  2. If byte is end-of-stream, return finished.

  3. If utf-8 bytes needed is 0, based on byte:

    0x00 to 0x7F

    Return a code point whose value is byte.

    0xC2 to 0xDF

    Set utf-8 bytes needed to 1 and utf-8 code point to byte − 0xC0.

    0xE0 to 0xEF
    1. If byte is 0xE0, set utf-8 lower boundary to 0xA0.

    2. If byte is 0xED, set utf-8 upper boundary to 0x9F.

    3. Set utf-8 bytes needed to 2 and utf-8 code point to byte − 0xE0.

    0xF0 to 0xF4
    1. If byte is 0xF0, set utf-8 lower boundary to 0x90.

    2. If byte is 0xF4, set utf-8 upper boundary to 0x8F.

    3. Set utf-8 bytes needed to 3 and utf-8 code point to byte − 0xF0.

    Otherwise

    Return error.

    Then (byte is in the range 0xC2 to 0xF4, inclusive) set utf-8 code point to utf-8 code point << (6 × utf-8 bytes needed) and return continue.

  4. If byte is not in the range utf-8 lower boundary to utf-8 upper boundary, inclusive, run these substeps:

    1. Set utf-8 code point, utf-8 bytes needed, and utf-8 bytes seen to 0, set utf-8 lower boundary to 0x80, and set utf-8 upper boundary to 0xBF.

    2. Prepend byte to stream.

    3. Return error.

  5. Set utf-8 lower boundary to 0x80 and utf-8 upper boundary to 0xBF.

  6. Increase utf-8 bytes seen by one and set utf-8 code point to utf-8 code point + (byte − 0x80) << (6 × (utf-8 bytes neededutf-8 bytes seen)).

  7. If utf-8 bytes seen is not equal to utf-8 bytes needed, return continue.

  8. Let code point be utf-8 code point.

  9. Set utf-8 code point, utf-8 bytes needed, and utf-8 bytes seen to 0.

  10. Return a code point whose value is code point.

Note

The constraints in the utf-8 decoder above match “Best Practices for Using U+FFFD” from the Unicode standard. No other behavior is permitted per the Encoding Standard (other algorithms that achieve the same result are obviously fine, even encouraged).

9.1.2 utf-8 encoder

utf-8's encoder's handler, given a stream and code point, runs these steps:

  1. If code point is end-of-stream, return finished.

  2. If code point is an ASCII code point, return a byte whose value is code point.

  3. Set count and offset based on the range code point is in:

    U+0080 to U+07FF, inclusive
    1 and 0xC0
    U+0800 to U+FFFF, inclusive
    2 and 0xE0
    U+10000 to U+10FFFF, inclusive
    3 and 0xF0
  4. Let bytes be a byte sequence whose first byte is (code point >> (6 × count)) + offset.

  5. Run these substeps while count is greater than 0:

    1. Set temp to code point >> (6 × (count − 1)).

    2. Append to bytes 0x80 | (temp & 0x3F).

    3. Decrease count by one.

  6. Return bytes bytes, in order.

10. Legacy single-byte encodings

An encoding where each byte is either a single code point or nothing, is a single-byte encoding. Single-byte encodings share the decoder and encoder. Index single-byte, as referenced by the single-byte decoder and single-byte encoder, is defined by the following table, and depends on the single-byte encoding in use. All but two single-byte encodings have a unique index.

Name Index
ibm866 index-ibm866.txt
iso-8859-2 index-iso-8859-2.txt
iso-8859-3 index-iso-8859-3.txt
iso-8859-4 index-iso-8859-4.txt
iso-8859-5 index-iso-8859-5.txt
iso-8859-6 index-iso-8859-6.txt
iso-8859-7 index-iso-8859-7.txt
iso-8859-8 index-iso-8859-8.txt
iso-8859-8-i
iso-8859-10 index-iso-8859-10.txt
iso-8859-13 index-iso-8859-13.txt
iso-8859-14 index-iso-8859-14.txt
iso-8859-15 index-iso-8859-15.txt
iso-8859-16 index-iso-8859-16.txt
koi8-r index-koi8-r.txt
koi8-u index-koi8-u.txt
macintosh index-macintosh.txt
windows-874 index-windows-874.txt
windows-1250 index-windows-1250.txt
windows-1251 index-windows-1251.txt
windows-1252 index-windows-1252.txt
windows-1253 index-windows-1253.txt
windows-1254 index-windows-1254.txt
windows-1255 index-windows-1255.txt
windows-1256 index-windows-1256.txt
windows-1257 index-windows-1257.txt
windows-1258 index-windows-1258.txt
x-mac-cyrillic index-x-mac-cyrillic.txt
Note

iso-8859-8 and iso-8859-8-i are distinct encoding names, because iso-8859-8 has influence on the layout direction. And although historically this might have been the case for iso-8859-6 and "iso-8859-6-i" as well, that is no longer true.

10.1 single-byte decoder

Single-byte encodings's decoder's handler, given a stream and byte, runs these steps:

  1. If byte is end-of-stream, return finished.

  2. If byte is an ASCII byte, return a code point whose value is byte.

  3. Let code point be the index code point for byte − 0x80 in index single-byte.

  4. If code point is null, return error.

  5. Return a code point whose value is code point.

10.2 single-byte encoder

Single-byte encodings's encoder's handler, given a stream and code point, runs these steps:

  1. If code point is end-of-stream, return finished.

  2. If code point is an ASCII code point, return a byte whose value is code point.

  3. Let pointer be the index pointer for code point in index single-byte.

  4. If pointer is null, return error with code point.

  5. Return a byte whose value is pointer + 0x80.

11. Legacy multi-byte Chinese (simplified) encodings

11.1 gbk

11.1.1 gbk decoder

gbk's decoder is gb18030's decoder.

11.1.2 gbk encoder

gbk's encoder is gb18030's encoder with its gbk flag set.

Note

Not fully aliasing gbk with gb18030 is a conservative move to decrease the chances of breaking legacy servers and other consumers of content generated with gbk's encoder.

11.2 gb18030

11.2.1 gb18030 decoder

gb18030's decoder has an associated gb18030 first, gb18030 second, and gb18030 third (all initially 0x00).

gb18030's decoder's handler, given a stream and byte, runs these steps:

  1. If byte is end-of-stream and gb18030 first, gb18030 second, and gb18030 third are 0x00, return finished.

  2. If byte is end-of-stream, and gb18030 first, gb18030 second, or gb18030 third is not 0x00, set gb18030 first, gb18030 second, and gb18030 third to 0x00, and return error.

  3. If gb18030 third is not 0x00, run these substeps:

    1. Let code point be null.

    2. If byte is in the range 0x30 to 0x39, inclusive, set code point to the index gb18030 ranges code point for (((gb18030 first − 0x81) × 10 + gb18030 second − 0x30) × 126 + gb18030 third − 0x81) × 10 + byte − 0x30.

    3. Let buffer be a byte sequence consisting of gb18030 second, gb18030 third, and byte, in order.

    4. Set gb18030 first, gb18030 second, and gb18030 third to 0x00.

    5. If code point is null, prepend buffer to stream and return error.

    6. Return a code point whose value is code point.

  4. If gb18030 second is not 0x00, run these substeps:

    1. If byte is in the range 0x81 to 0xFE, inclusive, set gb18030 third to byte and return continue.

    2. Prepend gb18030 second followed by byte to stream, set gb18030 first and gb18030 second to 0x00, and return error.

  5. If gb18030 first is not 0x00, run these substeps:

    1. If byte is in the range 0x30 to 0x39, inclusive, set gb18030 second to byte and return continue.

    2. Let lead be gb18030 first, let pointer be null, and set gb18030 first to 0x00.

    3. Let offset be 0x40 if byte is less than 0x7F and 0x41 otherwise.

    4. If byte is in the range 0x40 to 0x7E, inclusive, or 0x80 to 0xFE, inclusive, set pointer to (lead − 0x81) × 190 + (byteoffset).

    5. Let code point be null if pointer is null and the index code point for pointer in index gb18030 otherwise.

    6. If code point is null and byte is an ASCII byte, prepend byte to stream.

    7. If code point is null, return error.

    8. Return a code point whose value is code point.

  6. If byte is an ASCII byte, return a code point whose value is byte.

  7. If byte is 0x80, return code point U+20AC.

  8. If byte is in the range 0x81 to 0xFE, inclusive, set gb18030 first to byte and return continue.

  9. Return error.

11.2.2 gb18030 encoder

gb18030's encoder has an associated gbk flag (initially unset).

gb18030's encoder's handler, given a stream and code point, runs these steps:

  1. If code point is end-of-stream, return finished.

  2. If code point is an ASCII code point, return a byte whose value is code point.

  3. If the gbk flag is set and code point is U+20AC, return byte 0x80.

  4. Let pointer be the index pointer for code point in index gb18030.

  5. If pointer is not null, run these substeps:

    1. Let lead be pointer / 190 + 0x81.

    2. Let trail be pointer % 190.

    3. Let offset be 0x40 if trail is less than 0x3F and 0x41 otherwise.

    4. Return two bytes whose values are lead and trail + offset.

  6. If gbk flag is set, return error with code point.

  7. Set pointer to the index gb18030 ranges pointer for code point.

  8. Let byte1 be pointer / 10 / 126 / 10.

  9. Set pointer to pointerbyte1 × 10 × 126 × 10.

  10. Let byte2 be pointer / 10 / 126.

  11. Set pointer to pointerbyte2 × 10 × 126.

  12. Let byte3 be pointer / 10.

  13. Let byte4 be pointerbyte3 × 10.

  14. Return four bytes whose values are byte1 + 0x81, byte2 + 0x30, byte3 + 0x81, byte4 + 0x30.

12. Legacy multi-byte Chinese (traditional) encodings

12.1 big5

12.1.1 big5 decoder

big5's decoder has an associated big5 lead (initially 0x00). big5's decoder's handler, given a stream and byte, runs these steps:

  1. If byte is end-of-stream and big5 lead is not 0x00, set big5 lead to 0x00 and return error.

  2. If byte is end-of-stream and big5 lead is 0x00, return finished.

  3. If big5 lead is not 0x00, let lead be big5 lead, let pointer be null, set big5 lead to 0x00, and then run these substeps:

    1. Let offset be 0x40 if byte is less than 0x7F and 0x62 otherwise.

    2. If byte is in the range 0x40 to 0x7E, inclusive, or 0xA1 to 0xFE, inclusive, set pointer to (lead − 0x81) × 157 + (byteoffset).

    3. If there is a row in the table below whose first column is pointer, return the two code points listed in its second column (the third column is irrelevant):

      Pointer Code points Notes
      1133 U+00CA U+0304 Ê̄ (LATIN CAPITAL LETTER E WITH CIRCUMFLEX AND MACRON)
      1135 U+00CA U+030C Ê̌ (LATIN CAPITAL LETTER E WITH CIRCUMFLEX AND CARON)
      1164 U+00EA U+0304 ê̄ (LATIN SMALL LETTER E WITH CIRCUMFLEX AND MACRON)
      1166 U+00EA U+030C ê̌ (LATIN SMALL LETTER E WITH CIRCUMFLEX AND CARON)
      Note

      Since indexes are limited to single code points this table is used for these pointers.

    4. Let code point be null if pointer is null and the index code point for pointer in index big5 otherwise.

    5. If code point is null and byte is an ASCII byte, prepend byte to stream.

    6. If code point is null, return error.

    7. Return a code point whose value is code point.

  4. If byte is an ASCII byte, return a code point whose value is byte.

  5. If byte is in the range 0x81 to 0xFE, inclusive, set big5 lead to byte and return continue.

  6. Return error.

12.1.2 big5 encoder

big5's encoder's handler, given a stream and code point, runs these steps:

  1. If code point is end-of-stream, return finished.

  2. If code point is an ASCII code point, return a byte whose value is code point.

  3. Let pointer be the index big5 pointer for code point.

  4. If pointer is null, return error with code point.

  5. Let lead be pointer / 157 + 0x81.

  6. Let trail be pointer % 157.

  7. Let offset be 0x40 if trail is less than 0x3F and 0x62 otherwise.

  8. Return two bytes whose values are lead and trail + offset.

13. Legacy multi-byte Japanese encodings

13.1 euc-jp

13.1.1 euc-jp decoder

euc-jp's decoder has an associated euc-jp jis0212 flag (initially unset) and euc-jp lead (initially 0x00).

euc-jp's decoder's handler, given a stream and byte, runs these steps:

  1. If byte is end-of-stream and euc-jp lead is not 0x00, set euc-jp lead to 0x00, and return error.

  2. If byte is end-of-stream and euc-jp lead is 0x00, return finished.

  3. If euc-jp lead is 0x8E and byte is in the range 0xA1 to 0xDF, inclusive, set euc-jp lead to 0x00 and return a code point whose value is 0xFF61 + byte − 0xA1.

  4. If euc-jp lead is 0x8F and byte is in the range 0xA1 to 0xFE, inclusive, set the euc-jp jis0212 flag, set euc-jp lead to byte, and return continue.

  5. If euc-jp lead is not 0x00, let lead be euc-jp lead, set euc-jp lead to 0x00, and run these substeps:

    1. Let code point be null.

    2. If lead and byte are both in the range 0xA1 to 0xFE, inclusive, set code point to the index code point for (lead − 0xA1) × 94 + byte − 0xA1 in index jis0208 if the euc-jp jis0212 flag is unset and in index jis0212 otherwise.

    3. Unset the euc-jp jis0212 flag.

    4. If byte is not in the range 0xA1 to 0xFE, inclusive, prepend byte to stream.

    5. If code point is null, return error.

    6. Return a code point whose value is code point.

  6. If byte is an ASCII byte, return a code point whose value is byte.

  7. If byte is 0x8E, 0x8F, or in the range 0xA1 to 0xFE, inclusive, set euc-jp lead to byte and return continue.

  8. Return error.

13.1.2 euc-jp encoder

euc-jp's encoder's handler, given a stream and code point, runs these steps:

  1. If code point is end-of-stream, return finished.

  2. If code point is an ASCII code point, return a byte whose value is code point.

  3. If code point is U+00A5, return byte 0x5C.

  4. If code point is U+203E, return byte 0x7E.

  5. If code point is in the range U+FF61 to U+FF9F, inclusive, return two bytes whose values are 0x8E and code point − 0xFF61 + 0xA1.

  6. If code point is U+2022, set it to U+FF0D.

  7. Let pointer be the index pointer for code point in index jis0208.

  8. If pointer is null, return error with code point.

  9. Let lead be pointer / 94 + 0xA1.

  10. Let trail be pointer % 94 + 0xA1.

  11. Return two bytes whose values are lead and trail.

13.2 iso-2022-jp

13.2.1 iso-2022-jp decoder

iso-2022-jp's decoder has an associated iso-2022-jp decoder state (initially ASCII), iso-2022-jp decoder output state (initially ASCII), iso-2022-jp lead (initially 0x00), and iso-2022-jp output flag (initially unset).

iso-2022-jp's decoder's handler, given a stream and byte, runs these steps, switching on iso-2022-jp decoder state:

ASCII

Based on byte:

0x1B

Set iso-2022-jp decoder state to escape start and return continue.

0x00 to 0x7F, excluding 0x0E, 0x0F, and 0x1B

Unset the iso-2022-jp output flag and return a code point whose value is byte.

end-of-stream

Return finished.

Otherwise

Unset the iso-2022-jp output flag and return error.

Roman

Based on byte:

0x1B

Set iso-2022-jp decoder state to escape start and return continue.

0x5C

Unset the iso-2022-jp output flag and return code point U+00A5.

0x7E

Unset the iso-2022-jp output flag and return code point U+203E.

0x00 to 0x7F, excluding 0x0E, 0x0F, 0x1B, 0x5C, and 0x7E

Unset the iso-2022-jp output flag and return a code point whose value is byte.

end-of-stream

Return finished.

Otherwise

Unset the iso-2022-jp output flag and return error.

Katakana

Based on byte:

0x1B

Set iso-2022-jp decoder state to escape start and return continue.

0x21 to 0x5F

Unset the iso-2022-jp output flag and return a code point whose value is 0xFF61 + byte − 0x21.

end-of-stream

Return finished.

Otherwise

Unset the iso-2022-jp output flag and return error.

Lead byte

Based on byte:

0x1B

Set iso-2022-jp decoder state to escape start and return continue.

0x21 to 0x7E

Unset the iso-2022-jp output flag, set iso-2022-jp lead to byte, iso-2022-jp decoder state to trail byte, and return continue.

end-of-stream

Return finished.

Otherwise

Unset the iso-2022-jp output flag and return error.

Trail byte

Based on byte:

0x1B

Set iso-2022-jp decoder state to escape start and return error.

0x21 to 0x7E
  1. Set the iso-2022-jp decoder state to lead byte.

  2. Let pointer be (iso-2022-jp lead − 0x21) × 94 + byte − 0x21.

  3. Let code point be the index code point for pointer in index jis0208.

  4. If code point is null, return error.

  5. Return a code point whose value is code point.

end-of-stream

Set the iso-2022-jp decoder state to lead byte, prepend byte to stream, and return error.

Otherwise

Set iso-2022-jp decoder state to lead byte and return error.

Escape start
  1. If byte is either 0x24 or 0x28, set iso-2022-jp lead to byte, iso-2022-jp decoder state to escape, and return continue.

  2. Prepend byte to stream.

  3. Unset the iso-2022-jp output flag, set iso-2022-jp decoder state to iso-2022-jp decoder output state, and return error.

Escape
  1. Let lead be iso-2022-jp lead and set iso-2022-jp lead to 0x00.

  2. Let state be null.

  3. If lead is 0x28 and byte is 0x42, set state to ASCII.

  4. If lead is 0x28 and byte is 0x4A, set state to Roman.

  5. If lead is 0x28 and byte is 0x49, set state to Katakana.

  6. If lead is 0x24 and byte is either 0x40 or 0x42, set state to lead byte.

  7. If state is non-null, run these substeps:

    1. Set iso-2022-jp decoder state and iso-2022-jp decoder output state to state.

    2. Let output flag be the iso-2022-jp output flag.

    3. Set the iso-2022-jp output flag.

    4. Return continue, if output flag is unset, and error otherwise.

  8. Prepend lead and byte to stream.

  9. Unset the iso-2022-jp output flag, set iso-2022-jp decoder state to iso-2022-jp decoder output state and return error.

13.2.2 iso-2022-jp encoder

iso-2022-jp's encoder has an associated iso-2022-jp encoder state which is one of ASCII, Roman, and jis0208 (initially ASCII).

iso-2022-jp's encoder's handler, given a stream and code point, runs these steps:

  1. If code point is end-of-stream and iso-2022-jp encoder state is not ASCII, prepend code point to stream, set iso-2022-jp encoder state to ASCII, and return three bytes 0x1B 0x28 0x42.

  2. If code point is end-of-stream and iso-2022-jp encoder state is ASCII, return finished.

  3. If iso-2022-jp encoder state is ASCII and code point is an ASCII code point, return a byte whose value is code point.

  4. If iso-2022-jp encoder state is Roman and code point is an ASCII code point, excluding U+005C and U+007E, or is U+00A5 or U+203E, run these substeps:

    1. If code point is an ASCII code point, return a byte whose value is code point.

    2. If code point is U+00A5, return byte 0x5C.

    3. If code point is U+203E, return byte 0x7E.

  5. If code point is an ASCII code point, and iso-2022-jp encoder state is not ASCII, prepend code point to stream, set iso-2022-jp encoder state to ASCII, and return three bytes 0x1B 0x28 0x42.

  6. If code point is either U+00A5 or U+203E, and iso-2022-jp encoder state is not Roman, prepend code point to stream, set iso-2022-jp encoder state to Roman, and return three bytes 0x1B 0x28 0x4A.

  7. If code point is U+2022, set it to U+FF0D.

  8. Let pointer be the index pointer for code point in index jis0208.

  9. If pointer is null, return error with code point.

  10. If iso-2022-jp encoder state is not jis0208, prepend code point to stream, set iso-2022-jp encoder state to jis0208, and return three bytes 0x1B 0x24 0x42.

  11. Let lead be pointer / 94 + 0x21.

  12. Let trail be pointer % 94 + 0x21.

  13. Return two bytes whose values are lead and trail.

13.3 shift_jis

13.3.1 shift_jis decoder

shift_jis's decoder has an associated shift_jis lead (initially 0x00).

shift_jis's decoder's handler, given a stream and byte, runs these steps:

  1. If byte is end-of-stream and shift_jis lead is not 0x00, set shift_jis lead to 0x00 and return error.

  2. If byte is end-of-stream and shift_jis lead is 0x00, return finished.

  3. If shift_jis lead is not 0x00, let lead be shift_jis lead, let pointer be null, set shift_jis lead to 0x00, and then run these substeps:

    1. Let offset be 0x40, if byte is less than 0x7F, and 0x41 otherwise.

    2. Let lead offset be 0x81, if lead is less than 0xA0, and 0xC1 otherwise.

    3. If byte is in the range 0x40 to 0x7E, inclusive, or 0x80 to 0xFC, inclusive, set pointer to (leadlead offset) × 188 + byteoffset.

    4. Let code point be null, if pointer is null, and the index code point for pointer in index jis0208 otherwise.

    5. If code point is null and pointer is in the range 8836 to 10528, inclusive, return a code point whose value is 0xE000 + pointer − 8836.

      Note

      This is interoperable legacy from Windows known as EUDC.

    6. If code point is null and byte is an ASCII byte, prepend byte to stream.

    7. If code point is null, return error.

    8. Return a code point whose value is code point.

  4. If byte is an ASCII byte or 0x80, return a code point whose value is byte.

  5. If byte is in the range 0xA1 to 0xDF, inclusive, return a code point whose value is 0xFF61 + byte − 0xA1.

  6. If byte is in the range 0x81 to 0x9F, inclusive, or 0xE0 to 0xFC, inclusive, set shift_jis lead to byte and return continue.

  7. Return error.

13.3.2 shift_jis encoder

shift_jis's encoder's handler, given a stream and code point, runs these steps:

  1. If code point is end-of-stream, return finished.

  2. If code point is an ASCII code point or U+0080, return a byte whose value is code point.

  3. If code point is U+00A5, return byte 0x5C.

  4. If code point is U+203E, return byte 0x7E.

  5. If code point is in the range U+FF61 to U+FF9F, inclusive, return a byte whose value is code point − 0xFF61 + 0xA1.

  6. If code point is U+2022, set it to U+FF0D.

  7. Let pointer be the index shift_jis pointer for code point.

  8. If pointer is null, return error with code point.

  9. Let lead be pointer / 188.

  10. Let lead offset be 0x81, if lead is less than 0x1F, and 0xC1 otherwise.

  11. Let trail be pointer % 188.

  12. Let offset be 0x40, if trail is less than 0x3F, and 0x41 otherwise.

  13. Return two bytes whose values are lead + lead offset and trail + offset.

14. Legacy multi-byte Korean encodings

14.1 euc-kr

14.1.1 euc-kr decoder

euc-kr's decoder has an associated euc-kr lead (initially 0x00).

euc-kr's decoder's handler, given a stream and byte, runs these steps:

  1. If byte is end-of-stream and euc-kr lead is not 0x00, set euc-kr lead to 0x00 and return error.

  2. If byte is end-of-stream and euc-kr lead is 0x00, return finished.

  3. If euc-kr lead is not 0x00, let lead be euc-kr lead, let pointer be null, set euc-kr lead to 0x00, and then run these substeps:

    1. If byte is in the range 0x41 to 0xFE, inclusive, set pointer to (lead − 0x81) × 190 + (byte − 0x41).

    2. Let code point be null, if pointer is null, and the index code point for pointer in index euc-kr otherwise.

    3. If code point is null and byte is an ASCII byte, prepend byte to stream.

    4. If code point is null, return error.

    5. Return a code point whose value is code point.

  4. If byte is an ASCII byte, return a code point whose value is byte.

  5. If byte is in the range 0x81 to 0xFE, inclusive, set euc-kr lead to byte and return continue.

  6. Return error.

14.1.2 euc-kr encoder

euc-kr's encoder's handler, given a stream and code point, runs these steps:

  1. If code point is end-of-stream, return finished.

  2. If code point is an ASCII code point, return a byte whose value is code point.

  3. Let pointer be the index pointer for code point in index euc-kr.

  4. If pointer is null, return error with code point.

  5. Let lead be pointer / 190 + 0x81.

  6. Let trail be pointer % 190 + 0x41.

  7. Return two bytes whose values are lead and trail.

15. Legacy miscellaneous encodings

15.1 replacement

Note

The replacement encoding exists to prevent certain attacks that abuse a mismatch between encodings supported on the server and the client.

15.1.1 replacement decoder

replacement's decoder has an associated replacement error returned flag (initially unset).

replacement's decoder's handler, given a stream and byte, runs these steps:

  1. If replacement error returned flag is unset, set the replacement error returned flag and return error.

  2. Return finished.

15.1.2 replacement encoder

replacement's encoder is utf-8's encoder.

15.2 Common infrastructure for utf-16be and utf-16le

To convert a code unit to bytes using a utf-16be flag, run these steps:

  1. Let byte1 be code unit >> 8.

  2. Let byte2 be code unit & 0x00FF.

  3. Then return the bytes in order:

    utf-16be flag is set

    byte1, then byte2.

    utf-16be flag is unset

    byte2, then byte1.

15.2.1 shared utf-16 decoder

Note

A byte order mark has priority over a label as it has been found to be more accurate in deployed content. Therefore it is not part of the shared utf-16 decoder algorithm but rather the decode algorithm.

shared utf-16 decoder has an associated utf-16 lead byte and utf-16 lead surrogate (both initially null), and utf-16be decoder flag (initially unset).

shared utf-16 decoder's handler, given a stream and byte, runs these steps:

  1. If byte is end-of-stream and either utf-16 lead byte or utf-16 lead surrogate is not null, set utf-16 lead byte and utf-16 lead surrogate to null, and return error.

  2. If byte is end-of-stream and utf-16 lead byte and utf-16 lead surrogate are null, return finished.

  3. If utf-16 lead byte is null, set utf-16 lead byte to byte and return continue.

  4. Let code unit be the result of:

    utf-16be decoder flag is set

    (utf-16 lead byte << 8) + byte.

    utf-16be decoder flag is unset

    (byte << 8) + utf-16 lead byte.

    Then set utf-16 lead byte to null.

  5. If utf-16 lead surrogate is not null, let lead surrogate be utf-16 lead surrogate, set utf-16 lead surrogate to null, and then run these substeps:

    1. If code unit is in the range U+DC00 to U+DFFF, inclusive, return a code point whose value is 0x10000 + ((lead surrogate − 0xD800) << 10) + (code unit − 0xDC00).

    2. Prepend the sequence resulting of converting code unit to bytes using utf-16be decoder flag to stream and return error.

  6. If code unit is in the range U+D800 to U+DBFF, inclusive, set utf-16 lead surrogate to code unit and return continue.

  7. If code unit is in the range U+DC00 to U+DFFF, inclusive, return error.

  8. Return code point code unit.

15.2.2 shared utf-16 encoder

shared utf-16 encoder has an associated utf-16be encoder flag (initially unset).

shared utf-16 encoder's handler, given a stream and code point, runs these steps:

  1. If code point is end-of-stream, return finished.

  2. If code point is in the range U+0000 to U+FFFF, inclusive, return the sequence resulting of converting code point to bytes using utf-16be encoder flag.

  3. Let lead be ((code point − 0x10000) >> 10) + 0xD800, converted to bytes using utf-16be encoder flag.

  4. Let trail be ((code point − 0x10000) & 0x3FF) + 0xDC00, converted to bytes using utf-16be encoder flag.

  5. Return a byte sequence of lead followed by trail.

15.3 utf-16be

15.3.1 utf-16be decoder

utf-16be's decoder is shared utf-16 decoder with its utf-16be decoder flag set.

15.3.2 utf-16be encoder

utf-16be's encoder is shared utf-16 encoder with its utf-16be encoder flag set.

15.4 utf-16le

Note

Both "utf-16" and "utf-16le" are labels for utf-16le to deal with deployed content.

15.4.1 utf-16le decoder

utf-16le's decoder is shared utf-16 decoder.

15.4.2 utf-16le encoder

utf-16le's encoder is shared utf-16 encoder.

15.5 x-user-defined

Note

While technically this is a single-byte encoding, it is defined separately as it can be implemented algorithmically.

15.5.1 x-user-defined decoder

x-user-defined's decoder's handler, given a stream and byte, runs these steps:

  1. If byte is end-of-stream, return finished.

  2. If byte is an ASCII byte, return a code point whose value is byte.

  3. Return a code point whose value is 0xF780 + byte − 0x80.

15.5.2 x-user-defined encoder

x-user-defined's encoder's handler, given a stream and code point, runs these steps:

  1. If code point is end-of-stream, return finished.

  2. If code point is an ASCII code point, return a byte whose value is code point.

  3. If code point is in the range U+F780 to U+F7FF, inclusive, return a byte whose value is code point − 0xF780 + 0x80.

  4. Return error with code point.

16. Browser UI

Browsers are encouraged to not enable overriding the encoding of a resource. If such a feature is nonetheless present, browsers should not offer either utf-16be or utf-16le as option due to aforementioned security issues. Browsers also should disable this feature if the resource was decoded using either utf-16be or utf-16le.

17. References

[DOM]
W3C DOM4, Anne van Kesteren, Aryeh Gregor, Ms2ger, Alex Russell, Robin Berjon. W3C.
[HTML]
(Non-normative) HTML5, Ian Hickson, Robin Berjon, Steve Faulkner, Travis Leithead, Erika Doyle Navara, Edward O'Connor, Silvia Pfeiffer. W3C.
[RFC2119]
Key words for use in RFCs to Indicate Requirement Levels, Scott Bradner. IETF.
[UNICODE]
Unicode Standard. Unicode Consortium.
[URL]
(Non-normative) URL, Anne van Kesteren. W3C.
[WEBIDL]
Web IDL, Cameron McCormack. W3C.
[XML]
(Non-normative) Extensible Markup Language, Tim Bray, Jean Paoli, C. M. Sperberg-McQueen et al.. W3C.

18. Revision log

A detailed list of changes, including diffs, can be found in the github commit log.

19. Acknowledgments

There have been a lot of people that have helped make encodings more interoperable over the years and thereby furthered the goals of this standard. Likewise many people have helped making this standard what it is today.

With that, many thanks to Alan Chaney, Alexander Shtuchkin, Allen Wirfs-Brock, Aneesh Agrawal, Arkadiusz Michalski, Asmus Freytag, Ben Noordhuis, Boris Zbarsky, Cameron McCormack, Charles McCathieNeville, David Carlisle, Domenic Denicola, Dominique Hazaël-Massieux, Doug Ewell, Erik van der Poel, 譚永鋒 (Frank Yung-Fong Tang), Geoffrey Sneddon, Glenn Maynard, Gordon P. Hemsley, Henri Sivonen, Ian Hickson, James Graham, John Tamplin, Joshua Bell, 村井純 (Jun Murai), 신정식 (Jungshik Shin), Jxck, 강 성훈 (Kang Seonghoon), 川幡太一 (Kawabata Taichi), Ken Lunde, Ken Whistler, Kenneth Russell, 田村健人 (Kent Tamura), Leif Halvard Silli, Makoto Kato, Mark Callow, Mark Crispin, Mark Davis, Martin Dürst, Masatoshi Kimura, Ms2ger, Nigel Tao, Norbert Lindenberg, Øistein E. Andersen, Peter Krefting, Philip Jägenstedt, Philip Taylor, Richard Ishida, Robbert Broersma, Robert Mustacchi, Ryan Dahl, Shawn Steele, Simon Montagu, Simon Pieters, Simon Sapin, Vyacheslav Matva, and 成瀬ゆい (Yui Naruse) for being awesome.

This standard is written by Anne van Kesteren (Mozilla, annevk@annevk.nl). The API chapter was initially written by Joshua Bell (Google).

Per CC0, to the extent possible under law, the editors have waived all copyright and related or neighboring rights to this work.