1 Preface

While encodings have been defined to some extent, implementations have not always implemented them in the same way, have not always used the same labels, and often differ in dealing with undefined and former proprietary areas of encodings. This specification attempts to fill those gaps so that new implementations do not have to reverse engineer encoding implementations of the market leaders and existing implementations can converge.

In particular, this specification defines the encodings, their algorithms to go from bytes to code points and back, and their canonical names and identifying labels. This specification also defines an API to expose part of the encoding algorithms to JavaScript.

Historically encodings and their specifications (if any) were kept track of by the IANA Character Sets registry. This specification renders that registry obsolete.

2 Conformance

All diagrams, examples, and notes in this specification are non-normative, as are all sections explicitly marked non-normative. Everything else in this specification is normative.

The key words "MUST", "MUST NOT", "REQUIRED", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in the normative parts of this document are to be interpreted as described in RFC2119. For readability, these words do not appear in all uppercase letters in this specification. [RFC2119]

Conformance requirements phrased as algorithms or specific steps may be implemented in any manner, so long as the end result is equivalent. (In particular, the algorithms defined in this specification are intended to be easy to follow, and not intended to be performant.)

User agents may impose implementation-specific limits on otherwise unconstrained inputs, e.g. to prevent denial of service attacks, to guard against running out of memory, or to work around platform-specific limitations.

3 Terminology

Hexadecimal numbers are prefixed with "0x".

In equations, all numbers are integers, addition is represented by "+", subtraction by "−", multiplication by "×", division by "/", calculating the remainder of a division (also known as modulo) by "%", arithmetic left shifts by "<<", arithmetic right shifts by ">>", bitwise AND by "&", and bitwise OR by "|".

A byte is a sequence of eight bits, represented as a double-digit hexadecimal number in the range 0x00 to 0xFF.

A code point is a Unicode code point and is represented as a four-to-six digit hexadecimal number, typically prefixed with "U+". In equations and indexes code points are prefixed with "0x". [UNICODE]

A scalar value is a code point that is not in the range U+D800 to U+DFFF.

The ASCII whitespace are code points U+0009, U+000A, U+000C, U+000D, and U+0020.

The ASCII digits are code points in the range U+0030 to U+0039.

A string is a sequence of code points.

Comparing two strings in an ASCII case-insensitive manner means comparing them exactly, code point for code point, except that the characters in the range U+0041 to U+005A (i.e. LATIN CAPITAL LETTER A to LATIN CAPITAL LETTER Z) and the corresponding characters in the range U+0061 to U+007A (i.e. LATIN SMALL LETTER A to LATIN SMALL LETTER Z) are considered to also match.

A token is a piece of data, such as a byte or code point.

A stream represents an ordered sequence of tokens. End-of-stream is a special token that signifies no more tokens are in the stream.

When a token is read from a stream, the first token in the stream must be returned and subsequently removed, and end-of-stream must be returned otherwise.

When one or more tokens are prepended to a stream, those tokens must be inserted, in given order, before the first token in the stream.

Inserting the sequence of tokens 💩 in a stream " hello world", results in a stream "💩 hello world". The next token to be read would be &.

When one or more tokens are pushed to a stream, those tokens must be inserted, in given order, after the last token in the stream.

4 Encodings

An encoding defines a mapping from a scalar value sequence to a byte sequence (and vice versa). Each encoding has a name, and one or more labels.

4.1 Encoders and decoders

Each encoding has an associated decoder and encoder. Each decoder and encoder have a handler algorithm. A handler algorithm takes an input stream and a token, and returns finished, one or more tokens, error optionally with a code point, or continue.

An error mode as used below is either replacement (default) or fatal for a decoder and one of fatal (default) or HTML for an encoder.

An XML processor would set error mode to fatal. [XML]

HTML exists as error mode due to URLs and HTML forms requiring a non-terminating legacy encoder. The HTML error mode causes a sequence to be emitted that cannot be distinguished from legitimate input and can therefore lead to silent data loss. Developers are strongly encouraged to use the utf-8 encoding to prevent this from happening. [URL] [HTML]

To run an encoding's decoder or encoder encoderDecoder with input stream input, output stream output, and error mode mode, run these steps:

1. If mode is not given, set it to replacement, if encoderDecoder is a decoder, and fatal otherwise.

2. Let encoderDecoderInstance be a new encoderDecoder.

3. While true:

   1. Let result be the result of processing the result of reading from input for encoderDecoderInstance, input, output, and mode.

   2. If result is not continue, return result.

   3. Otherwise, do nothing.

To process a token token for an encoding's encoder or decoder instance encoderDecoderInstance, stream input, output stream output, and error mode mode, run these steps:

1. If mode is not given, set it to replacement, if encoderDecoder is a decoder instance, and fatal otherwise.

2. Let result be the result of running encoderDecoderInstance's handler on token.

3. If result is continue or finished, return result.

4. Otherwise, if result is one or more tokens, push result to output.

5. Otherwise, if result is error, switch on mode and run the associated steps:

   replacement

   Push U+FFFD to output.

   HTML

   Prepend U+0026, U+0023, followed by the shortest sequence of ASCII digits representing result's code point in base ten, followed by U+003B to input.

   fatal

   Return error.

6. Return continue.

4.2 Names and labels

The table below lists all encodings and their labels user agents must support. User agents must not support any other encodings or labels.

Authors must use the utf-8 encoding and must use the ASCII case-insensitive "utf-8" label to identify it.

New protocols and formats, as well as existing formats deployed in new contexts, must use the utf-8 encoding exclusively. If these protocols and formats need to expose the encoding's name or label, they must expose it as "utf-8".

To get an encoding from a string label, run these steps:

1. Remove any leading and trailing ASCII whitespace from label.

2. If label is an ASCII case-insensitive match for any of the labels listed in the table below, return the corresponding encoding, and failure otherwise.

In violation of section 1.4 of Unicode Technical Standard #22 this is a much simpler and more restrictive matching algorithm, as that is found to be necessary to be compatible with deployed content.

All encodings and their labels are also available as non-normative encodings.json resource.

5 Indexes

Most legacy encodings make use of an index. An index is an ordered list of pointers and corresponding code points. Within an index pointers are unique and code points can be duplicated.

An efficient implementation likely has two indexes per encoding. One optimized for its decoder and one for its encoder.

To find the pointers and their corresponding code points in an index, let lines be the result of splitting the resource's contents on U+000A. Then remove each item in lines that is the empty string or starts with U+0023. Then the pointers and their corresponding code points are found by splitting each item in lines on U+0009. The first subitem is the pointer (as a decimal number) and the second is the corresponding code point (as a hexadecimal number). Other subitems are not relevant.

The index code point for pointer in index is the code point corresponding to pointer in index, or null if pointer is not in index.

The index pointer for code point in index is the first pointer corresponding to code point in index, or null if code point is not in index.

These are the indexes defined by this specification, excluding index single-byte:

The index gb18030 ranges code point for pointer is the return value of these steps:

1. If pointer is greater than 39419 and less than 189000, or pointer is greater than 1237575, return null.

2. Let offset be the last pointer in index gb18030 ranges that is equal to or less than pointer and let code point offset be its corresponding code point.

3. Return a code point whose value is code point offset + pointer − offset.

The index gb18030 ranges pointer for code point is the return value of these steps:

1. Let offset be the last code point in index gb18030 ranges that is equal to or less than code point and let pointer offset be its corresponding pointer.

2. Return a pointer whose value is pointer offset + code point − offset.

The index shift_jis pointer for code point is the return value of these steps:

1. Let index be index jis0208 excluding all pointers in the range 8272 to 8835.

2. Return the index pointer for code point in index.

All indexes are also available as non-normative indexes.json resource. (index gb18030 ranges has a slightly different format here, to be able to represent ranges.)

6 Specification hooks

The algorithms decode, utf-8 decode, utf-8 decode without BOM, encode, and utf-8 encode are intended for usage by other specifications. utf-8 decode is to be used by new formats. The get an encoding algorithm can be used first to turn a label into an encoding.

To decode a byte stream stream using fallback encoding encoding, run these steps:

1. Let buffer be an empty byte sequence.

2. Let BOM seen flag be unset.

3. Read bytes from stream into buffer until either buffer contains three bytes or read returns end-of-stream.

4. For each of the rows in the table below, starting with the first one and going down, if the first bytes of buffer match all the bytes given in the first column, then set encoding to the encoding given in the cell in the second column of that row and set BOM seen flag.

   Byte order mark	Encoding
   0xEF 0xBB 0xBF	utf-8
   0xFE 0xFF	utf-16be
   0xFF 0xFE	utf-16le

   For compatibility with deployed content, the byte order mark (also known as BOM) is considered more authoritative than anything else.

5. If BOM seen flag is unset prepend buffer to stream.

6. Otherwise, if BOM seen flag is set, encoding is not utf-8, and buffer contains three bytes, prepend the last byte of buffer to stream.

7. Let output be a code point stream.

8. Run encoding's decoder with stream and output.

9. Return output.

To utf-8 decode a byte stream stream, run these steps:

1. Let buffer be an empty byte sequence.

2. Read three bytes from stream into buffer.

3. If buffer does not match 0xEF 0xBB 0xBF, prepend buffer to stream.

4. Let output be a code point stream.

5. Run utf-8's decoder with stream and output.

6. Return output.

To utf-8 decode without BOM a byte stream stream, run these steps:

1. Let output be a code point stream.

2. Run utf-8's decoder with stream and output.

3. Return output.

To encode a code point stream stream using encoding encoding, run these steps:

1. Let output be a byte stream.

2. Run encoding's encoder with stream, output, and HTML.

3. Return output.

This is mostly a legacy hook for URLs and HTML forms. Layering utf-8 encode on top is safe as it never triggers errors. [URL] [HTML]

To utf-8 encode a code point stream stream, return the result of encoding stream using encoding utf-8.

If the input to encode or utf-8 encode stems from a DOMString, the convert a DOMString to a sequence of Unicode characters from Web IDL is to be used first.

7 API

This section uses terminology from the DOM, Typed Arrays, and Web IDL. Non-browser implementations are not required to implement this API. [DOM] [TYPEDARRAY] [WEBIDL]

function encodeArrayOfStrings(strings, encoding) {
  var encoder, encoded, len, i, bytes, view, offset;

  encoder = new TextEncoder(encoding);
  encoded = [];

  len = Uint32Array.BYTES_PER_ELEMENT;
  for (i = 0; i < strings.length; i += 1) {
    len += Uint32Array.BYTES_PER_ELEMENT;
    encoded[i] = new TextEncoder(encoding).encode(strings[i]);
    len += encoded[i].byteLength;
  }

  bytes = new Uint8Array(len);
  view = new DataView(bytes.buffer);
  offset = 0;

  view.setUint32(offset, strings.length);
  offset += Uint32Array.BYTES_PER_ELEMENT;
  for (i = 0; i < encoded.length; i += 1) {
    len = encoded[i].byteLength;
    view.setUint32(offset, len);
    offset += Uint32Array.BYTES_PER_ELEMENT;
    bytes.set(encoded[i], offset);
    offset += len;
  }
  return bytes.buffer;
}

function decodeArrayOfStrings(buffer, encoding) {
  var decoder, view, offset, num_strings, strings, i, len;

  decoder = new TextDecoder(encoding);
  view = new DataView(buffer);
  offset = 0;
  strings = [];

  num_strings = view.getUint32(offset);
  offset += Uint32Array.BYTES_PER_ELEMENT;
  for (i = 0; i < num_strings; i += 1) {
    len = view.getUint32(offset);
    offset += Uint32Array.BYTES_PER_ELEMENT;
    strings[i] = decoder.decode(
      new DataView(view.buffer, offset, len));
    offset += len;
  }
  return strings;
}

7.1 Type ScalarValueString

This is a temporary definition until IDL proper is updated to include it. typedef is used for now to keep IDL validity.

typedef DOMString ScalarValueString;

ScalarValueString is identical to DOMString except that convert a DOMString to a sequence of Unicode characters is used subsequently when converting to an IDL value. Ergo, the IDL value is a sequence of scalar values.

Only use ScalarValueString if the subsystem deals in scalar values rather than code points. When in doubt, use DOMString.

7.2 Interface TextDecoder

dictionary TextDecoderOptions {
  boolean fatal = false;
  boolean ignoreBOM = false;
};

dictionary TextDecodeOptions {
  boolean stream = false;
};

[Constructor(optional DOMString label = "utf-8", optional TextDecoderOptions options),
 Exposed=Window,Worker]
interface TextDecoder {
  readonly attribute DOMString encoding;
  readonly attribute boolean fatal;
  readonly attribute boolean ignoreBOM;
  DOMString decode();
  DOMString decode(ArrayBufferView input, optional TextDecodeOptions options);
};

A TextDecoder object has an associated encoding, decoder, stream, ignore BOM flag (initially unset), BOM seen flag (initially unset), error mode (initially replacement), and streaming flag (initially unset).

A TextDecoder object also has an associated serialize stream algorithm, that given a stream stream, runs these steps:

1. Let output be the empty string.

2. While true:

   1. Let token be the result of reading from stream.

   2. If encoding is one of utf-8, utf-16be, and utf-16le, and ignore BOM flag and BOM seen flag are unset, run these subsubsteps:

      1. If token is U+FEFF, set BOM seen flag.

      2. Otherwise, if token is not end-of-stream, set BOM seen flag and append token to output.

      3. Otherwise, return output.

   3. Otherwise, if token is not end-of-stream, append token to output.

   4. Otherwise, return output.

This algorithm is intentionally different with respect to BOM handling from the decode algorithm used by the rest of the platform to give API users more control.

decoder = new TextDecoder([label = "utf-8" [, options]])

Returns a new TextDecoder object.

If label is either not a label or is a label for replacement, throws a TypeError.

decoder . encoding

Returns encoding's name.

fatal . fatal

Returns true if error mode is fatal, and false otherwise.

ignoreBOM . ignoreBOM

Returns true if ignore BOM flag is set, and false otherwise.

decoder . decode([input [, options]])

Returns the result of running encoding's decoder. If options's stream is set to true the method can be invoked multiple times to process a fragmented stream.

If the error mode is fatal set and encoding's decoder returns error, throws an "EncodingError".

The TextDecoder(label, options) constructor must run these steps:

1. Let encoding be the result of getting an encoding from label.

2. If encoding is failure or replacement, throw a TypeError.

3. Let dec be a new TextDecoder object.

4. Set dec's encoding to encoding.

5. If options's fatal member is true, set dec's error mode to fatal.

6. If options's ignoreBOM member is true, set dec's ignore BOM flag.

7. Return dec.

The encoding attribute must return encoding's name.

The fatal attribute must return true if error mode is fatal, and false otherwise.

The ignoreBOM attribute must return true if ignore BOM flag is set, and false otherwise.

The decode(input, options) method must run these steps:

1. If the streaming flag is unset, set decoder to a new encoding's decoder, set stream to a new stream, and unset the BOM seen flag.

2. If options's stream is true, set the streaming flag, and unset the streaming flag otherwise.

3. If input is given, then given input's buffer, byteOffset, and byteLength, push byteLength bytes from buffer, starting at byteOffset, to stream.

4. Let output be a new stream.

5. While true:

   1. Let token be the result of reading from stream.

   2. If token is end-of-stream and the streaming flag is set, return output, serialized.

   3. Otherwise, run these subsubsteps:

      1. Let result be the result of processing token for decoder, stream, output, and error mode.

      2. If result is finished, return output, serialized.

      3. Otherwise, if result is error, throw an "EncodingError"

      4. Otherwise, do nothing.

7.3 Interface TextEncoder

dictionary TextEncodeOptions {
  boolean stream = false;
};

[Constructor(optional DOMString utfLabel = "utf-8"),
 Exposed=Window,Worker]
interface TextEncoder {
  readonly attribute DOMString encoding;
  Uint8Array encode(optional ScalarValueString input = "", optional TextEncodeOptions options);
};

A TextEncoder object has an associated encoding, encoder, stream, and streaming flag (initially unset).

A TextEncoder object also has an associated serialize stream algorithm, that given a stream stream, runs these steps:

1. Let buffer be a byte sequence.

2. Keep reading from stream into buffer until end-of-stream.

3. Return a Uint8Array object wrapping an ArrayBuffer containing bytes.

encoder = new TextEncoder([utfLabel = "utf-8"])

Returns a new TextEncoder object.

If utfLabel is not a label for utf-8, utf-16be, or utf-16le, throws a TypeError.

encoder . encoding

Returns encoding's name.

encoder . encode([input [, options]])

Returns the result of running encoding's encoder. If options's stream is set to true, the method can be invoked multiple times to process a fragmented stream.

The TextEncoder(utfLabel) constructor must run these steps:

1. Let encoding be the result of getting an encoding from utfLabel.

2. If encoding is failure, or is none of utf-8, utf-16be, and utf-16le, throw a TypeError.

3. Let enc be a new TextEncoder object.

4. Set enc's encoding to encoding.

5. Return enc.

The encoding attribute must return encoding's name.

The encode(input, options) method must run these steps:

1. If the streaming flag is unset, set encoder to a new encoding's encoder and set stream to a new stream.

2. If options's stream is true, set the streaming flag, and unset the streaming flag otherwise.

3. Push input to stream.

4. Let output be a new stream.

5. While true:

   1. Let token be the result of reading from stream.

   2. If token is end-of-stream and the streaming flag is set, return output, serialized.

   3. Otherwise, run these subsubsteps:

      1. Let result be the result of processing token for encoder, stream, output.

      2. If result is finished, return output, serialized.

      3. Otherwise, do nothing.

      These encodings cannot return error.

8 The encoding

8.1 utf-8

8.1.1 utf-8 decoder

utf-8's decoder's has an associated utf-8 code point, utf-8 bytes seen, and utf-8 bytes needed (all initially 0), a utf-8 lower boundary (initially 0x80), and a utf-8 upper boundary (initially 0xBF).

utf-8's decoder's handler, given a stream and byte, runs these steps:

1. If byte is end-of-stream and utf-8 bytes needed is not 0, set utf-8 bytes needed to 0 and return error.

2. If byte is end-of-stream, return finished.

3. If utf-8 bytes needed is 0, based on byte:

   0x00 to 0x7F

   Return a code point whose value is byte.

   0xC2 to 0xDF

   Set utf-8 bytes needed to 1 and utf-8 code point to byte − 0xC0.

   0xE0 to 0xEF

   1. If byte is 0xE0, set utf-8 lower boundary to 0xA0.

   2. If byte is 0xED, set utf-8 upper boundary to 0x9F.

   3. Set utf-8 bytes needed to 2 and utf-8 code point to byte − 0xE0.

   0xF0 to 0xF4

   1. If byte is 0xF0, set utf-8 lower boundary to 0x90.

   2. If byte is 0xF4, set utf-8 upper boundary to 0x8F.

   3. Set utf-8 bytes needed to 3 and utf-8 code point to byte − 0xF0.

   Otherwise

   Return error.

   Then (byte is in the range 0xC2 to 0xF4) set utf-8 code point to utf-8 code point << (6 × utf-8 bytes needed) and return continue.

4. If byte is not in the range utf-8 lower boundary to utf-8 upper boundary, run these substeps:

   1. Set utf-8 code point, utf-8 bytes needed, and utf-8 bytes seen to 0, set utf-8 lower boundary to 0x80, and set utf-8 upper boundary to 0xBF.

   2. Prepend byte to stream.

   3. Return error.

5. Set utf-8 lower boundary to 0x80 and utf-8 upper boundary to 0xBF.

6. Increase utf-8 bytes seen by one and set utf-8 code point to utf-8 code point + (byte − 0x80) << (6 × (utf-8 bytes needed − utf-8 bytes seen)).

7. If utf-8 bytes seen is not equal to utf-8 bytes needed, continue.

8. Let code point be utf-8 code point.

9. Set utf-8 code point, utf-8 bytes needed, and utf-8 bytes seen to 0.

10. Emit a code point whose value is code point.

The constraints in the utf-8 decoder above match “Best Practices for Using U+FFFD” from the Unicode standard. No other behavior is permitted per the Encoding Standard (other algorithms that achieve the same result are obviously fine, even encouraged).

8.1.2 utf-8 encoder

utf-8's encoder's handler, given a stream and code point, runs these steps:

1. If code point is end-of-stream, return finished.

2. If code point is in the range U+0000 to U+007F, return a byte whose value is code point.

3. Set count and offset based on the range code point is in:

   U+0080 to U+07FF

   1 and 0xC0

   U+0800 to U+FFFF

   2 and 0xE0

   U+10000 to U+10FFFF

   3 and 0xF0

4. Let bytes be a byte sequence whose first byte is (code point >> (6 × count)) + offset.

5. Run these substeps while count is greater than 0:

   1. Set temp to code point >> (6 × (count − 1)).

   2. Append to bytes 0x80 | (temp & 0x3F).

   3. Decrease count by one.

6. Return bytes bytes, in order.

9 Legacy single-byte encodings

An encoding where each byte is either a single code point or nothing, is a single-byte encoding. Single-byte encodings share the decoder and encoder. Index single-byte, as referenced by the single-byte decoder and single-byte encoder, is defined by the following table, and depends on the single-byte encoding in use. All but two single-byte encodings have a unique index.

iso-8859-8 and iso-8859-8-i are distinct encoding names, because iso-8859-8 has influence on the layout direction. And although historically this might have been the case for iso-8859-6 and "iso-8859-6-i" as well, that is no longer true.

9.1 single-byte decoder

Single-byte encodings's decoder's handler, given a stream and byte, runs these steps:

1. If byte is end-of-stream, return finished.

2. If byte is in the range 0x00 to 0x7F, return a code point whose value is byte.

3. Let code point be the index code point for byte − 0x80 in index single-byte.

4. If code point is null, return error.

5. Return a code point whose value is code point.

9.2 single-byte encoder

Single-byte encodings's encoder's handler, given a stream and code point, runs these steps:

1. If code point is end-of-stream, return finished.

2. If code point is in the range U+0000 to U+007F, return a byte whose value is code point.

3. Let pointer be the index pointer for code point in index single-byte.

4. If pointer is null, return error with code point.

5. Return a byte whose value is pointer + 0x80.

10 Legacy multi-byte Chinese (simplified) encodings

10.1 gb18030

10.1.1 gb18030 decoder

gb18030's decoder has an associated gb18030 first, gb18030 second, and gb18030 third (all initially 0x00).

gb18030's decoder's handler, given a stream and byte, runs these steps:

1. If byte is end-of-stream and gb18030 first, gb18030 second, and gb18030 third are 0x00, return finished.

2. If byte is end-of-stream, and gb18030 first, gb18030 second, or gb18030 third is not 0x00, set gb18030 first, gb18030 second, and gb18030 third to 0x00, and return error.

3. If gb18030 third is not 0x00, run these substeps:

   1. Let code point be null.

   2. If byte is in the range 0x30 to 0x39, set code point to the index gb18030 ranges code point for (((gb18030 first − 0x81) × 10 + gb18030 second − 0x30) × 126 + gb18030 third − 0x81) × 10 + byte − 0x30.

   3. Let buffer be a byte sequence consisting of gb18030 second, gb18030 third, and byte, in order.

   4. Set gb18030 first, gb18030 second, and gb18030 third to 0x00.

   5. If code point is null, prepend buffer to stream and return error.

   6. Return a code point whose value is code point.

4. If gb18030 second is not 0x00, run these substeps:

   1. If byte is in the range 0x81 to 0xFE, set gb18030 third to byte and return continue.

   2. Prepend gb18030 second followed by byte to stream, set gb18030 first and gb18030 second to 0x00, and return error.

5. If gb18030 first is not 0x00, run these substeps:

   1. If byte is in the range 0x30 to 0x39, set gb18030 second to byte and return continue.

   2. Let lead be gb18030 first, let pointer be null, and set gb18030 first to 0x00.

   3. Let offset be 0x40 if byte is less than 0x7F and 0x41 otherwise.

   4. If byte is in the range 0x40 to 0x7E or 0x80 to 0xFE, set pointer to (lead − 0x81) × 190 + (byte − offset).

   5. Let code point be null if pointer is null and the index code point for pointer in index gb18030 otherwise.

   6. If pointer is null, prepend byte to stream.

   7. If code point is null, return error.

   8. Return a code point whose value is code point.

6. If byte is in the range 0x00 to 0x7F, return a code point whose value is byte.

7. If byte is 0x80, return code point U+20AC.

8. If byte is in the range 0x81 to 0xFE, set gb18030 first to byte and return continue.

9. Return error.

10.1.2 gb18030 encoder

gb18030's encoder's handler, given a stream and code point, runs these steps:

1. If code point is end-of-stream, return finished.

2. If code point is in the range U+0000 to U+007F, return a byte whose value is code point.

3. Let pointer be the index pointer for code point in index gb18030.

4. If pointer is not null, run these substeps:

   1. Let lead be pointer / 190 + 0x81.

   2. Let trail be pointer % 190.

   3. Let offset be 0x40 if trail is less than 0x3F and 0x41 otherwise.

   4. Return two bytes whose values are lead and trail + offset.

5. Set pointer to the index gb18030 ranges pointer for code point.

6. Let byte1 be pointer / 10 / 126 / 10.

7. Set pointer to pointer − byte1 × 10 × 126 × 10.

8. Let byte2 be pointer / 10 / 126.

9. Set pointer to pointer − byte2 × 10 × 126.

10. Let byte3 be pointer / 10.

11. Let byte4 be pointer − byte3 × 10.

12. Return four bytes whose values are byte1 + 0x81, byte2 + 0x30, byte3 + 0x81, byte4 + 0x30.

10.2 hz-gb-2312

This encoding is considered for removal.

10.2.1 hz-gb-2312 decoder

hz-gb-2312's decoder has an associated hz-gb-2312 flag (initially unset) and hz-gb-2312 lead (initially 0x00).

hz-gb-2312's decoder's handler, given a stream and byte, runs these steps:

1. If byte is end-of-stream and hz-gb-2312 lead is not 0x00, set hz-gb-2312 lead to 0x00 and return error.

2. If byte is end-of-stream and hz-gb-2312 lead is 0x00, return finished.

3. If hz-gb-2312 lead is 0x7E, set hz-gb-2312 lead to 0x00, and based on byte:

   0x7B

   Set the hz-gb-2312 flag and return continue.

   0x7D

   Unset the hz-gb-2312 flag and return continue.

   0x7E

   Return code point U+007E.

   0x0A

   Return continue.

   Otherwise

   Prepend byte to stream and return error.

4. If hz-gb-2312 lead is not 0x00, let lead be hz-gb-2312 lead, set hz-gb-2312 lead to 0x00, and then run these substeps:

   1. If byte is in the range 0x21 to 0x7E, let code point be the index code point for (lead − 1) × 190 + (byte + 0x3F) in index gb18030.

   2. If byte is 0x0A, unset the hz-gb-2312 flag.

   3. If code point is null, return error.

   4. Return a code point whose value is code point.

5. If byte is 0x7E, set hz-gb-2312 lead to 0x7E and return continue.

6. If the hz-gb-2312 flag is set:

   1. If byte is in the range 0x20 to 0x7F, set hz-gb-2312 lead to byte and return continue.

   2. If byte is 0x0A, unset the hz-gb-2312 flag.

   3. Return error.

7. If byte is in the range 0x00 to 0x7F, return a code point whose value is byte.

8. Return error.

10.2.2 hz-gb-2312 encoder

hz-gb-2312's encoder has an associated hz-gb-2312 flag.

hz-gb-2312's encoder's handler, given a stream and code point, runs these steps:

1. If code point is end-of-stream, return finished.

2. If code point is in the range U+0000 to U+007F and the hz-gb-2312 flag is set, prepend code point to stream, unset the hz-gb-2312 flag, and return two bytes 0x7E 0x7D.

3. If code point is 0x007E, return two bytes 0x7E 0x7E.

4. If code point is in the range U+0000 to U+007F, return a byte whose value is code point.

5. Let pointer be the index pointer for code point in index gb18030.

6. If pointer is null, return error with code point.

7. If the hz-gb-2312 flag is unset, prepend code point to stream, set the hz-gb-2312 flag, and return two bytes 0x7E 0x7B.

8. Let lead be pointer / 190 + 1.

9. Let trail be pointer % 190 − 0x3F.

10. If either lead or trail is less than 0x21, return error with code point.

11. Return two bytes whose values are lead and trail.

11 Legacy multi-byte Chinese (traditional) encodings

11.1 big5

11.1.1 big5 decoder

big5's decoder has an associated big5 lead (initially 0x00). big5's decoder's handler, given a stream and byte, runs these steps:

1. If byte is end-of-stream and big5 lead is not 0x00, set big5 lead to 0x00 and return error.

2. If byte is end-of-stream and big5 lead is 0x00, return finished.

3. If big5 lead is not 0x00, let lead be big5 lead, let pointer be null, set big5 lead to 0x00, and then run these substeps:

   1. Let offset be 0x40 if byte is less than 0x7F and 0x62 otherwise.

   2. If byte is in the range 0x40 to 0x7E or 0xA1 to 0xFE, set pointer to (lead − 0x81) × 157 + (byte − offset).

   3. If there is a row in the table below whose first column is pointer, return the two code points listed in its second column (the third column is irrelevant):

      Pointer	Code points	Notes
      1133	U+00CA U+0304	Ê̄ (LATIN CAPITAL LETTER E WITH CIRCUMFLEX AND MACRON)
      1135	U+00CA U+030C	Ê̌ (LATIN CAPITAL LETTER E WITH CIRCUMFLEX AND CARON)
      1164	U+00EA U+0304	ê̄ (LATIN SMALL LETTER E WITH CIRCUMFLEX AND MACRON)
      1166	U+00EA U+030C	ê̌ (LATIN SMALL LETTER E WITH CIRCUMFLEX AND CARON)

      Since indexes are limited to single code points this table is used for these pointers.

   4. Let code point be null if pointer is null and the index code point for pointer in index big5 otherwise.

   5. If pointer is null and byte is in the range 0x00 to 0x7F, prepend byte to stream.

   6. If code point is null, return error.

   7. Return a code point whose value is code point.

4. If byte is in the range 0x00 to 0x7F, return a code point whose value is byte.

5. If byte is in the range 0x81 to 0xFE, set big5 lead to byte and return continue.

6. Return error.

11.1.2 big5 encoder

big5's encoder's handler, given a stream and code point, runs these steps:

1. If code point is end-of-stream, return finished.

2. If code point is in the range U+0000 to U+007F, return a byte whose value is code point.

3. Let pointer be the index pointer for code point in index big5.

4. If pointer is null, return error with code point.

5. Let lead be pointer / 157 + 0x81.

6. If lead is less than 0xA1, return error with code point.

   Avoid returning Hong Kong Supplementary Character Set extensions literally.

7. Let trail be pointer % 157.

8. Let offset be 0x40 if trail is less than 0x3F and 0x62 otherwise.

9. Return two bytes whose values are lead and trail + offset.

12 Legacy multi-byte Japanese encodings

12.1 euc-jp

12.1.1 euc-jp decoder

euc-jp's decoder has an associated euc-jp jis0212 flag (initially unset) and euc-jp lead (initially 0x00).

euc-jp's decoder's handler, given a stream and byte, runs these steps:

1. If byte is end-of-stream and euc-jp lead is not 0x00, set euc-jp lead to 0x00, and return error.

2. If byte is end-of-stream and euc-jp lead is 0x00, return finished.

3. If euc-jp lead is 0x8E and byte is in the range 0xA1 to 0xDF, set euc-jp lead to 0x00 and return a code point whose value is 0xFF61 + byte − 0xA1.

4. If euc-jp lead is 0x8F and byte is in the range 0xA1 to 0xFE, set the euc-jp jis0212 flag, set euc-jp lead to byte, and return continue.

5. If euc-jp lead is not 0x00, let lead be euc-jp lead, set euc-jp lead to 0x00, and run these substeps:

   1. Let code point be null.

   2. If lead and byte are both in the range 0xA1 to 0xFE, set code point to the index code point for (lead − 0xA1) × 94 + byte − 0xA1 in index jis0208 if the euc-jp jis0212 flag is unset and in index jis0212 otherwise.

   3. Unset the euc-jp jis0212 flag.

   4. If byte is not in the range 0xA1 to 0xFE, prepend byte to stream.

   5. If code point is null, return error.

   6. Return a code point whose value is code point.

6. If byte is in the range 0x00 to 0x7F, return a code point whose value is byte.

7. If byte is 0x8E, 0x8F, or in the range 0xA1 to 0xFE, set euc-jp lead to byte and return continue.

8. Return error.

12.1.2 euc-jp encoder

euc-jp's encoder's handler, given a stream and code point, runs these steps:

1. If code point is end-of-stream, return finished.

2. If code point is in the range U+0000 to U+007F, return a byte whose value is code point.

3. If code point is U+00A5, return byte 0x5C.

4. If code point is U+203E, return byte 0x7E.

5. If code point is in the range U+FF61 to U+FF9F, return two bytes whose values are 0x8E and code point − 0xFF61 + 0xA1.

6. Let pointer be the index pointer for code point in index jis0208.

7. If pointer is null, return error with code point.

8. Let lead be pointer / 94 + 0xA1.

9. Let trail be pointer % 94 + 0xA1.

10. Return two bytes whose values are lead and trail.

The index jis0212 is not used by the euc-jp encoder due to lack of widespread support.

12.2 iso-2022-jp

12.2.1 iso-2022-jp decoder

iso-2022-jp's decoder has an associated iso-2022-jp state (initially ASCII state), iso-2022-jp jis0212 flag (initially unset), and iso-2022-jp lead (initially 0x00).

iso-2022-jp's decoder's handler, given a stream and byte, runs these steps, switching on iso-2022-jp state:

ASCII state

Based on byte:

0x1B

Set iso-2022-jp state to escape start state and return continue.

0x00 to 0x7F

Return a code point whose value is byte.

end-of-stream

Return finished.

Otherwise

Return error.

Escape start state

1. If byte is either 0x24 or 0x28, set iso-2022-jp lead to byte, iso-2022-jp state to escape middle state, and return continue.

2. If byte is not end-of-stream, prepend byte to stream.

3. Set iso-2022-jp state to ASCII state and return error.

Escape middle state

1. Let lead be iso-2022-jp lead and set iso-2022-jp lead to 0x00.

2. If lead is 0x24 and byte is either 0x40 or 0x42, unset the iso-2022-jp jis0212 flag, set iso-2022-jp state to lead state, and return continue.

3. If lead is 0x24 and byte is 0x28, set iso-2022-jp state to escape final state and return continue.

4. If lead is 0x28 and byte is either 0x42 or 0x4A, set iso-2022-jp state to ASCII state and return continue.

5. If lead is 0x28 and byte is 0x49, set iso-2022-jp state to Katakana state and return continue.

6. Let buffer be byte if byte is end-of-stream, and two bytes lead byte otherwise.

7. Prepend buffer to stream.

8. Set iso-2022-jp state to ASCII state and return error.

Escape final state

1. If byte is 0x44, set the iso-2022-jp jis0212 flag, set iso-2022-jp state to lead state, and return continue.

2. Let buffer be two bytes 0x28 byte, if byte is end-of-stream, and three bytes 0x24 0x28 byte otherwise.

3. Prepend buffer to stream.

4. Set iso-2022-jp state to ASCII state and return error.

Lead state

Based on byte:

0x0A

Set iso-2022-jp state to ASCII state and return code point U+000A.

0x1B

Set iso-2022-jp state to escape start state and return continue.

end-of-stream

Return finished.

Otherwise

Set iso-2022-jp lead to byte, iso-2022-jp state to trail state, and return continue.

Trail state

1. Set the iso-2022-jp state to lead state.

2. If byte is end-of-stream, return error.

3. Let code point be null and let pointer be (iso-2022-jp lead − 0x21) × 94 + byte − 0x21.

4. If iso-2022-jp lead and byte are both in the range 0x21 to 0x7E, set code point to the index code point for pointer in index jis0208, if the iso-2022-jp jis0212 flag is unset, and in index jis0212 otherwise.

5. If code point is null, return error.

6. Return a code point whose value is code point.

Katakana state

Based on byte:

0x1B

Set iso-2022-jp state to escape start state and return continue.

0x21 to 0x5F

Return a code point whose value is 0xFF61 + byte − 0x21.

end-of-stream

Return finished.

Otherwise

Return error.

12.2.2 iso-2022-jp encoder

iso-2022-jp's encoder has an associated iso-2022-jp state.

iso-2022-jp's encoder's handler, given a stream and code point, runs these steps:

1. If code point is end-of-stream, return finished.

2. If code point is in the range U+0000 to U+007F, or is U+00A5 or U+203E, and iso-2022-jp state is not ASCII state, prepend code point to stream, set iso-2022-jp state to ASCII state, and return three bytes 0x1B 0x28 0x42.

3. If code point is in the range U+0000 to U+007F, return a byte whose value is code point.

4. If code point is U+00A5, return byte 0x5C.

5. If code point is U+203E, return byte 0x7E.

6. If code point is in the range U+FF61 to U+FF9F and iso-2022-jp state is not Katakana state, prepend code point to stream, set iso-2022-jp state to Katakana state, and return three bytes 0x1B 0x28 0x49.

7. If code point is in the range U+FF61 to U+FF9F, return a byte whose value is code point − 0xFF61 + 0x21.

8. Let pointer be the index pointer for code point in index jis0208.

9. If pointer is null, return error with code point.

10. If iso-2022-jp state is not lead state, prepend code point to stream, set iso-2022-jp state to lead state, and return three bytes 0x1B 0x24 0x42.

11. Let lead be pointer / 94 + 0x21.

12. Let trail be pointer % 94 + 0x21.

13. Return two bytes whose values are lead and trail.

The index jis0212 is not used by the iso-2022-jp encoder due to lack of widespread support.

12.3 shift_jis

12.3.1 shift_jis decoder

shift_jis's decoder has an associated shift_jis lead (initially 0x00).

shift_jis's decoder's handler, given a stream and byte, runs these steps:

1. If byte is end-of-stream, shift_jis lead is not 0x00, set shift_jis lead to 0x00 and return error.

2. If byte is end-of-stream and shift_jis lead is 0x00, return finished.

3. If shift_jis lead is not 0x00, let lead be shift_jis lead, let pointer be null, set shift_jis lead to 0x00, and then run these substeps:

   1. Let offset be 0x40, if byte is less than 0x7F, and 0x41 otherwise.

   2. Let lead offset be 0x81, if lead is less than 0xA0, and 0xC1 otherwise.

   3. If byte is in the range 0x40 to 0x7E or 0x80 to 0xFC, set pointer to (lead − lead offset) × 188 + byte − offset.

   4. Let code point be null, if pointer is null, and the index code point for pointer in index jis0208 otherwise.

   5. If code point is null and pointer is in the range 8836 to 10528, return a code point whose value is 0xE000 + pointer − 8836.

      This is interoperable legacy from Windows known as EUDC.

   6. If pointer is null, prepend byte to stream.

   7. If code point is null, return error.

   8. Return a code point whose value is code point.

4. If byte is in the range 0x00 to 0x80, return a code point whose value is byte.

5. If byte is in the range 0xA1 to 0xDF, return a code point whose value is 0xFF61 + byte − 0xA1.

6. If byte is in the range 0x81 to 0x9F or 0xE0 to 0xFC, set shift_jis lead to byte and return continue.

7. Return error.

12.3.2 shift_jis encoder

shift_jis's encoder's handler, given a stream and code point, runs these steps:

1. If code point is end-of-stream, return finished.

2. If code point is in the range U+0000 to U+0080, return a byte whose value is code point.

3. If code point is U+00A5, return byte 0x5C.

4. If code point is U+203E, return byte 0x7E.

5. If code point is in the range U+FF61 to U+FF9F, return a byte whose value is code point − 0xFF61 + 0xA1.

6. Let pointer be the index shift_jis pointer for code point.

7. If pointer is null, return error with code point.

8. Let lead be pointer / 188.

9. Let lead offset be 0x81, if lead is less than 0x1F, and 0xC1 otherwise.

10. Let trail be pointer % 188.

11. Let offset be 0x40, if trail is less than 0x3F, and 0x41 otherwise.

12. Return two bytes whose values are lead + lead offset and trail + offset.

13 Legacy multi-byte Korean encodings

13.1 euc-kr

13.1.1 euc-kr decoder

euc-kr's decoder has an associated euc-kr lead (initially 0x00).

euc-kr's decoder's handler, given a stream and byte, runs these steps:

1. If byte is end-of-stream and euc-kr lead is not 0x00, set euc-kr lead to 0x00 and return error.

2. If byte is end-of-stream and euc-kr lead is 0x00, return finished.

3. If euc-kr lead is not 0x00, let lead be euc-kr lead, let pointer be null, set euc-kr lead to 0x00, and then run these substeps:

   1. If lead is in the range 0x81 to 0xC6, let temp be (26 + 26 + 126) × (lead − 0x81), and then set pointer to the result of the equation below, depending on byte:

      0x41 to 0x5A

      temp + byte − 0x41

      0x61 to 0x7A

      temp + 26 + byte − 0x61

      0x81 to 0xFE

      temp + 26 + 26 + byte − 0x81

   2. If lead is in the range 0xC7 to 0xFE and byte is in the range 0xA1 to 0xFE, set pointer to (26 + 26 + 126) × (0xC7 − 0x81) + (lead − 0xC7) × 94 + (byte − 0xA1).

   3. Let code point be null, if pointer is null, and the index code point for pointer in index euc-kr otherwise.

   4. If pointer is null, prepend byte to stream.

   5. If code point is null, return error.

   6. Return a code point whose value is code point.

4. If byte is in the range 0x00 to 0x7F, return a code point whose value is byte.

5. If byte is in the range 0x81 to 0xFE, set euc-kr lead to byte and return continue.

6. Return error.

13.1.2 euc-kr encoder

euc-kr's encoder's handler, given a stream and code point, runs these steps:

1. If code point is end-of-stream, return finished.

2. If code point is in the range U+0000 to U+007F, return a byte whose value is code point.

3. Let pointer be the index pointer for code point in index euc-kr.

4. If pointer is null, return error with code point.

5. If pointer is less than (26 + 26 + 126) × (0xC7 − 0x81), run these substeps:

   1. Let lead be pointer / (26 + 26 + 126) + 0x81.

   2. Let trail be pointer % (26 + 26 + 126).

   3. Let offset be 0x41 if trail is less than 26, 0x47 if trail is less than 26 + 26, and 0x4D otherwise.

   4. Return two bytes whose values are lead and trail + offset.

6. Set pointer to pointer − (26 + 26 + 126) × (0xC7 − 0x81).

7. Let lead be pointer / 94 + 0xC7.

8. Let trail be pointer % 94 + 0xA1.

9. Return two bytes whose values are lead and trail.

14 Legacy miscellaneous encodings

14.1 replacement

The replacement encoding exists to prevent certain attacks that abuse a mismatch between encodings supported on the server and the client.

14.1.1 replacement decoder

replacement's decoder has an associated replacement error returned flag (initially unset).

replacement's decoder's handler, given a stream and byte, runs these steps:

1. If replacement error returned flag is unset, set the replacement error returned flag and return error.

2. Return finished.

14.1.2 replacement encoder

replacement's encoder is utf-8's encoder.

14.2 Common infrastructure for utf-16be and utf-16le

In violation of the Unicode standard, which does not allow for handling a byte order mark in its definition of utf-16be and utf-16le, checking and using a byte order mark happens before an encoding to decode a byte stream is chosen, as seen in the decode algorithm.

To convert a code unit to bytes using a utf-16be flag, run these steps:

1. Let byte1 be code unit >> 8.

2. Let byte2 be code unit & 0x00FF.

3. Then return the bytes in order:

   utf-16be flag is set

   byte1, then byte2.

   utf-16be flag is unset

   byte2, then byte1.

14.2.1 shared utf-16 decoder

shared utf-16 decoder has an associated utf-16 lead byte and utf-16 lead surrogate (both initially null), and utf-16be flag (no initial value).

shared utf-16 decoder's handler, given a stream and byte, runs these steps:

1. If byte is end-of-stream and either utf-16 lead byte or utf-16 lead surrogate is not null, set utf-16 lead byte and utf-16 lead surrogate to null, and return error.

2. If byte is end-of-stream and utf-16 lead byte and utf-16 lead surrogate are null, return finished.

3. If utf-16 lead byte is null, set utf-16 lead byte to byte and return continue.

4. Let code unit be the result of:

   utf-16be flag is set

   (utf-16 lead byte << 8) + byte.

   utf-16be flag is unset

   (byte << 8) + utf-16 lead byte.

   Then set utf-16 lead byte to null.

5. If utf-16 lead surrogate is not null, let lead surrogate be utf-16 lead surrogate, set utf-16 lead surrogate to null, and then run these substeps:

   1. If code unit is in the range U+DC00 to U+DFFF, return a code point whose value is 0x10000 + ((lead surrogate − 0xD800) << 10) + (code unit − 0xDC00).

   2. Prepend the sequence resulting of converting code unit to bytes using utf-16be flag to stream and return error.

6. If code unit is in the range U+D800 to U+DBFF, set utf-16 lead surrogate to code unit and return continue.

7. If code unit is in the range U+DC00 to U+DFFF, return error.

8. Return code point code unit.

14.2.2 shared utf-16 encoder

shared utf-16 encoder has an associated utf-16be flag.

shared utf-16 encoder's handler, given a stream and code point, runs these steps:

1. If code point is end-of-stream, return finished.

2. If code point is in the range 0x00 to 0xFFFF, return the sequence resulting of converting code point to bytes using utf-16be flag.

3. Let lead be ((code point − 0x10000) >> 10) + 0xD800, converted to bytes using utf-16be flag.

4. Let trail be ((code point − 0x10000) & 0x3FF) + 0xDC00, converted to bytes using utf-16be flag.

5. Return a byte sequence of lead followed by trail.

14.3 utf-16be

14.3.1 utf-16be decoder

utf-16be's decoder is shared utf-16 decoder with its utf-16be flag set.

14.3.2 utf-16be encoder

utf-16be's encoder is shared utf-16 encoder with its utf-16be flag set.

14.4 utf-16le

In violation of the Unicode standard, "utf-16" is a label for utf-16le rather than its own standalone encoding.

14.4.1 utf-16le decoder

utf-16le's decoder is shared utf-16 decoder with its utf-16be flag unset.

14.4.2 utf-16le encoder

utf-16le's encoder is shared utf-16 encoder with its utf-16be flag unset.

14.5 x-user-defined

While technically this is a single-byte encoding, it is defined separately as it can be implemented algorithmically.

14.5.1 x-user-defined decoder

x-user-defined's decoder's handler, given a stream and byte, runs these steps:

1. If byte is end-of-stream, return finished.

2. If byte is in the range 0x00 to 0x7F, return a code point whose value is byte.

3. Return a code point whose value is 0xF780 + byte − 0x80.

14.5.2 x-user-defined encoder

x-user-defined's encoder's handler, given a stream and code point, runs these steps:

1. If code point is end-of-stream, return finished.

2. If code point is in the range U+0000 to U+007F, return a byte whose value is code point.

3. If code point is in the range U+F780 to U+F7FF, return a byte whose value is code point − 0xF780 + 0x80.

4. Return error with code point.

References

Acknowledgments

There have been a lot of people that have helped make encodings more interoperable over the years and thereby furthered the goals of this standard. Likewise many people have helped making this standard what it is today.

Ideally they are all listed here so please contact the editor with any omissions.

With that, many thanks to Alan Chaney, Alexander Shtuchkin, Allen Wirfs-Brock, Ben Noordhuis, Boris Zbarsky, Cameron McCormack, Charles McCathieNeville, David Carlisle, Dominique Hazaël-Massieux, Doug Ewell, Erik van der Poel, 譚永鋒 (Frank Yung-Fong Tang), Sam Sneddon, Glenn Maynard, Gordon P. Hemsley, Henri Sivonen, Ian Hickson, James Graham, John Tamplin, Joshua Bell, 신정식 (Jungshik Shin), 강 성훈 (Kang Seonghoon), 川幡太一 (Kawabata Taichi), Ken Lunde, Kenneth Russell, Leif Halvard Silli, Makoto Kato, Mark Callow, Mark Davis, Martin Dürst, Masatoshi Kimura, Ms2ger, Nigel Tao, Norbert Lindenberg, Øistein E. Andersen, Peter Krefting, Philip Jägenstedt, Philip Taylor, Robbert Broersma, Robert Mustacchi, Ryan Dahl, Shawn Steele, Simon Montagu, Simon Pieters, Simon Sapin, Vyacheslav Matva, and 成瀬ゆい (Yui Naruse) for being awesome.