W3C

Character Model for the World Wide Web 1.0

W3C Working Draft 28 September 2001

This version:
http://www.w3.org/TR/2001/WD-charmod-20010928
Latest version:
http://www.w3.org/TR/charmod
Previous version:
http://www.w3.org/TR/2001/WD-charmod-20010126
Editors:
Martin J. Dürst, W3C <duerst@w3.org>
François Yergeau, Alis Technologies, Inc. <yergeau@alis.com>
Misha Wolf, Reuters Ltd. <misha.wolf@reuters.com>
Asmus Freytag, ASMUS, Inc. <asmus@unicode.org>
Tex Texin, Progress Software Corp. <texin@progress.com>
Richard Ishida, Xerox <Richard.Ishida@gbr.xerox.com>

Abstract

This Architectural Specification provides authors of specifications, software developers, and content developers a common reference for interoperable text manipulation on the World Wide Web. Topics addressed include encoding identification, early uniform normalization, string identity matching, string indexing, and URI conventions, building on the Universal Character Set, defined jointly by Unicode and ISO/IEC 10646. Some introductory material on characters and character encodings is also provided.

Status of this Document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. The latest status of this series of documents is maintained at the W3C.

This is a W3C Working Draft published between the first Last Call Working Draft of 26 January 2001 and a planned second Last Call. This interim publication is used to document the progress made on addressing the comments received during the first Last Call. A list of last call comments with their status can be found in the disposition of comments (Members only).

Work is still ongoing on addressing the comments received during the first Last Call. We do not encourage comments on this Working Draft; instead we ask reviewers to wait for the second Last Call. We will announce the second Last Call on the W3C Internationalization public mailing list (www-international@w3.org; subscribe). Comments from the public and from organizations outside the W3C may be sent to www-i18n-comments@w3.org (archive). Comments from W3C Working Groups may be sent directly to the Internationalization Interest Group (w3c-i18n-ig@w3.org), with cross-posting to the originating Group, to facilitate discussion and resolution.

Due to the architectural nature of this document, it affects a large number of W3C Working Groups, but also software developers, content developers, and writers and users of specifications outside the W3C that have to interface with W3C specifications.

This document is published as part of the W3C Internationalization Activity by the Internationalization Working Group (Members only), with the help of the Internationalization Interest Group. The Internationalization Working Group will not allow early implementation to constrain its ability to make changes to this specification prior to final release. Publication as a Working Draft does not imply endorsement by the W3C Membership. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress".

For information about the requirements that informed the development of important parts of this specification, see Requirements for String Identity Matching and String Indexing. A list of current W3C Recommendations and other technical documents can be found at the W3C Web site.

Table of Contents

1 Introduction
    1.1 Goals and Scope
    1.2 Background
    1.3 Terminology and Notation
2 Conformance
3 Characters
    3.1 Perceptions of Characters
    3.2 Digital Encoding of Characters
    3.3 Transcoding
    3.4 Strings
    3.5 Reference Processing Model
    3.6 Choice and Identification of Character Encodings
    3.7 Character Escaping
4 Early Uniform Normalization
    4.1 Motivation
    4.2 Definitions for W3C Text Normalization
    4.3 Responsibility for Normalization
5 Compatibility and Formatting Characters
6 String Identity Matching
7 String Indexing
8 Character Encoding in URI References
9 Referencing the Unicode Standard and ISO/IEC 10646

Appendix A Examples of Characters, Keystrokes and Glyphs

Acknowledgements (Non-Normative)
References
    Normative References
    Other References
Change Log (Non-Normative)


1 Introduction

1.1 Goals and Scope

The goal of this document is to facilitate use of the Web by all people, regardless of their language, script, writing system, and cultural conventions, in accordance with the W3C goal of Universal Access. One basic prerequisite to achieve this goal is to be able to transmit and process the characters used around the world in a well-defined and well-understood way.

The main target audience of this document is W3C specification developers. This document defines conformance requirements for other W3C specifications. This document and parts of it can also be referenced from other W3C specifications.

Other audiences of this document include software developers, content developers, and authors of specifications outside the W3C. Software developers and content developers implement and use W3C specifications. This document defines some conformance requirements for software developers and content developers that implement and use W3C specifications. It also helps software developers and content developers to understand the character-related provisions in other W3C specifications.

The character model described in this document provides authors of specifications, software developers, and content developers with a common reference for consistent, interoperable text manipulation on the World Wide Web. Working together, these three groups can build a more international Web.

Topics addressed include encoding identification, early uniform normalization, string identity matching, string indexing, and URI conventions. Some introductory material on characters and character encodings is also provided.

Topics not addressed or barely touched include collation (sorting), fuzzy matching and language tagging. Some of these topics may be addressed in a future version of this specification.

At the core of the model is the Universal Character Set (UCS), defined jointly by The Unicode Standard [Unicode] and ISO/IEC 10646 [ISO/IEC 10646]. In this document, Unicode is used as a synonym for the Universal Character Set. The model will allow Web documents authored in the world's scripts (and on different platforms) to be exchanged, read, and searched by Web users around the world.

All W3C specifications must conform to this document (see section 2 Conformance). Authors of other specifications (for example, IETF specifications) are strongly encouraged to take guidance from it.

Since other W3C specifications will be based on some of the provisions of this document, without repeating them, software developers implementing W3C specifications must conform to these provisions.

1.2 Background

This section provides some historical background on the topics addressed in this document.

Starting with Internationalization of the Hypertext Markup Language ([RFC 2070]), the Web community has recognized the need for a character model for the World Wide Web. The first step towards building this model was the adoption of Unicode as the document character set for HTML.

The choice of Unicode was motivated by the fact that Unicode:

  • is the only universal character repertoire available,
  • covers the widest possible range,
  • provides a way of referencing characters independent of the encoding of a resource,
  • is being updated/completed carefully,
  • is widely accepted and implemented by industry.

W3C adopted Unicode as the document character set for HTML in [HTML 4.0]. The same approach was later used for specifications such as [XML 1.0] and [CSS2]. Unicode now serves as a common reference for W3C specifications and applications.

The IETF has adopted some policies on the use of character sets on the Internet (see [RFC 2277]).

When data transfer on the Web remained mostly unidirectional (from server to browser), and where the main purpose was to render documents, the use of Unicode without specifying additional details was sufficient. However, the Web has grown:

  • Data transfers among servers, proxies, and clients, in all directions, have increased,
  • Non-ASCII characters are being used in more and more places,
  • Data transfers between different protocol/format elements (such as element/attribute names, URI components, and textual content) have increased,
  • More and more APIs are defined, not just protocols and formats.

In short, the Web may be seen as a single, very large application ([Nicol]), rather than as a collection of small independent applications.

While these developments strengthen the requirement that Unicode be the basis of a character model for the Web, they also create the need for additional specifications on the application of Unicode to the Web. Some aspects of Unicode that require additional specification for the Web include:

  • Choice of encoding forms (UTF-8, UTF-16, UTF-32)
  • Counting characters, measuring string length in the presence of variable-length encodings and combining characters)
  • Duplicate encodings (e.g. precomposed vs decomposed)
  • Use of control codes for various purposes (e.g. bidirectionality control, symmetric swapping, etc.)

It should be noted that such properties also exist in legacy encodings (where legacy encoding is taken to mean any character encoding not based on Unicode), and in many cases have been inherited by Unicode in one way or another from such legacy encodings.

The remainder of this document presents additional specifications and requirements to ensure an interoperable character model for the Web, taking into account earlier work (from W3C, ISO and IETF).

1.3 Terminology and Notation

For the purpose of this specification, the producer of text data is the sender of the data in the case of protocols, and the tool that produces the data in the case of formats. The recipient of text data is the software module that receives the data.

Note: A software module may be both a recipient and a producer.

Unicode code points are denoted as U+hhhh, where hhhh is a sequence of at least four, and at most six hexadecimal digits.

2 Conformance

In this document, requirements are expressed using the key words "MUST", "MUST NOT", "REQUIRED", "SHALL" and "SHALL NOT". Recommendations are expressed using the key words "SHOULD", "SHOULD NOT" and "RECOMMENDED". "MAY" and "OPTIONAL" are used to indicate optional features or behaviour. These keywords are used in accordance with [RFC 2119].

This specification places conformance requirements on specifications, on software and on Web content. To aid the reader, all requirements are preceded by '[X]' where 'X' is one of 'S' for specifications, 'I' for software implementations, and 'C' for Web content. These markers indicate the relevance of the requirement and allow the reader to quickly locate relevant requirements using the browser's search function. [S][I][C] In order to conform to this document, each of these (content, software or specification) MUST satisfy all requirements that apply to it, as specified in the text. For instance, software does not have to satisfy requirements applying only to specifications in order to be conformant.

[S] Every W3C specification MUST:

  1. [S] conform to the requirements applicable to specifications;
  2. [S] specify that implementations MUST conform to the requirements applicable to software; and
  3. [S] specify that content created according to that specification MUST conform to the requirements applicable to content.

[S] If an existing W3C specification does not conform, then the next version of that specification MUST be modified in order to conform.

Where this specification contains a procedural description, it is to be understood as a way to specify the desired external behavior. [I] Implementations MAY use other ways of achieving the same results, as long as observable behavior is not affected.

3 Characters

3.1 Perceptions of Characters

3.1.1 Introduction

The word character is used in many contexts, with different meanings. Human cultures have radically differing writing systems, leading to radically differing concepts of a character. Such wide variation in end user experience can, and often does, result in misunderstanding. This variation is sometimes mistakenly seen as the consequence of imperfect technology. Instead, it derives from the great flexibility and creativity of the human mind and the long tradition of writing as an important part of the human cultural heritage.

The developers of W3C specifications, and the developers of software based on those specifications, are likely to be more familiar with usages they have experienced and less familiar with the wide variety of usages in an international context. Furthermore, within a computing context, characters are often confused with related concepts, resulting in incomplete or inappropriate specifications and software.

This section examines some of these contexts, meanings and confusions.

3.1.2 Units of a Writing System, and Units of Aural Rendering

The glossary entry in [Unicode 3.0] gives:

Character. (1) The smallest component of written language that has semantic values; refers to the abstract meaning and or shape ...

In some scripts, characters have a close relationship to phonemes, while in others they are closely related to meanings. Even when characters (loosely) correspond to phonemes, this relationship may not be simple, and there is rarely a one-to-one correspondence between character and phoneme.

Example: Japanese hiragana and katakana are syllabaries, not phonemic alphabets. A character in these scripts corresponds not to a phoneme, but to a syllable.

Example: Korean Hangul combines symbols for phonemes into square syllabic blocks. Depending on the user and the application, either the individual symbols or the syllabic clusters can be considered to be characters.

Example: Indic scripts use semi-regular or irregular ways to combine consonants and vowels into clusters. Depending on the user and the application, either individual consonants or vowels, or the consonant or consonant-vowel clusters can be perceived as characters.

[S][I] Specification writers and software developers MUST NOT assume that there is a one-to-one correspondence between characters and phonemes.

3.1.3 Units of Visual Rendering

Visual rendering introduces the notion of a glyph. Glyphs are defined by [ISO/IEC 9541] as "a recognizable abstract graphic symbol which is independent of a specific design". There is not a one-to-one correspondence between characters and glyphs:

  • A single character can be represented by multiple glyphs (each glyph is then part of the representation of that character). These glyphs may be physically separated from one another.
  • A single glyph may represent a sequence of characters (this is the case with ligatures, among others).
  • A character may be rendered with very different glyphs depending on the context.
  • A single glyph may represent different characters (e.g. capital Latin A, capital Greek A and capital Cyrillic A).

Each glyph can be represented by a number of different glyph images; a set of glyph images makes up a font. Glyphs can be construed as the basic units of organization of the visual rendering of text, just as characters are the basic unit of organization of encoded text.

See Appendix A for examples of the complexities of character to glyph mapping.

[S][I] Specification writers and software developers MUST NOT assume a one-to-one mapping between character codes and units of displayed text.

Some scripts, in particular Arabic and Hebrew, are written from right to left. Text including characters from these scripts can run in both directions and is therefore called bidirectional text (see examples A.6 to A.8).

The Unicode Standard requires that characters be stored and interchanged in logical order. [S] Protocols, data formats and APIs MUST store, interchange or process text data in logical order. [S] Where protocols or APIs support selection, they MUST support logical selection (range of character offsets), and they SHOULD support visual selection (area on display), because many users prefer visual selection. Visual selection can give rise to discontiguous logical selections and vice versa (see examples A.6 to A.8). [S] Specifications of protocols and APIs that involve selection MUST provide for discontiguous logical selections, at least to the extent necessary to support scripts involving character reordering. [S] Such specifications MAY, however, make actual implementation of discontiguous logical selections optional for implementations that choose not to support scripts involving character reordering. This is analogous to HTML4 not requiring the bidirectional algorithm from those implementations that choose not to support bidirectional scripts (cf. section 8.2 of [HTML 4.0]).

3.1.4 Units of Input

In keyboard input, it is not always the case that keystrokes and input characters correspond one-to-one. A limited number of keys can fit on a keyboard and many writing systems have too many characters to allow such a correspondence and must rely on more complex input methods, which transform keystroke sequences into character sequences. Other languages may make it necessary to input some characters with special modifier keys. See Appendix A for examples of non-trivial input.

[S][I] Specification writers and software developers MUST NOT assume that a single keystroke results in a single character, nor that a single character can be input with a single keystroke (even with modifiers), nor that keyboards are the same all over the world.

3.1.5 Units of Collation

String comparison, as used in sorting and searching for instance, is based on collation units which are not in general in a one-to-one relationship to encoded characters. String comparison can aggregate a character sequence into a single collation unit with its own position in the sorting order, can separate a single character into multiple collation units, and can distinguish various aspects of a character (case, presence of diacritics, etc.) to be sorted separately (multi-level sorting).

Example: In traditional Spanish sorting, the letter sequences "ch" and "ll" are treated as atomic collation units. Although Spanish sorting, and to some extent Spanish everyday use, treat "ch" as a single unit, current digital encodings treat it as two letters, and keyboards do the same (the user types 'c', then 'h').

Example: In most languages, the letter æ is sorted as two consecutive collation units: a and e.

Example: The sorting of text written in a bicameral script (i.e. a script which has distinct upper and lower case letters) is usually required to ignore case differences in a first pass; case is then used to break ties in a later pass.

Example: Treatment of accented letters in sorting is dependent on the script or language in question. The letter "ö" is considered as a modified "o" in French, as an extra character quite related to "o" in German and as a letter completely independent from "o" (and sorting after "z") in Swedish.

Note: collation methods may also vary by application (e.g. bilingual dictionaries are sorted differently than telephone books in Japanese).

[S][I] Even though software fundamentally works on the level of encoded characters, the capabilities of software to sort (or otherwise process) text MUST NOT be restricted to those collation units that have a one-to-one relation to encoded characters. [I] Software developers MUST NOT merely use a one-to-one mapping as their string-compare function, as for example in sorting operations.

3.1.6 Units of Storage

Computer storage and communication rely on units of physical storage and information interchange, such as bits and bytes (also known as octets, as nowadays the word bytes is generally considered to mean 8-bit bytes). A frequent error in specifications and implementations is the equating of characters with units of physical storage. The mapping between characters and such units of storage is actually quite complex, and is discussed in the next section, 3.2 Digital Encoding of Characters.

[S][I] Specification writers and software developers MUST NOT assume a one-to-one relationship between characters and units of physical storage.

3.1.7 Summary

The term character is used differently in a variety of contexts and often leads to confusion when used outside of these contexts. In the context of the digital representations of text, a character can be defined informally as a small logical unit of text. Text is then defined as sequences of characters. While such an informal definition is sufficient to create or capture a common understanding in many cases, it is also sufficiently open to create misunderstandings as soon as details start to matter. In order to write effective specifications, protocol implementations, and software for end users, it is very important to understand that these misunderstandings can occur.

[S] Specifications using the term character MUST specify which of the possible meanings they intend. [S] Specifications SHOULD avoid the use of the term character if a more specific term is available.

3.2 Digital Encoding of Characters

To be of any use in computers, in computer communications and in particular on the World Wide Web, characters must be encoded. In fact, much of the information processed by computers over the last few decades has been encoded text, exceptions being images, audio, video and numeric data. To achieve text encoding, a large variety of encoding schemes have been devised, which can loosely be defined as mappings between the character sequences that users manipulate and the sequences of bits that computers manipulate.

Given the complexity of text encoding and the large variety of schemes for character encoding invented throughout the computer age, a more formal description of the encoding process is useful. The process of defining a text encoding can be described as follows (see [UTR #17] for a more detailed description):

  1. A set of characters to be encoded is identified. The characters are pragmatically chosen to express text and to efficiently allow various text processes in one or more target languages. They may not correspond precisely to what users perceive as letters and other characters. The set of characters is called a repertoire.

  2. Each character in the repertoire is then associated with a (mathematical, abstract) non-negative integer, the code point (also known as a character number or code position). The result, a mapping from the repertoire to the set of non-negative integers, is called a coded character set (CCS).

  3. To enable use in computers, a suitable base datatype is identified (such as a byte, a 16-bit unit of storage or other) and a character encoding form (CEF) is used, which encodes the abstract integers of a CCS into sequences of the code units of the base datatype. The encoding form can be extremely simple (for instance, one which encodes the integers of the CCS into the natural representation of integers of the chosen datatype of the computing platform) or arbitrarily complex (a variable number of code units, where the value of each unit is a non-trivial function of the encoded integer).

  4. To enable transmission or storage using byte-oriented devices, a serialization scheme or character encoding scheme (CES) is next used. A CES is a mapping of the code units of a CEF into well-defined sequences of bytes, taking into account the necessary specification of byte-order for multi-byte base datatypes and including in some cases switching schemes between the code units of multiple CCSes (an example is ISO 2022). A CES, together with the CCSes it is used with, is identified by an IANA charset identifier. Given a sequence of bytes representing text and a charset identifier, one can in principle unambiguously recover the sequence of characters of the text.

Note: The term character encoding is somewhat ambiguous, as it is sometimes used to describe the actual process of encoding characters and sometimes to denote a particular way to perform that process (as in this file is in the X character encoding). Context normally allows the distinction of those uses, once one is aware of the ambiguity.

Note: Unfortunately, there are some important cases of charset identifiers that denote a range of slight variants of an encoding scheme, where the differences may be crucial (e.g. the well-known yen/backslash case) and may vary over time. In those cases, recovery of the character sequence from a byte sequence is not totally unambiguous. See the [XML Japanese profile] for examples of such ambiguous charsets.

In very simple cases, the whole encoding process can be collapsed to a single step, a trivial one-to-one mapping from characters to bytes; this is the case, for instance, for US-ASCII and ISO-8859-1.

Text data is said to be in a Unicode encoding form if it is encoded in UTF-8, UTF-16 or UTF-32.

3.3 Transcoding

Transcoding is the process of converting text data from one Character Encoding Form to another. Transcoders work only at the level of character encoding and do not parse the text; consequently, they do not deal with character escapes such as numeric character references (see 3.7 Character Escaping) and do not fix up embedded character encoding information (for instance in an XML declaration or in an HTML <meta> element).

Note: Transcoding may involve one-to-one, many-to-one, one-to-many or many-to-many mappings. In addition, the storage order of characters varies between encodings: some, such as Unicode, prescribe logical ordering while others use visual ordering; among encodings that have separate diacritics, some prescribe that they be placed before the base character, some after. Because of these differences in sequencing characters, transcoding may involve reordering: thus XYZ may map to yxz.

A normalizing transcoder is a transcoder that converts from a legacy encoding to a Unicode encoding form and ensures that the result is in Unicode Normalization Form C (see 4.2.1 Unicode-normalized Text). For most legacy encodings, it is possible to construct a normalizing transcoder; it is not possible to do so if the encoding's repertoire contains characters not in Unicode.

3.4 Strings

Various specifications use the notion of a string, sometimes without defining precisely what is meant and sometimes defining it differently from other specifications. The reason for this variability is that there are in fact multiple reasonable definitions for a string, depending on one's intended use of the notion; the term string is used for all these different notions because these are actually just different views of the same reality: a piece of text stored inside a computer. This section provides specific definitions for different notions of string which may be reused elsewhere.

Byte string: A string viewed as a sequence of bytes representing characters in a particular encoding. This corresponds to a CES. As a definition for a string, this definition is most often useless, except when the textual nature is unimportant and the string is considered only as a piece of opaque data with a length in bytes. Specifications in general SHOULD NOT define a string as a byte string.

Code unit string: A string viewed as a sequence of code units representing characters in a particular encoding. This corresponds to a CEF. This definition is useful in APIs that expose a physical representation of string data. Example: For the [DOM Level 1], UTF-16 was chosen based on widespread implementation practice.

Character string: A string viewed as a sequence of characters, each represented by a code point in [Unicode]. This is usually what programmers consider to be a string, although it may not match exactly what most users perceive as characters. This is the highest layer of abstraction that ensures interoperability with very low implementation effort. [S] This definition is generally the most useful and SHOULD be used by most specifications, following the examples of Production [2] of [XML 1.0], the SGML declaration of [HTML 4.01], and the character model of [RFC 2070].

Example: Consider the string comprising the characters U+10330 GOTHIC LETTER AHSA, U+2260 NOT EQUAL TO and U+0030 DIGIT ZERO, encoded in UTF-16 in big-endian byte order. The rows of the following table show the string viewed as a character string, code unit string and byte string, respectively:

Character string U+10330 U+2260 U+0030
Code unit string D800 DF30 2260 0030
Byte string D8 00 DF 30 22 60 00 30

Note: It is also possible to view a string as a sequence of graphemes. In this case the string is divided into text units that correspond to the user's perception of where character boundaries occur in a visually rendered text. However, there is no standard rule for the segmentation of text in this way, and the segmentation will vary from language to language and even from user to user. Examples of possible approaches can be found in sections 5.12 and 5.15 of [Unicode 3.0].

3.5 Reference Processing Model

Many Internet protocols and data formats, most notably the very important Web formats HTML, CSS and XML, are based on text. In those formats, everything is text but the relevant specifications impose a structure on the text, giving meaning to certain constructs so as to obtain functionality in addition to that provided by plain text. HTML and XML are markup languages, defining entities entirely composed of text but with conventions allowing the separation of this text into markup and character data. Citing from [XML 1.0], section 2.4:

Text consists of intermingled character data and markup. [...] All text that is not markup constitutes the character data of the document.

For the purposes of this section, the important aspect is that everything is text, that is, a sequence of characters.

Since its early days, the Web has seen the development of a Reference Processing Model, first described for HTML in [RFC 2070]. This model was later embraced by XML and CSS. It is applicable to any data format or protocol that is text-based as described above. The essence of the Reference Processing Model is the use of Unicode as a common reference. Use of the Reference Processing Model by a specification does not, however, require that implementations actually use Unicode. The requirement is only that the implementations behave as if the processing took place as described by the Model.

[S] A specification conforms to the Reference Processing Model if all of the following apply:

  • [S] The specification MUST be defined in terms of Unicode characters, not bytes or glyphs.
  • [S] The specification SHOULD NOT restrict the range of characters that can be used. The total Unicode range of code points from 0 to 0x10FFFF inclusive should be usable; any exceptions should be listed and justified.
  • [S] The specification MAY allow use of any character encoding which can be transcoded to Unicode for its text entities. [S] It MAY choose to disallow or deprecate some encodings and to make others mandatory. [S] Independent of the actual encoding, the specified behavior MUST be the same as if the processing happened as follows:
    • [S] the encoding of any text entity received by the application implementing the specification MUST be determined and the text entity MUST be interpreted as a sequence of Unicode characters. [S] This MUST be equivalent to transcoding the entity to some Unicode encoding form and receiving it in that encoding form;
    • [S] all processing MUST take place on this sequence of Unicode characters;
    • [S] if text is output by the application, the sequence of Unicode characters MUST be encoded using an encoding chosen among those allowed by the specification.
  • [S] If the specification is such that multiple text entities are involved (such as an XML document referring to external parsed entities), it MAY choose to allow these entities to be in different character encodings. [S] In all cases, this Reference Processing Model MUST be applied to all entities.

[S] All specifications that involve text MUST specify processing according to the Reference Processing Model.

Note: All specifications that derive from [XML 1.0] automatically inherit this Reference Processing Model. XML is entirely defined in terms of Unicode characters and mandates the UTF-8 and UTF-16 encodings while allowing any other encoding for parsed entities.

Note: When specifications choose to allow encodings other than Unicode encodings, implementers should be aware that the correspondence between the characters of a legacy encoding and Unicode characters may in practice depend on the software used for transcoding. See the [XML Japanese profile] for examples of such inconsistencies.

3.6 Choice and Identification of Character Encodings

Because encoded text cannot be interpreted and processed without knowing the encoding, it is vitally important that the character encoding scheme (see 3.2 Digital Encoding of Characters) is known at all times and places where text is exchanged or processed. [S]Specifications MUST either specify a unique encoding, or provide mechanisms such that the encoding of text can always be reliably identified. [S]When designing a new protocol, format or API, specifications SHOULD mandate a unique character encoding.

Mandating a unique character encoding is simple, efficient, and robust. There is no need for specifying, producing, transmitting, and interpreting encoding tags. At the receiver, the encoding will always be understood. There is also no ambiguity if data is transferred non-electronically and later has to be converted back to a digital representation.  Even when there is a need for compatibility with existing data, systems, protocols and applications, multiple encodings can often be dealt with at the boundaries or outside a protocol, format, or API. The DOM ([DOM Level 1]) is an example of where this was done. The advantages of choosing a unique encoding become more important the smaller the pieces of text used are and the closer to actual processing the specification is.

[S] When a unique encoding is mandated, the encoding MUST be UTF-8, UTF-16 or UTF-32. [S] If compatibility with ASCII is desired, UTF-8 (see [RFC 2279]) is RECOMMENDED; on the Internet, the IETF Charset Policy [RFC 2277] specifies that "Protocols MUST be able to use the UTF-8 charset". [S] For APIs, UTF-16 (see [RFC 2781]) or UTF-32 are RECOMMENDED.

Note: The [XML 1.0] specification requires all conforming XML processors to accept both UTF-16 and UTF-8.

3.6.1 Character Encoding Identification

[S] If the unique encoding approach is not chosen, then protocols MUST provide character encoding identification mechanisms. [S] These mechanisms SHOULD be along the lines of the [MIME] Internet specification. The MIME charset parameter is defined such that it provides sufficient information to unambiguously decode the sequence of bytes of the payload into a sequence of characters. The values are drawn from the IANA charset registry [IANA].

Note: The term charset derives from "character set", an expression with a long and tortured history (see [Connolly] for a discussion). [S] Specifications SHOULD avoid using the expression "character set", as well as the term "charset", except when the latter is used to refer to the MIME charset parameter or its IANA-registered values. The terms character encoding or character encoding scheme are recommended. In XML, the XML declaration or the text declaration contains a pseudo-attribute called encoding which actually identifies the charset.

The IANA charset registry is the official list of names and aliases for character encodings on the Internet.

[S] If the unique encoding approach is not taken, specifications SHOULD mandate the use of the IANA charset names, and in particular the names labeled in the registry as MIME preferred names, to designate character encodings in protocols, data formats and APIs. [S] The "x-" convention for unregistered names SHOULD NOT be used, having led to abuse in the past (use of "x-" for character encodings that were widely used, even long after there was an official registration). [I][C] Content developers and software that tag textual data MUST use one of the names mandated by the appropriate specification (e.g. the XML specification when editing XML text) and SHOULD use the MIME preferred name of an encoding to tag data in that encoding. [I][C] An IANA-registered charset name MUST NOT be used to tag textual data in an encoding other than the one identified in the IANA registration of that name.

[S] If the unique encoding approach is not chosen, specifications MUST designate at least one of the UTF-8 and UTF-16 encoding forms of Unicode as admissible encodings and SHOULD choose at least one of UTF-8 or UTF-16 as mandated encoding forms (encoding forms that MUST be supported by implementations of the specification). [S] They MAY define either UTF-8 or UTF-16 as a default (or both if they define suitable means of distinguishing them), but they MUST NOT use any other character encoding as a default. [S] Also, they MUST NOT use unreliable heuristics.

[I] Receiving software MUST determine the encoding from available information according to appropriate specifications. [I] When a IANA-registered charset name is recognized, receiving software MUST interpret the received data according to the encoding associated with the name in the IANA registry. [I] When no charset is provided the receiving software MUST adhere to the default encoding(s) specified in the specification.

[I] Receiving software MAY recognize as many encodings (names and aliases) as appropriate. A field-upgradeable mechanism may be appropriate for this purpose. Certain encodings are more or less associated with certain languages (e.g. Shift-JIS with Japanese); trying to support a given language or set of customers may mean that certain encodings have to be supported. The encodings that need to be supported may change over time. This document does not give any advice on which encoding may be appropriate or necessary for the support of any given language.

[I] Software MUST completely implement the mechanisms for character encoding identification and SHOULD implement them in such a way that they are easy to use (for instance in HTTP servers). [I] On interfaces to other protocols, software SHOULD support conversion between Unicode encoding forms as well as any other necessary conversions. [C] Content MUST make use of the offered facilities by always indicating character encoding; where the facilities offered for character encoding identification include defaults (e.g. in [XML 1.0]), relying on such defaults is sufficient to satisfy this identification requirement.

Because of the layered Web architecture (e.g. formats used over protocols), there may be multiple and at times conflicting information about character encoding. [S] Specifications MUST define conflict-resolution mechanisms (e.g. priorities) for these cases; [I][C] software and content MUST follow them carefully.

3.6.2 Private Use Code Points

Unicode designates certain ranges of code points for private use: the Private Use Area (U+E000-F8FF) and planes 15 and 16 (U+F0000-FFFFD and U+100000-10FFFD). These code points are guaranteed to never be allocated to standard characters, and are available for use by private agreement between a producer and a recipient. However, their use is strongly discouraged, since private agreements do not scale on the Web. Code points from different private agreements may collide, and a private agreement and therefore the meaning of the code points can quickly get lost.

[S] Specifications MUST NOT define any assignments of private use code points. [S] Conformance to a specification MUST NOT require the use of PUA characters. [S] Specifications SHOULD NOT provide mechanisms for agreement on private use code points between parties and MUST NOT require the use of such mechanisms. [S][I] Specifications and implementations SHOULD be designed in such a way as to not disallow the use of these code points by private arrangement. As an example, XML does not disallow the use of private use code points.

[S] Specifications MAY define markup to allow the transmission of symbols not in Unicode or to identify specific variants of Unicode characters.

Example: [MathML2], in section 3.2.9, defines an element mglyph for mathematical symbols not in Unicode.

Example: [SVG], in section 10.14, defines an element altGlyph which allows the identification of specific display variants of Unicode characters.

3.7 Character Escaping

In text-based protocols or formats where characters can be either part of character data or of markup (see 3.5 Reference Processing Model), it is often the case that certain characters are designated as having certain specific protocol/format functions in certain contexts (e.g. "<" and "&" serve as markup delimiters in HTML and XML). These syntax-significant characters cannot be used to represent themselves in text data in the same way as all other characters do. Also, often formats are represented in an encoding that does not allow to represent all characters directly.

To express syntax-significant or unrepresentable characters, a technique called escaping is used. This works by creating an additional syntactic construct, defining additional characters or defining character sequences that have special meaning. Escaping a character means expressing it using such a construct, appropriate to the format or protocol in which the character appears; expanding an escape (or unescaping) means replacing it with the character that it represents.

Certain guidelines apply to the way specifications define character escapes. [S] These guidelines MUST be followed when designing new W3C protocols and formats and SHOULD be followed as much as possible when revising existing protocols and formats.

  • [S] Specifications MUST NOT invent a new escaping mechanism if an appropriate one already exists.
  • [S] The number of different ways to escape a character SHOULD be minimized (ideally to one). [A well-known counter-example is that for historical reasons, both HTML and XML have redundant decimal (&#ddddd;) and hexadecimal (&#xhhhh;) escapes.]
  • [S] Explicit end delimiters MUST be provided. [S] Escapes such as \uABCD where the end delimiter is a space or any character other than [01-9A-F] SHOULD be avoided: it is not clear visually, and it can cause an editor to insert spurious line-breaks when word-wrapping on spaces. Forms like SPREAD's &UABCD; [SPREAD] or XML's &#xhhhh;, where the escape is explicitly terminated by a semicolon, are much better.
  • [S] Escaped characters SHOULD be acceptable wherever unescaped characters are; this does not preclude that a syntax-significant character, when escaped, loses its significance in the syntax. [S] In particular, escaped characters SHOULD be acceptable in identifiers and comments.

Certain guidelines apply to content developers, as well as to software that generates content:

  • [I][C] Escapes SHOULD be avoided when the characters to be expressed are representable in the character encoding of the document.
  • [I][C] Since character set standards usually list character numbers as hexadecimal, content developers SHOULD use the hexadecimal form of escapes when there is one.

4 Early Uniform Normalization

4.1 Motivation

As explained at length in Requirements for String Identity Matching and String Indexing [CharReq], the existence, in many character encoding schemes, of multiple representations for what users perceive as the same string makes it necessary to define character data normalization. Without a precise specification, it is not possible to determine reliably whether or not two strings are identical. Such a specification must take into account character encoding, the way normalization is to be performed and where or when (by sender or recipient) to perform it.

String identity is central to the correct functioning of much software, and in particular of large parts of the Web infrastructure (protocols, formats, etc.). Incorrect string matching can have far reaching consequences, including the creation of security holes. Consider a contract, encoded in XML, for buying goods: each item sold is described in an <artículo> element; unfortunately, "artículo" is subject to different representations in the character encoding of the contract. Suppose that the contract is viewed and signed by means of a user agent that looks for <artículo> elements, extracts them (matching on the element name), presents them to the user and adds up their prices. If different instances of the <artículo> element happen to be represented differently in a particular contract, then the buyer and seller may see (and sign) different contracts if their respective user agents perform string identity matching differently, which is fairly likely in the absence of a well-defined specification. The absence of a well-defined specification also means that there is no way to resolve the ensuing contractual dispute.

The Unicode Consortium provides four standard normalization forms (see Unicode Normalization Forms [UTR #15]). For use on the Web, this document defines W3C Text Normalization by picking the most appropriate of these (NFC) and additionally addressing the issues of legacy encodings and of character escapes (which can denormalize text when unescaped).

Roughly speaking, NFC is defined such that combining character sequences (a base character followed by one or more combining characters) are replaced, as far as possible, by canonically equivalent precomposed characters. Text in a Unicode encoding form is said to be in NFC if it doesn't contain any combining sequence that could be replaced and if any remaining combining sequence is in canonical order.

This document also specifies that normalization is to be performed early (by the sender) as opposed to late (by the recipient). The reasons for that choice are manifold:

  • Almost all legacy data as well as data created by current software is normalized (using NFC).

  • The number of Web components that generate or transform text is considerably smaller than the number of components that receive text and need to perform matching or other processes requiring normalized text.

  • Current receiving components (browsers, XML parsers, etc.) implicitly assume early normalization by not performing normalization themselves. This is a vast legacy.

  • Web components that generate and process text are in a much better position to do normalization than other components; in particular, they may be aware that they deal with a restricted repertoire only.

  • Not all components of the Web that implement functions such as string matching can reasonably be expected to do normalization. This, in particular, applies to very small components and components in the lower layers of the architecture.

  • Forward-compatibility issues can be dealt with more easily: less software needs to be updated, namely only the software that generates newly introduced characters.

  • It improves matching in cases where the character encoding is partly undefined, such as URIs [RFC 2396] in which non-ASCII bytes have no defined meaning.

  • It is a prerequisite for comparison of encrypted strings (see [CharReq], section 2.7).

  • It simplifies definitions and implementations for string indexing (see [CharReq], section 4.6).

  • It increases interoperability and predictability when string data is exposed in an API.

4.2 Definitions for W3C Text Normalization

4.2.1 Unicode-normalized Text

Text data is, for the purposes of this specification, Unicode-normalized if it is in a Unicode encoding form and is in Unicode Normalization Form C (according to revision 18 of [UTR #15]).

4.2.2 Fully Normalized Text

Text data is fully normalized if:

  1. the data is Unicode-normalized and does not contain any character escapes whose unescaping would cause the data to become no longer Unicode-normalized; or

  2. the data is in a legacy encoding and, after transcoding to a Unicode encoding form by a normalizing transcoder, the data satisfies clause 1 above.

In the remainder of this specification, normalized is used to mean fully normalized, unless otherwise indicated.

Note: A consequence of this definition is that legacy text (i.e. text in a legacy encoding) is always normalized unless i) a normalizing transcoder cannot exist for that encoding (e.g. because the repertoire contains characters not in Unicode) or ii) the text contains escapes which, once expanded, result in un-normalized text.

Note: Full normalization is specified against the context of a markup language (or the absence thereof), which specifies the form of escapes. For plain text (no escapes) in a Unicode encoding form, full normalization and Unicode-normalization are equivalent.

4.2.3 Examples

The string "suçon", expressed as the sequence of five characters U+0073 U+0075 U+00E7 U+006F U+006E and encoded in a Unicode encoding form, is both Unicode-normalized and fully normalized. The same string encoded in a legacy encoding for which there exists a normalizing transcoder would be fully normalized but not Unicode-normalized (since not in a Unicode encoding form).

In an XML or HTML context, the string "suc&#x00E7;on" is also both fully normalized and, if encoded in a Unicode encoding form, Unicode-normalized. Expanding "&#x00E7;" yields "suçon" as above, which contains no replaceable combining sequence.

The string "suçon", expressed as the sequence of six characters U+0073 U+0075 U+0063 U+0327 U+006F U+006E (U+0327 is the COMBINING CEDILLA) and encoded in a Unicode encoding form, is neither Unicode-normalized (since the combining sequence U+0063 U+0327 is replaceable by the precomposed U+00E7) nor fully normalized (since in a Unicode encoding form but not Unicode-normalized).

In an XML or HTML context, the string "suc&#x0327;on" is not fully normalized, regardless of encoding form, because expanding "&#x0327;" yields the sequence "suc¸on" which is not Unicode-normalized (c¸ is replaceable by ç). Unicode-normalization, however, is defined only for plain text, doesn't know that &#x0327; represents a character in XML or HTML and considers it just a sequence of characters. Therefore, the string "suc&#x0327;on" in a Unicode encoding form is Unicode-normalized since it contains no replaceable combining sequence. (The latter example does not imply that Unicode-normalization is sufficient to meet the normalization requirements of the Web; it just illustrates a case where Unicode-normalization and full normalization differ).

The string "<elem>/ foobar</elem>", where the '/' immediately after '<elem>' stands for the character U+0338 COMBINING LONG SOLIDUS OVERLAY, is neither Unicode-normalized nor fully normalized, since the U+0338 / combines with the '>' (yielding U+226F NOT GREATER-THAN).

Note: From this example, it follows that it is impossible to produce a normalized XML or HTML document containing the character U+0338 COMBINING LONG SOLIDUS OVERLAY immediately following an element tag, comment, CDATA section or processing instruction. It is noteworthy that U+0338 COMBINING LONG SOLIDUS OVERLAY also combines with '<', yielding U+226E NOT LESS-THAN. Consequently, U+0338 COMBINING LONG SOLIDUS OVERLAY should remain excluded from XML identifiers.

4.3 Responsibility for Normalization

[I] Producers MUST produce text data in normalized form, unless they are willing to accept the consequences (loss of integrity and security, high probability of rejection by recipients) of un-normalized data. For the purpose of W3C specifications and their implementations, the producer of text data is the sender of the data in the case of protocols and the tool that produces the data in the case of formats.

Note: As an optimization, it is perfectly acceptable for a system to define the producer to be the actual producer (e.g. a small device) together with a remote component (e.g. a server serving as a kind of proxy) to which normalization is delegated. In such a case, the communications channel between the device and proxy server is considered to be internal to the system, not part of the Web. Only data normalized by the proxy server is to be exposed to the Web at large.

Note: Normalization is the responsibility of the producer as a whole. This specification does not assign responsibility for normalization to any particular component of the producer (for instance a DOM implementation).

Note: Implementers of producer software are encouraged to delegate normalization to their respective data sources wherever possible. Examples of data sources are operating systems, libraries, and keyboard drivers.

[I] The recipients of text data MUST verify the normalization of data they receive and reject un-normalized data, unless they are willing to accept the consequences (loss of integrity and security) of un-normalized data. [I] Recipients MUST NOT normalize the data that they receive. [I] Recipients which transcode text data from a legacy encoding to a Unicode encoding form MUST use a normalizing transcoder.

Note: The prohibition of normalization by recipients is necessary to avoid the security issues mentionned in section 4.1.

[I] When a recipient returns un-normalized text to a sender (e.g. to indicate an error or fault), that recipient MAY return the text without normalizing it.

[I] If a software module functions as both a producer and a recipient of text data (e.g. a browser/editor), normalization MUST be applied in the producer part but MUST NOT be applied in the recipient part.

[I] Intermediate (recipient/producer) components whose role involves modification of text data MUST ensure that their modifications do not result in denormalization of any data exposed (sent on the network, saved to disk, returned in an API call, etc.).

Note: Consequently, an intermediate component, in a system that packages a payload in some control information, may modify the control information without having to renormalize the payload.

[I] In addition, software MUST behave as if normalization took place after each modification, so that any subsequent matching, indexing or other normalization-sensitive operations always behave as if they were dealing with normalized data.

Example: If the "z" is deleted from the (normalized) string "cz¸" (where "¸" represents a combining cedilla, U+0327), normalization is necessary to turn the denormalized result "c¸" into the properly normalized "ç". Analogous cases exist for insertion and concatenation. If the software that deletes the "z" later uses the string in a normalization-sensitive operation, it needs to normalize the string before this operation to ensure correctness; otherwise, normalization may be deferred until the data is exposed.

[I] Intermediate components whose role does not involve modification of the data (e.g. caching proxies) MUST NOT reject un-normalized data and MUST NOT perform normalization.

[S] In specifications of markup languages, syntax-significant characters MUST be chosen that do not combine with any other characters in NFC. This is to avoid problems such as U+0338 COMBINING LONG SOLIDUS OVERLAY combining with the '<' and '>' delimiters in XML (see the last example in section 4.2.3 Examples above.

5 Compatibility and Formatting Characters

This specification does not address the suitability of particular characters for use in markup languages, in particular formatting characters and compatibility equivalents. For detailed recommendations about the use of compatibility and formatting characters, see [UXML].

[S] Specifications SHOULD exclude compatibility characters in the syntactic elements (markup, delimiters, identifiers) of the formats they define.

6 String Identity Matching

One important operation that depends on early normalization is string identity matching [CharReq], which is a subset of the more general problem of string matching. There are various degrees of specificity for string matching, from approximate matching such as regular expressions or phonetic matching, to more specific matches such as case-insensitive or accent-insensitive matching and finally to identity matching. In the Web environment, where multiple encodings are used to represent strings, including some encodings which allow multiple representations for the same thing, identity is defined to occur if and only if the compared strings contain no user-identifiable distinctions. This definition allows strings to match when they differ only in their encodings (including escapes) or when strings in the same encoding differ only by their use of precomposed and decomposed character sequences, yet does not make equivalent strings that differ in case or accentuation.

To avoid unnecessary conversions and, more importantly, to ensure predictability and correctness, it is necessary for all components of the Web to use the same identity testing mechanism. [S][I] To meet this requirement and support the above definition of identity, string identity matching MUST be performed as if the following steps were followed:

  1. [S][I] Early uniform normalization to fully normalized form, as defined in 4.2.2 Fully Normalized Text

  2. [S][I] Conversion to a common encoding of UCS, if necessary

  3. [S][I] Expansion of all escapes

  4. [S][I] Binary comparison

[S][I] In accordance with section 4 Early Uniform Normalization, the first step MUST be performed by the producers of the strings to be compared. This ensures 1) that the identity matching process can produce correct results using the next three steps and 2) that a minimum of effort is spent on solving the problem.

[S][I] Forms of string matching other than identity SHOULD be based on the above steps. Taking into account normalization and escapes is necessary so that, for example, a case-insensitive match of 'suçon' against 'SU&#xC7;ON' or against 'SUC¸ON' returns TRUE.

Note: The expansion of escapes (step 3 above) is dependent on context, i.e. on which markup or programming language is considered to apply when the string matching operation is performed. Consider a search for the string "suçon" in an XML document containing "su&#xE7;on" but not "suçon". If the search is performed in a plain text editor, the context is plain text (no markup or programming language applies), the &#xE7; escape is not recognized, hence not expanded and the search fails. If the search is performed in an XML browser, the context is XML, the escape (defined by XML) is expanded and the search succeeds.

An intermediate case would be an XML editor that purposefully provides a view of an XML document with entity references left unexpanded. In that case, a search over that pseudo-XML view will deliberately not expand entities: in that particular context, entity references are not considered escapes and need not be expanded.

7 String Indexing

There are many situations where a software process needs to access a substring or to point within a string and does so by the use of indices, i.e. numeric "positions" within a string. Where such indices are exchanged between components of the Web, there is a need for an agreed-upon definition of string indexing in order to ensure consistent behavior. The requirements for string indexing are discussed in [CharReq], section 4. The two main questions that arise are: "What is the unit of counting?" and "Do we start counting at 0 or 1?".

Depending on the particular requirements of a process, the unit of counting may correspond to any of the definitions of a string provided in section 3.4 Strings (which all follow the pattern "a <foo string> is a sequence of <unit>s"). In particular:

  • [S][I] Character string is RECOMMENDED. (Example: [XPath])
  • [S][I] Code unit string MAY be used if the choice of character string would severely impact the efficiency of internal operations. (Example: use of UTF-16 in [DOM Level 1]).
  • Counting graphemes will become a good option where user interaction is the primary concern, once a suitable definition is widely accepted.

It is noteworthy that there exist other, non-numeric ways of identifying substrings which have favorable properties. For instance, substrings based on string matching are quite robust against small edits; substrings based on document structure (in structured formats such as XML) are even more robust against edits and even against translation of a document from one human language to another. [S] Consequently, specifications that need a way to identify substrings or point within a string SHOULD provide ways other than string indexing to perform this operation. [I][C] Users of such specifications (software developers, content developers) SHOULD prefer those other ways whenever possible.

Experience shows that more general, flexible and robust specifications result when individual characters are understood and processed as substrings, identified by a position before and a position after the substring. Understanding indices as boundary positions between the counting units also makes it easier to relate the indices resulting from the different string definitions. [S] Specifications SHOULD use this form of indexing, regardless of the choice of counting units. [S] APIs in addition SHOULD NOT specify single character or single encoding-unit arguments.

Example: uppercase('ß') cannot return the proper result ('SS') if the return type of the uppercase function is defined to be a single character.

The issue of index origin, i.e. whether we count from 0 or 1, actually arises only after a decision has been made on whether it is the units themselves that are counted or the positions between the units. [S] With the latter, starting with an index of 0 for the position at the start of the string is the RECOMMENDED solution, with the last index then being equal to the number of counting units in the string.

8 Character Encoding in URI References

According to the definition in [RFC 2396], URI references are restricted to a subset of US-ASCII. This RFC also specifies an escaping mechanism to encode arbitrary byte values, using the %HH convention. However, because the RFC does not define the mapping from characters to bytes, the %HH convention by itself is of limited use. This chapter defines how to address this issue in W3C specifications in a way consistent with the model defined in this document and with deployed practice.

[I] W3C specifications that define protocol or format elements (e.g. HTTP headers, XML attributes,...) whose role is that they be interpreted as URI references (or specific subsets of URI references, such as absolute URI references, URIs,...) MUST allow these protocol or format elements to contain characters disallowed by the URI syntax. The disallowed characters include all non-ASCII characters, plus the excluded characters listed in Section 2.4 of [RFC 2396], except for the number sign (#) and percent sign (%) characters and the square bracket characters re-allowed in [RFC 2732].

When passing such protocol or format elements to software components that cannot deal with anything else than characters legal in URI references, a conversion is needed. [S] W3C specifications MUST define when this conversion is to be made. [S] The conversion MUST take place as late as possible, i.e. all characters should be preserved as long as possible. [S] W3C specifications MUST specify that conversion to a URI reference is carried out as follows:

  1. [S] Each disallowed character is converted to UTF-8, resulting in one or more bytes.
  2. [S] The resulting bytes are escaped using the URI escaping mechanism (that is, each byte is converted to %HH, where HH is the byte value expressed using hexadecimal notation).
  3. [S] The original character is replaced by the resulting character sequence.

Example: The string "http://www.w3.org/People/Dürst/" is not a legal URI because the character "ü" is not allowed in URIs. The representation of "ü" in UTF-8 consists of two bytes with the values 0xC3 and 0xBC. The string is therefore converted to "http://www.w3.org/People/D%C3%BCrst/".

Note: [I-D URI-I18N] recently proposed the term Internationalized Resource Identifiers (IRI). A future version of this specification will adopt this term if it proves to be useful.

Request for feedback: There is currently a small but serious difference between [I-D URI-I18N] and the provisions above. US-ASCII characters which are not allowed in URIs (such as space, backslash, and curly brackets) are also not allowed in IRIs, but are allowed in protocol and format elements as defined above because they are escaped (as described above) during the conversion to a legal URI. We are requesting feedback on how to deal with this discrepancy.

Note: The intent of this chapter is not to limit URI references to a subset of US-ASCII characters forever, but to ensure that W3C technology correctly and predictably interacts with systems that are based on the definition of URI references while taking advantage of the capabilities of W3C technology to handle the UCS repertoire.

Note: The ability to use URI references with encodings other than UTF-8 is not affected by this chapter. However, in such a case, the %HH-escaped form has always to be used. As an example, if the HTTP server at www.w3.org used only ISO-8859-1, the above string, when used in a protocol or format element defined by a W3C specification, would always have to be given in the escaped form, "http://www.w3.org/People/D%FCrst/" ('ü' in ISO-8859-1 is 0xFC).

Note: Many current W3C specifications already contain provisions in accordance with this chapter. For [XML 1.0], see Section 4.2.2, External Entities. For [HTML 4.01], see Appendix B.2.1: Non-ASCII characters in URI attribute values, which also contains some provisions for backwards compatibility. For [XLink], see Section 5.4, Locator Attribute (href). Further information and links can be found at [Info URI-I18N].

[S] A W3C specification that defines new syntax for URIs, such as a new kind of fragment identifier, MUST specify that characters outside the US-ASCII repertoire are encoded in URIs using UTF-8 and %HH-escaping. This makes such new syntax fully interoperable with the above provisions.

Example: [XPointer] defines fragment identifiers for XML documents. An XPointer to an element with the ID résumé is xpointer(id('résumé')). The resulting fragment identifier is xpointer(id('r%C3%A9sum%C3%A9')) as the escaped UTF-8 of 'é' is %C3%A9. A URI reference to that element in a document called doc.xml is doc.xml#xpointer(id('r%C3%A9sum%C3%A9')). Using [XLink], and assuming the encoding of the XML document can represent 'é' directly, this can be written as xlink:href="doc.xml#xpointer(id('résumé'))". This looks very convenient and obvious, but it should be noted that it is only possible due to the fact that the encoding is aligned between representation (in XLink href) and interpretation (in XPointer).

9 Referencing the Unicode Standard and ISO/IEC 10646

Specifications often need to make references to the Unicode standard or International Standard ISO/IEC 10646. Such references must be made with care, especially when normative. The questions to be considered are:

  • Which standard should be referenced?
  • How to reference a particular version?
  • When to use versioned vs unversioned references?

ISO/IEC 10646  is developed and published jointly by ISO (the International Organisation for Standardisation) and IEC (the International Electrotechnical Commission). The Unicode Standard is developed and published by the Unicode Consortium, an organization of major computer corporations, software producers, database vendors, national governments, research institutions, international agencies, various user groups, and interested individuals. The Unicode Standard is comparable in standing to W3C Recommendations.

ISO/IEC 10646 and Unicode define exactly the same CCS (same repertoire, same code points) and encoding forms. They are actively maintained in synchrony by liaisons and overlapping membership between the respective technical committees. In addition to the jointly defined CCS and encoding forms, the Unicode Standard adds normative and informative lists of character properties, normative character equivalence and normalization specifications, a normative algorithm for bidirectional text and a large amount of useful implementation information. In short, Unicode adds semantics to the characters that ISO/IEC 10646 merely enumerates. Conformance to Unicode implies conformance to ISO/IEC 10646, see [Unicode 3.0] Appendix C.

[S] Since specifications in general need both a definition for their characters and the semantics associated with these characters, specifications SHOULD include a reference to the Unicode Standard, whether or not they include a reference to ISO/IEC 10646.  By providing a reference to The Unicode Standard implementers can benefit from the wealth of information provided in the standard and on the Unicode Consortium Web site. 

The fact that both ISO/IEC 10646 and Unicode are evolving (in synchrony) raises the issue of versioning: should a specification refer to a specific version of the standard, or should it make a generic reference, so that the normative reference is to the version current at the time of reading the specification? In general the answer is both. [S] A generic reference MUST be made if it is desired that characters allocated after a specification is published are usable with that specification. [S] A specific reference MAY be included to ensure that functionality depending on a particular version is available and will not change over time (an example would be the set of characters acceptable as Name characters in [XML 1.0], which is an enumerated list that parsers must implement to validate names).

Note: See http://www.unicode.org/unicode/standard/versions/#Citations for guidance on referring to specific versions of Unicode.

A generic reference can be formulated in two ways:

  1. By explicitly including a generic entry in the bibliography section of a specification and simply referring to that entry in the body of the specification. Such a generic entry contains text such as "... as it may from time to time be revised or amended".
  2. By including a specific entry in the bibliography and adding text such as "... as it may from time to time be revised or amended" at the point of reference in the body of the specification.

It is an editorial matter, best left to each specification, which of these two formulations is used. Examples of the first formulation can be found in the bibliography of this specification (see the entries for [ISO/IEC 10646] and [Unicode]). Examples of the latter, as well as a discussion of the versioning issue with respect to MIME charset parameters for UCS encodings, can be found in [RFC 2279] and [RFC 2781].

[S] All generic references to Unicode MUST refer to [Unicode 3.0] or later. [S] Generic references to ISO/IEC 10646 MUST be written such that they make allowance for the future publication of additional parts of the standard. They MUST refer to [ISO/IEC 10646-1:2000] or later, including any amendments.

Appendix A: Examples of Characters, Keystrokes and Glyphs

A few examples will help make sense of all this complexity (which is mostly a reflection of the complexity of human writing systems). Let us start with a very simple example: a user, equipped with a US-English keyboard, types "Foo", which the computer encodes as 16-bit values (the UTF-16 encoding of Unicode) and displays on the screen.

Example A.1: Basic Latin
Keystrokes Shift-f o o
Input characters F o o
Encoded characters (byte values in hex) 0x0046 0x006F 0x006F
Display Foo

The only complexity here is the use of a modifier (Shift) to input the capital F.

A slightly more complex example is a user typing "çé" on a French-Canadian keyboard, which the computer again encodes in UTF-16 and displays. We assume that this particular computer uses a fully composed form of UTF-16.

Example A.2: Latin with diacritics
Keystrokes ¸ c é
Input characters ç é
Encoded characters (byte values in hex) 0x00E7 0x00E9
Display çé

A few interesting things are happening here: when the user types the cedilla (¸), nothing happens except for a change of state of the keyboard driver; the cedilla is a dead key. When the driver gets the c, it provides a complete ç character to the system, which represents it as a single 16-bit code unit and displays a ç glyph. The user then presses the dedicated é key, which results in, again, a character represented by two bytes. Most systems will display this as one glyph, but it is also possible to combine two glyphs (the base letter and the accent) to obtain the same rendering.

On to a Japanese example: our user employs an input method to type "nihongo in Kanji characters", which the computer encodes in UTF-16 and displays.

Example A.3: Japanese
Keystrokes n i h o n g o <space><return>
Input characters Kanji character ni Kanji character hon Kanji character go
Encoded characters (byte values in hex) 0x65E5 0x672C 0x8A9E
Display nihongo in Kanji characters

The interesting aspect here is input: the user types Latin characters, which are converted on the fly to kana (not shown here), and then to kanji when the user requests conversion by pressing <space>; the kanji characters are finally sent to the application when the user presses <return>. The user has to type a total of nine keystrokes before the three characters are produced, which are then encoded and displayed rather trivially.

An Arabic example will show different phenomena:

Example A.4: Arabic
Keystrokes Arabic lam Arabic alif Arabic lam-alif Arabic ghayn Arabic ghayn
Input characters Arabic lam Arabic alif Arabic lam Arabic alif Arabic ghayn Arabic ghayn
Encoded characters (byte values in hex) 0x0644 0x0627 0x0644 0x0627 0x0639 0x0639
Display A few Arabic letters

Here the first two keystrokes each produce an input character and an encoded character, but the pair is displayed as a single glyph (Arabic lam-alif, a lam-alif ligature). The next keystroke is a lam-alif, which some Arabic keyboards have; it produces the same two characters which are displayed similarly, but this second lam-alif is placed to the left of the first one. The last two keystrokes produce two identical characters which are rendered by two different glyphs (a medial form followed to its left by a final form). We thus have 5 keystrokes producing 6 characters and 4 glyphs laid out right-to-left.

A final example in Tamil, typed with an ISCII keyboard, will illustrate some additional phenomena:

Example A.5: Tamil
Keystrokes Tamil ta Tamil aa Tamil na Tamil virama Tamil ka Tamil o
Input characters Tamil ta Tamil aa Tamil na Tamil virama Tamil ka Tamil o
Encoded characters (byte values in hex) 0x0B9F 0x0BBE 0x0B99 0x0BCD 0x0B95 0x0BCB
Display Tango in Tamil letters

Here input is straightforward, but note that contrary to the preceding accented Latin example, the diacritic Tamil virama (virama, vowel killer) is entered after the Tamil na to which it applies. Rendering is interesting for the last two characters. The last one (Tamil o) clearly consists of two glyphs which surround the glyph of the next to last character (Tamil ka).

A number of operations routinely performed on text can be impacted by the complexities of the world's writing systems. An example is the operation of selecting text on screen by a pointing device in a bidirectional (bidi) context. First, let's have some bidi text, in this case Arabic letters (written right-to-left) mixed with Arabic-Hindi digits (left-to-right):

Example A.6: Bidirectional text
In memory Arabic ayn Arabic dal Arabic dal <space> Arabic mim Arabic alif Arabic   ra Arabic sin <space> Arabic-Hindi digit one Arabic-Hindi digit nine Arabic-Hindi   digit nine Arabic-Hindi digit eight 
On screen Example of Arabic text

In the presence of bidi text, two possible selection modes must be considered. The first is logical selection mode, which selects all the characters logically located between the end-points of the user's mouse gesture. Here the user selects from between the first and second letters of the second word to the middle of the number. Logical selection looks like this:

Example A.7: Example of logical selection
In memory
Arabic ayn Arabic dal Arabic dal <space> Arabic mim  Arabic alif Arabic ra Arabic sin <space> Arabic-Hindi digit one Arabic-Hindi digit nine  Arabic-Hindi digit nine Arabic-Hindi digit eight 
On screen Example of logical selection

It is a consequence of the bidirectionality of the text that a single, continuous logical selection in memory results in a discontinuous selection appearing on the screen. This discontinuity, as well as the somewhat unintuitive behavior of the cursor, makes many users prefer a visual selection mode, which selects all the characters visually located between the end-points of the user's mouse gesture. With the same mouse gesture as before, we now obtain:

Example A.8: Example of visual selection
In memory
Arabic ayn Arabic dal Arabic dal <space> Arabic mim  Arabic alif Arabic ra Arabic sin  <space>  Arabic-Hindi digit one Arabic-Hindi digit nine   Arabic-Hindi digit nine Arabic-Hindi digit eight 
On screen Example of visual selection

In this mode, popular with users, a single visual selection range results in two logical ranges, which MUST be accommodated by protocols, APIs and implementations.

Acknowledgements

Special thanks go to Ian Jacobs for ample help with editing. Tim Berners-Lee and James Clark provided important details in the section on URIs. The W3C I18N WG and IG, as well as others, provided many comments and suggestions.

References

Normative References

IANA
Internet Assigned Numbers Authority, Official Names for Character Sets. (See ftp://ftp.isi.edu/in-notes/iana/assignments/character-sets.)
ISO/IEC 10646
ISO/IEC 10646-1:2000, Information technology -- Universal Multiple-Octet Coded Character Set (UCS) -- Part 1: Architecture and Basic Multilingual Plane, as, from time to time, amended, replaced by a new edition or expanded by the addition of new parts. (See http://www.iso.ch/ for the latest version.)
ISO/IEC 10646-1:2000
ISO/IEC 10646-1:2000, Information technology -- Universal Multiple-Octet Coded Character Set (UCS) -- Part 1: Architecture and Basic Multilingual Plane. (See http://www.iso.ch/iso/en/CatalogueDetailPage.CatalogueDetail?CSNUMBER=29819.)
MIME
Multipurpose Internet Mail Extensions (MIME). Part One: Format of Internet Message Bodies, N. Freed, N. Borenstein, RFC 2045, November 1996, <http://www.ietf.org/rfc/rfc2045.txt>. Part Two: Media Types, N. Freed, N. Borenstein, RFC 2046, November 1996. Part Three: Message Header Extensions for Non-ASCII Text, K. Moore, RFC 2047, November 1996. Part Four: Registration Procedures, N. Freed, J. Klensin, J. Postel, RFC 2048, November 1996. Part Five: Conformance Criteria and Examples, N. Freed, N. Borenstein, RFC 2049, November 1996.
RFC 2119
S. Bradner, Key words for use in RFCs to Indicate Requirement Levels, IETF RFC 2119. (See http://www.ietf.org/rfc/rfc2119.txt.)
RFC 2396
T. Berners-Lee, R. Fielding, L. Masinter, Uniform Resource Identifiers (URI): Generic Syntax, IETF RFC 2396, August 1998. (See http://www.ietf.org/rfc/rfc2396.txt.)
RFC 2732
R. Hinden, B. Carpenter, L. Masinter, Format for Literal IPv6 Addresses in URL's, IETF RFC 2732, 1999. (See http://www.ietf.org/rfc/rfc2732.txt.)
UTR #15
Mark Davis, Martin Dürst, Unicode Normalization Forms, Unicode Technical Report #15. (See http://www.unicode.org/unicode/reports/tr15/ for the latest revision). Revision 18 (November 1999) is at http://www.unicode.org/unicode/reports/tr15/tr15-18.html.
Unicode
The Unicode Consortium, The Unicode Standard -- Version 3.0, ISBN 0-201-61633-5, as updated from time to time by the publication of new versions. (See http://www.unicode.org/unicode/standard/versions/ for the latest version and additional information on versions of the standard and of the Unicode Character Database).
Unicode 3.0
The Unicode Consortium, The Unicode Standard -- Version 3.0, ISBN 0-201-61633-5. (See http://www.unicode.org/unicode/standard/versions/Unicode3.0.html.)

Other References

CharReq
Martin J. Dürst, Requirements for String Identity and Character Indexing Definitions for the WWW, W3C Working Draft. (See http://www.w3.org/TR/WD-charreq.)
Connolly
D. Connolly, Character Set Considered Harmful, W3C Note. (See http://www.w3.org/MarkUp/html-spec/charset-harmful.)
CSS2
Bert Bos, Håkon Wium Lie, Chris Lilley, Ian Jacobs, Eds., Cascading Style Sheets, level 2 (CSS2 Specification), W3C Recommendation. (See http://www.w3.org/TR/REC-CSS2.)
DOM Level 1
Vidur Apparao et al., Document Object Model (DOM) Level 1 Specification, W3C Recommendation. (See http://www.w3.org/TR/REC-DOM-Level-1/.)
HTML 4.0
Dave Raggett, Arnaud Le Hors, Ian Jacobs, Eds., HTML 4.0 Specification, W3C Recommendation, 18-Dec-1997 (See http://www.w3.org/TR/REC-html40-971218/.)
HTML 4.01
Dave Raggett, Arnaud Le Hors, Ian Jacobs, Eds., HTML 4.01 Specification, W3C Recommendation, 24-Dec-1999. (See http://www.w3.org/TR/html401/.)
I-D URI-I18N
Larry Masinter, Martin Dürst, Internationalized Resource Identifiers (IRI), Internet-Draft, January 2001, expired. (See http://www.w3.org/International/2001/01/draft-masinter-url-i18n-07.txt.)
Info URI-I18N
Internationalization: URIs and other identifiers. (See http://www.w3.org/International/O-URL-and-ident.)
ISO/IEC 9541-1
ISO/IEC 9541-1:1991, Information technology -- Font information interchange -- Part 1: Architecture. (See http://www.iso.ch/iso/en/CatalogueDetailPage.CatalogueDetail?CSNUMBER=17277 for the latest version.)
MathML2
David Carlisle, Patrick Ion, Robert Miner, Nico Poppelier, Eds., Mathematical Markup Language (MathML) Version 2.0, W3C Recommendation, 21 February 2001. (See http://www.w3.org/TR/MathML2/.)
Nicol
Gavin Nicol, The Multilingual World Wide Web, Chapter 2: The WWW As A Multilingual Application. (See http://www.mind-to-mind.com/i18n/articles/multilingual/multilingual-www.html.)
RFC 2070
F. Yergeau, G. Nicol, G. Adams, M. Dürst, Internationalization of the Hypertext Markup Language, IETF RFC 2070, January 1997. (See http://www.ietf.org/rfc/rfc2070.txt.)
RFC 2277
H. Alvestrand, IETF Policy on Character Sets and Languages, IETF RFC 2277, BCP 18, January 1998. (See http://www.ietf.org/rfc/rfc2277.txt.)
RFC 2279
F. Yergeau, UTF-8, a transformation format of ISO 10646, IETF RFC 2279, January 1998. (See http://www.ietf.org/rfc/rfc2279.txt.)
RFC 2781
P. Hoffman, F. Yergeau, UTF-16, an encoding of ISO 10646, IETF RFC 2781, February 2000. (See http://www.ietf.org/rfc/rfc2781.txt.)
SPREAD
SPREAD - Standardization Project for East Asian Documents Universal Public Entity Set. (See http://www.ascc.net/xml/resource/entities/index.html)
SVG
Jon Ferraiolo, Ed., Scalable Vector Graphics (SVG) 1.0 Specification, W3C Recommendation, 4 September 2001. (See http://www.w3.org/TR/SVG/.)
UTR #17
Ken Whistler, Mark Davis, Character Encoding Model, Unicode Technical Report #17. (See http://www.unicode.org/unicode/reports/tr17/.)
UXML
Martin Dürst and Asmus Freytag, Unicode in XML and other Markup Languages, Unicode Technical Report #20 and W3C Note. (See http://www.w3.org/TR/unicode-xml.)
XLink
Steve DeRose, Eve Maler, David Orchard, Eds, XML Linking Language (XLink) Version 1.0, W3C Recommendation, 27 June 2001. (See http://www.w3.org/TR/xlink/.)
XML 1.0
Tim Bray, Jean Paoli, C. M. Sperberg-McQueen, Eve Maler, Eds., Extensible Markup Language (XML) 1.0, W3C Recommendation. (See http://www.w3.org/TR/REC-xml.)
XML Japanese profile
MURATA Makoto Ed., XML Japanese Profile, W3C Note. (See http://www.w3.org/TR/japanese-xml/.)
XPath
James Clark, Steve DeRose, Eds, XML Path Language (XPath) Version 1.0, W3C Recommendation, 16 November 1999. (See http://www.w3.org/TR/xpath.)
XPointer
Steve DeRose, Eve Maler, Ron Daniel Jr., Eds, XML Pointer Language (XPointer) Version 1.0, W3C Candidate Recommendation, 11 September 2001. (See http://www.w3.org/TR/2001/CR-xptr-20010911/.)

Change Log (Non-Normative)

Changes since http://www.w3.org/TR/2001/WD-charmod-20010126/

Normalization: changed from "recipients MUST NOT normalize" to "recipients MUST check and reject un-normalized data".

Clarified conformance model, in particular introduced [S][I][C] specifiers for requirements. Still needs some work.

Made numerous other changes listed in Character Model for the World Wide Web 1.0 Last Call Comments (Members only).

Fixed countless typos and unclear/ambiguous sentences.

Updated references.