The goal of this document is to facilitate use of the Web by all people, regardless of their language, script, writing system, and cultural conventions, in accordance with the W3C goal of universal access. One basic prerequisite to achieve this goal is to be able to transmit and process the characters used around the world in a well-defined and well-understood way.

The main target audience of this document is W3C specification developers. This document defines conformance requirements for other W3C specifications. This document and parts of it can also be referenced from other W3C specifications.

Other audiences of this document include software developers, content developers, and authors of specifications outside the W3C. Software developers and content developers implement and use W3C specifications. This document defines some conformance requirements for software developers and content developers that implement and use W3C specifications. It also helps software developers and content developers to understand the character-related provisions in other W3C specifications.

The character model described in this document provides authors of specifications, software developers, and content developers with a common reference for consistent, interoperable text manipulation on the World Wide Web. Working together, these three groups can build a more international Web.

Topics addressed include encoding identification, early uniform normalization, string identity matching, string indexing, and URI conventions. Some introductory material on characters and character encodings is also provided.

Topics not addressed or barely touched include collation (sorting), fuzzy matching and language tagging. Some of these topics may be addressed in a future version of this specification.

At the core of the model is the Universal Character Set (UCS), defined jointly by The Unicode Standard and ISO/IEC 10646 . In this document, Unicode is used as a synonym for the Universal Character Set. The model will allow Web documents authored in the world's scripts (and on different platforms) to be exchanged, read, and searched by Web users around the world.

All W3C specifications must conform to this document (see section ). Authors of other specifications (for example, IETF specifications) are strongly encouraged to take guidance from it.

Since other W3C specifications will be based on some of the provisions of this document, without repeating them, software developers implementing W3C specifications must conform to these provisions.

This section provides some historical background on the topics addressed in this document.

Starting with Internationalization of the Hypertext Markup Language , the Web community has recognized the need for a character model for the World Wide Web. The first step towards building this model was the adoption of Unicode as the document character set for HTML.

The choice of Unicode was motivated by the fact that Unicode:

is the only universal character repertoire available,

W3C adopted Unicode as the document character set for HTML in . The same approach was later used for specifications such as XML 1.0 and CSS2 . Unicode now serves as a common reference for W3C specifications and applications.

The IETF has adopted some policies on the use of character sets on the Internet (see ).

When data transfer on the Web remained mostly unidirectional (from server to browser), and where the main purpose was to render documents, the use of Unicode without specifying additional details was sufficient. However, the Web has grown:

Data transfers among servers, proxies, and clients, in all directions, have increased.

In short, the Web may be seen as a single, very large application (see ), rather than as a collection of small independent applications.

While these developments strengthen the requirement that Unicode be the basis of a character model for the Web, they also create the need for additional specifications on the application of Unicode to the Web. Some aspects of Unicode that require additional specification for the Web include:

Choice of encoding forms (UTF-8, UTF-16, UTF-32).

It should be noted that such properties also exist in legacy encodings (where legacy encoding is taken to mean any character encoding not based on Unicode), and in many cases have been inherited by Unicode in one way or another from such legacy encodings.

The remainder of this document presents additional specifications and requirements to ensure an interoperable character model for the Web, taking into account earlier work (from W3C, ISO and IETF).

For the purpose of this specification, the producer of text data is the sender of the data in the case of protocols, and the tool that produces the data in the case of formats. The recipient of text data is the software module that receives the data.

Unicode code points are denoted as U+hhhh, where "hhhh" is a sequence of at least four, and at most six hexadecimal digits.

To be of any use in computers, in computer communications and in particular on the World Wide Web, characters must be encoded. In fact, much of the information processed by computers over the last few decades has been encoded text, exceptions being images, audio, video and numeric data. To achieve text encoding, a large variety of encoding schemes have been devised, which can loosely be defined as mappings between the character sequences that users manipulate and the sequences of bits that computers manipulate.

Given the complexity of text encoding and the large variety of schemes for character encoding invented throughout the computer age, a more formal description of the encoding process is useful. The process of defining a text encoding can be described as follows (see for a more detailed description):

A set of characters to be encoded is identified. The characters are pragmatically chosen to express text and to efficiently allow various text processes in one or more target languages. They may not correspond precisely to what users perceive as letters and other characters. The set of characters is called a repertoire.

In very simple cases, the whole encoding process can be collapsed to a single step, a trivial one-to-one mapping from characters to bytes; this is the case, for instance, for US-ASCII and ISO-8859-1.

Text data is said to be in a Unicode encoding form if it is encoded in UTF-8, UTF-16 or UTF-32.

Transcoding is the process of converting text data from one Character Encoding Form to another. Transcoders work only at the level of character encoding and do not parse the text; consequently, they do not deal with character escapes such as numeric character references (see ) and do not adjust embedded character encoding information (for instance in an XML declaration or in an HTML meta element).

A normalizing transcoder is a transcoder that converts from a legacy encoding to a Unicode encoding form and ensures that the result is in Unicode Normalization Form C (see ). For most legacy encodings, it is possible to construct a normalizing transcoder; it is not possible to do so if the encoding's repertoire contains characters not in Unicode.

Various specifications use the notion of a string, sometimes without defining precisely what is meant and sometimes defining it differently from other specifications. The reason for this variability is that there are in fact multiple reasonable definitions for a string, depending on one's intended use of the notion; the term string is used for all these different notions because these are actually just different views of the same reality: a piece of text stored inside a computer. This section provides specific definitions for different notions of string which may be reused elsewhere.

Byte string: A string viewed as a sequence of bytes representing characters in a particular encoding. This corresponds to a CES. As a definition for a string, this definition is most often useless, except when the textual nature is unimportant and the string is considered only as a piece of opaque data with a length in bytes. SSpecifications in general SHOULD NOT define a string as a byte string.

Code unit string: A string viewed as a sequence of code units representing characters in a particular encoding. This corresponds to a CEF. This definition is useful in APIs that expose a physical representation of string data. Example: For the DOM , UTF-16 was chosen based on widespread implementation practice.

Character string: A string viewed as a sequence of characters, each represented by a code point in Unicode . This is usually what programmers consider to be a string, although it may not match exactly what most users perceive as characters. This is the highest layer of abstraction that ensures interoperability with very low implementation effort. SThe character string definition of a string is generally the most useful and SHOULD be used by most specifications, following the examples of Production [2] of XML 1.0 , the SGML declaration of HTML 4.0 , and the character model of RFC 2070 .

Consider the string GOTHIC LETTER DAGS different from DIGIT ZERO comprising the characters U+10333 GOTHIC LETTER DAGS, U+2260 NOT EQUAL TO and U+0030 DIGIT ZERO, encoded in UTF-16 in big-endian byte order. The rows of the following table show the string viewed as a character string, code unit string and byte string, respectively:

Character string	U+10333 GOTHIC LETTER DAGS				U+2260 NOT EQUAL TO		U+0030 ZERO
Code unit string	D800		DF33		2260		0030
Byte string	D8	00	DF	33	22	60	00	30

Many Internet protocols and data formats, most notably the very important Web formats HTML, CSS and XML, are based on text. In those formats, everything is text but the relevant specifications impose a structure on the text, giving meaning to certain constructs so as to obtain functionality in addition to that provided by plain text. HTML and XML are markup languages, defining entities entirely composed of text but with conventions allowing the separation of this text into markup and character data. Citing from the XML 1.0 specification , section 2.4:

Text consists of intermingled character data and markup. [...] All text that is not markup constitutes the character data of the document.

For the purposes of this section, the important aspect is that everything is text, that is, a sequence of characters.

Since its early days, the Web has seen the development of a Reference Processing Model, first described for HTML in RFC 2070 . This model was later embraced by XML and CSS. It is applicable to any data format or protocol that is text-based as described above. The essence of the Reference Processing Model is the use of Unicode as a common reference. Use of the Reference Processing Model by a specification does not, however, require that implementations actually use Unicode. The requirement is only that the implementations behave as if the processing took place as described by the Model.

A specification conforms to the Reference Processing Model if all of the following apply:

SAll specifications that involve text MUST specify processing according to the Reference Processing Model.

Because encoded text cannot be interpreted and processed without knowing the encoding, it is vitally important that the character encoding scheme (see ) is known at all times and places where text is exchanged or processed. SSpecifications MUST either specify a unique encoding, or provide character encoding identification mechanisms such that the encoding of text can always be reliably identified. SWhen designing a new protocol, format or API, specifications SHOULD mandate a unique character encoding.

In text-based protocols or formats where characters can be either part of character data or of markup (see ), it is often the case that certain characters are designated as having certain specific protocol/format functions in certain contexts (e.g. < and & serve as markup delimiters in HTML and XML). These syntax-significant characters cannot be used to represent themselves in text data in the same way as all other characters do. Also, often formats are represented in an encoding that does not allow to represent all characters directly.

To express syntax-significant or unrepresentable characters, a technique called escaping is used. This works by creating an additional syntactic construct, defining additional characters or defining character sequences that have special meaning. Escaping a character means expressing it using such a construct, appropriate to the format or protocol in which the character appears; expanding an escape (or unescaping) means replacing it with the character that it represents.

Certain guidelines apply to the way specifications define character escapes. SThe guidelines in this document relating to the definition of character escapes MUST be followed when designing new W3C protocols and formats and SHOULD be followed as much as possible when revising existing protocols and formats.

Certain guidelines apply to content developers, as well as to software that generates content:

As explained at length in Requirements for String Identity Matching and String Indexing , the existence, in many character encoding schemes, of multiple representations for what users perceive as the same string makes it necessary to define character data normalization. Without a precise specification, it is not possible to determine reliably whether or not two strings are identical. Such a specification must take into account character encoding, the way normalization is to be performed and where or when (by sender or recipient) to perform it.

String identity is central to the correct functioning of much software, and in particular of large parts of the Web infrastructure (protocols, formats, etc.). Incorrect string matching can have far reaching consequences, including the creation of security holes. Consider a contract, encoded in XML, for buying goods: each item sold is described in an artículo element; unfortunately, artículo is subject to different representations in the character encoding of the contract. Suppose that the contract is viewed and signed by means of a user agent that looks for artículo elements, extracts them (matching on the element name), presents them to the user and adds up their prices. If different instances of the artículo element happen to be represented differently in a particular contract, then the buyer and seller may see (and sign) different contracts if their respective user agents perform string identity matching differently, which is fairly likely in the absence of a well-defined specification. The absence of a well-defined specification also means that there is no way to resolve the ensuing contractual dispute.

The Unicode Consortium provides four standard normalization forms (see Unicode Normalization Forms ). For use on the Web, this document defines W3C Text Normalization by picking the most appropriate of these (NFC) and additionally addressing the issues of legacy encodings and of character escapes (which can denormalize text when unescaped).

Roughly speaking, NFC is defined such that combining character sequences (a base character followed by one or more combining characters) are replaced, as far as possible, by canonically equivalent precomposed characters. Text in a Unicode encoding form is said to be in NFC if it doesn't contain any combining sequence that could be replaced and if any remaining combining sequence is in canonical order.

This document also specifies that normalization is to be performed early (by the sender) as opposed to late (by the recipient). The reasons for that choice are manifold:

Almost all legacy data as well as data created by current software is normalized (using NFC).

This section defines the responsibilities for normalization for various components and situations, based on the goal of early uniform normalization.

CAll content on the Web MUST be fully normalized.

IProducers MUST produce text data in normalized form, unless they are willing to accept the consequences (loss of integrity and security, high probability of rejection by recipients) of un-normalized data. For the purpose of W3C specifications and their implementations, the producer of text data is the sender of the data in the case of protocols and the tool that produces the data in the case of formats.

As an optimization, it is perfectly acceptable for a system to define the producer to be the actual producer (e.g. a small device) together with a remote component (e.g. a server serving as a kind of proxy) to which normalization is delegated. In such a case, the communications channel between the device and proxy server is considered to be internal to the system, not part of the Web. Only data normalized by the proxy server is to be exposed to the Web at large, as shown in the illustration below:

Illustration of a text producer defined as including a proxy. Illustration of a text producer defined as including a proxy.

IThe recipients of text data MUST verify the normalization of data they receive and reject un-normalized data, unless they are willing to accept the consequences (loss of integrity and security) of un-normalized data. IRecipients MUST NOT normalize the data that they receive. IRecipients which transcode text data from a legacy encoding to a Unicode encoding form MUST use a normalizing transcoder.

IWhen a recipient returns un-normalized text to a sender (e.g. to indicate an error or fault), that recipient MAY return the text without normalizing it.

IIf a software module functions as both a producer and a recipient of text data (e.g. a browser/editor), normalization MUST be applied in the producer part but MUST NOT be applied in the recipient part.

IIntermediate (recipient/producer) components whose role involves modification of text data MUST ensure that their modifications do not result in denormalization of any data exposed (sent on the network, saved to disk, returned in an API call, etc.).

ISoftware MUST behave as if normalization took place after each modification, so that any subsequent matching, indexing or other normalization-sensitive operations always behave as if they were dealing with normalized data.

IIntermediate components whose role does not involve modification of the data (e.g. caching proxies) MUST NOT reject un-normalized data and MUST NOT perform normalization.

SIn specifications of markup languages, syntax-significant characters MUST be chosen that do not combine with any other characters in NFC. This is to avoid problems such as U+0338 COMBINING LONG SOLIDUS OVERLAY combining with the < and > delimiters in XML (see the last example in section above).