Character Sets and Encodings

This page provides some orientation for newcomers to Web internationalization who don't really know where to start. The aim is to ease you gently into some of the material on the site.

You can find a selection of more detailed articles using the links to the right. Once you get some ideas from this page, you will probably just use Learn to internationalize, or the site search.

What's it about?

A character set is a collection of letters and symbols used in a writing system. For example, the ASCII character set covers letters and symbols for English text, ISO-8859-6 covers letters and symbols needed for many languages based on the Arabic script, and the Unicode character set contains characters for most of the living languages and scripts in the world.

Characters in a character set are stored as one or more bytes in a computer. Each byte or sequence of bytes represents a given character. A character encoding is the key that maps a particular byte or sequence of bytes to particular characters that the font renders as text.

There are many different character encodings. If the wrong encoding is applied to the bytes in memory, the result will be unintelligible text. It is therefore important, if people are to read your content, that you correctly label the character encoding used.

Learn more...

Character encodings for beginners explains some of the basic concepts about character encodings, and why you should care.

Character encodings: Essential concepts provides explanations of terminology such as Unicode, character sets, coded character sets, character encodings, the document character set, and character escapes.

Choosing an encoding

Everyone developing content, whether content authors or programmers, should use the UTF-8 character encoding, unless there are very special reasons for using something else. (If you decide to not use UTF-8, you must choose one of the few encodings that are interoperably implemented across all browsers.)

 

Declaring and applying an encoding

Content developers and programmers must ensure that the character encoding used for a document or page is declared in the right way.

You must also ensure that your data is saved in the encoding you have chosen, it is not sufficient to just label it.

(Note that with XHTML, encoding declarations are not always straightforward; they require an understanding of 'standards' vs. 'quirks' modes, and the impact of the XML declaration.)

Content developers and webmasters may also need to ensure that the server delivers content with the correct character encoding declarations, since server settings can override in-document declarations.

Escapes

Escapes are a way of representing a character using only ASCII text. They provide a way of representing characters that are not available in the character encoding you are using, or a way of avoiding the use of the character for other reasons (such as when they may conflict with syntax). You should be clear on when and how these escapes should be used.

 

 

Web addresses

Web addresses can also include non-ASCII characters. The user does little other than click on the appropriate link or enter the text as they see it, the heavy lifting is done by the user agent, but you may be interested to know how this works.

Specification developers should design their specifications so that non-ASCII web addresses can be used.

Learn more...

HTML & CSS authors