Accesskey n skips to in-page navigation. Skip to the content start.

s_gotoW3cHome Internationalization
 

Character encodings for beginners

Intended audience: content authors, users, and anyone who is unsure about what a character encoding is, and wants a brief summary of how it affects them.

Updated 2010-09-08 20:47

Question

What is a character encoding, and why should I care?

Answer

First, why should I care?

If you use anything other than the most basic characters needed for English, people may not be able to read your text unless you say what character encoding you used.

For example, you may intend the text to look like this:

mojibake1.gif

but it may actually display like this:

mojibake2.gif

Not only does inadequate encoding information spoil the readability of displayed text, but it may mean that your data cannot be found by a search, or reliably processed in a number of other ways.

So what's a character encoding?

Words and sentences in text are created from characters. Examples of characters include the Latin letter á or the Chinese ideograph or the Devanagari character .

You may not be able to see some of the characters in this page because you don't have the necessary fonts. If you click on the place where you expected to see a character you will link to a graphic version. This page is encoded in UTF-8.

Characters are grouped into a character set (also called a repertoire). This is then called a coded character set when each character is assigned a particular number, called a code point. These code points will be represented in the computer by one or more bytes.

Basically, this means that all characters are stored in computers using a code, like the ciphers used in espionage. A character encoding is a key to unlock (ie. crack) the code. It is a set of mappings between the bytes representing numbers in the computer and characters in the coded character set. Without the key, the data looks like garbage.

The misleading term charset is often used to refer to what are in reality character encodings. You should be aware of this usage, but stick to using the term character encodings whenever you can.

Unfortunately, there are many different character sets and character encodings, ie. many different ways of mapping between bytes, code points and characters. The following panel provides more detail for those who are interested.

For example, in the coded character set called ISO 8859-1 (also known as Latin1) the decimal code point value for the letter é is 233. In ISO 8859-5, the same code point represents the Cyrillic character щ. These character sets contain fewer than 256 characters and map code points to byte values directly. So a code point with the value 233 is represented by a single byte with a value of 233. Note however that that byte may represent either é or щ, depending on the context.

There are other ways of handling characters from a range of scripts. For example, with the Unicode character set, you can represent both characters in the same set. In fact, Unicode contains, in a single set, most characters you are likely to ever need. While the value of 233 still represents the é, the Cyrillic character щ now has a code point value of 1097. This is too large a number to be represented by a single byte*. If you use the character encoding for Unicode text called UTF-8, щ will be represented by two bytes, but the code point value is not simply derived from the value of the two bytes spliced together – some more complicated decoding is needed. Other Unicode characters map to one, three or four bytes in the UTF-8 encoding.

UTF-8 is the most widely used way to represent Unicode text in web pages. But UTF-8 is only one of the possible ways of encoding Unicode characters. In other words, a single code point in the Unicode character set can actually be mapped to different byte sequences, depending on which encoding was used for the document. Unicode code points can be mapped to bytes using any one of the encodings called UTF-8, UTF-16 or UTF-32. The Devanagari character , with code point 2325 (which is 915 in hexadecimal notation), will be represented by two bytes when using the UTF-16 encoding (09 15), three bytes with UTF-8 (E0 A4 95), or four bytes with UTF-32 (00 00 09 15).

* Most bytes these days are made up of 8 bits. There are only 28 (ie. 256) unique ways of combining 8 bits.

Note also that code point numbers are commonly expressed in hexadecimal notation - ie. base 16. For example, 233 in hexadecimal form is E9. Unicode code point values are typically written in the form U+00E9.

There can be further complications beyond those described in the panel above (such as byte order and escape sequences), but the detail described there shows why it is important that the application you are working with knows which character encoding is appropriate for your data, and knows how to handle that encoding.

Most of the time, however, you will not need to know the details. You will just need to be sure that you consider the advice in the section How does this affect me? below.

How do fonts fit into this?

A font is a collection of glyph definitions, ie. definitions of shapes used to display characters.

Once your application has worked out what characters it is dealing with, it will then look in the font for glyphs in order to display or print those characters. (Of course, if the encoding information was wrong, it will be looking up glyphs for the wrong characters.)

A given font will usually cover a single character set, or in the case of a large character set like Unicode, just a subset of all the characters in the set. If your font doesn't have a glyph for a particular character, some applications will look for the missing character in other fonts on your system (which will mean that the glyph will look different from the surrounding text, like a ransom note). Otherwise you will typically see a square box, a question mark or some other character instead. For example:

mojibake3.gif

How does this affect me?

As a content author or developer, you need to choose the best character encoding for your content or data. Unicode encodings are often a good choice here, since you can use a single encoding to handle pretty much any character you are likely to meet. This greatly simplifies things. Using Unicode throughout your system also removes the need to track and convert between various character encodings.

Content authors need to check what encoding their editor or scripts are saving text in, and how to save text in the encoding of your choice. Note, however, that just declaring a different encoding won't change the bytes; you need to save the text in that encoding too.

Content authors also need to find out how to declare the character encoding used for the document format they are working with. You may also need to check that your server is serving documents with the right HTTP declarations.

Developers need to ensure that the various parts of the system can communicate with each other, understand which character encodings are being used, and support all the necessary encodings and characters.

The links below provide some further reading on these topics.

By the way

The article Character encodings: Essential concepts is worth a read at this point, as it provides some gentle introductions to related topics, such as Unicode, UTF-8, Character sets, coded character sets, and encodings, the document character set, character escapes and the HTTP header.

Tell us what you think (English).

Subscribe to an RSS feed.

New resources

Home page news

Twitter (Home page news)

‎@webi18n

Further reading

By: Richard Ishida, W3C.

Valid XHTML 1.0!
Valid CSS!
Encoded in UTF-8!

Content first published 2008-01-31 15:30. Last substantive update 2010-09-08 20:47 GMT. This version 2010-09-08 20:47 GMT

For the history of document changes, search for qa-what-is-encoding in the i18n blog.