When selecting a page encoding, consider both current and future localization requirements, and the benefits of using the same encoding across all pages and all languages. These considerations make the use of Unicode an attractive choice for the following reasons:

UTF-8 and UTF-16 are both Unicode encodings. Since support for Unicode is currently limited to UTF-8 in many user agents, UTF-8 is usually the appropriate Unicode encoding. However, as user agent support for UTF-16 expands, UTF-16 will become an increasingly viable alternative.

Although there are other multi-script encodings (such as ISO-2022 and GB18030), Unicode generally provides the best combination of user agent and script support.

There are some situations where selecting a Unicode encoding is not practical. If content is encoded in a native encoding (legacy content or content originating from an external source) and the system lacks functionality for converting content between encodings, Unicode may greatly complicate implementation. If such a site is only required to serve single-script pages (containing languages that can be represented by a single native encoding), then the cost of using a Unicode encoding may outweigh the benefits. In this case, a native encoding (such as ISO-8859-1, Shift-JIS, etc.) may be a better choice.

Be sure to select an encoding that covers most all? of the characters required for the content, and (if it is a form) all of the characters that must be accepted as input.

Not all user agents support all page encodings, so it is important to understand which user agents must be able to render the page, and be sure that they have adequate support for the page encoding you have selected.

In general, user agents are most likely to support the commonly-used native character encodings for the major languages used on the web. Support for less commonly used encodings depends on the user agent. Older user agents, or user agents that operate under severe memory limitations, may not support UTF-8.

It is important to note that support for a given encoding does not necessarily imply support for all writing systems that encoding supports. For example, a user agent might support UTF-8, but not correctly display bidirectional Arabic text encoded in UTF-8. To display a page correctly, a user agents must support both the page encoding and the writing system.

.Point to an updated version of the table in hints & tips

The IANA charset registry shows a name plus a list of aliases for each registered charset value. One of these is identified as the preferred MIME name. Wherever you declare the character encoding, use the preferred MIME name in the charset value.

This maximizes the likelihood of interoperability.

Since the HTTP Content-Type header has precedence, and is also the easiest information to retrieve (user-agents do not have to parse the resource to get it), it is typically the preferred way to provide the character encoding for an HTML/XHTML document.

According to the HTML specification, in a case of conflict the HTTP charset declaration has the highest priority of all means of declaring the character set.

The reason you should do this 'where practical' is that pages may need to be viewed from a hard drive or CD, or some other way that doesn't involve interaction with the server. To provide for such cases, you should also declare the character encoding within the document itself.

Care should also be taken to ensure that the server-side settings are maintained if the file is moved or the server technology is changed.

To do this, use an XML declaration as shown below at the top of the file, and assign an IANA charset name to the encoding label of the declaration.

The checklist item above uses the phrase where practical because authors serving XHTML as text/html often choose not to include the XML declaration. This is because the declaration can cause display problems for some HTML browsers. For example, anything that appears before the DOCTYPE declaration forces Internet Explorer browsers, including version 6, into quirks mode rather than standards mode.

Because an XHTML document served as text/html is actually handled as HTML, the XML declaration is not actually required when the document is served. Note, however, that the XHTML specification recommends the use of both XML declaration and the meta charset declaration when XHTML is served as text/html.

In theory it is not necessary to specify the character encoding in an XML declaration for documents encoded in UTF-8 and UTF-16, since the XML parser treats these as the default. In practise, however, it is a good idea to label the document explicitly (as shown in the example above). For example, developers, testers, or translation production managers may want to perform a visual check of a document, or process the document using tools other than XML parsers.

You should declare the encoding inside the document even if the HTTP Content-Type parameter has been sent by the server. This ensures that the character encoding is always declared, even if the document is at some point not read from that server (eg. a local copy is read from disk, or the file is moved to another server that is not set up to serve the Content-Type parameter).

According to the XHTML 1.0 specification, in XHTML-conforming user agents, the value of the encoding declaration of the XML declaration takes precedence over the meta charset statement. It has a lower priority, however, than the HTTP Content-Type parameter.

To do this, use an XML declaration as shown below at the top of the file, and assign an IANA charset name to the encoding label of the declaration.

If you are serving XHTML as application/xhtml+xml, the encoding attribute is mandatory unless you are using UTF-8 or UTF-16 or declaring the encoding in the HTTP header.

In theory it is not necessary to specify the character encoding in an XML declaration for documents encoded in UTF-8 and UTF-16, since the XML parser treats these as the default. In practise, however, it is a good idea to label the document explicitly (as shown in the example above). For example, developers, testers, or translation production managers may want to perform a visual check of a document, or process the document using tools other than XML parsers.

You should declare the encoding inside the document even if the HTTP Content-Type parameter has been sent by the server. This ensures that the character encoding is always declared, even if the document is at some point not read from that server (eg. a local copy is read from disk, or the file is moved to another server that is not set up to serve the Content-Type parameter).

According to the XHTML 1.0 specification, in XHTML-conforming user agents, the value of the encoding declaration of the XML declaration takes precedence over the meta charset statement. It has a lower priority, however, than the HTTP Content-Type parameter.

The meta charset declaration is not needed when XHTML is served as application/xhtml+xml.

To do this, assign an IANA charset name as the charset value of a meta http-equiv statement.

This is good practise even if the character encoding has already been specified in the HTTP Content-Type parameter or any XML declaration (in XHTML). It ensures that the character encoding is always declared even if the document is at some point not read from that server (eg. a local copy is read from disk, or the file is moved to another server that is not set up to serve the Content-Type parameter).

Note that the XHTML specification recommends that the character encoding be declared in both the meta charset declaration and the XML declaration.

Note also that you should include a character encoding declaration even if your document uses a basic Latin encoding such as ISO 8859-1. For example, Japanese user agents will default to a Japanese encoding that does not include the accented letters, so they may not see you text correctly unless you specified the encoding.

In case of conflict, the Content-Type charset declaration and the XML declaration have precedence over the meta charset statement, according to the HTML 4.01 and XHTML 1.0 specifications. Is this true in practise? esp wrt IE?

This maximizes the likelihood that non-ASCII characters will be correctly recognized by the user agent.

The HTML spec says "The meta declaration must only be used when the character encoding is organized such that ASCII-valued bytes stand for ASCII characters (at least until the meta element is parsed). " How true is this?

Provide an overview of how to use escapes: hex / decimal NCRs, and entities - use is ok for invisible characters or others - explain what the exceptions might be

Since character set standards usually list character numbers as hexadecimal.

tbd

Discuss

Note that <font> and <basefont> tags are deprecated in the HTML4.01 Recommendation.

Easier maintenance

Faster translation AND localization.

Describe the evils of using <font> to cheat on the charset and represent other scripts.

Also don't assume that the font you've chosen will contain the characters needed for localized pages

tbd

tbd

This sets the default language for the whole document. It can be overridden for portions of the document as required.

Give reasons for use.

Give an example.

What about the use of the meta statement?

Give reasons for use. Extremely important for screen readers. Adaptation of styles.

Give an example.

Need to think about this - don't think it is supported by browsers.

Do we include detail here or under section on links?

Note that the HTML spec still says rfc1766, but this has been made obsolete by rfc3066.

Explain the basic principles here.

RFC3066 specifies that the two letter codes should be used where available, since this aids interoperability, and increases the likelihood of general recognition by browsers.

Because difficult to localize - at least CSS is likely to minimize required changes. Note that you can have a different style sheet per language. Give some examples and make it clear that this doesn't refer to rtl and ltr values.

This belongs in the CSS repository.

Because difficult to localize Note that before and after values are common in CSS3.

This will cause block elements and table columns to start on the right and flow from right to left. All block elements in the document will inherit this setting unless it is explicitly overridden.

No dir attribute is needed for documents that have a base directionality of ltr, since this is the default.

Having established the directionality at the html tag level, you should not use the dir attribute on other elements unless you want to change the directionality for that element. Unnecessary use of the dir attribute impacts bandwidth and potentially creates unnecessary additional work for page maintenance.

Although the HTML specification recommends the use of the dir attribute on the html element, this guideline is motivated more by practical considerations relating to user agent behavior.

According to the Microsoft article Authoring HTML for Middle Eastern Content, the following behaviors can only be expected in Internet Explorer 5 if the dir attribute is on the html element, rather than the body element.

check whether similar things apply to other user agents

Visual ordering of text was common for old user agents that didn't support the Unicode bidirectional algorithm. Text was stored in the source code in the same order you would expect to see it displayed. This also involved such things as disabling any line wrapping, explicit right-alignment of text in paragraphs and table cells, and reverse-ordering of table columns when translating from English to a language using a bidi script. The result is very fragile code that is difficult to maintain. For example, if you want to add a few words in the middle of a paragraph, you would have to move text to and from every line that followed it in the paragraph.

Visually ordered bidirectional HTML does not conform to the HTML specification.

With logical ordering text is stored in memory in the order in which it would normally be typed (and usually pronounced). The Unicode bidirectional algorithm is then applied by the browser to render the correct visual display.

Visual ordering isn't really seen much for Arabic. Since the Arabic letters are all joined up there was a stronger motivation on the part of Arabic implementers to enable the logical ordering approach.

It is usually best to use an Unicode encoding, such as UTF-8. This technique applies if, for some reason, you choose to serve your Hebrew page in an ISO encoding instead.

According to RFC1555 and RFC1556, there are special conventions for the use of charset parameter values to indicate bidirectional treatment in MIME mail, in particular to distinguish between visual, implicit, and explicit directionality. Visual refers to the practise of typing in the Hebrew characters in reverse order and preventing automatic line breaks. Formatting the document visually in this way is typically done to ensure reasonable display on older user agents that do not handle bidirectionality. Such documents do not conform to the HTML specification. Implicit is also called logical ordering, and refers to an approach where all characters in memory in the order in which it would normally be typed. Correct ordering for display is then done by a special algorithm (this is the preferred approach). Explicit refers to the use of explicit markers in the text to indicate directional changes.

The charset parameter value ISO-8859-8 for Hebrew denotes visual ordering, ISO-8859-8-i denotes implicit bidirectionality, and ISO-8859-8-e denotes explicit directionality.

Because HTML uses the Unicode bidirectional algorithm, conforming documents encoded using ISO 8859-8 must be labeled as ISO-8859-8-i. Explicit directional control is also possible with HTML, but cannot be expressed with ISO 8859-8, so "ISO-8859-8-e" should not be used.

Contrary to what is said in RFC1555 and RFC1556, ISO-8859-6 (Arabic) is not visual ordering.

Because directionality is an integral part of the document structure, markup should always be used to set the directionality for a document or chunk of information, or to indicate places in the text where the Unicode bidi algorithm is insufficient to achieve desired directionality.

The CSS2 specification recommends the use of markup for bidi text in HTML. In fact it goes as far as to say that conforming HTML user agents may ignore CSS bidi properties. This is because the HTML specification clearly defines the expected behavior of user agents with respect to the bidi markup.

See CSS vs. markup for bidi support for a fuller explanation.

Once you have established the appropriate directionality for the html element you will only need to apply bidi markup to a block element if you want that element's directionality to be different. The same applies for inline markup. Do not use inline bidi markup unless the Unicode bidi algorithm is insufficient on its own.

The following Arabic example shows bad usage. None of the dir attributes are needed if dir="rtl" was added to the html element. Removing them will significantly simplify the document, and reduce bandwidth - which may be an important consideration in countries where Arabic is spoken.

The Unicode Bidirectional Algorithm is applied to text that is stored in logical order, and determines the appropriate display direction of a sequence of characters. It does this on the basis of semantics associated with those characters by the Unicode Standard.

Occasionally the Unicode bidirectional algorithm is not sufficient to correctly order chunks of embedded text. Alternatively, you may want to override the effects of the bidirectional algorithm for a part of the page. In these cases you can apply additional markup to produce the ordering you want.

The following example illustrates the effect of applying a change in directionality to a block level element using the dir attribute.

Note, in particular, that the positions of the image and punctuation in the example above change relative to the text, because the overall directional flow has been changed. Note also, however, that the Hebrew characters are still read in the same direction. Their sequence is determined by the Unicode bidirectional algorithm, not by the dir attribute.

The content of all nested block elements will inherit directionality (unless of course a nested element explicitly changes its directionality using dir). Remember that the base directionality for a document should already be established by the html element. There is no need to add dir attributes to block level elements unless you want to apply a different direction to that set by the html tag or an explicit setting on a parent block element.

Visual user agents that support bidirectional display will typically right-align block elements in a rtl context, and vice versa. (See the example above.)

The dir attribute setting also affects the flow of columns in a table.

Note how the cells inherit the directionality set for the table. This produces the alignment of text in the cell, the order of text relative to the number, and the position of the question mark.

Note also that in most browsers, unlike other block elements, adding a dir attribute to the table will not cause the table to be aligned differently. It will only affect the order of columns and table content. If you want the table to be aligned with the other side of the content area you will need to wrap the table in another block element (eg. a div) that carries a dir attribute.Check that this applies for Mac browsers.

dir causes problem on DT element in IE6

You need to be familiar with the concepts in What you need to know about the bidi algorithm and inline markup to understand this technique.

Unfortunately, the bidirectional algorithm may not always produce the desired result with regard to the placement of punctuation. For instance, the overall context of the example below is LTR. If we introduce some punctuation between the Arabic and Latin letters it will produce the following (undesirable) result.

The exclamation mark is part of the Arabic phrase and should have appeared to its left. It appears to the right because it is between an Arabic and Latin character and the overall paragraph direction is LTR. It is therefore treated as part of the English text.

An easy way to fix this is to insert the Unicode character U+200F, called the RIGHT-TO-LEFT MARK, after the exclamation mark. There is a similar character, U+200E, called the LEFT-TO-RIGHT MARK.

The best way to represent these characters is with the pre-defined HTML character entities, &rlm; and &lrm;.

Now with two strong RTL characters on either side, the exclamation mark too will be treated as part of the RTL directional run and we will get the following (correct) result.

Note that it is possible to use actual Unicode characters or Numeric Character References (ie. ‎ and ‏) rather than the character entities mentioned above. The character entity is recommended because it provides maximum clarity in the code. A character code would not be visible, and a numeric value may be easily mistaken.

Actually that's not quite true. It looks fine in a LTR paragraph (see above), but not in a RTL context, where the entity name falls foul of the same problem! see below. You may be able to avoid this in some cases by breaking the line - as long as this doesn't introduce unwanted spaces.

You need to be familiar with the concepts in What you need to know about the bidi algorithm and inline markup to understand this technique.

The Unicode characters RLM (right-to-left mark) and LRM (left-to-right mark) can be useful to achieve the correct ordering of text items that are only separated by directionally neutral characters. We will show two examples of this.

In our first example, below, the list order is incorrect because the first two Arabic words should be reversed and the intervening comma, which is part of the English text, should appear immediately to the right of the first word. The reason for the failure is that, with a strongly typed right-to-left (RTL) character on either side, the bidirectional algorithm sees the neutral comma as part of the Arabic text.

The correct result was obtained by simply placing a &lrm; entity immediately after the comma. This has the effect of placing the neutral comma between two strongly typed characters, one left-to-right and the other right-to-left. Because neutral characters in this position take on the directionality of the overall context (here the paragraph), the bidi algorithm will now see it as part of the English left-to-right flow and will see the two Arabic words as separate.

In the second example, this time in a RTL Hebrew paragraph, the beginning of the sentence looks a real mess. This is because the text from W3C to Consortium is seen as a single directional run of LTR characters. (The second parenthesis from the right falls between LTR and RTL characters, so assumes the directionality of the paragraph - RTL.)

It is very simple to obtain the correct result. Simply put a &rlm; entity immediately after the hyphen. This causes the hyphen and the nearby parenthesis to be seen as part of the paragraph's text flow.

Note that the dir attribute is not appropriate to resolve this case.

At a simple level the Unicode bidirectional algorithm takes care of the reordering of inline text, but where there is nesting of directionality the dir attribute may need to be used.

The Unicode bidirectional algorithm organizes characters into directional runs - sequences of characters with the same directionality. Directionally neutral characters such as spaces and punctuation take on the directionality of surrounding characters, allowing directional runs to span several words. In the example below there are three directional runs - English, Arabic, and English. These are ordered according to the prevailing directionality of the paragraph - in this case left-to-right.

Unfortunately, the bidirectional algorithm alone does not produce the desired result if one of the directional runs contains mixed direction text, as can be seen in the following example.

The incorrect line of text is coded as a simple sequence of characters without any inline markup. Note that the order of the two Hebrew words is correct, but the text W3C should appear on the left hand side of the quotation and the comma should appear between the Hebrew text and W3C.

To get the correct result we have to create a new embedding level by surrounding the text within the quote marks with a span element and setting its dir attribute to rtl as shown here. (The language information has been omitted to make the example clearer.)

This causes the comma to take on the same RLT directionality as the whole span, and orders the Hebrew directional runs appropriately.

Note that we have used a span element to carry the dir attribute in this case. If the quote had already been surrounded by an element, the dir attribute should be attached to that. A span element should only be used where there is nothing else available.

Note also that we placed the span element inside the quotation marks, since these are a part of the English text.

Note that it may make sense to use markup rather than control codes, but it certainly doesn't make editing any easier unless the editing tool understands the markup you are applying and reorders the text appropriately.

@@ make sure to refer to the title element

There are a number of control characters in Unicode that can be used to create the same effect as markup for bidirectional text. These are:

Both Unicode in Markup Languages and the HTML 4.01 specification advise against using these when markup is available, and they particularly advise against mixing control codes and markup.

The references below need checking (esp for surviving ref to CSS)

Summarise and point to the bidi space Q&A

The shape of the glyphs used for a pair of mirrored characters will be determined at run time according to the directional context in which they appear.

bdo stands for bidirectional override. This inline element can be used to override the Unicode bidirectional algorithm if the dir attribute doesn't produce the desired result or if you want to produce a different result.

These represent two special characters in Unicode that can be used after the neutral character whose directionality is ambiguous. Problems typically arise for punctuation that falls between characters in a bidirectional script and characters in a non-bidirectional script. The entities are Unicode characters that are strongly typed, so they help disambiguate the context for the Unicode bidirectional algorithm.

The two digit year can cause confusion in parts of the world where multiple calendars are used. In these locations it may not be clear whether you are referring to the year using the local calendar or not.

Ambiguous dates cause confusion.

These make it clear for the user what order and format to use for data entry, and guarantee that the time will be recognized by the script, database or person that receives the information.