What you need to know about the bidi algorithm and inline markup

Base direction (directional context)

The result of the bidirectional algorithm will depend on the overall base direction of the paragraph, block or page in which it is applied. The base direction is an important concept. It establishes a directional context which the bidi algorithm refers to at various points to decide how to handle the text.

In HTML the base direction is either set explicitly by the nearest parent element that uses the dir attribute, or, in the absence of such an attribute, is inherited from the default direction of the document, which is left-to-right.

To set the default direction of the whole HTML document to right-to-left, add dir="rtl" to the html tag. This will mean that all elements in the document will inherit a base direction of LTR, unless the dir attribute is used on an element to change the base direction within that element's scope.

Characters and directional typing

We already know that a sequence of Latin characters is rendered (ie. displayed) one after the other from left to right (we can see that in this paragraph). On the other hand, the bidi algorithm will render a sequence of Arabic or Hebrew characters one after the other from right to left.

This is independent of the current base direction, and works because each character in Unicode has an associated directional property. Most letters are strongly typed as LTR. Letters from right-to-left scripts are strongly typed as RTL.

Directional runs

When text with different directionality is mixed inline, the bidi algorithm makes a separate directional run out of each sequence of continguous characters with the same directionality.

So in the following example there are three directional runs:

Another way of looking at this is that changes in direction mark the boundaries of directional runs.

Note that you don't need any markup or styling to make this happen.

Here's the important bit: the order in which directional runs are displayed across the page depends on the current base direction.

In the example above, which has an overall context (ie. base direction) of LTR, you would read 'bahrain', then 'مصر', then 'kuwait'.

If you change the directional context of the example above by specifying that the html element or a parent element is RTL, you will change the order of the directional runs.

The characters in both cases are stored in memory in exactly the same order, but the visual ordering of the directional runs, when displayed, is reversed.

Neutral characters

Spaces and punctuation are not strongly typed as either LTR or RTL in Unicode, because they may be used in either type of script. They are therefore classed as neutral or weak characters. Characters are usually classified as 'weak' when they are associated with numbers. A small number of characters punctuation characters are initially classed as weak, but in a non-numeric context are treated like neutrals. In consquence, in this article we will refer to all punctuation as neutral characters.

This is where things begin to get interesting. When the bidi algorithm encounters characters with neutral directional properties (such as spaces and punctuation) it works out how to handle them by looking at the surrounding characters.

A neutral character between two strongly typed characters with the same directional type will also assume that directionality. So a neutral character between two RTL characters will be treated as a RTL character itself, and will have the effect of extending the directional run. This is why the three arabic words (including the intevening spaces, which as neutrals take on the direction of the surrounding characters) in the following example are read from right to left as a single directional run. (The arrows show the reading order.)

Note that you still don't need any markup or styling for this. And that there are still only three directional runs here.

The really interesting part comes when a space or punctuation falls between two strongly typed characters with different directionality, ie. at the boundary between directional runs. In such a case the neutral character (or characters) will be treated as if they have the same directionality as the base direction.

Even if there are several neutral characters between the two strongly typed characters, they will all be treated in the same way.

The implications of all this will become clearer as we work through the examples in the next section.

Numbers

Numbers in RTL scripts run left-to-right within the right-to-left flow, but they are handled a little differently than words by the bidi algorithm in that they always run left-to-right. They are said to have weak directionality. The two examples in the picture illustrate this difference.

The first example uses European digits, '1234', the second expresses the same number using Arabic-Indic digits, ١٢٣٤. In both cases, the digits in the number are read left-to-right.

Because it is weakly typed, the number is seen as part of the Arabic text, so the two Arabic words that surround the number are treated as part of the same directional run - even though the sequence of digits runs LTR on screen.

Note also that, alongside a number, certain otherwise neutral characters, such as currency symbols, will be treated as part of the number rather than a neutral. There are some other slight differences in the way numbers are handled that we don't need to discuss here.

Neutrals that appear at the wrong side of a directional run

We have seen that the bidirectional algorithm can cope well with a single level of bidirectional text, and that you could produce the result below without any additional markup or intervention:

Unfortunately, neutrals between different directional runs can sometimes be misinterpreted. Let's type some punctuation at the end of the Arabic phrase in the last example. By default we will see the following:

The quotation marks look OK, but the exclamation mark is in the wrong position. It should appear at the end of the Arabic text, ie. to the left, like this:

Given our understanding of the bidi algorithm we can easily understand why this happened. Because the exclamation mark was typed in between the last RTL letter 'ب' (on the left)‌ and the LTR letter 'i' (of the word 'in') its directionality is determined by the base direction of the paragraph, ie. LTR in this case. (Note that it makes no difference that there are actually two punctuation characters and a space in this position - they are all neutrals and so are all affected the same way.)

Because the exclamation mark is seen as LTR it joins the directional run that includes the text 'in Arabic'.

So how do we get the punctuation in the right place? We'll explain in a moment, but first let's take a look at another common problem.

Nesting base direction

If you have a situation where embedded text, such as a quotation, is also bidirectional, then you will need help. The next picture shows a Latin sentence that contains a Hebrew quote which, in turn, contains both Hebrew and Latin text. This is how it would appear if you rely solely on the bidirectional algorithm.

The order of the two Hebrew words is correct, but the because the text 'W3C' is part of the Hebrew phrase, it should appear on the left hand side of the quotation and the comma should appear between the Hebrew text and 'W3C'. In other words, the desired result is:

The problem arises because the directional flows are being ordered according to the LTR base direction of the paragraph. Inside the Hebrew quotation, however, the correct default ordering should be RTL.

To resolve this problem we need to explicitly change the base direction of the embedded phrase (ie. open a new embedding level).

Note: The examples shown here a fairly simple. Such sentences could commonly have more than two directional runs, in which case the issue is more obvious. Take, for instance, the following example where the top line shows the expected rendering, but the second line shows the default treatment using just the bidi algorithm.

A simple solution

A simple way to resolve both of the problems we just mentioned is to explicitly change the base direction of the embedded phrase. In HTML this would be done by enclosing the quotation in markup and assigning it a directionality of RTL using the dir attribute.

Note carefully how the span tag falls inside the quote marks - these are part of the surrounding English text.

Note also that this is likely to be a simple solution because it is likely that there is already markup around the embedded phrase. Such markup may be used to semantically label the text, to add language markup, or perhaps add a class attribute to apply appropriate styling. In the example above, a span element was used to declare the language, so adding the dir attribute is simple.

In markup languages other than HTML you may find a similar attribute to dir which you can use to produce the correct effect. If you don't have such an attribute you may have to resort to individually styling the appropriate inline markup, but it would probably be better to lobby your markup developer to provide you with one.

What if I can't use markup?

There are some situations where you may not be able to use the markup described in the previous section. In HTML these include the title element and any attribute value.

In these situations you can use invisible Unicode characters that produce the same results.

To replicate the effect of the markup described in the example above related to nested base directions, we can use pairs of characters to surround the embedded text. The first character is one of U+202B RIGHT-TO-LEFT EMBEDDING (RLE) or U+202A LEFT-TO-RIGHT EMBEDDING (LRE). This corresponds to the markup <span dir="rtl"> or <span dir="ltr">, respectively. The second character is U+202C POP DIRECTIONAL FORMATTING (PDF). This corresponds to the </span> in the markup. Below you can see how to apply this to the previous example.

These control characters should only be used for inline phrases, not for block elements such as paragraphs. In general, it is recommended that you use markup where it is available, rather than these character pairs, because it is easier to see and therefore manage the markup, and it is consistent with the approach used for block elements. Where markup is not available, of course, this is the only option.

When it comes to dealing with the misplaced neutrals described earlier, you can use the same approach. There is, however, a simpler alternative that works for cases such as the one shown, and is in fact recommended by the Unicode Standard rather than a pair of controls for these simple cases.

This involves placing an invisible, strongly-typed RTL Unicode character, after the exclamation mark. This puts our neutral punctuation between two strongly typed RTL characters, which results in the neutral becoming RTL too, and therefore the exclamation mark becomes a continuation of the right-to-left directional run.

The character designed for this purpose is the Unicode character U+200F RIGHT-TO-LEFT MARK (RLM). There is also a similar character, U+200E LEFT-TO-RIGHT MARK (LRM).

Note that in the example just shown the Arabic text is no longer marked up for language or styling. Also, because the character is invisible you may prefer to actually type in a numeric character reference (‎) as we did here, or, if available, a character entity (such as ‎ in HTML).

Adjacent, same-direction directional runs that are incorrectly ordered

Neutrals between same directional runs can also sometimes be misinterpreted. In our next example the list order is incorrect. The first two Arabic words should be reversed and the intervening comma, which is part of the English text, should appear immediately to the right of the first word.

What was wanted was:

The reason for the failure is that, with a strongly typed right-to-left (RTL) character on either side, the bidirectional algorithm sees the neutral comma as part of the Arabic text. It is interpreting the first two arabic words and the comma as a list in Arabic. In fact it is part of the English text, and should mark the boundary of two directional runs in Arabic.

In the previous section the neutral character thought it was part of the directional context established by the base direction, but wasn't; in this section the neutral character thinks it is part of the directional run, when it is really part of the overall context! No-one said life was simple...

Putting markup around the comma is a bit like cracking an egg with a hammer in this case.

A simple solution is to use an invisible, strongly-typed RLM Unicode character, next to the comma. This puts our neutral punctuation between strongly typed RTL and LTR characters and forces it to take on the directionality of the base direction, which is the left-to-right of the English text. That breaks the Arabic words into two separate directional runs, which are then ordered LTR in accordance with the base direction of the paragraph.

In the following example an escaped version of the character has been added after the exclamation mark and the result looks fine:

More examples

The examples we have used so far have been English and LTR based. The same principles apply for RTL text in languages such as Hebrew and Arabic.

Example: Unfortunately, on its own the bidirectional algorithm creates a real mess of the following text, which is in a right-to-left paragraph. (The red superscript numbers are just part of the diagram, not the text, and are there to identify the parentheses.)

Here's what we ought to see.

This problem was about adjacent, same-direction directional runs that are incorrectly ordered.

Although it may not be immediately obvious, actually the solution is trivial. Just insert an RLM after 'W3C' and you're done. It's really that simple!

If you're not convinced, here's the explanation. Unfortunately this will take a little longer to write than the fix.

Initially the parenthesis labelled  ¹ was between two LTR-typed characters, so its directionality was also LTR. This makes 'W3C (World Wide Web Consortium" a single directional run. (Don't worry about the shape of the parentheses for now - this will be explained shortly.)

The insertion of the RLM after 'W3C' changes the directionality of the parenthesis. Now it is between strongly-typed LTR and RTL characters, and so it takes on the directionality of the base direction, ie. the RTL direction of the paragraph as a whole. This makes sense if you see the parentheses as part of the Hebrew sentence syntax. The other parenthesis is also RTL, since it already appears between Latin and Hebrew characters.

This means we now have three directional runs at the beginning of the text. In memory they are ordered as follows: 'W3C', a RTL parenthesis, and the text "World Wide Web Consortium". Since the base direction for this paragraph is right-to-left, those runs are ordered from right to left - giving us the order we expect.

Example: The picture below shows what you are likely to see when relying solely on the bidirectional algorithm to display a MAC address number in a right-to-left context.

The next picture shows what you are likely to see when relying solely on the bidirectional algorithm.

This is particularly worrisome, since it's not obvious that the non-hinted order is incorrect.

Although there are more characters involved, this problem is about neutrals that appear at the wrong side of a directional run. The same approach can be used to fix this. You can either put markup around the MAC address and use dir to set a different base direction, or you can put an LRM at the beginning of the number.

Example: The picture below shows the the unexpected result of displaying a telephone number in a right-to-left context, where the area code is surrounded by parentheses, and where the number appears at the beginning of a line or after some right-to-left text.

The next picture shows what you expect to see.

Because these are numbers, the order applied by the bidirectional algorithm is slightly different from what we've seen before, but the fix is essentially the same. The correct rendering can be produced by adding a LRM character or escape just before the first parenthesis, or by using markup around the number and setting the base direction to LTR.

Mirrored characters

You may have noticed that, in addition to changing position, one of the parentheses in the previous example actually changed shape, too. This was completely automatic, and happens because these characters are what are known as mirrored characters in Unicode.

Mirrored characters are usually pairs of characters, such as parentheses, brackets, and the like, whose shape when displayed is dependent upon whether it is part of a LTR or RTL context. You do not have to change the character for the shape to change.

The ends of an opening parenthesis always face in the direction of the text flow. In the picture below, the parenthesis circled in red faces to the right in the top line because it is being treated as the opening parenthesis of some Latin text. In the lower version of the text, the same character (again circled in red) is treated as a opening parenthesis related to the Hebrew text (ie. the expanded name follows the acronym in reading order), and therefore faces the other way.

This means that, whether the stored content is in Arabic/Hebrew or Latin script, you would use the same LEFT PARENTHESIS character at the beginning of the parenthesized text. In other words, treat mirrored characters as if any word left in the name meant 'opening', and right meant 'closing'.