View source for Activity Streams direction notes

<html>
<style>
h2 { margin-top: 3em; }
h3 { margin-top: 2em; }
#toc { float:right; margin: 0 0 15px 15px; width: 40%; }
#firstHeading { display:none; }
h1 { font-size: 2em; line-height: 1.5em; font-family: 'Helvetica Neue', 'Segoe UI', helvetica, sans-serif; }
body {
    font-family: 'Helvetica Neue', 'Segoe UI', sans-serif;
    font-size: 1.1em;
    font-size: 1.1rem;
    font-weight: 300;
    line-height: 1.7;
    -webkit-text-size-adjust: 100%;
    -moz-text-size-adjust: 100%;
    -ms-text-size-adjust: 100%;
    text-size-adjust: 100%;
	color: #000;
}
div#content { margin-left: 9em; }
.tocsection-1 > a { display: none; }
.pass { color: green; font-weight: bold; }
.partial { color: orange; font-weight: bold; }
.fail { color:red; font-weight: bold; }
</style>
</html>

=Activity Streams direction notes=

<html>
<p style="color:red; font-size: 120%;">This is a draft for discussion and may be updated at any time.</p>
</html>

This page gathers observations about handling of text direction in Activity Streams. '''It doesn't make recommendations, it just aims to draw together useful background information, questions and lines of thought to help the Social Media WG determine the best course of action to support text direction in their spec.'''  It was written by r12a, and these are personal thoughts, not endorsed by the i18n WG.

In order to support text in right-to-left scripts, it is necessary to be able to:

# establish the overall base direction for a paragraph
# change the direction for a range of inline text where needed

'''These notes are only about the former''' as it relates to the JSON-based format used for Activity Streams.  Indications of direction change inline would just be carried over from whatever the original author used.

Three possible approaches have been discussed:

# use an object level direction property
# rely on the first strong directional character in the string to indicate base direction
# add direction metadata as a part of each string

This page contains thoughts about each of those, and for the second, attempts to understand what would be required in order to embed a name, summary or content property value in a target Web page for user consumption.

'''Note: it is difficult to represent bidi examples satisfactorily. For example, "פעילות הבינאום, W3C" doesn't actually show the expected position of 'W3C' when displayed, but also doesn't represent the order of characters in memory, since the first such character is פ. The way a string of that kind looks in the examples on this page should be ignored (although you can select them and get the right code point sequence).'''



==Ascertaining the base direction of strings as input to JSON==

Take a string typed into an HTML form input field such as the following:

[[File:rtl_hash_he.png]]

Unless the correct RTL base direction is applied to this string, it will be displayed as 

[[File:ltr_hash_he.png]]

The sequence of characters in memory is as follows. The point to note is that the sequence starts with LTR characters.

&#x202D;# b i d i &#x00A0; פ ע י ל ו ת&#x00A0;  ה ב י נ א ו ם , &#x00A0; W 3 C&#x202C;

If such a string is stored in JSON, it is important to capture the directional context in which it was created, ie. that the base direction for the paragraph was set to RTL.  If heuristics are not being used to establish the base direction of the stored JSON strings, it is important to detect the base direction for all RTL text, not just examples like that above.

If this is text created using an HTML form, in order to establish the base direction you need to know the computed direction of the form field into which the text was typed.  That direction may be established by inheritance from a parent element, such as the html tag, or may be set by the user with keyboard shortcuts.

This information about the base direction of the string needs to be captured and stored in some way by the JSON, so that the string can be displayed correctly by a consumer of the JSON data.


In the case where the string input by the user contains multiple paragraphs (ie. multiple lines separated by line breaks in a form input), we only need to know the base direction of the string as a whole. Any differences in base direction introduced between such paragraphs in a single input field, eg. such as in 

[[File:multiple_base_directions.png|300x75px]]

would need to be introduced by the user anyway, and the mechanism used for that would be part of the captured string.  In other words, we only need to capture and store the base direction set for the input as a whole.  

Note, however, that if the input field is a textarea with direction set to auto, it is expected that the base direction will be determined for each line (paragraph) on the basis of the first strong character in that line.

==Storing the base direction in JSON==



===Using an object level direction property===

Aside from any concerns about 'aspirational solutions', this approach doesn't work well for Activity Streams because one object can contain several string values, which may require base direction to be set to both RTL and LTR, eg. 

<pre>
{
  "@context": {
    "@value": "http://www.w3.org/ns/activitystreams",
     },
  "name": "Internationalization Activity, W3C",
  "type": "Note",
  "summary": "פעילות הבינאום, W3C"
  }
</pre>

In particular, when dealing with translated versions of strings the input sources may be different, and the base direction may therefore also have been different.

<pre>
{
  "@context": {
    "@value": "http://www.w3.org/ns/activitystreams",
    },
  "nameMap": {
    "he": "פעילות הבינאום, W3C",
    "ar": "نشاط التدويل، W3C"
    "es": "Actividad de internationalización, W3C",
    }
  "type": "Note",
  "summaryMap": {
    "he": "פעילות הבינאום, W3C",
    "ar": "نشاط التدويل، W3C"
    "es": "Actividad de internationalización, W3C",
    }
  }
</pre>

===Relying on first-strong characters===

One way to store the base direction for a string is to follow the convention that the first strong directional character in a string indicates the base direction for the whole string.

In the case of a string such as 

<pre>
"summary": "פעילות הבינאום, W3C"
</pre>

which starts with a RTL character, this is straightforward as long as the original base direction was also RTL.

In the case of this example (where the characters are shown in memory order), 

 "content": "&#x202D;# b i d i   ن ش ا ط   ا ل ت د و ي ل ، W 3 C&#x202C;"

which is a RTL phrase that started with a LTR strong character, '''it would be necessary to add a RTL strong character''', such as RLM to the start of the string to indicate the expected base direction of the string when consumed later.

(Needs further thought: This may not fully cover the multi-'paragraph' input, since if the input field is an HTML textarea control with direction set to auto, then the user inputting the data would expect each line to have the base direction automatically set by the first strong character in that line.  The only way to reproduce that rendering after the JSON has been consumed in the future would presumably be to apply auto direction to the output also. This cannot be represented by RLM or LRM at the start of the string, but on the other hand, each line ought to already start with the appropriate first-strong character needed to replicate the original input, so it may be that this is a non-issue.)

Btw, there are sometimes variations on the theme of first-strong detection. For example, HTML has rules about detecting the first strong character in markup that cause it to skip over certain things at the beginning of the string.


====Consuming JSON strings that use first-strong characters to indicate base direction====

Let's suppose that some JSON strings will eventually be displayed to an end user as part of a Web page. When the text is inserted into a target document, the application that reads the JSON needs to also use first-strong heuristics to detect the direction, and then build markup around the string to ensure that it is presented appropriately to the user. (This is probably easiest to achieve by using `dir=auto` on a block element around the string, or using `span dir=auto` or `bdi` around inline presentations.)

The point here is that '''the AS specification needs to make it clear''' that the AS JSON model uses the first-strong heuristic approach to represent the base direction, so that the consuming application knows where to look for the information about base direction that it will need while rendering the string in the destination. This is important because there are other heuristic approaches that ignore the first-strong character.  (Twitter, for example, does NOT currently use first-strong to determine the direction of text in a tweet.  It looks instead at the overall content of the tweet. Also, currently both Twitter and Facebook appear to actively strip out directional control characters before creating the HTML that supports their display. See https://www.w3.org/International/wiki/Bidi_in_social_media.) 

Btw, there are likely to be some special rules applied to the presentation of strings in some contexts. For example, Twitter locates, and treats specially, certain sequences of characters that begin with @ or #, isolating them and giving them an inline direction of LTR if they contain LTR characters.

====Strings that are enclosed in markup====

Unlike the name property, the summary and content properties in AS can contain markup. Here we are looking at a string that begins and ends with HTML markup. For example,

<pre>
"content" : "&lt;p>نشاط التدويل، W3C&lt;/p>"
</pre>

A consumer application that looks for the first strong character in such a string will always encounter a LTR character first, and so would need to, either:

# skip the markup, or
# would have to encounter a directional control character that had been put at the start of the string in order to represent the base direction.

If the incoming HTML happens to already contain a `dir` attribute on the surrounding tags, eg. 

<pre>
"content" : "&lt;span dir='rtl'>نشاط التدويل، W3C</span>"
</pre>

then, when the consuming application renders the text, the information in the `dir` attribute will presumably override any directional markup that might have been added to the very start of the JSON string.

It may be worth noting that putting a control character such as RLM before a &lt;p> tag would have no effect on the rendering of a string in HTML, since Unicode controls are only effective within the current paragraph (ie. they are inline constructs), and the &lt;p> tag initiates a new paragraph.

===Adding direction metadata as a part of each string===

This was a suggestion made by one of the Social Web WG members near the end of the joint telecon with i18n. The idea is that each string would have a structure which allowed for an optional direction field to be associated with it.

This stores the direction out of band, but unlike the object level direction property described above, it is only associated with a single string.

==Some tentative conclusions==

# using an object level direction property is probably not appropriate for Activity Streams objects, since one object can contain more than one string
# for the in-band solution (ie. first-strong)
## the application that creates the JSON strings needs to do more than just squirt the string into the JSON array, it needs to identify the base direction by examining the context of the string being input, and it may need to modify the strings where the first-strong heuristic would otherwise provide a misleading result. 
## applications taking data from the JSON strings need to know that they should apply first-strong heuristics in order to establish the requisite base direction for the string as a whole.