Relation to ITS 1.0

ITS 2.0 has the following relations to ITS 1.0:

It adopts and maintains the following principles from ITS 1.0:

It adopts the use of data categories to define discrete units of functionality It adopts the separation of data category definition from the mapping of the data category to a given content format It adopts the conformance principle of ITS1.0 that an implementation only needs to implement one data category to claim conformance to ITS 2.0 ITS 2.0 supports all ITS 1.0 data category definitions and adds new definitions. ITS 2.0 adds a number of new data categories not found in ITS 1.0. While ITS 1.0 addressed only XML, ITS 2.0 specifies implementations of data categories in both XML and HTML5. Where ITS 1.0 data categories are implemented in XML, the implementation must be conformant with the ITS 1.0 approach to XML to claim conformance to ITS 2.0.

New Principles

ITS 2.0 also adds the following principles and features not found in ITS 1.0:

ITS 2.0 data categories are intended to be format neutral, with support for XML, HTML5, and NIF: a data category implementation only needs to support a single content format mapping in order to support a claim of ITS 2.0 conformance. ITS 2.0 provides algorithms to generate NIF out of HTML5 or XML with ITS 2.0 metadata. A global implementation of ITS 2.0 requires at least the XPath version 1.0. Other versions of XPath or other query languages (e.g., CSS selectors) can be expressed via a dedicated queryLanguage attribute.

As of the time of this writing, the new data categories included in ITS 2.0 are:

Below needs to be updated before each publication before last call. Domain Disambiguation Locale Filter Translation Agent Provenance Text Analysis Annotation External Resource Target Pointer Id Value Preserve Space Localization Quality Issue Localization Quality Précis MT Confidence Allowed Characters Storage Size

Typical Problems

The following examples sketch one of the issues that currently hinder efficient XML-related localization: the lack of a standard, declarative mechanism that identifies which parts of an XML document need to be translated. Tools often cannot automatically perform this identification.

Document with partially translatable content

In this document it is difficult to distinguish between those string elements that are translatable and those that are not. Only the addition of an explicit flag could resolve the issue.

Document with partially translatable content

Even when metadata are available to identify non-translatable text, the conditions may be quite complex and not directly indicated with a simple flag. Here, for instance, only the text in the nodes matching the expression //component[@type!='image']/data[@type='text'] is translatable.

Potential Users of ITS

The ITS specification aims to provide different types of users with information about what markup should be supported to enable worldwide use and effective internationalization and localization of content. The following paragraphs sketch these different types of users, and their usage of ITS. In order to support all of these users, the information about what markup should be supported to enable worldwide use and effective localization of content is provided in this specification in two ways:

abstractly in the data category descriptions: concretely in the ITS schemas:

Ways to Use ITS

The ITS specification proposes several mechanisms for supporting worldwide use and effective internationalization and localization of content. We will sketch them below by looking at them from the perspectives of certain user types. For the purpose of illustration, we will demonstrate how ITS can indicate that certain parts of content should or should not be translated.

A content author uses an attribute on a particular element to say that the text in the element should not be translated.

Use of ITS by content author

The its:translate="no" attributes indicate that the path and the cmd elements should not be translated.

A content author or information architect uses markup at the top of the document to identify a particular type of element or context in which the content should not be translated.

Use of ITS by information architect

The translateRule element is used in the header of the document to indicate that none of the path or cmd elements should be translated.

A processor may insert markup at the top of the document which links to ITS information outside of the document.

Use of ITS by processor

A rules element is inserted in the header of the document. It has a XLink href attribute used to link to an ITS external rule document.

ITS rule file shared by different documents

The rules element contains several ITS rules that are common to different documents. One of them is a translateRule element that indicates that no path or cmd element should be translated.

A schema developer integrates ITS markup declarations in his schema to allow users to indicate that specific parts of the content should not be translated.

Following schema example has to updated once we have final XSD schema for ITS 2.0 An XSD schema with ITS declaration

The declarations for the translate attribute is added to a group of common attributes commonAtts. This allows to use the translate attribute within the documents like in .

The first two approaches above can be likened to the use of CSS in . Using a style attribute, an XHTML content author may assign a color to a particular paragraph. That author could also have used the style element at the top of the page to say that all paragraphs of a particular class or in a particular context would be colored red.

Support for legacy HTML content

ITS 2.0 does not define how to use ITS in HTML versions prior version 5. Users are encouraged to migrate their content to HTML5 or XHTML. While it is possible to use its-* attributes introduced for HTML5 in older versions of HTML (such as 3.2 or 4.01) and pages using these attributes will work without any problems, its-* attributes will be marked as invalid in validators.

Local Approach

The document in shows how a content author may use the ITS translate attribute to indicate that all content inside the author element should be protected from translation. Translation tools that are aware of the meaning of this attribute can then screen the relevant content from the translation process.

ITS markup on elements in an XML document (local approach)

For this example to work, the schema developer will need to add the translate attribute to the schema as a common attribute or on all the relevant element definitions. Note how there is an expectation in this case that inheritance plays a part in identifying which content does have to be translated and which does not. Tools that process this content for translation will need to implement the expected inheritance.

Global Approach

The document in shows a different approach to identifying non-translatable content, similar to that used with a style element in , but using an ITS-defined element called rules. It works as follows: A document can contain a rules element (placed where it does not impact the structure of the document, e.g., in a “head” section). It contains one or more ITS rule elements (for example translateRule). Each of these specific elements contains a selector attribute. As its name suggests, this attribute selects the node or nodes to which a corresponding ITS information pertains. The values of ITS selector attributes are XPath absolute location paths (or CSS selectors if queryLanguage is set to "css"). Information for the handling of namespaces in these path expressions is taken from namespace declarations at the current rule element.

Caveat Related to XSLT-based Processing of ITS Selector Attributes

The values of ITS selector attributes are XPath absolute location paths. Accordingly, the following is a legitimate value:

myElement/descendant-or-self::*/@*

Unfortunately, values like this cause trouble when they are used in XSLT-based processing of ITS where the values of the ITS selector attributes are used as values of match attributes of XSLT templates. The reason for this is the following: match attributes may only contain a restriction/subset of XPath expressions, so-called patterns.

Basically the following restrictions hold for patterns:

only axes "child" or "attribute" allowed "//" or "/" possible id() or key() function possible predicates possible

Using only XSLT patterns in ITS selector attributes helps to avoid this issue. In many cases, this is possible by using patterns with predicates. The value above may for example be rewritten as follows:

*[self::myElement]/@* | myElement//*/@*

ITS global markup in an XML document (rule-based approach)

For this approach to work, the schema developer needs to add the rules element and associated markup to the schema. In some cases global rules may be sufficient to allow the schema developer to avoid adding other ITS markup (such as an translate attribute) to the elements and attributes in the schema. However, it is likely that authors will want to use attributes on markup from time to time to override the general rule.

For specification of the Translate data category information, the contents of the rules element would normally be designed by an information architect familiar with the document format and familiar with, or working with someone familiar with, the needs of the localization group.

The global, rule-based approach has the following benefits:

Content authors do not have to concern themselves with creating additional markup or verifying that the markup was applied correctly. ITS data categories are associated with sets of nodes (for example all p elements in an XML instance) Changes can be made in a single location, rather than by searching and modifying local markup throughout a document (or documents, if the rules element is stored as an external entity) ITS data categories can designate attribute values as well as elements. It is possible to associate ITS markup with existing markup (for example the term element in DITA)

The commonality in both examples above is the markup translate='no'. This piece of ITS markup can be interpreted as follows:

it pertains to the Translate data category the attribute translate holds a value of no

The ITS selector attribute allows:

ITS data category attributes to appear in global rules (even outside of an XML document or schema) ITS data categories attributes to pertain to sets of XML nodes (for example all p elements in an XML document) ITS markup to pertain to attributes ITS markup to associate with existing markup (for example the term element in DITA)

Global, Rule-based Selection

Global, rule-based selection is implemented using the rules element. It contains zero or more rule elements. Each rule element has a mandatory selector attribute. This attribute and all other possible attributes on rule elements are in the empty namespace and used without a prefix.

If there is more than one rules element in an XML document, the rules from each section are to be processed at the same precedence level. The rules sections are to be read in document order, and the ITS rules with them processed sequentially. The versions of these rules elements MUST NOT be different.

Depending on the data category and its usage, there are additional attributes for adding information to the selected nodes, or for pointing to existing information in the document. For example, the Localization Note data category can be used for adding notes to selected nodes, or for pointing to existing notes in the document. For the former purpose, a locNote element can be used. For the latter purpose, a locNotePointer attribute can be used.

Each data category allows users to add information to the selected nodes except for language information. Pointing to existing information is not possible for data categories that express a closed set of values, that is: Translate, Directionality, Locale Filter, and Elements Within Text.

The functionalities of adding information and pointing to existing information are mutually exclusive. That is: markup for pointing and adding MUST NOT appear in the same rule element.

Global rules can appear in the XML document they will be applied to, or in a separate XML document. The precedence of their processing depends on these variations. See also .

Local Selection in an XML Document

Local selection in XML documents is realized with ITS local attributes, the ruby element, or the span element. span serves just as a carrier for the local ITS attributes and a container for ruby.

The content model of span permits arbitrary nesting of ruby markup, since the rt element can contain span. An application of ruby, however, MUST not use such arbitrary nesting.

The data category determines what is being selected. The necessary data category specific defaults are described in .

Defaults for various data categories

By default the content of all elements in a document is translatable. The attribute its:translate="no" in the head element means that the content of this element, including child elements, should not be translated. The attribute its:translate="yes" in the title element means that the content of this element, should be translated (overriding the its:translate="no" in head). Attribute values of the selected elements or their children are not affected by local translate attributes. By default they are not translatable.

The default directionality of a document is left-to-right. The its:dir="rtl" in the quote element means that the directionality of the content of this element, including child elements and attributes, is right-to-left. Note that xml:lang indicates only the language, not the directionality.

The dir and translate attributes are not listed in the ITS attributes to be used in HTML5. The reason is that these two attributes are available in HTML5 natively, so there is no need to provide them as its- attributes. The definition of the two attributes in HTML5 is compatibly, that is it provides the same values and interpretation, as the definition for the two data categories Translate and Directionality.

Choosing Query Language

Rule elements have attributes which contain asbolute and relative selectors. Interpretation of these selectors depends on the actual query languge. The query language is set by queryLanguage attribute on rules element. If queryLanguge is not specified XPath 1.0 is used as a default query language.

XPath 1.0

XPath 1.0 is identified by xpath value in queryLanguage attribute.

CSS Selectors

CSS Selectors are identified by css value in queryLanguage attribute.

Additional query languages

ITS processors MAY support additional query languages. For each additional query language processor MUST define:

identifier of query language used in queryLanguage; rules for evaluating absolute selector to collection of nodes; rules for evaluating relative selector to collection of nodes.

Future versions of this specification MAY define additional query languages. The following query language identifiers are reserved: xpath, css, xpath2, xpath3, xquery, xquery3, xslt2, xslt3.

Variables in selectors

A param element (or several ones) can be placed as the first child element(s) of the rules element to define the default values of variables used in the various selectors used in the rules.

Implementation MUST support the param element for all query languages it supports and which at the same time define how variables are bind for evaluation of selector expression. Implementations SHOULD also provide means for changing the default values of the param elements. Such means are implementation-specific.

The param element has a required name attribute. The value of the name attribute is a QName, see . The content of the element is a string used as default value for the corresponding variable.

Using the param element to define the default value of a variable in a selector attribute.

The param element defines the default value for the $LCID variable. In this case, only the msg element with the attribute lcid set to 0x049 is seen as translatable.

In XSLT-based applications, it may make sense to map ITS parameters directly to XSLT parameters. To avoid naming conflicts one can use a prefix with the parameter name's value to distinguish between the ITS parameters and the XSLT parameters.

Definition

The Translate data category expresses information about whether the content of an element or attribute should be translated or not. The values of this data category are yes (translatable) or no (not translatable).

Implementation

The Translate data category can be expressed with global rules, or locally on an individual element. For elements, the data category information inherits to the textual content of the element, including child elements, but excluding attributes. The default is that elements are translatable and attributes are not.

GLOBAL: The translateRule element contains the following:

All selector related definitions has to be update to reflect queryLanguage A required selector attribute. It contains an absolute selector which selects the nodes to which this rule applies. A required translate attribute with the value yes or no. The Translate data category expressed globally

The translateRule element specifies that the elements code must not be translated.

LOCAL: The following local markup is available for the Translate data category:

A translate attribute with the value yes or no.

It is not possible to override the Translate data category settings of attributes using local markup. This limitation is consistent with the advised practice of not using translatable attributes. If attributes need to be translatable (e.g., an HTML alt attribute), then this must be declared globally.

The Translate data category expressed locally

The local its:translate="no" specifies that the content of panelmsg must not be translated.

The Translate data category expressed locally in HTML5

The local translate="no" attribute specifies that the content of span must not be translated.

Definition

The Localization Note data category is used to communicate notes to localizers about a particular item of content.

This data category can be used for several purposes, including, but not limited to:

Tell the translator how to translate parts of the content Expand on the meaning or contextual usage of a specific element, such as what a variable refers to or how a string will be used in the user interface Clarify ambiguity and show relationships between items sufficiently to allow correct translation (e.g., in many languages it is impossible to translate the word enabled in isolation without knowing the gender, number and case of the thing it refers to.) Indicate why a piece of text is emphasized (important, sarcastic, etc.)

Two types of informative notes are needed:

An alert contains information that the translator must read before translating a piece of text. Example: an instruction to the translator to leave parts of the text in the source language. A description provides useful background information that the translator will refer to only if they wish. Example: a clarification of ambiguity in the source text.

Editing tools may offer an easy way to create this type of information. Translation tools can be made to recognize the difference between these two types of localization notes, and present the information to translators in different ways.

Implementation

The Localization Note data category can be expressed with global rules, or locally on an individual element. For elements, the data category information inherits to the textual content of the element, including child elements, but excluding attributes.

GLOBAL: The locNoteRule element contains the following:

A required selector attribute. It contains an absolute selector which selects the nodes to which this rule applies. A required locNoteType attribute with the value description or alert.

Exactly one of the following:

A locNote element that contains the note itself and allows for local ITS markup. A locNotePointer attribute that contains a relative selector pointing to a node that holds the localization note. A locNoteRef attribute that contains a URI referring to the location of the localization note. A locNoteRefPointer attribute that contains a relative selector pointing to a node that holds the URI referring to the location of the localization note. The locNote element

The locNoteRule element associates the content of the locNote element with the message with the identifier 'DisableInfo' and flags it as important. This would also work if the rule was in an external file, allowing to provide notes without modifying the source document.

The locNotePointer attribute

The locNotePointer attribute is a relative selector pointing to a node that holds the note.

The locNoteRef attribute

The locNoteRule element specifies that the message with the identifier 'NotFound' has a corresponding explanation note in an external file. The URI for the exact location of the note is stored in the locNoteRef attribute.

The locNoteRefPointer attribute

The locNoteRefPointer attribute contains a relative selector pointing to a node that holds the URI referring to the location of the note.

LOCAL: The following local markup is available for the Localization Note data category:

One of the following:

A locNote attribute that contains the note itself. A locNoteRef attribute that contains a URI referring to the location of the localization note. An optional locNoteType attribute with the value description or alert. If the locNoteType attribute is not present, the type of localization note will be assumed to be description. The Localization Note data category expressed locally The Localization Note data category expressed locally in HTML5

It is generally recommended to avoid using attributes to store text, however, in this specific case, the need to provide the notes without interfering with the structure of the host document is outweighing the drawbacks of using an attribute.

Definition

The Terminology data category is used to mark terms and optionally associate them with information, such as definitions. This helps to increase consistency across different parts of the documentation. It is also helpful for translation.

Existing terminology standards such as and its derived formats are about coding terminology data, while the ITS Terminology data category simply allows to identify terms in XML documents and optionally to point to corresponding information.

Implementation

The Terminology data category can be expressed with global rules, or locally on an individual element. There is no inheritance. The default is that neither elements nor attributes are terms.

GLOBAL: The termRule element contains the following:

A required selector attribute. It contains an absolute selector which selects the nodes to which this rule applies. A required term attribute with the value yes or no.

None or exactly one of the following:

A termInfoPointer attribute that contains a relative selector pointing to a node that holds the terminology information. A termInfoRef attribute that contains a URI referring to the resource providing information about the term. A termInfoRefPointer attribute that contains a relative selector pointing to a node that holds the URI referring to the location of the terminology information. Usage of the termInfoPointer attribute Usage of the termInfoRef attribute Usage of the termInfoRefPointer attribute

LOCAL: The following local markup is available for the Terminology data category:

A term attribute with the value yes or no. An optional termInfoRef attribute that contains a URI referring to the resource providing information about the term. The Terminology data category expressed locally The Terminology data category expressed locally in HTML5

Definition

The Directionality data category allows the user to specify the base writing direction of blocks, embeddings and overrides for the Unicode bidirectional algorithm. It has four values: ltr, rtl, lro and rlo.

ITS defines only the values of the Directionality data category and their inheritance. The behavior of text labeled in this way may vary, according to the implementation. Implementers are encouraged, however, to model the behavior on that described in the CSS 2.1 specification or its successor. In such a case, the effect of the data category's values would correspond to the following CSS rules:

Data category value: ltr (left-to-right text)

CSS rule: *[dir="ltr"] { unicode-bidi: embed; direction: ltr}

Data category value: rtl (right-to-left text)

CSS rule: *[dir="rtl"] { unicode-bidi: embed; direction: rtl}

Data category value: rlo (left-to-right override)

CSS rule: *[dir="lro"] { unicode-bidi: bidi-override; direction: ltr}

Data category value: rlo (right-to-left text)

CSS rule: *[dir="rlo"] { unicode-bidi: bidi-override; direction: rtl}

More information about how to use this data category is provided by .

Implementation Examples for HTML5 need to be added; some values need to added to dir to reflect HTML5.

The Directionality data category can be expressed with global rules, or locally on an individual element. For elements, the data category information inherits to the textual content of the element, including child elements and attributes. The default is that both elements and attributes have the directionality of left-to-right.

GLOBAL: The dirRule element contains the following:

A required selector attribute. It contains an absolute selector which selects the nodes to which this rule applies. A required dir attribute with the value ltr, rtl, lro or rlo. Document which needs global rules for directionality

In this document the right-to-left directionality is marked using a direction attribute with a value rtlText.

The Directionality data category expressed with global rules

The dirRule element indicates that all elements with an attribute direction="rtlText" have right-to-left content.

LOCAL: The following local markup is available for the Directionality data category:

A dir attribute with the value ltr, rtl, lro or rlo. The Directionality data category expressed locally

On the first quote element, the its:dir="rtl" attribute indicates a right-to-left content.

The Directionality data category expressed locally in HTML5

Definition

The Ruby data category is used for a run of text that is associated with another run of text, referred to as the base text. Ruby text is used to provide a short annotation of the associated base text. It is most often used to provide a reading (pronunciation) guide.

Implementation Examples for HTML5 need to be added;

The Ruby data category can be expressed with global rules, or locally. There is no inheritance.

GLOBAL: The rubyRule element contains the following:

A required selector attribute. It contains an absolute selector which selects the nodes to which this rule applies. This is the ruby base text. An optional rubyPointer attribute that contains a relative selector pointing to a node that corresponds to the ruby element. An optional rpPointer attribute that contains a relative selector pointing to a node that corresponds to the ruby parenthesis. An optional rubyText element that contains the ruby text. An optional rtPointer attribute that contains a relative selector pointing to a node that corresponds to the ruby text.

Where legacy formats do not contain ruby markup, it is still possible to associate ruby text with a specified range of document content using the rubyRule element.

Adding ruby text with a rubyRule element

LOCAL: In a document, the Ruby data category is realized with a ruby element. It contains the following:

The ruby base text or span element that contains the ruby base text and allows for local ITS markup. An rp element that contains the ruby parenthesis. It is used in case of simple markup to specify characters that can denote the beginning and end of ruby text when user agents do not have other ways to present ruby text distinctively from the base text. An rt element that contains the ruby text and allows for local ITS markup.

All these elements share the attributes of the span element.

The Ruby data category expressed locally

The structure of the content model for the ruby element is identical with the structure of ruby markup as defined in .

The structure of ruby defined in section 5.4 of is also compliant with ruby defined in this specification.

Need to reevaluate above statement related to ODF.

Definition

The element langRule is used to express the language of a given piece of content. The langPointer attribute points to the markup which expresses the language of the text selected by the selector attribute. This markup MUST use values that conform to . The recommended way to specify language identification is to use xml:lang. The langRule element is intended only as a fall-back mechanism for documents where language is identified with another construct.

Pointing to language information via langRule

The following langRule element expresses that the content of all p elements (including attribute values and textual content of child elements) are in the language indicated by mylangattribute, which is attached to the p elements, and expresses language using values conformant to .

The Language Information data category only provides for rules to be expressed at a global level. Locally users are able to use xml:lang (which is defined by XML) or an attribute specific to the format in question (as in ).

xml:lang is the preferable means of language identification. To ease the usage of xml:lang, a declaration for this attribute is part of the non-normative XML DTD and XML Schema document for ITS markup declarations. There is no declaration of xml:lang in the non-normative RELAX NG document for ITS, since in RELAX NG it is not necessary to declare attributes from the XML namespace.

Applying the Language Information data category to xml:lang attributes using global rules is not necessary, since xml:lang is the standard way to specify language information in XML. xml:lang is defined in terms of RFC 3066 or its successor ( is the "Best Common Practice" for language identification and encompasses and its successors.)

Add something about HTML5 lang

Implementation

The Language Information data category can be expressed only with global rules. For elements, the data category information inherits to the textual content of the element, including child elements and attributes. There is no default.

GLOBAL: The langRule element contains the following:

A required selector attribute. It contains an absolute selector which selects the nodes to which this rule applies. A required langPointer attribute that contains a relative selector pointing to a node that contains language information.

Definition

The Elements Within Text data category reveals if and how an element affects the way text content behaves from a linguistic viewpoint. This information is for example relevant to provide basic text segmentation hints for tools such as translation memory systems. The values associated with this data category are:

yes : The element and its content are part of the flow of its parent element. For example the element strong in :

<strong>Appaloosa horses</strong> have spotted coats.

nested : The element is part of the flow of its parent element, its content is an independent flow. For example the element fn in :

Palouse horses<fn>A Palouse horse is the same as an Appaloosa.</fn> have spotted coats.

no : The element splits the text flow of its parent element and its content is an independent text flow. For example the element p when inside the element li in DITA or XHTML:

<li>Palouse horses: <p>They have spotted coats.</p> <p>They have been bred by the Nez Perce.</p> </li>

Implementation

The Elements Within Text data category can be expressed with global rules, or locally on an individual element. There is no inheritance. The default is that elements are not within text.

GLOBAL: The withinTextRule element contains the following:

A required selector attribute. It contains an absolute selector which selects the nodes to which this rule applies. A required withinText attribute with the value yes, no or nested. Specifying elements within text with a withinTextRule element

LOCAL: The following local markup is available for the Elements Within Text data category:

A withinText attribute with the values yes, no or nested. The Elements Within Text data category expressed locally The Elements Within Text data category expressed locally in HTML5

Definition

The Domain data category is used to identify the topic or subject of a given content. Such information allows to make more relevant lingusitic choices during various processes.

Examples of usage include:

Allowing machine translation systems to select the most appropriate engine and rules to translate the content. Providing a general indication of what terminology collection should be used by a translator.

This data category addresses various challenges:

Often domain-related information already exist in the document (e.g. keywords in the HTML meta element). The Domain data category provides a mechanism to point to this information. There are many flat or structured lists of domain related values, keywords, key phrases, classification codes, ontologies, etc. The Domain data category does not propose its own given list. Instead it provides a mapping mechanism to associate the values in the document with the values used by the consumer tool.

Implementation

The Domain data category can be expressed only with global rules. For elements, the data category information inherits to the textual content of the element, including child elements and attributes. There is no default.

The information provided by this data category is a comma-separated list of one or more values which is obtained by applying the following algorithm:

Set the initial value of the resulting string as a empty string. Get the list of nodes resulting of the evaluation of the domainPointer attribute. For each node: If the node value contains a COMMA (U+002C): Split the node value into separate strings using the COMMA (U+002C) as separator. For each string: Trim the leading and trailing white spaces of the string. Check if there is a mapping for the string: If one is found: Add the corresponding value to the result string. Otherwise (if no mapping is found): Add the string to the result string. If the node value does not contain a COMMA (U+002C): Trim the leading and trailing white spaces of the string. Check if there is a mapping for the string: If one if found: Add the corresponding value to the result string. Otherwise (if no mapping is found): Add the string to the result string. Return the resulting string.

GLOBAL: The domainRule element contains the following:

A required selector attribute. It contains an absolute selector which selects the nodes to which this rule applies. A required domainPointer attribute that contains a relative selector pointing to a node that contains the domain information. An optional domainMapping attribute that contains a comma separated list of mappings between values in the content and consumer tool specific values. The left part of the pair is part of the source content and unique within the mapping. The right part of the mapping belongs to the consumer tool. Several left parts can map to a single right part. The values in the left or the right part of the mapping may contain spaces; in that case they MUST be delimited by quotation marks, that is pairs of APOSTROPHE (Unicode code point U+0027) or QUOTATION MARK (U+0023).

Although the domainMapping attribute it is optional, its usage is recommended. Many commercial machine translation systems use their own domain definitions; the domainMapping attribute will foster interoperability between these definitions and metadata items like DC.subject in Web pages or other types of content.

Values used in the domainMapping attribute are arbitrary strings. In some consumer systems or existing content, the domain may be identified via an URI like http://example.com/domains/automotive. The domainMapping allows for using URIs too. For the mapping, they are regarded as ordinary string values.

The domainRule element

The domainRule element expresses that the content of the HTML body element is in the domain expressed by the HTML meta element with the name attribute, value keywords. The domainPointer attribute points to that meta element.

The domainRule element

The domainRule element expresses that the content of the HTML body element is in the domain expressed by associated values. The domainPointer attribute points to the values in the source content. The domainMapping attribute contains the comma separated list of mappings. In the example, automotive is available in the source content, and auto is used within the consumer tool, e.g. a machine translation system.

In source content, if available, it is recommended to use dublin core subject as the metadata term for domain information. In HTML, this can be achieved via a meta element with the name="keywords" attribute or name="dcterms.subject" attribute.

In the area of machine translation (e.g. machine translation systems or systems harvesting content for machine translation training), there is no agreed upon set of value sets for domain. Nevertheless it is recommended to use a small set of values both in source content and within consumer tools, to foster interoperability. If larger value sets are needed (e.g. detailed terms in the law or medical domain), mappings to the smaller value set needed for interoperability should be provided. An example would be a domainMapping attribute for generalizing the law domain: domainMapping="'criminal law' law, 'property law' law, 'contract law' law".

It is possible to have more than one domain associated with a piece of content. For example, if the consumer tool is a statistical machine translation engine, it could include corpora from all domains available in the source content in training the machine translation engine.

The consumer machine translation engine might choose to ignore the domain and take a one size fits all approach, or may be selective in which domains to use, based on the range of content marked with domain. For example, if the content has hundreds of sentences marked with domain 'automotive' and 'medical', but only a couple of sentences marked with additional domains 'criminal law' and 'property law', the consumer tool may opt to include its domains 'auto' and 'medicine', but not 'law', since the extra training resources does not justify the improvement in the output.

Definition

The Disambiguation data category is used to indicate occurrences of specific concepts that may require special handling in the localization of the document.

This data category can be used for several purposes, including, but not limited to:

Informing translation systems that this fragment of text may not be literally translated, but subject to specific proper name translation rules or official translations, as well as a very specific meaning of the phrases. Informing content management and translation systems about the type of the underlying entity in order to enable processing based on a specific type of the target, for example, when handling personal names, product names or geographic names, chemical compounds, protein names and similar.

Disambiguation is achieved by associating a selected fragment of text with an external web resource that can be referenced by a translation or linguistic review agent in order to access the correct meaning or lexical use of the text and thereby informing its translation.

A fragment of text can be disambiguated at different granularities, i.e. as a lexical concept, as an ontology concept, or as a named entity.

As a lexical concept, the external reference can provide synonyms and example usage, e.g. using service such as Wordnet.

As an ontology concept, the external reference can provide a formal conceptual definition within a framework of related concepts.

As a named entity, the external reference can provide a description of the real world entity the text intends to convey. For instance, the word 'City' in 'I am going to the City' may be disambiguated in one of the WordNet synsets that can be represented by 'city', an ontology concept of a City that could represent a subclass of a “PopulatedPlace” in the conceptual granularity level, or the central area of a particular city, e.g. City of London, as interpreted in the entity granularity level. Linked data network, such as DBpedia, increasing interlink ontological and named entity definitions for the same things as authored in different languages, offering a mechanism to locate translations from the source language description.

Two types of disambiguation are needed to identify:

The previous sentence needs to be re-worded Disambiguation type class, which describes the type class of the underlying concept or entity of the fragment. Disambiguation, which describes the actual underlying external resource that conveys the intended meaning of the fragment.

Text analysis engines, such as named entity recognizers, named entity, concept and word sense disambiguation components can offer an easy way to create this information. Content management tools can present and visualize this information or use it to index their content. Machine translations systems may use it for training and translation when dealing with proper names and edge cases.

Implementation

The Disambiguation data category can be expressed with global rules, or locally on an individual element. The information applies to the textual content of the element. There is no inheritance. The entity type follows inheritance rules.

The two last sentences above seem contradictory.

GLOBAL: The disambiguationRule element contains the following:

A required selector attribute that contains an absolute selector which selects the nodes to which this rule applies.

Either:

A disambigSource attribute that contains a string representing the disambiguation identifier collection source.

Exactly one of the following:

A disambigIdent attribute that contains a string that represents the disambiguation identifier for the disambiguation target that is valid within the specified disambiguation source.

A disambigIdentPointer attribute that contains a relative selector pointing to a node that represents a unique identifier for the disambiguation target.

Or:

Exactly one of the following:

A disambigIdentRef attribute that contains an URI that represents a unique identifier for the disambiguation target.

A disambigIdentRefPointer attribute that contains a relative selector pointing to a node that holds a URI that represents a unique identifier for the disambiguation target.

None or exactly one of the following:

A disambigClassPointer attribute that contains a relative selector pointing to a node specifying the entity type class behind the selector.

A disambigClassRef attribute that contains a URI, specifying the type class of the concept or entity behind the selector.

A disambigClassRefPointer attribute that contains a relative selector pointing to a node that holds a URI that specifies the entity type class behind the selector.

An optional disambigGranularity attribute that contains a string, specifying the granularity level of the disambiguation. The value can be one of the following identifiers: lexicalConcept, ontologyConcept, or entity.

Below will need a test case in the test suite. Sentence below is awkward

When using a disambiguation rule, the user MUST use one of the use cases for disambiguation: specifying the target type, or specifying the target identity. For the latter, the user MUST use only one of the two addressing modes:

Using disambigSource and one of disambigIdent or disambigIdentPointer to specify the collection and the identifier itself. Using one of disambigIdentRef or disambigIdentRefPointer using a URI for the disambiguation target. Usage of disambigClassRef, disambigGranularity, disambigIdentRef, disambigSource and disambigIdent for both entity and word sense disambiguation.

LOCAL: The following local markup is available for the Disambiguation data category:

An optional disambigClassRef attribute that contains a URI, specifying the type class of the concept or entity behind the selector. An optional disambigGranularity attribute that contains a string, specifying the granularity level of the disambiguation. The value can be one of the following identifiers: lexicalConcept, ontologyConcept, or entity

Either:

A disambigSource attribute that contains a string representing the disambiguation identifier collection source. A disambigIdent attribute that contains a string, representing the disambiguation identifier for the disambiguation target that is valid within the specified disambiguation source.

Or:

A disambigIdentRef attribute that contains a URI that represents a unique identifier for the disambiguation target.

The user MUST use only one of the two addressing modes for "target identity" disambiguation:

Using disambigSource and disambigIdent to specify the collection and the identifier itself. Using disambigIdentRef using a URI for the disambiguation target Local mixed usage of Usage of disambigClassRef, disambigGranularity, and disambigIdentRef in HTML.

For referring to disambigClassRef values, implementors are encouraged to use an existing repository of entity types as long as they satisfy their requirements. For example, the Named Entity Recognition and Disambiguation ontology (NERD): http://nerd.eurecom.fr/ontology

Furthermore, valid target types depend on the disambiguation granularity: types of entities are distinct from types of lexical concepts or ontology concepts. While this distinction exists, the specification does not prescribe a way of automatically inferring a disambiguation level from a target type.

When serializing the ITS mark-up in HTML5, the preferred way is to serialize in RDFa Lite or Microdata due to the existing search and crawling infrastructure that is able to consume this kind of data.

Local mixed usage of entityTypeSourceRef, enttiyTypeRef, disambigSourceRef, disambigIdentRef in HTML+RDFa Lite.

See for the companion document with the mapping data.

Companion document, having the mapping data for .

Definition

The Locale Filter data category specifies that a node is only applicable to certain locales.

This data category can be used for several purposes, including, but not limited to:

Include a legal notice only in locales for certain regions. Drop editorial notes from all localized output.

The Locale Filter data category associates with each selected node a list of extended language ranges conforming to . The list is comma-separated and can include the wildcard extended language range *. The list can also be empty. Whitespace surrounding language ranges is ignored.

To express that all locales should be included, one can use the wildcard * for the language range. To express that the content should not be included in any local, one can use the empty value.

Implementation

The Locale Filter data category can be expressed with global rules, or locally on an individual element. For elements, the data category information inherits to the textual content of the element, including child elements and attributes. The default is that the language range is *.

Implementations MUST NOT combine lists of language ranges from multiple rules or local attributes.

GLOBAL: The localeFilterRule element contains the following:

A required selector attribute. It contains an absolute selector which selects the nodes to which this rule applies. A required localeFilterList attribute with a comma-separated list of extended language ranges, or an empty string value. The Locale Filter data category expressed globally

The localeFilterRule element specifies that certain legal notice elements should only be shown in the specified locales. Note that using the extended language range *-CA in the localeFilterList attribute would cover all Canadian locales, including various minority languages in Canada.

The Locale Filter data category expressed globally

The localeFilterRule element specifies that editorial remarks should be removed from all translations.

LOCAL: The following local markup is available for the Locale Filter data category:

A localeFilterList attribute with a comma-separated list of extended language ranges, or an empty string value. The Locale Filter data category expressed locally

Definition Early draft of this data category; additional data categories for provenance might be added, or below definition might be changed. The definition of this data category is not yet reflected in the data category overview table in .

The Translation Provenance Agent data category is used to communicate the identity of agents that have been involved in the translation of the content or the revision of the translated contend. This allows translation and translation revision consumers, such as post-editors or translation quality reviewers, to assess how the performance of these agents may impact the quality of the translation. Translation and translation revision agents can be identified as a person, a piece of software or an organization that has been involved in providing a translation that resulted in the selected content.

This data category offers three types of information. First, it allows to identity translation agents. Second, it allows to identify revision agents. Third, if provenance information is needed that includes temporal information about processes or requires agents that support a wider range of activities, the data category offers a mechanism to refer to external, RDF-based provenance descriptions based on the provenance data model .

Translation or translation revision tools, such as machine translation agents or CAT tools, may offer an easy way to create this information. Translation tools can then present this information to post-editors or translation process managers. Web applications may to present such information to consumers of translated documents.

Implementation No agreement yet on whether such usage of global rules, that is for identifiyng just one or a small set of elements, is something to recommend. See also issue-51.

The Translation Agent Provenance data category can be expressed with global rules, or locally on individual elements. For elements, the data category information inherits to the textual content of the element, including child elements, but excluding attributes.

GLOBAL: The transProvRule element contains the following:

A required selector attribute. It contains an absolute selector which selects the nodes to which this rule applies.

At least one of the following:

Exactly one of the following:

A translationProvenanceRecordsRef attribute. Its value is a URI pointing to the translationProvenanceRecord element containing the list of translation provenance records related to the content selected via the selector attribute.

A translationProvenanceRecordsRefPointer attribute that contains a relative selector pointing to a node with the exact same semantics as translationProvenanceRecordsRef.

Human translation provenance information specified by exactly one of the following:

A transPerson attribute that contains a string identifying a human translation agent.

A transPersonRef attribute that contains an IRI referring to a resource that identifies a human translation agent.

A transPersonPointer attribute that contains a relative selector pointing to a node with the exact same semantics as transPerson.

A transPersonRefPointer attribute that contains a relative selector pointing to a node with the exact same semantics as transPersonRef.

Organizational translation provenance information specified by exactly of the following:

A transOrg attribute that contains a string identifying an organization acting as a translation agent.

A transOrgRef attribute that contains an IRI referring to a resource that identifies an organization acting as a translation agent.

A transOrgPointer attribute that contains a relative selector pointing to a node with the exact same semantics as transOrg.

A transOrgRefPointer attribute that contains a relative selector pointing to a node with the exact same semantics as transOrgRef.

Translation tool provenance related information specified by exactly one of the following:

A transTool attribute that contains a string identifying a software tool that was used in translating the selected content.

A transToolRef attribute that contains an IRI referring to a resource that identifies a software tool that was used in the translation.

A transToolPointer attribute that contains a relative selector pointing to a node with the exact same semantics as transTool.

A transToolRefPointer attribute that contains a relative selector pointing to a node with the exact same semantics as transToolRef.

Human translation revision provenance related information specified by exactly one of the following:

A transRevPerson attribute that contains a string identifying a human translation revision agent.

A transRevPersonRef attribute that contains an IRI referring to a resource that identifies a human translation revision agent.

A transRevPersonPointer attribute that contains a relative selector pointing to a node with the exact same semantics as transRevPerson.

A transRevPersonRefPointer attribute that contains a relative selector pointing to a node with the exact same semantics as transRevPersonRef.

Organizational revision translation related provenance information specified by exactly of the following:

A transRevOrg attribute that contains a string identifying an organization acting as a translation revision agent.

A transRevOrgRef attribute that contains an IRI referring to a resource that identifies an organization acting as a translation revison agent.

A transRevOrgPointer attribute that contains a relative selector pointing to a node with the exact same semantics as transRevOrg.

A transRevOrgRefPointer attribute that contains a relative selector pointing to a node with the exact same semantics as transRevOrgRef.

Translation tool revision provenance related information specified by exactly one of the following:

A transRevTool attribute that contains a string identifying a software tool that was used in revising the translation of the selected content.

A transRevToolRef attribute that contains an IRI referring to a resource that identifies a software tool that was used in revising the translation of the selected content.

A transRevToolPointer attribute that contains a relative selector pointing to a node with the exact same semantics as transRevTool.

A transRevToolRefPointer attribute that contains a relative selector pointing to a node with the exact same semantics as transRevToolRef.

A reference to external, RDF-based provenance description specified by exactly one of the following:

A provRef attribute that that contains one or more space (U+0020) separated Provenance URI, each referring to a resource that identifies a different provenance entity record defined by the provenance data model.

A provRefPointer attribute that contains a relative selector pointing to a node with the exact same semantics as provRef.

Below note is taken from the quality issue data category. Same question applies: Why should below only say "do not apply to HMTL as local markup"? There is local markup for direct annotation in XML too.

The attributes translationProvenanceRecordsRefPointer, transPersonPointer, transPersonRefPointer, transOrgPointer, transOrgRefPointer, transToolPointer, transToolRefPointer, transRevPersonPointer, transRevPersonRefPointer, transRevOrgPointer, transRevOrgRefPointer, transRevToolPointer, transRevToolRefPointer and provRefPointer do not apply to HTML as local markup is provided for direct annotation in HTML.

The Translation Agent Provenance data category used globally.

This example shows how the provenance of the par and the legalnotice elements in this XML document is different. Therefore it is recorded in separate transProvRule elements.

The Translation Agent Provenance data category used globally with pointer attributes.

This example expresses the same provenance information as , but the provenance information for the par element is stored differently, inside a format specific element my-provenance-info. The first transProvRule element and its attributes transToolRefPointer, transOrgPointer, transRevToolRefPointer, transRevOrgPointer and provRefPointer are used to point to the information inside that my-provenance-info element.

Not sure if we need the standoff version globally. We don't have it with quality either. Thoughts? The Translation Agent Provenance data category used globally with standoff provenance records.

This example expresses the same plus some additional provenance information as , but the provenance information is realized standoff within translationProvenanceRecords elements. The transProvRule elements with the translationProvenanceRecordsRef attributes point to translationProvenanceRecords related to the par and legalnotice elements. The legalnotice element has been revised two times. Hence, the related translationProvenanceRecords element contains two translationProvenanceRecord child elements. The second translationProvenanceRecord child element provides information about the second revison.

Annotating provenance information in HTML5 with transProvRule element

The transProvRule element resides in a separate file () that associates the provenance information with a selected span of content in the HTML document.

External rule document associated with an HTML5 document

This document is used in :

LOCAL: Using the inline markup to represent the data category locally is limited to a single occurrence for a given content (e.g. one cannot have different transToolRef attributes applied to the same span of text because the inner-most one would override the others). A local standoff markup is provided to allow such cases.

The following local markup is available for the Translation Agent Provenance data category:

Either (inline markup): at least one of the following, with the same semantics as the corresponding attributes at the transProvRule element:

Human translation provenance information specified by exactly a transPerson or a transPersonRef attribute.

Organizational translation provenance information specified by exactly a transOrg or a transOrgRef attribute.

Translation tool provenance related information specified by exactly a transTool or a transToolRef attribute.

Human translation revision provenance related information specified by exactly a transRevPerson or a transRevPersonRef attribute.

Organizational revision translation related provenance information specified by exactly a transRevOrg or a transRevOrgRef attribute.

Translation tool revision provenance related information specified by exactly a transRevTool or a transRevToolRef attribute.

A reference to external, RDF-based provenance description specified by a provRef attribute.

Or (standoff markup):

A translationProvenanceRecordsRef attribute. Its value is a URI pointing to the translationProvenanceRecords element containing the list of provenance information related to this content.

An element translationProvenanceRecords (or <span its-translation-provenance-records> in HTML) which contains:

One or more elements translationProvenanceRecord (or <span its-translation-provenance-record> in HTML), each of which contains at least one of the following, with the same semantics as the corresponding attributes at the transProvRule element:

Human translation provenance information specified by exactly a transPerson or a transPersonRef attribute.

Organizational translation provenance information specified by exactly a transOrg or a transOrgRef attribute.

Translation tool provenance related information specified by exactly a transTool or a transToolRef attribute.

Human translation revision provenance related information specified by exactly a transRevPerson or a transRevPersonRef attribute.

Organizational revision translation related provenance information specified by exactly a transRevOrg or a transRevOrgRef attribute.

Translation tool revision provenance related information specified by exactly a transRevTool or a transRevToolRef attribute.

A reference to external, RDF-based provenance description specified by a provRef attribute.

Important:

When the attributes transPerson, transPersonRef, transOrg, transOrgRef, transTool, transToolRef, transRevPerson, transRevPersonRef, transRevOrg, transRevOrgRef, transRevTool, transRevToolRef and provRef (or their equivalent representations) are used in in a standoff manner, the information they carry pertains to the content of the element that refers to the standoff annotation, not to the content of the element translationProvenanceRecord (or <span translation-provenance-record>in HTML) where they are declared.

The order of translationProvenanceRecord elements inside translationProvenanceRecords is significant: it reflects the temporal order of revisions. This is demonstrated e.g. in .

Annotating provenance information in XML with local inline markup

The provenance related attributes at the par and legalnotice elements are used to associate the provenance information directly with the content of theses elements.

Annotating provenance information in HTML with local inline markup

In this example several spans of content are associated with provenance information.

Annotating provenance information in XML with local standoff markup

The following example shows a document using local standoff markup to encode several pieces of provenance information. The par and legalnotice elements delemit the content to markup. They hold translationProvenanceRecordsRef attributes that point to the related translationProvenanceRecords elements. The legalnotice element has been revised two times. Hence, the related translationProvenanceRecords element contains two translationProvenanceRecord child elements. The second translationProvenanceRecord child element provides information about the second revison.

Annotating provenance information in XML with local standoff markup and a global rule

The following example shows a document using local standoff markup to encode several pieces of provenance information. But because, in this case, the par or the legal notice elements do not allow attributes from another namespace we cannot use translationProvenanceRecordsRef directly. Instead, a global rule is used to map the function of translationProvenanceRecordsRef to a non-ITS construct, here the ref attribute of the par or legal notice elements.

Annotating provenance information in HTML with local standoff markup

The following example shows a document using local standoff markup to encode provenance information. The p elements delimits the content to markup. It holds a its-translation-provenance-records-ref attribute that points to the standoff information inside the script element.

TODO for above: Finalize how HTML should work: use its-* attributes for standoff markup or markup inside the script element.

Definition

The External Resource data category indicates that a node represents or references potentially translatable data in a resource outside the document. Examples of such resources are external images and audio or video files.

Implementation

The External Resource data category can be expressed only with global rules. There is no inheritance. There is no default.

GLOBAL: The externalResourceRefRule element contains the following:

A required selector attribute. It contains an absolute selector which selects the nodes to which this rule applies. A required externalResourceRefPointer attribute that contains a relative selector pointing to a node that provides the URI of the external resource. The externalResourceRefRule element

The externalResourceRefRule element expresses that the imagedata, audiodata and videodata elements contain references to external resources. These references are expressed via a fileref attribute. The externalResourceRefPointer attribute points to that attribute.

Two externalResourceRefRule elements used for external resources associated with HTML5 video elements

The two externalResourceRefRule elements select the src and the poster attributes at HTML5 video elements. These attributes identify different external resources, and at the same time contain the references to these resources. For this reason, the externalResourceRefPointer attributes point to the value of src and poster respectively. The underlying HTML5 document is given in .

An HTML5 document that can be used for .

Definition

Some formats, such as those designed for localization or for multilingual resources, hold the same content in different languages inside a single document. The Target Pointer data category is used to associate the node of a given source content (i.e. the content to be translated) and the node of its corresponding target content (i.e. the source content translated into a given target language).

This specification makes no provision regarding the presence of the target nodes or their content: A target node may or may not exist and it may or may not have content.

This data category can be used for several purposes, including but not limited to:

Extract the source content to translate and put back the translation at its proper location.

Compare source and target content for quality verification.

Re-use existing translations when localizing the new version of an existing document.

Access aligned bi-lingual content to build memories, or to train machine translation engines.

In general, it is recommended to avoid developing formats where the same content is stored in different languages in the same document, unless for very specific use cases. See the best practices Working with multilingual documents from for further guidance.

Implementation

The Target Pointer data category can be expressed only with global rules. There is no inheritance. There is no default.

GLOBAL: The targetPointerRule element contains the following:

A required selector attribute. It contains an absolute selector which selects the nodes to which this rule applies. A required targetPointer attribute. It contains a relative selector that points to the node for the target content corresponding to the selected source node.

The source node and the target node may be of different types, but the target node must be able to contain the same content of the source node (e.g. an attribute node cannot be the target node of a source node that is an element with children).

Defining the target location of a source content with the targetPointerRule element

Definition

The Id Value data category indicates a value that can be used as unique identifier for a given part of the content.

The recommended way to specify a unique identifier is to use xml:id (See the best practice Defining markup for unique identifiers from ). The idValueRule element is intended only as a fall-back mechanism for documents where unique identifiers are available with another construct.

Providing a unique identifier that is maintained in the original document can be use for several purposes, for example:

Allow automated alignment between different versions of the source document, or between source and translated documents.

Improve the confidence in leveraged translation for exact matches.

Provide back-tracking information between displayed text and source material when testing or debugging.

The Id Value data category only provides for rules to be expressed at a global level. Locally, users are able to use xml:id (which is defined by XML) or an attribute specific to the format in question (as in ).

Applying the Id Value data category to xml:id attributes using global rules is not necessary, since xml:id is the recommended way to specify an identifier in XML.

Implementation

The id Value data category can be expressed only with global rules. There is no inheritance. There is no default.

GLOBAL: The idValueRule element contains the following:

A required selector attribute. It contains an absolute selector which selects the nodes to which this rule applies. A required idValue attribute. It contains an XPath expression which constructs a string corresponding to the identifier of the node to which this rule applies. The identifier MUST be unique at least within the document. If the attribute xml:id is present for the selected node, the value of the xml:id attribute MUST take precedence over the idValue value. Pointing to an ID value with the idValueRule element

The idValueRule element indicates that the unique identifier for each <text> element is the value of the attribute name of its parent element.

Constructing ID values using the idValueRule element.

The idValue attribute allows to build composite values based on different attributes, element or event hard-coded text. Any of the String functions offered by XPath can be used. In the document below, the two elements <text> and <desc> are translatable, but they have only one corresponding identifier, the name attribute in their parent element.

To make sure the identifier is unique for both the content of <text> and the content of <desc>, the XPath expression concat(../@name, '_t') gives the identifier "settingsMissing_t" for the content of <text> and the expression concat(../@name, '_d') gives the identifier "settingsMissing_d" for the content of <desc>.

Using xml:id and idValueRule

When an xml:id attribute is present for a node selected by an idValueRule element, the value of xml:id takes precedence over the value defined by the idValueRule element. In the example below, the unique ID to use is “btnAgain” for the first <res> element, and “retryTip” for the second <res> element.

Definition

The Preserve Space data category indicates how whitespace should be handled in content. The possible values for this data category are "default" and "preserve" and carry the same meaning as the corresponding values of the xml:space attribute. The default value is "default".

Implementation

The Preserve Space data category can be expressed with global rules, or locally using the xml:space attribute. For elements, the data category information inherits to the textual content of the element, including child elements and attributes.

The Preserve Space data category is not applicable to HTML5 documents because xml:space (and by extension Preserve Space) has no effect in documents parsed as text/html.

GLOBAL: The preserveSpaceRule element contains the following:

A required selector attribute. It contains an absolute selector which selects the nodes to which this rule applies. A required space attribute with the value "default" or "preserve". The Preserve Space data category expressed globally

The preserveSpaceRule element specifies that whitespace in all verse elements must be treated literally.

LOCAL: The xml:space attribute, as defined in section 2.10 of , maps exactly to the Preserve Space data category.

The Preserve Space data category expressed locally

The standard xml:space attribute specifies that the whitespace in the verse element must be treated literally.

Definition

The Localization Quality Issue data category is used to express information related to localization quality assessment tasks. Such tasks can be conducted on the translation of some source text into a target language or on the source text itself where its quality may impact on the localization process.

This data category can be used in a number of ways, including the following example scenarios:

An automatic quality checking tool flags a number of potential quality issues in an XML or HTML file and marks them up using ITS 2.0 markup. Other tools in the workflow then examine this markup and decide whether the file needs to be reviewed manually or passed on for further processing without a manual review stage.

A quality assessment process identifies a number of issues and adds the ITS markup to a rendered HTML preview of an XML file along with CSS styling that highlights these issues. The resulting HTML file is then sent back to the translator to assist his or her revision efforts.

A human reviewer working with a web-based tool adds quality markup, including comments and suggestions, to a localized text as part of the review process. A subsequent process examines this markup to ensure that changes were made.

The data category defines four pieces of information:

Information Description Value Notes Type A set of broad types of issues into which tool-specific issues can be categorized. One of the values defined in list of type values. ITS 2.0-compliant tools that use these categories MUST map their internal values to these types. If the type of the issue is set to uncategorized, a comment MUST be specified as well. Comment A human-readable description of the quality issue. Text Severity A decimal value representing the severity of the issue, as defined by the model generating the metadata. A decimal value between 0.0 and 100.0 (inclusive), with higher values indicating greater severity. It is up to tools to map the values of this to their own system to this scale. If needed, the original value can be passed along using a custom namespace for XML, or a data- attribute for HTML. Profile Reference A reference to a document describing the quality assessment model used for the issue. A URI pointing to the reference document. The use of resolvable URI is strongly recommended as it provides a way for human evaluators to learn more about the quality issues in use.

Implementation

The Localization Quality Issue data category can be expressed with global rules, or locally on individual elements. For elements, the data category information inherits to the textual content of the element, including child elements, but excluding attributes.

GLOBAL: The locQualityIssueRule element contains the following:

A required selector attribute. It contains an absolute selector which selects the nodes to which this rule applies.

At least one of the following:

Exactly one of the following:

A locQualityIssuesRef attribute. Its value is a URI pointing to the locQualityIssues element containing the list of issues related to this content.

A locQualityIssuesRefPointer attribute that contains a relative selector pointing to a node with the exact same semantics as locQualityIssuesRef.

Exactly one of the following:

A locQualityIssueType attribute that implements the type information.

A locQualityIssueTypePointer attribute that contains a relative selector pointing to a node with the exact same semantics as locQualityIssueType.

Exactly one of the following:

A locQualityIssueComment attribute that implements the comment information.

A locQualityIssueCommentPointer attribute that contains a relative selector pointing to a node with the exact same semantics as locQualityIssueComment.

None or exactly one of the following:

A locQualityIssueSeverity attribute that implements the severity information.

A locQualityIssueSeverityPointer attribute that contains a relative selector pointing to a node with the exact same semantics as locQualityIssueSeverity.

None or exactly one of the following:

A locQualityIssueProfileRef attribute that implements the profile reference information.

A locQualityIssueProfileRefPointer attribute that contains a relative selector pointing to a node with the exact same semantics as locQualityIssueProfileRef.

Why does below say "do not apply to HMTL as local markup"? There is local markup for direction annotation in XML too.

The attributes locQualityIssuesRefPointer, locQualityIssueTypePointer, locQualityIssueCommentPointer, locQualityIssueSeverityPointer and locQualityIssueProfileRefPointer do not apply to HTML as local markup is provided for direct annotation in HTML.

Annotating an issue in XML with locQualityIssueRule element

The locQualityIssueRule element associates the issue information with a selected span of content.

Using locQualityIssueRule to map equivalent markup

The locQualityIssueRule element defines what constructs are equivalent to the native ITS markup for the different pieces of information of the data category.

Annotating an issue in HTML5 with locQualityIssueRule element

The locQualityIssueRule element resides in a separate file () that associates the issue information with a selected span of content in the HTML document.

External rule document associated with an HTML5 document

This document is used in :

LOCAL: Using the inline markup to represent the data category locally is limited to a single occurrence for a given content (e.g. one cannot have different locQualityIssueType attributes applied to the same span of text because the inner-most one would override the others). A local standoff markup is provided to allow such cases.

The following local markup is available for the Localization Quality Issue data category:

Either (inline markup):

At least one of the following attributes:

A locQualityIssueType attribute that implements the type information.

A locQualityIssueComment attribute that implements the comment information.

An optional locQualityIssueSeverity attribute that implements the severity information.

An optional locQualityIssueProfileRef attribute that implements the profile reference information.

Or (standoff markup):

A locQualityIssuesRef attribute. Its value is a URI pointing to the locQualityIssues element containing the list of issues related to this content.

An element locQualityIssues (or <span loc-quality-issues> in HTML) which contains:

Should locQualityIssues also be defined for global rules? It seems not to be specific to local.

One or more elements locQualityIssue (or <span its-loc-quality-issue> in HTML), each of which contains:

At least one of the following attributes:

A locQualityIssueType attribute that implements the type information.

A locQualityIssueComment attribute that implements the comment information.

An optional locQualityIssueSeverity attribute that implements the severity information.

An optional locQualityIssueProfileRef attribute that implements the profile reference information.

Important: When the attributes locQualityIssueType, locQualityIssueComment, locQualityIssueSeverity and locQualityIssueProfileRef (or their equivalent representations) are used in in a standoff manner, the information they carry pertains to the content of the element that refers to the standoff annotation, not to the content of the element locQualityIssue (or <span loc-quality-issue>in HTML) where they are declared.

Annotating an issue in XML with local inline markup

The attributes locQualityIssueType, locQualityIssueComment and locQualityIssueSeverity are used to associate the issue information directly with a selected span of content.

Annotating an issue in HTML with local inline markup

In this example several spans of content are associated with a quality issue.

Annotating an issue in XML with local standoff markup

The following example shows a document using local standoff markup to encode several issues. The mrk element delimits the content to markup and holds a locQualityIssuesRef attribute that points to the locQualityIssues element where the issues are listed.

Annotating an issue in XML with local standoff markup and a global rule

The following example shows a document using local standoff markup to encode several issues. But because, in this case, the mrk element does not allow attributes from another namespace we cannot use locQualityIssuesRef directly. Instead, a global rule is used to map the function of locQualityIssuesRef to a non-ITS construct, here the ref attribute of any mrk elements that has its attribute type set to "x-itslq".

Annotating an issue in HTML with local standoff markup

The following example shows a document using local standoff markup to encode several issues. The span element delimits the content to markup and holds a loc-quality-issues-ref attribute that points to a special span element where the issues are listed within a set of other special span elements.

TODO for above: Finalize how HTML should work: use its-* attributes for standoff markup or markup inside the script element.

Definition

The Localization Quality Précis data category is used to express an overall measurement of the localization quality of a document or an item in a document.

This data category allows to specify a quality score or a voting result for a given item or document, as well as to indicate what constitutes a passing score or vote. It also allows to point to a profile describing the quality assessment model used for the scoring or the voting.

Implementation

The Localization Quality Précis data category can be expressed with global rules, or locally on individual elements. For elements, the data category information inherits to the textual content of the element, including child elements, but excluding attributes.

GLOBAL: The locQualityPrecisRule element contains the following:

A required selector attribute. It contains an absolute selector which selects the nodes to which this rule applies.

Exactly one of the following:

Exactly one of the following:

A locQualityPrecisScore attribute. Its value is an integer between 0 and 100 (inclusive) with higher values indicating a better score.

A locQualityPrecisScorePointer attribute that contains a relative selector pointing to a node with the exact same semantics as locQualityPrecisScore.

Exactly one of the following:

A locQualityPrecisVote attribute. Its value is a signed integer with higher values indicating a better vote.

A locQualityPrecisVotePointer attribute that contains a relative selector pointing to a node with the exact same semantics as locQualityPrecisVote.

None or exactly one of the following:

A locQualityPrecisThreshold attribute. Its value is a signed integer which indicates the lowest score or vote value that constitutes a passing score or a passing vote in the profile used.

A locQualityPrecisThresholdPointer attribute that contains a relative selector pointing to a node with the exact same semantics as locQualityPrecisThreshold.

None or exactly one of the following:

A locQualityPrecisProfileRef attribute. Its value is a URI pointing to the reference document describing the quality assessment model used for the scoring.

A locQualityPrecisProfileRefPointer attribute that contains a relative selector pointing to a node with the exact same semantics as locQualityPrecisProfileRef.

The attributes locQualityPrecisScorePointer, locQualityPrecisThresholdPointer, and locQualityPrecisProfileRefPointer do not apply to HTML as local markup is provided for direct annotation in HTML.

The Localization Quality Précis data category expressed globally in XML

The following example shows how to use the locQualityPrecisRule element to specify the score, threshold and profile for a document.

Using pointers to map the Localization Quality Précis data category in XML

The following example shows how the locQualityPrecisVotePointer, locQualityPrecisThresholdPointer and locQualityPrecisProfileRefPointer can be used to map the data category to an equivalent markup.

The Localization Quality Précis data category expressed globally in HTML

The following example shows how to use the locQualityPrecisRule element to specify the score, threshold and profile for an HTML document.

External rule document associated with an HTML5 document

This document is used in :

LOCAL: The following local markup is available for the Localization Quality Précis data category:

Exactly one of the following:

A locQualityPrecisScore attribute. Its value is an integer between 0 and 100 (inclusive) with higher values indicating a better score.

A locQualityPrecisVote attribute. Its value is a signed integer with higher values indicating a better vote.

An optional locQualityPrecisThreshold attribute. Its value is a signed integer which indicates the lowest score or vote that constitutes a passing score or a passing vote in the profile used.

An optional locQualityPrecisProfileRef attribute. Its value is a URI pointing to the reference document describing the quality assessment model used for the scoring.

The Localization Quality Précis data category expressed locally in XML

The locQualityPrecisScore, locQualityPrecisThreshold and locQualityPrecisProfileRef are used to score the quality of the document.

The Localization Quality Précis data category expressed locally in HTML

The its-loc-quality-precis-score, its-loc-quality-precis-threshold and its-loc-quality-precis-profile-ref are used to score the quality of the document.

Definition

The MT Confidence data category is used to communicate the self-reported confidence of a specific machine translation engine. It is not intended as comparable between machine translation engines and platforms. It is solely for providing self-reported confidence by the specific system that produced the actually used raw machine translation. This data category does NOT aim to establish any sort of correlation between the self-reported confidence and either human evaluation of MT usefulness, or post-editing cognitive effort. For harmonization’s sake, MT Confidence is provided as a (rational) number from the interval <0;1>.

Implementers are expected to interpret the floating point number and present it to human and other consumers in other convenient forms, such as percentage (0-100%) with up to 2 decimal digits, font or background color coding etc.

This data category can be used for several purposes, including, but not limited to:

Automated sorting of raw machine translated text for further processing based on empirically set thresholds.

Provide readers of machine translated text with self-reported relative accuracy prediction.

Provide translators, post-editors, reviewers and proofreaders with self-reported relative accuracy prediction.

Human consumers using often machine translation for the same source should be able to predict usefulness of a machine translated segments at a glance.

MT confidence can be displayed e.g.:

on websites machine translated on the fly,

by simple translation editors, and Computer Aided Translation (CAT) tools.

Implementation

The MT Confidence data category can be expressed with global rules, or locally on individual elements. For elements, the data category information inherits to the textual content of the element, including child elements, but excluding attributes.

GLOBAL: The mtConfidenceRule element contains the following:

A required selector attribute. It contains an absolute selector which selects the nodes to which this rule applies.

A required mtProducer attribute that contains a human readable string identifying the Machine Translation Platform, e.g. "Bing Translator", "Google Translate", "DCU Matrex", "vanilla Moses" etc.

An optional mtEngine attribute that contains a string uniquely identifying a specific MT engine on a platform given in mtProducer. Some examples of values are:

A BCP 47 language tag with t-extension, e.g. ja-t-it for an Italian to Japanese MT engine

A Domain as per the

A privately structured string, eg. Domain:IT-Pair:IT-JA, IT-JA:Medical, etc.

Global usage of mtConfidenceRule, mtProducer, and mtEngine (specified by BCP 47 t-extension) along with local usage of mtConfidenceScore Global usage of mtConfidenceRule, mtProducer, and mtEngine (specified with a sample privately structured string) along with local usage of mtConfidenceScore

LOCAL: the following local markup is available for the MT Confidence data category:

An mtProducer attribute that contains a string identifying the Machine Translation Platform, e.g. “Bing Translator”, “Google Translate”, “DCU Matrex”, “vanilla Moses” etc.

An mtEngine attribute that contains a string uniquely identifying a specific MT engine on a platform given in mtProducer. Some examples of values are given for the global definition of MT Confidence.

The MT Confidence data category expressed locally The MT Confidence data category expressed locally in HTML5

Definition

The Allowed Characters data category is used to specify what characters are allowed in a given content.

This data category can be used for various purposes, including the following examples:

Limit the characters which may be used in the UI of a game because of some special font restrictions. Prevent illegal characters to be entered for text content that are file or directory names. Control what characters can be used when translating examples of login name in a content.

The set of characters that are allowed is specified using a regular expression. That is, each character in the selected content MUST be included in the set specified by the regular expression.

The regular expression is a character class construct as defined in the section Character Classes of XML Schema , with the assumption that the . metacharacter matches also CARRIAGE RETURN (U+000D) and LINE FEED (U+000F). That is with the dot-all option set.

Example of expressions (shown as XML source):

"[abc]" : allows the characters 'a', 'b' and 'c'. "[a-c]" : allows the characters 'a', 'b' and 'c'. "[a-zA-Z]" : allows the characters from 'a' to 'z' and from 'A' to 'Z'. "[^abc]" : allows any characters except 'a', 'b', and 'c'. "[^a-c]" : allows any characters except 'a', 'b', and 'c'. "\w" : allows any character except the set of "punctuation", "separator" and "other" characters. "[ --[<>:"\\/|\?*]]" : allows only the characters valid for Windows file names. "." : allows any character. "" : allows no character. "[a-ÿ-[\s]]" : allows all characters between U+0061 and U+00FF except the characters SPACE (U+0020), TABULATION (U+0009), CARRIAGE RETURN (U+000D) and LINE FEED (U+000F).

Implementation

The Allowed Characters data category can be expressed with global rules, or locally on individual elements. For elements, the data category information inherits to the textual content of the element, including child elements, but excluding attributes.

GLOBAL: The allowedCharactersRule element contains the following:

A required selector attribute. It contains an absolute selector which selects the nodes to which this rule applies.

Exactly one of the following:

A allowedCharacters attribute that contains the regular expression indicating the allowed characters.

A allowedCharactersPointer attribute that contains a relative selector pointing to a node with the exact same semantics as allowedCharacters.

The Allowed Characters data category expressed globally in XML

The allowedCharactersRule element states that the translated content of elements content must not contain the characters * and +.

Mapping the Allowed Characters data category in XML

The attribute allowedCharactersPointer is used to map the data category to the non-ITS attribute set in this document. The attribute has the same semantics as allowedCharacters.

LOCAL: the following local markup is available for the Allowed Characters data category:

A allowedCharacters attribute that contains the regular expression indicating the allowed characters.

The Allowed Characters data category expressed locally in XML

The local allowedCharacters attribute specifies that the translated content of element panelmsg must contain only Unicode characters between U+0020 and U+00FE.

The Allowed Characters data category expressed locally in HTML

The local its-allowed-characters attribute specifies that the translated content of element code must not contain the characters other than 'a' to 'z' in any case and the characters underscore and minus.

Definition

The Storage Size data category is used to specify the maximum storage size of a given content.

This data category can be used for various purposes, including the following examples:

Verify during translation if a string fits into a fixed-size database field. Control the size of a string that is stored in a fixed-size memory buffer at run-time.

The storage size is expressed in bytes and is provided along with the character set encoding used to store the content.

Implementation

The Storage Size data category can be expressed with global rules, or locally on individual elements. There is no inheritance. The default value of the character set encoding is UTF-8.

GLOBAL: The storageSizeRule element contains the following:

A required selector attribute. It contains an absolute selector which selects the nodes to which this rule applies.

Exactly one of the following:

A storageSize attribute. It contains the maximum number of bytes the text of the selected node is allowed in storage.

A storageSizePointer attribute that contains a relative selector pointing to a node with the exact same semantics as storageSize.

None or exactly one of the following:

A storageEncoding attribute. It contains the name of the character set encoding used to calculate the number of bytes of the selected text. The name MUST be one of the names or aliases listed in the IANA Character Sets registry . The default value is "UTF-8".

A storageEncodingPointer attribute that contains a relative selector pointing to a node with the exact same semantics as storageEncoding.

An optional lineBreakType attribute. It indicates what type of line breaks the storage uses. The possible values are: cr for CARRIAGE RETURN (U+000D), lf for LINE FEED (U+000A), crlf for CARRIAGE RETURN (U+000D) followed by LINE FEED (U+000A), or nel for NEXT LINE (U+0085). The default value is lf.

The Storage Size data category expressed globally in XML

The storageSizeRule element is used to specify that, when encoded in ISO-8859-1, the content of the country element must not be more than 25 bytes. The name "Papouasie-Nouvelle-Guinée" is 25 character long and fits because all characters in ISO-8859-1 are encoded as a single byte.

Mapping the Storage Size data category in XML

The storageSizePointer attribute is used to map the non-ITS attribute max to the same functionality as storageSize. There is no character set encoding specified, so the default UTF-8 is assumed. Note that, while the name "Papouasie-Nouvelle-Guinée" is 25 character long, the character 'é' is encoded into two bytes in UTF-8. Therefore this name is one byte too long to fit in its storage destination.

LOCAL: the following local markup is available for the Storage Size data category:

A storageSize attribute. It contains the maximum number of bytes the text of the selected node is allowed in storage.

An optional storageEncoding attribute. It contains the name of the character set encoding used to calculate the number of bytes of the selected text. The name MUST be one of the names or aliases listed in the IANA Character Sets registry . The default value is "UTF-8".

An optional lineBreakType attribute. It indicates what type of line breaks the storage uses. The possible values are: cr for CARRIAGE RETURN (U+000D), lf for LINE FEED (U+000A), crlf for CARRIAGE RETURN (U+000D) followed by LINE FEED (U+000A), or nel for NEXT LINE (U+0085). The default value is lf.

The Storage Size data category expressed locally in XML

The storageSize attribute allows to specify different the maximum storage sizes throughout the document.

The Storage Size data category expressed locally in HTML

The its-storage-size is used here to specify the maximum number of bytes the two editable strings can have in UTF-8.