Tamil Gap Analysis

W3C Working Draft

This version:
https://www.w3.org/TR/2021/WD-taml-gap-20210210/
Latest published version:
https://www.w3.org/TR/taml-gap/
Latest editor's draft:
https://w3c.github.io/iip/gap-analysis/taml-gap
Previous version:
https://www.w3.org/TR/2021/WD-taml-gap-20210126/
Editor:
(W3C)
Participate:
GitHub w3c/iip
File a bug
Commit history
Pull requests

Abstract

This document describes and prioritises gaps for the support of the Tamil script on the Web and in eBooks. In particular, it is concerned with text layout. It checks that needed features are supported in W3C specifications, in particular HTML and CSS and those relating to digital publications. It also checks whether the features have been implemented in browsers and ereaders. This is a preliminary analysis.

Status of This Document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at https://www.w3.org/TR/.

This document describes and prioritises gaps for the support of the Tamil script on the Web and in eBooks. In particular, it is concerned with text layout. It checks that needed features are supported in W3C specifications, in particular HTML and CSS and those relating to digital publications. It also checks whether the features have been implemented in browsers and ereaders. This document complements the document Tamil Layout Requirements, which describes the requirements for areas where gaps appear. It is linked to from the language matrix that tracks Web support for many languages.

The editor's draft of this document is being developed by the India International Program Layout Task Force, part of the W3C Internationalization Interest Group. It is published by the Internationalization Working Group. The end target for this document is a Working Group Note.

This document was published by the Internationalization Working Group as a Working Draft.

GitHub Issues are preferred for discussion of this specification.

Publication as a Working Draft does not imply endorsement by the W3C Membership.

This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

This document was produced by a group operating under the 1 August 2017 W3C Patent Policy. The group does not expect this document to become a W3C Recommendation. W3C maintains a public list of any patent disclosures made in connection with the deliverables of the group; that page also includes instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy.

This document is governed by the 15 September 2020 W3C Process Document.

1. Introduction

The W3C needs to make sure that the text layout and typographic needs of scripts and languages around the world are built in to technologies such as HTML, CSS, SVG, etc. so that Web pages and eBooks can look and behave as people expect around the world.

This page documents difficulties people encounter when trying to use the Tamil language in the Tamil script on the Web.

Having identified an issue, it investigates the current status with regards to web specifications and implementations by user agents (browsers, e-readers, etc.), and attempts to prioritise the severity of the issue for web users.

A summary of this report and others can be found as part of the language matrix.

For a description of the Tamil script see the (non-W3C) page Tamil, which summarises aspects of the orthography and typographic features, including relevant Unicode characters and their use.

1.1 Work flow

This version of the document is a preliminary analysis

Gap analysis work usually starts with a preliminary analysis, conducted quickly by one or a small group of experts. Then a more detailed analysis is carried out, involving a wider range of experts. The detailed analysis may involve the development of tests, in order to illustrate issues and track results for browsers. The next phase is ongoing maintenance. It is expected that the resulting document will not be frozen: as gaps are fixed, this should be noted in the document. It is also possible that new gaps are noticed or arise, and they can be added to this document when that happens.

As the gap analysis develops, the requirements for features that are problematic should be described in the companion document, Tamil Layout Requirements. Links to the appropriate part of that document should be added to this document as the material is created. Note that the requirements document should not contain any technology-specific information: all of that belongs here.

1.2 Prioritization

This document not only describes gaps, it also attempts to prioritise them in terms of the impact on the local user. The prioritisation is indicated by colour.

Key:

It is important to note that these colours do not indicate to what extent a particular features is broken. They indicate the impact of a broken or missing feature on the content author or end user.

Basic styling is the level that would be generally accepted as sufficient for most Web pages. Advanced level support would include additional features one might expect to include in ebooks or other advanced typographic formats. There may be features of a script or language that are not supported on the Web, but that are not generally regarded as necessary (usually archaic or obscure features). In this case, the feature can be described here, but the status should be marked as OK.

The decision as to what priority level is assigned to a described gap is down to the experts doing the gap analysis. It may not always be straightforward to decide. If a given section in this document refers to more than one feature that is broken, each with different impacts on Web users, the priority for the section should be the lowest denominator.

A cell can be scored as OK if the feature in question is specified in an appropriate specification, and is supported by user agents. A specification that is in CR or later and has two implementations in 'major' browsers will count. This means that the feature may not be supported in all browsers yet. (At some point in the future we may try to distinguish, visually, whether support is available in a specification but still pending in major browsers or applications.)

2. Text direction

See also General page layout & progression for features such as column layout, page turning direction, etc. that are affected by text direction.

2.1 Vertical text

Are the script requirements for vertically oriented text met? What about if you mix vertical text with scripts that are normally only horizontal? Do you need a switch to use different characters in vertical vs. horizontal text? Does the browser support short runs of horizontal text in vertical lines (tate-chu-yoko in Japanese) as expected? Is the orientation of characters and the directional ordering of characters supported as needed? See available information or check for currently needed data.

#70 Upright text in vertical lines doesn't segment correctly

Upright characters in vertical lines of text don't break correctly, due to the problem of detecting syllable boundaries correctly mentioned above. Marking this as advanced, since this is not a common use case.

See requirements at: Indic Layout Requirements

2.2 Bidirectional text

If this script runs right-to-left, are there any issues when handling that? Is bidirectional text adequately supported? What about numbers and expressions? Do the Unicode bidi controls and HTML markup provide the support needed? Is isolation of directional runs problematic? See available information or check for currently needed data.

3. Characters and phrases

3.1 Characters & encoding

Are there any character repertoire issues preventing use of this script on the Web? Do variation selectors need attention? Are there any other encoding-related issues? See available information or check for currently needed data.

3.2 Fonts

Do the standard fallback fonts used in browsers (eg. serif, sans-serif, cursive, etc.) match expectations? Are special font or OpenType features needed for this script that are not available? See available information or check for currently needed data.

3.3 Font styles, weight, etc

This covers ways of modifying the glyphs, such as for italicisation, bolding, oblique, etc. Do italic fonts lean in the right direction? Is synthesised italicisation problematic? Are there other problems relating to bolding or italicisation - perhaps relating to generalised assumptions of applicability? See available information or check for currently needed data.

3.4 Glyph shaping and positioning

Does the script in question require additional user control features to support alterations to the position or shape of glyphs, for example adjusting the distance between the base text and diacritics, or changing the glyphs used in a systematic way? Do you need to be able to compose/decompose conjuncts, or show characters that are otherwise hidden, etc? See available information or check for currently needed data.

3.5 Cursive text

If this script is cursive (eg. Arabic, N’Ko, Syriac, etc), are there problems or needed features related to the handling of cursive text? Do cursive links break if parts of a word are marked up or styled? Do Unicode joiner and non-joiner characters behave as expected? See available information or check for currently needed data.

3.6 Baselines, line-height, etc

Does the browser support requirements for baseline alignment between mixed scripts and in general? See available information or check for currently needed data.

#71 Differences in physical size for fonts affects baseline alignment

By and large, Indian scripts in browsers are rendered with default fonts provided by back-end Operating Systems. If there are major differences in physical size for a particular font size, which most of the times are, it shows the mixed-script text in an un-organized way. This obviously affects all the base-line attributes.

3.7 Transforming characters

Does your script need special text transforms that are not supported? Does your script convert letters to uppercase, capitalised and lowercase alternatives according to your typographic needs? Do you need to to convert between half-width and full-width presentation forms? See available information or check for currently needed data.

3.8 Grapheme/word segmentation & selection

This is about how text is divided into graphemes, words, sentences, etc., and behaviour associated with that. Do Unicode grapheme clusters appropriately segment character units for your script? When you double- or triple-click on the text, is the expected range of characters highlighted? When you move through the text with the cursor, or backspace, etc. do you see the expected behaviour? (Some of the answers to these questions may be picker up in other sections, such as line-breaking, or initial-letter styling.) See available information or check for currently needed data.

#74 Spans around partial orthographic syllables can't be styled

If you put a span around the ம first character of the two character syllable மொ in மொழி (eg. in order to color or otherwise style it), the half part of o vowel-sign no longer appears at the start of the word, but looks like incorrect inputting at the wrong place.

Apparently this requirement is a bit difficult to cater to as it requires the knowledge to be transferred by the font to the rendering engine about which part of the glyph is attributed to which code-point in storage. However, if it can be achieved, nothing like it.

#72 Grapheme clusters fail to represent syllabic conjuncts

The Unicode concept of 'grapheme cluster' currently fails to represent the small number of conjuncts that are used in modern Tamil, ie. kṣa க்ஷ and the two alternative sequences for srī, ஶ்ரீ and ஸ்ரீ. This means that various editing operations, line breaking algorithms, vertical text, etc. are liable to break text at the wrong point when those conjuncts are used. For more details, see the relevant sections.

Indic Layout Requirements provides a grammar for indian orthographic syllable boundaries which works for the consonant clusters in Tamil which don't use conjuncts.

CSS uses the concept of 'typographic character unit', rather than grapheme cluster, in its specs with the explanation that these cases are beyond the scope of the grapheme cluster concept and that implementations should provide appropriate support.

See requirements at: Indic Layout Requirements

3.9 Inline features & punctuation

Are there specific problems related to punctuation or the interaction of the text with punctuation (for example separation of punctuation from previous text, but allowing no line break between)? Are there issues related to handling of abbreviation, ellipsis, or iteration? Are there problems related to bracketing information or demarcating things such as proper nouns, etc? See available information or check for currently needed data.

3.10 Text decoration

This is about ways of marking text (see also specific sections dedicated to quotations and inline notes/annotations). Is it possible to express emphasis or highlight content as expected? Bold, italic and under-/over-lines are not always appropriate, and some scripts have their own unique ways of doing things, that are not in the Western tradition at all. Text delimiters mark certain items or sections off from the main text, such as book names in Chinese, quotations, head markers in Tibetan, etc, and often involve the use of punctuation. Is there any behaviour that isn't well supported, such as overlines for numeric digits in Syriac? Are there issues about the positioning or use of underlines? Some aspects related to the drawing of lines alongside or through text involve local typographic considerations. Do underlines need to be broken in special ways for this script? Do you need support for additional line shapes or widths? Does the distance or position of the lines relative to the text need to vary in ways that are not achievable? Are lines correctly drawn relative to vertical text? See available information or check for currently needed data.

#75 Underline and overline need to clear glyph ascenders and descenders

Indian language text has some signs called as Matras which sometimes join above the shirorekha or below the normal baseline.

Applications should ensure that the underline and overline when getting rendered, should adaquetly be taken into consideration. Chrome browser seemed to be perfectly rendering the underline and overline feature by breaking the same where a matra/vowel sign occurs. Firefox also handles it a bit differently by appropriately lowering or heightening the respective lines.

3.11 Quotations

Are there any issues when dealing with quotations marks, especially when nested? Should block quotes be indented or handled specially? See available information or check for currently needed data.

3.12 Inline notes & annotations

The ruby spec currently specifies an initial subset of requirements for fine-tuning the typography of phonetic and semantic annotations of East Asian text, including furigana, pinyin and zhuyin fuhao systems. Is is adequate for what it sets out to do? What other controls will be needed in the future? What about other types of inline annotation, such as warichu? (For referent-type notes such as footnotes, see below.) See available information or check for currently needed data.

3.13 Data formats & numbers

If the script has its own set of number digits, are there any issues in how they are used? Does the script or language use special format patterns that are problematic (eg. 12,34,000 in India)? What about date/time formats and selection - and are non-Gregorian calendars needed? Do percent signs and other symbols associated with number work correctly, and do numbers need special decorations, (like in Ethiopic or Syriac)? How about the management of personal names, addresses, etc. in web pages: are there issues? See available information or check for currently needed data.

#76 Data formats need to be controllable by users

Both ASCII and Tamil Numerals are used in Tamil. Most of the user community identifies with both. However, mono-lingual (mono-script rather) readers may want a fully localized web page including numbers, date, time and calendar to name a few.

4. Lines and Paragraphs

4.1 Line breaking

Does the browser capture the rules about the way text in your script wraps when it hits the end of a line? Does line-breaking wrap whole 'words' at a time, or characters, or something else (such as syllables in Tibetan and Javanese)? What characters should not appear at the end or start of a line, and what should be done to prevent that? See available information or check for currently needed data.

#78 Break anywhere fails on conjuncts

line-break:anywhere causes lines to break inside words. It should break lines on grapheme cluster boundaries for all consonant clusters apart from the 3 special conjuncts. Chrome doesn't support the anywhere value. Firefox and Safari behave as expected.

The exceptions are the sequences க்ஷ k͓ʂ, and ஶ்ரீ ʃ͓ɾī / ஸ்ரீ s͓ɾī (which are synonyms). These sequences should not be broken during line breaking. Correct line breaking of these conjunct-forming sequences are not supported by default by Unicode grapheme clusters (which split them in two), and requires the application of tailored rules.

Test: line-break:anywhere should not break shri or ksha conjuncts.

Firefox is ok for shri and for ksha without a vowel-sign, but in ரிக்ஷா leaves க் on previous line, still shaped for half a conjunct. Safari is ok for shri in HTML, but leaves ஸ் behind in textarea; for ksha, in textarea leaves க் behind, in HTML initially moves whole word to next line, then puts ரி back at end of line as you decrease the window width.

Similar results are produced for word-break: break-all, except that Chrome supports this property and value. Chrome wraps ஸ்ரீந as a single unit and ரிக்ஷா as a unit.

The impact of this is advanced, although it would be good to fix it.

#77 Dandas are wrapped alone to the beginning of a line

In general, there are no issues with ordinary Tamil line-breaking. Line-break opportunities occur at the spaces between words, and characters that are not supposed to appear at a line start do not.

The exception to the latter is that [U+0964 DEVANAGARI DANDA] and [U+0965 DEVANAGARI DOUBLE DANDA] will wrap alone to the beginning of a line if there is a space between them and the previous word. This is the case for all browsers.

Test: A line should not start with a danda character even if it is separated from the previous word by a space.

The impact of this is advanced, although it would be good to fix it.

#73 Dandas are wrapped alone to the beginning of a line

Tamil uses full stop to represent the end of the sentence. However if the user wants to use the danda or double danda, as per to Unicode recommendation, they have to come from Devanagari block of Unicode. Devanagari phrase separator । U+0964 DEVANAGARI DANDA or ॥ U+0965 DEVANAGARI DOUBLE DANDA are encoded in the DEVANAGARI block with the intent that they should be used as common punctuation for all the major scripts of India including Tamil.

The properties of purna viram and deergh viram should be the same as the properties of FullStop or other punctuation marks, and a new line should not begin with purna viram and deergh viram.

See also hyphenation below.

4.2 Hyphenation

Is hyphenation used for your script, or something else? If hyphenation is used, does it work as expected? (Note, this is about line-end hyphenation when text is wrapped, rather than use of the hyphen and related characters as punctuation marks.) See available information or check for currently needed data.

#79 Hyphenation isn't supported

None of the major browsers supports Tamil hyphenation out of the box. This is a problem for text in narrow columns, because Tamil words tend to be long.

In the case of Tamil, simple dictionary lookup is not enough, because the language is highly inflexional and a significant element of morphological analysis is needed in addition to other Tamil-specific orthographic rules for placement of break opportunities.

Tamil also needs to hyphenate without adding a visual marker, as shown in the picture where yellow text indicates hyphenated words that have been wrapped.

hyphenation_ta

Specs: css-text-3 provides the hyphens property, but browsers need to implement the actual mechanism for processing the text.

(It is possible to produce manual hyphenation, but given the number of words that need to be hyphenated in a typical Tamil text, this isn't going to be very useful. The html spec defines the wbr element, which could be used because it doesn't produce a hyphen mark, it only marks a potential break point. ­ does produce a hyphen, so it isn't helpful for Tamil.)

css-text-4 provides the hyphenate-character property, which should allow authors to indicate that Tamil should have no visible marker for hyphenation.

Tests & results: interactive test, hyphens:auto will produce hyphenation for Tamil words

Gecko, Blink, and Webkit all fail to hyphenate the Tamil text when hyphens is set to auto.

Santhosh Thottingal has written a JavaScript-based tool for hyphenating Indian scripts, which mostly relies on syllable-breaks plus a few additional rules.

Priority: The impact of this is basic, because of the difficulties of handling text in narrow columns.

4.3 Text alignment & justification

When text in a paragraph needs to have flush lines down both sides, does it follow the rules for your script? Does the script need assistance to conform to a grid pattern? Does your script allow punctuation to hang outside the text box at the start or end of a line? Where adjustments are need to make a line flush, how is that done? Do you shrink/stretch space between words and/or letters? Are word baselines stretched, as in Arabic? What about paragraph indents, or the need for logical alignment keywords, such as start/end, rather than left/right? See available information or check for currently needed data.

#80 Inter-character spacing is not adjusted for narrow columns

Justification of Tamil in narrow columns of text can lead to large blank spaces between words, or a loss of justification when only one word fits on a line. To mitigate this, especially in the absence of hyphenation, lines that are justified in Tamil should automatically add inter-character adjustments.

Those adjustments, however, should equally expand the space between all unconnected, spacing glyphs (including the space between various vowel-signs and their base), rather than solely put space around syllables or grapheme clusters.

When only one word fits on a line, that word should be stretched to fit the whole line.

None of the major browsers automatically does anything other than stretch inter-character space at the moment.

Test: When a narrow column means that only one word fits on a line, the word will be stretched to fit the whole width of the column

The impact of this is marked as advanced, because oversized gaps in text seem to be fairly common in Tamil printed materials, but perhaps this should be basic instead?

4.4 Letter spacing

Some scripts create emphasis or other effects by spacing out the words, letters or syllables in a word. Are there requirements for this script/language that are unsupported? (For justification related spacing, see below.) See available information or check for currently needed data.

#81 Text units for letter-spacing are incorrect

Tamil is an alphasyllabary in nature and Akshar is the writing system to write the language. It is very common in the Tamil script to represent/brake words based on Akshara, which is instinctively recognized by users of the script. The same requirement is given here.

As the W3C specification points to Unicode Text Segmentation (TR 29), it is observed that most of the browsers support it (e.g. Chrome and Firefox) when word is spaced by akshar. Also, in cases where there is wrong Akshara formation e.g. Consonant+Matra+Matra, the breaking seems to stack ill formed akshara into one set instead of clearly breaking it separate. This breaking behaviour needs to improve.

4.5 Lists, counters, etc.

The CSS Counter Styles specification describes a limited set of simple and complex styles for counters to be used in list numbering, chapter heading numbering, etc.The rules plus more counter styles (totalling around 120 for over 30 scripts) are listed in the document Ready-made Counter Styles. Do these cover your needs? Are the details correct? Are there other aspects related to counters and lists that need to be addressed? See available information or check for currently needed data.

#82 Local counter styles are not well supported

Two CSS counter styles are defined in the document Ready-made Counter Styles: tamil (numeric) and ancient-tamil (additive). The numeric style is defined also in the CSS Counter Styles specification, the other relies on the user-defined mechanism specified in that spec in order to be applied.

Test results for the numeric counter style indicate that it is only supported by Firefox. Chrome and Safari do not support it.

An ancient-tamil additive counter style is defined in Ready-made Counter Styles, but test results indicate that no browsers support it.

Both of these counter style could be defined by users if the Counter Styles spec's generic mechanism for defining counter styles was implemented. The spec is essentially done, but so far only Firefox has implemented it. See tests and results for implementation of this mechanism.

There is also a question about whether an alphabetic counter style needs to be defined for Tamil.

The impact of the lack of support cited here is mediated by the tendency to use western counters, however there appears to be a desire for native styles (at least the numeric) to be made available. So the impact is being marked as advanced.

4.6 Styling initials

Does the browser or ereader correctly handle special styling of the initial letter of a line or paragraph, such as for drop caps or similar? How about the size relationship between the large letter and the lines alongide? where does the large letter anchor relative to the lines alongside? is it normal to include initial quote marks in the large letter? is the large letter really a syllable? etc. Are all of these things working as expected? See available information or check for currently needed data.

#84 initial-letter positioning is not widely available

Having selected the correct text for highlighting, it is important to ensure proper alignment of the baseline and height of the initial letter highlight relative to the other lines of text. This doesn't work well without help from the dedicated CSS properties, initial-letters and the initial-letters-align. Unfortunately, only Safari supports the first property, and it requires the -webkit prefix, so this is still an immature feature.

Safari aligns the alphabetic baseline of the highlighted text with that on the specified number of lines. The relationship between the highlighted letters and the first line of the paragraph appears to be based on cap height, but is not clear. The requirements for that relationship are not yet really clear, despite the information in Indian Layout Requirements.

The impact here is advanced, since it is mainly needed for advanced layouts.

#83 ::first-letter selection fails on conjuncts only

To select the characters to which styling is applied, it is possible to manually mark up the initial letters and apply styling to them. However, it is more flexible and easier to use the CSS ::first-letter pseudo-element.

Test ::first-letter selection will keep conjuncts together.

The tests show that Firefox, Chrome, and Safari all handle consonant plus pulli, and consonant plus vowel-sign combinations correctly, include prescript vowel-signs. They also handle typical punctuation that occurs at the beginning of a Tamil paragraph.

Chrome and Safari, however, fail to treat as a unit the KSHA conjunct (although it's not clear that any Tamil words begin with that sequence of characters) and SHRI conjuncts, whether the latter is spelled with ஸ or ஶ. The latter conjunct does begin a few words, such as ஶ்ரீநகர். Firefox selects the whole conjunct in all cases.

The impact here is advanced, since the impact of the failures cited here on the user is likely to be very small, especially since they can resort to markup in the rare cases where the conjuncts are not properly handled.

5. Page & book layout

5.1 General page layout & progression

How are the main text area and ancilliary areas positioned and defined? Are there any special requirements here, such as dimensions in characters for the Japanese kihon hanmen? The book cover for scripts that are read right-to-left scripts is on the right of the spine, rather than the left. Is that provided for? When content can flow vertically and to the left or right, how do you specify the location of objects, text, etc. relative to the flow? For example, keywords 'left' and 'right' are likely to need to be reversed for pages written in English and page written in Arabic. Do tables and grid layouts work as expected? How do columns work in vertical text? Can you mix block of vertical and horizontal text correctly? Does text scroll in the expected direction? Other topics that belong here include any local requirements for things such as printer marks, tables of contents and indexes. See available information or check for currently needed data.

5.2 Footnotes, endnotes, etc.

Does your script have special requirements for footnotes, endnotes or other necessary annotations of this kind in the way needed for your culture? (There is a section above for purely inline annotations, such as ruby or warichu. This section is more about annotation systems that separate the reference marks and the content of the notes.) See available information or check for currently needed data.

5.3 Page headers, footers, etc.

Are there special conventions for page numbering, or the way that running headers and the like are handled? See available information or check for currently needed data.

5.4 Forms & user interaction

Are vertical form controls well supported? In right-to-left scripts, is it possible to set the base direction for a form field? Is the scroll bar on the correct side? etc. See available information or check for currently needed data.

6. Other

6.1 Culture-specific features

Sometimes a script or language does things that are not common outside of its sphere of influence. This is a loose bag of additional items that weren't previously mentioned. This section may also be relevant for observations related to locale formats (such as number, date, currency, format support).

6.2 What else?

There are many other CSS modules which may need review for script-specific requirements, not to mention the SVG, HTML, Speech, MathML and other specifications. What else is likely to cause problems for worldwide deployment of the Web, and what requirements need to be addressed to make the Web function well locally?

Show summary