– DRAFT –
WHATWG, i18n, APA joint meeting

APA + i18n + WHATWG Joint Meeting

Symbols and Ruby semantics

<matatk> w3c/html-ruby#24

matatk: Matthew, Janina's co-chair at APA and a member of the TAG
… sharing an example of using symbols to help illustrate text on a page
… Video is a major application
… People who have trouble with text, use video, but have trouble with text like titles etc. in video

janina0: There's several hundred thousand people who struggle to use text, and using symbols is helpful. Want to make this possible as they currently have trouble using the Web

matatk: [shows a pancake recipe marked up with bliss symbolics as annotations]
… It's a language that evolves over time.
… Bliss is very comprehensive

<Fazio> this work directly correlates to WCAG 2.1's Success Criterion 1.3.4: Identify Input Purpose which has an autocomplete requirement

matatk: but there are also other symbolic languages
… people generally only know one set of these
… Bliss codifies some concepts, like our dictionary
… authors can look up a concept and get the symbol
… ARASAAC is more pictorial
… Most of the time there's a 1-1 correspondance
… But sometimes 1-2 or 2-1 with Bliss
… We're proposing to do this stuff by using existing standards most of the way, and some proposed changes to ruby which will enable us to do the rest
… How we expect it to work is, the concepts are going to be effectively the unicode codepoint strings that correspond to the Bliss "letters" that make up the concept
… Bliss, or a subset of it, is going into Unicode (not official yet but on its way)
… And because Bliss is effectively a superset, we can use those codepoints as a key into the concepts, and there can be a font that draws it into the right symbol set for the user
… So we're using the font to do the mapping
… Because the symbols are rendered above the text to which they relate, suggestion was to use Ruby to render it
… However, that's not what the <ruby> element is for
… But there's a change being proposed for a "type" attribute, which in conjunction with lang= attribute, could identify annotations as being bliss and render appropriately

jcraig: type attribute is not specifically related to Bliss, but rather distinguish whether it's whether it's a phonetic key or a supplemental type

jcraig: We think combining supplemental with bliss lang, could identify without having any detrimental effect on CJK usage

matatk: There's type attribute proposed, if that become part of the spec, we think we won't need further spec changes to do this as well
… so this is why we are involved, because we would need that spec change to work

<Fazio> Is any coordination being done with AAATE, ATIA, or other organizations that specialize in AAC?

florian: Yes, the type attribute as being considered, primary purpose is to affect the accessibility rendering of this
… there are use cases where you want to read the ruby and the base, and cases where you don't
… in some cases reading one or other is only conveying half the info
… but others it's phonetic annotation, and you should only read once
… that's the fundamental purpose of this -- avoid reading it twice, because if you do it wrong it can be annoying to impossible to understand depending on the content
… So about this, what is the intended accessibility tree and screenreading effect that you expect for content like this?

<jcraig> The issue related to the proposed `type` attribute on Ruby is: w3c/html-ruby#24

florian: You mentioned using lang attribute, maybe that's all we need. Depends on what you want for audio rendering.

matatk: Bliss is a language, but we're only using it for list of concepts
… so not doing any translation, using the font simply for mapping
… none of the other symbols have language codes

matatk: in terms of accessibility tree, we hadn't thought ths would be directly of use to screenreader users
… as audience is people reading the text and using images to help understand the text
… but it's an item in the page

<Fazio> this should be a joint effort with Augmentative Alternative Communication manufacturers for compatibility so a user can import symbol sets as part of their Assistive Tech

matatk: I suppose my answer, I would expect it to work, essentially as a bunch of glyphs so you would get the Unicode names for each codepoint
… so I expect the screenreader would announce those names

Florian: [example]

matatk: Yeah. That's my immediate impression.

matatk: If Bliss goes into Unicode (as we expect), only about 1300 are going in (7000-ish symbols) because we don't want to change unicode when adding to it
… hope to create new compounds from the existing codepoints
… similar to creating variations on emojis

janina0: Didn't think about using with screenreaders. Don't think it has been tried.

jcraig: Haven't tried, but shouldn't be a problem. Do screenreaders + emoji all the time.

<Zakim> zcorpan, you wanted to ask if the type attribute impacts the a11y mapping

zcorpan: What is a11y mapping for ruby with a type attribute?
… for mode with reading base and not pronunciation ...?

florian: One is to read both in sequence
… other is more subtle, I'll give an example in Japanese
… when meant to read once, it's not "read the base" or "read the annotation", it's "read it once" and necessary information is in both
… Japanese furigana is phonetic information, but it is incomplete
… because Japanese is pitch language, some words differ by pitch alone
… base text can disambiguate
… on the flip side, if you hvae a base character, it can be read in different ways
… phonetic annotation disambiguates which reading to use

zcorpan: So would have to expose both

florian: Yes

florian: Basically need "read a single pronunciation from both this info" vs "read both in sequence"

Murata: [introduces self]

<xfq> Text to Speech of Electronic Documents Containing Ruby: User Requirements

Murata: I'm skeptical about introducing new type
… If you have a dictionary for text-to-speech, you know the readings
… if the attached ruby doesn't match any of the possibilities, then it is not a phonetic hint
… and in that case, both should be read aloud

anne: So you can derive from the text?

MURATA: From text and a dictionary. If the ruby matches one of the readings, then it is phonetic.
… Then you just choose that possibility from your dictionary, producing much better result.
… If it doesn't match what's in the dictionary, it is not a phonetic hint.
… And you have to read both the base and the ruby.

anne: Presumably all TTS needs a dictionary

MURATA: However dictionaries have been hstorically very limited
… but now things are changing. Dictionaries are getting much better than before.
… When we use ruby for phonetics, I'm skeptical

Fazio: Initially I was excited about this work. I mentioned last year to head of assistive tech of ? and he expressed concern about creating symbol sets here
… Alternative communication sets have a unique science to them
… There's a drawn-out process to create these
… we need to involve created of these symbol sets in this work
… so that we don't repurpose symbol sets for other purposes
… [missed]
… It's bring your own device, Internet recognizes it
… we will run into ton of internationalization problems

<duerst> [There may be the case that an author comes up with a new way to read some combination of characters that's not in the dictionary, even if the dictionary is quite big. But then the argument may be that both should be read, to make sure the listener gets the fact that this is a new reading.]

Fazio: For example, we saw a symbol that looks like person getting feet massaged on a table, but it was emergency room for not feeling well
… so symbols are not universal

matatk: +1

<Zakim> duerst, you wanted to ask about language-specific versions of bliss and to ask whether the idea is really to use the same codes for bliss and arasaac

duerst: First question, the sybols on the left and right, did I understand you want to have bliss encoding, but display as on the right?

matatk: Yes, Unicode is encoding Bliss.

duerst: There's a 1-1 mapping defined...

anne: The codepoints create glyphs, use font to display

florian: And doesn't have to be 1-1, because can use ligatures if necessary

duerst: You say Bliss is a language, but I suspect that there are also influences from the host language.
… These people live in a community.
… Things like syntax and word order

matatk: The various symbols sets that exist generally are designed with the cultural norms and sensitivies of a particular culture in mind
… It's up to the page author to pick the appropriate symbols.
… We're not trying to translate here, just mapping from a concept that exists in Bliss
… Things like word order don't come into it, because would follow word order of the page.
… We're just adding pictograms to help the reader understand.

duerst: Maybe you need to mark it up as "bliss for English" or "bliss for whatever"

r12a_: Or maybe tag the glyph variations
… We're talking about using symbols for illustration

<Zakim> eemeli, you wanted to ask Isn't Bliss more appropriate a script rather than a language?

eemeli: Following on from what Martin said, recognizing that Bliss forms its own language, my understanding is its use as ruby it would be much more helpful to consider Bliss to be a script, rather than a language.
… The examples you show are annotations on English grammar and structure and words, presenting English as Bliss, rather than Bliss as a language itself.
… I think that aligns also with the understanding of how text-to-speech would handle this.
… If the Bliss symbols are representing English, then the English content might indeed be the only thing that needs to be spoken out.

janina0: Bliss is not a spoken language

duerst: It's closer to a sign language, yeah?

janina0: It's a symbolic language

anne: yeah. But question is, if someone tries to read it with a screenreader, what happens?

<Zakim> jcraig, you wanted to respond to Matt's comment about how screen reader users get localized symbols (including emoji, but presumably Bliss, too) and to mention low vision users use screen readers, too and to briefly mention one potential implementation for accessibility and to mention to Murata-san that it seems I have misattributed your view in the "consensus" notes from our lunch yesterday: w3c/html-ruby#24 (comment)

jcraig: There's quite a few things to review

jcraig: Reading to Matt's comments about localizing symbols, such as emoji
… The accessibility use case, even for spoken representation of this, isn't just blind users, but also low-vision users
… who may be able to see the symbols on the screen

<Fazio> wouldn't screen readers need some kind of ignore tag for the bliss symbol because it would be redundant

jcraig: You can point at a symbol and ask it to speak
… There are different ways it's useful for a textual representation to be used even if not for blind user
… But also acknowledge that a fully blind user wouldn't use this, it's primarily a visual or symbolic representation

jcraig: Regarding localization, sometimes it's tied to the framework, sometimes tied to Unicode representation
… but there's alternative text for a symbol or for a combination of symbols
… Those get localized by the system into their own strings, which can have grammar as well
… So each symbol could have a representation as well
… it's closer to a sign language than spoken language
… that said ASL and other SL have their own grammar
… There's a separate transliterated sign language thats separate from ASL

jcraig: Murata-san, I thought we'd concluded that a type attribute would be needed!
… so I was surprised

jcraig: Simon had asked about implementation details, but this could be attributed strings. may not work for all platforms, but could work on Apple at least

r12a_: As Janina said, there's no phonetic value to these glyph sets
… the phonetic value is in the text they're associated with, and that's what would get read by a reader
… The Bliss symbols can be used as a language because there's grammar, tense, etc. built into the way they are used when used as pure symbols
… but the way used here, they are illustrations for individual words
… different usage pattern
… and that raises concern that one of the fundamental things we've discussed is a type for ruby

<Fazio> symbols usually illustrate sentences or actions not words

<Fazio> for AAC users

r12a_: but the issue we were trying to solve was largely what Murata was discussing, which is how to pronounce stuff
… not an open-ended classification
… Do you pronounce the base or note, or some extra magic, or what
… That's about pronunciation
… and that worries me that we diverting ruby for other purposes

florian: The type attribute we've been discussing, it's not very mature
… next proposed step was "maybe write an explainer"
… we'll be exploring, but still exploring
… as to Murata-san's comment, I partly agree but only partly
… As dictionaries get better, number of cases where UA can guess increase
… but I don't think exhaustive complete dictionary can exist
… There will always be cases they can't guess, and always UAs that have worse dictionaries than others
… so if author can specify, there's a use case for that
… but that's what explainer will be about

florian: As for here, the type attribute, if it comes to be, doesn't classify for purpose of classifying. It's to help accessibility tree
… based on what you're doing, you're displaying them
… primary goal is achieved

<Fazio> for my memory, I have 2 comments: decorative, phrases not words

florian: if you want to classify beyond that, for what purpose?
… maybe you want to hide them, but then can just use class
… or maybe not
… but if we are classifying things, need to answer "to what purpose"
… our purpose was pronuncation

r12a_: Additional concern wrt using ruby is, if we are using the glyph sets in Japanese, then yes, we may be able to distinguish whether pronunciation, but we already use ruby for text on both sides for descriptions as well as phonetics
… so seems if we had 3 types of ruby, doesn't feel clean

florian: Ruby can be nested infinitely, theoretically

Fazio: First that comes to mind is, symbol set is non-text content
… shouldn't it be marked as decorative?
… it's redundancy of text that's already there
… why read it

Fazio: Secondly, discussing symbols for words, but that's dangerous
… if we think about in terms of words, we have autocomplete
… someone using AAC, doesn't have symbols for items, more so for actions
… so phone number would have image of making a phone call rather than a phone

<xfq> If it is encoded in Unicode, it becomes text content, it's just cannot be read (or maybe it can be read using the Unicode names of the characters).

Fazio: plus if you have symbols for every word on page, ad nauseum

fantasai: To Richard's point about distinguishing between Bliss as its own language vs as an annotation, and whether English, Chinese, or other variation on Bliss...

<jyasskin> I liked eemeli's point that this might be a script.

fantasai: Maybe the language tag needs some thought on what it's doing - bliss with an English variant - zbl-en for example, or en-bliss i.e. English in bliss symbols, are two different concepts.

fantasai: I've heard several descriptions of what we're doing here. Need more info so the language tags can be correct. Maybe we need multiple patterns so the language tags can be correct.

fantasai: re type attribute, this was to distinguish the different types of pronunication we need. I don't think it would be problematic to introduce new types as long as they're reasonably exclusive of each other.

matatk7: Next steps is explainer (s)

<eemeli> English in Bliss would be en-Blis.

DOM Localization

<jcraig> en-us-zbl?

DOM Localization

eemeli: Wanted to discuss whether to do this thing, and which things
… and where to advance the work and where to do that

eemeli: On Tuesday had breakout on DOM localization. I'll do a recap of what that means.

<xfq> Breakout session on DOM localization

eemeli: we'd like to introduce localization as a capability of the Web Platform

Slideset: https://docs.google.com/presentation/d/1ON2yeocyDSVr8r0cg8ZlhgpFfsvloznq1pJqYSADJs8/edit and archived PDF copy

eemeli: Right now when you produce anything for more than one locale, you end up needing to find a solution for localization from the many available libraries out there
… This ends up being slightly problematic, because nearly all of these eventually limit what is expressable in the UI that you present to your users
… Most localization are just bits of strings, present in one locale or another. This is just text replacement.
… This covers 80% of the work in most applications
… Most of the rest is inserting a string into a placeholder.
… Easily handled by simple solution, so often that's what's done.
… What remains it the last few % of strings that don't fit this model.
… E.g. "Number of unread messages: 3" rather than "You have 3 unread messages." vs "You have 1 unread message." vs "You have no unread messages."
… Expression in human language thatis more natural vs what works for templating.
… And the complexity of this varies a lot across languages.
… These complexities are handled by systems that were built incrementally as the add capability for handling additional grammatical variations.
… These end up limiting expressibility of content that needs to be localized.
… It would be good improve sittuation of the Web for this!

eemeli: Last few years we've been working on a new message formatting specification called "Unicode Message Format"
… which is a basis on which we ought to build a solution for the Web

<xfq> https://messageformat.unicode.org/

eemeli: but it's not a whole solution
… What exists now is part of UTS35, defines what a single message might look like
… including dates, etc.
… but doesn't say how to integrate with HTML content
… e.g. a link that you might want to localize
… How fundamentally do you combine a corpus of messages with a page?
… This is DOM localization project!
… For Mozilla we're particularly interested in this, because we discovered that various systems we used are not conducive to our use cases
… including the developer experience

<xfq> MessageFormat spec

eemeli: so this experience is one of the things we built message format on

eemeli: What we would like to start introducing as part of HTML is a page could refer to localization resources
… Much the same way as referring to external style resources
… and then refer to those messages
… which then become part of the rendered page itself.
… In the simplest case, it's a string entered into the content of the element
… but could possibly include elements within itself (e.g. links)
… and could also affect not just the content of an element, but also attribute values
… We already have in HTML spec translatable attributes
… which correspond with sorts of content we want to make localizable
… This of course relies on having something like [slide] a resource file format for localizable content
… where you can have key to value mappings
… which the run-time system can then handle

eemeli: Now is the time to build the details of this at W3C, building on top of the work at Unicode CLDR

<xfq> translatable attributes: https://html.spec.whatwg.org/multipage/dom.html#translatable-attributes

eemeli: And that has led to these big questions [slide]

1. Should we work on introducing localization as a capability of the Web platform?

2. Is the i18nWG an appropriate form for this standardization?

3. How specific to localization should the HTML capabilities be?

eemeli: For example, we could limit capabilities or expand them, which changes the design

matatk7: I see the use cases, and I've seen them in native apps wher emore app-oriented content
… where the line is drawn as to whether goes into page or into message file, wondering how you draw the line
… also, example you gave is a button
… how would that relate to attribute to attributes like aria-label

<Zakim> matatk, you wanted to ask about line-drawing and accessible names

eemeli: Do we want to build a system for UI-type strings, or do we want to build a system that is well-supportive of longer-form content that may be localizable.
… Valid question, both answers are valid. Have to decide.
… wrt aria-label, they are considered translatable, so you would put those in the attribute localization

<Zakim> jyasskin, you wanted to ask why this ought to be done on the client

eemeli: [shows syntax]

jyasskin: Generally think this is a good idea. But why should it be done on the client side?

eemeli: Because it can, and because it's not always clear what is best done on client or server
… if we design the system to work on the client, it becomes easy to move that work to server side
… In a context where web page is interactive and user could enter a number, and process on the client side where could result in a message that is dynamically displayed
… then need to manage that localization
… But if we start with server-side assumption, much more conflict and mismatch.

florian: This looks like something that could be client-side or server-side, better to design to be able to do either way

florian: You mentioned injecting not just plain text, but possibly marked-up text
… because markup might depend on language

<duerst> [[sorry this is late] Status of Blissymbols in Unicode: see https://www.unicode.org/roadmaps/bmp/, where Blissymbols are listed as a script for which proposals have been formally submitted to the UTC or to WG2, with a link to https://www.unicode.org/L2/L2023/23138-n5228-blissymbols.pdf.]

florian: and in some cases might need to transform the tree
… we need to not accidentally re-invent XSLT
… We should not try to create generic transforms, and that's not our goal.

eemeli: Would like to make that goal / non-goal clear.
… I am an i18n/l10n guy, so not my place to say what the maximum limits are

florian: If we wanted XSLT, we'd use XSLT. Let's not accidentally go there.

fantasai: I think it's good to produce correctly and robustly localizable web applications. I think it needs a specific WG to do the work.

fantasai: I agree with florian that we should not be trying to create a generic translation system.

fantasai: It should be based on localization, and stick to that.

eemeli: So intent is, to liimt work to "what does syntax of a file like this look like"
… not how it is used
… or how it is used or integrated in HTML

fantasai: I think you need to think of these all together.

fantasai: And i18n is not the right group.

florian: I expect interest with with i18nWG wraps, so you should draw from this community
… but not only
… start in incubation
… but pull in these folks

r12a_: HTML Bidi doc from Aharon Lanin started here, but then moved into HTML for implementation

<Zakim> zcorpan, you wanted to say there's probably demand for a templating system in general

zcorpan: There are other use cases for replacing strings in a tree, e.g. templating
… and there are proposals for that "DOM Parts"
… maybe it would make sense to integrate, or layer one on top of the other, so we don't have incompatible templating systems

eemeli: So that would expand space of the solution to beyond localization, to be about templating and other factors

zcorpan: Yes, but templating could be designed for templating, and maybe this uses templating
… might not increase complexity for localization

<Zakim> jcraig, you wanted to ask how this would work across non-leaf-node element boundaries like `<x>You have <y>12</y> messages.</x>` and to say this looks sort of like client-side lproj, which may provide some benefit auto-translation if multiple provided languages were considered when translating to a non-provided language.

anne: For this to happen, the other thing needs to happen first

jcraig: My first point echoes Simon and Anne. At some point you'll want to have something to localize that's not just a flat string
… Another thing is, what's benefit of this over server-side translate
… Not clear to me if this is client-side with replacement strings
… but future benefit with if you provide English, Spanish, Urdu, maybe you can produce better translate for 4th language via tools
… Not clear what real problem to solve

fantasai: The example that was given earlier, if you're generating on the client side... if all you have is simple strings that don't need morphological changes, you can template in on the server side the generation of the labels....
… but if you have something that needs linguistic processing - like number of messages - you have to have that on the client side because it needs dynamic info. And then if you are doing it on the server side too you are duplicating on both sides.
… and you can send just the language that the user is in. E.g. this is an English page, so it can just link to the English localization resource.
… But the framework of being able to solve these dynamic localization problems is the focus here.

anne: There's also Uncode MessageFormat going on?

eemeli: Yes, this is building on that technology
… But MessageFormat is about a single message
… plaintext
… whereas we're dealing with more complex content, can include markup, attributes, etc.
… The resource fileformat this this is an example of, this shows how to combine keys and values
… MessageFormat is only about a single key-value pair
… I tried to push for an expanded scope
… but they did not want to be in the business of defining a file format

r12a_: I develop lots of content, and one is family history. I don't want that to go through a server for privacy concerns. So glad there would be a client-side solution.

fantasai: responding to how this integrates with templating
… if there's a templating effort, these should be worked on together, because the needs of localization should be taken into consideration by the templating system and vice versa
… templating system needs localization knowledge, and localization needs to be compatible with templating approach.

anne: Templating might take awhile, first need to define insertion points
… but prior iterations, it does have all the bits you want to mutate. Children, attributes, etc.

<Zakim> jyasskin, you wanted to ask if we need a way to define the MessageFormat functions

jyasskin: MessageFormat has to insert child nodes, need to make consistent with templating stuff
… IIRC MessageFormat also has environment-defined functions. Would that be part of that integration?

eemeli: Yes, eventually. But not a first-pass solution.
… a lot of this can be advanced iteratively
… We can introduce part of the capability, and partial support for what is expressible in MessageFormat
… in a way that can allow for later extension to support more

florian: There are numerous localization frameworks that started with hypothesis that was too simple
… and ended up horrendously complex because they had to accumulate complexity ad-hoc
… we want to avoid that situation

eemeli: Worked out compromise with MessageFormat
… some work in DOM Parts for definign some placeholders
… if we build on top of that, it would become very easy to end up with an iterative mess or two different templating systems to support, which nobody wants
… MessageFormat syntax also supports function calls, formatting, selector functions, and other complexities necessary for linguistic variability of human language
… but also tried to limit, not everything we can imagine, but things which are necessary

eemeli: A bit challenging to figure out how to solve? Need a new WG?

florian: Suggest going informally for awhile, then make a WG.

<xfq> Explainer: https://github.com/w3c/i18n-discuss/blob/gh-pages/explainers/message-resources.md

eemeli: Explainer, proposed solution, including ABNF and a data model description on how to interpret meaning of the file
… for DOM Localization parts, not absolutely clear to me, should I be filing a WHATWG issue or ...?
… Is that work something that ought to wait for MessageResource work to be further along

anne: Filing an issue against WHATWG HTML or maybe DOM would be better
… Earlier is better, because then if ppl interested, they can get involved

eemeli: Usage of MessageFormat needs to be accounted for in MessageFormat design
… but things like comments and metadata for communication between developers and translators

matatk6: Lots of work, but next steps seem to be clear.

xfq: We have a preliminary explainer, which we can incubate at i18n. Once the discussions are more thorough, we can consider forming a working group or other next steps. Everyone is welcome to join the i18n WG discussions.