Public Last Call #2 Comments
Character Model for the World Wide Web 1.0
W3C Working Draft 30 April 2002

Comment (received 2002-04-30) -- split/concat example looks good, I think (LCI-191)

I'm not quite sure I understand it... maybe it's just my browser's poor rendering... but that string, 'cz', just looks like a normal ASCII string. I don't see how the accent thingy pops up when you delete the 'z'.

Comment (received 2002-05-14) -- normalization-sensitive unclear

I found the definition of normalization-sensitive in section 4.4 unclear.

Comment (received 2002-05-14) -- UTF-7

Is UTF-7 a unicode encoding form? (I am pretty ignorant about UTF-7 but I believe it exists and is a UCS).

Comment (received 2002-05-14) -- Typo

The last sentence of the fourth example in section 3.1.5 has the character 'o' should it be 'ö' instead.

Comment (received 2002-05-24) -- mapping between character codes and units of displayed text

In '[S] [I] Specifications and software MUST NOT assume a one-to-one mapping between character codes and units of displayed text.', does 'character codes' mean characters or character codes or both? Please clarify.

Comment (received 2002-05-27) -- Endorsement from RDF Core

For the sections 3.4, 4, 6, 9, C, D RDF Core endorses the last call working draft. We have found earlier drafts helpful in identifying how best to meet our responsibilities to RDF users world wide. (However, we do not intend to address all the requirements of these sections in the version of the RDF recommendations currently in working draft).

Comment (received 2002-05-27) -- breadth of scope

Concerning sections 1 and 2 RDF Core is concerned that the scope of charmod is overly broad. In particular, there appears to be no acknowledgement that some languages being defined by W3C working groups may not be intended as web languages and hence not have a need to address internationalization issues. There may be an implicit (and false) assumption that all W3C recommendations specify (only) web languages with processing models.

Comment (received 2002-05-28) -- RE: breadth of scope

Comment (received 2002-05-27) -- non-universality of processing model

For the section 3.5 RDF Core WG notes that the language is somewhat offputting for us as specification developers given that our specification explicitly does not have a processing model. We have no particular suggestions about this, nor would we object if the I18N WG chose not to address this issue.

Comment (received 2002-05-28) -- RE: non-universality of processing model

Comment (received 2002-05-27) -- no dependency on IRI draft

The main concern of the RDF Core WG is section 8. Any normative section of charmod MUST NOT depend on the IETF IRI draft which is not finished and is not yet stable. We draw attention to 'SHOULD use Internationalized Resource Identifiers (IRI) [I-D IRI]'. The IRI draft is only a draft, the reference to it is not normative, and the strength of this SHOULD dependency appears excessive ('not optional'). In particular, the IRI draft does not adequately address IRI equality (not merely functional equivalence in retrieval). Moreover, the bidi section presents a learning curve which developers are unlikely to want to climb before IRI has consensus around it; We have found the text in Xlink section 5.4 and XML Erratum 26 adequately clear for some of the IRI questions, particularly those that are most pressing for RDF and believe that charmod should merely:

- reiterate such text;

- reiterate the early uniform normalization model for the iris when regarded as unicode strings

Comment (received 2002-05-28) -- Use of word 'byte'

Proposal: Change 'the word bytes is generally considered to mean 8-bit bytes' to 'the word bytes is generally considered to mean 8 bits.'

Comment (received 2002-05-24) -- Re: 2nd Last Call for the Character Model for the WWW

I've tried to reconcile our original comments [1], your latest spec [2], and the disposition of issues [3]. Fortunately, we had few comments and most were FYI but I fear I've failed on some of the substantive ones.

[1] http://lists.w3.org/Archives/Public/www-i18n-comments/2001Feb/0017

[2] http://www.w3.org/TR/2002/WD-charmod-20020430

[3] http://www.w3.org/International/Group/charmod-lc/

For instance, LCC-117 [4] was summarized as, 'Section 3.6.2 (Private Use Code Points): Disagreement with our approach'. [5] LCI-95's disposition is 'N - Y S'. I presume this means you don't agree with the comment, there's no change, the issue is closed, and it was a substantive issue. But I don't know if we didn't explain ourselves well, or why you disagreed? So when I consider the original text 'Specifications MUST NOT provide mechanisms for private agreement between parties.' [6] I can see was was of concern. When I check the latest version I see 'Specifications SHOULD NOT provide mechanisms for agreement on private use code points between parties and MUST NOT require the use of such mechanisms.' [7] So that seems to have changed -- after a lot of time and confusion on my part?!

[4] http://www.w3.org/International/Group/charmod-lc/#LCC-117

[5] http://www.w3.org/International/Group/charmod-lc/#LCI-95

[6] http://www.w3.org/TR/2001/WD-charmod-20010126/#sec-Encodings

[7] http://www.w3.org/TR/2002/WD-charmod-20020430/#sec-PrivateUse

Comment (received 2002-09-10) -- Re: Private Use Code Points: Disagreement with our approach

If the wording has been changed to SHOULD NOT, I don't plan to persue this any further.

Donald

Comment (received 2002-09-23) -- Re: Private Use Code Points: Disagreement with our approach

Now that's it's archive in the WG archives, please consider it 'official' <smile/>.

Comment (received 2002-05-24) -- Re: 2nd Last Call for the Character Model for the WWW

Please do not state 'All W3C specs must conform.' I think you should:

a. state in the STATUS that the intent is that this will be used for W3C specifications.

b. state, 'any spec wishing to conform must ...' and how, when, and what unforeseen exceptions might be permitted becomes is a matter of W3C policy -- perhaps following Ian's suggestion.

Comment (received 2002-05-24) -- Re: 2nd Last Call for the Character Model for the WWW

[In the table of Last Call #1 comments,] [t]here's a YYYE for defining 'logical order' but I don't see it. (I do see 'logical selection model' is this the same?)

Comment (received 2002-05-24) -- Re: 2nd Last Call for the Character Model for the WWW

In a couple of places it appears you use an accented i in an example. This is hard to see that it is not a normal 'i', particularly when italicized.

Comment (received 2002-05-31) -- Conformance of new vs. old specs

Section 2, 'Conformance', contains the following statements:

[S] Every W3C specification MUST:

1. conform to the requirements applicable to specifications,

2. specify that implementations MUST conform to the requirements applicable to software, and

3. specify that content created according to that specification MUST conform to the requirements applicable to content.

[S] If an existing W3C specification does not conform to the requirements in this document, then the next version of that specification SHOULD be modified in order to conform.

It seems strange that 'Every...specification MUST...conform to the requirements', but that existing specifications that do not conform 'SHOULD be modified'. While we assume that the intent is to require conformance by *new* specifications without mandating updates to existing specifications solely for conformance reasons, the wording is certainly surprising and could be made clearer.

Comment (received 2002-05-31) -- determining relevant language for sorting

Section 3.1.5, 'Units of collation', contains the statement 'Note that, where searching or sorting is done dynamically, particularly in a multilingual environment, the 'relevant language' should be determined to be that of the current user, and may thus differ from user to user.'

While we agree that the user's language is frequently the most reasonable choice to be used in determining a collation to be used for various operations, it is most certainly not *always* the best choice. This is particularly true when massive amounts of data have been placed into a repository of some sort (e.g., a database) using the semantics of the data itself. For example, database systems often enhance retrieval performance through the use of special structures ('indexes') that are created long in advance of knowing what user might be retrieving the data. In such cases, it might be determined that the 'best' default language is the language of the data, or of the repository, or of some other entity.

To ensure that the statement in Section 3.1.5 is interpreted to allow this situation, the phrase 'should be determined' should ;^) be replaced with 'SHOULD be determined'. By formalizing the term ('SHOULD'), the Character Model document properly recognizes that some applications require the ability to have different defaults.

Comment (received 2002-05-31) -- How to avoid use of the term 'character'?

Section 3.1.7, 'Summary', contains a paragraph that reads: '[S] When specifications use the term 'character' it MUST be clear which of the possible meanings they intend. [S] Specifications SHOULD avoid the use of the term 'character' if a more specific term is available.'

This paragraph would be considerably more useful if it either contained a list of the possible meanings or contained a pointer to another location in the document that provides such a list.

Comment (received 2002-05-31) -- proprietary charset identifiers

Section 3.2, 'Digital Encoding of Characters', list element 4, contains the phrase '... is identified by an IANA charset identifier.'

In fact, there are a great many CESes that are identified by charset identifiers that are not assigned by IANA at all, but that are 'created' by proprietary means (e.g., corporations). The Character Model specification must not prohibit the use of CESes identified by charset identifiers assigned through other means.

To correct this, simply change '...is identified by an IANA charset identifier.' to '...is identified by a unique identifier, such as an IANA charset identifier.'

Comment (received 2002-06-13) -- Re: proprietary charset identifiers

[...] I assure you that I am not satisfied with the decision [...]

Comment (received 2002-05-31) -- DISCUSSION OF SUBSEQUENT ITEMS

Section 4.4, 'Responsibility for Normalization', is the section in which the Query WG is most interested. Intense discussions have taken place in the past over the subject of when normalization should, should not, must, or must not be performed, and what components of an environment have responsibilities in that area. Implementers of data repositories that might contain vast quantities of data (e.g., database systems) have expressed particular concerns about this, observing that some applications involve the need to store data very quickly, but retrieve it in a less urgent fashion, while other applications place severe demands on retrieval but have fewer constraints on storing data. In other words, the demands of users of applications, not a rigid policy, must govern *some* aspects of the decision about when normalization is performed and by whom.

Comment (received 2002-05-29) -- Objection to prohibition against receiver from normalizing text

Section 4.4, 'Responsibility for Normalization', specifies one requirement (on web content) that states '[C] In order to conform to this specification, all text content on the Web MUST be in include-normalized form and SHOULD be in fully-normalized form.' While this statement expresses a clearly desirable situation, it is 100% guaranteed that the web will *never* contain only include-normalized text. The character model MUST (irony noted) recognize that fact and give guidance for dealing with such data. A preferred alternative, currently prohibited by the Character Model document, is to allow a *consuming* application to do the normalization. [The Character Model] currently prohibits such action, as addressed in the next comment. This prohibition is unreasonable and we believe that its inclusion will dramatically inhibit adoption of the Character Model by products and implementors.

Comment (received 2002-05-31) -- Prohibition against normalizing suspect text

In Section 4.4, 'Responsibility for Normalization', another requirement (on implementations and on specifications) states: '[S] [I] A text-processing component that receives suspect text MUST NOT perform any normalization-sensitive operations unless it has first confirmed through inspection that the text is in normalized form, and MUST NOT normalize the suspect text. Private agreements MAY, however, be created within private systems which are not subject to these rules, but any externally observable results MUST be the same as if the rules had been obeyed.'

We do not object to the observation that normalization-sensitive operations are best performed on normalized text. However, the requirement (as stated) clearly prohibits a consuming application from normalize non-normalized text that it receives. We are quite opposed to this prohibition for a variety of reasons.

One problem is that the requirement doesn't make it clear what the consuming application's behavior must be, but it seems reasonable to conclude that the consuming application must reject the un-normalized text. Of course, such an application that silently rejects such text is unlikely to be considered user-friendly, so we might guess that an error can be raised in some manner. But that makes the application very unhelpful in general, since users (e.g., of web browsers) often wish to access text regardless of how rigidly it conforms to the Character Model's requirements.

The 'Private agreements' clause starts off in a promising manner, but then requires that the results of such agreements remain unhelpful to users of the applications, since the application is not allowed to produce 'observable results' based on handling un-normalized text.

So, what can be done to support applications (and specifications) that must deal with text cannot always be guaranteed to be normalized? We very much want certain classes of applications to be allowed to do normalization on un-normalized text and we are willing to participate in discussions that identify those classes of applications.

A rather cynical way out of this dilemma that can be imagined is for an application (e.g., a database management system) to 'read' suspect text and then 'create' brand new normalized text that just happens to be character-for-character identical to the un-normalized text it received. That obviously implies degenerating into games just to get around conformance requirements; instead, we must fix the specifications and requirements themselves.

Comment (received 2002-05-31) -- Prohibition against interim unnormalized states

Section 4.4, 'Responsibility for Normalization', contains a requirement that states: '[I] A text-processing component which modifies text and performs normalization-sensitive operations MUST behave as if normalization took place after each modification, so that any subsequent normalization-sensitive operations always behave as if they were dealing with normalized text.'

We believe that many implementors, on grounds of performance considerations, disagree with the requirement that normalization take place after each operation. While we recognize that the Note following the quoted requirement suggests a way to ease the performance issue (using what we call 'local normalization'), we believe that a couple of good examples will help ease implementors' concerns. More importantly, we believe that some application requirements are best satisfied by allowing such (normalization-sensitive) operations on text that has not yet been proven to be normalized. A requirement that such operations on such text result in fully-normalized text poses the unreasonable additional burden of doing complete normalization of possibly huge text without a clear application need to do so.

We urge the document's editors to modify the requirements to recognize the facts that some applications require the ability to use and manipulate un-normalized text and that operations on such text need not necessarily be normalized. In XQuery, we have a normalize function to allow applications to *force* normalization when they require normalized text and we believe that this is a useful way forward.

Comment (received 2002-06-06) -- CharMod restricts closed systems

In the effort to improve interoperability of text exchange across open applications on the Web, the Character Model should not restrict the ability for closed systems to leverage the Web and Web-based technologies The term 'closed system', as used in this document, refers to a system designed to support organizations communicating among themselves based on a contract into which all parties have explicitly entered.

The background of this spec states that 'the Web may be seen as a single, very large application...rather than as a collection of small independent applications.' Based on this premise, it is understandable why the CharMod spec chooses to require early normalization in a single canonical form. However, the Web and technologies that have developed to support the Web have also provided enormous value to closed systems, including intranet and extranet scenarios. The relationship between the evolution of the World Wide Web and its use in closed/private systems has been a mutually beneficial one. Private systems have benefited from the efficiencies of applying Web-developed standards and tools, which has in-turn increased the demand and support for these Web-enabling components. The current Character Model spec threatens to break this relationship by forcing restrictions on tools that are commonly used in closed systems, in order to exclusively support the goals of the open system Web.

It is apparent that the I18N WG has solid reasons for preferring Normalization Form C as an interchange format for the Web; however, it is not likely to be the optimal choice for all applications. There are many legacy systems (both applications and operating systems) that use a decomposed character normalization. It will be difficult for organizations to justify why they should adopt CharMod-based technologies (such as XML 1.1 over XML 1.0), which require transcoding to a less optimal normalization form with no benefit for their closed system. This is likely to lead to fractured use of technologies such as XML 1.0/1.1.

Finally, XML plays an important role as a data-interchange format in scalable, loosely coupled systems. The Character Model reduces XML to a format applicable only to natural language communication in one particular normalization form. This is unfortunate considering that vastly more bytes of machine-to-machine XML are transmitted than are people-to-people or people-machine bytes.

The restrictions mandated by the Character Model limit the use of the Web and Web-based technologies for a large base of users. While supporting the vision for the Web as a 'single, very large application', the limitations to other uses of the Web does not appear to support the Character Model's goal to 'facilitate the use of the Web by all people'.

Comment (received 2003-03-15) -- Satisfied

Comment (received 2002-06-06) -- Inconsistent/Redundant Requirements for W3C Spec Conformance

'[S] Every W3C specification MUST:

1. conform to the requirements applicable to specifications,

2. specify that implementations MUST conform to the requirements applicable to software, and

3. specify that content created according to that specification MUST conform to the requirements applicable to content.

[S] If an existing W3C specification does not conform to the requirements in this document, then the next version of that specification SHOULD be modified in order to conform.'

CONCERN: Stating that all specs 'MUST' conform, but that non-conforming specs 'SHOULD' be modified appears to be inconsistent.

RECOMMENDATION: The conformance model should only apply to future specs (including future versions of current specs), instead of specifying different conformance levels for existing and future versions.

Comment (received 2003-03-15) -- Satisfied

Comment (received 2002-06-06) -- W3C Spec Conformance

'[S] Every W3C specification MUST:

1. conform to the requirements applicable to specifications,

2. specify that implementations MUST conform to the requirements applicable to software, and

3. specify that content created according to that specification MUST conform to the requirements applicable to content.

[S] If an existing W3C specification does not conform to the requirements in this document, then the next version of that specification SHOULD be modified in order to conform.'

CONCERN: The CharMod's requirement that all specs and related implementations conform to the entire CharMod spec will force non-NFC based applications to perform round trip transcoding to/from NFC in order to use the Web, even in closed system scenarios (e.g. extranets). This will also affect intranet scenarios as corporate systems are forced to jump through hoops in order to satisfy text processors (such as XML parsers) that are required to reject non-NFC text. The costs certainly outweigh the benefits for closed systems. However, it is clear that a recommended conformance level would improve open system interoperability.

RECOMMENDATION: Replace conformance paragraph and included list with the following sentence:

'[S] Future W3C specifications (including future versions of existing specifications) MUST reference this specification as W3C recommended guidance for interoperable Web applications.'

Comment (received 2003-03-15) -- Satisfied

Comment (received 2002-06-06) -- Full Range of Unicode Code Points Not Allowed in XML

'[S] Specifications SHOULD allow the use of the full range of Unicode code points from U+0000 to U+10FFFF inclusive; code points above U+10FFFF MUST NOT be used.'

CONCERN: If this is truly a goal for text on the Web, users should understand why XML is unable to achieve this. As a high profile W3C spec, readers are likely to notice the inconsistent message. Does this mean that I18N believes that XML (1.1 or some later version) should support the characters 0x0-0x1F?

RECOMMENDATION: If XML 1.1 is unable to achieve this goal, the Character Model spec should either remove this requirement or explain the discrepancy.

Comment (received 2003-03-15) -- Satisfied

Comment (received 2002-06-06) -- Definition of 'Fully-Normalized'

'Text is fully-normalized if:

1. the text is in a Unicode encoding form, is include-normalized and none of the constructs comprising the text begin with a composing character or a character escape representing a composing character; or

2. the text is in a legacy encoding and, if it were transcoded to a Unicode encoding form by a normalizing transcoder, the resulting text would satisfy clause 1 above.'

CONCERN: Based on previous definitions, 'Unicode-normalized' may be a more precise term than 'Unicode encoding form' (if the implication is that full normalization requires include-normalization, which requires Unicode normalization as defined in 4.2.1.

RECOMMENDATION: Refer to text that is 'Unicode-normalized' (possibly linked to the definition [in 4.2.1]), instead of 'Unicode encoding form'.

Comment (received 2003-03-15) -- Satisfied

Comment (received 2002-06-06) -- Mandating NFC for All Web Content

'[C] In order to conform to this specification, all text content on the Web MUST be in include-normalized form and SHOULD be in fully-normalized form.

[S] Specifications of text-based formats and protocols MUST, as part of their syntax definition, require that the text be in normalized form.'

CONCERN: This restriction is currently applied to 'all text content on the Web' when only content intended for interoperability in open systems will necessarily benefit from it. Although the first requirement for content producers ([C]) could be interpreted to have no impact on intranet scenarios, the second requirement for specifications ([S]) will impact the tools that intranet scenarios have been depending on.

RECOMMENDATION: Replace above text with text similar to:

'[C] In order to conform to this specification, all text content on the Web intended for consumption by foreign systems MUST be in include-normalized form and SHOULD be in fully-normalized form.

[S] Specifications of text-based formats and protocols MUST, as part of their syntax definition, reference the above requirement.'

Comment (received 2003-03-15) -- Satisfied

Comment (received 2002-06-06) -- Text-Processors MUST Perform Normalization Checking

'[S] [I] A text-processing component that receives suspect text MUST NOT perform any normalization-sensitive operations unless it has first confirmed through inspection that the text is in normalized form, and MUST NOT normalize the suspect text. Private agreements MAY, however, be created within private systems which are not subject to these rules, but any externally observable results MUST be the same as if the rules had been obeyed.'

CONCERN: This requirement will force technologies such as XML parsers to be tied to the latest list of NFC disallowed diacritic characters in order to check normalization. Additionally, in some cases ('MAYBE' cases) NFC checks require text processors to scan backwards through a text stream in order to confirm normalization status. This will require major architectural changes for any processors designed to break a text stream into separate smaller windows for efficient processing, because no previously processed buffer can be thrown away until it is no longer needed to confirm the validity of any diacritic code points at the start of the next buffer. Text processors that expand character entities today at least have the ability to note the ‘&’ flag. It is also worth noting that optimizers of normalization checks will observe that all code points < 0x341 are always allowable. This would result in non-English based texts being disproportionately impacted by normalization checks. Finally, this requirement forces the redefinition of XML to allow for only NFC text.

RECOMMENDATION: Replace the above text with text similar to:

'[S] [I] Text-processing components MAY include an option to verify that suspect text is in normalized form. Text-processing components MUST NOT normalize the suspect text without specific direction.'

Comment (received 2003-03-15) -- Satisfied

Comment (received 2002-06-06) -- Content Producers and Proxies

'NOTE: As an optimization, it is perfectly acceptable for a system to define the producer to be the actual producer (e.g. a small device) together with a remote component (e.g. a server serving as a kind of proxy) to which normalization is delegated. In such a case, the communications channel between the device and proxy server is considered to be internal to the system, not part of the Web. Only data normalized by the proxy server is to be exposed to the Web at large, as shown in the illustration below:'

CONCERN: Although this note seems to allow closed systems to define their own boundaries, these systems will still be prevented from leveraging technologies based on CharMod, without first normalizing.

RECOMMENDATION: If the CharMod spec continues to mandate that all text processors must check normalization, this note should point out that such processors could not be used between devices and their proxies.

Comment (received 2003-03-15) -- Satisfied

Comment (received 2002-06-06) -- Web Repositories

'A similar case would be that of a Web repository receiving content from a user and noticing that the content is not properly normalized. If the user so requests, it would certainly be proper for the repository to normalize the content on behalf of the user, the repository becoming effectively part of the producer for the duration of that operation.'

CONCERN: As noted in issue, 'Content Producers and Proxies', this scenario is not possible if XML is to be used between the user and the Web repository. This scenario also seems to imply that users may very liberally interpret the boundaries of content production. Could one also claim that the transfer from one repository to another repository was also contained within the realm of content production? What if the second repository was the final destination for the content; does this mean the content never in fact needed to be normalized?

This scenario seems to encourage users to get around normalization requirements where impractical or inappropriate, yet leaves them with a confusing message and no legal tools to work with (since tools such as XML parsers will only accept normalized text anyway).

RECOMMENDATION: Delete this paragraph.

Comment (received 2003-03-15) -- Satisfied

Comment (received 2002-06-06) -- IRIs

'[S] W3C specifications that define protocol or format elements (e.g. HTTP headers, XML attributes, etc.) which are to be interpreted as URI references (or specific subsets of URI references, such as absolute URI references, URIs, etc.) SHOULD use Internationalized Resource Identifiers (IRI) [I-D IRI] (or an appropriate subset thereof).'

CONCERN: Although other W3C specifications support goals similar to those of the IRI proposal, we hesitate to endorse this section of the CharMod spec until the IRI draft has undergone further review.

RECOMMENDATION: Considering the W3C practice to keep the maturity level of a technical report within one level of any technical report on which it depends, the Character Model should not be considered for Recommendation until the IRI proposal has reached RFC status. Although the precedent has typically referred to W3C dependencies, it seems reasonable that any dependency on a spec outside the W3C should be judged by criteria at least as strong as those imposed on the W3C.

Comment (received 2002-05-30) -- XML Protocol LC#2 review question on implementation testing

In reviewing the Charmod LC#2, the XML Protocol WG has a request in relation to implementation conformance.

- Section 2 Conformance, 3rd paragraph, last sentence

- '[S] [I] [C] In order to conform to this document, specifications MUST NOT violate any requirements preceded by [S], software MUST NOT violate any requirements preceded by [I], and content MUST NOT violate any requirements preceded by [C].'

- The XML Protocol WG has produced a protocol specification, which makes various testable assertions, as well as a Collection of Tests each showing whether an assertion is implemented in the protocol processor. As such, the tests do not check for conformance to other specifications (e.g. XML 1.0). The XML Protocol WG asks the I18n WG to comment on whether the [I] implementation requirements of Charmod apply to the XMLP Test Collection.

Comment (received 2002-05-30) -- XMLP WG response to Charmod LC#2

Goals and Scope, second last paragraph

'All W3C specifications must conform to this document (see section 2 Conformance).'

This statement seems too comprehensive and probably needs qualification. What about existing W3C specifications or a W3C specification whose status is very close to LC (e.g. XMLP's)?

Suggest: 'All W3C specifications published after [a certain date or event such as this Charmod becoming a recommendation] must conform to this document (... etc).'

Comment (received 2002-05-30) -- XMLP WG response to Charmod LC#2

Responsibility for Normalization, 8th paragraph, 1st sentence

'[S] Specifications of text-based formats and protocols MUST, as part of their syntax definition, require that the text be in normalized form'

In our previous response to the Charmod WD review [1], we said

'The XML Protocol processor will defer to applications any normalization (early or late) that may be required for sending and/or receiving application payload(s).'

We received confirmation from i18n WG that that is acceptable:

    > XML protocol need not normalize application payloads

    > or check to insure that they are normalized

    Correct.

The XMLP WG would like to see clarification of this [S] requirement in relation to the payload's normalization or lack thereof.

Comment (received 2002-05-30) -- XMLP WG response to Charmod LC#2

Responsibility for Normalization, 8th paragraph, 1st sentence

'[S] Specifications of text-based formats and protocols MUST, as part of their syntax definition, require that the text be in normalized form'

In a previous review, we asked the question: 'May intermediaries re-send payloads (either normalized or un-normalized) untouched, even though they may change the protocol envelope?' From this requirement, do we take it that re-sent text is to be normalized when forwarded? However, please see comment [C062].

Comment (received 2002-05-30) -- XMLP WG response to Charmod LC#2

Responsibility for Normalization, 9th paragraph, 1st sentence

'[S] [I] A text-processing component that receives suspect text MUST NOT perform any normalization-sensitive operations unless it has first confirmed through inspection that the text is in normalized form,and MUST NOT normalize the suspect text'

It would be helpful to give the reason(s) for the prohibition against normalizing the suspect text.

Comment (received 2002-05-30) -- XMLP WG response to Charmod LC#2

Responsibility for Normalization, 9th paragraph, 1st sentence

'...and MUST NOT normalize the suspect text'

In a previous review, we asked the question: 'If un-normalized text is rejected and returned to sender, does it have to be normalized before transmission?' From this requirement, do we take it that rejected un-normalized text is to remain un-normalized when returned?

Comment (received 2002-05-30) -- XMLP WG response to Charmod LC#2

Responsibility for Normalization, last [S] requirement

'[S] Specifications that define a mechanism (for example an API or a defining language) for producing a document SHOULD require that the final output of this mechanism be normalized.'

Is a document text-based format? Is this requirement covered by the earlier one - '[S] Specifications of text-based formats and protocols MUST, as part of their syntax definition, require that the text be in normalized form.'? However, the earlier requirement is stronger ('MUST') than this one ('SHOULD').

Comment (received 2002-05-30) -- Comments on Character Model

[S] [I] Software that sorts or searches text for users MUST do so on the basis of appropriate collation units and ordering rules for the relevant language and/or application.

Hmm, there are cases where you just don't know the language, and even if you do, is this a requirement in the general case for things like XQuery? I think there are scenarios where it's reasonable to say a particular module shall order things by Unicode character number order and that's all there is to it. I think this should be rewritten to say that IF strings are being collated, they MUST be collated EITHER in the order appropriate to the language they're in, or if that's not possible by unicode character number.

Comment (received 2003-05-01) -- Satisfied

Comment (received 2002-05-30) -- Comments on Character Model

[S] When designing a new protocol, format or API, specifications SHOULD mandate a unique character encoding.

No. If the format is in XML and has likely usage scenarios which include creation by humans, this is a good enough reason to just go by the XML rules. For example, I habitually compose XML documents in ISO-8859-1, which suits my needs as a user of European languages. I see no reason whatsoever why a specification should invalidate either my habits or those of a Japanese author who wants to use some flavor of JIS.

OK, I guess this argument could fall under the exception clause of SHOULD, but I'd go so far as to add

[S] When designing an XML-based protocol which is apt to be authored by humans, specifications MUST NOT limit the use of character encodings beyond the rules provided by XML.

Comment (received 2004-01-24) -- Satisfied

Comment (received 2002-05-30) -- Comments on Character Model

The paragraph beginning

'[S] If the unique encoding approach is not chosen, specifications MUST designate at least one of the UTF-8 and UTF-16 encoding forms of Unicode as admissible... '

is fine, but if the format uses XML, then XML's rules cover this and in fact require that UTF-8 and -16 are both admissable; which takes priority over the language here and this should be noted.

Comment (received 2004-01-24) -- Satisfied

Comment (received 2002-05-30) -- Comments on Character Model

I am unable to develop an intelligent opinion as to the cost-benefit trade-off of Early Uniform Normalization and will remain unable to do so without hard information as to the cost. For example, if there was a C-language library available unencumbered by licensing issues which had a memory footprint smaller than say 10k and which ran at I/O speeds, you could reasonably argue that this is a cost effectively equal to zero. On the other hand, if E.U.N. requires a memory footprint of 256K or, worse, understanding and linking to the entire ICU library (blecch), the cost is likely to be unacceptable in a large class of applications.

There's a normalizer demo at Unicode.org referenced from Appendix D, which suggests that a few hundred lines of Java suffice, but I haven't had time to build to tables or to really think about whether they are being done in the best possible way.

I think my blockage on this point will be shared by the AC members who will eventually be asked to express on opinion on E.U.N. So I think somebody owes the world the gift of a few quantitative research results on these numbers.

Comment (received 2004-01-24) -- Satisfied

Comment (received 2002-05-30) -- Comments on Character Model

List item 4. 'Testing for bit-by-bit identity.'

<pedantry intensity='severe'>This may be the way you do it but I think it's the wrong way to talk about it. The point about Unicode is that it says is a character is an thingie identified by number which has a bunch of properties. At the end of the day, what you want people to do is to normalize the data in computer storage to a series of non-negative integers and when testing for equality, if you have two sequences of non-negative integers which are equal in length and pairwise equal in value, then you have equality. It is is conceivable in theory that the integer values are stored differently in two parts of the same program; and in practice, who knows what lurks inside a Perl 'scalar', and and what really happens when perl processes the '==' operator?. So I think that item 4 should say the strings are pairwise numerically equal by code point and leave it at that.</pedantry>

Comment (received 2003-05-01) -- Dissatisfied

Comment (received 2002-05-30) -- Comments on Character Model

[S] Since specifications in general need both a definition for their characters and the semantics associated with these characters, specifications SHOULD include a reference to the Unicode Standard, whether or not they include a reference to ISO/IEC 10646.

Change SHOULD to MUST. There's no excuse for doing a spec that talks about this stuff without referencing Unicode. Among other things, it's easy to buy the Unicode spec, and the spec is useful; neither of these things are true about the ISO version.

Comment (received 2003-05-01) -- Dissatisfied

Comment (received 2002-05-30) -- Comments on Character Model

'[S] Protocols, data formats and APIs MUST store, interchange or process text data in logical order' - shouldn't that be [S] [I] - software should do this too? In fact, arguably this should be [S] [I] [C].

Comment (received 2003-05-01) -- Satisfied

Comment (received 2002-05-30) -- Comments on Character Model

Mind you, it seems that the boundaries between [S] [I] and [C] are pretty fuzzy. If I were editing this thing, I'd just drop the whole notation and rely on getting the normative language right about what must be done, relying on the spec/data/software authors to follow the normative language that reasonably applies to them.

Comment (received 2002-05-30) -- Comments on Character Model

There is a problem in the phrase beginning 'also known as octets'... it seems backward; the reason we talk about 'octets' is that some bytes *used to be* non-8-bit; the fact that they're all 8-bit now means that the term 'octet' is probably a bit redundant. Perhaps the wording is correct but my brain obstinately insists on reading it backward so a little editorial cleanup is in order.

Comment (received 2003-05-01) -- Satisfied

Comment (received 2002-05-30) -- Comments on Character Model

The bullet point beginning '[S] Escape syntax SHOULD either require explicit end delimiters' is fine, but the charmod document itself doesn't actually comply per section 1.3's description of the U+hhhh notation. It might be elegant to cite the containing document as an example of non-compliance :)

Comment (received 2002-06-18) -- Re: Absence of explicit end delimiters makes Charmod non-compliant

Of course, thank you.

Comment (received 2002-05-30) -- Comments on Character Model

[L]ist item '2' uses the term 'legacy encoding', since it's defined shouldn't it be in bold?

Comment (received 2003-05-01) -- Satisfied

Comment (received 2002-05-30) -- Comments on Character Model

[In] 4.2.2 (second NOTE) [and] 4.2.3 (first NOTE) the phrase '(or the absence thereof)' baffles me no matter how many times I read it... please clarify a bit.

Comment (received 2003-05-01) -- Satisfied

Comment (received 2002-05-30) -- Comments on Character Model

'[C] In order to conform to this specification, all text content on the web MUST...' er, shouldn't this be [I] as well, since a lot of that content is produced by software? But see my comment to 3.1.3 above [since split into C073 and C074].

Comment (received 2004-01-24) -- Satisfied

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Goals and Scope, last paragraph

'Since other W3C specifications will be based on some of the provisions of this document, without repeating them, software developers implementing W3C specifications must conform to these provisions.'

Unclear what 'these provisions' (end of sentence) are since the first part of the sentence refers to only 'some of the provisions'. That is, should software developers implementing W3C specifications conform to some or all of these provisions?

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Background, 3rd paragraph, 2nd bullet

'covers the widest possible range,'

Unicode covers the widest possible range of what? Characters? Languages? Scripts? Writing notations?

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Background, 3rd paragraph, 3rd bullet

'provides a way of referencing characters independent of the encoding of a resource,'

Unclear what the 'resource' is. What is the relationship between the characters being referenced and the 'resource'?

Is this the intent? - 'provides a way to reference characters independent of the encoding of the characters,'

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Background, 4th paragraph, last sentence

'Unicode now serves as a common reference for W3C specifications and applications.'

Unclear what sort of 'reference' is meant.

Is this the intent? - 'Unicode now serves as a common reference character set for W3C specifications and applications'

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Background, 8th paragraph, last bullet

'Use of control codes for various purposes (e.g. bidirectionality control, symmetric swapping, etc.).'

It would be useful to have links to reference material that explain the issues.

E.g. 'Use of control codes for various purposes (e.g. bidirectionality control [Unicode Standard 13.2], symmetric swapping [Unicode Standard 13.3], etc.).'

Comment (received 2003-02-18) -- Satisfied

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Background, 9th paragraph, 1st sentence

'It should be noted that such properties also exist in legacy encodings (where legacy encoding is taken to mean any character encoding not based on Unicode), and in many cases have been inherited by Unicode in one way or another from such legacy encodings.'

Unclear what 'such properties' are. The previous sentence talks about 'aspects of Unicode' with no mention of 'properties'.

Is this the intent? - 'It should be noted that such aspects also exist in legacy encodings (where legacy encoding is taken to mean any character encoding not based on Unicode), and in many cases have been inherited by Unicode in one way or another from such legacy encodings.'

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Conformance, 1st NOTE, 1st sentence

'RFC 2119 makes it clear that requirements that use SHOULD are not optional ...'

Inconsistent usage of term 'requirements'. The first paragraph of this Conformance section makes a distinction between 'requirements' and 'recommendations'. It says that 'requirements are expressed using the key words 'MUST', ... etc.'. This NOTE talks of 'requirements that use SHOULD ...'

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Conformance, 3rd Paragraph, last sentence

'[S] [I] [C] In order to conform to this document, specifications MUST NOT violate any requirements preceded by [S], software MUST NOT violate any requirements preceded by [I], and content MUST NOT violate any requirements preceded by [C].'

How will conformance be enforced? Are the the conformance requirements in this document testable for violations?

Comment (received 2004-02-07) -- Satisfied.

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Conformance, 5th Paragraph, 1st sentence

'[S] If an existing W3C specification does not conform to the requirements in this document, then the next version of that specification SHOULD be modified in order to conform'

This lowered (to SHOULD) conformance requirement seems to contradict that in the preceding paragraph which states that '[S] Every W3C specification MUST conform to the requirements applicable to specifications, ...'

Comment (received 2004-02-07) -- Satisfied.

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Conformance, 5th Paragraph, 1st sentence

'[S] If an existing W3C specification does not conform to the requirements in this document, then the next version of that specification SHOULD be modified in order to conform'

Current wording says that in order to conform, the next version is to be modified, i.e. without stating nature of modification.

Is this the intent? - '[S] If an existing W3C specification does not conform to the requirements in this document, then the next version of that specification SHOULD be modified so that it then becomes conformant.'

Comment (received 2004-02-07) -- Satisfied.

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Conformance, 6th Paragraph, last sentence

'[I] Where this specification contains a procedural description, it MUST be understood as a way to specify the desired external behavior. Implementations MAY use other ways of achieving the same results, as long as observable behavior is not affected.'

'way' in the first sentence refers to 'a way to specify' whereas in the second sentence, the 'other ways' are 'ways of achieving' what is specified. Also current wording 'as long as observable behavior is not affected' is probably not the correct requirement.

Is this the intent? - '[I] Where this specification contains a procedural description, it MUST be understood as a way to specify the desired external behavior. Implementations MAY use different means of achieving the same results, as long as observable behavior is as described.'

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Introduction, 2nd EXAMPLE, 1st sentence

'Korean Hangul is a featural syllabary ...'

Would be helpful to define 'featural syllabary' and explain distinction between a 'syllabary' and 'featural syllabary'. The 1st and 2nd examples give the impression that the distinction is in arranging 'into square syllabic blocks'.

Comment (received 2003-02-18) -- Satisfied

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Introduction, 2nd EXAMPLE, 1st sentence

'... that combines symbols for individual sounds of the language ...'

Are these 'individual sounds of the language' phonemes or syllables?

Is this the intent? - '... that combines symbols for individual phonemes [or syllables] of the language ...'

Comment (received 2003-02-18) -- Satisfied

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Introduction, 3rd EXAMPLE, 1st sentence

'Indic scripts are abugidas.'

Would be helpful to indicate definition of 'abugidas' explicitly. E.g. 'Indic scripts are abugidas where each consonant letter carries an inherent vowel that is eliminated or replaced using semi-regular or irregular ways to combine consonants and vowels into clusters.'

Comment (received 2003-02-18) -- Satisfied

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Introduction, 4th EXAMPLE, 1st sentence

'Arabic script is an example of an abjad.'

Would be helpful to indicate definition of 'abjad' explicitly. E.g. 'Arabic script is an example of an abjad where short vowel sounds are typically not written at all.'

Comment (received 2003-02-18) -- Satisfied

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Introduction, 2nd last paragraph, 1st sentence

'The developers of W3C specifications, and the developers of software based on those specifications, are likely to be more familiar with usages they have experienced and less familiar with the wide variety of usages in an international context.'

In both instances of 'usages', it is unclear 'usages' of what are intended.

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Units of visual rendering, 3rd paragraph, 1st sentence

'[S] [I] Specifications and software MUST NOT assume a one-to-one mapping between character codes and units of displayed text.'

Inconsistency issue? This sentence speaks of mapping between 'character codes' whereas the third sentence of the first paragraph of 3.1.3 (There is not a one-to-one correspondence between characters and glyphs) speaks of mapping between 'characters', not 'character codes'. Also, in all the other 3.1.x sections, the [S][I] requirements are about non one-to-one correspondence between 'characters', not 'character codes'.

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Units of visual rendering, 5th paragraph, 3rd sentence

'The Unicode Standard [Unicode] requires that characters be stored and interchanged in logical order.'

Would be helpful to define 'logical order' or to provide link to reference material such as Unicode Standard, Section 2.2 where it is defined.

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Units of collation, 5th EXAMPLE, 1st sentence

'In Thai the sequence U+0E44 U+0E01 must be sorted as if it was written U+0E01 U+0E44.'

Would be helpful to show the actual glyphs for U+0E44 and U+0E01.

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Summary, 1st paragraph, 2nd and 3rd sentences

'In the context of the digital representations of text, a character can be defined informally as a small logical unit of text. Text is then defined as sequences of characters.'

'Character' and 'text' are defined circularly.

Comment (received 2003-02-18) -- Satisfied

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Character encoding identification, 9th paragraph, 2nd sentence

'[S] Specifications MAY define either UTF-8 or UTF-16 as a default encoding form (or both if they define suitable means of distinguishing them), but they MUST NOT use any other character encoding as a default.'

Since specifications 'MUST NOT use any other character encoding as a default' other than 'either UTF-8 or UTF-16' should the beginning of the sentence be '[S] Specifications MUST define either UTF-8 or UTF-16 as a default encoding form... ' ?

Comment (received 2003-02-18) -- Satisfied

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Character encoding identification, 12th paragraph, last sentence

'[I] On interfaces to other protocols, software SHOULD support conversion ...'

In the phrase 'to other protocols', which is the base protocol that the 'other protocols' are being distinguished from?

Is this the intent? - '[I] On interfaces to protocols, software SHOULD support conversion ...'

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Character encoding identification, 12th paragraph, last sentence

'[I] On interfaces to other protocols, software SHOULD support conversion between Unicode encoding forms as well as any other necessary conversions.'

Should it be 'between Unicode encoding forms' or 'to Unicode encoding forms' or 'both between and to Unicode encoding forms'?

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Character Escaping, 1st paragraph, 3rd sentence

'There is also a need, often satisfied by the same or similar mechanisms, to express characters not directly representable in the character encoding of instances of the language.'

Why 'instances of the language' and not just 'the language' ?

Comment (received 2003-02-18) -- Re: Your comments on the Character Model [C080-C086, C090-C100, C102-C105, C107-C111]

It is basically my own lack of understanding of the rationale, i.e. why don't languages have inherently associated character encodings whereas language instances do. It boils down to not knowing the difference between a language and a language instance.

I suggest that additional explanation or examples be given for the sentence in question.

Comment (received 2003-02-18) -- Your comments on the Character Model [C080-C086, C090-C100, C102-C105, C107-C111]

Comment (received 2003-06-23) -- Satisfied

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Character Escaping, 1st paragraph, last sentence

' ... a language's syntax, which is itself expressed as characters represented at the character encoding level.'

Why is a language's syntax expressed as characters 'represented at the character encoding level' and not just as characters in the sense of abstract symbols?

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Character Escaping, 4th [S] requirement, 2nd and last sentences

'Escape syntaxes where the end is determined by a character outside the set of characters admissible in the character escape itself SHOULD be avoided. ... Forms like SPREAD's &UABCD; [SPREAD] or XML's &#xhhhh;, where the character escape is explicitly terminated by a semicolon, are much better.'

The examples of good forms ('where the character escape is explicitly terminated by a semicolon') in the last sentence seem to exhibit the characteristics ('where the end is determined by a character outside the set of characters admissible in the character escape itself') of escape syntaxes that SHOULD be avoided.

Comment (received 2004-02-07) -- Satisfied.

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Character Escaping, 6th [S] requirement, 1st sentence

'[S] Escaped characters SHOULD be acceptable wherever unescaped characters are; ...'

What are 'unescaped characters'? Any character not expressed in the escaping mechanism? Seems to say that escaped characters SHOULD be acceptable wherever a character is acceptable (since a character normally is not expressed in the escaping mechanism).

Is this the intent? - '[S] Escaped characters SHOULD be acceptable wherever their unescaped forms are; ...'

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Character Escaping, 6th [S] requirement, last sentence

'In particular, escaped characters SHOULD be acceptable in identifiers and comments...'

What if the identifier syntax is defined to be of a set that does not include the character which is escaped?

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Fully-normalized text, 5th paragraph, last sentence

'Many languages will benefit from defining more boundaries...'

It would be helpful to give examples of the 'more boundaries'.

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

General Examples, 3rd paragraph, 1st sentence

'The string suc¸on (U+0073 U+0075 U+0063 U+0327 U+006F U+006E), where U+0327 is the COMBINING CEDILLA, encoded in a Unicode encoding form, is neither ...'

The string ... 'is not ...' because there is no 'nor' alternative.

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

General Examples, 5th paragraph, 1st sentence

'...the string suc¸on (U+0073 U+0075 U+0063 U+0327 U+006F U+006E) which is not include-normalized ('c¸' is replaceable by 'ç').'

Should it be this? - '...the string suc¸on (U+0073 U+0075 U+0063 U+0327 U+006F U+006E) which is not Unicode-normalized ('c¸' is replaceable by 'ç').'

Comment (received 2002-06-04) -- TAG comments on Character Model for the World Wide Web 1.0

We believe that specifications SHOULD NOT add rules for character encoding beyond what is provided in XML. They MUST NOT restrict character sets beyond what XML allows. In other words, the TAG disagrees with the current wording of the recommendation at the beginning of section 3.6 that says a specification should mandate a unique encoding. We believe a specification must not mandate a single encoding to the exclusion of UTF8/16.

For some machine-to-machine routing protocol, we accept that restricting the encoding to UTF8/16 would be acceptable. But for specifications designed for editing by humans (such as MathML), we believe that this restriction should not be imposed.

Comment (received 2004-01-19) -- Satisfied. TBray: Might be worthwhile to point to RFC3470.

Comment (received 2002-05-27) -- Comments on charmod from Chris

Numbers for each conformance requirements clause would greatly aid referencing them.

Comment (received 2004-01-19) -- TBray: I hope there will be stable, addressable normative text. I want URIs to identify conformance requirements.

Comment (received 2002-05-27) -- Comments on charmod from Chris

Please at least link to an accessible representation of 'foreign' characters rather than merely providing raster images of them. The text of this specification does not conform to itself, since it iuses bytes (pixels) to represent Unicode characters. Its also less than optimal wrt WAI guidelines. Apendix B is a lot better. But if the concern is to ensure correct rendering on legacy browsers, at least provide a link to the actual unicode sample, as characters and markup.

Comment (received 2004-01-19) -- Satisfied.

Comment (received 2004-01-26) -- Chris Lilley: In general you went further than I expected, but please provide text that can be linked to for remaining text in bitmaps.

Comment (received 2002-05-27) -- Comments on charmod from Chris

Much of this document is a statement of existing good design practice. Many existing W3C specifications implement large parts of it. This is good. Care should be taken with MUSTs which make W3C Recs non-conforming. For example, XML 1.0 and 1.1 are non conforming.

Comment (received 2004-01-19) -- Satisfied.

Comment (received 2002-05-27) -- Comments on charmod from Chris

These sections (collectively 'character 101'):

3 Characters.

5 Compatibility and Formatting Characters.

6 String Identity Matching.

7 String Indexing

9 Referencing the Unicode Standard and ISO/IEC 10646

taken as a group, are great, in general, and should be collected together with appropritate intrioductory and reference material as a separate document and move to Proposed Rec once it exits Last Call. There is already a large body of existing implementation of these concepts in W3C Recs.

Section 4 Early Uniform Normalization is very important, but affects a lot of specifications and needs, I believe, a CR period as does section 8 Character Encoding in URI References (perhaps - there is some exprerience for the latter). Thus, I suggest splitting the document so that sections 4 and 8 can move to CR without delaying sections 3,5,6,7,9 which are needed as a Rec ASAP!

Comment (received 2004-01-19) -- TAG is [very] happy with the WG's disposition re: comment C119. RF: Note that it hasn't been split yet. TB, TBL, RF: Leave open until this is actually done.

Comment (received 2002-05-27) -- Comments on charmod from Chris

The portions about collation and sorting (for example 3.1.5 Units of collation) are sparse, vague, and anecdotal which contrasts strangely with the MUSTs; this section should be removed and returned for further work to produce a separate architectural specification on collation that has crisp, well thought out conformance criteria. The maturity of the collation parts does not match that of the 'character 101', normalization and URI reference parts.

Comment (received 2004-01-26) -- Satisfied

Comment (received 2002-05-27) -- Comments on charmod from Chris

'[S ] Specifications MUST be defined in terms of Unicode characters, not bytes or glyphs.'

Yes in general, but not exclusively. The MUST should be retainmed, but the scope of the statement tightened up. Specifications *when they talk about characters* MUST be defined in terms of Unicode characters, not bytes or glyphs. Specifications are allowed to talk about bytes if that is what they are representing (eg, PNG which defines a byte stream) or glyphs (for example the SVG glyph element, which is very clearly defined as a glyph because that is its purpose in life. Although its 'unicode' attribute is, indeed, defined in terms of (a string of) Unicode characters.

Comment (received 2004-01-26) -- Satisfied

Comment (received 2002-05-27) -- Comments on charmod from Chris

'[S] Specifications SHOULD allow the use of the full range of Unicode code points from U+0000 to U+10FFFF inclusive; code points above U+10FFFF MUST NOT be used.'

So XML is not conforming, since it disallows for example U+0000 ?

Comment (received 2004-01-26) -- Satisfied

Comment (received 2002-05-27) -- Comments on charmod from Chris

'[S] Specifications SHOULD NOT provide mechanisms for agreement on private use code points between parties and MUST NOT require the use of such mechanisms. '

svg glyph with a unicode='&#xFE00;' is that a private agreement (aand hence in contravention)? If you disallow it, though, you break the following

'[S] [I] Specifications and implementations SHOULD be designed in such a way as to not disallow the use of private use code points by private arrangement.'

and in practice, duisallowing it would merely encourage mapping glyphs to the ascii code range wheras they should use the correct unicode code point or, if none, the PUA. Related point, avoid using character mechanisms for things that are not characters ('pi' fonts). Use small inline graphics instead.

Comment (received 2004-01-26) -- Satisfied

Comment (received 2002-05-27) -- Comments on charmod from Chris

'[S] Escaped characters SHOULD be acceptable wherever unescaped characters are; this does not preclude that a syntax-significant character, when escaped, loses its significance in the syntax. In particular, escaped characters SHOULD be acceptable in identifiers and comments.'

XML should allow NCRs everywhere, for example inside element and attribute names?

Comment (received 2004-01-26) -- Satisfied

Comment (received 2002-05-27) -- Comments on charmod from Chris

'[S] W3C specifications MUST define when the conversion from IRI references to URI references (or subsets thereof) takes place, in accordance with Internationalized Resource Identifiers (IRI) [I-D IRI].'

Why not go further and say that the IRI form is used in the document instance and the hexified URI form when it goes over the wire? It would be bad if different XML namespaces defined different processing here.

Comment (received 2002-05-27) -- Comments on charmod from Chris

'Conformance to Unicode implies conformance to ISO/IEC 10646, see [Unicode 3.0] Appendix C.

[S] Since specifications in general need both a definition for their characters and the semantics associated with these characters, specifications SHOULD include a reference to the Unicode Standard, whether or not they include a reference to ISO/IEC 10646. By providing a reference to The Unicode Standard implementers can benefit from the wealth of information provided in the standard and on the Unicode Consortium Web site.'

That is a bit weak. Say explicitly that a reference to 10646 without a reference to Unicode implies no character semantics, no bidi processing no character case information etc etc. Also, since one is a strict superset of the other, provide a rationale why a specification should ever provide a reference to 10646 since a reference to Unicode exactly covers the same CCS?

Comment (received 2004-02-07) -- Satisfied.

Comment (received 2002-07-03) -- 'private agreements don't scale on the web'

Well, they can scale. The real problem is that they don't interoperate.

Comment (received 2002-07-03) -- Readability of tables

The second table in this section is much more readable than the first, by using '-' instead of 'N', since '-' is visually different from 'Y'. Recommend using same for first table.

Comment (received 2002-07-03) -- Spelling

Incorrect spellings: occuring, reponsibilities English spellings (US is mandated for W3C specs): recognise, standardisation, normalised, organisation, behaviour

Comment (received 2002-07-03) -- Give example of transcoding

Comment (received 2002-07-05) -- Re: Give example of transcoding

It would be useful if 3.3 gave an example of where transcoding is used, since this is a frequently misunderstood point with regards to XML and HTML. People (and some UAs) think that the encoding also specifies the repertoire/CCS.

Something along the lines of:

'For example, in XML and HTML, documents are always in Unicode, but they may be delivered to a user agent in an encoding for another coded character set (indicated by the encoding attribute in XML, and the HTTP content-type header in HTML). The user agent then transcodes the characters of the incoming document stream into Unicode code points. For example, a document delivered with encoding iso-8859-2 may contain the string 'ő&#x0151;' where the first character (LATIN SMALL LETTER O WITH DOUBLE ACUTE) is at code point 0xf5 in iso-8859-2. This will be transcoded so that there will be two identical characters at (Unicode) code point 0x0151 in the document as processed by the user agent.'

Comment (received 2003-05-01) -- Re: Your comments on the Character Model [C130, C131]

Comment (received 2003-05-09) -- RE: Your comments on the Character Model [C130, C131]

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Character encoding identification, 9th paragraph, last sentence

'[S] Specifications MUST NOT propose the use of heuristics to determine the encoding of data.'

It would be helpful to either give examples of the undesirable 'heuristics' or the reasons for banning 'use of heuristics'.

Comment (received 2004-02-07) -- Satisfied.

Comment (received 2002-05-31) -- WSArch WG review of Charmod LC #2

Character encoding identification, 9th paragraph, last sentence

'[S] Specifications MUST NOT propose the use of heuristics to determine the encoding of data.'

Would the absence of a BOM in UTF-8 encoding be considered use of heuristics for identifying encoding?

Comment (received 2004-02-07) -- Satisfied.

Comment (received 2002-06-28) -- XSL WG Comments on Chairacter Model WD

'[S] Every W3C specification MUST conform to the requirements applicable to specifications, specify that implementations MUST conform to the requirements applicable to software, and specify that content created according to that specification MUST conform to the requirements applicable to content. [S] If an existing W3C specification does not conform to the requirements in this document, then the next version of that specification SHOULD be modified in order to conform.'

Why is it that every spec MUST but subsequent specs only SHOULD meet this requirement? Is the intent here to permit current non-conforming specs to maintain backwards compatibiltiy in future releases? This is an XSL requirement.

Comment (received 2002-06-28) -- XSL WG Comments on Chairacter Model WD

'[S] Protocols, data formats and APIs MUST store, interchange or process text data in logical order.'

This appears to be higher level of conformance than necessary for interoperability. Why do internals need to be dictated as long as there is exernal conformance?

Comment (received 2002-06-28) -- XSL WG Comments on Chairacter Model WD

'Note that, where searching or sorting is done dynamically, particularly in a multilingual environment, the 'relevant language' should be determined to be that of the current user, and may thus differ from user to user.'

Suggest the 'should' become 'SHOULD' to bring it to the level of a recommendation.

Comment (received 2002-06-28) -- XSL WG Comments on Chairacter Model WD

'[S] When specifications use the term 'character ' it MUST be clear which of the possible meanings they intend. [S] Specifications SHOULD avoid the use of the term 'character' if a more specific term is available.'

In 3.1.7 it is stated that specifications must make it clear 'which of the possible meanings' of the word 'character' is intended. But it's not explicit what the 'possible meanings' are. Where do we read that one of the possible meanings is 'a Unicode code point'?

There should be examples of this as the spec itself is an offender here. While it could be explicitly stated in line with their own requirement, We read it that the possible meanings were 'Units of aural rendering', 'Units of visual rendering', 'Units of input', 'Units of collation' and 'Units of storage'.

It is very difficult to conform to the second requirement in 3.1.7, as is illustrated by the fact that the Character Model document itself fails to conform to it: see the immediately following section heading.

Comment (received 2002-06-28) -- XSL WG Comments on Chairacter Model WD

'A CES, together with the CCSes it is used with, is identified by an IANA charset identifier. Given a sequence of bytes representing text and a charset identifier, one can in principle unambiguously recover the sequence of characters of the text.'

There are other legal identifiers other than IANA. We should not be restricted to these.

Comment (received 2002-06-28) -- XSL WG Comments on Chairacter Model WD

'Certain guidelines apply to content developers, as well as to software that generates content: ... [I] [C] Choose an encoding for the document that maximizes the opportunity to directly represent characters and minimizes the need to represent characters by markup means such as character escapes. In general, if the first encoding choice is not satisfactory, Unicode is the next best choice, for its large character repertoire and its wide base of support.'

The last bullet immediately before the section heading of section 4 seems strange. Grammatically, it is hard to parse, and is in the imperative mood which is not used elsewhere. Semantically, the statement that 'If the first encoding choice is not satisfactory, Unicode is the next best choice' seems very odd. Surely (a) Unicode is always the first choice, and (b) Unicode is not an encoding? Also the term 'satisfactory' is far too vague for a specification. We also question the appropriateness of these 'guidelines' in the spec body. They seem more appropriate for a note or appendix.

Comment (received 2002-06-28) -- XSL WG Comments on Chairacter Model WD

We have a concern about the guideline preventing new character escaping syntax.

Comment (received 2002-06-28) -- XSL WG Comments on Chairacter Model WD

'[C] In order to conform to this specification, all text content on the Web MUST be in include-normalized form and SHOULD be in fully-normalized form.'

The impacts of this requirement on XSLT and other infoset 'pipeline' type processes are still unclear to us.

For instance, XSLT and many other specifications are designed around an infoset 'pipeline' so that various processes can transform, augment, or otherwise manipulate content. A final step in a pipeline often involves serialization of the infoset. It appears to us that serialization of an infoset according to the Character Model may result in either significant manipulation of the data within that infoset (resulting in a loss of data fidelity) or failure to serialize. In either case, an upstream process such as an XSLT transformation cannot trust that its output can be successfully processed further on in the pipeline, without adopting normalization rules at the infoset level as well. The practical inability to limit normalization to text content on the Web concerns us. The implications of this are not adequately discussed in the Character Model spec.

Without a clear idea of the implications of the Character Model upon the tendency to rely on the XML Information Set instead of upon text for composing processes within a system, we cannot agree to the mandate for normalization.

Comment (received 2002-06-28) -- XSL WG Comments on Chairacter Model WD

'[I] A text-processing component which modifies text and performs normalization-sensitive operations MUST behave as if normalization took place after each modification, so that any subsequent normalization-sensitive operations always behave as if they were dealing with normalized text.'

The fourth requirement in section 4.4 is labelled [I], but XPath implementations have to do what the XPath specification says, so this is actually an [S] requirement. The implication of this requirement is that functions such as concat() should perform normalization. This is both expensive and backwards-incompatible, we will have to examine whether it is something where the benefits exceed the costs. This also seems to violate the self-imposed limitation to only require conformance to observable behaviors. How XPaths are handled within an XSLT Processor should not be the subject of this spec as long as the results are conformant.

Comment (received 2002-06-28) -- XSL WG Comments on Chairacter Model WD

'[S] Specifications of text-based languages and protocols SHOULD define precisely the construct boundaries necessary to obtain a complete definition of full-normalization . These definitions MUST include at least the boundaries between markup and character data as well as entity boundaries (if the language has any include mechanism) and SHOULD include any other boundary that may create denormalization when instances of the language are processed.'

The requirement (still in 4.4) about defining construct boundaries is very unclear when applied to a language that performs dynamic manipulation of strings.

Comment (received 2002-09-10) -- Character Model Comments Clarifications

Comment (received 2002-09-24) -- RE: Character Model Comments Clarifications

Comment (received 2002-06-28) -- XSL WG Comments on Chairacter Model WD

'[S] Specifications MUST document any security issues related to normalization.'

The requirement 'Specifications MUST document any security issues related to normalization.' is untestable on its face and should be detailed.

Comment (received 2002-06-28) -- XSL WG Comments on Chairacter Model WD

'[S] [I] Forms of string matching other than identity matching SHOULD be performed as if the following steps were followed: Steps 1 to 3 for string identity matching. Matching the strings in a way that is appropriate to the application.'

It is unclear whether the procedure for string identity matching in section 6 establishes a requirement for expansion of %HH escapes in URIs, especially when comparing namespace URIs, where such expansion has not traditionally been performed. Section 8 should give guidance on this.

Comment (received 2002-07-12) -- The term 'UCS' vs. the term 'Unicode'

Sec. 1.1 says, inter alia, 'In this document, Unicode is used as a synonym for the Universal Character Set.' I believe the term 'UCS' would be better, because it is clearer and less subject to misconstruction.

It is clearer because the term 'Unicode' may reasonably be used to denote (a) the consortium of that name, (b) the Univeral Character Set defined by ISO/IEC 10646 and by the Unicode Standard, (c) the UCS taken together with the additional rules defined by the Unicode Standard, which Unicode does NOT share with ISO/IEC 10646, and (d) the Unicode Standard itself. Despite the explicit statement that in the character model spec the term 'Unicode' is used in sense (b), I suspect the common use, elsewhere, of the term in senses (a), (d), and especially (c), will necessarily color readers' perceptions of the meaning of the text.

The term 'UCS' is also less likely to convey to casual readers that it is really the Unicode Standard, not ISO/IEC 10646, which counts. It is true, as you have pointed out from time to time, that the Unicode Consortium and the responsible ISO/IEC technical committee have worked well for some time now in keeping the two standards aligned. I applaud that fact and the role some of you have individually played in making it happen. But I remember too the years in which the two organizations threatened to burden the world with two different and incompatible universal character sets, and the roles some of you played then, and I am unwilling that any W3C specification should risk conveying the idea that if the two standards should diverge, the Web or the W3C would naturally side with one or the other party.

It would not be appropriate to use the term 'ISO/IEC 10646' (or just '10646' for short) to refer to the UCS. It is also not appropriate to use the term 'Unicode'.

Please reconsider and use the neutral and unambiguous term 'UCS'.

Comment (received 2003-02-12) -- Dissatisfied

Comment (received 2002-07-12) -- ANSI X3.4 is missing

The spec refers several times to ASCII. In the context of a specification defining a character model, I assume that this term is used in its proper and narrow sense to denote the coded character set defined by American national standard ANSI X3.4. That American national standard should be included among the non-normative references.

Comment (received 2003-02-12) -- Dissatisfied

Comment (received 2002-07-12) -- Spanish 'ch' is not a letter sequence

Section 3.1.5 says 'EXAMPLE: In traditional Spanish sorting, the letter sequences 'ch' and 'll' are treated as atomic collation units. Although Spanish sorting, and to some extent Spanish everyday use, treat 'ch' as a single unit, current digital encodings treat it as two letters, and keyboards do the same (the user types 'c', then 'h').'

This is not what I learned in grade school. Sra. Robles was quite clear, and rather strict about it (and so of course I am sure that she is right and your informants must be wrong).

I believe the paragraph would be more accurate and clearer if it read 'EXAMPLE: In traditional Spanish sorting, the character sequences 'ch' and 'll' are treated as single letters and as atomic collation units. Although Spanish sorting, and to some extent Spanish everyday use, treat 'ch' as a single unit, current digital encodings treat it as two characters, and keyboards do the same (the user types 'c', then 'h').'

I don't know of any digital encoding whose specification provides any definition of 'letter', and thus I find it surprising and confusing to read that most such encodings treat 'ch' as two letters: I don't believe that any character set specifications or encodings can meaningfully be said to treat ANYTHING as ANY number of 'letters', since 'letter' is a concept foreign to their universe of discourse.

Comment (received 2003-02-12) -- Satisfied

Comment (received 2002-07-12) -- Counting languages

The paragraph which reads 'EXAMPLE: In most languages, the letter 'æ' is sorted as two consecutive collation units: 'a' and 'e'' [I wonder if that aesc will come through this HTML form ...] might be improved if the 'most languages' were changed to 'some languages'. In Old English, Old Norse, Norwegian, Danish, and Swedish, I believe that aesc is treated as a single collation unit; I don't know of any languages in which aesc occurs in native words which sorts it in the way you describe. Are you counting all the other languages in Western Europe as languages in which aesc is sorted as 'ae'? (Note that it does not matter whether the languages which sort aesc as 'ae' outnumber the others or not: the point to be made is that they exist. The term 'most' brings in an element of quantitative comparison which is distracting -- do Flemish and Dutch count as one language, or two, in this tally? -- and unnecessary. Hence my suggestion to eliminate 'most'.)

Comment (received 2003-02-12) -- Satisfied

Comment (received 2002-07-12) -- For 'o' read '&ouml;'

In 'In German certain applications treat the letter 'o' as if it were the sequence 'oe'' the quoted letter should be an o with an umlaut, surely? (Or else what applications are you thinking of?)

Comment (received 2003-02-12) -- Satisfied

Comment (received 2002-07-12) -- User control of collation, foreign matter

Thank you for specifying that '[S] [I] Software that allows users to sort or search text SHOULD allow the user to select alternative rules for collation units and ordering.' I am glad to see that you are not requiring that software use the collation rules of the user's language whether the user wants it to or not.

The requirement '[S] [I] When sorting and searching in the context of a particular language, it MUST be possible to deal gracefully with strings being compared that contain Unicode characters not normally associated with that language' appears to be unenforceably vague: the word 'gracefully' seems impossible to define sharply enough to allow objective determinations of whether a given spec or implementation conforms with this requirement or not. I would be loath to lose the word 'gracefully' from the text, but I don't believe it belongs in a conformance requirement.

Comment (received 2002-07-12) -- Deriving specs from specs, building specs on specs

In the sentence 'NOTE: All specifications that derive from the XML 1.0 specification [XML 1.0] automatically inherit this Reference Processing Model', I think the phrase 'derive from' is not quite right. One would not say that the XHTML spec 'derives from' the XML spec: it references it, uses it, builds a language on it, builds on it, cites it normatively (does it?), but it is specs like XML 1.1 which 'derive from' (stand in the relation of genealogical descent to) the XML 1.0 spec. Perhaps an acceptable wording would be 'all specifications which define applications of the XML 1.0 specification ...'?

Comment (received 2003-02-12) -- Satisfied

Comment (received 2002-07-12) -- Always reliable identification is a chimaera

The requirement '[S] Specifications MUST either specify a unique encoding, or provide character encoding identification mechanisms such that the encoding of text can always be reliably identified' is, I think, too strong. I do not believe that any identification mechanism can ALWAYS guarantee the correct identification of an encoding; if I am right, this requirement guarantees that no specification ever written has ever conformed, and no specification will ever conform, to the character model specification. Malicious users, incompetent users, ignorance or indifference on the part of those responsible for servers, and transcoders which understandably do not touch the internal labels on the data they transcode, can combine to defeat any labeling or encoding-identification scheme ever devised. Even the W3C server has been known, from time to time, to serve documents with the wrong character-encoding identification.

Please weaken this requirement so that it is achievable, or else XML 1.1 and every other spec now under development by the W3C will be blocked by this unrealistic counsel of perfection. The identification mechanisms of XML 1.0 are pretty good, if I say so myself. But they do not come close to meeting the requirement stated here. I think you've set the bar too high.

Comment (received 2003-02-12) -- Satisfied

Comment (received 2002-07-12) -- Heuristics

The spec says '[S] Specifications MUST NOT propose the use of heuristics to determine the encoding of data.' Is it your intent to outlaw the heuristics given by the XML 1.0 spec? If not, I believe the wording of this requirement is incorrect, and should be changed to something specifying that heuristics should be used only in the absence of usable labels. If it is your intent to disallow the heuristics defined in XML 1.0, I disagree, and believe the MUST NOT should be changed to something weaker, preferably a generic health warning.

Comment (received 2002-07-13) -- Re: Heuristics

Comment (received 2002-07-12) -- For 'insure' read 'ensure'

In the phrase 'it is insured that normalization is uniform' I believe 'insured' should be spelled 'ensured'. (Usage among native speakers of English is not completely uniform, but some prefer to reserve the verb 'insure' for financial transactions involving payments of fees in exchange for financial protection against risks, and to use 'ensure' when the meaning is 'to make certain' or 'to entail'.

Comment (received 2003-02-12) -- Satisfied

Comment (received 2002-07-12) -- Recommend Unicode for character escapes?

Section 3.7 - Character escapes - contains the paragraph:

[S] Whenever specifications define character escapes that allow the representation of characters using a number the number SHOULD be in hexadecimal notation.

I rather expected it to continue:

... and SHOULD represent the Unicode code point of the character.

I wondered if the Reference Preocessing Model (3.5) applied an implied mandate to use Unicode, but it seems to permit other encodings to be used, and therefore does not seem to supply the 'default' for 3.7.

Have I missed something?

Comment (received 2002-06-18) -- QA Review for Charmod

[...] I would like to know how you plan to enforce the use of charmod in other specifications by process, pubrules, charters? We are faced to the same question in QA WG.

Comment (received 2002-06-18) -- QA Review for Charmod

I found interesting the way you have declared the rules of Conformance for your specifications. I would like to know if there's a plan for a Test Suite or at least Examples and Techniques to demonstrate your technologies.

For example in the first statement (Testable assertion?), I had difficulty to define a binary test case, is it possible to have testable examples for each rule in a separate document. It will help people understand the statement you have defined.

Comment (received 2002-06-18) -- QA Review for Charmod

'[S] [I] Specifications and software MUST NOT assume that there is a one-to-one correspondence between characters and the sounds of a language.'

Here I clearly understand that there's not *necessary* correspondance but it could be sometimes. So it means for me that in the set of all possible values, some will be false, and so invalidate the one to one relationship.

I don't know if it's better, but you will tell me:

'[S] [I] Specifications and software MUST allow the correspondence between one phoneme and mutliple characters when necessary.'

Because for me it seems more testable and comprehensive for a developper or a specification editor.

As a general rule, even if it seems valid, avoid the use of MUST NOT when it's possible to define a MUST.

Comment (received 2003-05-05) -- Satisfied

Comment (received 2002-06-18) -- QA Review for Charmod

'[S] Protocols, data formats and APIs MUST store, interchange or process text data in logical order.'

Why only S (Specifications) ? Do you mean:

'[S] (Specifications of?) Protocols, data formats and APIs MUST store, interchange or process text data in logical order.'

or

'[I] Protocols, data formats and APIs MUST store, interchange or process text data in logical order.'

Comment (received 2002-06-18) -- QA Review for Charmod

'[S] When specifications use the term 'character' it MUST be clear which of the possible meanings they intend.'

Ambiguous definition, can you clarify? What's the meaning of 'be clear' in this context. The answer will depend on the people.

Do you mean?

'[S] When specifications use the term 'character', the specifications MUST define the possible meanings they intend.'

Comment (received 2003-05-05) -- Satisfied

Comment (received 2002-06-18) -- QA Review for Charmod

When you have multiple rules, please make it a list to be more readable.

Comment (received 2002-09-23) -- FW: Query about CharMod last call comment

Comment (received 2003-05-05) -- Satisfied

Comment (received 2002-07-12) -- Reliability of character encoding identification

Section 3.6 specifies that '[S] Specifications MUST either specify a unique encoding, or provide character encoding identification mechanisms such that the encoding of text can always be reliably identified.'

The XML Schema WG believes that this requirement, as formulated, is not met by any existing specifications and is unlikely ever to be met by any. Document producers, software implementors, and server administrators, working alone or in concert, have innumerable opportunities to render character-set labels false out of malice, ignorance, or indifference; if character-set labels are false, the encoding of the text can only rarely be reliably identified.

The word 'always' seems to suggest that encoding identification mechanisms must function even in the case of hostile users or misconfigured servers; that's not possible. Either the i18n WG should lower its expectations or it should express its expectations more clearly.

We believe a more correct standard would be to require that specifications provide mechanisms to ensure that it is POSSIBLE to get things right, or to ensure that with correct operation / under normal circumstances character encodings are reliably and correctly identified.

N.B. This comment is substantially similar to C157 and to comment 3.13 of our comments on the previous last-call draft.

Comment (received 2002-07-12) -- Heuristics considered useful

The rule '[S] Specifications MUST NOT propose the use of heuristics to determine the encoding of data' appears to mean that the XML 1.0 Recommendation does not conform to the character model spec, since in its Appendix F it proposes the use of heuristics for recognizing the character encoding being used well enough to bootstrap and read the XML declaration and any encoding declaration included within it. If this is the intent, we believe this rule should be scaled back to something more like what was in the first last-call spec: Specifications MUST NOT require or encourage the use of unreliable heuristics.' If this is not the intent, we believe the rule needs to be rewritten to be clearer.

Either way, it would be useful to define 'heuristics'; without a definition, it's hard to know exactly what constraint is intended to be expressed by this rule.

N.B. this comment is similar but not identical to C158.

Comment (received 2002-07-12) -- Converting to RFC-2396-style URIs

We note with some alarm that section 8 of the specification no longer contains an account of any algorithm for converting internationalized resource identifiers into uniform resource identifiers as defined by RFC 2396 and widely implemented. We believe that the algorithm which was presented in the first last-call draft should be restored, in order to allow other W3C specifications to refer to it as needed. (Note that the XML Schema 1.0 Recommendation does refer readers to section 8 of this document for information about conversion problems -- information which this version of the document no longer provides.) We do not believe the reference to section 2.2.5 of RFC 2718 serves as an adequate substitute: RFC 2718 is informational, not normative, and its account of the algorithm presupposes more familiarity with the family of URI specifications than it is reasonable to assume, even of writers of W3C specifications.

Comment (received 2002-07-12) -- Early uniform normalization

We note that this version of the character model specification does a much better job of motivating the choice of early uniform normalization than did the previous version, and we congratulate the i18n WG on this important improvement to the coherence and plausibility of the character model.

That said, we continue to have reservations about the Draconian rules enunciated in connection with early uniform normalization. We note that a number of working groups have expressed strong objections, and we urge the i18n WG to continue working toward consensus, and not to attempt to move forward until there is something much more like consensus within the community than we believe there is at present.

Comment (received 2002-07-12) -- The spelling of ISO's name

For 'International Organisation for Standardisation' read 'International Organization for Standardization' (or so says their Web site, http://www.iso.ch, and their documents).

Comment (received 2003-05-02) -- Satisfied

Comment (received 2002-05-27) -- Comments on charmod from Chris

Having defined the terms 'logical selection mode' and 'visual selection mode', please use them rather than the highly ambiguous 'discontiguous selections' and 'contiguous selections' [...]

Comment (received 2004-02-07) -- Satisfied

Comment (received 2002-05-27) -- Comments on charmod from Chris

'Logical selection looks like this:'

There should be a requirement after that [...]

[S][I] Specifications of protocols and APIs that involve selection of ranges MUST provide for text selection in logical selection mode.

Comment (received 2004-02-07) -- Dissatisfied: "The TAG resolved to press for further clarification. Text selection is indeed a means of processing, but no-one could infer "support text selection in logical mode" from "do all processing in logical mode", especially if the only mentions of text selection stress visual mode text selection."

Comment (received 2002-05-27) -- Comments on charmod from Chris

Also, should there not be something about copying that selection and pasting it somewhere else, that what you get is the logical selection?

Comment (received 2004-02-07) -- Dissatisfied: 'The TAG resolved to clarify the request; make it "making a logical selection, copying and pasting preserves the characters in logical order"'

Comment (received 2002-05-27) -- Comments on charmod from Chris

Similarly in the next part, I suggest rewording to remove the ambiguous phrase:

[S] Specifications of protocols and APIs that involve selection of ranges SHOULD provide for text selection in logical selection mode, at least to the extent necessary to support implementation of visual selection on screen on top of those protocols and APIs.

Comment (received 2004-02-07) -- Satisfied.

Comment (received 2002-05-27) -- Comments on charmod from Chris

Its not clear that this is such a strong requirement and it complicates processing, especially on handheld devices. Perhaps weaken to MAY? And say what happens when this funky visual selection getc copied and pasted - do you get a set of separate logical selections (if so how delimited)? A single visually ordered selection (yuk)? Something else?

Otherwise, the weaker requirement for contiguous visual selection is likely to merely encourage the use of visual storage or the disposal of logical storage once the visual result has been generated. Which would lead to text copied from visualy contiguous (logically discontiguous) selections being stored in visual order. Which is to be avoided.

Comment (received 2004-02-07) -- Dissatisfied: "The TAG resolved to pushback on the grounds of misunderstanding. I18N WG seem to have missed the point we were making, or we were not clear enough.

We were concerned about implementations *only* providing visual presentation and visual ordering, while the I18N WG seemed to take the comments as requiring only logical selection. This is not the case.

It seems that we all agree that implementations should provide logical selection; and charmod asks for visual selection as well, assuming a logical backing store.

Comment (received 2002-05-27) -- Comments on charmod from Chris

It would be a good idea to tie into WAI concerns by noting that accessibility tools, which access the DOM, should be able to get at logically ordered text and to know which parts are selected.

Comment (received 2004-02-07) -- Satisfied.

Comment (received 2002-07-12) -- fixed-length escapes

In contemplating the rule '[S] Escape syntax SHOULD either require explicit end delimiters or mandate a fixed number of characters in each character escape' I am uncertain whether you intend to outlaw the kinds of escapes defined by section 6.3 of ISO 2022 or not. ISO 2022 defines some fixed-length and some variable-length escape sequences, in which certain classes of characters are defined as final characters. These final characters might be viewed as explicit end delimiters, but they are not solely delimiters. They are part of the escape sequence and cannot be disregarded in establishing the meaning of the escape sequence.

I don't think I have a strong preference for making escape sequences of this kind legal or illegal here, but I think it probably needs to be clearer whether they are legal or not.

Comment (received 2002-07-13) -- Re: fixed-length escapes

Comment (received 2002-07-12) -- fixed-length escapes

[...] 'Escape syntaxes where the end is determined by a character outside the set of characters admissible in the character escape itself SHOULD be avoided' is a good provision, but at first glance it seemed to be saying that the terminating semicolon of entity and character references (which is 'a character outside the set of characters admissible in the character escape itself') was being deprecated. I think rephrasing might help, though I have not been able to draft a better alternative.

Comment (received 2002-05-27) -- Comments on charmod from Chris

'[S] If the unique encoding approach is not chosen, specifications MUST designate at least one of the UTF-8 and UTF-16 encoding forms of Unicode as admissible encodings and SHOULD choose at least one of UTF-8 or UTF-16 as mandated encoding forms (encoding forms that MUST be supported by implementations of the specification).'

Does that mean that, for example, saying UTF-8 is allowed and UTF-16 is disallowed and an encoding declaration is not required, is okay?

Comment (received 2002-05-27) -- Comments on charmod from Chris

Needs a little more on encodings that are a group of similar but not identical encodings, for example shift-jis.

Comment (received 2002-05-27) -- Comments on charmod from Chris

'Because of the layered Web architecture (e.g. formats used over protocols), there may be multiple and at times conflicting information about character encoding. [S] Specifications MUST define conflict-resolution mechanisms (e.g. priorities) for cases where there is multiple or conflicting information about character encoding.'

Yes. Better though to not define such layering; the XML MIME RFC messed this up by allowing the charset and the xml encoding declaration to differ and for the former to take precedence; this requires 'save as' to rewrite the XML otherwise it is no longer well formed.... better to require any transcoders to leave XML alone or to know how to rewrite the encoding declaration if they change the encoding.

Comment (received 2002-05-27) -- Comments on charmod from Chris

'Certain encodings are more or less associated with certain languages (e.g. Shift-JIS with Japanese); trying to support a given language or set of customers may mean that certain encodings have to be supported.'

The corollary should be clearly stated: do not assume that 'everyone' supports a favored but non-mandated encoding 'every parser I know supports Latin-1/Shift-JIS' is not true.

Comment (received 2002-07-23) -- Apple comments on Character Model

Apple shares Microsoft's concern that the Character Model is overly strict with respect to forcing NFC conformance on all Web-based applications. Apple generally follows the philosophy of 'be lenient in what you accept and strict in what you produce,' which seems to us to have proven a good model historically. Therefore, we would prefer a Character Model that recommends that content producers generate NFC, but that does not require that applications (including XML parsers and other content producers and proxies) must reject anything that is not NFC.

Comment (received 2002-06-28) -- XSL WG Comments on Chairacter Model WD

'[S] [I] A text-processing component that receives suspect text MUST NOT perform any normalization-sensitive operations unless it has first confirmed through inspection that the text is in normalized form, and MUST NOT normalize the suspect text . Private agreements MAY, however, be created within private systems which are not subject to these rules, but any externally observable results MUST be the same as if the rules had been obeyed.'

The exception for private agreements is crippled by the observable results restriction thus when all is said and done any suspect text will always remain.

Comment (received 2002-09-10) -- Character Model Comments Clarifications

Comment (received 2002-06-28) -- XSL WG Comments on Chairacter Model WD

'[S] [I] A text-processing component that receives suspect text MUST NOT perform any normalization-sensitive operations unless it has first confirmed through inspection that the text is in normalized form, and MUST NOT normalize the suspect text . Private agreements MAY, however, be created within private systems which are not subject to these rules, but any externally observable results MUST be the same as if the rules had been obeyed.'

Section 4.4 appears to require that XML be changed to disallow the use of a composing character as the first character in an entity. This change would be backwards incompatible. XSL WG specifications such as XSLT and XPath must continue to work with all XML well-formed documents.

Comment (received 2002-06-28) -- XSL WG Comments on Chairacter Model WD

'[S] [I] A text-processing component that receives suspect text MUST NOT perform any normalization-sensitive operations unless it has first confirmed through inspection that the text is in normalized form, and MUST NOT normalize the suspect text . Private agreements MAY, however, be created within private systems which are not subject to these rules, but any externally observable results MUST be the same as if the rules had been obeyed.'

Since the contents of an XML text node are 'suspect text' (there is nothing to prevent use of a composing character as the first character in a text node), section 4.4 appears to be saying that XPath must disallow operations such as substring() unless the text is inspected and found to be normalized. We do not believe that users want to pay the high cost of this feature.

Comment (received 2002-10-24) -- Re: 'x-' prefix on charset names

Yes, that is precisely what I meant. Thank you very much for taking my question to WG.