| TOC |
|
This document is an Internet-Draft and is subject to all provisions of section 3 of RFC 3667. By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she become aware will be disclosed, in accordance with RFC 3668.
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."
The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html.
This Internet-Draft will expire on April 1, 2005.
Copyright (C) The Internet Society (2004).
Many Internet application protocols include string-based lookup, searching, or sorting operations. However the problem space for searching and sorting international strings is large, not fully explored, and is outside the area of expertise for the Internet Engineering Task Force (IETF). Rather than attempt to solve such a large problem, this specification creates an abstraction framework so that application protocols can precisely identify a comparison function and the repertoire of comparison functions can be extended in the future.
1.
Introduction
1.1
Structure of this Document
1.2
Conventions Used in this Document
2.
Collation Definition and Purpose
2.1
Definition
2.2
Purpose
2.3
Sort Keys
3.
Collation Name Syntax
3.1
Basic Syntax
3.2
Wildcards
3.3
Ordering Direction
3.4
URIs
3.5
Naming Guidelines
4.
Collation Specification Requirements
4.1
Operations Supported
4.1.1
Equality
4.2
Substring
4.3
Ordering
4.4
Internal Canonicalization Algorithm
4.5
Use of Lookup Tables
4.6
Treatement of NULL Strings
4.7
Multi-Value Attributes
5.
Application Protocol Requirements
5.1
Character Encoding
5.2
Operations
5.3
Wildcards
5.4
Canonicalization Function
5.5
Disconnected Clients
5.6
Error Codes
5.7
Octet Collation
6.
Use by ACAP and Sieve
7.
Collation Registration
7.1
Collation Registration Procedure
7.2
Collation Registration Format
7.2.1
Registration Template
7.2.2
The collation Element
7.2.3
The name Element
7.2.4
The title Element
7.2.5
The functions Element
7.2.6
The specification Element
7.2.7
The submitter Element
7.2.8
The owner Element
7.2.9
The version Element
7.2.10
The UnicodeVersion Element
7.2.11
The UCAVersion Element
7.2.12
The UCAMatchLevel Element
7.3
DTD for Collation Registration
7.4
Structure of Collation Registry
7.5
Example Initial Registry Summary
8.
Guidelines for Expert Reviewer
9.
Initial Collations
9.1
ASCII Numeric Collation
9.1.1
ASCII Numeric Collation Description
9.1.2
ASCII Numeric Collation Registration
9.2
ASCII Casemap Collation
9.2.1
ASCII Casemap Collation Description
9.2.2
Legacy English Casemap Collation Registration
9.2.3
English Casemap Collation Registration
9.3
Nameprep Collation
9.3.1
Nameprep Collation Description
9.3.2
Nameprep Collation Registration
9.4
Basic Collation
9.4.1
Basic Collation Description
9.4.2
Basic Collation Registration
9.4.3
Basic Accent Sensitive Match Collation Registration
9.4.4
Basic Case Sensitive Match Collation Registration
9.5
Octet Collation
9.5.1
Octet Collation Description
9.5.2
Octet Collation Registration
10.
IANA Considerations
11.
Security Considerations
12.
Open Issues
13.
Change Log
13.1
Changes From -02
13.2
Changes From -01
13.3
Changes From -00
14.
References
14.1
Normative References
14.2
Informative References
§
Authors' Addresses
§
Intellectual Property and Copyright Statements
| TOC |
The ACAPNewman, C. and J. Myers, ACAP -- Application Configuration Access Protocol, November 1997.[11] specification introduced the concept of a comparator (which we call collation in this document), but failed to create an IANA registry. With the introduction of stringprepHoffman, P. and M. Blanchet, Preparation of Internationalized Strings ("stringprep"), December 2002.[6] and the Unicode Collation AlgorithmDavis, M. and K. Whistler, Unicode Collation Algorithm version 9, July 2002.[8], it is now time to create that registry and populate it with some initial values appropriate for an international community. This specification replaces and generalizes the definition of a comparator in ACAP and creates a collation registry.
@@@@ to be completed
The key words "MUST", "MUST NOT", "SHOULD", "SHOULD NOT", and "MAY" in this document are to be interpreted as defined in "Key words for use in RFCs to Indicate Requirement Levels"Bradner, S., Key words for use in RFCs to Indicate Requirement Levels, March 1997.[1].
The attribute syntax specifications use the Augmented Backus-Naur Form (ABNF)Crocker, D. and P. Overell, Augmented BNF for Syntax Specifications: ABNF, November 1997.[2] notation including the core rules defined in Appendix A. This also inherits ABNF rules from Language TagsAlvestrand, H., Tags for the Identification of Languages, January 2001.[5].
The term 'protocol' is used in this memo in a very generic sense, and includes things such as query languages.
| TOC |
A collation is a named function which takes two arbitrary length character strings (with the exception of the i;octetOctet Collation collation) as input and can be used to perform one or more of three basic comparison operations: equality test, substring match, and ordering test.
Collations provide a multi-protocol abstraction layer for comparison functions so the details of a particular comparison operation can be specified by someone with appropriate expertise independent of the application protocol that consumes that collation. This is similar to the way a charsetFreed, N. and J. Postel, IANA Charset Registration Procedures, October 2000.[14] separates the details of octet to character mapping from a protocol specification such as MIMEFreed, N. and N. Borenstein, Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies, November 1996.[9] or the way SASLMyers, J., Simple Authentication and Security Layer (SASL), October 1997.[10] separates the details of an authentication mechanism from a protocol specification such as ACAPNewman, C. and J. Myers, ACAP -- Application Configuration Access Protocol, November 1997.[11].
Here a small diagram to help illustrate the value of this abstraction layer:
+-------------------+ +-----------------+
| IMAP i18n SEARCH |--+ | Basic |
+-------------------+ | +--| Collation Spec |
| | +-----------------+
+-------------------+ | +-------------+ | +-----------------+
| ACAP i18n SEARCH |--+--| Collation |--+--| A stringprep |
+-------------------+ | | Registry | | | Collation Spec |
| +-------------+ | +-----------------+
+-------------------+ | | +-----------------+
| ...other protocol |--+ | | locale-specific |
+-------------------+ +--| Collation Spec |
+-----------------+
Thus IMAP, ACAP and future application protocols with international search capability simply specify how to interface to the collation registry instead of each protocol specification having to specify all the collations it supports.
One component of a collation is a canonicalization function which can be pre-applied to single strings and may enhance the performance of subsequent comparison operations. Normally, this is an implementation detail of collations, but at times it may be useful for an application protocol to expose collation canonicalization over protocol. Collation canonicalization can range from an identity mapping (e.g., the i;octet collation Section 9.5Octet Collation) to a mapping which makes the string unreadable to a human (e.g., the basic collation).
| TOC |
The collation name itself is a single US-ASCII string beginning with a letter and made up of letters, digits, or one of the following 4 symbols: "-", ";", "=" or ".". The name MUST NOT be longer than 254 characters.
collation-char = ALPHA / DIGIT / "-" / ";" / "=" / "." collation-name = ALPHA *253collation-char
The string a client uses to select a collation MAY contain a wildcard ("*") character which matches zero or more collation-chars. Wildcard characters MUST NOT be adjacent. Clients which support disconnected operation SHOULD NOT use wildcards to select a collation, but clients which provide collation operations only when connected to the server MAY use wildcards. If the wildcard string matches multiple collations, the server SHOULD select the collation with the broadest scope (preferably international scope), the most recent table versions and the greatest number of supported operations. A single wildcard character ("*") refers to the application protocol collation behavior that would occur if no explicit negotiation were used.
collation-wild = ("*" / (ALPHA ["*"])) *(collation-char ["*"])
; MUST NOT exceed 255 characters total
When used as a protocol element for ordering, the collation name MAY be prefixed by either "+" or "-" to explicitly specify an ordering direction. As mentioned previously, "+" has no effect on the ordering function, while "-" negates the result of the ordering function. In general, collation-order is used when a client requests a collation, and collation-sel is used with the server informs the client of the selected collation.
collation-sel = ["+" / "-"] collation-name collation-order = ["+" / "-"] collation-wild
Some protocols are designed to use URIs to refer to collations rather than simple tokens. A special section of the IANA web page is reserved for such usage. The "collation-uri" form is used to refer to a specific IANA registry entry for a specific named collation (the collation registration may not actually be present if it is experimental). The "collation-auri" form is an abstract name for an ordering, a comparator pattern or a vendor private comparator.
collation-uri = "http://www.iana.org/assignments/collation/"
collation-name ".xml"
collation-auri = ( "http://www.iana.org/assignments/collation/"
collation-order [".xml"]) / other-uri
other-uri = absoluteURI
; excluding the IANA collation namespace.
While this specification makes no absolute requirements on the structure of collation names, naming consistency is important, so the following initial guidelines are provided.
Collation names with an international audience typically begin with "i;". Collation names intended for a particular language or locale typically begin with a language tagAlvestrand, H., Tags for the Identification of Languages, January 2001.[5] followed by a ";". After the first ";" is normally the name of the general collation algorithm followed by a series of algorithm modifications separated by the ";" delimiter. Parameterized modifications will use "=" to delimit the parameter from the value. The version numbers of any lookup tables used by the algorithm SHOULD be present as parameterized modifications.
Collation names of the form *;vnd-domain.com;* are reserved for vendor-specific collations created by the owner of the domain name following the "vnd-" prefix. Registration of such collations (or the name space as a whole) with intended use of "Vendor" is encouraged when a public specification or open-source implementation is available, but is not required.
| TOC |
A collation specification MUST state which of the three basic functions are supported (equality, substring, ordering) and how to perform each of the supported functions on any two input character strings including empty strings (with the exception of the i;octetOctet Collation collation). Collations must be deterministic, i.e.given a collation with a specific name, and any two fixed input strings, the result MUST be the same for the same operation. Collations MUST be transitive.
The equality function always returns "match" or "no-match" when supplied valid input and MAY return "error" if the input strings are not valid character strings or violate other collation constraints.
The substring matching function determines if the first string is a substring of the second string. A collation which supports substring matching will automatically support the two special cases of substring matching: prefix and suffix matching if those special cases are supported by the application protocol. It returns "match" or "no-match" when supplied valid input and returns "error" when supplied invalid input.
Application protocols MAY return position information for substring matches. If this is done, the position information MUST include both the starting offset and the ending offset in the string. This is important because more sophisticated collations can match strings of unequal length (for example, a pre-composed accented character will match a decomposed accented character).
The ordering function determines how two character strings are ordered. It returns "-1" if the first string is listed before the second string according to the collation, "+1" if the second string is listed before the first string, and "0" if the two strings are equal. If the order of the two strings is reversed, the result of the ordering function of the collation MUST be reversed, i.e. results which would be "+1" are instead "-1" and results which would be "-1" are instead "+1", while results which would be "0" stay "0". In general, collations SHOULD NOT return "0" unless the two character sequences are identical.
Since ordering is normally used to sort a list of items, "error" is not a useful return value from the ordering function. Strings with errors that prevent the sorting algorithm from functioning correctly should sort to the end of the list. Thus if the first string is invalid while the second string is valid, the result will be "+1". If the second string is invalid while the first string is valid, the result will be "-1". If both strings are invalid, the result SHOULD match the result from the "i;octet" collation.
When the collation is used with a "+" prefix, the behavior is the same as when used with no prefix. When the collation is used with a "-" prefix, the result of the ordering function of the collation MUST be reversed.
A collation specification MUST describe the internal canonicalization algorithm. This algorithm can be applied to individual strings and the result strings can be stored to potentially optimize future comparison operations. A collation MAY specify that the canonicalization algorithm is the identity function. The output of the canonicalization algorithm MAY have no meaning to a human.
Collations which use more than one customizable lookup table in a documented format MUST assign numbers to the tables they use. This permits an application protocol command to access the tables used by a server collation.
Unless otherwise specified by the collation or application protocol, a NULL string (as opposed to an empty string) is equal only to another NULL string, a NULL string is not a substring of any other string, and a NULL string sorts to a position after all non-NULL strings, but before strings which generate errors.
Some application protocols will permit the use of multi-value attributes with a collation. This paragraph describes the rules that apply unless otherwise specified by the collation or application protocol. In the case of the equality and substring operation, the operations are applied over each pair of single values from the two inputs. If any combination produces an error, the result is an error. Otherwise, if any combination produces a "match", the result is a match. Otherwise the result is "no-match". For the ordering function, the smallest ordinal character string from the first set of values is compared to the smallest ordinal character string from the second set of values.
| TOC |
This section describes the requirements and issues that an application protocol which offers searching, substring matching and/or sorting and permits the use of characters outside the US-ASCII charset needs to consider.
The protocol specification has to make sure that it is clear on which characters (rather than just octets) the collations are used. This can be done by specifying the protocol itself in terms of characters (e.g. in the case of a query language), by specifying a single character encoding for the protocol (e.g. UTF-8 [3]Yergeau, F., UTF-8, a transformation format of ISO 10646, November 2003.), or by carefully describing the relevant issues of character encoding labeling and conversion. In the later case, details to consider include how to handle unknown charsets, any charsets which are mandatory-to-implement, any issues with byte-order that might apply, and any transfer encodings which need to be supported.
The protocol must specify which of the operations defined in this specification (equality matching, substring matching and ordering) can be invoked in the protocol, and how they are invoked. There may be more than one way to invoke an operation.
The protocol MUST provide a mechanism for the client to select the collation to use with equality matching, substring matching and ordering.
If the protocol provides positional information for the results of a substring match, that positional information MUST fully specify the substring in the result that matches independent of the length of the search string. For example, returning both the starting and ending offset of the match would suffice, as would the starting offset and a length. Returning just the starting offset is not acceptable. This rule is necessary because advanced collations can treat strings of different lengths as equal (for example, pre-composed and decomposed accented characters).
The protocol MUST specify whether it allows the use of wildcards in collation identifiers or not. If the protocol allows wildcards, then:
The protocol MUST specify how comparisons behave in the absence of explicit collation negotiation or when a collation of "*" is requested. The protocol MAY specify that the default collation used in such circumstances is sensitive to server configuration.
The protocol SHOULD provide a way to list available collations matching a given wildcard pattern or patterns.
If the protocol provides a canonicalization function for strings, then use of collations MAY be appropriate for that function. [Need to describe how that would be done.]
If the protocol supports disconnected clients, then a mechanism for the client to precisely replicate the server's collation algorithm is likely desirable. Thus the protocol MAY wish to provide a command to fetch lookup tables used by charset conversions and collations.
The protocol specification should consider assigning protocol error codes for the following circumstances:
If the protocol permits the use of the i;octetOctet Collation collation, it has to say so. The octet collation SHOULD NOT be used unless the protocol uses UTF-8 as its single character encoding.
If the protocol permits the use of collations with data structures beyond those described in this specification ([is the following a list of described data structures, or of undescribed data structures???] octet strings, NULL string, array of octet strings), the protocol MUST describe the default behavior for a collation with that data structure.
| TOC |
Both ACAPNewman, C. and J. Myers, ACAP -- Application Configuration Access Protocol, November 1997.[11] and SieveShowalter, T., Sieve: A Mail Filtering Language, January 2001.[15] are standards track specifications which used collations prior to the creation of this specification and registry. Those standards do not meet all the application protocol requirements described in Section 5Application Protocol Requirements. For backwards compatibility, those protocols use the "i;ascii-casemap" instead of "en;ascii-casemap". [have to check whether the following is true:] These protocols allow the use of the i;octetOctet Collation collation working directly on UTF-8 data as used in these protocols.
| TOC |
IANA will create a mailing list collation@iana.org which can be used for public discussion of collation proposals prior to registration. Use of the mailing list is encouraged but not required. The actual registration procedure will not begin until the completed registration template is sent to iana@iana.org. The IESG will appoint a designated expert who will monitor the collation@iana.org mailing list and review registrations forwarded from IANA. The designated expert is expected to tell IANA and the submitter of the registration within two weeks whether the registration is approved, approved with minor changes, or rejected with cause. When a registration is rejected with cause, it can be re-submitted if the concerns listed in the cause are addressed. Decisions made by the designated expert can be appealed to the IESG and subsequently follow the normal appeals procedure for IESG decisions.
Collation registrations in a standards track, BCP or IESG-approved experimental RFC are owned by the IETF, and changes to the registration follow normal procedures for updating such documents. Collation registrations in other RFCs are owned by the RFC author(s). Other collation registrations are owned by the individual(s) listed in the contact field of the registration and IANA will preserve this information. Changes to a registration MUST be approved by the owner. In the event the owner cannot be contacted for a period of one month and a change is deemed necessary, the IESG MAY re-assign ownership to an appropriate party.
Registration of a collation is done by sending a well-formed XML document that validates with collationreg.dtdDTD for Collation Registration.
Here is a template for the registration:
<?xml version='1.0'?> <!DOCTYPE rfc SYSTEM 'collationreg.dtd'> <collation rfc="YYYY" scope="i18n" intendedUse="common"> <name>collation name</name> <title>technical title for collation</title> <functions>equality order substring</functions> <specification>specification reference</specification> <owner>email address of owner or IETF</owner> <submitter>email address of submitter<submitter> <version>1</version> <UnicodeVersion>3.2</UnicodeVersion> <UCAVersion>3.1.1</UCAVersion> </collation>
The root of the registration document MUST be a <collation> element. The collation element contains the other elements in the registration, which are described in the following sub-subsections, in the order given here.
The <collation> element MAY include an "rfc=" attribute if the specification is in an RFC. The "rfc=" attribute gives only the number of the RFC, without any prefix, such as "RFC", or suffix, such as ".txt".
The <collation> element MUST include a "scope=" attribute, which MUST have one of the values "i18n", "local" or "other".
The <collation> element MUST include an "intendedUse=" attribute, which must have one fo the values "common", "limited", "vendor", or "deprecated". Collation specifications intended for "common" use are expected to reference standards from standards bodies with significant experience dealing with the details of international character sets.
Be aware that future revisions of this specification may add additional function types, as well as additional XML attributes and values. Any system which automatically parses these XML documents MUST take this into account to preserve future compatibility. A DTD for the current definition of the collation registration template is given in Section 7.3DTD for Collation Registration
The <name> element gives the precise name of the comparator. The <name> element is mandatory.
The <title> element give the title of the comparator. The <title> element is mandatory.
The <functions> element lists which of the three functions the comparator provides. The <functions> element is mandatory.
The <specification> element describes where to find the specification. The <specification> element is mandatory. It MAY have a URI attribute. [check that the following is really true; it reflects what is currently in the DTD; also, say what it means/in what cases it should be used] There may be more than one <specification> elements.
The <submitter> element provides an RFC 2822 email address for the person who submitted the registration. It is optional if the <owner> element contains an email address. [check that the following is really true; it reflects what is currently in the DTD; also, say what it means/in what cases it should be used] There may be more than one <submitter> elements.
The <owner> element contains either the four letters "IETF" or an email address of the owner of the registration. The <owner> element is mandatory. [check that the following is really true; it reflects what is currently in the DTD; also, say what it means/in what cases it should be used] There may be more than one <owner> elements.
The <version> element is included when the registration is likely to be revised or has been revised in such a way that the results change for certain input strings. The <version> element is optional.
The <UnicodeVersion> element indicates the version number of the UnicodeData file on which the collation is based. The <UnicodeVersion> element is optional.
The <UCAVersion> element specifics the version of the Unicode Collation Algorithm on which the collation is based. The <UCAVersion> element is optional.
The <UCAMatchLevel> element specifies the number of Unicode Collation Algorithm sort key levels used for the equality and substring operations. The <UCAMatchLevel> element is optional.
<!-
DTD for Collation Registration Document
Data types:
entity description
====== ===========
NUMBER [0-9]+
URI As defined in RFC YYYY
CTEXT printable ASCII text (no line-terminators)
TEXT character data
->
<!ENTITY % NUMBER "CDATA">
<!ENTITY % URI "CDATA">
<!ENTITY % CTEXT "#PCDATA">
<!ENTITY % TEXT "#PCDATA">
<!ELEMENT collation (name,title,functions,specification+,owner+,
submitter*,version?,UnicodeVersion?,
UCAVersion?,UCAMatchLevel?)>
<!ATTLIST collation
rfc %NUMBER; "0"
scope (i18n|local|other) #IMPLIED
intendedUse (common|limited|vendor|deprecated) #IMPLIED>
<!ELEMENT name (%CTEXT;)>
<!ELEMENT title (%CTEXT;)>
<!ELEMENT functions (%CTEXT;)>
<!ELEMENT specification (%TEXT;)>
<!ATTLIST specification
uri %URI; "">
<!ELEMENT owner (%CTEXT;)>
<!ELEMENT submitter (%CTEXT;)>
<!ELEMENT version (%CTEXT;)>
<!ELEMENT UnicodeVersion (%CTEXT;)>
<!ELEMENT UCAVersion (%CTEXT;)>
<!ELEMENT UCAMatchLevel (%CTEXT;)>
Once the registration is approved, IANA will store each XML registration document in a URL of the form http://www.iana.org/assignments/collation/collation-name.xml where collation-name is the contents of the name element in the registration. Both the submitter and the designated expert is responsible for verifying that the XML is well-formed and complies with the DTD. In the future, it is hoped IANA will take over XML verification responsibility from the designated expert.
IANA will also maintain a text summary of the registry under the name http://www.iana.org/assignments/collation/summary.txt. This summary is divided into four sections. The first section is for collations intended for common use. This section is intended for collation registrations published in IESG approved RFCs or for locally scoped collations from the primary standards body for that locale. The designated expert is encouraged to reject collation registrations with an intended use of "common" if the expert believes it should be "limited", as it is desirable to keep the number of "common" registrations small and high quality. The second section is reserved for limited use collations. The third section is reserved for registered vendor specific collations. The final section is reserved for deprecated collations.
The following is an example of how IANA might structure the initial registry summary.txt file:
Collation Functions Scope Reference
--------- --------- ----- ---------
Common Use Collations:
i;nameprep;v=1;uv=3.2 e, o, s i18n [RFC XXXX]
i;basic;uca=3.1.1;uv=3.2 e, o, s i18n [RFC XXXX]
i;basic;uca=3.1.1;uv=3.2;match=accent e, o, s i18n [RFC XXXX]
i;basic;uca=3.1.1;uv=3.2;match=case e, o, s i18n [RFC XXXX]
en;ascii-casemap e, o, s Local [RFC XXXX]
Limited Use Collations:
i;octet e, o, s Other [RFC XXXX]
i;ascii-numeric e, o Other [RFC XXXX]
Vendor Collations:
Deprecated Collations:
i;ascii-casemap e, o, s Local [RFC XXXX]
References
----------
[RFC XXXX] Newman, C., "Internet Application Protocol Collation
Registry", RFC XXXX, Sun Microsystems, October 2003.
| TOC |
The expert reviewer appointed by the IESG has fairly broad latitude for this registry. While a number of collations are expected (particularly customizations of the basic collation for localized use), an explosion of collations (particularly common use collations) is not desirable for widespread interoperability. However, it is important for the expert reviewer to provide cause when rejecting a registration, and when possible to describe corrective action to permit the registration to proceed. The following table includes some example reasons to reject a registration with cause:
| TOC |
This section describes an initial set of collations for the collation registry.
The "i;ascii-numeric" collation is a simple collation intended for use with arbitrary sized decimal numbers stored as octet strings of US-ASCII digits (0x30 to 0x39). It supports equality and ordering, but does not support the substring function. The algorithm is as follows:
The associated canonicalization algorithm is to truncate the input string at the first non-digit character.
<?xml version='1.0'?> <!DOCTYPE rfc SYSTEM 'collationreg.dtd'> <collation rfc="XXXX" scope="other" intendedUse="limited"> <name>i;ascii-numeric</name> <title>ASCII Numeric</title> <functions>equality order</functions> <specification>RFC XXXX</specification> <owner>IETF</owner> <submitter>chris.newman@sun.com<submitter> </collation>
The "en;ascii-casemap" collation is a simple collation intended for use with English language text in pure US-ASCII. It provides equality, substring and ordering functions. The algorithm first applies a canonicalization algorithm to both input strings which subtracts 32 (0x20) from all octet values between 97 (0x61) and 122 (0x7A) inclusive. The result of the collation is then the same as the result of the "i;octet" collation for the canonicalized strings. Care should be taken when using OS-supplied functions to implement this collation as this is not locale sensitive, but functions such as strcasecmp and toupper can be locale sensitive.
For historical reasons, in the context of ACAP and Sieve, the name "i;ascii-casemap" is a synonym for this collation.
<?xml version='1.0'?> <!DOCTYPE rfc SYSTEM 'collationreg.dtd'> <collation rfc="XXXX" scope="local" intendedUse="deprecated"> <name>i;ascii-casemap</name> <title>Legacy English Casemap</title> <functions>equality order substring</functions> <specification>RFC XXXX</specification> <owner>IETF</owner> <submitter>chris.newman@sun.com<submitter> </collation>
<?xml version='1.0'?> <!DOCTYPE rfc SYSTEM 'collationreg.dtd'> <collation rfc="XXXX" scope="local" intendedUse="common"> <name>en;ascii-casemap</name> <title>English Casemap</title> <functions>equality order substring</functions> <specification>RFC XXXX</specification> <owner>IETF</owner> <submitter>chris.newman@sun.com<submitter> </collation>
The "i;nameprep;v=1;uv=3.2" collation is an implementation of the nameprepHoffman, P. and M. Blanchet, Nameprep: A Stringprep Profile for Internationalized Domain Names (IDN), March 2003.[7] specification based on normalization tables from Unicode version 3.2. This collation applies the nameprep canoncialization function to both input strings and then returns the result of the i;octet collation on the canonicalized strings. While this collation offers all three functions, the ordering function it provides is inadequate for use by the majority of the world.
Version number 1 is applied to nameprep as specified in RFC 3491. If the nameprep specification is revised without any changes that would produce different results when given the same pair of input octet strings, then the version number will remain unchanged.
The table numbers for tables used by nameprep are as follows:
| Table Number | Table Name |
|---|---|
| 1 | UnicodeData-3.2.0.txt |
| 2 | Table B.1 |
| 3 | Table B.2 |
| 4 | Table C.1.2 |
| 5 | Table C.2.2 |
| 6 | Table C.3 |
| 7 | Table C.4 |
| 8 | Table C.5 |
| 9 | Table C.6 |
| 10 | Table C.7 |
| 11 | Table C.8 |
| 12 | Table C.9 |
<?xml version='1.0'?> <!DOCTYPE rfc SYSTEM 'collationreg.dtd'> <collation rfc="XXXX" scope="i18n" intendedUse="common"> <name>i;nameprep;v=1;uv=3.2</name> <title>Nameprep</title> <functions>equality order substring</functions> <specification>RFC XXXX</specification> <owner>IETF</owner> <submitter>chris.newman@sun.com<submitter> <version>1</version> <UnicodeVersion>3.2</UnicodeVersion> </collation>
The basic collation is intended to provide tolerable results for a number of languages for all three functions (equality, substring and ordering) so it is suitable as a mandatory-to-implement collation for protocols which include ordering support. The ordering function of the basic collation is the Unicode Collation AlgorithmDavis, M. and K. Whistler, Unicode Collation Algorithm version 9, July 2002.[8] version 9 (UCAv9).
The equality and substring functions are created as described in UCAv9 section 8. While that section is informative to UCAv9, it is normative to this collation specification.
This collation is based on Unicode version 3.2, with the following tables relevant:
0E40..0E44 ; Logical_Order_Exception # Lo [5] THAI CHARACTER SARA E..THAI CHARACTER SARA AI MAIMALAI 0EC0..0EC4 ; Logical_Order_Exception # Lo [5] LAO VOWEL SIGN E..LAO VOWEL SIGN AI # Total code points: 10
UCAv9 includes a number of configurable parameters and steps labelled as potentially optional. The following list summarizes the defaults used by this collation:
The exact collation name with these defaults is "i;basic;uca=3.1.1;uv=3.2". When a specification states that the basic collation is mandatory-to-implement, only this specific name is mandatory-to-implement.
In order to allow modification of the optional behaviors, the following ABNF is used for variations of the basic collation:
basic-collation = ("i" / Language-Tag) ";basic;uca=3.1.1;uv=3.2"
[";match=accent" / ";match=case"]
[";tailor=" 1*collation-char ]
If multiple modifiers appear, they MUST appear in the order described above. The modifiers have the following meanings:
- match=accent
- Both the first and second levels of the sort keys are considered relevant to the equality and substring operations (rather than the default of first level only). This makes the matching functions sensitive to accentual distinctions.
- match=case
- The first three levels of sort keys are considered relevant to the equality and substring operations. This makes the matching functions sensitive to both case and accentual distinctions.
The default weighting option is "non-ignorable". The "semi-stable" sort key option is not used by default.
The canonicalization algorithm associated with this collation is the output of step 3 of the UCAv9 algorithm (described in section 4.3 of the UCA specification). This canonicalization is not suitable for human consumption.
Finally, the UCAv9 algorithm permits the "allkeys" table to be tailored to a language. People who make quality tailorings are encouraged to register those tailorings using the collation registry. Tailoring names beginning with "x" are reserved for experimental use, are treated as "Limited use" and MUST NOT match wildcards if any registered collation is available that does match.
<?xml version='1.0'?> <!DOCTYPE rfc SYSTEM 'collationreg.dtd'> <collation rfc="XXXX" scope="i18n" intendedUse="common"> <name>i;basic;uca=3.1.1;uv=3.2</name> <title>Basic</title> <functions>equality order substring</functions> <specification>RFC XXXX</specification> <owner>IETF</owner> <submitter>chris.newman@sun.com<submitter> <UnicodeVersion>3.2</UnicodeVersion> <UCAVersion>3.1.1</UCAVersion> <UCAMatchLevel>1</UCAMatchLevel> </collation>
<?xml version='1.0'?> <!DOCTYPE rfc SYSTEM 'collationreg.dtd'> <collation rfc="XXXX" scope="i18n" intendedUse="common"> <name>i;basic;uca=3.1.1;uv=3.2;match=accent</name> <title>Basic Accent Sensitive Match</title> <functions>equality order substring</functions> <specification>RFC XXXX</specification> <owner>IETF</owner> <submitter>chris.newman@sun.com<submitter> <UnicodeVersion>3.2</UnicodeVersion> <UCAVersion>3.1.1</UCAVersion> <UCAMatchLevel>2</UCAMatchLevel> </collation>
<?xml version='1.0'?> <!DOCTYPE rfc SYSTEM 'collationreg.dtd'> <collation rfc="XXXX" scope="i18n" intendedUse="common"> <name>i;basic;uca=3.1.1;uv=3.2;match=case</name> <title>Basic Case Sensitive Match</title> <functions>equality order substring</functions> <specification>RFC XXXX</specification> <owner>IETF</owner> <submitter>chris.newman@sun.com<submitter> <UnicodeVersion>3.2</UnicodeVersion> <UCAVersion>3.1.1</UCAVersion> <UCAMatchLevel>3</UCAMatchLevel> </collation>
The "i;octet" collation is a simple and fast collation intended for use on binary octet strings rather than on character data. It is the only such collation; it is not possible to register additional collations with this property. Protocols that want to make this collation available have to do so by explicitly allowing it. If not explicitly allowed, it MUST NOT be used. It never returns an "error" result. It provides equality, substring and ordering functions.
The ordering algorithm is as follows:
This algorithm is roughly equivalent to the C library function memcmp with appropriate length checks added.
The matching function returns "match" if the sorting algorithm would return "0". Otherwise the matching function returns "no-match".
The substring function returns "match" if the first string is the empty string, or if there exists a substring of the second string of length equal to the length of the first string which would result in a "match" result from the equality function. Otherwise the substring function returns "no-match".
The associated canonicalization algorithm is the identity function.
This collation is defined with intendedUse="limited" because it can only be used by protocols that explicitly allow it.
<?xml version='1.0'?> <!DOCTYPE rfc SYSTEM 'collationreg.dtd'> <collation rfc="XXXX" scope="i18n" intendedUse="limited"> <name>i;octet</name> <title>Octet</title> <functions>equality order substring</functions> <specification>RFC XXXX</specification> <owner>IETF</owner> <submitter>chris.newman@sun.com<submitter> </collation>
| TOC |
Section 7Collation Registration defines how to register collations with IANA. This section should be carefully studied, and commented upon if necessary, by IANA before approval of this document for publication as an RFC.Section 9Initial Collations defines a list of predefined collations, which should be registered when this document is approved and published as an RFC.
| TOC |
Collations will normally be used with UTF-8 strings. Thus the security considerations for UTF-8Yergeau, F., UTF-8, a transformation format of ISO 10646, November 2003.[3] and stringprepHoffman, P. and M. Blanchet, Preparation of Internationalized Strings ("stringprep"), December 2002.[6] also apply and are normative to this specification.
| TOC |
See http://www.w3.org/2004/08/ietf-collation.
| TOC |
Add IANA comment to open issues. Otherwise this is just a re-publish to keep the document alive.
| TOC |
| TOC |
| [1] | Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, March 1997. |
| [2] | Crocker, D. and P. Overell, "Augmented BNF for Syntax Specifications: ABNF", RFC 2234, November 1997. |
| [3] | Yergeau, F., "UTF-8, a transformation format of ISO 10646", STD 63, RFC 3629, November 2003. |
| [4] | Berners-Lee, T., Fielding, R. and L. Masinter, "Uniform Resource Identifier (URI): Generic Syntax", draft-fielding-uri-rfc2396bis-07.txt (work in progress), April 2004. |
| [5] | Alvestrand, H., "Tags for the Identification of Languages", BCP 47, RFC 3066, January 2001. |
| [6] | Hoffman, P. and M. Blanchet, "Preparation of Internationalized Strings ("stringprep")", RFC 3454, December 2002. |
| [7] | Hoffman, P. and M. Blanchet, "Nameprep: A Stringprep Profile for Internationalized Domain Names (IDN)", RFC 3491, March 2003. |
| [8] | Davis, M. and K. Whistler, "Unicode Collation Algorithm version 9", July 2002. |
| TOC |
| [9] | Freed, N. and N. Borenstein, "Multipurpose Internet Mail Extensions (MIME) Part One: Format of Internet Message Bodies", RFC 2045, November 1996. |
| [10] | Myers, J., "Simple Authentication and Security Layer (SASL)", RFC 2222, October 1997. |
| [11] | Newman, C. and J. Myers, "ACAP -- Application Configuration Access Protocol", RFC 2244, November 1997. |
| [12] | Narten, T. and H. Alvestrand, "Guidelines for Writing an IANA Considerations Section in RFCs", BCP 26, RFC 2434, October 1998. |
| [13] | Resnick, P., "Internet Message Format", RFC 2822, April 2001. |
| [14] | Freed, N. and J. Postel, "IANA Charset Registration Procedures", BCP 19, RFC 2978, October 2000. |
| [15] | Showalter, T., "Sieve: A Mail Filtering Language", RFC 3028, January 2001. |
| TOC |
| Chris Newman | |
| Sun Microsystems | |
| 1050 Lakes Drive | |
| West Covina, CA 91790 | |
| US | |
| EMail: | chris.newman@sun.com |
| Martin Duerst (Note: Please write "Duerst" with u-umlaut wherever possible, for example as "Dürst" in XML and HTML.) | |
| W3C/Keio University | |
| 5322 Endo | |
| Fujisawa, Kanagawa 252-8520 | |
| Japan | |
| Phone: | +81 466 49 1170 |
| Fax: | +81 466 49 1171 |
| EMail: | mailto:duerst@w3.org |
| URI: | http://www.w3.org/People/D%C3%BCrst/ |
| TOC |
The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org.
This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Copyright (C) The Internet Society (2004). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.
Funding for the RFC Editor function is currently provided by the Internet Society.