5122 2007-10-01 19:40:35 +0000 [FT] Section 4 Tokenization constraint 2007-10-11 21:40:17 +0000 1 1 1 Unclassified XPath / XQuery / XSLT Full Text 1.0 Last Call drafts PC Linux CLOSED FIXED P2 normal --- 1 holstege jim.melton public-qt-comments oldest_to_newest 16956 0 holstege 2007-10-01 19:40:35 +0000 The definition of tokenization in section 4 includes the rule: "The tokenizer MUST, when tokenizing two equal items, identify the same tokens in each." This is too strong. The context in which the items arise may impact the tokenization of those items. As a simple example: the parent element may provide different xml:lang attributes. Other implementation-specific configuration information may apply to ancestors of the item and impact how the item itself is tokenized. 17152 1 holstege 2007-10-11 21:40:03 +0000 Agreed to at f2f meeting 11 Oct 2007: Change text to: The tokenizer SHOULD, when tokenizing two equal items, identify the same tokens in each. The cases where tokenization of two equal items does not identity the same tokens in each is implementation-defined.