6946 2009-05-23 17:06:06 +0000 [FT] Test Suite - Wildcards 2009-06-09 09:55:09 +0000 1 1 1 Unclassified XPath / XQuery / XSLT Full Text 1.0 Candidate Recommendation All All CLOSED FIXED P2 normal --- 1 christian.gruen jim.melton jmdyck public-qt-comments oldest_to_newest 25272 0 christian.gruen 2009-05-23 17:06:06 +0000 Dear Pat, dear all, I eventually decided to implement my own wildcard evaluator to disallow the support more sophisticated wildcard expressions by a regular expression matcher. As a consequence, I would now expect empty results for the following three results: [1] ftwildcard-q4.xq .//content ftcontains "task?" with wildcards As the question mark is not preceded by a period, the literal term "task?" will be matched against the text. [2] ftwildcard-q13.xq $cont ftcontains "specialist\." As the period is preceded by a backslash, the resulting literal term is "specialist." [3] ftwildcard-q14.xq $cont ftcontains "nex.\?" The period will be treated as a placeholder for one arbitrary character, and the question mark will be searched as literal. Indeed the "ftusecases.xml" documents contains the substring "task?", "specialist.", and "next?". However, if the full-text tokenizer interprets periods as sentence delimiters, and not as parts of tokens, the periods will not be passed on to the wildcard evaluator. This is why both of the following queries "task?" ftcontains "task?", "task?" ftcontains "task?" with wildcards will lead to the internal comparison "task" <-> "task?", and will both return false. The tokenizer could add periods and other special characters to the returned tokens. This, however, would lead to surprising results with normal ftcontains operations: "is this a task?" ftcontains "task" -> false Looking forward to your reply - thanks, Christian 25502 1 jmdyck 2009-06-08 16:34:59 +0000 > Indeed the "ftusecases.xml" documents contains the substring "task?", > "specialist.", and "next?". However, if the full-text tokenizer interprets > periods as sentence delimiters, and not as parts of tokens, the periods will > not be passed on to the wildcard evaluator. This is why both of the following > queries > > "task?" ftcontains "task?", > "task?" ftcontains "task?" with wildcards > > will lead to the internal comparison "task" <-> "task?", and will both return > false. But note that tokenization is applied, not just to the search item, but also to the query strings. If the latter's tokenization also interprets punctuation as token-delimiters, then the internal comparison will be "task" <-> "task" and the queries will return true. By the way, did you intend "with wildcards" for cases [2] and [3]? 25504 2 christian.gruen 2009-06-08 16:51:07 +0000 > But note that tokenization is applied, not just to the search item, but also > to the query strings. If the latter's tokenization also interprets > punctuation as token-delimiters, then the internal comparison will be > > "task" <-> "task" > > and the queries will return true. Thanks Michael. So I think I now understand that the full-text tokenizer itself must be completely aware to the wildcard syntax to return the correct tokens - which seems logical. > By the way, did you intend "with wildcards" for cases [2] and [3]? Yes, exactly - I was referring here to the two test suite files "ftwildcard-q13.xq" and "ftwildcard-q14.xq". I'll give an update to this bug report as soon as possible. Christian