11885 2011-01-27 08:59:00 +0000 [XQFTTS] english-stems.txt stemming dictionary 2011-06-28 07:28:14 +0000 1 1 1 Unclassified XPath / XQuery / XSLT Full Text 1.0 Proposed Recommendation PC Windows NT CLOSED FIXED P2 normal --- 1 tim jim.melton holstege paul public-qt-comments oldest_to_newest 44787 0 tim 2011-01-27 08:59:00 +0000 The file "english-stems.txt" contains stemming rules only for lower case text. However, the specification clearly states that the "Stemming Option must be applied before the Case Option and the Diacritics Option". So when tokenizing the string "Dogs and Cats" with stemming, the okens presented to the tokenizer must be "Dogs", "and", "Cats". The guidelines for running XQFTTS state that the "stemming-dictionary is a plain text file containing lines of whitespace-separated tokens. Each token on the line should stem to the first token on the line." Note that it is conceivable that the stemming dictionary might stem "AIDS" to "AIDS" but "aids" to "aid". This would be a useful test of the order of application of stemming and case options. Presumably the test suite doesn't currently test this. 45935 1 holstege 2011-02-22 19:44:09 +0000 Correct, stemming should be case-sensitive. I have added two additional tests ft-matchoptions-q5 and ft-matchoptions-q6 to test this case. Please indicate your satisfaction with this resolution by closing this bug. 45937 2 paul 2011-02-22 20:26:56 +0000 So how would one do case-insensitve stemming? 45954 3 tim 2011-02-23 10:18:56 +0000 (In reply to comment #1) > Correct, stemming should be case-sensitive. I have added two additional tests > ft-matchoptions-q5 and ft-matchoptions-q6 to test this case. Please indicate > your satisfaction with this resolution by closing this bug. Thanks. However, for some tests to pass, the english-stems.txt needs to have the following lines added. Dog Dogs Cat Cats Improve Improves Improving Improved Test Tests Testing Tested Plan Planning Conduct Conducting 45957 4 tim 2011-02-23 10:37:44 +0000 (In reply to comment #2) > So how would one do case-insensitve stemming? Assuming that lowercase(AB) = ab lowercase(Ab) = ab lowercase(aB) = ab lowercase(ab) = ab one would ensure that if the implementation's stemming algorithm was such that stem(AB) = AB then stem(Ab) = Ab stem(aB) = aB stem(ab) = ab Thus when the case option is case-insensitive, applying the case option to the stem would always return 'ab' for each of AB, Ab, aB and ab. 48566 5 holstege 2011-05-17 15:29:47 +0000 Made the additions to english-stems.txt. 49244 6 tim 2011-06-07 07:43:05 +0000 english-stems.txt is still missing Test Tests Testing Tested which is required for the following tests. ftstaticcontext-q2 ftstaticcontext-q4 ftstaticcontext-q5 stemming-queries-results-q1 stemming-queries-results-q1b xquery-xpath-composability-queries-results-q2 50385 7 holstege 2011-06-28 04:14:41 +0000 Done, per comment 6. 50388 8 tim 2011-06-28 07:28:14 +0000 Confirmed fixed. Thanks.