Christian,

I can address a few of these:

[3] ft-3.4-examples-q1.xq:

Query yields a boolean, whereas the result contains elements

--Yes. Fixed, this one and one that follows. The results now contain true.

[4] ftstaticcontext-results-q1.txt:

Result file is incorrect (superfluous: line 377-end)

--Removed. Static context queries will be replaced.

[5] ftstaticcontext-results-q2.txt:

Result file is incorrect (superfluous: line 280-end)

--Removed. Static context queries will be replaced.

[6] ftstaticcontext-results-q6.txt:

All books expected as result ('approved'/'approve' -> 'approv')

--Removed. Static context queries will be replaced.

[18] ftwildcard-q2.xq

<book number="2"> is missing

--This query looks for "site." site followed a period (not an indicator). There is a sites. The title ends in site but does not end in a period. I don't see "site." in Book 2.

--The language document says: The "without wildcards" option finds tokens without recognizing wildcard indicators and qualifiers. Periods, question marks, asterisks, plus signs, and two numbers separated by a comma, both enclosed by curly braces, are always recognized as ordinary text characters.

--Might you have been thinking the period was an indicator or I am missing your point?

[19] ftwildcard-q5.xq

<book number="1"> expected as result (as it contains the word "task")

--I was clearly confused about what word I was searching on, so I changed the word in query to match the description to "site.?" That word is not in any book for the same reasons described above, unless again I am missing oyur point.

[29] ft-3.4.5-examples-q1.xq

true expected (or lowercase option missing)

--Yes, indeed the lowercase option was missing. I added it. 

As always thanks to you Christian.

Pat

Dear Pat,

thanks a lot for the prompt reply! As usual, just a small comment..

[18] ftwildcard-q2.xq
[19] ftwildcard-q5.xq

> --This query looks for "site." site followed a period (not an indicator). 
> There is a sites. The title ends in site but does not end in a period. I 
> don't see "site." in Book 2. [...]

This is an interesting point for discussion. I had another look into the XQFT Tokenization section (4.1). If I get it right, the tokenizer won't care about characters which are not part of tokens; so I would expect the two following queries to return true:

  'a b' ftcontains 'a.b' 
  'a.b' ftcontains 'a b' 

This is why I would expect the dot in "site." to be ignored in the default (without wildcards) mode. - Please tell me if I got something wrong.

Thanks again,

Christian, BaseX Team 
http://www.basex.org

Christian, I think I disagree with your argument in comment 2 (http://www.w3.org/Bugs/Public/show_bug.cgi?id=6469#c2).  There, you said: 

>This is an interesting point for discussion. I had another look into the XQFT
>Tokenization section (4.1). If I get it right, the tokenizer won't care about
>characters which are not part of tokens; so I would expect the two following
>queries to return true:

>  'a b' ftcontains 'a.b' 
>  'a.b' ftcontains 'a b' 

I have just finished re-reading all of section 4.1 very carefully and I didn't find any suggestion that punctuation was never (part of) a token.  In the first of your two examples (quoted above), the first search context 'a b' has, I believe, two tokens, 'a' and 'b'. There, I think we are in agreement.  However, the search pattern 'a.b' has, I think, three tokens, 'a', '.', and 'b'.  (Caution: tokenization is implementation-defined and I'm presuming a tokenizer with certain behaviors that I believe are common in western-world language products.)  Because there is no token corresponding to '.' in the search context, I believe that query must return false. 

Similarly, in the second example, the search context has three tokens and the search pattern has two.  Because the search context does not have two tokens 'a' and 'b' that are adjacent, I believe that this query must also return false.  However, if the query had been written:
   'a.b' ftcontains 'a' ftand 'b'
then it would of course return true. 

If I'm wrong, please correct my analysis!

P.S., I'm working today on the remaining issues in your bug report to resolve the ones I can and assign the others. 

Hi Jim,

thank you for commenting the tokenization issue. If I get it right, tokenization of punctuation eventually is implementation dependent as well, so - as long as it is not specified if punctuation is to be treated as own token - the two discussed queries..

  'a b' ftcontains 'a.b' 
  'a.b' ftcontains 'a b'

..can either return true or false. Concerning the examples in 4.1.1, punctuation is ignored in the tokenization process - or, to put it differently, space and punctuation is treated the same way here:

  "Ford Mustang 2000, 65K, excellent [...]"
  -> Ford(1) Mustang(2) 2000(3), 65K(4), excellent(5)

So the following queries..

  "Ford Mustang 2000, 65K" ftcontains "2000 65K"
  "Ford Mustang 2000 65K" ftcontains "2000, 65K"

..should return true for these examples. What do you think?

Christian, BaseX Team 
http://www.basex.org

..another question (I hope I didn't overlook this point in the specification): can spaces be handled as tokens as well? In this case, we would run into trouble with many position filters in the test suite..

 'A B' ftcontains ('A' ftand 'B') distance at most 0 words
  -> true/false?

Christian, I think that I still disagree. The only way that "a.b" could contain "a b" or vice versa is if the tokenizer recognized "." as a token separator and not as a token. Because tokenization is so completely implementation-defined, anything is possible. Maybe Pat or Mary will have some good ideas about this, 'cause I have too little real-world Full Text experience to be very certain. 

W.R.T. your question about whether spaces can be recognized as tokens: Again, because tokenization is so completely implementation-defined, it's possible. However, I do not believe that a tokenizer that did that would survive in the marketplace, so I don't believe we need to accommodate that possibility. 

Therefore, your query:
   'A B' ftcontains ('A' ftand 'B') distance at most 0 words
would, IMHO, always return true. 

In this comment, I'm going to respond to some more of your bug reports:

[1] ft-3.2-examples-q5.xq:

You're right about having only a minimum of queries with scoring, because it's
very difficult to predict what the results might be.  The XML Query WG and XSL
WG have a sort of policy that requires us to place every example in the spec
into the test suite, and this is one of those.  I'm not entirely sure how to
resolve this dilemma other than to either (1) change the comparison specified
in the catalog to "inspect" or (2) add a second possible result of "no file at
all".  The TF will discuss this issue. 


[2] ft-3.3-examples-q1.xq:
You're right. This should be changed in the spec and in the test. 


[7] FTPrimary-FTWords-any-q4b.xq:
You are obviously correct.  Fixed.  (Note that this result had all 9 paragraphs
and should have had none, but the result of the next test you cite had no
paragraphs and should have had all 9.  I suspect the result files got
reversed.) 


[8] FTPrimary-FTWords-anyword-q4b.xq
Again, obviously correct. Fixed.  (Note that this result had no paragraphs and
should have had all 9, but the result of the previous test you cite had all 9
paragraphs and should have had none.  I suspect the result files got reversed.)



[9] FTPrimary-FTWords-anyword-q2b_result.xml
Good catch! Fixed. 


[10] FTWords/*.xml
I appreciate the comment, but it would be extremely helpful if you could
identify the specific locations. 


[11] FTPrimary-FTWords-phrase-q3a.xq - -q4a.xq
I agree.  I've fixed the problems in each of the tests in slightly different
ways. 
I also think that FTPrimary-FTWords-phrase-q4b.xq had the wrong result; I
believe that its result file should contain only an empty <paragraphs> element.
Do you agree?


[12] FTOr-badexpr1.xq
Good question; the TF will discuss this. 


[13] FTNot-q1.xq
I think I agree with you, but the TF should discuss this.  Of course, the
question is 'Does book[para ftcontains ftnot "Ninja"] mean "find all books that
have a para element that doesn't contain 'Ninja'" or "Find all books that do
not have a para element that contains 'Ninja'"?'  You believe that it's the
former, and I think you're probably right. 


[15] FTNot-q2.xq / -q3.xq / -q4.xq / -q5.xq
I agree; I have fixed q2 and q3, but will ask the test author to fix q4 and q5
(because the result will change from .xml to .txt and the comparison from XML
to Fragment). 


And, unfortunately, that's all I have time to address today. With luck, other
TF members will be able to work on the remaining issues soon. 

Hi Jim,

thanks again for the quick reply, and sorry for persisting on the tokenizer issue. Yes, I'll be interested what the other TF members say about this as I'm still not sure how to interpret the examples in 4.1.1. As stated before, using a Western tokenizer and assuming that commas and periods are treated the same way, I would expect all of the following queries to yield true (otherwise, the position information, which is shown in parentheses in this section, would not make much sense):

   "Ford Mustang 2000, 65K" ftcontains "2000 65K"
   "Ford Mustang 2000, 65K" ftcontains "2000.65K"
   "Ford Mustang 2000  65K" ftcontains "2000.65K"

Concerning #10 and #11:

> [10] FTWords/*.xml
> I appreciate the comment, but it would be extremely helpful
> if you could identify the specific locations. 

Hm, there might have been around 10 files with missing spaces. You could try search and replace in this directory for the following strings, I hope that the three replacements will cover all cases:

 "checked.<"        ->  "checked. <"
 "ue</nt>is"        ->  "ue</nt> is"
 "on</nt>produces"  ->  "on</nt> produces"

> [11] FTPrimary-FTWords-phrase-q3a.xq - -q4a.xq
> I agree.  I've fixed the problems in each of the tests in
> slightly different ways. 
> I also think that FTPrimary-FTWords-phrase-q4b.xq had the wrong result;
> I believe that its result file should contain only an empty <paragraphs>
> element. Do you agree?

In my version from Jan 24, the result was already an empty <paragraphs> element; but I haven't checked the latest version.

Christian, BaseX Team 
http://www.basex.org

Hi Christian.

Re:
[18] ftwildcard-q2.xq
[19] ftwildcard-q5.xq

> --This query looks for "site." site followed a period (not an indicator). 
> There is a sites. The title ends in site but does not end in a period. I 
> don't see "site." in Book 2. [...]

This is an interesting point for discussion. I had another look into the XQFT
Tokenization section (4.1). If I get it right, the tokenizer won't care about
characters which are not part of tokens; so I would expect the two following
queries to return true:

  'a b' ftcontains 'a.b' 
  'a.b' ftcontains 'a b' 

This is why I would expect the dot in "site." to be ignored in the default
(without wildcards) mode. - Please tell me if I got something wrong.

--I see your point. I am an end user and I sometimes project my preferences where they don't belong. I would love to have search engine that can find special characters, in this case to allow me to fund "site." as a token. But you are absolutely correct that depending on the tokenization, finding "site" would be just as valid, and that would return Books 1 and 2. So I have added a qualifier to the descriptions, changed the comparator values to Inspect, and added second output files to q2 and q5.

Thanks again for your comments. It is very important to get this right.

Pat

Christian,

We addressed one more item during the FTTF telecon today:

[13] FTNot-q1.xq

This one returns all <book> elements in which the <para> element does not
contain the word "Ninja". In the result, even the books are listed which have
no <para> element; I would only expect 4 instead of 9 titles as result.

--The Task Force agreed with your comment and added 4 to the expected result.

Pat

[12] Empty sequence is allowed, and produces no matches.  Added a different test case to test the case where the expression returns something that can't be treated as xs:string*.

[15] Correct.

[16](a,b) Correct. Fixed results.

[17] Correct. Fixed results.

[20] Correct. Swapped queries.

[21]/[22] Correct, confused with distance. Fixed math.

[23] Fixed queries: intended to have single scope.

[24] Fixed result.

[25] Result already fixed.

[26], [27] Result fixed.

Christian, I believe this resolves everything you reported in this bug. If you agree, could you please close this bug. Thank you.

Dear Pat, thanks for your comments. I'll close the bug (issue [28], however, is still open). - Christian