XQuery and XPath Full Text 1.0

1.1 Full-Text Search and XML

As XML becomes mainstream, users expect to be able to search their XML documents. This requires a standard way to do full-text search, as well as structured searches, against XML documents. A similar requirement for full-text search led ISO to define the SQL/MM-FT [SQL/MM] standard. SQL/MM-FT defines extensions to SQL to express full-text searches providing functionality similar to that defined in this full-text language extension to XQuery 1.0 and XPath 2.0.

XML documents may contain highly structured data (fixed schemas, known types such as numbers, dates), semi-structured data (flexible schemas and types), markup data (text with embedded tags), and unstructured data (untagged free-flowing text). Where a document contains unstructured or semi-structured data, it is important to be able to search using Information Retrieval techniques such as scoring and weighting.

Full-text search is different from substring search in many ways:

1. A full-text search searches for tokens and phrases rather than substrings. A substring search for news items that contain the string "lease" will return a news item that contains "Foobar Corporation releases the 20.9 version ...". A full-text search for the token "lease" will not.

2. There is an expectation that a full-text search will support language-based searches which substring search cannot. An example of a language-based search is "find me all the news items that contain a token with the same linguistic stem as 'mouse'" (finds "mouse" and "mice"). Another example based on token proximity is "find me all the news items that contain the tokens 'XML' and 'Query' allowing up to 3 intervening tokens".

3. Full-text search must address the vagaries and nuances of language. Search results are often of varying usefulness. When you search a web site for cameras that cost less than $100, this is an exact search. There is a set of cameras that matches this search, and a set that does not. Similarly, when you do a string search across news items for "mouse", there is only 1 expected result set. When you do a full-text search for all the news items that contain the token "mouse", you probably expect to find news items containing the token "mice", and possibly "rodents", or possibly "computers". Not all results are equal. Some results are more "mousey" than others. Because full-text search may be inexact, we have the notion of score or relevance. We generally expect to see the most relevant results at the top of the results list.

[Definition: Full-text queries are performed on tokens and phrases. Tokens and phrases are produced via tokenization.] Informally, tokenization breaks a character string into a sequence of tokens, units of punctuation, and spaces.

Tokenization, in general terms, is the process of converting a text string into smaller units that are used in query processing. Those units, called tokens, are the most basic text units that a full-text search can refer to. Full-text operators typically work on sequences of tokens found in the target text of a search. These tokens are characterized by integers that capture the relative position(s) of the token inside the string, the relative position(s) of the sentence containing the token, and the relative position(s) of the paragraph containing the token. The positions typically comprise a start and an end position.

Tokenization, including the definition of the term "tokens", SHOULD be implementation-defined. Implementations SHOULD expose the rules and sample results of tokenization as much as possible to enable users to predict and interpret the results of tokenization. Tokenization is defined more formally in 4.1 Tokenization.

[Definition: A token is a non-empty sequence of characters returned by a tokenizer as a basic unit to be searched. Beyond that, tokens are implementation-defined.] [Definition: A phrase is an ordered sequence of any number of tokens. Beyond that, phrases are implementation-defined.]

[Definition: A sentence is an ordered sequence of any number of tokens. Beyond that, sentences are implementation-defined. A tokenizer is not required to support sentences.]

[Definition: A paragraph is an ordered sequence of any number of tokens. Beyond that, paragraphs are implementation-defined. A tokenizer is not required to support paragraphs.]

Some XML elements represent semantic markup, e.g., <title>. Others represent formatting markup, e.g., <b> to indicate bold. Semantic markup serves well as token boundaries. Some formatting markup serves well as token boundaries, for example, paragraphs are most commonly delimited by formatting markup. Other formatting markup may not serve well as token boundaries. Implementations are free to provide implementation-defined ways to differentiate between the markup's effect on token boundaries during tokenization. In the absence of an implementation-defined way to differentiate, element markup (start tags, end tags, and empty-element tags) creates token boundaries.

A sample tokenization is used for the examples in this document. The results might be different for other tokenizations.

Tokenization enables functions and operators that operate on a part or the root of the token (e.g., wildcards, stemming).

Tokenization enables functions and operators which work with the relative positions of tokens (e.g., proximity operators).

This specification focuses on functionality that serves all languages. It also selectively includes functionalities useful within specific families of languages. For example, searching within sentences and paragraphs is useful to many western languages and to some non-western languages, so that functionality is incorporated into this specification.

Certain aspects of language processing are described in this specification as implementation-defined or implementation-dependent.

• [Definition: Implementation-defined indicates an aspect that may differ between implementations, but must be specified by the implementor for each particular implementation.]

• [Definition: Implementation-dependent indicates an aspect that may differ between implementations, is not specified by this or any W3C specification, and is not required to be specified by the implementor for any particular implementation.]

1.2 Organization of this document

This document is organized as follows. We first present a high level syntax for the XQuery and XPath Full Text 1.0 language along with some examples. Then, we present the syntax and examples of the basic primitives in the XQuery and XPath Full Text 1.0 language. This is followed by the semantics of the XQuery and XPath Full Text 1.0 language. The appendix contains a section that provides an EBNF for the XPath 2.0 Grammar with Full-Text Extensions, an EBNF for XQuery 1.0 Grammar with Full-Text Extensions, acknowledgements and a glossary.

1.3 A word about namespaces

Certain namespace prefixes are predeclared by XQuery 1.0 and, by implication, by this specification, and bound to fixed namespace URIs. These namespace prefixes are as follows:

• xml = http://www.w3.org/XML/1998/namespace

• xs = http://www.w3.org/2001/XMLSchema

• xsi = http://www.w3.org/2001/XMLSchema-instance

• fn = http://www.w3.org/2005/xpath-functions

• local = http://www.w3.org/2005/xquery-local-functions

In addition to the prefixes in the above list, this document uses the prefix err to represent the namespace URI http://www.w3.org/2005/xqt-errors, This namespace prefix is not predeclared and its use in this document is not normative. Error codes that are not defined in this document are defined in other XQuery 1.0 and XPath 2.0 specifications, particularly [XML Path Language (XPath) 2.0] and [XQuery 1.0 and XPath 2.0 Functions and Operators].

Finally, this document uses the prefix fts to represent a namespace containing a number of functions used in this document to describe the semantics of XQuery and XPath Full Text functions. There is no requirement that these functions be implemented, therefore no URI is associated with that prefix.

2.1 Processing Model

A full-text contains expression (2.2 Full-Text Contains Expression) is composed of several parts:

1. An XPath 2.0 or XQuery 1.0 expression (RangeExpr) that specifies the sequence of items to be searched. [Definition: Those items are called the search context.]

2. The full-text selection to be applied (3 Full-Text Selections). Full-text selections are, syntactically and semantically, fully composable and contain:

   • Required:

     • Tokens and phrases for which a search is performed (3.2 Search Tokens and Phrases).

   • Optional:

     • Match options, such as indicators for case sensitivity and stop words (3.4 Match Options);

     • Boolean full-text operators, that compose a full-text selection from simpler full-text selections (3.5 Logical Full-Text Operators);

     • Other full-text operators that are constraints on the positions of matches, such as indicators for distance between tokens and for the cardinality of matches (3.6 Positional Filters and 3.3 Cardinality Selection); and

     • The weighting information. Each individual search term in a full-text selection may be annotated with optional weight information. This information may be used during the evaluation of the full-text selections to calculate scoring, information that quantifies the relevance of the result to the given search criteria.

3. An optional XPath 2.0 or XQuery 1.0 expression (UnionExpr) that specifies the set of nodes, descendents of the RangeExp, whose contents must be ignored for the purpose of determining a match during the search (3.7 Ignore Option).

The results of the evaluation of the full-text selection operators are instances of the AllMatches model, which complements the XQuery Data Model (XDM) for processing full-text queries. An AllMatches instance describes all possible solutions to the full-text query for a given search context item. Each solution is described by a Match instance. A Match instance contains the tokens from the search context that must be included (described using StringInclude instances which model the positive terms) and the tokens from search context item that must be excluded (described using StringExclude instances which model the negative terms). Each negative or positive term is modeled as a tuple: the position of the query token or phrase in the full-text selection, and a TokenInfo structure that describes a set of tokens in the text string which match the query token or phrase.

Figure 1 provides a schematic overview of the XQuery and XPath Full Text processing steps that are discussed in detail below. Some of these steps are completely outside the domain of XQuery; in Figure 1, these are depicted outside the black line that represents the boundaries of the language. The diagram only shows the central pieces of the XQuery Processing Model (see Section 2.2 Processing Model^XQ), however zooms in on the Execution Engine where the processing of the full-text extensions takes place. The full-text processing steps are labeled as FTn within the diagram and are referenced within the text.

Like all XQuery expressions, an FTContainsExpr returns an XDM Instance (see Fig. 1). With the exception of FTWords, which consumes TokenInfos, all full-text selections are closed under the AllMatches data model, i.e., their input and output are AllMatches instances. Tokenization transforms an XDM instance into TokenInfos, which ultimately get converted into AllMatches instances by the evaluation of full-text selections. Thus, the evaluation of nested full-text and XQuery expressions instances moves back and forth between these two models.

The resulting AllMatches instance obtained by the evaluation of an FTContainsExpr is converted into a Boolean value before being returned to the enclosing XPath or XQuery operation as follows. If at least one member of the disjunction contains only positive terms then value returned is true. If all members of the disjunction contain negative terms the result is false.

Weighting information, in an implementation-dependent fashion, may be used when calculating the scoring information computed and made available by FTContainsExpr to the optional score construct.

Given the components of a given full-text contains expression, the evaluation algorithm will proceed according to the following steps, also referenced in the processing model diagram as steps FTn (see Fig. 1):

1. Evaluate the search context expression (resulting in the sequence of search context items), the ignore option, if any (resulting in the set of ignored nodes), and any other XQuery/XPath exprssions nested within the full-text contains expression. (FT1)

2. Tokenize the query string(s). (FT2.1)

3. For each search context item:

   1. Delete the ignored nodes from the search context item.

   2. Tokenize the result of the previous step. This produces a sequence of tokens. (FT2.2) Note that implementations may (as an optimization) perform tokenization as part of the External Processing that is described in the XQuery Processing Model, when an XML document is parsed into an Infoset/PSVI and ultimately into a XQuery Data Model instance.

   3. Evaluate the FTSelection against the tokens of the search context. (FT3, FT4)

4. Convert the topmost AllMatches instances into a Boolean value. (FT5)

   The additional scoring information (also part of FT5) that is produced by the evaluation of the full-text contains expression is implementation-dependent and is not specified in this document. The scoring information is made available at the same time the Boolean value is returned.

(A more detailed version of the above procedure appears in Section 4.3 FTContainsExpr.)

Section 3 Full-Text Selections describes the syntax and the informal semantics of full-text operators. Their formal semantics as well as the formal definition of the AllMatches data model are given in Section 4 Semantics.

2.2 Full-Text Contains Expression

[Definition: A full-text contains expression is a expression that evaluates a sequence of items against a full-text selection. ]

As a syntactic construct, a full-text contains expression (grammar symbol: FTContainsExpr) behaves like a comparison expression (see Section 3.5.2 General Comparisons^XQ). This grammar rule introduces FTContainsExpr.

A full-text contains expression may be used anywhere a ComparisonExpr may be used. The ftcontains operator has higher precedence than other comparison operators, so the results of ftcontains expressions may be compared without enclosing them in parentheses.

for $b in /books/book
where $b/title ftcontains ("dog" with stemming) ftand "cat" 
return $b/author

/books/book[title ftcontains ("dog" with stemming) ftand "cat"]/author

/books/book[price < 50 and title ftcontains ("train" with stemming)]

/books/book[title ftcontains "dog" ftand "cat" ne
            content ftcontains ("train" with stemming)]

2.3 Score Variables

Besides specifying a match of a full-text query as a Boolean condition, full-text query applications typically also have the ability to associate scores with the results. [Definition: The score of a full-text query result expresses its relevance to the search conditions.]

XQuery and XPath Full Text extends the languages of XQuery 1.0 and XPath 2.0 further by adding optional score variables to the for and let clauses of FLWOR expressions.

The production for the extended for clause in XQuery 1.0 follows.

In XPath 2.0, the SimpleForClause is extended similarly.

When a score variable is present in a for clause the evaluation of the expression following the in keyword not only needs to determine the result sequence of the expression, i.e., the sequence of items which are iteratively bound to the for variable. It must also determine in each iteration the relevance "score" value of the current item and bind the score variable to that value.

The semantics of scoring and how it relates to second-order functions is discussed in Section 4.4 Scoring.

In the following example book elements are determined that satisfy the condition [content ftcontains "web site" ftand "usability" and .//chapter/title ftcontains "testing"]. The scores assigned to the book elements are returned.

for $b score $s 
    in /books/book[content ftcontains "web site" ftand "usability" 
                   and .//chapter/title ftcontains "testing"]
return $s

The example above is also a legal example of the XPath 2.0 extension.

Scores are typically used to order results, as in the following, more complete example.

for $b score $s 
    in /books/book[content ftcontains "web site" ftand "usability"]
where $s > 0.5
order by $s descending
return <result>  
          <title> {$b//title} </title> 
          <score> {$s} </score> 
       </result>

Note that the score variable gets one score value for each item in the value of the expression after the in keyword, regardless of the number of FTContainsExprs in that expression. In the following example, two separate full-text contains expressions are used to select the matching paragraphs. There is still just one score for each para returned. The highest scoring paragraphs will be returned first:

for $p score $s in //book[title ftcontains "software"]/para[. ftcontains "usability"]
     order by $s descending
  return $p

The following more elaborate example uses multiple score variables to return the matching paragraphs ordered so that those from the highest scoring books precede those from the lowest scoring books, where the highest scoring paragraphs of each book are returned before the lower scoring paragraphs of that book:

for $b score $score1 in //book[title ftcontains "software"]
    order by $score1 descending
return
    for $p score $score2 in $b/para[. ftcontains "usability"]
       order by $score2 descending
    return $p

The score variable is bound to a value which reflects the relevance of the match criteria in the full-text selections to the items returned by the respective RangeExprs. The calculation of relevance is implementation-dependent, but score evaluation must follow these rules:

1. Score values are of type xs:double in the range [0, 1].

2. For score values greater than 0, a higher score must imply a higher degree of relevance

Similarly to their use in a for clause, score variables may be specified in a let clause. A score variable in a let clause is also bound to the score of the expression evaluation, but in the let clause one score is determined for the complete result.

The production for the extended let clause follows.

When using the score option in a for clause the expression following the in keyword has the dual purpose of filtering, i.e., driving the iteration, and determining the scores. It is possible to separately specify expressions for filtering and scoring by combining a simple for clause with a let clause that uses scoring. The following is an example of this.

for $b in /books/book[.//chapter/title ftcontains "testing"]
let score $s := $b/content ftcontains "web site" ftand "usability" 
order by $s descending
return <result score="{$s}">{$b}</result>

This example returns book elements with chapter titles that contain "testing". Along with the book elements scores are returned. These scores, however, reflect whether the book content contains "web site" and "usability".

Note that it is not a requirement of the score of an FTContainsExpr to be 0, if the expression evaluates to false, nor to be non-zero, if the expression evaluates to true. Hence, in the example above it is not possible to infer the Boolean value of the FTContainsExpr in the let clause from the calculated score of a returned result element. For instance, an implementation may want to assign a non-zero score to a book that contained "web site", but not "usability", as this may be considered more relevant than a book that does not contain "web site" or "usability".

The expression ExprSingle associated with the score variable is passed to the scoring algorithm. The scoring algorithm calculates the score value based on the passed expression (not on the value returned by evaluating the expression). The set of expressions supported by the scoring algorithm is implementation-defined. If an expression not supported by the scoring algorithm is passed to the scoring algorithm, the result is implementation-defined.

The use of score variables introduces a second-order aspect to the evaluation of expressions which cannot be emulated by (first-order) XQuery functions. Consider the following replacement of the clause let score $s := FTContainsExpr

let $s := score(FTContainsExpr)

where a function score is applied to some FTContainsExpr. If the function score were first-order, it would only be applied to the result of the evaluation of its argument, which is one of the Boolean constants true or false. Hence, there would be at most two possible values such a score function would be able to return and no further differentiation would be possible.

for $b in /books/book
let score $s := $b/content ftcontains ("web site" weight 0.5)
                                ftand ("usability" weight 2)
return <result score="{$s}">{$b}</result>

2.4 Extensions to the Static Context

The XQuery Static Context is extended with a component for each full-text match option group. The settings of these components can be changed by using the following declaration syntax in the Prolog.

Match options modify the match semantics of full-text expressions. They are described in detail in Section 3.4 Match Options. When a match option is specified explicitly in a full-text expression, it overrides the setting of the respective component in the static context.

3.1 Primary Full-Text Selections

[Definition: A primary full-text selection is the basic form of a full-text selection. It specifies tokens and phrases as search conditions (FTWords), optionally followed by a cardinality constraint (FTTimes). An FTSelection in parentheses and the FTExtensionSelection are also a primary full-text selections.]

3.2 Search Tokens and Phrases

FTWords finds matches that contain the specified tokens and phrases.

FTWords consists of two parts: a mandatory FTWordsValue part and an optional FTAnyallOption part. FTWordsValue specifies the tokens and phrases that must be contained in the matches. FTAnyallOption specifies how containment is checked.

In general, the tokens and phrases in FTWordsValue are specified using a nested XQuery expression. To simplify notation, the enclosing braces may be omitted if FTWordsValue consists of a single literal.

The following rules specify how an FTWordsValue matches tokens and phrases. First, the FTWordsValue is converted to a sequence of strings as though it were an argument to a function with the expected type of xs:string*. Then, each of those strings is tokenized into a sequence of tokens as described in Section 4.1 Tokenization. Then, FTAnyallOption is checked.

If FTAnyallOption is "any", the sequence of tokens for each string is considered as a phrase, i.e. a match is found in the tokenized form of the text being searched, whenever that form contains a subsequence of tokens that corresponds to the sequence of query tokens in an implementation-defined way and that subsequence of tokens covers consecutive token positions in the tokenized text. If the value of the FTWordsValue contains more than one string, the different strings are considered to be alternatives, i.e. the resulting matches must contain at least one of the generated phrases.

If FTAnyallOption is "all", the sequence of tokens for each string is considered as a phrase. The resulting matches must contain all of the generated phrases.

If FTAnyallOption is "phrase", the tokens from all the strings are concatenated in a single sequence, which is considered as a phrase. The resulting matches must contain the generated phrase.

If FTAnyallOption is "any word", the tokens from all the strings are combined into a single set. The resulting matches must contain at least one of the tokens in the set.

If FTAnyallOption is "all words", the tokens from all the strings are combined into a single set. The resulting matches must contain all of the tokens in the set.

If the FTWordsValue evaluates to a single string, the use of "any", "all", and "phrase" in FTAnyallOption produces the same results.

If FTAnyallOptions is omitted, "any" is the default.

The following expression returns the sample book element, because its title element contains the token "Expert":

//book[./title ftcontains "Expert"]

The following expression returns the sample book element, because its title element contains the phrase "Expert Reviews":

//book[./title ftcontains "Expert Reviews"]

The following expression returns the sample book element, because its title element contains the two tokens "Expert" and "Reviews":

//book[./title ftcontains {"Expert", "Reviews"} all]

The following expression returns false for our sample document, because the p element doesn't contain the phrase "Web Site Usability" although it contains all of the tokens in the phrase:

//book//p ftcontains "Web Site Usability"

The following expression returns book numbers of book elements by "Marigold" with a title about "Web Site Usability", sorting them in descending score order:

for $book in /books/book[.//author ftcontains "Marigold"] 
let score $score := $book/title ftcontains "Web Site Usability" 
where $score > 0.8 
order by $score descending
return $book/@number

3.3 Cardinality Selection

[Definition: A cardinality selection consist of an FTWords followed by the FTTimes postfix operator.] A cardinality selection selects matches for which the operand FTWords is matched a specified number of times.

A cardinality selection limits the number of different matches of FTWords within the specified range. The semantics of FTRange are described in 3.6.3 Distance Selection.

In the document fragment "very very big":

1. The FTWords "very big" has 1 match consisting of the second "very" and "big".

2. The FTWords {"very", "big"} all has 2 matches; one consisting of the first "very" and "big", and the other containing the second "very" and "big".

3. The FTWords {"very", "big"} any has 3 matches.

The following expression returns the example book element's number, because the book element contains 2 or more occurrences of "usability":

//book[. ftcontains "usability" occurs at least 2 times]/@number

The following expression returns the empty sequence, because there are 3 occurrences of {"usability", "testing"} any in the designated title:

//book[@number="1" and title ftcontains {"usability", 
"testing"} any occurs at most 2 times] 

3.4 Match Options

Full-text match options modify the matching behaviour of the primary full-text selection to which they are applied.

[Definition: Match options modify the set of tokens in the query, or how they are matched against tokens in the text.]

[Definition: Each of the seven alternatives of production FTMatchOption corresponds to one match option group. ] The match options from any given group are mutually exclusive, i.e., only one of these settings can be in effect, whereas match options of different groups can be combined freely.

Note that, along with the syntax rules above, there is an extra-grammatical constraint, multiple-match-options , which needs to be considered, if multiple match options are specified. It states that within a single FTMatchOptions at most one match option of any given match option group may be specified. For example, if the FTCaseOption "lowercase" is specified, then "uppercase" cannot also be specified as part of the same FTMatchOptions.

Although match options only take effect in the application of FTWords, the syntax also allows to specify match options that modify the non-primitive full-text selection "(" FTSelection ")". Such a higher-level match option provides a default for the respective match option group for any embedded FTPrimary, just as match option declarations in the Prolog provide default match options for the whole query.

Match options are propagated through the query via the static context. For each of the seven match option groups, the static context has a component that contains one option from that group. The seven settings are initialized by the implementation in accordance with the table in Appendix C Static Context Components, and are modified by any FTOptionDecls in the Prolog. The resulting settings are then propagated unchanged to every FTContainsExpr in the module (including those in VarDecls and FunctionDecls, and including any that happen to be nested within another FTContainsExpr). At any given FTContainsExpr, the settings from the static context are copied to the FTContainsExpr's inner settings, which are then propagated down the syntax tree. At each FTPrimaryWithOptions, the locally specified match options (if any) overwrite the corresponding inner setting(s). At each FTWords, the inner settings are used as the effective match options for tokenizing the query strings and matching them against the tokens in the text. (These inner settings could be seen as a parallel set of components in the static context, but Section 4 Semantics models them as structures that get passed as parameters to various semantic functions.)

Thus, when a match option appears in an FTSelection, it applies to the associated FTPrimary, but not to any FTContainsExprs that happen to be embedded within that FTPrimary. Instead, for a nested FTContainsExpr, the default match options are those declared in the Prolog or, if not declared in the Prolog, then supplied by the implementation's initial values.

[Definition: The order in which effective match options for an FTWords are applied is called the match option application order.] This order is significant because match options are not always commutative. For example, synonym(stem(word)) is not always the same as stem(synonym(word)).

The match option application order is subject to some constraints:

1. The Language Option must be applied first

2. The Stemming Option must be applied before the Case Option and the Diacritics Option

Aside from these constraints, the full order of the application of match options is implementation-defined.

More information on their semantics is given in 4.2.6 Match Options Semantics.

If no match options declarations are present in the prolog and the implementation does not define any overwriting of the static context components for the match options, the query:

/books/book/title ftcontains "usability" 

is, assuming "de" is the implementation-defined default language, equivalent to the query:

/books/book/title ftcontains "usability" 
    language "de"
    without wildcards
    without thesaurus
    without stemming
    case insensitive 
    diacritics insensitive 
    without stop words

We describe each match option group in more detail in the following sections.

//book[@number="1"]//editor ftcontains "salon de the"
with default stop words language "fr"

//book[@number="1"]/title ftcontains "improv.*" with wildcards

//book[@number="1"]/title ftcontains ".?site" with wildcards

//book[@number="1"]/p ftcontains "w.ll" with wildcards

//book[@number="1"]/p ftcontains "w.ll" without wildcards

count(.//book/content ftcontains "duties" with
thesaurus at "http://bstore1.example.com/UsabilityThesaurus.xml"
relationship "UF")>0

doc("http://bstore1.example.com/full-text.xml")
/books/book[count(./content ftcontains "web site components" with
thesaurus at "http://bstore1.example.com/UsabilityThesaurus.xml"
relationship "NT" at most 2 levels)>0]

doc("http://bstore1.example.com/full-text.xml")
/books/book[count(. ftcontains "Merrygould" with thesaurus at
"http://bstore1.example.com/UsabilitySoundex.xml" relationship
"sounds like")>0]

/books/book[@number="1"]/title ftcontains "improve" with stemming 

//book[@number="1"]/title ftcontains "Usability" lowercase 

//book[@number="1"]/title ftcontains "usability" case insensitive

//book[@number="1"]//editor ftcontains "Vera" diacritics insensitive

//book[@number="1"]/editors ftcontains "Vera" diacritics sensitive

/books/book[@number="1"]//p ftcontains "propagation of errors"
with stemming with stop words ("a", "the", "of") 

/books/book[@number="1"]//p ftcontains "propagating errors" 
with stop words ("few")

/books/book[@number="1"]//p ftcontains "propagation of errors" 
with stemming without stop words

doc("http://bstore1.example.com/full-text.xml")
/books/book[count(.//content ftcontains "planning then 
conducting" with stop words at 
"http://bstore1.example.com/StopWordList.xml")>0]

doc("http://bstore1.example.com/full-text.xml")
/books/book[count(.//content ftcontains "planning then conducting"
with stop words at "http://bstore1.example.com/StopWordList.xml"
except ("the", "then"))>0]

declare namespace exq = "http://example.org/XQueryImplementation";

declare ft-option option exq:diacritics "short-vowel insensitive"

declare namespace exq = "http://example.org/XQueryImplementation";

//para[. ftcontains
         ("Kinder" ftand "Platz" distance exactly 1 words)
         with stemming
         option exq:compounds "distance=1" ]

3.5 Logical Full-Text Operators

Full-text selections can be combined with the logical connectives ftor (full-text or), ftand (full-text and), not in (mild not), and ftnot (unary full-text not).

//book[.//author ftcontains "Millicent" ftor "Voltaire"]

//book[@number="1"]/title ftcontains ("usability" ftand "testing")

//book/author ftcontains "Millicent" ftand "Montana"

/books/book ftcontains "usability" not in "usability testing"

//book[. ftcontains ftnot "usability"]

//book ftcontains "information" ftand
"retrieval" ftand ftnot "information retrieval"

//book[. ftcontains "web site usability" ftand 
ftnot "usability testing"]

3.6 Positional Filters

[Definition: Positional filters are postfix operators that serve to filter matches based on various constraints on their positional information.]

Recall that the grammar rule for FTSelection allows an arbitrary number of positional filters to follow an FTOr. Multiple adjacent positional filters are applied from left to right, i.e., the first filter is applied to the result of the FTOr, the second is applied to the result of that first application, and so on.

//book/title ftcontains ("web site" ftand "usability") ordered

//book[@number="1"] ftcontains ("Montana" ftand "Millicent") ordered