XQuery 1.0 and XPath 2.0 Full-Text 1.0

1.1 Full-Text Search and XML

As XML becomes mainstream, users expect to be able to search their XML documents. This requires a standard way to do full-text search, as well as structured searches, against XML documents. A similar requirement for full-text search led ISO to define the SQL/MM-FT [SQL/MM] standard. SQL/MM-FT defines extensions to SQL to express full-text searches providing functionality similar to that defined in this full-text language extension to XQuery 1.0 and XPath 2.0.

XML documents may contain highly structured data (fixed schemas, known types such as numbers, dates), semi-structured data (flexible schemas and types), markup data (text with embedded tags), and unstructured data (untagged free-flowing text). Where a document contains unstructured or semi-structured data, it is important to be able to search using Information Retrieval techniques such as scoring and weighting.

Full-text search is different from substring search in many ways:

1. A full-text search searches for tokens and phrases rather than substrings. A substring search for news items that contain the string "lease" will return a news item that contains "Foobar Corporation releases the 20.9 version ...". A full-text search for the token "lease" will not.

2. There is an expectation that a full-text search will support language-based searches which substring search cannot. An example of a language-based search is "find me all the news items that contain a token with the same linguistic stem as "mouse" (finds "mouse" and "mice"). Another example based on token proximity is "find me all the news items that contain the tokens "XML" and "Query" allowing up to 3 intervening words.

3. Full-text search must address the vagaries and nuances of language. Search results are often of varying usefulness. When you search a web site for cameras that cost less than $100, this is an exact search. There is a set of cameras that matches this search, and a set that does not. Similarly, when you do a string search across news items for "mouse", there is only 1 expected result set. When you do a full-text search for all the news items that contain the token "mouse", you probably expect to find news items containing the token "mice", and possibly "rodents", or possibly "computers". Not all results are equal. Some results are more "mousey" than others. Because full-text search may be inexact, we have the notion of score or relevance. We generally expect to see the most relevant results at the top of the results list.

   As XQuery and XPath evolve, they may apply the notion of score to querying structured data. For example, when making travel plans or shopping for cameras, it is sometimes useful to get an ordered list of near matches in addition to exact matches. If XQuery and XPath define a generalized inexact match, we expect XQuery and XPath to utilize the scoring framework provided by XQuery and XPath Full-Text.

The following definitions apply to full-text search:

1. [Definition: Full-text queries are performed on tokens and phrases. Tokens and phrases are produced via tokenization.] Informally, tokenization breaks a character string into a sequence of words, units of punctuation, and spaces.

2. [Definition: A token is defined as a character, n-gram, or sequence of characters returned by a tokenizer as a basic unit to be searched. Each instance of a token consists of one or more consecutive characters. Beyond that, tokens are implementation-defined.] Note that consecutive tokens need not be separated by either punctuation or space, and tokens may overlap. [Definition: A phrase is an ordered sequence of any number of tokens. Beyond that, phrases are implementation-defined.]

   Note:

   In some natural languages, tokens and words can be used interchangeably.

3. Tokenization enables functions and operators that operate on a part or the root of the token (e.g., wildcards, stemming).

   Tokenization enables functions and operators which work with the relative positions of tokens (e.g., proximity operators).

   Tokenization also uniquely identifies sentences and paragraphs in which tokens appear. [Definition: A sentence is an ordered sequence of any number of tokens. Beyond that, sentences are implementation-defined. A tokenizer is not required to support sentences.] [Definition: A paragraph is an ordered sequence of any number of tokens. Beyond that, paragraphs are implementation-defined. A tokenizer is not required to support paragraphs.] Whatever a tokenizer for a particular language chooses to do, it must preserve the containment hierarchy: paragraphs contain sentences, which contain tokens.

   The tokenizer has to process two codepoint equal strings in the same way, i.e., it should identify the same tokens. Everything else about the behavior of the tokenizer is implementation-defined.

4. This specification focuses on functionality that serves all languages. It also selectively includes functionalities useful within specific families of languages. For example, searching within sentences and paragraphs is useful to many western languages and to some non-western languages, so that functionality is incorporated into this specification.

5. Some XML elements represent semantic markup, e.g., <title>. Others represent formatting markup, e.g., <b> to indicate bold. Semantic markup serves well as token boundaries. Some formatting markup serves well as token boundaries, for example, paragraphs are most commonly delimited by formatting markup. Other formatting markup may not serve well as token boundaries. Implementations are free to provide implementation-defined ways to differentiate between the markup's effect on token boundaries during tokenization.

Certain aspects of language processing are described in this specification as implementation-defined or implementation-dependent.

• [Definition: Implementation-defined indicates an aspect that may differ between implementations, but must be specified by the implementor for each particular implementation.]

• [Definition: Implementation-dependent indicates an aspect that may differ between implementations, is not specified by this or any W3C specification, and is not required to be specified by the implementor for any particular implementation.]

1.2 Organization of this document

This document is organized as follows. We first present a high level syntax for the XQuery 1.0 and XPath 2.0 Full-Text 1.0 language along with some examples. Then, we present the syntax and examples of the basic primitives in the XQuery 1.0 and XPath 2.0 Full-Text 1.0 language. This is followed by the semantics of the XQuery 1.0 and XPath 2.0 Full-Text 1.0 language. The appendix contains a section that provides an EBNF for the XPath 2.0 Grammar with Full-Text extensions, an EBNF for XQuery 1.0 Grammar with Full-Text extensions, acknowledgements and a glossary.

1.3 A word about namespaces

Certain namespace prefixes are predeclared by XQuery 1.0 and, by implication, by this specification, and bound to fixed namespace URIs. These namespace prefixes are as follows:

• xml = http://www.w3.org/XML/1998/namespace

• xs = http://www.w3.org/2001/XMLSchema

• xsi = http://www.w3.org/2001/XMLSchema-instance

• fn = http://www.w3.org/2005/xpath-functions

• local = http://www.w3.org/2005/xquery-local-functions

In addition to the prefixes in the above list, this document uses the prefix err to represent the namespace URI http://www.w3.org/2005/xqt-errors, This namespace prefix is not predeclared and its use in this document is not normative. Error codes that are not defined in this document are defined in other XQuery 1.0 and XPath 2.0 specifications, particularly [XML Path Language (XPath) 2.0] and [XQuery 1.0 and XPath 2.0 Functions and Operators].

Finally, this document uses the prefix fts to represent a namespace containing a number of functions used in this document to describe the semantics of XQuery 1.0 and XPath 2.0 Full-Text functions. There is no requirement that these functions be implemented, therefore no URI is associated with that prefix.

2.1 Processing Model

As part of the External Processing that is described in the XQuery Processing Model, when an XML document is parsed into an Infoset/PSVI and ultimately into a XQuery Data Model instance, a full-text process called tokenization is usually executed.

Tokenization, in general terms, is the process of converting a text string into smaller units that are used in query processing. Those units, called tokens, are the most basic text units that a full-text search can refer to. Full-text operators typically work on sequences of token occurrences found in the target text (nodes) of a search. These token occurrences are characterized by unique identifiers that capture the relative position of the token inside the string, the relative position of the sentence containing the token, and the relative position of the paragraph containing the token.

Tokenization, including the definition of the term "words", SHOULD be implementation-defined. Implementations SHOULD expose the rules and sample results of tokenization as much as possible to enable users to predict and interprete the results of tokenization. Tokenization MUST only conform to these constraints:

1. Each word MUST consist of one or more consecutive characters;

2. The tokenizer MUST preserve the containment hierarchy (e.g., paragraphs contain sentences, which contain words); and

3. The tokenizer MUST, when tokenizing two equal strings, identify the same tokens in each.

A sample tokenization is used for the examples in this document. The results might be different for other tokenizations.

A full-text contains expression (2.2 Full-text Contains Expression), evaluated within the normal Query Processing (XQuery Processing Model), is composed of several parts:

1. An XPath 2.0 or XQuery 1.0 expression (RangeExpr) that specifies the sequence of items to be searched. [Definition: Those items are called the search context.]

2. The full-text selection to be applied (3 Full-Text Selections). Full-text selections are, syntactically and semantically, fully composable and contain:

   • Required:

     • Words and phrases for which a search is performed (3.2 Search Tokens and Phrases).

   • Optional:

     • Match options, such as indicators for case sensitivity and stop words (3.3 Match Options);

     • Boolean full-text operators, that compose a full-text selection from simpler full-text selections (3.4 Logical Full-Text Operators);

     • Other full-text operators that are constraints on the positions of matches, such as indicators for distance between tokens and for the cardinality of matches (3.5 Positional Filters and 3.6 Cardinality Selection); and

     • The weighting information. Each individual search term in a full-text selection may be annotated with optional weight information. This information may be used during the evaluation of the full-text selections to calculate scoring, information that quantifies the relevance of the result to the given search criteria.

3. An optional XPath 2.0 or XQuery 1.0 expression (UnionExpr) that specifies the set of nodes, descendents of the RangeExp, which contents may be ignored for the purpose of determining a match during the search (3.7 Ignore Option).

The results of the evaluation of the full-text selection operators are instances of the AllMatches model, which complements the XQuery Data Model (XDM) for processing full-text queries. An AllMatches instance describes all possible solutions to the full-text query for a given search context item. Each solution is described by a Match instance. A Match instance contains the tokens from the search context that must be included (described using StringInclude instances which model the positive terms) and the tokens from search context item that must be excluded (described using StringExclude instances which model the negative terms). Each negative or positive term is modeled as a tuple: the position of the query word or phrase in the full-text selection, and a TokenInfo structure that describes a consecutive sequence of token occurrences in the text string which match the query word or phrase.

Figure 1 provides a schematic overview of the XQuery 1.0 and XPath 2.0 Full-Text processing steps that are discussed in detail below. Some of these steps are completely outside the domain of XQuery; in Figure 1, these are depicted outside the black line that represents the boundaries of the language. The diagram only shows the central pieces of the XQuery Processing Model (see Section 2.2 Processing Model^XQ), however zooms in on the Execution Engine where the processing of the Full-Text extensions takes place. The full-text processing steps are labeled as FTn within the diagram and are referenced within the text.

Like all XQuery expressions, an FTContainsExpr returns an XDM Instance (see Fig. 1). With the exception of FTWords, which consumes TokenInfos, all full-text selections are closed under the AllMatches data model, i.e., their input and output are AllMatches instances. Tokenization normally occurs at the time of parsing of the original XML documents, for example, during the Data Model Generation process (see Figure 1). But here it may also occur "on-the-fly" transforming an XDM instance into TokenInfos, which ultimately get converted into AllMatches instances by the evaluation of full-text selections. Thus, the evaluation of nested full-text and XQuery expressions instances moves back and forth between these two models.

The resulting AllMatches instance obtained by the evaluation of a Full Text expression is converted into a Boolean value before being returned to the enclosing XPath or XQuery operation as follows. If at least one member of the disjunction contains only positive terms then value returned is true. If all members of the disjunction contain negative terms the result is false.

Weighting information, in an implementation-dependent fashion, may be used when calculating the scoring information computed and made available by FTContainsExpr to the optional score construct.

Given the components of a given Full Text expression, the evaluation algorithm will proceed according to the following steps, also referenced in the processing model diagram as steps FTn (see Fig. 1):

1. Evaluate the search context expression, resulting in the set of search context items; (FT1 provides the evaluation of any XPath 2.0 or XQuery 1.0 expressions that generates or modifies the search context, as well as the query string(s) in a partially evaluated full-text selection)

2. Evaluate the (optional) ignore expression, resulting in the set of ignored nodes and virtually delete the ignore nodes from the search context nodes tree. (Included in FT1)

3. Apply the tokenization algorithm to query string(s).

4. For each search context item:

   1. Apply the tokenization algorithm in order to extract potentially matching terms together with their positional information. This step results in a sequence of token occurrences.

   2. Evaluate the simple "FTWord" operators in the full-text selection against the tokenized input. This results in a set of AllMatches instances. (FT3)

   3. Evaluate the rest of the full-text selection operator tree in a bottom up fashion. At each step the AllMatches instance produced by the previous steps are given as input, and a new instance of the AllMatches is obtained as output. At each step the FTMatchOptions are controlling the semantics of the application of the FTWords operator. (FT4)

5. Convert the AllMatches instance into a Boolean value. (FT5)

   The additional scoring information (also part of FT5) that is produced by the evaluation of the Full Text expression is implementation-dependent and is not specified in this document. The scoring information is made available at the same time the Boolean value is returned.

Section 3 Full-Text Selections describes the syntax and the informal semantics of Full Text operators. Their formal semantics as well as the formal definition of the AllMatches data model are given in Section 4 Semantics.

2.2 Full-text Contains Expression

[Definition: A full-text contains expression is a expression that evaluates a sequence of nodes against a full-text selection. ]

As a syntactic construct a full-text contains expression (grammar symbol: FTContainsExpr) behaves like a comparison expression (see Section 3.5.2 General Comparisons^XQ). This grammar rule introduces FTContainsExpr.

A full-text contains expression may be used anywhere a ComparisonExpr may be used. The ftcontains operator has higher precedence than other comparison operators, so the results of ftcontains expressions may be compared without enclosing them in parentheses.

for $b in /books/book
where $b/title ftcontains ("dog" with stemming) ftand "cat" 
return $b/author

/books/book[title ftcontains ("dog" with stemming) ftand "cat"]/author

/books/book[title ftcontains "dog" ftand "cat" ne
            content ftcontains ("train" with stemming)]

2.3 Score Variables

Besides specifying a match of a full-text search as a Boolean condition, full-text search applications typically also have the ability to associate scores with the results. [Definition: Scores express the relevance of those results to the full-text search conditions.]

XQuery 1.0 and XPath 2.0 Full-Text extends the languages of XQuery 1.0 and XPath 2.0 further by adding optional score variables to the for and let clauses of FLWOR expressions.

The production for the extended for clause follows.

When a score variable is present in a for clause the evaluation of the expression following the in keyword not only needs to determine the result sequence of the expression, i.e., the sequence of items which are iteratively bound to the for variable. It must also determine in each iteration the relevance "score" value of the current item and bind the score variable to that value.

The semantics of scoring and how it relates to second-order functions is discussed in Section 4.3.2 Scoring.

In the following example book elements are determined that satisfy the condition [content ftcontains "web site" ftand "usability" and .//chapter/title ftcontains "testing"]. The scores assigned to the book elements are returned.

for $b score $s 
    in /books/book[content ftcontains "web site" ftand "usability" 
                   and .//chapter/title ftcontains "testing"]
return $s

XPath 2.0 Full-Text extends the language of XPath 2.0 in the for expression in the same way: with optional score variables. The example above is also a legal example of the XPath 2.0 extension.

Scores are typically used to order results, as in the following, more complete example.

for $b score $s 
    in /books/book[content ftcontains "web site" ftand "usability"]
where $s > 0.5
order by $s descending
return <result>  
          <title> {$b//title} </title> 
          <score> {$s} </score> 
       </result>

Note that the score applies to the entire for expression. In the following example, two separate full-text contains expressions are used to select the matching paragraphs. There is still just one score for each para returned. The highest scoring paragraphs will be returned first:

for $p score $s in //book[title ftcontains "software"]/para[. ftcontains "usability"]
     order by $s descending
  return $p

The following more elaborate example uses multiple score variables to return the matching paragraphs ordered so that those from the highest scoring books precede those from the lowest scoring books, where the highest scoring paragraphs of each book are returned before the lower scoring paragraphs of that book:

for $b score $score1 in //book[title ftcontains "software"]
    order by $score1 descending
return
    for $p score $score2 in $b/para[. ftcontains "usability"]
       order by $score2 descending
    return $p

The score variable is bound to a value which reflects the relevance of the match criteria in the full-text selections to the nodes in the respective RangeExprs. The calculation of relevance is implementation-dependent, but score evaluation must follow these rules:

1. Score values are of type xs:double in the range [0, 1].

2. For score values greater than 0, a higher score must imply a higher degree of relevance

Similarly to their use in a for clause, score variables may be specified in a let clause. A score variable in a let clause is also bound to the score of the expression evaluation, but in the let clause one score is determined for the complete result. The let variable may be dropped from the let clause, if the score variable is present.

The production for the extended let clause follows.

While when using the score option in a for clause the expression following the in keyword has the dual purpose of filtering, i.e., driving the iteration, and determining the scores, it is possible to separately specify expressions for filtering and scoring by combining a simple for clause with a let clause that uses scoring. The following is an example of this.

for $b in /books/book[.//chapter/title ftcontains "testing"]
let score $s := $b/content ftcontains "web site" ftand "usability" 
order by $s descending
return <result score="{$s}">{$b}</result>

This example returns book elements with chapter titles that contain "testing". Along with the book elements scores are returned. These scores, however, reflect whether the book content contains "web site" and "usability".

Note that it is not a requirement of the score of an FTContainsExpr to be 0, if the expression evaluates to false, nor to be non-zero, if the expression evaluates to true. Hence, in the example above it is not possible to infer the Boolean value of the FTContainsExpr in the let clause from the calculated score of a returned result element. For instance, an implementation may want to assign a non-zero score to a book that contained only "web site", but not "usability", as this may be considered more relevant than a book that does not contain either of both.

The expression ExprSingle assigned to the score variable is passed to the scoring algorithm and is not evaluated directly. The scoring algorithm calculates the score value based on the passed expression (not on the value returned by evaluating the expression). The set of supported expressions is implementation-defined.

The use of score variables introduces a second-order aspect to the evaluation of expressions which cannot be emulated by (first-order) XQuery functions. Consider the following replacement of the clause let score $s := FTContainsExpr

let $s := score(FTContainsExpr)

where a function score is applied to some FTContainsExpr. If the function score were first-order, it would only be applied to the result of the evaluation of its argument, which is one of the Boolean constants true or false. Hence, there would be at most two possible values such a score function would be able to return and no further differentiation would be possible.

for $b in /books/book
let score $s := $b/content ftcontains ("web site" weight 0.5)
                                ftand ("usability" weight 2)
return <result score="{$s}">{$b}</result>

2.4 Extensions to the Static Context

The XQuery Static Context is extended by a component for each of the full-text match options. Thus, the default of a match option in a query may be changed by providing a setting in the static context using the following declaration syntax.

Match options modify the match semantics of full-text expressions. They are described in detail in Section 3.3 Match Options. When a match option is specified explicitly in a query, that setting overrides the setting of the respective match option in the static context.

3.1 Primary Full-Text Selections

[Definition: A primary full-text selection is the basic form of a full-text selection. It specifies words and phrases as search conditions (FTWords), optionally followed by a cardinality constraint (FTTimes). An FTSelection in parentheses is also a primary full-text selection.]

3.2 Search Tokens and Phrases

FTWords finds matches that contain the specified tokens and phrases.

FTWords consists of two parts: a mandatory FTWordsValue part and an optional FTAnyallOption part. FTWordsValue specifies the tokens and phrases that must be contained in the matches. FTAnyallOption specifies how containment is checked.

The FTWordsValue is converted as though it were an argument to a function with the expected type of "xs:string*".

In general, the tokens and phrases in FTWordsValue are specified using a nested XQuery expression. To simplify notation, the enclosing braces may be omitted if FTWordsValue consists of a single literal.

The following rules specify how the containment of the strings from the FTWordsValue sequence is checked. First, every string is tokenized into a sequence of tokens as described in Section 4.1 Tokenization. Then, FTAnyallOption is checked.

If FTAnyallOption is "any", the sequence of tokens for every string is considered as a phrase, i.e. the tokens must occur consecutively in the text in the specified order. If the sequence contains more than one string, the different strings are considered to be alternatives, i.e. the resulting matches must contain at least one of the generated phrases.

If FTAnyallOption is "all", the sequence of tokens for every string is considered as a phrase. The resulting matches must contain all of the generated phrases.

If FTAnyallOption is "phrase", the tokens from all the strings are concatenated in a single sequence, which is considered as a phrase. The resulting matches must contain the generated phrase.

If FTAnyallOption is "any word", the tokens from all the strings are combined into a single set. The resulting matches must contain at least one of the tokens in the set.

If FTAnyallOption is "all words", the tokens from all the strings are combined into a single set. The resulting matches must contain all of the tokens in the set.

If the FTWordsValue evaluates to a single string, the use of "any", "all", and "phrase" in FTAnyallOption produces the same results.

If FTAnyallOptions is omitted, "any" is the default.

The following expression returns the book element whose number is 1, because its title element contains the token "Expert":

/book[@number="1" and ./title ftcontains "Expert"]

The following expression returns the book element whose number is 1, because its title element contains the phrase "Expert Reviews":

/book[@number="1" and ./title ftcontains "Expert Reviews"]

The following expression returns the book element whose number is 1, because its title element contains two tokens "Expert" and "Reviews":

/book[@number="1" and ./title ftcontains {"Expert",
"Reviews"} all]

The following expression returns false, because the p element doesn't contain the phrase "Web Site Usability" although it contains all of the tokens in the phrase:

/book[@number="1"]//p ftcontains "Web Site Usability"

The following expression returns book numbers of book elements by "Marigold" with a title about "Web Site Usability", sorting them in descending score order:

for $book in /book[.//author ftcontains "Marigold"] 
let score $score := $book/title ftcontains "Web Site Usability" 
where $score > 0.8 
order by $score descending
return $book/@number

3.3 Match Options

Full-text match options modify the matching behaviour of the primary full-text selection to which they are applied.

[Definition: Match options modify the set of tokens in the query, or how they are matched against tokens in the text.]

[Definition: Each of the seven alternatives of production FTMatchOption corresponds to one match option group. ] The match options from any given group are mutually exclusive, i.e., only one of these settings can be in effect, whereas match options of different groups can be combined freely.

Note that, along with the syntax rules above, there is an extra-grammatical constraint, multiple-match-options , which needs to be considered, if multiple match options are specified. It states that within a single FTMatchOptions at most one match option of any given match option group may be specified. For example, if the FTCaseOption "lowercase" is specified, then "uppercase" cannot also be specified as part of the same FTMatchOptions.

Although match options only take effect in the application of FTWords, the syntax also allows to specify match options that modify the non-primitive full-text selection "(" FTSelection ")". Such a higher-level match option provides a default for the respective match option group for any embedded FTPrimary, just as the static context components corresponding to the match option groups provide default match options for the whole query. Details about these context components, including their default values, are given in Appendix C Static Context Components.

In other words, there is a tuple of seven effective match options, one from each group, which are propagated from top to bottom in the query syntax tree. For the top-level query the seven values are given by the static context and at each FTPrimary the locally (like postfix operators) specified match options may override these propagated values. Thus, any occurrence of an FTWords in a query is associated with seven effective match options, one from each group, that influence its matching.

The order in which effective match options for an FTWords are applied is subject to some constraints:

1. The Language Option must be applied first

2. The Stemming Option must be applied before the Case Option and the Diacritics Option

Aside from these constraints, the full order of the application of match options is implementation-defined. [Definition: This order is called the match option application order.]

More information on their semantics is given in 4.2.6 Match Options Semantics.

If no match options declarations are present in the prolog and the implementation does not define any overwriting of the static context components for the match options, the query:

/book/title ftcontains "usability" 

is, assuming "de" is the implementation-defined default language, equivalent to the query:

/book/title ftcontains "usability" case insensitive 
    diacritics insensitive 
    without stemming without thesaurus  
    without stop words language "de" without wildcards

We describe each match option group in more detail in the following sections.

/book[@number="1"]/title ftcontains "Usability" lowercase 

/book[@number="1"]/title ftcontains "usability" 
case insensitive 

/book[@number="1"]//editor ftcontains "Vera" diacritics insensitive

/book[@number="1"]/editors ftcontains "Vera" diacritics sensitive

/book[@number="1"]/title ftcontains "improve" with stemming 

count(.//book/content ftcontains "duties" with
thesaurus at "http://bstore1.example.com/UsabilityThesaurus.xml"
relationship "UF")>0

doc("http://bstore1.example.com/full-text.xml")
/books/book[count(./content ftcontains "web site components" with
thesaurus at "http://bstore1.example.com/UsabilityThesaurus.xml"
relationship "NT" at most 2 levels)>0]

doc("http://bstore1.example.com/full-text.xml")
/books/book[count(. ftcontains "Merrygould" with thesaurus at
"http://bstore1.example.com/UsabilitySoundex.xml" relationship
"sounds like")>0]

/book[@number="1"]//p ftcontains "propagation of errors"
with stemming with stop words ("a", "the", "of") 

/book[@number="1"]//p ftcontains "propagation of errors" 
with stemming without stop words

doc("http://bstore1.example.com/full-text.xml")
/books/book[count(.//content ftcontains "planning then 
conducting" with stop words at 
"http://bstore1.example.com/StopWordList.xml")>0]

doc("http://bstore1.example.com/full-text.xml")
/books/book[count(.//content ftcontains "planning then conducting"
with stop words at "http://bstore1.example.com/StopWordList.xml"
except ("the then"))>0]

/book[@number="1"]//editor ftcontains "salon de the"
with default stop words language "fr"

/book[@number="1"]/title ftcontains "improv.*" with
wildcards

/book[@number="1"]/title ftcontains ".?site" with
wildcards

/book[@number="1"]/p ftcontains "w.ll" with
wildcards

/book[@number="1"]/p ftcontains "w.ll" without wildcards

declare namespace exq = "http://example.org/XQueryImplementation";

declare ft-option option exq:diacritics "short-vowel insensitive"

declare namespace exq = "http://example.org/XQueryImplementation";

//para[. ftcontains "Kinder" ftand "Platz" 
        distance 1 words with stemming option exq:compounds "distance=1"

3.4 Logical Full-Text Operators

Full-text selections can be combined with the logical connectives ftor (full-text or), ftand (full-text and), not in (mild not), and ftnot (unary full-text not).

 /book[.//author ftcontains "Millicent" ftor
"Voltaire"] 

/book[@number="1"]/title ftcontains ("usability" ftand "testing")

/book/author ftcontains "Millicent" ftand "Montana"

/book ftcontains "usability" not in "usability
testing"

/book[. ftcontains ftnot "usability"]

/book ftcontains "information" ftand
"retrieval" ftand ftnot "information retrieval"

/book[. ftcontains "web site usability" ftand 
ftnot "usability testing"]

3.5 Positional Filters

[Definition: Positional filters are postfix operators that serve to filter matches based on various constraints on their positional information.]

Recall that the grammar rule for FTSelection allows an arbitrary number of positional filters to follow an FTOr. Multiple adjacent positional filters are applied from left to right, i.e., the first filter is applied to the result of the FTOr, the second is applied to the result of that first application, and so on.

/book/title ftcontains ("web site" ftand "usability")
ordered 

/book[@number="1"] ftcontains ("Montana" ftand
"Millicent") ordered 

/book/title ftcontains "web" ftand "site"
ftand "usability" window 5 words

/book ftcontains ("web" ftand "site" ordered)
ftand ("usability" ftor "testing") window 10 words

/book//title ftcontains "web site" ftand
"usability" window 3 words

/book[@number="1" and . ftcontains "efficient" 
ftand ftnot "and" window 3 words]