The sequence of nodes in the XQuery 1.0 and XPath 2.0 Data Model is
inadequate to support fully composable FTSelections.
Full-text operations, such as FTSelections, operate on linguistic
units, such as positions of tokens, and which are not captured
in the XQuery 1.0 and XPath 2.0 Data Model (XDM).
XQuery 1.0 and XPath 2.0 Full-Text adds relative token, sentence, and
paragraph position numbers via AllMatches. AllMatches make
FTSelections fully composable.
FTSelections
FTSelections are fully composable and may be nested arbitrarily under
other FTSelections. Each FTSelection may be associated with
match options (such as stemming and stop words) and score weights.
Since score weights are solely interpreted by the formal semantics
scoring function, they do not influence the semantics of
FTSelections. Therefore, score weights are not considered in
the formal semantics.
XML Representation
The XML representation of the
FTSelections used in the
fts:evaluate function
closely follows the grammar of the language. It can be viewed as
an XML representation of an abstract syntax tree (AST) of a
parsed full-text query. Every FTSelection
is represented as an XML element. Every nested FTSelection is
represented as a nested descendant element.
For binary FTSelections, e.g., FTAnd, the nested FTSelections
are represented in <left> and <right>
descendant elements. For unary FTSelections, a
<selection> descendant element is used. Additional
characteristics of FTSelections, e.g., the distance unit for
FTDistance, are stored in attributes.
<xs:schema
xmlns:xs="http://www.w3.org/2001/XMLSchema"
xmlns:fts="http://www.w3.org/2006/xquery-full-text"
targetNamespace="http://www.w3.org/2006/xquery-full-text"
elementFormDefault="qualified"
attributeFormDefault="unqualified">
<xs:include schemaLocation="AllMatches.xsd" />
<xs:include schemaLocation="MatchOptions.xsd" />
<xs:complexType name="FTSelection">
<xs:sequence>
<xs:choice>
<xs:element name="FTWords" type="fts:FTWords"/>
<xs:element name="FTAnd" type="fts:FTAnd"/>
<xs:element name="FTOr" type="fts:FTOr"/>
<xs:element name="FTUnaryNot" type="fts:FTUnaryNot"/>
<xs:element name="FTMildNot" type="fts:FTMildNot"/>
<xs:element name="FTOrder" type="fts:FTOrder"/>
<xs:element name="FTScope" type="fts:FTScope"/>
<xs:element name="FTContent" type="fts:FTContent"/>
<xs:element name="FTDistance" type="fts:FTDistance"/>
<xs:element name="FTWindow" type="fts:FTWindow"/>
<xs:element name="FTTimes" type="fts:FTTimes"/>
</xs:choice>
<xs:element ref="fts:matchOptions"
minOccurs="0"/>
<xs:element name="weight"
type="xs:double"
minOccurs="0"/>
</xs:sequence>
</xs:complexType>
<xs:element name="selection" type="fts:FTSelection"/>
<xs:complexType name="FTWords">
<xs:sequence>
<xs:element ref="searchItem"
minOccurs="0"
maxOccurs="unbounded"/>
</xs:sequence>
<xs:attribute name="type"
type="fts:FTWordsType"
use="required"/>
</xs:complexType>
<xs:element name="searchItem" type="fts:SearchItem"/>
<xs:complexType name="FTAnd">
<xs:sequence>
<xs:element name="left" type="fts:FTSelection"/>
<xs:element name="right" type="fts:FTSelection"/>
</xs:sequence>
</xs:complexType>
<xs:complexType name="FTOr">
<xs:sequence>
<xs:element name="left" type="fts:FTSelection"/>
<xs:element name="right" type="fts:FTSelection"/>
</xs:sequence>
</xs:complexType>
<xs:complexType name="FTUnaryNot">
<xs:sequence>
<xs:element name="selection" type="fts:FTSelection"/>
</xs:sequence>
</xs:complexType>
<xs:complexType name="FTMildNot">
<xs:sequence>
<xs:element name="selection" type="fts:FTSelection"/>
</xs:sequence>
</xs:complexType>
<xs:complexType name="FTOrder">
<xs:sequence>
<xs:element name="selection" type="fts:FTSelection"/>
</xs:sequence>
</xs:complexType>
<xs:complexType name="FTScope">
<xs:sequence>
<xs:element name="selection" type="fts:FTSelection"/>
</xs:sequence>
<xs:attribute name="type"
type="fts:ScopeType"
use="required"/>
<xs:attribute name="scope"
type="fts:ScopeSelector"
use="required"/>
</xs:complexType>
<xs:complexType name="FTContent">
<xs:sequence>
<xs:element name="selection" type="fts:FTSelection"/>
</xs:sequence>
<xs:attribute name="type"
type="fts:ContentMatchType"
use="required"/>
</xs:complexType>
<xs:complexType name="FTDistance">
<xs:sequence>
<xs:element name="range" type="fts:FTRangeSpec"/>
<xs:element name="selection" type="fts:FTSelection"/>
</xs:sequence>
<xs:attribute name="type"
type="fts:DistanceType"
use="required"/>
</xs:complexType>
<xs:complexType name="FTWindow">
<xs:sequence>
<xs:element name="selection" type="fts:FTSelection"/>
</xs:sequence>
<xs:attribute name="size"
type="xs:integer"
use="required"/>
<xs:attribute name="type"
type="fts:DistanceType"
use="required"/>
</xs:complexType>
<xs:complexType name="FTTimes">
<xs:sequence>
<xs:element name="range" type="fts:FTRangeSpec"/>
<xs:element name="selection" type="fts:FTWords"/>
</xs:sequence>
</xs:complexType>
<xs:simpleType name="FTWordsType">
<xs:restriction base="xs:string">
<xs:enumeration value="any"/>
<xs:enumeration value="all"/>
<xs:enumeration value="phrase"/>
<xs:enumeration value="any word"/>
<xs:enumeration value="all word"/>
</xs:restriction>
</xs:simpleType>
<xs:simpleType name="ScopeType">
<xs:restriction base="xs:string">
<xs:enumeration value="same"/>
<xs:enumeration value="different"/>
</xs:restriction>
</xs:simpleType>
<xs:simpleType name="ScopeSelector">
<xs:restriction base="xs:string">
<xs:enumeration value="paragraph"/>
<xs:enumeration value="sentence"/>
</xs:restriction>
</xs:simpleType>
<xs:simpleType name="DistanceType">
<xs:restriction base="xs:string">
<xs:enumeration value="paragraph"/>
<xs:enumeration value="sentence"/>
<xs:enumeration value="word"/>
</xs:restriction>
</xs:simpleType>
<xs:simpleType name="ContentMatchType">
<xs:restriction base="xs:string">
<xs:enumeration value="at start"/>
<xs:enumeration value="at end"/>
<xs:enumeration value="entire content"/>
</xs:restriction>
</xs:simpleType>
</xs:schema>
The evaluate function
The denotational semantics for the evaluation of
FTSelections is defined using the fts:evaluate
function. The function takes three
parameters: (1) an FTSelection, 2) a search
context node, and 3) the default set of match options
that apply to the evaluation of the FTSelection.
The fts:evaluate function returns the
AllMatches that is the result of evaluating the
FTSelection. When fts:evaluate is applied to some
FTSelection X, it calls the function
fts:ApplyX to build the resulting AllMatches.
If X is applied on nested FTSelections, the
fts:evaluate function is recursively called on these nested
FTSelections and the returned AllMatches are used in the evaluation of
fts:ApplyX.
The semantics for the fts:evaluate function
is given below.
declare function fts:evaluate (
$ftSelect as element(*, fts:FTSelection),
$searchContext as item(),
$matchOptions as element(fts:matchOptions),
$searchTokenNum as xs:integer )
as element(fts:allMatches)
{
if (fn:count($ftSelect/fts:matchOptions) > 0) then
(: First we deal with all match options that the :)
(: FTSelection might bear: we add the match options :)
(: in front of the current match options sequence :)
(: and pass the new sequence to the recursive call :)
let $newFTSelection :=
<fts:selection>{$ftSelect/*[fn:not(self::fts:matchOptions)]}</fts:selection>
return fts:evaluate($newFTSelection,
$searchContext,
element fts:matchOptions {
$ftSelect/fts:matchOptions/*,
$matchOptions/*
},
$searchTokenNum)
else if (fn:count($ftSelect/fts:weight) > 0) then
(: Weight has no bearing on semantics -- just :)
(: call "evaluate" on nested FTSelection :)
let $newFTSelection := $ftSelect/*[fn:not(self::fts:weight)]
return fts:evaluate($newFTSelection,
$searchContext,
$matchOptions,
$searchTokenNum)
else
typeswitch ($ftSelect/*[1])
case $nftSelection as element(fts:FTWords) return
(: Apply the FTWords in the search context :)
fts:ApplyFTWords($searchContext,
$matchOptions,
$nftSelection/@type,
$nftSelection/fts:searchItem,
$searchTokenNum + 1)
case $nftSelection as element(fts:FTAnd) return
let $left := fts:evaluate($nftSelection/fts:left,
$searchContext,
$matchOptions,
$searchTokenNum)
let $newSearchTokenNum := $left/@stokenNum
let $right := fts:evaluate($nftSelection/fts:right,
$searchContext,
$matchOptions,
$newSearchTokenNum)
return fts:ApplyFTAnd($left, $right)
case $nftSelection as element(fts:FTOr) return
let $left := fts:evaluate($nftSelection/fts:left,
$searchContext,
$matchOptions,
$searchTokenNum)
let $newSearchTokenNum := $left/@stokenNum
let $right := fts:evaluate($nftSelection/fts:right,
$searchContext,
$matchOptions,
$newSearchTokenNum)
return fts:ApplyFTOr($left, $right)
case $nftSelection as element(fts:FTUnaryNot) return
let $nested := fts:evaluate($nftSelection/fts:selection,
$searchContext,
$matchOptions,
$searchTokenNum)
return fts:ApplyFTUnaryNot($nested)
case $nftSelection as element(fts:FTMildNot) return
let $left := fts:evaluate($nftSelection/fts:left,
$searchContext,
$matchOptions,
$searchTokenNum)
let $newSearchTokenNum := $left/@stokenNum
let $right := fts:evaluate($nftSelection/fts:right,
$searchContext,
$matchOptions,
$newSearchTokenNum)
return fts:ApplyFTMildNot($left, $right)
case $nftSelection as element(fts:FTOrder) return
let $nested := fts:evaluate($nftSelection/fts:selection,
$searchContext,
$matchOptions,
$searchTokenNum)
return fts:ApplyFTOrder($nested)
case $nftSelection as element(fts:FTScope) return
let $nested := fts:evaluate($nftSelection/fts:selection,
$searchContext,
$matchOptions,
$searchTokenNum)
return fts:ApplyFTScope($nftSelection/@type,
$nftSelection/@scope,
$nested)
case $nftSelection as element(fts:FTContent) return
let $nested := fts:evaluate($nftSelection/fts:selection,
$searchContext,
$matchOptions,
$searchTokenNum)
return fts:ApplyFTContent($searchContext,
$matchOptions,
$nftSelection/@type,
$nested)
case $nftSelection as element(fts:FTDistance) return
let $nested := fts:evaluate($nftSelection/fts:selection,
$searchContext,
$matchOptions,
$searchTokenNum)
return fts:ApplyFTDistance($matchOptions,
$nftSelection/@type,
$nftSelection/fts:range,
$nested)
case $nftSelection as element(fts:FTWindow) return
let $nested := fts:evaluate($nftSelection/fts:selection,
$searchContext,
$matchOptions,
$searchTokenNum)
return fts:ApplyFTWindow($matchOptions,
$nftSelection/@type,
$nftSelection/@size,
$nested)
case $nftSelection as element(fts:FTTimes) return
let $nested := fts:evaluate($nftSelection/fts:selection,
$searchContext,
$matchOptions,
$searchTokenNum)
return fts:ApplyFTTimes($nftSelection/fts:range,
$nested)
default return ()
};
For
concreteness, assume that the FTSelection was
invoked inside an ftcontains expression such
as searchContext ftcontains ftselection. In order to
determine the
AllMatches result of ftselection, the
fts:evaluate function is invoked as
follows: fts:evaluate($ftselection,
$searchContext, $matchOptions, 0), where
$ftselection is the XML representation of the
ftselection and
$searchContext is bound to the result of
the evaluation of the XQuery expression
searchContext.
Initially, the
$searchTokensNum is 0, i.e., no
search tokens have been processed.
The variable $matchOptions is bound to the
list of match options as defined in the static context (see
Appendix ).
Match options embedded in
ftselection modify the match options collection as
evaluation proceeds.
Match options are applied to an FTSelection,
organized in a stack.
The top match option in the stack is
applied first.
The second match option is applied next.
Match
options are applied sequentially down to the bottom of the stack.
Ordering among match options is necessary because match
options are not always commutative. For example,
synonym(stem(word)) is not always the same as stem(synonym(word)).
Naturally, match options may be reordered when they commute, but
this is an optimization issue and is beyond the scope of this
document.
Given the invocation of: fts:evaluate($ftselection,
$searchContext, $matchOptions), evaluation proceeds as
follows. First, $ftselection is checked to see whether
a match option is applied 1) on a nested FTSelection, 2) on a
weight specification, 3) on an FTWords, or 4) on some other
FTSelection (case 4).
-
If $ftselection contains a match option,
then it modifies the context for the nested FTSelection.
Consequently, a new match option element is created and pushed onto
the top of the stack of match options. The
createOptionElement function used to create a
stack element corresponding to the match option creates a
data structure that stores the type of match option, such as
stemming, thesaurus, and the details
relating to the match option, such as the name of the thesaurus, the
stop words for which other tokens may be substituted. The context
match option created is added to
the top of the stack because, in the FTSelection, it was applied
before the other match options in the current match options stack.
The evaluate function is then invoked on the
nested FTSelection with the new match options stack. When the
function returns, the match option is popped from the stack, and the
result of the nested evaluate function is
returned. The match option is popped because the match options
do not apply to FTSelections outside its scope.
If $ftselection contains a weight
specification, then the specification is ignored because it
does not alter the semantics. The evaluate
function is recursively called on the nested FTSelection and the
resulting AllMatches is returned.
If $ftselection is an FTWords, then
it does
not have any nested FTSelections. Consequently, this is the base
of the recursive call, and the AllMatches result of the FTWords
is computed and returned. The AllMatches is computed by invoking
the ApplyFTWords function with the current
search context and other necessary information.
If $ftselection contains neither a match
option nor a weight specification and is not an FTWords, the
FTSelection performs a full-text operation, such as
&&, ||, window.
These operations are fully-compositional and may be
invoked on nested FTSelections. Consequently, evaluation proceeds
as follows.
First, the evaluate function is
recursively invoked on each nested FTSelection.
The result of
evaluating each nested FTSelection is an AllMatches.
The AllMatches are transformed into the resulting
AllMatches by applying the full-text operation corresponding to
FTSelection1 which is
generically named applyX for some type of
FTSelection X in the code.
For example, let
FTSelection1 be FTSelection2 &&
FTSelection3 . Here FTSelection2 and
FTSelection3 may themselves be arbitrarily nested
FTSelections. Thus, evaluate is invoked on
FTSelection2 and FTSelection3, and the
resulting AllMatches are transformed to the final AllMatches
using the ApplyFTAnd function corresponding to
&& .
The semantics of the ApplyX function for
each FTSelection kind X is given below.
Formal semantics functions
The formal semantics of the applyX
functions for each FTSelection kind X is specified by
six functions. How these six functions are computed is
implementation-dependent, but the functions must satisfy some
well-defined properties.
The getTokenInfo function is
described in Section 4.1
Tokenization.
The wordDistance function returns the
number of tokens that occur between the positions of the
TokenInfos $tokenInfo1 and
$tokenInfo2. For example, two consecutive
tokens have a distance of 0 tokens.
declare function fts:wordDistance (
$tokenInfo1 as element(fts:tokenInfo),
$tokenInfo2 as element(fts:tokenInfo),
$matchOptions as element(fts:matchOptions) )
as xs:integer external;
The getParaDistance function returns the number of
paragraphs between the TokenInfos
$tokenInfo1 and $tokenInfo2.
declare function fts:paraDistance (
$tokenInfo1 as element(fts:tokenInfo),
$tokenInfo2 as element(fts:tokenInfo),
$matchOptions as element(fts:matchOptions) )
as xs:integer external;
The sentenceDistance function
returns the number of sentences between the
TokenInfos $tokenInfo1 and
$tokenInfo2.
declare function fts:sentenceDistance (
$tokenInfo1 as element(fts:tokenInfo),
$tokenInfo2 as element(fts:tokenInfo),
$matchOptions as element(fts:matchOptions) )
as xs:integer external;
The isStartToken function
returns true if the TokenInfo $tokenInfo
describes a token whose start position is the first position of
the node $searchContext.
declare function fts:isStartToken (
$searchContext as item(),
$tokenInfo as element(fts:tokenInfo) )
as xs:boolean external;
The isEndToken function
returns true if the TokenInfo $tokenInfo
describes a token whose end position is the last position of
the node $searchContext.
declare function fts:isEndToken (
$searchContext as item(),
$tokenInfo as element(fts:tokenInfo) )
as xs:boolean external;
FTWords
An FTWords that consists of a single
search string consisting of a sequence of token to be
matched as a phrase is evaluated by
the applySearchTokensAsPhrase function. Its parameters
are 1) the search context, 2) the list of match options, 3) the search
string to be matched as a sequence of fts:searchToken
items, and 4) the position where the latter search string occurs in the
query.
(: simplified version not dealing with special match options :)
declare function fts:applySearchTokensAsPhrase (
$searchContext as item(),
$matchOptions as element(fts:matchOptions),
$searchTokens as element(fts:searchToken)*,
$queryPos as xs:integer )
as element(fts:allMatches)
{
<fts:allMatches stokenNum="{$queryPos}">
{
for $tokenInfo in
fts:matchTokenInfos(
$searchContext,
$matchOptions,
(),
$searchTokens )
return
<fts:match>
<fts:stringInclude queryPos="{$queryPos}">
{$tokenInfo}
</fts:stringInclude>
</fts:match>
}
</fts:allMatches>
};
If after the application of all the match options, the sequence
of search tokens returned for an FTWords is empty, an empty
AllMatches is returned.
The AllMatches corresponding to an
FTWords is a set of Matches. Each of the Matches
is associated with a start and an end position indicating where the corresponding
search tokens were found. For example, the AllMatches
result for the FTWords "Mustang" is given
below. To simplify the presentation in the figures we write
Pos: N, if the attributes
startPos and endPos are the same
with N being that position.
![FTWords example]()
There are five variations of FTWords depending on how the
tokens and phrases in the nested XQuery 1.0 and XPath 2.0
expression are matched.
When any word is specified, at
least one token in the tokenization of the nested expression must be
matched.
When all word is specified, all
tokens in the tokenization of the nested expression must be
matched.
When phrase is specified, all
tokens in the tokenization of the nested expression must be
matched as a phrase.
When any is specified, at least one
string atomic value in the nested expression must be
matched as a phrase.
When all is specified, all
string atomic values in the nested expression must be
matched as a phrase.
The semantics for FTWords when any word is specified
is given below. Since FTWords
does not have nested FTSelections, the
ApplyFTWords function does not take
AllMatches parameters corresponding to nested
FTSelection results.
declare function fts:MakeDisjunction (
$curRes as element(fts:allMatches),
$rest as element(fts:allMatches)* )
as element(fts:allMatches)
{
if (fn:count($rest) = 0)
then $curRes
else
let $firstAllMatches := $rest[1]
let $restAllMatches := fn:subsequence($rest, 2)
let $newCurRes := fts:ApplyFTOr($curRes,
$firstAllMatches)
return fts:MakeDisjunction($newCurRes,
$restAllMatches)
};
declare function fts:ApplyFTWordsAnyWord (
$searchContext as item(),
$matchOptions as element(fts:matchOptions),
$searchItems as element(fts:searchItem)*,
$queryPos as xs:integer )
as element(fts:allMatches)
{
(: Tokenization of search string has already occurred. :)
(: Get sequence of SearchTokens over all search items. :)
let $searchTokens := $searchItems/fts:searchToken
return
if (fn:count($searchItems) eq 0)
then <fts:allMatches stokenNum="0" />
else
let $allAllMatches :=
for $searchToken at $pos in $searchTokens
return fts:applySearchTokensAsPhrase($searchContext,
$matchOptions,
$searchToken,
$queryPos + $pos - 1)
let $firstAllMatches := $allAllMatches[1]
let $restAllMatches := fn:subsequence($allAllMatches, 2)
return fts:MakeDisjunction($firstAllMatches, $restAllMatches)
};
The tokenized search strings are passed to
ApplyFTWordsAnyWord as a sequence of
fts:searchItem, each containing the tokens of
a single search string. A single flattened sequence of all
tokens (of type fts:searchToken) over all
search items is constructed. For each of these,
the result of FTWords is computed using
applySearchTokensAsPhrase. Finally, the
disjunction of all resulting AllMatches is computed.
The semantics for FTWords when all word
is specified is similar to the above, however composes a
conjunction. It is given below.
declare function fts:MakeConjunction (
$curRes as element(fts:allMatches),
$rest as element(fts:allMatches)* )
as element(fts:allMatches)
{
if (fn:count($rest) = 0)
then $curRes
else
let $firstAllMatches := $rest[1]
let $restAllMatches := fn:subsequence($rest, 2)
let $newCurRes := fts:ApplyFTAnd($curRes,
$firstAllMatches)
return fts:MakeConjunction($newCurRes,
$restAllMatches)
};
declare function fts:ApplyFTWordsAllWord (
$searchContext as item(),
$matchOptions as element(fts:matchOptions),
$searchItems as element(fts:searchItem)*,
$queryPos as xs:integer )
as element(fts:allMatches)
{
(: Tokenization of search strings has already occurred. :)
(: Get sequence of SearchTokens over all search items :)
let $searchTokens := $searchItems/fts:searchToken
return
if (fn:count($searchTokens) eq 0)
then <fts:allMatches stokenNum="0" />
else
let $allAllMatches :=
for $searchToken at $pos in $searchTokens
return fts:applySearchTokensAsPhrase($searchContext,
$matchOptions,
$searchToken,
$queryPos + $pos - 1)
let $firstAllMatches := $allAllMatches[1]
let $restAllMatches := fn:subsequence($allAllMatches, 2)
return fts:MakeConjunction($firstAllMatches, $restAllMatches)
};
The semantics for FTWords if phrase is specified
is given below.
declare function fts:ApplyFTWordsPhrase (
$searchContext as item(),
$matchOptions as element(fts:matchOptions),
$searchItems as element(fts:searchItem)*,
$queryPos as xs:integer )
as element(fts:allMatches)
{
(: Get sequence of SearchTokenInfos over all search items :)
let $searchTokens := $searchItems/fts:searchToken
return
if (fn:count($searchTokens) eq 0)
then <fts:allMatches stokenNum="0" />
else
fts:applySearchTokensAsPhrase($searchContext,
$matchOptions,
$searchTokens,
$queryPos)
};
The ApplyFTWordsPhrase function
also flattens the sequence of search items to a sequence of
search tokens, but then calls
applySearchTokensAsPhrase on that
entire sequence, instead of calling it on each search token
individually. Hence, the sequence of all search tokens is
matched as a single phrase and the computed TokenInfos
are returned.
The semantics for FTWords when any is specified is
given below.
declare function fts:ApplyFTWordsAny (
$searchContext as item(),
$matchOptions as element(fts:matchOptions),
$searchItems as element(fts:searchItem)*,
$queryPos as xs:integer )
as element(fts:allMatches)
{
if (fn:count($searchItems) eq 0)
then <fts:allMatches stokenNum="0" />
else
let $firstSearchItem := $searchItems[1]
let $restSearchItem := fn:subsequence($searchItems, 2)
let $firstAllMatches :=
fts:ApplyFTWordsPhrase($searchContext,
$matchOptions,
$firstSearchItem,
$queryPos)
let $newQueryPos :=
if ($firstAllMatches//@queryPos)
then fn:max($firstAllMatches//@queryPos) + 1
else $queryPos
let $restAllMatches :=
fts:ApplyFTWordsAny($searchContext,
$matchOptions,
$restSearchItem,
$newQueryPos)
return fts:ApplyFTOr($firstAllMatches, $restAllMatches)
};
The FTWords with any
specified forms the disjunction of the AllMatches that
are the result of the matching of each search item as a phrase.
The semantics for FTWords when all is specified
is given below.
declare function fts:ApplyFTWordsAll (
$searchContext as item(),
$matchOptions as element(fts:matchOptions),
$searchItems as element(fts:searchItem)*,
$queryPos as xs:integer )
as element(fts:allMatches)
{
if (fn:count($searchItems) = 0)
then <fts:allMatches stokenNum="0" />
else
let $firstSearchItem := $searchItems[1]
let $restSearchItem := fn:subsequence($searchItems, 2)
let $firstAllMatches :=
fts:ApplyFTWordsPhrase($searchContext,
$matchOptions,
$firstSearchItem,
$queryPos)
return
if ($restSearchItem) then
let $newQueryPos :=
if ($firstAllMatches//@queryPos)
then fn:max($firstAllMatches//@queryPos) + 1
else $queryPos
let $restAllMatches :=
fts:ApplyFTWordsAll($searchContext,
$matchOptions,
$restSearchItem,
$newQueryPos)
return
fts:ApplyFTAnd($firstAllMatches, $restAllMatches)
else $firstAllMatches
};
The difference between all and
any is the use of conjunction instead of
disjunction.
The ApplyFTWords function combines
all of these functions.
declare function fts:ApplyFTWords (
$searchContext as item(),
$matchOptions as element(fts:matchOptions),
$type as fts:FTWordsType,
$searchItems as element(fts:searchItem)*,
$queryPos as xs:integer )
as element(fts:allMatches)
{
if ($type eq "any word")
then fts:ApplyFTWordsAnyWord($searchContext,
$matchOptions,
$searchItems,
$queryPos)
else if ($type eq "all word")
then fts:ApplyFTWordsAllWord($searchContext,
$matchOptions,
$searchItems,
$queryPos)
else if ($type eq "phrase")
then fts:ApplyFTWordsPhrase($searchContext,
$matchOptions,
$searchItems,
$queryPos)
else if ($type eq "any")
then fts:ApplyFTWordsAny($searchContext,
$matchOptions,
$searchItems,
$queryPos)
else fts:ApplyFTWordsAll($searchContext,
$matchOptions,
$searchItems,
$queryPos)
};
FTOr
The parameters of the ApplyFTOr
function are the two AllMatches parameters
corresponding to the results of the two nested
FTSelections. The search context and the match
options stack are not used by this function. The semantics is given
below.
declare function fts:ApplyFTOr (
$allMatches1 as element(fts:allMatches),
$allMatches2 as element(fts:allMatches) )
as element(fts:allMatches)
{
<fts:allMatches stokenNum="{fn:max(($allMatches1/@stokenNum,
$allMatches2/@stokenNum))}">
{$allMatches1/fts:match,$allMatches2/fts:match}
</fts:allMatches>
};
The ApplyFTOr function creates a new AllMatches
in which
Matches are the union of those found
in the input AllMatches.
Each Match represents one possible result of the corresponding
FTSelection. Thus, a Match from either
of the AllMatches is a result.
For example, consider the FTSelection
"Mustang" || "Honda". The
AllMatches corresponding to
"Mustang" and "Honda" are given below.
![FTOr input AllMatches 1]()
![FTOr input AllMatches 2]()
The AllMatches produced by
ApplyFTOr is given below.
![FTOr result AllMatches]()
FTAnd
The parameters of the ApplyFTAnd
function are the two AllMatches
corresponding to the results of the two nested
FTSelections. The search context and the match
options are not used by this function. The semantics is given below.
declare function fts:ApplyFTAnd (
$allMatches1 as element(fts:allMatches),
$allMatches2 as element(fts:allMatches) )
as element(fts:allMatches)
{
<fts:allMatches stokenNum="{fn:max(($allMatches1/@stokenNum,
$allMatches2/@stokenNum))}" >
{
for $sm1 in $allMatches1/fts:match
for $sm2 in $allMatches2/fts:match
return <fts:match>
{$sm1/*, $sm2/*}
</fts:match>
}
</fts:allMatches>
};
The result of the conjunction is a new
AllMatches that contains the "Cartesian
product" of the matches of the participating
FTSelections. Every resulting Match
is formed by the combination of the StringInclude components
and StringExclude from the
AllMatches of the nested FTSelection
. Thus every match contains the positions
to satisfy a Match from both original
FTSelections and excludes the positions that
violate the same Matches.
For example, consider the FTSelection
"Mustang" && "rust". The
source AllMatches are give below.
![FTAnd input AllMatches 1]()
![FTAnd input AllMatches 2]()
The AllMatches produced by ApplyFTAnd is
given below.
![FTAnd result AllMatches]()
FTUnaryNot
The parameters of the ApplyFTUnaryNot
function are 1) the search context, 2) the list of match
options, and 3) one AllMatches parameter corresponding to the
result of the nested FTSelection to be negated. The
search context and the match options are not used by this function. The
semantics is given below.
declare function fts:InvertStringMatch ( $strm as element(*,fts:StringMatch) )
as element(*,fts:StringMatch)
{
if ($strm instance of element(fts:stringExclude)) then
<fts:stringInclude queryPos="{$strm/@queryPos}">
{$strm/fts:tokenInfo}
</fts:stringInclude>
else
<fts:stringExclude queryPos="{$strm/@queryPos}">
{$strm/fts:tokenInfo}
</fts:stringExclude>
};
declare function fts:UnaryNotHelper ( $matches as element(fts:match)* )
as element(fts:match)*
{
if (fn:count($matches) = 0
then <match/>
else
for $sm in $matches[1]/*
for $rest in fts:UnaryNotHelper( fn:subsequence($matches, 2) )
return
<fts:match>
{
fts:InvertStringMatch($sm),
$rest/*
}
</fts:match>
};
declare function fts:ApplyFTUnaryNot (
$allMatches as element(fts:allMatches) )
as element(fts:allMatches)
{
<fts:allMatches stokenNum="{$allMatches/@stokenNum}">
{
fts:UnaryNotHelper($allMatches/fts:match)
}
</fts:allMatches>
};
The generation of the resulting
AllMatches of an FTUnaryNot resembles the transformation
of a negation of prepositional formula in DNF back to DNF.
The negation of AllMatches requires
the inversion of all the conditions on the nodes encoded
by the AllMatches.
In the InvertStringMatch function above,
this inversion occurs as follows.
The function fts:invertStringMatch
inverts a stringInclude into a stringExclude and
vice versa.
The function fts:UnaryNotHelper transforms
the source Matches into the resulting
Matches by forming the combinations of the
inversions of a StringInclude or StringExclude
component over the source Matches into new Matches.
For example, consider the FTSelection
! ("Mustang" || "Honda"). The
source AllMatches is given below:
![FTUnaryNot input AllMatches]()
The FTUnaryNot transforms the StringIncludes to
StringExcludes as illustrated below.
![FTUnaryNot result AllMatches]()
FTMildNot
The parameters of the ApplyFTMildNot
function are the two AllMatches parameters corresponding
to the results of the two nested FTSelections. The
search context and the match options stack are not
used by this function. The semantics is given below.
declare function fts:ApplyFTMildNot (
$allMatches1 as element(fts:allMatches),
$allMatches2 as element(fts:allMatches) )
as element(fts:allMatches)
{
if (fn:count($allMatches2//fts:stringExclude) gt 0) then
fn:error("Invalid expression on the right-hand side of a not-in")
else
<fts:allMatches stokenNum="{$allMatches1/@stokenNum}">
{
let $posSet2 := $allMatches2/fts:match/fts:stringInclude/@queryPos
return
$allMatches1/fts:match[
every $pos1 in ./fts:stringInclude/@queryPos,
$pos2 in $posSet2
satisfies $pos1 ne $pos2]
}
</fts:allMatches>
};
The resulting AllMatches contains Matches
of the first operand that do not mention in their
stringInclude components positions in a StringInclude component
in the AllMatches of the second operand.
For example, consider the FTSelection
("Ford" mildnot "Ford
Mustang"). The
source AllMatches for the left-hand side argument is given below.
![FTMildnot input AllMatches 1]()
The
source AllMatches for the right-hand side argument is given below.
![FTMildnot input AllMatches 2]()
The FTMildNot will transform these to an empty
AllMatches because both position 1 and position 27 from
the first AllMatches contain only TokenInfos from
stringInclude components of the second
AllMatches.
FTOrder
The parameters of the ApplyFTOrder
function are 1) the search context, 2) the list of match
options, and 3) one AllMatches parameter corresponding to
the result of the nested FTSelections. The evaluation
context and the match options are not used by this function.
The semantics is given below.
declare function fts:ApplyFTOrder (
$allMatches as element(fts:allMatches) )
as element(fts:allMatches)
{
<fts:allMatches stokenNum="{$allMatches/@stokenNum}">
{
for $match in $allMatches/fts:match
where every $stringInclude1 in $match/fts:stringInclude,
$stringInclude2 in $match/fts:stringInclude
satisfies (($stringInclude1/fts:tokenInfo/@startPos <=
$stringInclude2/fts:tokenInfo/@startPos)
and
($stringInclude1/@queryPos <=
$stringInclude2/@queryPos))
or
(($stringInclude1/fts:tokenInfo/@startPos>=
$stringInclude2/fts:tokenInfo/@startPos)
and
($stringInclude1/@queryPos >=
$stringInclude2/@queryPos))
return
<fts:match>
{
$match/fts:stringInclude,
for $stringExcl in $match/fts:stringExclude
where every $stringIncl in $match/fts:stringInclude
satisfies (($stringExcl/fts:tokenInfo/@startPos <=
$stringIncl/fts:tokenInfo/@startPos)
and
($stringExcl/@queryPos <=
$stringIncl/@queryPos))
or
(($stringExcl/fts:tokenInfo/@startPos >=
$stringIncl/fts:tokenInfo/@startPos)
and
($stringExcl/@queryPos >=
$stringIncl/@queryPos))
return $stringExcl
}
</fts:match>
}
</fts:allMatches>
};
The resulting AllMatches contains the Matches for which the start positions
in the StringInclude elements are in the order of the
query positions of their query strings.
StringExcludes that preserve the order (with respect to their start positions) are also retained.
For example, consider the FTSelection
("great" && "condition")
ordered. The source AllMatches is given below.
![FTOrder input AllMatches]()
![FTOrder input AllMatches]()
![FTOrder input AllMatches]()
The AllMatches for FTOrder are given below.
![FTOrder result AllMatches]()
![FTOrder result AllMatches]()
FTScope
The parameters of the ApplyFTScope
function are 1) the search context, 2) the list of match
options, 3) the type of the scope (same or different), 4) the
linguistic unit (sentence or paragraph), and 5) one
AllMatches parameter corresponding to the result of the
nested FTSelections. The search context and the
match options are not used by this function. The function
definitions depend on the type of the scope (paragraph,
sentence) and the scope predicate (same, different).
The semantics of same sentence is given below.
declare function fts:ApplyFTScopeSameSentence (
$allMatches as element(fts:allMatches) )
as element(fts:allMatches)
{
<fts:allMatches stokenNum="{$allMatches/@stokenNum}">
{
for $match in $allMatches/fts:match
where every $stringInclude1 in $match/fts:stringInclude,
$stringInclude2 in $match/fts:stringInclude
satisfies $stringInclude1/fts:tokenInfo/@startSent =
$stringInclude2/fts:tokenInfo/@startSent
and $stringInclude1/fts:tokenInfo/@startSent =
$stringInclude1/fts:tokenInfo/@endSent
and $stringInclude2/fts:tokenInfo/@startSent =
$stringInclude2/fts:tokenInfo/@endSent
and $stringInclude1/fts:tokenInfo/@startSent > 0
and $stringInclude2/fts:tokenInfo/@startSent > 0
return
<fts:match>
{
$match/fts:stringInclude,
for $stringExcl in $match/fts:stringExclude
where
$stringExcl/fts:tokenInfo/@startSent = 0
or
($stringExcl/fts:tokenInfo/@startSent =
$stringExcl/fts:tokenInfo/@endSent
and
(every $stringIncl in $match/fts:stringInclude
satisfies $stringIncl/fts:tokenInfo/@startSent =
$stringExcl/fts:tokenInfo/@startSent) )
return $stringExcl
}
</fts:match>
}
</fts:allMatches>
};
An AllMatches returned by the scope same sentence
contains those Matches whose StringIncludes span only a single
sentence and all span the same sentence. In these Matches only
those StringExcludes are retained that also only span a single
sentence, which is, in case there are StringIncludes in that Match,
the same as the one spanned by the StringIncludes.
The semantics of different sentence
is given below.
declare function fts:ApplyFTScopeDifferentSentence (
$allMatches as element(fts:allMatches) )
as element(fts:allMatches)
{
<fts:allMatches stokenNum="{$allMatches/@stokenNum}">
{
for $match in $allMatches/fts:match
where every $stringInclude1 in $match/fts:stringInclude,
$stringInclude2 in $match/fts:stringInclude
satisfies $stringInclude1 = $stringInclude2
or $stringInclude1/fts:tokenInfo/@endSent <
$stringInclude2/fts:tokenInfo/@startSent
or $stringInclude2/fts:tokenInfo/@endSent <
$stringInclude1/fts:tokenInfo/@startSent
return
<fts:match>
{
$match/fts:stringInclude,
for $stringExcl in $match/fts:stringExclude
where every $stringIncl in $match/fts:stringInclude
satisfies $stringExcl/fts:tokenInfo/@endSent <
$stringIncl/fts:tokenInfo/@startSent
or $stringIncl/fts:tokenInfo/@endSent <
$stringExcl/fts:tokenInfo/@startSent
return $stringExcl
}
</fts:match>
}
</fts:allMatches>
};
An AllMatches returned by the scope different sentence
contains those Matches that have no two StringIncludes covering
the same sentence.
In these Matches only those StringExcludes are retained that also do not
cover a common sentence with one of the StringIncludes.
The semantics of same paragraph is analogous to same
sentence and is given below.
declare function fts:ApplyFTScopeSameParagraph (
$allMatches as element(fts:allMatches) )
as element(fts:allMatches)
{
<fts:allMatches stokenNum="{$allMatches/@stokenNum}">
{
for $match in $allMatches/fts:match
where every $stringInclude1 in $match/fts:stringInclude,
$stringInclude2 in $match/fts:stringInclude
satisfies $stringInclude1/fts:tokenInfo/@startPara =
$stringInclude2/fts:tokenInfo/@startPara
and $stringInclude1/fts:tokenInfo/@startPara =
$stringInclude1/fts:tokenInfo/@endPara
and $stringInclude2/fts:tokenInfo/@startPara =
$stringInclude2/fts:tokenInfo/@endPara
and $stringInclude1/fts:tokenInfo/@startPara > 0
and $stringInclude2/fts:tokenInfo/@endPara > 0
return
<fts:match>
{
$match/fts:stringInclude,
for $stringExcl in $match/fts:stringExclude
where
$stringExcl/fts:tokenInfo/@startPara = 0
or
($stringExcl/fts:tokenInfo/@startPara =
$stringExcl/fts:tokenInfo/@endPara
and
(every $stringIncl in $match/fts:stringInclude
satisfies $stringIncl/fts:tokenInfo/@startPara =
$stringExcl/fts:tokenInfo/@startPara) )
return $stringExcl
}
</fts:match>
}
</fts:allMatches>
};
The semantics of different paragraph is analogous to
different sentence and is given below.
declare function fts:ApplyFTScopeDifferentParagraph (
$allMatches as element(fts:allMatches) )
as element(fts:allMatches)
{
<fts:allMatches stokenNum="{$allMatches/@stokenNum}">
{
for $match in $allMatches/fts:match
where every $stringInclude1 in $match/fts:stringInclude,
$stringInclude2 in $match/fts:stringInclude
satisfies $stringInclude1 = $stringInclude2
or $stringInclude1/fts:tokenInfo/@endPara <
$stringInclude2/fts:tokenInfo/@startPara
or $stringInclude2/fts:tokenInfo/@endPara <
$stringInclude1/fts:tokenInfo/@startPara
return
<fts:match>
{
$match/fts:stringInclude,
for $stringExcl in $match/fts:stringExclude
where every $stringIncl in $match/fts:stringInclude
satisfies $stringExcl/fts:tokenInfo/@endPara <
$stringIncl/fts:tokenInfo/@startPara
or $stringIncl/fts:tokenInfo/@endPara <
$stringExcl/fts:tokenInfo/@startPara
return $stringExcl
}
</fts:match>
}
</fts:allMatches>
};
The semantics for the general case is given below.
declare function fts:ApplyFTScope (
$type as fts:ScopeType,
$selector as fts:ScopeSelector,
$allMatches as element(fts:allMatches) )
as element(fts:allMatches)
{
if ($type eq "same" and $selector eq "sentence")
then fts:ApplyFTScopeSameSentence($allMatches)
else if ($type eq "different" and $selector eq "sentence")
then fts:ApplyFTScopeDifferentSentence($allMatches)
else if ($type eq "same" and $selector eq "paragraph")
then fts:ApplyFTScopeSameParagraph($allMatches)
else fts:ApplyFTScopeDifferentParagraph($allMatches)
};
For example, consider the FTSelection
("Mustang" && "Honda") same
paragraph. The source AllMatches is given below.
![FTScope input AllMatches]()
The FTScope returns an empty AllMatches because neither
Match contains TokenInfos from a single sentence.
FTContent
The parameters of the ApplyFTContent
function are 1) the search context, 2) the match options,
3) the type of the content
match at the start of the current node, at the end of it,
or its entire content, and 4) one
AllMatches parameter corresponding to the result of the
nested FTSelections. The semantics is given below.
declare function fts:ApplyFTContent (
$searchContext as item(),
$matchOptions as element(fts:matchOptions),
$type as fts:ContentMatchType,
$allMatches as element(fts:allMatches) )
as element(fts:allMatches)
{
if ($type eq "entire content") then
let $temp1 := fts:ApplyFTWordDistanceExactly(
$matchOptions,
$allMatches,
1)
let $temp2 := fts:ApplyFTContent(
$searchContext,
$matchOptions,
$temp1,
"at start")
let $temp3 := fts:ApplyFTContent(
$searchContext,
$matchOptions,
$temp2,
"at end")
return $temp3
else
<fts:allMatches stokenNum="{$allMatches/@stokenNum}">
{
for $match in $allMatches/fts:match
where if ($type eq "at start") then
some $si in $match/fts:stringInclude
satisfies fts:isStartToken($searchContext,
$si/fts:tokenInfo)
else (: $type eq "at end" :)
some $si in $match/fts:stringInclude
satisfies fts:isEndToken($searchContext,
$si/fts:tokenInfo)
return $match
}
</fts:allMatches>
};
The evaluation of scope functions depends on
the type of the content match.
entire match is evaluated as
distance exactly 0 words at start at end, i.e., all the
StringIncludes must match every token in the content of the current
search context node.
at start retains only
Matches that contain a StringInclude that matches the
first token. This is checked using the semantic function
fts:isStartToken.
at end retains the
Matches that contain a StringInclude that matches the
last token. This is checked using the semantic function
fts:isEndToken.
The parameters of the
ApplyFTDistance function are 1) the
search context, 2) the list of match options, 3) one
AllMatches parameter corresponding to the result of the
nested FTSelections, 4) the unit of the distance (tokens,
sentences, paragraphs), and 5) the range specified.
The search context is not used by this function. The
function definitions depend on the distance units and the range
specifications.
The semantics of case word distance exactly N
is given below.
declare function fts:ApplyFTWordDistanceExactly(
$matchOptions as element(fts:matchOptions),
$allMatches as element(fts:allMatches),
$n as xs:integer )
as element(fts:allMatches)
{
<fts:allMatches stokenNum="{$allMatches/@stokenNum}">
{
for $match in $allMatches/fts:match
let $sorted := for $si in $match/fts:stringInclude
order by $si/fts:tokenInfo/@startPos ascending
return $si
where
if (fn:count($sorted) le 1) then fn:true() else
every $idx in 1 to fn:count($sorted) - 1
satisfies fts:wordDistance(
$sorted[$idx]/fts:tokenInfo,
$sorted[$idx+1]/fts:tokenInfo,
$matchOptions) = $n
return
<fts:match>
{
$match/fts:stringInclude,
for $stringExcl in $match/fts:stringExclude
where some $stringIncl in $match/fts:stringInclude
satisfies fts:wordDistance(
$stringIncl/fts:tokenInfo,
$stringExcl/fts:tokenInfo,
$matchOptions) = $n
return $stringExcl
}
</fts:match>
}
</fts:allMatches>
};
The semantics of word distance at least N is given
below.
declare function fts:ApplyFTWordDistanceAtLeast (
$matchOptions as element(fts:matchOptions),
$allMatches as element(fts:allMatches),
$n as xs:integer )
as element(fts:allMatches)
{
<fts:allMatches stokenNum="{$allMatches/@stokenNum}">
{
for $match in $allMatches/fts:match
let $sorted := for $si in $match/fts:stringInclude
order by $si/fts:tokenInfo/@startPos ascending
return $si
where
if (fn:count($sorted) le 1) then fn:true() else
every $index in (1 to fn:count($sorted) - 1)
satisfies fts:wordDistance(
$sorted[$index]/fts:tokenInfo,
$sorted[$index+1]/fts:tokenInfo,
$matchOptions) >= $n
return
<fts:match>
{
$match/fts:stringInclude,
for $stringExcl in $match/fts:stringExclude
where some $stringIncl in $match/fts:stringInclude
satisfies fts:wordDistance(
$stringIncl/fts:tokenInfo,
$stringExcl/fts:tokenInfo,
$matchOptions) >= $n
return $stringExcl
}
</fts:match>
}
</fts:allMatches>
};
The semantics of word distance at most N is given
below.
declare function fts:ApplyFTWordDistanceAtMost (
$matchOptions as element(fts:matchOptions),
$allMatches as element(fts:allMatches),
$n as xs:integer )
as element(fts:allMatches)
{
<fts:allMatches stokenNum="{$allMatches/@stokenNum}">
{
for $match in $allMatches/fts:match
let $sorted := for $si in $match/fts:stringInclude
order by $si/fts:tokenInfo/@startPos ascending
return $si
where
if (fn:count($sorted) le 1) then fn:true() else
every $index in (1 to fn:count($sorted) - 1)
satisfies fts:wordDistance(
$sorted[$index]/fts:tokenInfo,
$sorted[$index+1]/fts:tokenInfo,
$matchOptions) <= $n
return
<fts:match>
{
$match/fts:stringInclude,
for $stringExcl in $match/fts:stringExclude
where some $stringIncl in $match/fts:stringInclude
satisfies fts:wordDistance(
$stringIncl/fts:tokenInfo,
$stringExcl/fts:tokenInfo,
$matchOptions) <= $n
return $stringExcl
}
</fts:match>
}
</fts:allMatches>
};
The semantics of word distance from M to N is given
below.
declare function fts:ApplyFTWordDistanceFromTo (
$matchOptions as element(fts:matchOptions),
$allMatches as element(fts:allMatches),
$m as xs:integer,
$n as xs:integer )
as element(fts:allMatches)
{
<fts:allMatches stokenNum="{$allMatches/@stokenNum}">
{
for $match in $allMatches/fts:match
let $sorted := for $si in $match/fts:stringInclude
order by $si/fts:tokenInfo/@startPos ascending
return $si
where
if (fn:count($sorted) le 1) then fn:true() else
every $index in (1 to fn:count($sorted) - 1)
satisfies fts:wordDistance(
$sorted[$index]/fts:tokenInfo,
$sorted[$index+1]/fts:tokenInfo,
$matchOptions) >= $m
and
fts:wordDistance(
$sorted[$index]/fts:tokenInfo,
$sorted[$index+1]/fts:tokenInfo,
$matchOptions) <= $n
return
<fts:match>
{
$match/fts:stringInclude,
for $stringExcl in $match/fts:stringExclude
where some $stringIncl in $match/fts:stringInclude
satisfies fts:wordDistance(
$stringIncl/fts:tokenInfo,
$stringExcl/fts:tokenInfo,
$matchOptions) >= $m
and
fts:wordDistance(
$stringIncl/fts:tokenInfo,
$stringExcl/fts:tokenInfo,
$matchOptions) <= $n
return $stringExcl
}
</fts:match>
}
</fts:allMatches>
};
The semantics of sentence distance exactly N
is given below.
declare function fts:ApplyFTSentenceDistanceExactly (
$matchOptions as element(fts:matchOptions),
$allMatches as element(fts:allMatches),
$n as xs:integer )
as element(fts:allMatches)
{
<fts:allMatches stokenNum="{$allMatches/@stokenNum}">
{
for $match in $allMatches/fts:match
let $sorted := for $si in $match/fts:stringInclude
order by $si/fts:tokenInfo/@startPos ascending
return $si
where
if (fn:count($sorted) le 1) then fn:true() else
every $index in (1 to fn:count($sorted) - 1)
satisfies fts:sentenceDistance(
$sorted[$index]/fts:tokenInfo,
$sorted[$index+1]/fts:tokenInfo,
$matchOptions) = $n
return
<fts:match>
{
$match/fts:stringInclude,
for $stringExcl in $match/fts:stringExclude
where some $stringIncl in $match/fts:stringInclude
satisfies fts:sentenceDistance(
$stringIncl/fts:tokenInfo,
$stringExcl/fts:tokenInfo,
$matchOptions) = $n
return $stringExcl
}
</fts:match>
}
</fts:allMatches>
};
The semantics of sentence distance at least N
is given below.
declare function fts:ApplyFTSentenceDistanceAtLeast (
$matchOptions as element(fts:matchOptions),
$allMatches as element(fts:allMatches),
$n as xs:integer )
as element(fts:allMatches)
{
<fts:allMatches stokenNum="{$allMatches/@stokenNum}">
{
for $match in $allMatches/fts:match
let $sorted := for $si in $match/fts:stringInclude
order by $si/fts:tokenInfo/@startPos ascending
return $si
where
if (fn:count($sorted) le 1) then fn:true() else
every $index in (1 to fn:count($sorted) - 1)
satisfies fts:sentenceDistance(
$sorted[$index]/fts:tokenInfo,
$sorted[$index+1]/fts:tokenInfo,
$matchOptions) >= $n
return
<fts:match>
{
$match/fts:stringInclude,
for $stringExcl in $match/fts:stringExclude
where some $stringIncl in $match/fts:stringInclude
satisfies fts:sentenceDistance(
$stringIncl/fts:tokenInfo,
$stringExcl/fts:tokenInfo,
$matchOptions) >= $n
return $stringExcl
}
</fts:match>
}
</fts:allMatches>
};
The semantics of sentence distance at most N
is given below.
declare function fts:ApplyFTSentenceDistanceAtMost (
$matchOptions as element(fts:matchOptions),
$allMatches as element(fts:allMatches),
$n as xs:integer )
as element(fts:allMatches)
{
<fts:allMatches stokenNum="{$allMatches/@stokenNum}">
{
for $match in $allMatches/fts:match
let $sorted := for $si in $match/fts:stringInclude
order by $si/fts:tokenInfo/@startPos ascending
return $si
where
if (fn:count($sorted) le 1) then fn:true() else
every $index in (1 to fn:count($sorted) - 1)
satisfies fts:sentenceDistance(
$sorted[$index]/fts:tokenInfo,
$sorted[$index+1]/fts:tokenInfo,
$matchOptions) <= $n
return
<fts:match>
{
$match/fts:stringInclude,
for $stringExcl in $match/fts:stringExclude
where some $stringIncl in $match/fts:stringInclude
satisfies fts:sentenceDistance(
$stringIncl/fts:tokenInfo,
$stringExcl/fts:tokenInfo,
$matchOptions) <= $n
return $stringExcl
}
</fts:match>
}
</fts:allMatches>
};
The semantics of sentence distance from M to N
is given below.
declare function fts:ApplyFTSentenceDistanceFromTo (
$matchOptions as element(fts:matchOptions),
$allMatches as element(fts:allMatches),
$m as xs:integer,
$n as xs:integer )
as element(fts:allMatches)
{
<fts:allMatches stokenNum="{$allMatches/@stokenNum}">
{
for $match in $allMatches/fts:match
let $sorted := for $si in $match/fts:stringInclude
order by $si/fts:tokenInfo/@startPos ascending
return $si
where
if (fn:count($sorted) le 1) then fn:true() else
every $index in (1 to fn:count($sorted) - 1)
satisfies fts:sentenceDistance(
$sorted[$index]/fts:tokenInfo,
$sorted[$index+1]/fts:tokenInfo,
$matchOptions) >= $m
and
fts:sentenceDistance(
$sorted[$index]/fts:tokenInfo,
$sorted[$index+1]/fts:tokenInfo,
$matchOptions) <= $n
return
<fts:match>
{
$match/fts:stringInclude,
for $stringExcl in $match/fts:stringExclude
where some $stringIncl in $match/fts:stringInclude
satisfies fts:sentenceDistance(
$stringIncl/fts:tokenInfo,
$stringExcl/fts:tokenInfo,
$matchOptions) >= $m
and
fts:sentenceDistance(
$stringIncl/fts:tokenInfo,
$stringExcl/fts:tokenInfo,
$matchOptions) <= $n
return $stringExcl
}
</fts:match>
}
</fts:allMatches>
};
The semantics of paragraph distance exactly N
is given below.
declare function fts:ApplyFTParagraphDistanceExactly (
$matchOptions as element(fts:matchOptions),
$allMatches as element(fts:allMatches),
$n as xs:integer )
as element(fts:allMatches)
{
<fts:allMatches stokenNum="{$allMatches/@stokenNum}">
{
for $match in $allMatches/fts:match
let $sorted := for $si in $match/fts:stringInclude
order by $si/fts:tokenInfo/@startPos ascending
return $si
where
if (fn:count($sorted) le 1) then fn:true() else
every $index in (1 to fn:count($sorted) - 1)
satisfies fts:paraDistance(
$sorted[$index]/fts:tokenInfo,
$sorted[$index+1]/fts:tokenInfo,
$matchOptions) = $n
return
<fts:match>
{
$match/fts:stringInclude,
for $stringExcl in $match/fts:stringExclude
where some $stringIncl in $match/fts:stringInclude
satisfies fts:paraDistance(
$stringIncl/fts:tokenInfo,
$stringExcl/fts:tokenInfo,
$matchOptions) = $n
return $stringExcl
}
</fts:match>
}
</fts:allMatches>
};
The semantics of paragraph distance at least N
is given below.
declare function fts:ApplyFTParagraphDistanceAtLeast (
$matchOptions as element(fts:matchOptions),
$allMatches as element(fts:allMatches),
$n as xs:integer )
as element(fts:allMatches)
{
<fts:allMatches stokenNum="{$allMatches/@stokenNum}">
{
for $match in $allMatches/fts:match
let $sorted := for $si in $match/fts:stringInclude
order by $si/fts:tokenInfo/@startPos ascending
return $si
where
if (fn:count($sorted) le 1) then fn:true() else
every $index in (1 to fn:count($sorted) - 1)
satisfies fts:paraDistance(
$sorted[$index]/fts:tokenInfo,
$sorted[$index+1]/fts:tokenInfo,
$matchOptions) >= $n
return
<fts:match>
{
$match/fts:stringInclude,
for $stringExcl in $match/fts:stringExclude
where some $stringIncl in $match/fts:stringInclude
satisfies fts:paraDistance(
$stringIncl/fts:tokenInfo,
$stringExcl/fts:tokenInfo,
$matchOptions) >= $n
return $stringExcl
}
</fts:match>
}
</fts:allMatches>
};
The semantics of paragraph distance at most N
is given below.
declare function fts:ApplyFTParagraphDistanceAtMost (
$matchOptions as element(fts:matchOptions),
$allMatches as element(fts:allMatches),
$n as xs:integer )
as element(fts:allMatches)
{
<fts:allMatches stokenNum="{$allMatches/@stokenNum}">
{
for $match in $allMatches/fts:match
let $sorted := for $si in $match/fts:stringInclude
order by $si/fts:tokenInfo/@startPos ascending
return $si
where
if (fn:count($sorted) le 1) then fn:true() else
every $index in (1 to fn:count($sorted) - 1)
satisfies fts:paraDistance(
$sorted[$index]/fts:tokenInfo,
$sorted[$index+1]/fts:tokenInfo,
$matchOptions) <= $n
return
<fts:match>
{
$match/fts:stringInclude,
for $stringExcl in $match/fts:stringExclude
where some $stringIncl in $match/fts:stringInclude
satisfies fts:paraDistance(
$stringIncl/fts:tokenInfo,
$stringExcl/fts:tokenInfo,
$matchOptions) <= $n
return $stringExcl
}
</fts:match>
}
</fts:allMatches>
};
The semantics of paragraph distance from M to N
is given below.
declare function fts:ApplyFTParagraphDistanceFromTo (
$matchOptions as element(fts:matchOptions),
$allMatches as element(fts:allMatches),
$m as xs:integer,
$n as xs:integer )
as element(fts:allMatches)
{
<fts:allMatches stokenNum="{$allMatches/@stokenNum}">
{
for $match in $allMatches/fts:match
let $sorted := for $si in $match/fts:stringInclude
order by $si/fts:tokenInfo/@startPos ascending
return $si
where
if (fn:count($sorted) le 1) then fn:true() else
every $index in (1 to fn:count($sorted) - 1)
satisfies fts:paraDistance(
$sorted[$index]/fts:tokenInfo,
$sorted[$index+1]/fts:tokenInfo,
$matchOptions) >= $m
and
fts:paraDistance(
$sorted[$index]/fts:tokenInfo,
$sorted[$index+1]/fts:tokenInfo,
$matchOptions) <= $n
return
<fts:match>
{
$match/fts:stringInclude,
for $stringExcl in $match/fts:stringExclude
where some $stringIncl in $match/fts:stringInclude
satisfies fts:paraDistance(
$stringIncl/fts:tokenInfo,
$stringExcl/fts:tokenInfo,
$matchOptions) >= $m
and
fts:paraDistance(
$stringIncl/fts:tokenInfo,
$stringExcl/fts:tokenInfo,
$matchOptions) <= $n
return $stringExcl
}
</fts:match>
}
</fts:allMatches>
};
The resulting AllMatches contains
Matches of the operand that satisfy the condition that
the distance
for every pair of consecutive StringIncludes is within the specified interval,
where the distance is measured in tokens, sentences, or paragraphs from the end
of the preceding StringInclude to the start of the next.
In the general case, the semantics is given below.
declare function fts:ApplyFTDistance (
$matchOptions as element(fts:matchOptions),
$type as fts:DistanceType,
$range as element(fts:range),
$allMatches as element(fts:allMatches) )
as element(fts:allMatches)
{
if ($type eq "word") then
if ($range/@type eq "exactly") then
fts:ApplyFTWordDistanceExactly($matchOptions,
$allMatches,
$range/@n)
else if ($range/@type eq "at least") then
fts:ApplyFTWordDistanceAtLeast($matchOptions,
$allMatches,
$range/@n)
else if ($range/@type eq "at most") then
fts:ApplyFTWordDistanceAtMost($matchOptions,
$allMatches,
$range/@n)
else fts:ApplyFTWordDistanceFromTo($matchOptions,
$allMatches,
$range/@m,
$range/@n)
else if ($type eq "sentence") then
if ($range/@type eq "exactly") then
fts:ApplyFTSentenceDistanceExactly($matchOptions,
$allMatches,
$range/@n)
else if ($range/@type eq "at least") then
fts:ApplyFTSentenceDistanceAtLeast($matchOptions,
$allMatches,
$range/@n)
else if ($range/@type eq "at most") then
fts:ApplyFTSentenceDistanceAtMost($matchOptions,
$allMatches,
$range/@n)
else fts:ApplyFTSentenceDistanceFromTo($matchOptions,
$allMatches,
$range/@m,
$range/@n)
else if ($range/@type eq "exactly") then
fts:ApplyFTParagraphDistanceExactly($matchOptions,
$allMatches,
$range/@n)
else if ($range/@type eq "at least") then
fts:ApplyFTParagraphDistanceAtLeast($matchOptions,
$allMatches,
$range/@n)
else if ($range/@type eq "at most") then
fts:ApplyFTParagraphDistanceAtMost($matchOptions,
$allMatches,
$range/@n)
else fts:ApplyFTParagraphDistanceFromTo($matchOptions,
$allMatches,
$range/@m,
$range/@n)
};
For example, consider the FTDistance selection
("Ford Mustang" &&
"excellent") distance at most 3 words.
The Matches of the source AllMatches for
("Ford Mustang" &&
"excellent") are given below.
![FTDistance input AllMatches]()
Continued on next diagram
![FTDistance input AllMatches]()
Continued on next diagram
![FTDistance input AllMatches]()
Continued on next diagram
![FTDistance input AllMatches]()
Continued on next diagram
![FTDistance input AllMatches]()
Continued on next diagram
![FTDistance input AllMatches]()
The result for the FTDistance selection
consists of only the first Match (with positions 1, 2, and 5), because only
for this Match the word distance between consecutive TokenInfos is always less than or equal to 3. It is 1 for the first pair and 3 for the second.
FTWindow
The parameters of the
ApplyFTWindow function are 1) the
search context, 2) the list of match options, 3) the unit of
type fts:DistanceType, 4) a size, and 5) one
AllMatches parameter
corresponding to the result of the nested FTSelections.
The search context is not used by this function. For each unit
type a function is defined as follows.
The semantics of window N words is given below.
declare function fts:ApplyFTWordWindow (
$matchOptions as element(fts:matchOptions),
$allMatches as element(fts:allMatches),
$n as xs:integer )
as element(fts:allMatches)
{
<fts:allMatches stokenNum="{$allMatches/@stokenNum}">
{
for $match in $allMatches/fts:match
let $minpos := fn:min($match/*/fts:tokenInfo/@startPos),
$maxpos := fn:max($match/*/fts:tokenInfo/@endPos)
for $windowStartPos in ($minpos to $maxpos - $n + 1)
let $windowEndPos := $windowStartPos + $n - 1
where fn:min($match/fts:stringInclude/fts:tokenInfo/@startPos)
>= $windowStartPos
and fn:max($match/fts:stringInclude/fts:tokenInfo/@endPos)
<= $windowEndPos
return
<fts:match>
{
$match/fts:stringInclude,
for $stringExclude in $match/fts:stringExclude
where $stringExclude/fts:tokenInfo/@startPos >= $windowStartPos
and $stringExclude/fts:tokenInfo/@endPos <= $windowEndPos
return $stringExclude
}
</fts:match>
}
</fts:allMatches>
};
The semantics of window N sentences is given below.
declare function fts:ApplyFTSentenceWindow (
$matchOptions as element(fts:matchOptions),
$allMatches as element(fts:allMatches),
$n as xs:integer )
as element(fts:allMatches)
{
<fts:allMatches stokenNum="{$allMatches/@stokenNum}">
{
for $match in $allMatches/fts:match
let $minpos := fn:min($match/*/fts:tokenInfo/@startSent),
$maxpos := fn:max($match/*/fts:tokenInfo/@endSent)
for $windowStartPos in ($minpos to $maxpos - $n + 1)
let $windowEndPos := $windowStartPos + $n - 1
where fn:min($match/fts:stringInclude/fts:tokenInfo/@startSent)
>= $windowStartPos
and fn:max($match/fts:stringInclude/fts:tokenInfo/@endSent)
<= $windowEndPos
return
<fts:match>
{
$match/fts:stringInclude,
for $stringExclude in $match/fts:stringExclude
where $stringExclude/fts:tokenInfo/@startSent >= $windowStartPos
and $stringExclude/fts:tokenInfo/@endSent <= $windowEndPos
return $stringExclude
}
</fts:match>
}
</fts:allMatches>
};
The semantics of word N paragraphs is given below.
declare function fts:ApplyFTParagraphWindow (
$matchOptions as element(fts:matchOptions),
$allMatches as element(fts:allMatches),
$n as xs:integer )
as element(fts:allMatches)
{
<fts:allMatches stokenNum="{$allMatches/@stokenNum}">
{
for $match in $allMatches/fts:match
let $minpos := fn:min($match/*/fts:tokenInfo/@startPara),
$maxpos := fn:max($match/*/fts:tokenInfo/@endPara)
for $windowStartPos in ($minpos to $maxpos - $n + 1)
let $windowEndPos := $windowStartPos + $n - 1
where fn:min($match/fts:stringInclude/fts:tokenInfo/@startPara)
>= $windowStartPos
and fn:max($match/fts:stringInclude/fts:tokenInfo/@endPara)
<= $windowEndPos
return
<fts:match>
{
$match/fts:stringInclude,
for $stringExclude in $match/fts:stringExclude
where $stringExclude/fts:tokenInfo/@startPara >= $windowStartPos
and $stringExclude/fts:tokenInfo/@endPara <= $windowEndPos
return $stringExclude
}
</fts:match>
}
</fts:allMatches>
};
The resulting AllMatches contains
Matches of the operand that satisfy the condition that
there exists a sequence of the specified number of
consecutive (token, sentence, or paragraph) positions, such
that all StringIncludes are within that window, and the
StringExcludes retained are also within that window.
The semantics for the general function is given below.
declare function fts:ApplyFTWindow (
$matchOptions as element(fts:matchOptions),
$type as fts:DistanceType,
$size as xs:integer,
$allMatches as element(fts:allMatches) )
as element(fts:allMatches)
{
if ($type eq "word") then
fts:ApplyFTWordWindow($matchOptions,
$allMatches,
$size)
else if ($type eq "sentence") then
fts:ApplyFTSentenceWindow($matchOptions,
$allMatches,
$size)
else
fts:ApplyFTParagraphWindow($matchOptions,
$allMatches,
$size)
};
For example, consider the FTWindow selection
("Ford Mustang" &&
"excellent") window 10 words.
The Matches of the source AllMatches for
("Ford Mustang" &&
"excellent") are given below.
![FTWindow AllMatches]()
Continued on next diagram
![FTWindow AllMatches]()
Continued on next diagram
![FTWindow AllMatches]()
Continued on next diagram
![FTWindow AllMatches]()
Continued on next diagram
![FTWindow AllMatches]()
Continued on next diagram
![FTWindow AllMatches]()
The result for the FTWindow selection
consists of only the first, the fifth, and the sixth
Matches because their respective window sizes are 5, 4,
and 9.
FTTimes
The parameters of the ApplyFTTimes
function are 1)an FTRange specification, and 2)
parameter corresponding to the result of the nested
FTWords.
The function definitions depend on the range
specification FTRange to limit the number of
occurrences.
The general semantics is given below.
declare function fts:FormCombinations (
$sms as element(fts:match)*,
$times as xs:integer )
as element(fts:match)*
{
if ( $times eq 1 ) then $sms
else if (fn:count($sms) lt $times) then ()
else if (fn:count($sms) eq $times) then
<fts:match>{$sms/*}</fts:match>
else (
fts:FormCombinations(fn:subsequence($sms, 2), $times),
for $combination in
fts:FormCombinations(fn:subsequence($sms, 2), $times - 1)
return
<fts:match>
{
$sms[1]/*,
$combination/*
}
</fts:match>
)
};
declare function fts:FormRange (
$sms as element(fts:match)*,
$l as xs:integer,
$u as xs:integer,
$stokenNum as xs:integer )
as element(fts:allMatches)
{
if ($l > $u) then ()
else
let $am1 := <fts:allMatches stokenNum="{$stokenNum}">
{fts:FormCombinations($sms, $l)}
</fts:allMatches>
let $am2 := <fts:allMatches stokenNum="{$stokenNum}">
{fts:FormCombinations($sms, $u+1)}
</fts:allMatches>
return fts:ApplyFTAnd($am1,
fts:ApplyFTUnaryNot($am2))
};
The semantics of occurs exactly N times is given
below.
declare function fts:ApplyFTTimesExactly (
$allMatches as element(fts:allMatches),
$n as xs:integer )
as element(fts:allMatches)
{
fts:FormRange($allMatches/match, $n, $n, $allMatches/@stokenNum)
};
The semantics of occurs at least N times is given below.
declare function fts:ApplyFTTimesAtLeast (
$allMatches as element(fts:allMatches),
$n as xs:integer )
as element(fts:allMatches)
{
<fts:allMatches stokenNum="{$allMatches/@stokenNum}">
{fts:FormCombinations($allMatches/fts:match, $n)}
</fts:allMatches>
};
The semantics of occurs at most N times is given
below.
declare function fts:ApplyFTTimesAtMost (
$allMatches as element(fts:allMatches),
$n as xs:integer )
as element(fts:allMatches)
{
fts:FormRange($allMatches/fts:match, 0, $n, $allMatches/@stokenNum)
};
The semantics of occurs from M to N times is given below.
declare function fts:ApplyFTTimesFromTo (
$allMatches as element(fts:allMatches),
$m as xs:integer,
$n as xs:integer )
as element(fts:allMatches)
{
fts:FormRange($allMatches/fts:match, $m, $n, $allMatches/@stokenNum)
};
The way to ensure that
there are at least N different matches of an
FTSelection is to ensure that at least N of
its Matches occur simultaneously. This is similar to
forming their conjunction by combining N distinct
Matches into one simple match. Therefore, the AllMatches
for the selection condition specifying the range qualifier
at least N contains the possible
combinations of N simple matches of the
operand and one Match for each combination
negating the rest of the simple matches. This operations
is performed in the function
fts:FormCombinations.
The range [l, u] is represented by the condition
at least l and not at least l+1.This transformation
is performed in the function
fts:FormRange.
The semantics for the general case is given below.
declare function fts:ApplyFTTimes (
$range as element(fts:range),
$allMatches as element(fts:allMatches) )
as element(fts:allMatches)
{
if (fn:count($allMatches//fts:stringExclude) gt 0) then
fn:error(fn:QName('http://www.w3.org/2005/xqt-errors', 'err:XPST0003'))
else if ($range/@type eq "exactly") then
fts:ApplyFTTimesExactly($allMatches, $range/@n)
else if ($range/@type eq "at least") then
fts:ApplyFTTimesAtLeast($allMatches, $range/@n)
else if ($range/@type eq "at most") then
fts:ApplyFTTimesAtMost($allMatches, $range/@n)
else fts:ApplyFTTimesFromTo($allMatches,
$range/@m,
$range/@n)
};
The above function performs a sanity check to ensure that the nested
AllMatches is a result of the evaluation of FTWords as defined in
the grammar.
FTWordsSelection(FTWords FTTimes?) | ("(" FTSelection ")")
Otherwise, an error
is raised.
For example, consider the FTTimes selection
"Mustang" occurs at least 2 times. The source
AllMatches of the FTWords
selection "Mustang" is given below.
![FTTimes input AllMatches]()
The result consists of the pairs of the Matches.
![FTTimes result AllMatches]()