Copyright © 2012 W3C® (MIT, ERCIM, Keio), All Rights Reserved. W3C liability, trademark and document use rules apply.
This document specifies requirements and use cases for Full-Text Search for use in XQuery 3.0 [XQuery 3.0: An XML Query Language] and XPath 3.0 [XML Path Language (XPath) 3.0].
This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.
This is a First Public Working Draft as described in the Process Document. It was jointly developed by the W3C XML Query Working Group and the W3C XSL Working Group, each of which is part of the XML Activity. This document will eventually be published as a Working Group Note to persistently record the Requirements that guided the development of XQuery and XPath Full Text 3.0 as a W3C Recommendation.
This document includes, for each requirement, a corresponding status, indicating the current situation of the requirement in XQuery and XPath Full Text 3.0 at the time that the spec was most recently published on 13 December 2011. Organizations and individuals should review this document to determine whether or not the requirements provided meet the needs of the full-text community. If additional requirements are identified, they may be added to these requirements in a future publication.
A future publication of this document will incorporate a number of Use Cases that assist the Working Groups in determining whether a candidate requirement is, in fact, a real requirement and illustrating various problems that XQuery and XPath Full Text 3.0 is intended to address.
Please report errors in this document using W3C's public Bugzilla system (instructions can be found at http://www.w3.org/XML/2005/04/qt-bugzilla). If access to that system is not feasible, you may send your comments to the W3C XSLT/XPath/XQuery public comments mailing list, public-qt-comments@w3.org. It will be very helpful if you include the string “[FT30req]” in the subject line of your report, whether made in Bugzilla or in email. Please use multiple Bugzilla entries (or, if necessary, multiple email messages) if you have more than one comment to make. Archives of the comments and responses are available at http://lists.w3.org/Archives/Public/public-qt-comments/.
Publication as a Working Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.
This document was produced by groups operating under the 5 February 2004 W3C Patent Policy. W3C maintains a public list of any patent disclosures made in connection with the deliverables of the XML Query Working Group and also maintains a public list of any patent disclosures made in connection with the deliverables of the XSL Working Group; those pages also include instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy.
1 Goals
2 Requirements
2.1 Terminology
2.2 General
Requirements
2.2.1 Backward compatibility
2.2.2 Extension compatibility
2.3 Full Text Search
Functionality
2.3.1 Feature names for XQuery require-feature and
prohibit-feature
2.3.2 Language Identifiers
2.3.3 Match highlighting
2.3.4 Optional URI for stemming algorithms
2.3.5 Refine language about levels in FTThesaurus
Option
2.3.6 Tokenize items
2.3.7 Count occurrences of search terms
2.3.8 Return score values
2.3.9 Search on punctuation
2.3.10 Snippets
2.4 Editorial
Changes
2.4.1 Irrevocable Stop Words
2.4.2 Rename TokenInfo
A References
A.1 Non-Normative
The goal of XQuery and XPath Full Text 3.0 is to extend XQuery and XPath Full Text 1.0 with additional functionality in response to requests from users and implementors.
The following key words are used throughout the document to specify the extent to which an item is a requirement for the work of the XML Query Working Group:
This word means that the item is an absolute requirement.
This word means that there may exist valid reasons not to treat this item as a requirement, but the full implications should be understood and the case carefully weighed before discarding this item.
This word means that an item deserves attention, but further study is needed to determine whether the item should be treated as a requirement.
When the words MUST, SHOULD, or MAY are used in this technical sense, they occur as a hyperlink to these definitions. These words will also be used with their conventional English meaning, in which case there is no hyperlink. For instance, the phrase "the full implications should be understood" uses the word "should" in its conventional English sense, and therefore occurs without the hyperlink.
Each requirement also includes a status section, indicating its current situation in the XML-Query family of specifications. Three status levels are available:
This indicates that the requirement, according to
its original formulation, has been completely met. Optional
clarificatory text may follow.
This indicates that the requirement has been
partially met according to its original formulation. When this
happens, explanatory text is provided to better clarify the current
scope of the requirement.
This indicates that the requirement, according to
its original formulation, has not been met. If this is the case,
explanatory text is provided.
XQuery and XPath Full Text 3.0 MUST be compatible with XQuery and XPath 3.0 extensions developed by the XML Query Working Group and the XSL Working Group.
Status: this requirement has been
met.
XQuery and XPath Full Text 3.0 MUST add feature names for XQuery require-feature and prohibit-feature to include the names defined in the "http://www.w3.org/2011/xquery-features" namespace.
Status: this requirement has been
met.
XQuery and XPath Full Text 3.0 MUST specify in the Language Option how to handle multiple language identifiers for the same language, including languages represented by both two- and three- letter identifiers.
Status: this requirement has been
met.
XQuery and XPath Full Text 3.0 MUST support adding marker elements around token matches. How the element will be marked MAY be specified by an EQName or a function item as argument.
Status: this requirement has not been met.
The Working Group has yet to decide whether it will be a
requirement for this or any other version.
XQuery and XPath Full Text 3.0 MUST define an optional URI to identify stemming algorithms. Issue raised in Bugzilla Bug 9680.
Status: this requirement has not been met.
The Working Group has yet to decide whether it will be a
requirement for this or any other version.
XQuery and XPath Full Text 3.0 MUST refine language about levels in FTThesaurus Option. Issue raised in Bugzilla Bug 11444.
Status: this requirement has not been met.
The Working Group has yet to decide whether it will be a
requirement for this or any other version.
XQuery and XPath Full Text 3.0 MUST support explicitly tokenizing an item and returning a sequence of strings.
Status: this requirement has not been met.
The Working Group has yet to decide whether it will be a
requirement for this or any other version.
XQuery and XPath Full Text 3.0 MUST enable counting the number of occurrences of search terms specified in a full-text expression.
Status: this requirement has not been met.
The Working Group has yet to decide whether it will be a
requirement for this or any other version.
XQuery and XPath Full Text 3.0 MUST allow explicit access to score values that have been assigned to items by an FTContains expression, making the score values available for merging and other computations.
Status: this requirement has not been met.
The Working Group has yet to decide whether it will be a
requirement for this or any other version.
XQuery and XPath Full Text 3.0 MUST support searching on punctuation. For example, searching on tokens that contain punctuation such as PB&J and document.xml.
Status: this requirement has not been met.
The Working Group has yet to decide whether it will be a
requirement for this or any other version.
XQuery and XPath Full Text 3.0 MUST support displaying snippets (match tokens in context). Snippets are brief segments of text surrounding matches displayed in search results to enable the user to better judge the usefulness of a search result.
Status: this requirement has not been met.
The Working Group has yet to decide whether it will be a
requirement for this or any other version.
XQuery and XPath Full Text 3.0 MUST specify in the Stop Word Option that implementations may apply stop word lists during indexing and be unable to comply with query-time requests to not apply those stop words.
Status: this requirement has been
met.
XQuery and XPath Full Text 3.0 MUST rename TokenInfo to something like TokenSpan or TokenRange. Issue raised in Bugzilla Bug 9541.
Status: this requirement has not been met.
The Working Group has yet to decide whether it will be a
requirement for this or any other version.