Formal Standards for Meta-Search
May 6, 1996
Meta-search engines, tools that simultaneously search multiple conventional search engines and integrate the results, are becoming increasingly popular. Today at least three meta-search engines are in wide use: SavvySearch, MetaCrawler and WebCompass; many more are in development. Users report that these tools are very helpful in their Web navigation endeavors. In some cases, the search engines that are queried by meta-search engines find this behavior beneficial. Lesser known search engines enjoy the publicity and extra awareness that the meta-search tools raise. Meta-search engines also serve as additional entrances into sites whose search engines index local content only.
In other cases, as recently suggested on the robots mailing list, meta-search engines appear to work against the advertiser supported business model adopted by some of the larger search sites. Related problems that have surfaced are the increased strain on the Internet and various search engines, and reformatting of results. One solution to the advertising problem that has been suggested is propagation of advertisements produced by search engines into the meta-search results. Another solution might involve intermediate result pages which give search engines an opportunity to display advertisements for each of their links that is followed.
Ultimately it should be up to search engine providers to decide how and under what conditions their resources are used, and each will probably have a unique opinion. Perhaps these problems are best addressed with the introduction of a formal standard for meta-search tools. A standard for meta-searchers could exist as an extension to the existing robot exclusion standard, or as an entirely new mechanism (how about SavvyNotWanted.txt?) Below is a partial list of questions that I believe should be considered when designing such a standard:
This list has probably overlooked some important issues. The next step is consulting the many search engine providers and identifying their concerns.