Fifth International World Wide Web Conference

May 6-10, 1996, Paris, France

Panel10: Efficiency of Internet Indexing

Thursday 9 May, 1996 - 11:30-13:00

Panel Chair

Nick Arnett
Internet Evangelist
Verity Inc.

Panel Members

Brief Outline

The proliferation of Web robots has led to increasingly redundant indexing, unnecessarily tying up CPU cycles and bandwidth. The Harvest project at the University of Colorado developed a more efficient system (presented at WWW2), but Harvest thus far has failed to achieve widespread adoption; it is not used by any of the major Web search services. "Robot wars" loom in the future, as webmasters and indexers fight over increasingly saturated resources. This is an example of the need for "cooper-tition" -- erstwhile competing developers must cooperate to allow the technology to grow to the next step.

The future direction of this technology is only clear in a negative sense -- the inefficiency of today's non-cooperative robots is the wrong direction. This panel will discuss Harvest and more recent projects that would address this issue. Lively discussion should result as the panelists attempt to reconcile the need for efficiency with the demands of competition.

Related topics will also be covered: standards to allow robots to identify and skip multiple copies of documents, aliases and symbolic links; requirements for robots on heavily trafficked enterprise networks.

mailto Mail to the Organizers

Created: 9 April 1996
Last updated: 26 April 1996