Rory Ward, Telescript Bard, rory_ward@genmagic.com
Barry Friedman, Code Dependent, barry@genmagic.com
General Magic Inc: http://www.genmagic.com
Submitted to the W3C Workshop on Distributed Indexing/Searching.

Web Indexing Using Telescript

Telescript is an open, object-oriented, remote programming language. It is a platform that enables the creation of active, distributed network applications. There are three simple concepts to the language: agents, places and "go". Agents "go" to places, where they interact with the place or other agents to get work done on a user's behalf. Agents are in fact mobile programs capable of transporting themselves from place to place in a Telescript network. Telescript contains mechanisms to ensure the safety and security of mobile programs and the servers that host them.

Telescript documentation and software is available at General Magic's web site. General Magic is proposing a Common Agent Platform that would allow the development of interoperable agents using a variety of programming languages. See the position paper at http://www.genmagic.com/internet/cap/w3c-paper.htm for more information.

Telescript technology offers the ability to distribute indexing to the sites where the content resides. A search engine operator can develop an agent that can travel to the content site, build an index that meets their needs, and return with the index information.

Each Telescripted web site implements an open, standardized programmatic interface for agents to access local web content. The resident agents do their work autonomously via this API. Their resource consumption and content access is controlled by the site using facilities provided in the Telescript language. Any indexing agent developer can take advantage of the programmatic facilities at Telescripted web sites.

An indexing agent can remain at the web site that is hosting it, and send update agents to its origin when the content is changed (modified, added or removed). The site operator can control the the rate and size of updates.

As Telescripted Web sites become available, they advertise themselves to a central directory. The directory can be used by agent developers to dynamically discover new Telescripted Web sites.

Advantages of this approach

• Scales very well: The resources needed to index web content is spread between the content providers.
• Always up to date: Agents stay at web sites and update the central repository when local content changes. These updates can be scheduled by the agent developer or by the web site operator.
• Only modified content is reindexed: Because agents remain local to the content, they only reindex pages that have changed. Therefore, after sending the initial index, only deltas need to be transmitted.
• Decreased load on the Internet: Because agents are close to the resources they need, there is no need to transmit every page across the network. Only the index data needed by the agent's developer needs to get transmitted.
• Robot access differentiated from browsing: Because indexers are not using HTTP to index the web, robot access can be differentiated from browsing. This helps the site operators and their advertisers.
• Site configurable and controllable: The site operator can control the resources used by agents and the content accessed. The site can restrict who can send agents to the site and what security policies are applied.
• Open platform for indexing: Any index agent can take advantage of the programmatic interface available at a web site. This allows for different indexing algorithms to be employed.
• Dynamic discovery of Active Web Sites: A central directory of active web sites that is always up to date allows indexing agents to dynamically discover new sites.
• Avoids copyright issues: Accomplishing the indexing on the local server under the web site owner's control avoids copyright violations and related liabilities for the crawler operator.
• Multiple varieties of architectures: As personal web servers and other kinds of "low-level" sites proliferate and become ubiquitous, it seems likely that indexing hierarchies will arise. Using Telescript agents inherently allows for the kinds of flexibility necessary to effect and maintain such a distributed autonomous hierarchical architecture.