The First Web, and Beyond

Santiago Ortiz, Data Scientist

Web Crawlers

A Web crawler is a bot that, starting with a list of one or more Websites (seed), downloads them and adds all their links to the same list, and continues downloading and adding links, probably without encountering any limit (Websites are created faster than a bot’s speed).

Crawlers are exploration algorithms that start in tiny regions of a vast space and move blindly in all possible directions. They are a good metaphor for human pursuit of knowledge. Bots were critical in the early years of the Web, since they were the only way to know what was out there and the only method to index all (accessible) sites.

The First Web, and Beyond

It makes sense (at least poetic sense) to start exploring the entire Web beginning with the first Website ever created, published by Tim Berners-Lee in 1992. In a way, this is a journey that starts in the past and moves toward the present. This project, commissioned by The Webby Awards, is an homage to the Web.