Layering the Indexing System

In the first part of the presentation, I spoke about the top-level only: the user interface. In the follow slides, I focus on the lowest level: the page fetcher.

Every search-engine, nowadays, has to scan the Net by itself to look for new and updated pages. Scanning takes a lot of time, while most pages hardly every change. It would be nice to combine this work done by all engines to one generalized system.