Sites are Different

Current engines only allow sites be either fully opened, or full closed for indexing. With an intermediate fetching level, we can add more features.

We can allow sites to report changes in pages by themselves. For instance, if you have a site with 100,000 static pages, then it is not pleasant to have an engine checking all those pages every day. Better is to install a small software package (for instance part of the Apache server) which locally detects changes and reports them to the spider on its own initiative.

In this model, even sites which require registration or payment can be indexed. With little effort, they can post lists of words to the fetcher, which will report these to the spiders.