Re: [Boost-users] Could searching and indexing Boost docs mightwork better?

Subject: Re: [Boost-users] Could searching and indexing Boost docs mightwork better?
From: Cory Nelson (phrosty_at_[hidden])
Date: 2010-09-20 04:53:07

On Mon, Sep 20, 2010 at 12:57 AM, John Maddock <boost.regex_at_[hidden]> wrote:
>> Does anyone on this list have (or have a friend with) SEO expertise? They
>> would know how to address this. Otherwise,
>> Paul, maybe you could try to ask Google themselves?
>
>> The problem is that bots are currently blocked from accessing the
>> documentation pages because they are considered too expensive. I've
>> reduced the cost of serving the documentation pages, but apparently
>> that's not good enough because the regression test results are still
>> served from zipfiles. Someone needs to sort that out with whoever runs
>> the regression testing system.
>
> Surely the test result pages aren't accessed that much? Â As long as they're
> blocked from the bots they shouldn't take up too much CPU time even when
> zipped?
>
> IMO it's unacceptible not to have the Boost documentation indexed by the web
> crawlers - would it be possible to allow indexing of current release *only*
> - and see how that goes?
>
> Cheers, John.

A single sitemap would work for all the big search engines.  A sitemap
lets you specify how often something is updated -- their spiders
should index it far less often if you specify a long term.

You might be able to do the same thing with HTTP cache headers.
Depending on your web server you might be able to configure entire
subdirs to cache until the end of time.

-- 
Cory Nelson
http://int64.org