Re: [Boost-users] Could searching and indexing Boost docs mightwork better?

Subject: Re: [Boost-users] Could searching and indexing Boost docs mightwork better?
From: John Maddock (boost.regex_at_[hidden])
Date: 2010-09-20 03:57:59

> Does anyone on this list have (or have a friend with) SEO expertise? They 
> would know how to address this. Otherwise,
> Paul, maybe you could try to ask Google themselves?

>The problem is that bots are currently blocked from accessing the
>documentation pages because they are considered too expensive. I've
>reduced the cost of serving the documentation pages, but apparently
>that's not good enough because the regression test results are still
>served from zipfiles. Someone needs to sort that out with whoever runs
>the regression testing system.

Surely the test result pages aren't accessed that much?  As long as they're 
blocked from the bots they shouldn't take up too much CPU time even when 
zipped?

IMO it's unacceptible not to have the Boost documentation indexed by the web 
crawlers - would it be possible to allow indexing of current release 
*only* - and see how that goes?

Cheers, John.