Re: [Boost-users] Could searching and indexing Boost docs mightwork better?

Subject: Re: [Boost-users] Could searching and indexing Boost docs mightwork better?
From: Paul A. Bristow (pbristow_at_[hidden])
Date: 2010-09-20 05:12:55

> -----Original Message-----
> From: boost-users-bounces_at_[hidden] [mailto:boost-users-
> bounces_at_[hidden]] On Behalf Of Cory Nelson
> Sent: Monday, September 20, 2010 9:53 AM
> To: boost-users_at_[hidden]
> Subject: Re: [Boost-users] Could searching and indexing Boost docs mightwork
> better?
> 
> On Mon, Sep 20, 2010 at 12:57 AM, John Maddock <boost.regex_at_[hidden]>
> wrote:
> >> Does anyone on this list have (or have a friend with) SEO expertise?
> >> They would know how to address this. Otherwise, Paul, maybe you could
> >> try to ask Google themselves?
> >
> >> The problem is that bots are currently blocked from accessing the
> >> documentation pages because they are considered too expensive. I've
> >> reduced the cost of serving the documentation pages, but apparently
> >> that's not good enough because the regression test results are still
> >> served from zipfiles. Someone needs to sort that out with whoever
> >> runs the regression testing system.
> >
> > Surely the test result pages aren't accessed that much?  As long as
> > they're blocked from the bots they shouldn't take up too much CPU time
> > even when zipped?
> >
> > IMO it's unacceptible not to have the Boost documentation indexed by
> > the web crawlers - would it be possible to allow indexing of current
> > release *only* - and see how that goes?

I agree that this should be the thing we try to achieve immediately.
 
> A single sitemap would work for all the big search engines.  A sitemap lets you
> specify how often something is updated -- their spiders should index it far less
> often if you specify a long term.
> 
> You might be able to do the same thing with HTTP cache headers.
> Depending on your web server you might be able to configure entire subdirs to
> cache until the end of time.

Long term, site maps looks the *right* way to do this - but it would best be done by someone who knows what they are doing (definitely NOT me!!).

http://en.wikipedia.org/wiki/Site_map

http://www.sitemaps.org/

Shall I start a new thread on boost-users asking for help on generating a site map?

Paul

---
Paul A. Bristow,
Prizet Farmhouse, Kendal LA8 8AB  UK
[phone]  07714330204
pbristow_at_[hidden]