Re: [boost] boost and google search

Subject: Re: [boost] boost and google search
From: Rene Rivera (grafikrobot_at_[hidden])
Date: 2011-12-21 00:19:38

On 12/20/2011 2:20 PM, Daniel James wrote:
> On 19 December 2011 05:32, Rene Rivera<grafikrobot_at_[hidden]>  wrote:
>>
>>
>> I've just put in the change for #1 that I mentioned (the redirect codes).
>
> It was using 302, which as far as I know is the right code to use.
> Google certainly did the right thing ('boost 1.48' was appearing for
> release urls pretty soon).

Hm? I thought it was returning the default of "301" for those. But 
reading the Apache docs now tells me I was wrong in that belief :-\ But 
reading the HTTP spec, and the commentary about what browsers likely do 
and should do.. It seems like 303 is a better choice for clients instead 
of 302 while having the same effect on crawlers. Particularly the 
difference about 302 not "technically" allowing the caching of the 
target page. While the 303 explicitly allows the caching of the target. 
Of course, it also says that most clients treat 302 as if it where a 303 
anyway. But it seems that being explicit here covers all the clients.

>> I'll see about crawling the old docs soon.
>
> I don't think it's a good idea to block them. It can be useful to
> search for old documentation and often if you block google's bot it
> will still return links to those pages based on the pages which link
> to them (it did this for the documentation when it was blocked
> before).

Good point.

> The correct thing to do is to structure the site so that it directs
> google (and users) to the more useful pages. I've been working on
> this, and although no one seems to have noticed,

For what it's worth.. I did notice :-)

> the current search
> results are a lot better than they used to be. From the beginning of
> this thread:
>
> On 23 November 2011 15:34, Edward Rankin<erankin_at_[hidden]>  wrote:
>>
>> For example
>> I just searched for boost,foreach and a documentation from 1.35 was the
>> first hit. It would be great if there is a way to tell google to update
>> their links to the latest version.
>
> If you search for it now, 1.48 comes first (at least it does for me).

OK, I'll abandon the crawl prevention on old docs. And reassess in the 
future what else we can do.

-- 
-- Grafik - Don't Assume Anything
-- Redshift Software, Inc. - http://redshift-software.com
-- rrivera/acm.org (msn) - grafik/redshift-software.com
-- 102708583/icq - grafikrobot/aim,yahoo,skype,efnet,gmail