Re: [boost] Should we kill the YahooGroups files area?

From: Caleb Epstein (caleb.epstein_at_[hidden])
Date: 2005-09-08 10:34:34

On 9/7/05, Jeremy Maitin-Shepard <jbms_at_[hidden]> wrote:
> 
> Beman Dawes <bdawes_at_[hidden]> writes:
> 
> > I'm not seeing any obvious way to bulk download. Anyone else have any
> > ideas?
> 
> Perhaps wget -r will work?

Nope. You need to authenticate by submitting a form, so wget -r doesn't 
work. Thankfully there are Perl modules (sorry, Dave) that make this none 
too difficult.

See the attached Perl script "ygspider" which can be used to spider the 
files from a Yahoo! group given the group name, a Yahoo! username and 
password. It requires the WWW::Mechanize Perl module to work. You can run 
"perldoc ygspider" or "ygspider --help" to get detailed help, but the basic 
usage is:

ygspider groupname username password

Beware that this script probably violates Yahoo! TOS and I seem to have 
managed to get my home IP address banned in my numerous test runs, but not 
before I sucessfully mirrored all of the Boost files.

Let me know if I can send these someplace for better hosting. They use about 
22 MiB of disk space in total.

-- 
Caleb Epstein
caleb dot epstein at gmail dot com