Re: [Boost-testing] Too many results?

From: Martin Wille (mw8329_at_[hidden])
Date: 2007-09-27 14:34:36

Rene Rivera wrote:
> I've been watching my Linux machine take incrementally more and more 
> time processing the regression results and uploading them to the OSL 
> server. If I'm lucky not too many new results come in, and it takes 3.5 
> hours (it used to take about half that). Still not entirely bad... But 
> the really bad part is that the generated results, which are compressed 
> into a ZIP file, takes more than 200M. It takes a long time to upload 
> that through my limited pipe. So I'm looking for suggestions on how to 
> improve the turn around time, and of course suggestions on how to reduce 
> the over all resources needed are welcome... Note, I'm looking for 
> things we can do quickly.
> 
> The one though I've had so far is to use "rsync" instead of uploading a 
> ZIP, and them zipping on the OSL server. But the uncompressed sources 
> are more than 1.5Gig so it would make for close to 2Gig of disk space 
> use on the OSL server. Which is something I'd rather avoid.
> 
> So, any clever ideas out there?

You could try (tar and) rzip instead of (tar and) bzip2. rzip uses the
same compression algorithm as rsync. rzip is good in finding blocks of
identical data even if these blocks are at locations far from each other
in the to-be-compressed file. We've seen it do wonders on certain data.

HTH,
m