Re: [Boost-testing] Test processing changes...

From: Rene Rivera (grafikrobot_at_[hidden])
Date: 2008-05-11 01:49:34

K. Noel Belcourt wrote:
> Hi Rene,
> 
> On May 9, 2008, at 12:21 PM, Rene Rivera wrote:
> 
>> Rene Rivera wrote:
>>> I made a slight change to check for this variance (version
>>> independent) so it should work better now.
>> Noel & David, any problems running the test processing now?
> 
> Looks like the Sandia-gcc results are posting okay.

Hm, except it looks like you switched those back to not using the 
process_jam_log.py?

> On a semi-related note, Do you have any ideas on how we can speed up  
> the posting of test results to the web page?  On occasion, it seems  
> like there's a 12 hour lag, or more, from when my tests finish, to  
> when the results appear on the web page.

12 hours is about right given it's taking the result processing 6, or 
more, hours on average per cycle. The easy way is to speed things up is 
to move the result processing to a faster machine, and with a faster 
connection to the internet.

Currently the machine running them is a 1.6Ghz 32bit x86 Ubuntu Linux 
with 2.5GB RAM. The upstream connection is a rather small, and shared, 
1/2 T1. It takes approximately 6GB of disk to maintain the results. The 
resulting doubly compressed archive of the results produced is currently 
going at 277MB and it expands, from a bz2 to a zip, on the server to 
527MB. The N-1 run too about 7 hours to process, so if we go with that 
cycle that 3 runs a day at least, or 831MB a day to upload. Which then 
takes at best 3.15 hours just for the upload. It also means it eats up 
about 26GB of upstream bandwidth a month.

The hard way to speed things up is to replace the entire reporting 
system with an incremental database-backed equivalent. But I have yet to 
find a system that is already available. I place special emphasis on 
*equivalent* as there are system available out there. But their 
reporting is rather lacking for anything past the basic reporting of 
individual test results.

-- 
-- Grafik - Don't Assume Anything
-- Redshift Software, Inc. - http://redshift-software.com
-- rrivera/acm.org (msn) - grafik/redshift-software.com
-- 102708583/icq - grafikrobot/aim,yahoo,skype,efnet,gmail