Re: [Boost-testing] Test results processing transition..

Subject: Re: [Boost-testing] Test results processing transition..
From: Tom Kent (lists_at_[hidden])
Date: 2015-01-14 21:11:16

On Thu, Jan 8, 2015 at 8:27 AM, Rene Rivera <grafikrobot_at_[hidden]> wrote:

> On Thu, Jan 8, 2015 at 7:07 AM, Tom Kent <lists_at_[hidden]> wrote:
>
>> I had my machine go down last week, and I'm only now getting back to
>> this. I just tried a couple runs and had two issues:
>>
>> 1) It takes a *long* time to run >30min....I think the previous version
>> was <5min on the same machine. Did it use to be an incremental process?
>> Most of results don't change between runs, so does it need to re-process
>> everything? Can it just check timestamp (or maybe checksum) of the result
>> sets?
>>
>
> It shouldn't be longer, except for the first time. As it's running the
> exact same commands, just wrapped in a shell script. Can you tell which
> part is taking so long?
>

I've added some timing outputs before each step in the script (and manually
gaged some with a stopwatch)...here's what I found:

After cleaning out my directory (getting everything from scratch) and
running (roughly adding the numbers to the left manually) here's what I saw:
$ cat times_first_run.log
build_setup Fri Jan 9 22:26:11 UTC 2015 --- 0s
git_update boost_root Fri Jan 9 22:26:11 UTC 2015 --- 10.5m
git_update boost_regresssion Fri Jan 9 22:36:50 UTC 2015 --- 1s
git_update boost_bb Fri Jan 9 22:36:51 UTC 2015 --- 3s
update_tools Fri Jan 9 22:36:54 UTC 2015 --- 0s
bootstrap Fri Jan 9 22:36:54 UTC 2015 --- 30s
build bb Fri Jan 9 22:37:27 UTC 2015 --- 5m
build_results develop Fri Jan 9 22:42:31 UTC 2015 --- 17m
build_results master Fri Jan 9 22:59:35 UTC 2015 --- 13m
upload_results develop Fri Jan 9 23:12:48 UTC 2015 --- 30s
upload_results master Fri Jan 9 23:13:20 UTC 2015 --- 30s
complete Fri Jan 9 23:13:56 UTC 2015

Immediately running aging at the end of the first output resulted in:
$ cat times_second_run.log
build_setup Mon Jan 12 14:09:14 UTC 2015 --- 0s
git_update boost_root Mon Jan 12 14:09:14 UTC 2015 --- 2.5m
git_update boost_regresssion Mon Jan 12 14:11:47 UTC 2015 --- 1s
git_update boost_bb Mon Jan 12 14:11:48 UTC 2015 --- 1s
update_tools Mon Jan 12 14:11:49 UTC 2015 --- 0s
bootstrap Mon Jan 12 14:11:49 UTC 2015 --- 30s
build bb Mon Jan 12 14:12:25 UTC 2015 --- 2.5m
build_results develop Mon Jan 12 14:15:01 UTC 2015 --- 13m
 - downloading + unzipping files (2 downloaded) --- 30s
 - reading xml, merging results, generating links --- 11.5m
 - writing documents --- 1m
build_results master Mon Jan 12 14:28:11 UTC 2015 --- 9.5m
 - downloading + unzipping files (0 downloaded) --- 5s
 - reading xml, merging results, generating links --- 9m
 - writing documents --- 30s
upload_results develop Mon Jan 12 14:37:38 UTC 2015 --- 30s
upload_results master Mon Jan 12 14:38:12 UTC 2015 --- 30s
complete Mon Jan 12 14:38:54 UTC 2015

Are those times similar to what you're seeing? This was a mult-core machine
with 3.5GB of RAM, so nothing too tiny.

Tom

> 2) When I completed uploading, the results page then gave a "Error
>> extracting file: The specified zipfile was not found." instead of showing
>> the matrix. Do we know what causes this? Is it possibly because two people
>> were uploading at the same time? Could we have the script place a small
>> lock file into the upload directory that the script could check for before
>> attempting an upload, then delete when completing (or have it time out if a
>> upload fails)?
>>
>
> It already does something like a lock file to try and prevent such
> problems. So not sure what is going on.
>
> --
> -- Rene Rivera
> -- Grafik - Don't Assume Anything
> -- Robot Dreams - http://robot-dreams.net
> -- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail
>
> _______________________________________________
> Boost-Testing mailing list
> Boost-Testing_at_[hidden]
> http://lists.boost.org/mailman/listinfo.cgi/boost-testing
>