|
|
Boost Testing : |
From: Markus Schöpflin (markus.schoepflin_at_[hidden])
Date: 2007-10-05 04:36:16
Martin Wille wrote:
> Beman Dawes wrote:
>> The regression.py --incremental switch is great for those who cycle
>> tests often or have a slow machine. But --incremental is imperfect. It
>> can cause stale results for a variety of reasons.
>>
>> In the long run it would be best to fix --incremental's problems. But
>> for the short run, perhaps we could mitigate those problems by asking
>> testers using --incremental to schedule a job to occasionally delete
>> their bin.v2 tree. Say once a week. That wouldn't totally solve
>> --incremental's problems, but it might reduce them to a more tolerable
>> level.
>
> ISTR there's at least one tester for whom a single full run takes a week
> or even longer.
That would be me. The regression tests for Tru64 are run on an old Digital
Workstation 500au, having a 500 MHz processor and one GB RAM.
I'm now down to only testing the latest released compiler on that platform,
which takes about 48 hours for a full run. Testing the current trunk on
this compiler version and the one before takes a full week on this machine.
For a long time I have been using the following in the script driving the
regression tests:
DAY_OF_THE_WEEK=$(date "+%w")
if [ ${DAY_OF_THE_WEEK} -eq 6 -o ${DAY_OF_THE_WEEK} -eq 0 ] ; then
INCREMENTAL=''
else
INCREMENTAL='--incremental'
fi
regression.py --v2 $INCREMENTAL ...
This results in all tests started at Saturday or Sunday to be
non-incremental, and it works pretty well.
>> Another approach would be to build the "once a week" logic right into
>> regression.py. That ensures compliance without testers having to do
>> anything, but still allows testers to schedule their own bin.v2 removals
>> more often if they wish.
>>
>> Thoughts?
As long as regression.py doesn't do the test cycling itself, it would be
pretty pointless to try and implement a once a week logic in there,
wouldn't it?
But once regression.py would be able to cycle the tests on its own, it
would be fine with me if the script scheduled a full run every week or so.
> I'm violently against automatizing the mentioned result-purging or any
> other expensive operation.
>
> The testers need to be able to adjust use of resources according to the
> needs of their environment.
>
> Automatically forced full runs would result in testers stopping such
> runs and delaying them until a more suitable time, or it would result in
> testers no longer using regression.py (FWIW, I also use only parts of
> that script because it tried to upload a lot of stuff without my consent).
That's one thing which is bothering me too. It isn't clearly stated what
will be uploaded by the script, and currently I think it is uploading too
much by default. What I would expect is that _only_ the regression results
located in the XML files are uploaded, and nothing else.
> Automating things like you suggested might be acceptable for machines
> that are 100% dedicated to running Boost tests. I don't think this is
> the case for the majority of the testers.
>
> Additionally, the underlying problems with incremental runs will never
> get fixed if result tables clean up automatically after a while.
> Problems need to be solved at the point at which they are created, not
> at the point at which they become visible.
Markus