|
|
Boost Testing : |
From: John Maddock (john_at_[hidden])
Date: 2007-10-04 12:41:48
Martin Wille wrote:
> Beman Dawes wrote:
>> The regression.py --incremental switch is great for those who cycle
>> tests often or have a slow machine. But --incremental is imperfect.
>> It can cause stale results for a variety of reasons.
>>
>> In the long run it would be best to fix --incremental's problems. But
>> for the short run, perhaps we could mitigate those problems by asking
>> testers using --incremental to schedule a job to occasionally delete
>> their bin.v2 tree. Say once a week. That wouldn't totally solve
>> --incremental's problems, but it might reduce them to a more
>> tolerable level.
>
> ISTR there's at least one tester for whom a single full run takes a
> week or even longer.
Oh :-((((
>> Another approach would be to build the "once a week" logic right into
>> regression.py. That ensures compliance without testers having to do
>> anything, but still allows testers to schedule their own bin.v2
>> removals more often if they wish.
>>
>> Thoughts?
>
> I'm violently against automatizing the mentioned result-purging or any
> other expensive operation.
> Additionally, the underlying problems with incremental runs will never
> get fixed if result tables clean up automatically after a while.
> Problems need to be solved at the point at which they are created, not
> at the point at which they become visible.
Well, personally I have the motivation to try and find the problem and solve
it *if* we can create a reproducable test system (ie a small one that
doesn't take all day to run).
John.