|
|
Boost Testing : |
From: Martin Wille (mw8329_at_[hidden])
Date: 2007-10-04 12:33:11
Beman Dawes wrote:
> The regression.py --incremental switch is great for those who cycle
> tests often or have a slow machine. But --incremental is imperfect. It
> can cause stale results for a variety of reasons.
>
> In the long run it would be best to fix --incremental's problems. But
> for the short run, perhaps we could mitigate those problems by asking
> testers using --incremental to schedule a job to occasionally delete
> their bin.v2 tree. Say once a week. That wouldn't totally solve
> --incremental's problems, but it might reduce them to a more tolerable
> level.
ISTR there's at least one tester for whom a single full run takes a week
or even longer.
> Another approach would be to build the "once a week" logic right into
> regression.py. That ensures compliance without testers having to do
> anything, but still allows testers to schedule their own bin.v2 removals
> more often if they wish.
>
> Thoughts?
I'm violently against automatizing the mentioned result-purging or any
other expensive operation.
The testers need to be able to adjust use of resources according to the
needs of their environment.
Automatically forced full runs would result in testers stopping such
runs and delaying them until a more suitable time, or it would result in
testers no longer using regression.py (FWIW, I also use only parts of
that script because it tried to upload a lot of stuff without my consent).
Automating things like you suggested might be acceptable for machines
that are 100% dedicated to running Boost tests. I don't think this is
the case for the majority of the testers.
Additionally, the underlying problems with incremental runs will never
get fixed if result tables clean up automatically after a while.
Problems need to be solved at the point at which they are created, not
at the point at which they become visible.
Regards,
m