Boost logo

Boost Testing :

From: Markus Schöpflin (markus.schoepflin_at_[hidden])
Date: 2007-10-05 04:36:16

Martin Wille wrote:

> Beman Dawes wrote:
>> The --incremental switch is great for those who cycle
>> tests often or have a slow machine. But --incremental is imperfect. It
>> can cause stale results for a variety of reasons.
>> In the long run it would be best to fix --incremental's problems. But
>> for the short run, perhaps we could mitigate those problems by asking
>> testers using --incremental to schedule a job to occasionally delete
>> their bin.v2 tree. Say once a week. That wouldn't totally solve
>> --incremental's problems, but it might reduce them to a more tolerable
>> level.
> ISTR there's at least one tester for whom a single full run takes a week
> or even longer.

That would be me. The regression tests for Tru64 are run on an old Digital
Workstation 500au, having a 500 MHz processor and one GB RAM.

I'm now down to only testing the latest released compiler on that platform,
which takes about 48 hours for a full run. Testing the current trunk on
this compiler version and the one before takes a full week on this machine.

For a long time I have been using the following in the script driving the
regression tests:

   DAY_OF_THE_WEEK=$(date "+%w")

   if [ ${DAY_OF_THE_WEEK} -eq 6 -o ${DAY_OF_THE_WEEK} -eq 0 ] ; then
   fi --v2 $INCREMENTAL ...

This results in all tests started at Saturday or Sunday to be
non-incremental, and it works pretty well.

>> Another approach would be to build the "once a week" logic right into
>> That ensures compliance without testers having to do
>> anything, but still allows testers to schedule their own bin.v2 removals
>> more often if they wish.
>> Thoughts?

As long as doesn't do the test cycling itself, it would be
pretty pointless to try and implement a once a week logic in there,
wouldn't it?

But once would be able to cycle the tests on its own, it
would be fine with me if the script scheduled a full run every week or so.

> I'm violently against automatizing the mentioned result-purging or any
> other expensive operation.
> The testers need to be able to adjust use of resources according to the
> needs of their environment.
> Automatically forced full runs would result in testers stopping such
> runs and delaying them until a more suitable time, or it would result in
> testers no longer using (FWIW, I also use only parts of
> that script because it tried to upload a lot of stuff without my consent).

That's one thing which is bothering me too. It isn't clearly stated what
will be uploaded by the script, and currently I think it is uploading too
much by default. What I would expect is that _only_ the regression results
located in the XML files are uploaded, and nothing else.

> Automating things like you suggested might be acceptable for machines
> that are 100% dedicated to running Boost tests. I don't think this is
> the case for the majority of the testers.
> Additionally, the underlying problems with incremental runs will never
> get fixed if result tables clean up automatically after a while.
> Problems need to be solved at the point at which they are created, not
> at the point at which they become visible.


Boost-testing list run by mbergal at