[Boost-testing] Re: [testing] Possible changes to testing..

From: Misha Bergal (mbergal_at_[hidden])
Date: 2005-03-09 21:13:57

Rene Rivera <grafik.list_at_[hidden]> writes:

[...]
>>> From my point of view, this is what needs to be worked on next 
>> (http://www.crystalclearsoftware.com/cgi-bin/boost_wiki/wiki.pl?Boost.Testing)
>> # Incremental testing is not reliable
>> # Tests are run for compilers for which they are known to fail.
>> This needs to be done no matter whether we use BuildBot or something
>> else.
>
> Yes. And I, Dave, Volodya, and maybe others, are thinking about how to
> fix those problems. But we can't fix, what we don't know how to fix
> ;-) 
> -- Concretely of course, abstractly we have some ideas.

One of the approaches would be to replace  Boost.Build option
--dump-tests with --testing with the following changes in Boost.Build
behavior:

1. The list of disabled tests will be read from some file (specified
   as option in command line)

2. The test rules will check if the test is in the list of disabled
   tests and just won't set up the test and its dependencies if the
   test is disabled. (This allows to skip tests/libs marked as
   unusable/failed for particular toolsets).

3. For every "component" directory used in the rules Boost.Build will
   create/touch an "updated on this run" file even if that "component"
   has been updated (I understand that this is a little bit vague, but
   I am not very familiar with Boost.Build and its terminology). This
   allows run incremental but to not upload the results of obsolete
   tests.

I'll see if I have some time to prototype above.

-- 
Misha Bergal
MetaCommunications Engineering