Re: [boost] [testing] Refactoring serialization tests, and possibly others.

From: troy d. straszheim (troy_at_[hidden])
Date: 2005-09-20 07:58:47

Robert Ramey wrote:
> Basically you're correct  on all of this.
> 
> Rene Rivera wrote:
> 
>>After having run two cycles of tests for Boost with the
>>mingw-3_4_2-stlport-5_0 configuration and having it take more than 14
>>hours on a 2.2Ghz+1GB machine, most of that in the Boost.Serialization
>>library[*]. And reading some of the recent discussion about the desire
>>to expand testing to include cross-version compatibility and
>>cross-compiler compatibility, and hence having the number of tests
>>multiply possibly exponentially. I am seriously concerned that we are
>>going in the wrong direction when it comes to structuring tests.
> 
> 
> This was the basis of my suggestion that we run a complete set only very 
> occasionally.

I thought that was given.  Sorry if I didn't make that clear when 
suggesting widening the current strategy.  Clearly this isn't for 
everybody.

[snip]

> So there are a number of things that might be looked into
> 
> a) Reduce the combinations of the serializaton tests.
> b) Don't use libraries to test other libraries.  That is don't re-test one 
> library (.e.g. serialization) just because some other library that it 
> depends upon (e.g. mpl) has changed.
> c) Define a two separate test Jamfiles -
>     i) normal test
>     ii) carpet bombing mode
> e) Maybe normal mode can be altered on frequent basis when I just want to 
> test a new feature. or just one test.
> f) Include as part of the installation instructions for users an exhaustive 
> test mode.  That is a user how downloads and installs the package would have 
> the option of producing the whole test results on his own platform and 
> sending in his results.  This would have a couple of advantages
>   i) It would be sure that all new platforms are tested
>   ii) I would ensure that the user has everyting installed correctly

a) I haven't thought about yet.  I agree that the scheme could be better 
thought out, but it looks tricky to do it right, and my big issue right 
now, for a client, is to be able to test platform-independent archives 
(from different architectures) against each other in carpet-bomb mode. 
Since it turns out to be trivial to also accommodate testing different 
boost versions and compilers, I'm offering it up.

b) is out of my hands.

c) i) I have refactored the "normal tests" in places to support the 
existence of a carpet bomb mode, without lengthening the current "normal 
mode" tests.  I've also been switching tests over to the autoregistering 
unit test framework as I go.

c) ii) I have been working hard on.  Carpet bombing is controlled by an 
environment variable (which I am considering changing to 
BOOST_SERIALIZATION_CARPET_BOMB).  The tarball I posted recently is now 
very out-of-date, I have a lot of changes and have noticed a few 
additional things.  For instance one needs a second class A that can be 
serialized by platform-independent archives, and the build environment 
will have to do this switching.  Some primitives in the existent class A 
need to be removed in the platform independent case, and the use of 
std::rand() has to go, as you won't get the same stream of numbers on 
all platforms, so you use boost::random RNGs, and you need to be sure 
that you (re)seed the RNGs at appropriate times.

d) If there's a carpet bombing mode, what do you call normal mode? 
Covering Fire?

e) Sounds convenient.

f) Whatever I come up with for c) ii) should be pretty easy to package 
like that, if you guys actually want to use it.

-t