Re: [boost] [testing] Refactoring serialization tests, and possibly others.

From: Rene Rivera (grafik.list_at_[hidden])
Date: 2005-09-20 08:57:02

troy d. straszheim wrote:

Strange, I hadn't realized you where actively working to change the 
serialization test, from reading the previous threads :-\

> Robert Ramey wrote:

>>So there are a number of things that might be looked into
>>
>>a) Reduce the combinations of the serializaton tests.
>>b) Don't use libraries to test other libraries.  That is don't re-test one 
>>library (.e.g. serialization) just because some other library that it 
>>depends upon (e.g. mpl) has changed.
>>c) Define a two separate test Jamfiles -
>>    i) normal test
>>    ii) carpet bombing mode
>>e) Maybe normal mode can be altered on frequent basis when I just want to 
>>test a new feature. or just one test.
>>f) Include as part of the installation instructions for users an exhaustive 
>>test mode.  That is a user how downloads and installs the package would have 
>>the option of producing the whole test results on his own platform and 
>>sending in his results.  This would have a couple of advantages
>>  i) It would be sure that all new platforms are tested
>>  ii) I would ensure that the user has everyting installed correctly
> 
> a) I haven't thought about yet.  I agree that the scheme could be better 
> thought out, but it looks tricky to do it right, and my big issue right 
> now, for a client, is to be able to test platform-independent archives 
> (from different architectures) against each other in carpet-bomb mode. 
> Since it turns out to be trivial to also accommodate testing different 
> boost versions and compilers, I'm offering it up.

One thing I should make clear about my suggestion to reduce the 
combinations... Is that it doesn't mean reducing the number of types 
that get serialized, rather increasing those and having more complex and 
comprehensive test.

> b) is out of my hands.

It's likely out of everyones hands. This is a build system issue with 
regards to preventing the normal header dependency scanning. I don't 
think it's a road we want to follow as it overall destabilizes the 
correctness of test results.

(c) smart_ptr already does this by running an expanded set of test if 
you: "cd libs/smart_ptr/test && bjam test".

> c) i) I have refactored the "normal tests" in places to support the 
> existence of a carpet bomb mode, without lengthening the current "normal 
> mode" tests.  I've also been switching tests over to the autoregistering 
> unit test framework as I go.
> 
> c) ii) I have been working hard on.  Carpet bombing is controlled by an 
> environment variable (which I am considering changing to 
> BOOST_SERIALIZATION_CARPET_BOMB). 

We really don't want to use environment variables any more. Try making 
it based on a command line switch. For example we might want to 
standardize on some set of testing level options, and incorporate them 
into the build system. Perhaps:

     --test-level=basic
     --test-level=regression
     --test-level=complete

You can easily test for the options with:

     if --test-level=basic in $(ARGV)

> d) If there's a carpet bombing mode, what do you call normal mode? 
> Covering Fire?

:-)

-- 
-- Grafik - Don't Assume Anything
-- Redshift Software, Inc. - http://redshift-software.com
-- rrivera/acm.org - grafik/redshift-software.com
-- 102708583/icq - grafikrobot/aim - Grafik/jabber.org