Re: [boost] [testing] Refactoring serialization tests, and possibly others.

From: Robert Ramey (ramey_at_[hidden])
Date: 2005-10-12 11:10:04

troy d. straszheim wrote:
> On Tue, Sep 20, 2005 at 05:35:57PM +0200, troy d. straszheim wrote:
...
I have one concern about this.

Originally, I had larger test modules.  I breaking it down into smaller 
tests worked out better for me.  But then I wasn't using the unit test 
framework.  Since then, I see that the unit test framework is a better 
choice for this.  Each test program attempts to focus on one feature of the 
library.  But within that test, various ways of using the feature are 
excercised.  When a bug is discovered, a small section is added to the test 
so that the things will keep moving forward.  This turns out to be a good 
fit with the unit test framework and I'm very pleased to see things moving 
in this direction.

My original tests had one feature tests and internally tested for each 
archive.  This resulted in very long compile times and choked more than one 
compiler. Moving to smaller tests fixed that.

So I would like to see more or less the same setup but finer grained control 
over what gets tested.  I Run the current carpet bomb testing on my local 
machine - a 2.4GHz Windows XP machine on the following compilers.   Borland 
5.51, Borland 5.64, Comeau 4.3, gcc 3.3-cygwin, MSVC 6.5, VC 7.1 and VC 8.0. 
This takes about 4 hours for the serialization library alone.  This is in 
debug mode.  I also run release mode from time to time at it takes about 
the same amount of time.

Running the serialization tests for one compiler typically takes about an 
hour.
So it is a lot of time - but its not totally out of control - but it is 
heading there.

There are a number of ideas on how to deal with this which I won't rehash 
here except for one.

I would like to see the "Getting Started" or installation or whatever it is 
included a "configuration/system validation phase" whereby running the tests 
is a normal part of the boost installation procedure.  This would result in 
the following:

a) all tests would be run on all platforms actually being used.
b) Amount of testing would actually be increased by many fold as it would be 
done by all who installed the package
c) Variety of testing would be larger
d) Installation problems would be trapped on installation rather than when 
users are having problems with their code.
e) The testing load would be typically be 1-2 hrs per installation - not an 
unreasonable burden to ensure that one may save days of frustration tracking 
down what turns out to be an installation configuration problem.

Robert Ramey