[boost] [testing] Refactoring serialization tests, and possibly others.

From: Rene Rivera (grafik.list_at_[hidden])
Date: 2005-09-19 21:38:00

After having run two cycles of tests for Boost with the 
mingw-3_4_2-stlport-5_0 configuration and having it take more than 14 
hours on a 2.2Ghz+1GB machine, most of that in the Boost.Serialization 
library[*]. And reading some of the recent discussion about the desire 
to expand testing to include cross-version compatibility and 
cross-compiler compatibility, and hence having the number of tests 
multiply possibly exponentially. I am seriously concerned that we are 
going in the wrong direction when it comes to structuring tests.

 From looking at the tests for serialization I think we are 
over-testing, and we are past the point of exhausting testing resources. 
Currently this library takes the approach of carpet bombing the testing 
space. The current tests follow this overall structure:

     [feature tests] x [archive types] x [char/wchar] x [DLL/not-DLL]

Obviously this will never scale.

My first observation is that it doesn't seem that those axis look like 
independent features to me. That is, for example, the char/wchar 
functionality doesn't depend on the feature getting tested, or at least 
it shouldn't. And I can't imagine the library is structure internally in 
that way. To me it doesn't make sense to test "array" saving with each 
of the 3 archive types since the code for serialization of the "array" 
is the same in all situations. Hence it would makes more sense to me to 
structure the tests as:

     [feature test] x [xml archive type] x [char] x [not-DLL]

     [text archive tests] x [char] x [non-DLL]

     [binary archive tests] x [non-DLL]

     [wchar tests] x [non-DLL]

     [DLL tests]

Basically it's structured to test specific aspects of the library not to 
test each aspect against each other aspect. Some benefits as I see them:

* Reduced number of tests means faster turn around on testing.
* It's much easier to add tests for other aspects as one only has to 
concentrate on a few tests instead of many likely unrelated aspects.
* The tests can be expanded to test the aspects more critically. For 
example the DLL tests can be very specific as to what aspect of DLL vs 
non-DLL they test.
* It is easier to tell what parts of the library are breaking when the 
tests are specific.

[*] That's just a CPU time observation, not to mention that it takes 
2.15GB of disk space out of a total of 5.09GB. And it's only one 
compiler variation.

-- 
-- Grafik - Don't Assume Anything
-- Redshift Software, Inc. - http://redshift-software.com
-- rrivera/acm.org - grafik/redshift-software.com
-- 102708583/icq - grafikrobot/aim - Grafik/jabber.org