Boost logo

Boost :

From: Rene Rivera (grafik.list_at_[hidden])
Date: 2005-09-19 21:38:00

After having run two cycles of tests for Boost with the
mingw-3_4_2-stlport-5_0 configuration and having it take more than 14
hours on a 2.2Ghz+1GB machine, most of that in the Boost.Serialization
library[*]. And reading some of the recent discussion about the desire
to expand testing to include cross-version compatibility and
cross-compiler compatibility, and hence having the number of tests
multiply possibly exponentially. I am seriously concerned that we are
going in the wrong direction when it comes to structuring tests.

 From looking at the tests for serialization I think we are
over-testing, and we are past the point of exhausting testing resources.
Currently this library takes the approach of carpet bombing the testing
space. The current tests follow this overall structure:

     [feature tests] x [archive types] x [char/wchar] x [DLL/not-DLL]

Obviously this will never scale.

My first observation is that it doesn't seem that those axis look like
independent features to me. That is, for example, the char/wchar
functionality doesn't depend on the feature getting tested, or at least
it shouldn't. And I can't imagine the library is structure internally in
that way. To me it doesn't make sense to test "array" saving with each
of the 3 archive types since the code for serialization of the "array"
is the same in all situations. Hence it would makes more sense to me to
structure the tests as:

     [feature test] x [xml archive type] x [char] x [not-DLL]

     [text archive tests] x [char] x [non-DLL]

     [binary archive tests] x [non-DLL]

     [wchar tests] x [non-DLL]

     [DLL tests]

Basically it's structured to test specific aspects of the library not to
test each aspect against each other aspect. Some benefits as I see them:

* Reduced number of tests means faster turn around on testing.
* It's much easier to add tests for other aspects as one only has to
concentrate on a few tests instead of many likely unrelated aspects.
* The tests can be expanded to test the aspects more critically. For
example the DLL tests can be very specific as to what aspect of DLL vs
non-DLL they test.
* It is easier to tell what parts of the library are breaking when the
tests are specific.

[*] That's just a CPU time observation, not to mention that it takes
2.15GB of disk space out of a total of 5.09GB. And it's only one
compiler variation.

-- Grafik - Don't Assume Anything
-- Redshift Software, Inc. -
-- rrivera/ - grafik/
-- 102708583/icq - grafikrobot/aim - Grafik/

Boost list run by bdawes at, gregod at, cpdaniel at, john at