From: Rene Rivera (grafik.list_at_[hidden])
Date: 2005-09-19 21:38:00
After having run two cycles of tests for Boost with the
mingw-3_4_2-stlport-5_0 configuration and having it take more than 14
hours on a 2.2Ghz+1GB machine, most of that in the Boost.Serialization
library[*]. And reading some of the recent discussion about the desire
to expand testing to include cross-version compatibility and
cross-compiler compatibility, and hence having the number of tests
multiply possibly exponentially. I am seriously concerned that we are
going in the wrong direction when it comes to structuring tests.
From looking at the tests for serialization I think we are
over-testing, and we are past the point of exhausting testing resources.
Currently this library takes the approach of carpet bombing the testing
space. The current tests follow this overall structure:
[feature tests] x [archive types] x [char/wchar] x [DLL/not-DLL]
Obviously this will never scale.
My first observation is that it doesn't seem that those axis look like
independent features to me. That is, for example, the char/wchar
functionality doesn't depend on the feature getting tested, or at least
it shouldn't. And I can't imagine the library is structure internally in
that way. To me it doesn't make sense to test "array" saving with each
of the 3 archive types since the code for serialization of the "array"
is the same in all situations. Hence it would makes more sense to me to
structure the tests as:
[feature test] x [xml archive type] x [char] x [not-DLL]
[text archive tests] x [char] x [non-DLL]
[binary archive tests] x [non-DLL]
[wchar tests] x [non-DLL]
Basically it's structured to test specific aspects of the library not to
test each aspect against each other aspect. Some benefits as I see them:
* Reduced number of tests means faster turn around on testing.
* It's much easier to add tests for other aspects as one only has to
concentrate on a few tests instead of many likely unrelated aspects.
* The tests can be expanded to test the aspects more critically. For
example the DLL tests can be very specific as to what aspect of DLL vs
non-DLL they test.
* It is easier to tell what parts of the library are breaking when the
tests are specific.
[*] That's just a CPU time observation, not to mention that it takes
2.15GB of disk space out of a total of 5.09GB. And it's only one
-- -- Grafik - Don't Assume Anything -- Redshift Software, Inc. - http://redshift-software.com -- rrivera/acm.org - grafik/redshift-software.com -- 102708583/icq - grafikrobot/aim - Grafik/jabber.org