Review: Boost.Test

From: Beman Dawes (bdawes_at_[hidden])
Date: 2001-12-13 13:43:29

I would like to see Boost.Test accepted, but only after resolution of major 
issues.

Since some of the issues prevented full testing of Boost.Test, I think 
there should be a final mini-review after the major issues are resolved.

*  Documentation improvements.  Several reviewers mentioned document 
problems.  I've noticed additional problems, which I'll communicate 
privately.  Dave Abrahams has offered to do a pass through the docs, and I 
think they should be posted for further comments after he and Gennadiy have 
worked them over.

*  Working Jamfiles are desperately needed for both library building and 
running the test programs.  Without these, it is very hard to get an 
overall picture of what is working and what isn't.

*  Portability issues need to be addressed.  The errors that turned up on 
the first round of Win32 testing looked fairly trivial, but until they are 
actually fixed we can't tell for sure that all platforms will be able to 
use Boost.Test. Because Boost.Test is used for both Boost and user testing, 
we have to hold it to a very high standard for portability.

*  The Boost regression tests have to actually be run using the new 
Boost.Test.  Because we can't stop regression testing, we can't switch over 
to the new Test library until the regression testing is working smoothly. 
(That means finishing the project to run regression tests with Jam, IMO.)

*  The issue of compiled versus header implementation needs to be better 
resolved.  If existing uses aren't compatible with the new Boost.Test, we 
need to figure out and document the transition path for current 
users.  This is a serious problem.  If it isn't possible to provide a 
light-weight version of test tools which is implemented on top of unit 
tests, it might be necessary to have a second light-weight implementation 
(as headers).  Hopefully that won't have to happen; perhaps all that will 
be needed is a forwarding header that in effect does what 
"example/online_test.cpp" does.

*  The floating point comparison issues are tricky, yet haven't been 
reviewed fully because the refactoring came up during the actual review. 
(I'm not arguing against refactoring, just saying that it needs further 
review both from the general Boost standpoint and from numerics experts.)

*  The public (rather than detail) header files expose functions which 
aren't documented, presumably because their usual use is via the macro 
wrappers.  Either these should be documented or moved into a detail 
namespace.  (I think someone else made the same point.)

*  The details of test outputs for each possible type of failure, and for 
each level of logging, are quite important to users.  I'm sorry no reviewer 
(including me!) has commented on this aspect of Boost.Test:-(

While the issues above are serious and need to be resolved, they shouldn't 
obscure the fact that Boost.Test is a very nice testing package indeed.

Thanks,

--Beman