Review: Boost.Test library

From: Ron Garcia (garcia_at_[hidden])
Date: 2001-12-12 17:57:17

Review of the Boost Test Library:

I vote to NOT accept the Boost Test Library.  My primary reason is the
state of the documentation.  While I don't think that the library is
particularly unacceptable, I think it's necessary to emphasize the
importance of good, clear, complete documentation as a component of a
good library.  I understand that improving the documentation has
been discussed, but I would like to see at least a first round of these 
improvements before accepting
the library. 

DOCUMENTATION:

I would like to see more of a distinction between user documentation
and reference documentation.  The reference documentation is crucial
for expositing the interfaces in the library, and the interfaces are
one of the most important components of a library.  The user
documentation is also very important, especially in this case. Having
a test library that is easy to use and easy to get started with is
important as incentive to begin developing testcases early and
developing them as often as possible.

A less important nit on the documentation:  I really liked the layout
and design of the current test library's documentation much better.  Sure it 
wasn't much in the way of eye candy, but the simplicity actually appeals to me
much more.

I would really like to see the Test Library Design Page completed.

SEPARATE BINARY LIBRARY:

It appears that in all cases one must link tto he new test framework as
an external library.  Is that true, and if so, is this necessary?  I
did notice a mention of compilation speed as an issue, but
I'm curious as to how great of an extent this is an issue?  In
particular, I am significantly less likely to use the simple execution
framework for uniform reporting of exceptional conditions  if I need to link 
yet another
library with any application with which I wish to use it.  The
simplicity of using the current execution tools -- add an include
file, add a #define (for test tools), and change the name of main --
is worth it to me in some situations at the possible expense of some
compile time. I also wonder if a different factoring could make some features 
(especially at the execution tools level) available as a simple include while 
a separate library is still mandated for the full blown tool?

In addition, I am wondering how soon Boost.Build will replace the
regression.cpp test framework?  Does the current framework have hooks
for linking in libraries with test files?  

Given that this library is implemented as .cpp files, there seems to
be a lack of build materials, ie. boost.build, makefiles,
etc. available. I'd like it to be as easy as possible to get started
using the library.

PREPROCESSOR MACROS:

I see that the library includes a great deal of macros for testing.
Are all of them really necessary?  I realize that the macros for
explicit tests are needed for embedding line number and file
information, but what about the ones for creating test cases, test
suites, etc.  Is this done for some consistency reason?  

EXIT_SUCCESS: 
I would like to see boost::exit_success and boost::exit_failure
(defined in boost/cstdlib.hpp) preferred over the C macros.  Perhaps someday
these values could find their way into Standard C++.

BOOST.FUNCTION SUPPORT:

As previously mentioned, I too would like to see Test Cases that can
take Boost.Function objects.  In some situations it improves the test
case code immensely.  If Boost.Function is tested without using that
component of the test library, (or maybe not using the test library at
all), then there is not too great a concern about cyclic
dependencies. The gain for people
writing unit tests would be worth it!

CONCLUSION:
To reiterate what I previously said, the library is pretty good, but I
think that the documentation should be more than an afterthought.  I
would like to see a more final version of the documentation before 
accepting the library.  I'd also like to see the other issues considered
as well.  In my opinion this is a very important library.  

ron