Boost.Test Formal Review Results

From: Jeremy Siek (jsiek_at_[hidden])
Date: 2001-12-19 23:58:37

The new Boost.Test library is not currently ready for inclusion in
Boost mainly due to the documentation.  However the new Boost.Test is
a good library and vitally needed by the Boost community. It will be
given a "delayed acceptance", which means that it will be checked into
a CVS branch where Gennadiy and others can collaborate on improving
the documentation and resolving the remaining issues. Once that
process is complete, we will have another mini-review, and if that
goes well Boost.Test will be added to the Boost library collection.

Below are my comments on the library, followed by a summary/overview
of other boost members' comments.

My comments:

index.html:
  The descriptions of the components are too vague. The words
  "monitor" and "control" are used, but the reader is left wondering
  what kind of monitoring, and what kind of control. Also, need
  to explain in which situations one would use one component, and
  not another component.

execution_monitor.htm
  Would be good to start with an example to motivate the need for an
  execution monitor, and to show how to use it. Also, the introduction
  does not flow very well. The train of thought jumps around.
  Also it should be longer. Also, the docs should compare the
  execution monitor with the program execution monitor to explain
  what the difference is, and when to use which.

prg_exec_monitor.htm
  Good example.
  Which exceptions are caught by the provided main, and how are
    they reported.
  Should compare and contrast with the test execution monitor, giving
    concrete examples.

test_tools.htm
  Make sure the document the backwards compatibility macros.

  BOOST_CHECK_EQUAL
    What are the requirements on the types of the parameters left and
right?
    I suppose there must be an operator<<(ostream, left_t)
    and operator<<(ostream, right_t) defined. Are there any other
    requirements also missing?

test_exec_monitor.htm
  The docs for this component are quite good.

unit_test_framework.htm

   How does the unit test library handle testing class templates and
      function templates? Typically one want to test with several
      variations on the template paremeters, in a systematic way.
      Would be good to have an example of this.

  Explain the situations when a user would choose function_test_case
  versus the class_test_case.

  Need to separate out what is reference material and what is
  tutorial material. Also, need to make sure there is tutorial
  and/or examples for every macro/function in the framework.
  Also, be careful to only document the public parts of the
  library, or perhaps separate parts of the interface into
  layers. For example, to use the macros, does the casual user
  need to know about the underlying class that gets created?

  Are the BOOST_TEST_CASE, BOOST_PARAM_TEST_CASE, and BOOST_TEST_SUITE
  macro's really necessary? Then only save a little typing.

  I am very concerned with how BOOST_PARAM_TEST_CASE works. When
  trying to use this for the first time, I didn't realize that
  the parameter containers have to be global variables. I made
  them local, and then ran into memory problems. Even if this
  were documented well, I think it still poses a serious usability
  problem, especially when dealing with template classes, where the
  parameters may need to vary. One solution would be to have the
  parameter range *copied* into the test object. Perhaps there
  are better solutions...

  The extension to use bind looks good, that should definitely
  be included.

  Need examples of using the test_suite, unit_test_log, unit_test_result

  Perhaps should break this page up into several pages, each one
  focusing on a single function/class, which more examples and
  explanation of each one.

getting_started.htm
  * perhaps should list the invariants of the class, and discuss
       which tests cover which invariants
  * forgot to test the case when length > strlen(s) for the
       constructor const_string(char const* s, size_t length). What
       should the specified behaviour of the constructor be?
  Would be good to have more full examples for class_test_case and
     for parameterized_class_test_case

   Break up the unit_test_example.cpp into peices showing
      complete individual tests.
   Create an example file for the const_string.

Framework compilation
  Need jamfiles for everything. Should build libraries (.a's).

Miscellaneous Comments:
  - The blue frame with the white border is not so great. It wastes space.
  - The "Benefits" sections tend not to be very informative. Should either
    drop them, or put more into them.

Testing:
  Would be nice to have documentation describing how the Boost.Test
  library was tested, describing which components are tested by which
  tests. The current docs have a few pointers to files, but this needs
  to be organized better, collected in one place, and made sure that
  it is complete.

Collected comments from others:

From Dave
  documentation wording improvements
  was the proposal for the BOOST_THROW macro resolved?

From Fernando, et. all
  the floating point equality checking issue, refactoring of the
  messages and the equality testing functions. Gennadiy has made
  changes to accomodate this, and we will need to look at the
  changes he made.

From Bill:
  - Documentation needs improving.
  - BTW, I strongly disagree with the statement "(though obviously this
      is not crucial to the acceptance or inclusion of this library)".
      Good documentation *is* a *requirement* for boost libraries.
  - timeout implementation portability issues

From Ron:
  documentation improvements
    want clean separation between "reference" and "tutorial"
documentation.
    (Ron was not talking about the separation between user and
implementation
     documentation)
  separate binary library: this needs to be documented more clearly.

From Beman:
  documentation
  jamfiles
  portability
  cleaner separation between "public" versus "detail" parts of the
    library
  details of test outputs

----------------------------------------------------------------------
 Jeremy Siek                          http://www.osl.iu.edu/~jsiek/
 Ph.D. Student, Indiana Univ. B'ton   email: jsiek_at_[hidden]
 C++ Booster (http://www.boost.org)   office phone: [phone]
----------------------------------------------------------------------