[Boost-users] [Boost.Test] Always leaving test code active in Release mode

Subject: [Boost-users] [Boost.Test] Always leaving test code active in Release mode
From: Ruediger Berlich (r.berlich_at_[hidden])
Date: 2012-10-30 14:05:59

Hi there,

we use Boost.Test to do Unit tests for the Geneva library, covering
parametric optimization. Code for tests is included directly in most
classes, so tests can be written "on-site", whenever a new feature is
added. However, in Release mode, usually test-functions are *not*
included (masked by an "#ifdef GEMTESTING" block), but can be switched
on by the user prior to compilation. When this happens, it means that
classes being tested have a different interface than classes without
the GEMTESTING-define set.

Users are meant to derive their own classes from the Geneva's core
classes and overload certain functions. They are also provided with a
test infrastructure, that allows them to test their own code without
too much trouble for compliance with the libraries rules (as just one
example, it is being tested whether their code can be serialiazed
using Boost.Serialization and whether de-serialized classes are
identical to the one being serialized).

However, when users compile our (Open Source) library with the
GEMTESTING-define set and then compile their own code without this
flag, they will include headers with a different interface than the
one being used to compile the library. This has led to some very hard
to detect failures.

We are looking for ways to circumvent this source of errors and so far
see three options:

a) We can give compiled libraries a different name when the test-code
is included (e.g. libgeneva-testing.so or the likes)
b) There could be sanity checks in the code, e.g. with a static
function returning different values depending on whether test code is
activated or not
c) We can always compile the test code, even in Release mode, by
removing the "#ifdef GEMTESTING", this leaving no ambiguities
regarding the API

Option a) will effectively double compilation times, as we will always
have to compile both library variants. It would also force us to make
many changes in the CMake-based build system. Option b) leaves a
source for frustration for our users, as they will only discover the
problem when compiling and/or running their code. Option c) is the
easiest to implement, but leaves open some questions (see below)

***
My question now is, what the trade-offs of option c) are (i.e., leaving all test
code in and always linking with the Boost.Test libraries). Do we loose
anything ? Will this affect our ability to debug the code ? Are there
runtime-penalties ? Note that none of the test-functions is called
during normal execution.
***

Of course any views on a good strategy for the above scenario are also
appreciated.

In any case thanks and best regards,
Ruediger