Re: [boost] Boost.Tests: Unit test defines own main() function, why?

From: Gennadiy Rozental (rogeeff_at_[hidden])
Date: 2007-12-24 01:45:33

"Jens Seidel" <jensseidel_at_[hidden]> wrote in message 
news:20071221132309.GA9314_at_imkf-pc073.imkf.tu-freiberg.de...
> Hi Gennadiy,
>
> first of all thanks for your comments. Some stuff is now a little bit
> more understandable other stuff not.
>
> I hope you didn't considered my mails as complaints, I just tried to
> express that some parts of Boost.Test need some improvement. To be
> honest I think that your library is a very important one and well done.
> I will definitively look into it more carefully (reading the code/the
> docs multiple days) as it is impossible to understand the current
> implementation compared to the old outdated documentation with current
> resources in let's say less than one day.
>
> Please note that I'm also willing to resolve these issues and will
> try some improvements to the documentation. I have to confess that
> I did not write much documentation myself but I'm (mostly as translator)
> involved in many Open Source documents such as Debian Reference, Debian
> Website and other documents (HOWTOs, manpages, ...). I know XML and SGML
> based tools really well (and maintain these partely myself) so I do not
> expect problems with Boost documentation system.
>
> I don't have much time to explain all my issues with Boost.Test now in
> detail (I still have to buy some presents :-) but I will at least
> answer your questions:
>
> On Fri, Dec 21, 2007 at 02:27:31AM -0500, Gennadiy Rozental wrote:
>> "Jens Seidel" <jensseidel_at_[hidden]> wrote in message
>> news:20071219090237.GA13691_at_imkf-pc073.imkf.tu-freiberg.de...
>> > And why are static and shared libraries handled differently? Why not
>> > just removing main also from static libraries?
>
> Let me again note that I think that the words "static" and "shared" do
> not match here as both libraries are indeed different ones (which share
> a lot of code). This *needs* to be fixed! The current situation is
> clearly NOT acceptable (no joke!).

They are not different in most part. The only part that is different is test 
runner signature and required init function signature.

>> 1. For backward compatibility
>
> Here you assume that most users do not use the shared library. OK, let's
> assume this for now.

Till 1.34.1 shared library was not officially supported at all.

>> 2. For enhanced usability in many cases. For people producing a lot of 
>> unit
>> tests adding extra function main()
>> implementation is unnecessary burden (not mentioning that they will have 
>> to
>> actually learn how implement it properly.
>
> Confusing users with all kind of combinations of BOOST_TEST_DYN_LINK,
> BOOST_TEST_MAIN, BOOST_TEST_ALTERNATIVE_INIT_API, usage or not usage
> of main and the init method is OK?

1. IMO majority of the users will never need to know anything about 
BOOST_TEST_ALTERNATIVE_INIT_API
2. What do you find confusing? BOOST_TEST_DYN_LINK and BOOST_TEST_MAIN 
macros serve separate and orthogonal purposes.
3. IMO majority of the users will never need to know anything about both 
init and main() functions.

> I'm sure many combination of these flags are just incompatible to each 
> other.

BOOST_TEST_MAIN and BOOST_TEST_DYN_LINK are for the most part orthogonal.

> Defining a one line main() method (containing also test module
> initialization API code) instead of using macros would simplify a lot
> in code and documentation (but would require a new API :-().

 And is not necessary in majority of use case IMO. It might simplify docs 
though.

>> 3. In many (if not the most) cases function main() handling is hidden
>> completely from the users and usage and both usage variants behave pretty
>> much the same. The only difference is presence of an extra macro
>> BOOST_TEST_DYN_LINK.
>
> Maybe. But I was never largely involved into Boost.Test and followed the
> first time mainly the normal starting instructions from the old docs
> (which are very easy!) and extended some parts later as needed
> (test_tools::predicate_result stuff, ...). So I consider my code the
> normal use case and you think it is very exotic as nearly no part of the
> documentation and your mails address my usage?

Your use case is still supported, unless I am missing something. It's just 
is not considered the most user friendly and accordingly might require extra 
leg work.

>> You coming to this from *nix prospective.
>
> Right.
>
>> On NT dll and lib are two quite
>> different ball games. Boost.Test opted to be consistent in between
>> platforms.
>
> But why not select the best of both systems? (To be honest I do not know
> anything with could be used from your proprietary world.)
> You did it the opposite way.

I did it in a way to make library portable in between different platforms. 
Being consistent in between different usage variants was not my primary 
priority (though would be nice to have). Usually usage variant is selected 
once based on some external restriction and users don't need to jump from 
one to another every few seconds.

>> > Did you also consider using a stable ABI (application binary interface,
>> > would allow a
>> > binary to be used with different library version without need to
>> > recompile)
>>
>> I suggest I write Boost.Test using C? In Boost.Test  a lot of staff 
>> happened
>
> The first "I" should probably be "You".
>
>> in headers. We might find it difficult.
>
> No, one can also use ABIs in C++. Included code is part of the binary if
> it comes from headers and fixed even if you move the binary to another
> system where different shared libraries may be installed. ABI affects
> exported symbols in shared libraries only. As most Boost libs are indeed
> header based ABI does not affect these.

Boost.Test is still evolving. My priority is not stable ABI, but backward 
compartibility in a sense that test modules that used to work should still 
work when built with new version of boost. Once dust is settled we might see 
stable ABI.

>> > or API (application programming interface, no need to change
>> > the source code to be able to compile against a new version of a
>> > library) (at least some time in the future)
>>
>> I do not plan any changes that should break backward compatibility. 
>> 1.34.1
>> did break it for very small number of use cases, which where never
>> officially supported in a first place (plus I believe 1 tool was found
>> unsafe and the interface was changes).
>
> A stable API is a very good think. But if the current interface is
> unusable (and also impossible to document) it may be useful.

I am not sure what interface you mean, but if you refer to test runner 
interface and shared library, than yes it was unusable. But shared library 
never was part of "contract".  It was never tested against and only existed 
on some platforms as a fluke of make system. There were no problems with pre 
1.34 interfaces for static library and they stay the same. There were new 
interfaces introduced for automated test units registration, but old one 
still supported.

> All Boost libraries need to pass an initial review. Is it possible to
> rewrite libs after it completely (as you did????) or would it make sense
> to ask the community if changes are worth to be done?

The public interfaces were not changed in most part, though the library did 
undergo major implementation rework somewhere in between 1.33 and 1.34. In a 
retrospect It might've been useful to do a mini review.

>> As far as I know there is no problem compiling/linking Boost.Test.
>
> I did not wrote about problems compiling Boost.Test but about using it!

I means problems compiling/linking of test modules with Boost.Test.

> I prefer of course to use Boost from the packages provided on my system.
> Even if Boost isn't installed by default it is possible to select it for
> installation during seconds on each modern system.

So select one usage variant (static or shared)  and stick with it. Why do 
you concerned with inter-variant compatibility.

>> > The problem is that the example in the official documentation (which is
>> > (or should be) on www.boost.org not somewhere outside) just contains:
>> > "the Unit Test Framework is responsible for supplying function main()"
>> >
>> > Even if you consider that to be outdated (again: Why do you not update
>> > it??????)
>
> This seems to be one of the most important questions: Why do you not
> replace http://www.boost.org/libs/test/doc/index.html with
> http://www.patmedia.net/~rogeeff/html/index.html?
> Does the content need to be transferred from HTML first or what steps are
> missing?

1. I do not how to do it
2. I do not know if it possible at all.
3. The later version is beta I am still working on.

>> > you should assume that most code around does not provide
>> > main() and fails that's why starting with recent versions.
>>
>> Only those who employ shared library and manual registration had an 
>> issue.
>
> Which is the default case in the old documentation. I also see some

No. shared library was *never* supported. for a long time the default and 
only case was static library. Next was "included" variant. Next I finally 
managed to support dynamic library in 1.34. Latest addition is an ability to 
build test module as shared library and run using external test runner. 
Users of static library and manual registration did not see any differences 
when upgraded to 1.34.

> advantages:
>
> * No usage of just another macro

Not a big advantage. In fact it rather disadvantage in this case:
1. Point of test case definition and registration are remote. You may just 
forget to register one and no one will notice.
2. It's repetitive: for each test case you need to repeat it second time for 
registration. And in case if it's defined in another test file you will need 
extern declaration as well.

> * It's easier to uncomment a test (only a single line needs a comment,
>  the compiler can still check the syntax of the not used test

It still pretty simple to comment out single test case. And in 1.35 there 
will be an ability to filter out test units by name.

> * I know the order of the execution of the tests for sure and can test
>  basic stuff at the beginning

In fact you don't. And you shouldn't. Proper test case should be self 
contained.

>> > How could I define my own main function which is compatible with shared
>> > and dynamic libraries? unit_test_main has different signatures!?
>>
>> 1. shared and dynamic libs are compatible ;)
>
> I assume you forgot "not" here. Otherwise you refer to code rewriting
> (automatic registration, ...)?

No. I did not. IMO shared == dynamic.

>> 2. Don't define init function at all.
>
> I need an init function. I use a logger and set some parts to verbose
> file output and compare as another test the generated output with
> expected one. I would nevertheless like to do this directly in main()
> without need for another init function.

You are better of with global fixture. How can you compare anything with 
expected output from init function?

>> 3. The test runner  function signature is now the same for both static 
>> and
>> dynamic library (see svn). I am yet to update the updated docs ;)
>
> Good.

Well almost. Init function signature is different. And it;s required as an 
argument in both cases.

>> > Again: Why are
>> > static and shared libraries handled differently? It shouldn't matter!
>>
>> It does on windows.
>
> But it does not have to differ! Most Open Source code from Linux can be
> compiled on Windows as well. Do you think part of the porting is to
> change interfaces just for fun? It is not necessary!

I did not change interfaces in between Linux and Windows. Static and Dynamic 
libraries interface does differ.

I am afraid that might sound like repetition, but let me retell the story 
again: Original Boost.Test library was working only as static library. The 
problem with dynamic libraries was:
* it can't included main()
* it can't have unresolved symbols.

Accordingly original initialization interface, that is based on externally 
defined init function was not going to work for shared libraries. I had 
several alternatives:

1. Forget about it. This is what we were doing before 1.34
2. Change all the interfaces plus require users to define function main(). 
This is what you suggest. This was problematic, since there were many users 
of Boost.Test already and many of them are content with macro usage and much 
prefer to avoid unnecessary work of function main() definition.
3. Make an alternative interface for use with shared libraries. and require 
users to implement function main oh their test modules. This solution makes 
static and shared library usages incompatible. But this is the least evil I 
found in this case. In addition I've changed the meaning of some of the 
macro in case of dynamic library so that function main can be generated in 
some cases automatically.

Hope this will clear it out,

Gennadiy