Re: [boost] [test] reporting problems from tests

From: Gennadiy Rozental (rogeeff_at_[hidden])
Date: 2008-04-14 02:46:38

"Maxim Yanchenko" <maximyanchenko_at_[hidden]> ???????/???????? ? ???????? 
?????????: news:loom.20080331T100703-724_at_post.gmane.org...
> Hi Gennadiy!

Hi, Maxim!

Sorry for delay with reply.

> I'm using Boost.Test 1.34.1.
>
> I have some questions:
>
>
> 1) Is it possible to customize the output?
> As I see in boost/test/impl/compiler_log_formatter.ipp:
>
> void
> compiler_log_formatter::print_prefix( std::ostream& output, const_string 
> file,
> std::size_t line )
> {
>    output << file << '(' << line << "): ";
> }
>
> This is MSVC compiler output syntax, and integrates with MSVS only.
> But many other compilers (at least gcc, suncc) and unix tools like grep 
> print
> their messages in the form "filename:line: ", and there are many 
> tools/IDEs
> supporting exactly this format but not "filename(line): ".
>
> Is it possible to customize it?
> Maybe by using Boost.Format, or any kind of config #define, or anything 
> else.

You mean: is it possible to customize the default? You can always write your 
own log format and apply it to your tests. Getting back to the default: I 
agree *nix users might be discriminated here. I do not want to that the 
default build behavior (for backward compartibility), but I am open to ifdef 
some alternative formats as defaults. Boost.Format dependency is not an 
option, but several simple popular formats can be implemented. Can you 
suggest the formats?

> 2) timeout.
> You working with timeout using SIGALRM, but this interferes affects 
> sleep(),
> alarm() etc.
> You can read about it here:
> http://www.opengroup.org/onlinepubs/000095399/functions/sleep.html
> To me, it looks dangerous as there are many applications calling sleep().
> What do you think?

a) SIGALRM is only use if you actually specified timeout during test 
invocation.
b) do you have any alternative for timeout implementation on *nix
c) do you know about any actual examples where it's causing issues?

> 3) --catch_system_errors
> Is it possible to set it to false by default?
No: backward compartibility.

> Everybody wants to have a core dump if the program crashed, the only 
> scenario
I am not. I'd rather see some output indicating program crashed immediately 
and not wait for core dump to be created. But that's me. I am sure many 
people have your preference.

> when cores are unwelcome - huge test suits with many-many binaries, when
> crashing tests can just exhaust disk space. But such huge test suits will 
> be
> run from scripts where the number of parameters doesn't matter.
>
> But when I run a single test manually - I (and everybody I know) expect it 
> to
> crash, to have a possibility to dive into the core with the debugger.

Unless error message indicate what the problem is immediately.

> You know the errors leading to crashes are usually very hard to reproduce, 
> so
> it's quite frustrating when you lose a chance to understand an error just
> because you forgot to type this "--catch_system_errors=no" or forgot to 
> setup
> environment variable.

Usually unit tests are expected to exhibit reproducible behavior. Random 
based unit tests have their value, but not that popular. You may want to 
have special harness for just these test that performs proper setup (record 
seed value for example)

> For now I just went to the boost sources and changed the code to make "no"
> default option.
> But I'm looking for more elegant solution.
> For example, why not to place all default values to config defines like
> BOOST_TEST_DEFAULT_SOMETHING?

This is doable. How about BOOST_TEST_DEFAULTS_TO_CORE_DUMP?

> 4) I finally found the problem in the crashing test - it crashed inside
> Boost.Test due to parallel calls to Boost.Test checks.
> Any plans to do anything with this?

Yeah, I've got any plans ;) Any interests in participating in them?

> 5) Printing objects in failed tests
> For example, what if I have BOOST_CHECK_EQUAL(a,b) and there is no 
> operator<<
> defined in the classes?
> Or they're defined, but do something that I don't want to see in this
> particular test output (because it dumps the object in binary form, for
> example)?
> Is it possible to customize this somehow, e.g. by passing a printing 
> functor as
> a third parameter, or any other way?

You can disable the printing completely by using
BOOST_TEST_DONT_PRINT_LOG_VALUE( the_type )

Regards,

Gennadiy