Re: [boost] Boost.Test updates in trunk: need for (mini) review?

Subject: Re: [boost] Boost.Test updates in trunk: need for (mini) review?
From: Vicente J. Botet Escriba (vicente.botet_at_[hidden])
Date: 2012-10-21 06:11:12

Le 20/10/12 05:03, Gennadiy Rozenal a écrit :
> Hi,
>
> It's been a long while since I merged any changes into boost release and by
> now there are whole bunch of new features there. Some are small, but some
> others are big ones such that they can change the way people use the library.
> I'd like to list them here (excluding bug fixes and removal of deprecated
> interfaces) and ask what should we do with these. Specifically let me know
> if you think these require some kind of (mini) review from community, but any
> other comments are welcome as well.
Hi,

thanks for taking the time to make this ready for release.

I would appreciate if all the new Boost.Test macros start by BOOST_TEST_ 
and you provide the equivalent macros for existing ones that don't 
follows this pattern.
I understand that you want short names, but avoid possible collisions is 
better than having short names. I could accept also BOOST_TF_ as Boost 
Test Framework or BOOST_TL_ as Boost Test Library.
> So here we go:
>
> I. New testing tool BOOST_CHECKA
>
> This tool is based on excellent idea from Kevlin Henney. I chose the name
> CHECKA for this tool due to the lack of better name, but I am open to
> suggestions. This testing tool capable of replacing whole bunch of existing
> other tools like BOOST_CHECK_EQUAL, BOOST_CHECK_GT etc. Usage is the most
> natural you can wish for:
>
> BOOST_CHECKA( var1 - var2 >= 12 );
>
> And the output will include as much information as we can get:
>
>   error: in "foo": check var1 - var2 >= 12 failed [23-15<12]
I like this a lot. How it works?
What about BOOST_TEST_CHECK (see above)?
>
> II. New "data driven test case" subsystem
>
> New data driven test case subsystem represent a generalization of parameterized
> test case and test case template and eventually will replace both. The idea is
> to allow user to specify an arbitrary (monomorphic or polymorphic) set of
> samples and run a test case on each of the samples. Samples can have different
> arity, thus you can have test cases with multiple parameters of different types.
> For now we support following dataset kinds (aside of generators none of dataset
> constructions routines performs *any* copying):
>
> a) singleton - dataset constructed out of single sample
>
> data::make(10) - singleton dataset with integer sample
> data::make("qwerty") - singleton dataset with string sample
>
> b) array - dataset constructed out of C array
>
> int a[] = {1,2,3,4,5,6};
> data::make(a) - dataset with 6 integer values
>
> c) collection - dataset constructed out of C++ forward iterable collection
>
> std::vector<double> v{1.2, 2.3, 3.4, 5.6 };
> data::make(v) - dataset with 6 double values
>
> d) join - dataset constructed by joining 2 datasets of the same type
>
> int a[] = {1,2,3};
> int b[] = {7,8,9};
> data::make(a) + data::make(b) - dataset with 6 integer values
>
> e) zip - dataset constructed by zipping 2 datasets of the same size, but not
>           necessarily the same type
>
> This dataset has an arity which is sum of argument dataset arities.
>
> int   a[] = {1,2,3};
> char* b[] = {"qwe", "asd", "zxc"};
>
> data::make(a) ^ data::make(b)
> dataset with 3 samples which are pairs of int and char*.
>
> f) grid - dataset constructed by "multiplying" 2 datasets of the same different
>            sizes and types type
>
> This dataset has an arity which is sum of argument dataset arities.
>
> int   a[] = {1,2,3};
> char* b[] = {"qwe", "asd"};
> double c[] = {1.1, 2.2};
>
> data::make(a) * data::make(b) * data::make(c)
> dataset with 12 samples which are tuples of int and char* and double.
>
> g) xrange generator dataset which produes samples in some range
>
> data::xrange( 0., 3., 0.4 ) - dataset with 8 double samples
> data::xrange<int>((data::begin=9, data::end=15)) - dataset with 6 int samples
> data::xrange( 1., 7.5 ) - dataset with 7 double samples
> data::xrange( 5, 0, -1 ) - dataset with 5 int samples
>
> h) random - generator dataset with unlimited number of random samples
>
> data::random(data::distribution = std::normal_distribution<>(5.,2))
> dataset with random double numbers following specified distribution
>
> data::random(( data::engine = std::minstd_rand(),
>                 data::distribution = std::discrete_distribution<>(),
>                 data::seed = 20UL ));
> dataset with random int numbers following specified distribution
>
> While all this interfaces can be used by itself to build complex datasets with
> various purposes, primary use case it was developped for is new data driven
> test case interface:
>
> BOOST_DATA_TEST_CASE( test_name, dataset, parameter_names... )
>
> Here are couple examples:
>
> int samples1[] = {1,2,3};
> BOOST_DATA_TEST_CASE( t1, samples1, sample )
> {
>     BOOST_CHECKA( foo(sample) > 0 );
> }
>
> Above test case is going to be executed 3 times with different sample values
>
> char* strs[] = {"qwe", "asd", "zxc", "mkl" };
>
> BOOST_DATA_TEST_CASE( t1, data::xrange(4) ^ strs ^ data::random(),
>                        intval, str, dblval )
> {
>     MyObj obj( dblval, str );
>
>     BOOST_CHECKA( obj.goo() == intval );
> }
>
> Above test case will be executed 4 times with different values of parameters
> intval, str, and dblval.
Yes, this will be very useful.

Which will be the values for intval? is the index of the tuple?

What about adding something like a product test case that will execute 
the test with the Cartesian product? Is this already available.
> Polymorphic datasets are still being developed, but should appear soon(ish).
>
> III. Auto-registered test unit decorators.
>
> Previously is was not possible and/or convenient to assign attributes to the
> automatically registered test units. To alleviate this I've introduced a
> notion of test unit decorator. These can be "attached" to any test unit
> similarly to like it is done in other languages
Could you give some references?
>
> Following decorators already implemented:
>      label - adds labels to a test unit
>      expected_failures - set expected failures for a test unit
Could you give an example showing the utility of this decorator?
>      timeout - sets timeout for a test unit
Very useful.
>      description - sets a test unit description
>      depends_on - sets a test unit dependency
What do you mean by depends_on? Is the execution of the test subject to 
the success of its dependencies or whether the test is enabled/disabled?
>      enable_if/disable_if - facilitates a test unit status change
>      fixture - assigns fixture to a test units
>
> Test unit description is new test unit attribute, which is reported by new
> list_content command line argument described below. Usage of labels covered
> below as well. enable_if/disable_if allow new, much more flexible test
> management. By adding enable_if/disable_if decorators to the test unit you can
> conditionally select which test units to run at construction time based on some
> compile time or run-time parameters.
I guess that this should be useful, but could you give a practical example?
> And we finally we have suite level fixtures
> which are set by attaching fixture decorator to a test suite (suite level
> fixtures executed once per test suite).
>
> Attachment of decorator is facilitated by using of BOOST_TEST_DECORATOR. Note
> that you can use any of '+', '-', '*' symbols to attach decorator (and any
> number of '*'):
Is there a difference between them? and if not why do you provide all of 
them?
>
>
>
> IV Support for "run by label" plus more improvements for filtered runs
>
> Previously you only had an ability to filter test unit to execute by name. Now
> you can attach labels to test units and create collections to run which are
> located in arbitrary positions in test tree. For example you can have special
> label for performance test cases, which is attached to all test units which are
> responsible for testing performance of your various components. Or you might
> want to collect exception safety tests etc. To filter test units by level you
> still use --run_test CLA. Labels are dented by @prefix:
>
> test.exe --run=@performance
>
> You can now repeat --run_test argument to specify multiple conditions:
I guess you mean --run.
>   
> test.exe --run=@performance,@exception_safety --run=prod/subsystem2
>
> In addition run by name/label now recognizes dependencies. So if test unit A
> depends on test unit B and test unit B is disabled/is not part of current run
> test unit A will not run as well.
>
> Finally you now have an ability to specify "negative" conditions, by prefixing
> name or label with !
>
> test.exe --run=!@performance
>
> This will run all test units which are not labeled with "performance".
>
> V. Support for failure context
>
> In many scenarios it is desirable to specify an additional information to a
> failure, but you do not need to see it if run is successful. Thus using
> regular print statements is undesirable. This becomes especially important
> when you develop common routine, which performs your testing and invoke it
> from multiple test units. In this case failure location is no help at all.
Yes this should be very useful.
> To alleviate this problem two new tools are introduced:
>    BOOST_TEST_INFO
>    BOOST_TEST_CONTEXT
>
> BOOST_TEST_INFO attaches context to next assertion which is executed.
> For example:
>
> BOOST_TEST_INFO( "a=" << a );
> BOOST_CHECKA( foo(a) == 0 );
Could the following be provided as well?

BOOST_CHECKA( foo(a) == 0, "a=" << a  );

> BOOST_CHECK_CONTEXT attaches context to all assertions within a scope:
>
> BOOST_CHECK_CONTEXT( "Starting test foo from subset " << ss ) {
>
> BOOST_CHECKA( foo(ss, a) == 1 );
> BOOST_CHECKA( foo(ss, b) == 2 );
>
> }
>
> VI. Two new command line arguments: list_context, wait_for_debugger
>
> Using list_context you can see a full or partial test tree of your test module.
> wait_for_debugger allows to force test module to wait till you attach debugger.
>
> VII. Colored output.
>
> Using new CLA --color_output you can turn on colored output for error and
> information messages
>
> VIII New production testing tools interface introduced
>
> This feature was covered in details in my presentation in BoostCon 2010. The
> gist if this is an ability to use Boost.Test testing tools interfaces in
> production (nothing to do with testing) code. User supplied implementation
> is plugged in
Could you tell us more about the utilities? Couldn't these utilities be 
moved to existing libraries or a specific library?
> IX. Number of smaller improvements:
>
> * Notion of framework shutdown. Allows to eliminate some fake memory leaks
> * Added checkpoints at fixture entry points, test case entry point and test case
> exit point for auto registered test cases
> * New FPE portable interfaces introduced and FPE handling is separated from
> system errors handling. You can detect FPE even if catch_system_error is false
> * Added an ability to erase registered exception translator
> * execution_monitor: new interface vexecute - to be used to monitor nullary
> functions with no result values
> * test_tree_visitor interface extended to facilitate visitors applying the same
> action to all test units
> * execution_monitor use typeid to report "real" exception type if possible.
> * New ability to redirect leaks report into a file
>
>
Wow, you have added a lot of new interesting features. I will check the 
documentation and start using it as soon as possible.

About if a mini review is needed it is up to you, but it is clear that a 
review will provide you some feedback before you deliver all these features.

Best,
Vicente