Re: [Boost-build] [run] need help with b2

Subject: Re: [Boost-build] [run] need help with b2
From: Raffi Enficiaud (raffi.enficiaud_at_[hidden])
Date: 2016-10-25 05:18:40

Le 25/10/16 à 06:57, Rene Rivera a écrit :
> On Mon, Oct 24, 2016 at 2:40 PM, Raffi Enficiaud
> <raffi.enficiaud_at_[hidden]
> <mailto:raffi.enficiaud_at_[hidden]>> wrote:
>
>     Dear all,
>
>     There are several threads in this ML that I raised and are left open
>     without any positive outcome on my side.
>
>
> Sorry about that.. I don't always have time to devote to responding as
> much as I would like (last week was one of those times).

Thank you for pinging!

>
>     I have some fixes in boost.test and I would like to avoid any
>     regression of the kind I had for 1.62 in the future. Those fixes are
>     ready to merge, but blocked by the b2 issues.
>
>     To avoid regressions, I need several improvements of the "run"
>     commands. Those would be to my opinion a good thing for b2:
>
>     - being able to [run binary] without being forced to rebuild the
>     binary, on all platforms (see my post "[run] error on Win32: trying
>     to create a binary out of target name", 19/10/16). Basically a "[run
>     ]" should be a command that is not so tightly coupled with a build
>     (but would trigger a build if needed).
>
>
> Since this is something that works on some platform and not on Windows..
> I'd consider it a bug. It will take some time to figure out a fix though
> as I don't usually test on Windows. But perhaps you can help me by
> seeing if we can come up with a minimal repro project. I checked into BB
> an example that has the above "multiple tests with one binary". Can you
> go to boost-root/tools/build/example/complex-testing and do a "b2 -a" on
> Windows (or other platforms) to see if that works or not? [1]

The test you wrote works fine on Windows. But I modified it to reproduce 
the error I am seeing (patch attached). I just added a dependency on 
another project that contains a library, with an additional requirement 
on boost.timer.

When the line
=========
<library>../../../../../libs/timer/build//boost_timer
=========

is removed from

=========
project check_libmessage
     : source-location .
     : requirements <link>static
                    # same as boost.test here
                    <library>../../../../../libs/timer/build//boost_timer
     ;
=========

things work as expected. In particular, the run commands do not try to 
generate a binary named after the target name.

>
>     - being able to [run binary] several times with several command line
>     options, without any aliasing/troubleshooting among those runs
>
>
> This already works assuming you build the exe once, as in the example I
> reference above, and you specify different target names for each set of
> CLI options combinations. Is there something else you are asking for?

This is exactly what I need, but the runner should understand that if 
the command line that is passed to the test has changed since last run, 
the test should be run again.

If I omit the "-a" on the b2 command line, the test is not reexecuted if 
the command line has changed since last run.
So the caching should be done on the dependency graph to avoid 
unnecessary builds, but also on the command line arguments. Those are 
first class citizens for a run command.

>
>     - being able to discover the output folder of a test (see post
>     "[run] parameter containing the build directory"), in order to
>     assert the postconditions of a run when some system behaviour is
>     expected
>
>
> The key problem with getting that directly is that of course it's not
> know until late in the process of generating the targets. As you know
> the location is determined when the final target is created. Anyway..

OTOH, the proposed solutions did not work for me (see related posts, in 
short: not being able to have the full path of the project because the 
project does not exist). Are those the consequences of a bug?

What about having specific placeholder variables, like VS is doing for 
%(intermediate-build-dir) (I do not recall the name exactly, but you get 
the point).

> One option for this is to run the tool that checks the post conditions
> as a separate step. And giving that post check tool as an input file
> argument the test it needs to check. For example:
>
> ==
> run success.cpp : arg1 arg2 : : : success-a ;
> run post.cpp : : success-a : : post-a ;
> ==

Sounds good to me. I believe all this machinery can go to a higher 
level/enhanced run command in the future.

> When post-a is run it will have a single argument with the full relative
> path to the "success-a.test" file. In your post check program you would
> take the directory of that path to get the location of the side-band
> generated data you are after. In that same complex-testing example above
> there's a working example of this also. [1]

great :) you answered my problem and I will test your proposed solution!

>     - being able to propagate easily the requirements, in case the tests
>     should not run on specific platforms (inside the thread "[run]
>     parameter containing the build directory")
>
>
> I'm not understanding what you are asking for that doesn't already
> exist. Can you give me a concrete, hopefully small, example of the problem?

Of course.

A target declaration is for me a "producer":
======
requirements_datasets = [ requires cxx11_decltype ... ] ;

exe smoke-ts-static : smoke-ts/basic-smoke-test.cpp
                       ../build//boost_unit_test_framework
                     : $(requirements_datasets) ;
======
that I then consume to create [run] rules, via an alias and an 
intermediate rule:

========
rule boost.test-smoke-ts-logger ( test-name-prefix : logger ? )
{
     logger ?= HRF ;

     # generate many run rules	
     return
       [ run smoke-ts-static : : : : $(test-name-prefix)-0 ]
}

alias "smoke-ts"
:
   [ boost.test-smoke-ts-logger bt-st-txml : XML ]
   [ boost.test-smoke-ts-logger bt-st-thrf : HRF ]
   [ boost.test-smoke-ts-logger bt-st-tjunit : JUNIT ]
;
========

If the requirements of "smoke-ts-static" are not met, I do not consider 
as a failure that the [run] rules cannot be executed.
 From a previous discussion in this ML, what I understood is that the 
requirements should be propagated up to the [run] command inside the 
"boost.test-smoke-ts-logger" through the alias "smoke-ts" in order to 
avoid any failure in any of the intermediate rule.

Something like

======
alias "smoke-ts"
: [ boost.test-smoke-ts-logger bt-st-txml : XML ]
: $(requirements_datasets)
;
======

should be written in that sense.
Maybe I did not understand properly, but in case I did, the requirements 
should be to my opinion propagated from produce to consumer automatically.

>
>     - being able to declare environment variables for [run ...] commands
>     (thread "[run] specifying environment variables for running tests")
>
>
> Yeah, that's currently not possible. As.. (a) the current test actions
> don't support setting env vars. (b) It would be rather messy to account
> for the various ways to set env vars on the various OS shells. (c) The
> "b2" core engine doesn't have support in it for setting env vars when
> executing actions. If we wanted to support this (c) would likely be the
> easiest to implement [2]. The usual solution would be to change your
> test programs to accept arguments to alter the behavior instead of using
> env vars.

Those features are (unfortunately for b2) part of the things I would 
like to have regression tests: boost.test programs can also handle 
environment variables - especially useful in CI - and this is what I 
would to be able to test.

There is this line (testing.jam:633)

$(LAUNCHER) "$(>)" $(ARGS) "$(INPUT_FILES)" > "$(output-file)" 2>&1 
$(.NULLIN)

that seems to be the key line for adding anything related to env 
variables (+ a "testing.env" property). But my very limited 
understanding of the rules/action interactions + the jam language 
prevents me to go deeper into this.

>
>     Those are the basics I use with cmake. I would really appreciate to
>     have those features in b2 directly and avoid a quick and dirty fix
>     with cmake on my private build agents.
>
>     I do not know how I may work this out with a b2 specialist, but I
>     have to say I am a bit blocked and I am starting feeling a bit
>     frustrated by the time it takes for me to get something done with b2.
>
>
> I should say that the current testing.jam implementation is horrible.
> Note, not blaming anyone though. It was an expedient port of the b1
> functionality that was enough to get the existing use cases working. And
> as such it operates in a rather nasty kludgy way. And this is certainly
> something worth redoing. If not now, certainly in a near future [3].

Thanks for putting those bullets into the backlog.

>
> [1]
> <https://github.com/boostorg/build/blob/develop/example/complex-testing/jamroot.jam>
> [2] <https://trello.com/c/HIfba2d9>
> [3] <https://trello.com/c/s7ceRe1H>
>
> --
> -- Rene Rivera
> -- Grafik - Don't Assume Anything
> -- Robot Dreams - http://robot-dreams.net <http://robot-dreams.net/>
> -- rrivera/acm.org
> <http://acm.org/> (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail
>
>
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build
>