On Mon, Oct 24, 2016 at 2:40 PM, Raffi Enficiaud <raffi.enficiaud@mines-paris.org> wrote:

Dear all,

There are several threads in this ML that I raised and are left open without any positive outcome on my side.

Sorry about that.. I don't always have time to devote to responding as much as I would like (last week was one of those times).

I have some fixes in boost.test and I would like to avoid any regression of the kind I had for 1.62 in the future. Those fixes are ready to merge, but blocked by the b2 issues.

To avoid regressions, I need several improvements of the "run" commands. Those would be to my opinion a good thing for b2:

- being able to [run binary] without being forced to rebuild the binary, on all platforms (see my post "[run] error on Win32: trying to create a binary out of target name", 19/10/16). Basically a "[run ]" should be a command that is not so tightly coupled with a build (but would trigger a build if needed).

Since this is something that works on some platform and not on Windows.. I'd consider it a bug. It will take some time to figure out a fix though as I don't usually test on Windows. But perhaps you can help me by seeing if we can come up with a minimal repro project. I checked into BB an example that has the above "multiple tests with one binary". Can you go to boost-root/tools/build/example/complex-testing and do a "b2 -a" on Windows (or other platforms) to see if that works or not? [1]

- being able to [run binary] several times with several command line options, without any aliasing/troubleshooting among those runs

This already works assuming you build the exe once, as in the example I reference above, and you specify different target names for each set of CLI options combinations. Is there something else you are asking for?

- being able to discover the output folder of a test (see post "[run] parameter containing the build directory"), in order to assert the postconditions of a run when some system behaviour is expected

The key problem with getting that directly is that of course it's not know until late in the process of generating the targets. As you know the location is determined when the final target is created. Anyway.. One option for this is to run the tool that checks the post conditions as a separate step. And giving that post check tool as an input file argument the test it needs to check. For example:

run success.cpp : arg1 arg2 : : : success-a ;

run post.cpp : : success-a : : post-a ;

When post-a is run it will have a single argument with the full relative path to the "success-a.test" file. In your post check program you would take the directory of that path to get the location of the side-band generated data you are after. In that same complex-testing example above there's a working example of this also. [1]

- being able to propagate easily the requirements, in case the tests should not run on specific platforms (inside the thread "[run] parameter containing the build directory")

I'm not understanding what you are asking for that doesn't already exist. Can you give me a concrete, hopefully small, example of the problem?

- being able to declare environment variables for [run ...] commands (thread "[run] specifying environment variables for running tests")

Yeah, that's currently not possible. As.. (a) the current test actions don't support setting env vars. (b) It would be rather messy to account for the various ways to set env vars on the various OS shells. (c) The "b2" core engine doesn't have support in it for setting env vars when executing actions. If we wanted to support this (c) would likely be the easiest to implement [2]. The usual solution would be to change your test programs to accept arguments to alter the behavior instead of using env vars.

Those are the basics I use with cmake. I would really appreciate to have those features in b2 directly and avoid a quick and dirty fix with cmake on my private build agents.

I do not know how I may work this out with a b2 specialist, but I have to say I am a bit blocked and I am starting feeling a bit frustrated by the time it takes for me to get something done with b2.

I should say that the current testing.jam implementation is horrible. Note, not blaming anyone though. It was an expedient port of the b1 functionality that was enough to get the existing use cases working. And as such it operates in a rather nasty kludgy way. And this is certainly something worth redoing. If not now, certainly in a near future [3].

[1] <https://github.com/boostorg/build/blob/develop/example/complex-testing/jamroot.jam>

[2] <https://trello.com/c/HIfba2d9>

[3] <https://trello.com/c/s7ceRe1H>

-- Rene Rivera
-- Grafik - Don't Assume Anything
-- Robot Dreams - http://robot-dreams.net
-- rrivera/acm.org (msn) - grafikrobot/aim,yahoo,skype,efnet,gmail