Re: [Boost-build] Using bjam testing rules, and boost.test with CI systems

Subject: Re: [Boost-build] Using bjam testing rules, and boost.test with CI systems
From: Johan Nilsson (r.johan.nilsson_at_[hidden])
Date: 2010-07-05 04:23:36

Anthony Foglia wrote:
> Sorry to wait so long to respond, but work interfered.

I assume this is not work then :)

>
> Johan Nilsson wrote:
>> Anthony Foglia wrote:
>>> Johan Nilsson wrote:
>>>> Anthony Foglia wrote:
>>>>> I'm setting up a continuous integration system (Hudson) for our
>>>>> code base.  It can parse the xml output from boost test.  The
>>>>> problem is, when I run the boost test code from the boost.build
>>>>> "run" rule, it appends an "EXIT STATUS: 0" line after the XML,
>>>>> which causes the parser to fail.  Is there a way to tell run to
>>>>> preserve the only the executable output, without the exit status
>>>>> line?
>>>>
>>>> I've been using Hudson + Boost.Test for quite some time now.
>>>>
>>>> IIUC you're redirecting standard output to a file, and use that
>>>> file as input to the xUnit/Boost.Test plugin. I suggest that you
>>>> instead explicitly configure Boost.Test to write the test log to
>>>> file in one of the following ways:
> [...]
>>>> - Add the command-line option "--log_sink=<path to file>" as an
>>>> argument to the test driver. Caveat: only supported lately IIRC,
>>>> haven't tried it myself.
>>>
>>> This would be (almost) perfect, but it appears it was introducted in
>>> 1.42, and we're still on 1.35.
>>>
>>> Ideally it would be a setting I could set for Hudson either directly
>>> to Boost.Test (like BOOST_TEST_OUTPUT_FORMAT environment variables),
>>> or to bjam.  But if I set it manually, then I have the problem of
>>> different test executables fighting over the output file (with no
>>> synchronization).  (Hence the "(almost)" above.)
>>>
>>> I feel that somehow I should be able to tell boost build's test
>>> functions that in the .output files I want just the output, not the
>>> output+status.
>>
>> You could always add an additional Hudson build step after running
>> the actual build + tests, that removes the offending status lines
>> from the logs using e.g. sed or grep (or whatever).
>
> That was my goal, but things are not that simple.  We have different
> libraries each with their own tests directories.  I could make aliases
> for them in the Jamroot, but then I have no easy way to know, in my
> Hudson build step, which files are generated to run sed on, other
> than a find command.[*]

That should be enough, shouldn't it? Or why not run a Ruby/Python script to 
find all files and postprocess them to another set of (wellformed) output 
files, and perhaps put them in a special directory structure.

>
> Even if it did work, the rule would basically be when the output file
> (e.g. test-executor.output) changes, create a new (saner) file from
> that (e.g. test-executor.pure-output).  Isn't that what a build
> system is designed for?  So shouldn't I be using boost-build for it?
>
> Do the boost library developers use boost-build and Boost.Test for
> testing?  How do they tie it into a continuous integration system, or
> do they not?

As for individual boost library development, I'd guess it's pretty much up 
to everyone to do as they please. There's nothing stopping them from running 
CI when developing their libraries (even if I suspect that most of them 
don't).

The official testing performed for the Boost C++ libraries (as in all of 
them "together") are the Boost Regression Tests, which I guess you know 
about. AFAIK this isn't run CI-style, rather on a fixed-interval basis 
(likely due to the huge turnaround times). IIRC there's been discussions on 
running tests (incrementally?) on a check-in basis but I don't know what's 
happening on that front.

You could checkout the Boost Regression Tests scripts to find out how the 
XML is generated/handled, see:

http://www.boost.org/development/running_regression_tests.html

Perhaps Ryppl (http://ryppl.org) will help out here in the future. Looks 
promising, even though I'd personally hoped for Mercurial rather than Git, 
and perhaps Boost.Build rather than CMake ...

>
> Can I write my own rule to take the .output files and generate
> .pure-output and .exitcode files from them?  I've never written my own
> rule, plus, it looks like I would want a generator that works with
> RUN_OUTPUT types, but because that target is hidden by the run and
> run-fail rules, I don't see how to get the name of that target, or how
> to use it without skipping the run and run-fail rules.

Checkout the Boost.Build extender manual if you have not already. You should 
be able to do this using a custom generator (if you've never written even a 
rule yourself, prepare for investing quite some time):

http://www.boost.org/doc/tools/build/doc/html/bbv2/extender.html

Before starting on the task though, I suggest you make a more explicit 
question on this mailing list to get advise from someone more knowledgable 
than me.

[I'd still personally go with the postprocessing approach until you can 
switch to a more recent Boost.Test version if you don't want to try out my 
other suggested approach]

HTH / Johan