Re: [Boost-build] Using bjam testing rules, and boost.test with CI systems

Subject: Re: [Boost-build] Using bjam testing rules, and boost.test with CI systems
From: Anthony Foglia (AFoglia_at_[hidden])
Date: 2010-07-02 18:22:59

Sorry to wait so long to respond, but work interfered.

Johan Nilsson wrote:
> Anthony Foglia wrote:
>> Johan Nilsson wrote:
>>> Anthony Foglia wrote:
>>>> I'm setting up a continuous integration system (Hudson) for our code
>>>> base.  It can parse the xml output from boost test.  The problem is,
>>>> when I run the boost test code from the boost.build "run" rule, it
>>>> appends an "EXIT STATUS: 0" line after the XML, which causes the
>>>> parser to fail.  Is there a way to tell run to preserve the only the
>>>> executable output, without the exit status line?
>>>
>>> I've been using Hudson + Boost.Test for quite some time now.
>>>
>>> IIUC you're redirecting standard output to a file, and use that file
>>> as input to the xUnit/Boost.Test plugin. I suggest that you instead
>>> explicitly configure Boost.Test to write the test log to file in one
>>> of the following ways:
[...]
>>> - Add the command-line option "--log_sink=<path to file>" as an
>>> argument to the test driver. Caveat: only supported lately IIRC,
>>> haven't tried it myself.
>>
>> This would be (almost) perfect, but it appears it was introducted in
>> 1.42, and we're still on 1.35.
>>
>> Ideally it would be a setting I could set for Hudson either directly
>> to Boost.Test (like BOOST_TEST_OUTPUT_FORMAT environment variables),
>> or to bjam.  But if I set it manually, then I have the problem of
>> different test executables fighting over the output file (with no
>> synchronization).  (Hence the "(almost)" above.)
>>
>> I feel that somehow I should be able to tell boost build's test
>> functions that in the .output files I want just the output, not the
>> output+status.
> 
> You could always add an additional Hudson build step after running the 
> actual build + tests, that removes the offending status lines from the 
> logs using e.g. sed or grep (or whatever).

That was my goal, but things are not that simple.  We have different 
libraries each with their own tests directories.  I could make aliases 
for them in the Jamroot, but then I have no easy way to know, in my 
Hudson build step, which files are generated to run sed on, other than a 
find command.[*]

Even if it did work, the rule would basically be when the output file 
(e.g. test-executor.output) changes, create a new (saner) file from that 
(e.g. test-executor.pure-output).  Isn't that what a build system is 
designed for?  So shouldn't I be using boost-build for it?

Do the boost library developers use boost-build and Boost.Test for 
testing?  How do they tie it into a continuous integration system, or do 
they not?

Can I write my own rule to take the .output files and generate 
.pure-output and .exitcode files from them?  I've never written my own 
rule, plus, it looks like I would want a generator that works with 
RUN_OUTPUT types, but because that target is hidden by the run and 
run-fail rules, I don't see how to get the name of that target, or how 
to use it without skipping the run and run-fail rules.

[*] Yes, Hudson will have to do its own directory walk to find the test 
output, but that's done by the xUnit plugin, and I can't piggyback on 
that find.

-- 
Anthony Foglia
Princeton Consultants
[phone] x233