Re: [Boost-build] Using bjam testing rules, and boost.test with CI systems

Subject: Re: [Boost-build] Using bjam testing rules, and boost.test with CI systems
From: Anthony Foglia (AFoglia_at_[hidden])
Date: 2010-07-06 19:47:13

Johan Nilsson wrote:
> Anthony Foglia wrote:
>> Sorry to wait so long to respond, but work interfered.
> 
> I assume this is not work then :)

Wrong.  This is work.  Just management hasn't cared that our CI system 
has been down for a year, so I've had to squirrel away time to set up a 
simpler one.

>> Johan Nilsson wrote:
>>> Anthony Foglia wrote:
>>>> I feel that somehow I should be able to tell boost build's test
>>>> functions that in the .output files I want just the output, not the
>>>> output+status.
>>>
>>> You could always add an additional Hudson build step after running
>>> the actual build + tests, that removes the offending status lines
>>> from the logs using e.g. sed or grep (or whatever).
>>
>> That was my goal, but things are not that simple.  We have different
>> libraries each with their own tests directories.  I could make aliases
>> for them in the Jamroot, but then I have no easy way to know, in my
>> Hudson build step, which files are generated to run sed on, other
>> than a find command.[*]
> 
> That should be enough, shouldn't it? Or why not run a Ruby/Python script 
> to find all files and postprocess them to another set of (wellformed) 
> output files, and perhaps put them in a special directory structure.

Yes, it should.  It just feels very kludgy to me.  Especially when I am 
close, except for a single line.  Plus, Hudson does issue a warning if 
the unit test output isn't new, so unless I do dependency analysis, I 
won't see those warnings if a test is accidentally deleted.  (It's 
simple dependency analysis, but generating files only if input files 
have been updated is exactly what a build system is designed for. 
Writing one to avoid a build systems quirks feels uncomfortably bad.)

I feel part of the reason boost-build isn't widely used is because it's 
unfriendly and de facto designed solely for the Boost libraries.  If it 
were easier to use, more general and less customized for Boost's quirks, 
  then perhaps more people would pick it up.

>> Even if it did work, the rule would basically be when the output file
>> (e.g. test-executor.output) changes, create a new (saner) file from
>> that (e.g. test-executor.pure-output).  Isn't that what a build
>> system is designed for?  So shouldn't I be using boost-build for it?
>>
>> Do the boost library developers use boost-build and Boost.Test for
>> testing?  How do they tie it into a continuous integration system, or
>> do they not?
> 
> As for individual boost library development, I'd guess it's pretty much 
> up to everyone to do as they please. There's nothing stopping them from 
> running CI when developing their libraries (even if I suspect that most 
> of them don't).
> 
> The official testing performed for the Boost C++ libraries (as in all of 
> them "together") are the Boost Regression Tests, which I guess you know 
> about. AFAIK this isn't run CI-style, rather on a fixed-interval basis 
> (likely due to the huge turnaround times). IIRC there's been discussions 
> on running tests (incrementally?) on a check-in basis but I don't know 
> what's happening on that front.

The part I'm trying to understand is the automated test running and 
parsing of the output, which wouldn't matter necessarily differ between 
compilations triggered by a check-in, or triggered by a clock.

> You could checkout the Boost Regression Tests scripts to find out how 
> the XML is generated/handled, see:
> 
> http://www.boost.org/development/running_regression_tests.html

Thanks for the pointer.  I've downloaded the code and started looking in 
the regression.py file.

It looks like Boost gets the results in email format, in which case, 
you'd want to append the exit code.  Still, I don't see any major 
obstacle to having their regression test handler do the concatenating of 
the output and an exit code stored in two files.

I've put in a bug report on this, though now that I see how Boost is 
using it, I don't expect any progress to be made on it.

> Perhaps Ryppl (http://ryppl.org) will help out here in the future. Looks 
> promising, even though I'd personally hoped for Mercurial rather than 
> Git, and perhaps Boost.Build rather than CMake ...

Very interesting proposal.  This implies Boost will switch to CMake as 
the preferred build tool, though I thought there was a thread on the 
Boost-users list two months ago implying that wasn't going to be the 
case.  (I can't seem to find it at the moment.)

>> Can I write my own rule to take the .output files and generate
>> .pure-output and .exitcode files from them?  I've never written my own
>> rule, plus, it looks like I would want a generator that works with
>> RUN_OUTPUT types, but because that target is hidden by the run and
>> run-fail rules, I don't see how to get the name of that target, or how
>> to use it without skipping the run and run-fail rules.
> 
> Checkout the Boost.Build extender manual if you have not already. You 
> should be able to do this using a custom generator (if you've never 
> written even a rule yourself, prepare for investing quite some time):
> 
> http://www.boost.org/doc/tools/build/doc/html/bbv2/extender.html

Yeah, I checked that out and it didn't really help.  The extender 
example is about parsing a new source type, and there might be a few 
little comments on adding a new target type.  But I would need to force 
my way into accessing an intermediate type, which I don't see.  Plus, 
there's a bunch of "magic strings" in the example that aren't quite 
obvious to me.  I'll try to write a more detailed email tomorrow.

> [I'd still personally go with the postprocessing approach until you can 
> switch to a more recent Boost.Test version if you don't want to try out 
> my other suggested approach]

I'd like to come up with a solution, however temporary, before we try 
upgrading Boost.  I'd rather have the tests to support us first.

-- 
Anthony Foglia
Princeton Consultants
[phone] x233