Re: [Boost-cmake] CTest and logfile scraping

From: troy d. straszheim (troy_at_[hidden])
Date: 2008-06-01 16:46:30

Hey Doug,

Very cool, none of that rather long mail ended up being contentious/confusing.
Smiles all round.

Doug Gregor wrote:

[snip]

> Well, I don't know if it's fragile or not (it's never caused problems
> for me), but there are other issues with not having testing targets be
> first-class citizens. It's hard to run tests for just one library
> without building the tests for *all* Boost libraries (as part of the
> Build stage) and then running CTest directly, e.g., with "ctest -R
> MPI". That's why we have the TEST_BOOST_* configuration options... but
> it's still messy.

I like those, as they do significantly reduce the time to generate makefiles
and build targets when not testing (there are >2k tests iirc, that's plenty
of statting to do when figuring out what to build), but yeah, that and the
-R stuff is messy,  'make test-Signals' would be nicer.

>> Also, ctest hardcodes a set of 'testing models': Nightly, Weekly,
>> Experimental, which are aribtrary, distracting, and couple the system
>> that runs the builds (the ctest side) to the system that displays them
>> (the dart/cdart/etc side).
> 
> Right, these testing models are meant to support Dart's model, which
> isn't necessarily the right model for Boost. I don't think their
> existence is a problem; having features that are important to one user
> group but not another is the price we pay for using someone else's
> software, but the benefits generally outweigh the costs.

I was a little harsh there.   I like your characterization better.

> [snip]
>> I think that covers the business of getting at the build results in a non-lossy
>> > fashion.
>> >
>> > Referring to Catalogue O' Worries entry #3, ideally one would like to
>> > post results
>> >
>> > - at the end of the build of each component (e.g. libBAR, libFOO),
>> > - at the end of the build of the *tests* for this component,
>> > - and at the end of the *run* of the tests for this component
>> >
>> > and to do so recursively, i.e. if libBAR depends on libFOO, the
>> > build/test/post of libFOO will be done automagically.
> 
> Just the build+post of libFOO will be done, not the testing of libFOO, right?

Hm.  I guess it isn't clear what 'ideally' means.
Yes, just the build/post of libFOO I suppose.  This is a little
fuzzy to me yet, let's see how it looks when we're actually implementing it.

I was getting at was that I think the right granularity here is
three posts per library, one after the build of the lib itself,
one after the build of the tests, one after the run of the tests.
Part of the thinking here is that testers will prefer a lower
and more constant load on their uplinks to
an intermittent but very large one.

>> >  This implies
>> > peppering the build dependency tree with intermediate targets
>> > that collect results and post them.

> CMake has most of the facilities we need to move testing of Boost into
> the build, which is one of the steps in this process. The
> Boost-specific test macros (boost_test_run, boost_test_compile, etc.)
> could be changed to use add_custom_command, to make testing a part of
> the normal build process. 

Good, that's what I was thinking, I'll play with this.  I have a python
wrapper script working for compile/link:

<cxx_create_object>
   <source>/home/troy/Projects/boost/branches/CMake/release/libs/thread/src/pthread/thread.cpp</source>
   <object>CMakeFiles/boost_thread-mt-shared-debug.dir/pthread/thread.cpp.o</object>
   <commandline>/usr/bin/c++ -DBOOST_ALL_NO_LIB=1 -Dboost_thread_mt_shared_debug_EXPORTS -fPIC -I/home/troy/Projects/boost/branches/CMake/release -g -DBOOST_THREAD_BUILD_DLL=1 -pthread -D_REENTRANT -o CMakeFiles/boost_thread-mt-shared-debug.dir/pthread/thread.cpp.o -c /home/troy/Projects/boost/branches/CMake/release/libs/thread/src/pthread/thread.cpp</commandline>
   <returncode>0</returncode>
   <stdout />
   <stderr />
</cxx_create_object>
<cxx_create_shared_library>
   <target>../../../lib/libboost_thread-gcc42-mt-d-1_35_1.so.1.35.1</target>
   <commandline>/usr/bin/c++ -fPIC -shared -Wl,-soname,libboost_thread-gcc42-mt-d-1_35_1.so.1.35.1 -o ../../../lib/libboost_thread-gcc42-mt-d-1_35_1.so.1.35.1 CMakeFiles/boost_thread-mt-shared-debug.dir/pthread/thread.cpp.o CMakeFiles/boost_thread-mt-shared-debug.dir/pthread/exceptions.cpp.o CMakeFiles/boost_thread-mt-shared-debug.dir/pthread/once.cpp.o -lpthread -lrt</commandline>
   <returncode>0</returncode>
   <stdout />
   <stderr />
</cxx_create_shared_library>

and so far integration into our existing cmake codebase is *very* clean.
Since it is trivial to HTTP POST things from python, it is looking like
adding some simple targets 'post-build-Signals', 'post-test-Signals' would
be a good way to proceed.

I'm worried about some of the auxiliary
> issues---handling timeouts, 

I think those are easy, you've got the python wrapper executing a subshell and
it can handle the timeout.

> avoiding those "would you like to start a
> debugger?" dialogs on Windows, 

I dunno.  Shouldn't Boost.Test itself be able to tell windows not to
sic Clippy on me when a test crashes?

> dealing with run-fail and compile-fail
> tests, etc.---that CTest already handles. 

Will have to take a look.  Since cmake just runs ctest in a subshell
(which itself runs cmake again, if I remember correctly) it may be
possible to just 'wrap' these as well.

I'll get the proof-of-concept code out later.

-t