Re: [Boost-cmake] Analysis of the current CMake system

Subject: Re: [Boost-cmake] Analysis of the current CMake system
From: Brad King (brad.king_at_[hidden])
Date: 2009-01-19 10:51:13

Hi Dave,

I think some of the confusion is because my original posting proposed
*two* different approaches to testing:

1.) Build all the (run) tests for one library into a single executable.
 Each test gets its own TU but the tests are linked together.  Execution
of the tests is delayed until test time at which point CTest runs the
executable over and over with different arguments for each test.

2.) Build the tests as individual executables, but not until *test*
time.  The idea is that we drive testing with CTest, but each test is a
*recursive invocation* of CTest with its --build-and-test feature.  This
feature drives the build of a test source file through the native
toolchain as part of running the test.  The output of the test includes
all the native toolchain stuff and the test executable's output.
However, every test is its own separate recursive invocation of ctest,
so its output is recorded separately from other tests.

Run-tests can use either #1 or #2.

Compile-only tests should use #2 since the interesting part of the test
is the compilation, and compile-fail tests can clearly not be linked to
other tests.

David Abrahams wrote:
> on Thu Jan 15 2009, Brad King <brad.king-AT-kitware.com> wrote:
>> The question here is whether one wants to test with the same tools users
>> might use to build the project.  If one user's tool doesn't provide
>> per-rule information then we need log-scraping to test it.  
> 
> Except that I contest your premise that no intrinsic per-rule
> information support implies log scraping.  If there is support for the
> use of replacement tools ("cl-wrapper" instead of "cl"), you can also
> avoid log scraping.

My argument is simply that if there *were no way* to get per-rule info
from a native build tool then log scraping is necessary.  I fully agree
that it may be possible to avoid it with sufficient effort for every
native tool.  Log scraping has been working well enough for us that
we've not been motivated to put in this effort.

If you can point me at documentation about how to do this in VS I'd love
to see it.  I know the Intel compiler does it, but that is a
full-fledged plugin that even supports its own project file format.  We
would probably need funding to do something that heavy-weight.

>>>   Frankly I'm not sure what logfile scraping has to do with the
>>>   structural problems you've mentioned.
>> I'm only referring to the test part of the anti-logscraping code.  The
>> python command wrappers are there to avoid log scraping, 
> 
> Sorry, I'm not up on the details of the system, so I don't know what
> "python command wrappers" refers to.

Currently in boost every test compilation command line is invoked
through a python command that wraps around the real command.  In the
current system this is necessary to avoid log scraping since the tests
are done during the main build.

>> but if the tests were run through CTest then no log scraping would be
>> needed.
> 
> Now I'm really confused.  On one hand, you say it's necessary to run
> tests through the native toolchains, and that implies log scraping.  On
> the other, you suggest running tests through CTest and say that doesn't
> imply log scraping.  I must be misinterpreting something.  Could you
> please clarify?

See approach #2 above.

>>> * Boost developers need the ability to change something in their
>>>   libraries and then run a test that checks everything in Boost that
>>>   could have been affected by that change without rebuilding and
>>>   re-testing all of Boost (i.e. "incremental retesting").
>> How does the current solution solve that problem (either Boost.Build or
>> the current CMake system)?
> 
> Boost.Build does it by making test results into targets that depend on
> successful runs of up-to-date test executables.  Test executables are
> targets that depend on boost library binaries and headers.

CTest will need some work to make this totally minimal.  Basically it is
missing timestamps to avoid re-running tests, which is probably why Troy
put the tests into the build in the current system.

As things stand now, the above approaches work as follows.  Approach #1
will compile/link test executables during the main build with full
dependencies.  Approach #2 will drive the individual native build system
for every test which has its own dependencies.  Both approaches will
still run every test executable though.  I'm sure we can address this
problem.

How does Boost.Build decide whether a compile-fail test needs to be
re-attempted?  Does its dependency scanning decide when something has
changed that could affect whether a new attempt at compilation could be
different?

>>> c) Adding a feature to a library requires modifying existing test code.
>> I don't understand what you mean here.  Are you saying that to test a
>> new feature, the test dispatcher needs to be updated to link in the new
>> test?  
> 
> I don't know what a test dispatcher is.  If you want to maximally
> isolate the tests for the new feature, you can put them in a new
> translation unit, but something has to call into that translation unit
> from main if the tests are going to run.

A test dispatcher is the 'main' to which you refer.  Every test gets its
own TU but they all link together.  The 'main' dispatches a call to each
test based on the first command line argument.

>> FYI, CMake provides a command to generate the dispatcher for you
>> (create_test_sourcelist).
> 
> Oh, nice; problem solved.

Thanks, and yes.

>>>> Problem (a) is automatically handled by the testing solution I propose
>>>> above since test results are recorded and reported individually.
>>> Sorry, what did you propose above?
>> Testing with ctest's --build-and-test feature.  The entire build and
>> execution of every test would be captured independently of other tests.
> 
> I don't see how that solves problem a).  If one TU of a test executable
> (corresponding to a feature) fails to compile, do you somehow build the
> executable with all the remaining TUs?

Approach #1 will not.  I was referring to approach #2.

>> However, I think our discussion above concludes that log-scraping
>> avoidance is not the main problem.  It can be made to work.
> 
> Sorry, just because I'm too literal-minded and want to be sure, do you
> mean "log scraping can be made to work," or "log scraping *avoidance*
> can be made to work?"

The latter.  We can avoid log scraping and still address the main
problems with the current system.

>> The code in question tells CMake to generate a python script that looks
>> like this (on Windows):
>>
>>   sys.path.append("c:\path\with\backslashes\to\some\file.txt")
>>   #                                        ^^ escape sequence?
> 
> Oh, that's easy enough.  Either use forward slashes or precede the
> string with r:
> 
>    sys.path.append(r"c:\path\with\backslashes\to\some\file.txt")

Sure, but I was just reporting a bug in that the current implementation
in boost doesn't do this.  It just puts in the raw backslashes.

-Brad