|
|
Boost Interest : |
Subject: Re: [Boost-cmake] Analysis of the current CMake system
From: Brad King (brad.king_at_[hidden])
Date: 2009-01-15 13:27:57
David Abrahams wrote:
> * logfile scraping is too hopelessly fragile to make for a good testing
> system, and there are better and possibly even easier alternatives.
The question here is whether one wants to test with the same tools users
might use to build the project. If one user's tool doesn't provide
per-rule information then we need log-scraping to test it. That doesn't
mean we can't test some tools without log-scraping though.
> Frankly I'm not sure what logfile scraping has to do with the
> structural problems you've mentioned.
I'm only referring to the test part of the anti-logscraping code. The
python command wrappers are there to avoid log scraping, but if the
tests were run through CTest then no log scraping would be needed.
> * Boost developers need the ability to change something in their
> libraries and then run a test that checks everything in Boost that
> could have been affected by that change without rebuilding and
> re-testing all of Boost (i.e. "incremental retesting").
How does the current solution solve that problem (either Boost.Build or
the current CMake system)?
>> The large number of high-level targets places many rules in the outer
>> make level which leads to very long startup times (look at
>> CMakeFiles/Makefile2, which make needs to parse many times).
>
> Just curious: why does make need to parse the same file many times?
I haven't looked at it in detail recently, but on quick inspection I
think it is now just twice. Each of the two times works with separate
rules so they could probably be split into two files. However, the file
has never been very big for any of our projects because we don't have a
huge number of top-level targets (since VS doesn't work in that case).
>> Boost needs to build libraries, documentation, and other files to be
>> placed in the install tree. Rules to build these parts can fit in
>> relatively few high-level targets and should certainly use them.
>
> Sorry, what "should certainly use" what? Rules should use the targets?
Bad wording on my part. I meant that it is fine to use top-level
targets to drive the build of libraries and documentation.
>> However, there are some disadvantages:
>>
>> (a) If one test fails to compile none of its tests can run
>> (b) A bad test may accidentally link due to symbols from another test
>
> c) Adding a feature to a library requires modifying existing test code.
I don't understand what you mean here. Are you saying that to test a
new feature, the test dispatcher needs to be updated to link in the new
test? FYI, CMake provides a command to generate the dispatcher for you
(create_test_sourcelist).
>> CTest doesn't do log-scraping
>> to detect errors. Every test gets run individually and its output is
>> recorded separately. Boost's current system puts the tests inside the
>> build and then jumps through hoops to avoid log-scraping of the
>> results.
>
> What kind of hoops?
It runs all the tests through python command wrappers to capture
individual output, and therefore has to generate its own compiler
command line invocations instead of using CMake's knowledge of the
native tools. Currently -c and -o options are hard-coded AFAICS.
>> This brings us to the log-scraping issue in general. CMake permits
>> users to build projects using their native tools, including Visual
>> Studio, Xcode and Makefiles. In order to make sure builds with these
>> tools work, the test system must drive builds through them too.
>> Testing with just one type of native tool is insufficient.
>
> That's at least somewhat debatable. To get results for all the
> different toolchains would require more testing resources, would it not?
Yes. In our model we ask users to contribute testing resources for the
toolchains they want supported which we don't have. If no one cares
enough about a platform/compiler to submit tests, we don't need to
support that platform.
>> Since the native tools do not support per-rule reporting log-scraping
>> is necessary.
>
> Also somewhat debatable. If we can get xcode to invoke
> "boost-g++-wrapper" instead of "g++," we can still get per-rule
> reporting, right?
If per-rule reporting is not available from a native tool we have to do
log-scraping. What's debatable is whether we can work around a lack of
explicit support in the native tools.
We could make this a CMake feature by teaching the generators to wrap
the compiler up with a tool we distribute with CMake. Then you won't
have to hack the compilation rule variables for Boost or depend on python.
>> Problem (a) is automatically handled by the testing solution I propose
>> above since test results are recorded and reported individually.
>
> Sorry, what did you propose above?
Testing with ctest's --build-and-test feature. The entire build and
execution of every test would be captured independently of other tests.
>> Problem (c) is a lack of convenience when the build error is subtle
>> enough to require the compiler command line to diagnose it (which in
>> my experience is very rare).
>
> Rare, but when you need it, you really need it.
Well, one could always reproduce the build locally or get help from the
person running the machine with the problem...hence "lack of convenience" :)
However, I think our discussion above concludes that log-scraping
avoidance is not the main problem. It can be made to work.
>> P.S. Boost's CMake code currently uses file(TO_NATIVE_PATH) but then
>> references the result variable in CMake code. This causes a build
>> tree with spaces in the path to create files in a directory with
>> escaped backslashes in its name. Also, will python deal with
>> non-escaped backslashes in windows paths inside the strings configured
>> into the scripts?
>
> I can tell you lots about python, but I don't understand your question.
> Could you rephrase?
The code in question tells CMake to generate a python script that looks
like this (on Windows):
sys.path.append("c:\path\with\backslashes\to\some\file.txt")
# ^^ escape sequence?
-Brad