|
|
Boost Interest : |
Subject: Re: [Boost-cmake] Analysis of the current CMake system
From: troy d. straszheim (troy_at_[hidden])
Date: 2009-01-15 14:23:08
Brad King wrote:
> David Abrahams wrote:
>> * logfile scraping is too hopelessly fragile to make for a good testing
>> system, and there are better and possibly even easier alternatives.
>
> The question here is whether one wants to test with the same tools users
> might use to build the project. If one user's tool doesn't provide
> per-rule information then we need log-scraping to test it. That doesn't
> mean we can't test some tools without log-scraping though.
>
I don't quite get "That doesn't mean we can't test some tools without log-scraping".
I see two different cases here. There's the developer working under visual
studio or emacs who wants to run some tests. This guy knows (or should know)
how to find what compile/link flags were used, chase down warnings and whatnot.
In this case the message "test X failed, go look at the logs" is fine. Let
the user use his tool.
Then there's the 'tester' who operates testing slave boxes that poll svn, build
and run things. The test slave operator doesn't care about which tool, just
that it works. The output of these slave runs ends up on a website somewhere,
and in this case the information about the build should be exhaustive: every
step, every flag, every warning, every command linee, every
pass/fail/warn/notrun/timeout should be available, so that the library author
can get as much info as possible w/o asking the slave operator to dig around
in his build.
>> Frankly I'm not sure what logfile scraping has to do with the
>> structural problems you've mentioned.
>
> I'm only referring to the test part of the anti-logscraping code. The
> python command wrappers are there to avoid log scraping, but if the
> tests were run through CTest then no log scraping would be needed.
One problem with the current implementation is that every test is a toplevel
target; this is easy to fix, if you just write the tests to flatfiles in the
build hierarchy (like ctest does it) and use a driver script to find the files
and run the tests (that is nice and clean for tests that involve just running
compiled binaries, I haven't thought about how it would work for compile and
compile-fail tests). I have since converted a different project (as large as
boost) to this method and am very happy with the results. You can easily
customize the test-running business... its just a python script, which seems to
me the right tool for the job, a scripting task. Getting information about
building/testing out to python, intact and in its entirety, makes communicating
with servers a real pleasure (using xmlrpc under python, for instance, to talk
to the xmlrpc plugin inside a Trac instance).
If I remember correctly the 'log scraping' was more about ctest having to scrape
the build log to find compile errors. Is this no longer necessary?
>> * Boost developers need the ability to change something in their
>> libraries and then run a test that checks everything in Boost that
>> could have been affected by that change without rebuilding and
>> re-testing all of Boost (i.e. "incremental retesting").
>
> How does the current solution solve that problem (either Boost.Build or
> the current CMake system)?
>
>>> The large number of high-level targets places many rules in the outer
>>> make level which leads to very long startup times (look at
>>> CMakeFiles/Makefile2, which make needs to parse many times).
>> Just curious: why does make need to parse the same file many times?
>
> I haven't looked at it in detail recently, but on quick inspection I
> think it is now just twice. Each of the two times works with separate
> rules so they could probably be split into two files. However, the file
> has never been very big for any of our projects because we don't have a
> huge number of top-level targets (since VS doesn't work in that case).
>
>>> Boost needs to build libraries, documentation, and other files to be
>>> placed in the install tree. Rules to build these parts can fit in
>>> relatively few high-level targets and should certainly use them.
>> Sorry, what "should certainly use" what? Rules should use the targets?
>
> Bad wording on my part. I meant that it is fine to use top-level
> targets to drive the build of libraries and documentation.
>
>>> However, there are some disadvantages:
>>>
>>> (a) If one test fails to compile none of its tests can run
>>> (b) A bad test may accidentally link due to symbols from another test
>> c) Adding a feature to a library requires modifying existing test code.
>
> I don't understand what you mean here. Are you saying that to test a
> new feature, the test dispatcher needs to be updated to link in the new
> test? FYI, CMake provides a command to generate the dispatcher for you
> (create_test_sourcelist).
>
>>> CTest doesn't do log-scraping
>>> to detect errors. Every test gets run individually and its output is
>>> recorded separately. Boost's current system puts the tests inside the
>>> build and then jumps through hoops to avoid log-scraping of the
>>> results.
>> What kind of hoops?
>
> It runs all the tests through python command wrappers to capture
> individual output, and therefore has to generate its own compiler
> command line invocations instead of using CMake's knowledge of the
> native tools. Currently -c and -o options are hard-coded AFAICS.
Right... do you see a different way to get this done? Again, the
thinking here was that:
1. This is really only necessary on build slaves, and those are either make or
nmake.
2. I wanted to be as close to the invocation of the compiler as possible, to
capture as much as possible (execution time, environment, arguments, exit
status, etc).
I suppose you could have cmake generate a target that first calls a python
script, passing the command line it is about to call and some information about
what it is doing ("i'm compiling foo.o from foo.cpp, the command line is
'blah'"), then execute the command, but then what do you do about the exit
status... (ah, I see you've suggested something better below)
>>> This brings us to the log-scraping issue in general. CMake permits
>>> users to build projects using their native tools, including Visual
>>> Studio, Xcode and Makefiles. In order to make sure builds with these
>>> tools work, the test system must drive builds through them too.
>>> Testing with just one type of native tool is insufficient.
>> That's at least somewhat debatable. To get results for all the
>> different toolchains would require more testing resources, would it not?
>
> Yes. In our model we ask users to contribute testing resources for the
> toolchains they want supported which we don't have. If no one cares
> enough about a platform/compiler to submit tests, we don't need to
> support that platform.
>
>>> Since the native tools do not support per-rule reporting log-scraping
>>> is necessary.
>> Also somewhat debatable. If we can get xcode to invoke
>> "boost-g++-wrapper" instead of "g++," we can still get per-rule
>> reporting, right?
>
> If per-rule reporting is not available from a native tool we have to do
> log-scraping. What's debatable is whether we can work around a lack of
> explicit support in the native tools.
>
> We could make this a CMake feature by teaching the generators to wrap
> the compiler up with a tool we distribute with CMake. Then you won't
> have to hack the compilation rule variables for Boost or depend on python.
Presumably the name of this tool (let's call it CWrap?) would have some
interface that you could implement however you like... and if that is
implementable in python, you've given your users lots of ways to tweak their
build system. We're OK with being dependent on python.
>
>>> Problem (a) is automatically handled by the testing solution I propose
>>> above since test results are recorded and reported individually.
>> Sorry, what did you propose above?
>
> Testing with ctest's --build-and-test feature. The entire build and
> execution of every test would be captured independently of other tests.
Or a python script that does what ctest's --build-and-test does...
IMV a lot more flexibility, a lot less code.
>>> Problem (c) is a lack of convenience when the build error is subtle
>>> enough to require the compiler command line to diagnose it (which in
>>> my experience is very rare).
>> Rare, but when you need it, you really need it.
>
> Well, one could always reproduce the build locally or get help from the
> person running the machine with the problem...hence "lack of convenience" :)
>
> However, I think our discussion above concludes that log-scraping
> avoidance is not the main problem. It can be made to work.
>
>>> P.S. Boost's CMake code currently uses file(TO_NATIVE_PATH) but then
>>> references the result variable in CMake code. This causes a build
>>> tree with spaces in the path to create files in a directory with
>>> escaped backslashes in its name. Also, will python deal with
>>> non-escaped backslashes in windows paths inside the strings configured
>>> into the scripts?
>> I can tell you lots about python, but I don't understand your question.
>> Could you rephrase?
>
> The code in question tells CMake to generate a python script that looks
> like this (on Windows):
>
> sys.path.append("c:\path\with\backslashes\to\some\file.txt")
> # ^^ escape sequence?
>
I'd have to look back. This stuff was indeed working on windows; anyhow, looks
like we can detangle a lot of this with some of your help making tweaks to cmake
itself.
-t