Re: [Boost-cmake] Analysis of the current CMake system

Subject: Re: [Boost-cmake] Analysis of the current CMake system
From: troy d. straszheim (troy_at_[hidden])
Date: 2009-01-21 12:50:27

Brad King wrote:
> troy d. straszheim wrote:
>> I don't quite get "That doesn't mean we can't test some tools without
>> log-scraping".
>>
>> I see two different cases here.  There's the developer working under
>> visual studio or emacs who wants to run some tests.  This guy knows (or
>> should know) how to find what compile/link flags were used, chase down
>> warnings and whatnot.  In this case the message "test X failed, go look
>> at the logs" is fine.  Let the user use his tool.
> 
> How do we know that a VS IDE build works unless we test it?  Testing it
> requires log scraping unless we write some kind of plugin, which may not
> be possible for all tools.  Even if we do log scraping for VS IDE
> builds, we can still use "CWrap" for generators where it can be implemented.

I think we're agreeing here.  There is testing the VS IDE build, and then there 
is testing the VS IDE build and trying to report every single thing that could 
go wrong in the course of the build to a cdashish server somewhere.  I'd assume 
that IDE builds would need to be tested by somebody sitting in front of the IDE. 
  If something goes wrong the IDE tells you and you go figure out what it is. 
make and nmake builds, running on slaves in a datacenter someplace, would need 
to report everything that goes  wrong.  I don't think the slave-testing-process 
should get complicated by IDEs that constantly get tested manually anyhow.

> 
> CTest has several parts.  One is to monitor builds, another is to run
> the tests.  Currently it uses log scraping to monitor builds since the
> native tools don't provide per-rule reports and we haven't created
> "CWrap" to work around the limitation.  For running the tests it has
> always separated output on a per-test basis.
> 

OK, we're on the same page.

>>> It runs all the tests through python command wrappers to capture
>>> individual output, and therefore has to generate its own compiler
>>> command line invocations instead of using CMake's knowledge of the
>>> native tools.  Currently -c and -o options are hard-coded AFAICS.
>> Right... do you see a different way to get this done?
> 
> Use --build-and-test and with "CWrap" enabled (or currently the python
> approximation of it).
> 

Ok.

>>> We could make this a CMake feature by teaching the generators to wrap
>>> the compiler up with a tool we distribute with CMake.  Then you won't
>>> have to hack the compilation rule variables for Boost or depend on
>>> python.
>> Presumably the name of this tool (let's call it CWrap?) would have some
>> interface that you could implement however you like... and if that is
>> implementable in python, you've given your users lots of ways to tweak
>> their build system.   We're OK with being dependent on python.
> 
> Whatever interface we create to tell the generators to do this can also
> specify the wrapper (hook) command.  We can provide a simple tool to be
> used by default and just specify a command-line interface for a custom tool.

Sounds good.

>>> Testing with ctest's --build-and-test feature.  The entire build and
>>> execution of every test would be captured independently of other tests.
>> Or a python script that does what ctest's --build-and-test does...
>> IMV a lot more flexibility, a lot less code.
> 
> How is duplicating functionality less code?  If --build-and-test is
> missing something, we can extend it.

Well, the ability to easily talk to anything that has python bindings and e.g. 
report results via XML-RPC to a trac server doesn't duplicate anything in ctest, 
and that'd be a real PITA to code up in C++.   To support this kind of thing, I 
could envision some python bindings to ctest: ctest does its thing, calls a 
python function (that it has been passed) as it collects results.  I'd have to 
think about this.

>>> The code in question tells CMake to generate a python script that looks
>>> like this (on Windows):
>>>
>>>   sys.path.append("c:\path\with\backslashes\to\some\file.txt")
>>>   #                                        ^^ escape sequence?
>>>
>> I'd have to look back.  This stuff was indeed working on windows;
>> anyhow, looks like we can detangle a lot of this with some of your help
>> making tweaks to cmake itself.
> 
> Python seems to leave the backslashes if the sequence doesn't happen to
> be a supported escape.  You may just be getting lucky.
> 

Yeah it'd be nice to not have to do all this, obviously bugs breed in this kind 
of stuff.

So the first order of business would be to remove this kind of thing:

set(CMAKE_CXX_COMPILE_OBJECT "\"${PYTHON_EXECUTABLE}\" \"${BOOST_TEST_DRIVER}\" 
<CMAKE_CURRENT_BINARY_DIR> cxx_compile_object <OBJECT> 
${CMAKE_CXX_COMPILE_OBJECT}" )

We're 100% agreed on the need for this, as far as I can see.  You may have ideas 
about the interface.  It gets executed as:

    wrapper build_dir opcode target arg0 arg1 arg2 ... argN

e.g.:

    mywrap.py /path/to/build create_shared_library libsomething.so gcc -shared 
-o libsomething.so somebody.o somebodyelse.o -lstdc++

'opcode' is one of cxx_compile_object, create_shared_library, 
create_static_library, link_executable.

The driver currently pickles all this information as well as the exit code, 
stderr and stdout, and elapsed time to a file, where it is later collected and 
sent to the reporting system.  IIRC a nonzero exit code indicates error, and 
output to stderr indicates warning, on most platforms (but there was some 
hackery involving windows).

-t