Re: [boost] Distributed Boost with CMake: proposal and volunteering

Subject: Re: [boost] Distributed Boost with CMake: proposal and volunteering
From: Daniel Pfeifer (daniel_at_[hidden])
Date: 2011-01-31 04:27:58

Am Samstag, den 29.01.2011, 18:00 -0500 schrieb Dave Abrahams:
> At Sat, 29 Jan 2011 12:21:54 +0100,
> Daniel Pfeifer wrote:
> * Did you do this all by yourself?
> * Is it based on prior Boost/CMake efforts?
> * Did you know about the overlapping effort at Ryppl?

It is based on the work of Douglas Gregor and Troy Straszheim. I
borrowed some ideas from the "MinGW cross compiling environment"
project, from ExternalProject.cmake and from Ryppl. So: yes, I know
about Ryppl.

> > The following features are currently implemented:
> > * Aggregate modules from different sources (CVS, SVN, GIT, tarballs,
> > ...).
> > * Build, Test, Install
> 
> * Have you verified that it builds, tests, and installs equivalent
>   targets to what is being done by bjam?  And if so, how?

No. But for the 'how' part of the question: I plan to make it possible
to test against the installed Boost libraries. So we can easily compare
the following three cases:
A built and tested with bjam
B built and installed with bjam, tested with CTest
C built with CMake, tested with CTest.

Do you think this is sufficient enough?

> > * Create a binary installer with selectable components for Windows.
> > * Create a source package (with the modules included) that can do
> > everything in this list (except the first entry).
> > * Create a Debian source package that can be uploaded to a Launchpad
> > PPA where it is built and packaged into many binary Debian packages.
> > * Build Documentation (the usual quickbook-doxygen-boostbook-chain).
> > * Tested on Windows (Visual Studio 10) and Ubuntu (GCC).
> 
> * What was the test methodology?

Maybe I should have written 'runs' instead of 'tested' since it does not
include unit tests. Boost.CMake can successfully aggregate the modules,
compile the source code, install the libraries and headers, generate
packages with dependancies, and run tests. Whether the results are
correct mainly depends on whether the individual CMakeLists.txt files
are correct.

> > * Precompiled headers (currently MSVC only).
> 
> I know what precompiled headers are, but I'm not sure specifically
> what it means that you're listing them here.

There is no built in support for PCHs in CMake (yet). What Boost.CMake
does in MSVC is simply:
* create a header file that includes all headers that you want to
precompile for your library or executable
* create a source file that includes this header file
* compile this source file with the /Yc flag
* compile all other source files with /Yu and /FI

In case of Boost.Math (which is the only library that makes use of PCH)
this looks as follows:

  boost_add_library(math_tr1
    PRECOMPILE <boost/math/special_functions.hpp>
    SOURCES ${sources}
    )

> > Please note that I am not proposing a development process. I am just
> > proposing a tool.
> > I am volunteering to extend and maintain the tool so that it can (help
> > to) drive the development process.
> 
> Whoa, seriously?

Seriously.

> Gratefully accepted!  However, what I'd really like
> is to get you and the other stakeholders who are investing in the same
> thing (e.g. Kitware) on the same page, so that it's not just up to you
> alone.  One of the main goals of the Ryppl project is to use tools
> that are maintained by more than one or two individual developers.

This is reasonable. Concerning reusing tools, I guess my approach goes
even farther than Ryppl. Example: In the prior Boost/CMake effort there
was a function to sort all modules depending on their dependancies. I
completely removed that. Build dependancies are handled by the build
tool (Visual Studio, GNU Make) while install dependancies are handled by
the installer (NSIS, APT, ...).

Ryppl unites [...] package management [...]. These are awesome features,
no question. However, I am happy with APT on Debian, Pacman on Arch, RPM
on Fedora, Ports on Mac... OK, I am not that happy on Windows, but that
is a different story. Even if Ryppl becomes the best distributed
cross-platform software management system you could imagine, it will not
replace any of the package management tools in place. All you could
expect is that someone creates a Ryppl-based Linux destro, or that Ryppl
causes confusion for end users (Should I install that Python module with
aptitude or with easy_install? Should I install that Ruby module with
yum or with gem? Should I install that boost library with pacman or with
ryppl?)

So my approach would be a little bit different: I want a system that is
able to serve the currently existing package management systems. I
started with DEB, because I am a Ubuntu user. I also made it work on
Windows and Visual Studio because I think this is still the most
important platform for most end users. I want to add support for RPM etc
in the future. This sounds like much more work (supporting many PMS
versus just one), but this work has to be done anyways. There already
are package managers that create Boost packages for different PMS. All
we need is to pull them on bord like: "Help us help you to automate your
work."

> > In the next couple of days I was going to write some documentation and
> > a tutorial how to migrate the libraries to CMake.
> 
> I'm a little confused.  From the results you've reported, it sounds
> like you've already done that migration.  So what's left to do?

You should see Boost.CMake as four parts:

1. A set of CMake functions to simplify writing libraries for Boost.
These include functions to generate documentation, define testcases, and
then some thin wrappers around corresponding functions already in CMake
(eg add_library -> boost_add_library). The goal of these functions is to
provide naming consistency and so on. These functions are not intended
to be reused for projects outside of Boost. All features, that might be
of interest for other projects too, should rather be pushed upstream to
CMake than implemented here.

2. A set of CMake functions to aggregate modules from differenct
sources.

3. A database of Boost modules and where to get the sources.

4. All those CMakeLists.txt files for projects that do not provide their
own.

I volunteer to take responsibility for 1 and 2. In my vision 3 will be
maintainded either by the individual authors of the modules or by the
release managers. Currently this part is quite simple since all modules
are just copied from a boost release. Once all modules provide their own
listfiles there will be no 4 any more.

What belongs into a projects listfile:
a) The name of the project must be set: 'boost_project("Date Time")'
b) Header files should be declared: 'boost_add_headers(<list>)'
c) boost_add_library / boost_add_executable
d) boost_add_documentation
e) boost_add_test

I did a), b) and c) for the most of the projects. I did d) and e) for at
least one project (which is Accumulators, since I started at the top).
I cannot migrate all projects completely because
* there are some headers where I do not know to which project they
belong to,
* I don't know all dependancies of each project (talking about the
headers here, compiling deps are easy) and
* it is huge amount of work.

The interface of the CMake functions I talked about in 1. of the "four
parts of Boost.CMake" is now more or less stable. I thought I document
them and then maybe call for help from the Guild. I will do what I can
in this process, but I require some help from others. Maybe I could
participate in this years Summer of Code, but I fear there will be not
enough work left for a whole summer :-).

cheers, Daniel