Re: [boost] Distributed Boost with CMake: proposal and volunteering

Subject: Re: [boost] Distributed Boost with CMake: proposal and volunteering
From: Dave Abrahams (dave_at_[hidden])
Date: 2011-01-31 11:34:43

At Mon, 31 Jan 2011 10:27:58 +0100,
Daniel Pfeifer wrote:
> 
> Am Samstag, den 29.01.2011, 18:00 -0500 schrieb Dave Abrahams:
> > At Sat, 29 Jan 2011 12:21:54 +0100,
> > Daniel Pfeifer wrote:
> > * Did you do this all by yourself?
> > * Is it based on prior Boost/CMake efforts?
> > * Did you know about the overlapping effort at Ryppl?
> 
> It is based on the work of Douglas Gregor and Troy Straszheim. I
> borrowed some ideas from the "MinGW cross compiling environment"
> project, from ExternalProject.cmake and from Ryppl. So: yes, I know
> about Ryppl.

Great.  At least we probably are not duplicating work.

> > > The following features are currently implemented:
> > > * Aggregate modules from different sources (CVS, SVN, GIT, tarballs,
> > > ...).
> > > * Build, Test, Install
> > 
> > * Have you verified that it builds, tests, and installs equivalent
> >   targets to what is being done by bjam?  And if so, how?
> 
> No. But for the 'how' part of the question: I plan to make it possible
> to test against the installed Boost libraries. So we can easily compare
> the following three cases:
> A built and tested with bjam
> B built and installed with bjam, tested with CTest
> C built with CMake, tested with CTest.
> 
> Do you think this is sufficient enough?

Sounds like it's a great start.  I keep thinking about a tool that
compares command-lines and does some kind of heuristically-informed
fuzzy matching to find likely problems, but I'm not sure we need to go
that far.  I *would* like to see something that verifies all the same
tests get run.

> > > * Create a binary installer with selectable components for Windows.
> > > * Create a source package (with the modules included) that can do
> > > everything in this list (except the first entry).
> > > * Create a Debian source package that can be uploaded to a Launchpad
> > > PPA where it is built and packaged into many binary Debian packages.
> > > * Build Documentation (the usual quickbook-doxygen-boostbook-chain).
> > > * Tested on Windows (Visual Studio 10) and Ubuntu (GCC).
> > 
> > * What was the test methodology?
> 
> Maybe I should have written 'runs' instead of 'tested' since it does not
> include unit tests. Boost.CMake can successfully aggregate the modules,
> compile the source code, install the libraries and headers, generate
> packages with dependancies, and run tests. Whether the results are
> correct mainly depends on whether the individual CMakeLists.txt files
> are correct.

Naturally.  We need some strategy that can give us reasonable
confidence in that correctness.

> > > Please note that I am not proposing a development process. I am just
> > > proposing a tool.
> > > I am volunteering to extend and maintain the tool so that it can (help
> > > to) drive the development process.
> > 
> > Whoa, seriously?
> 
> Seriously.
> 
> > Gratefully accepted!  However, what I'd really like
> > is to get you and the other stakeholders who are investing in the same
> > thing (e.g. Kitware) on the same page, so that it's not just up to you
> > alone.  One of the main goals of the Ryppl project is to use tools
> > that are maintained by more than one or two individual developers.
> 
> This is reasonable. Concerning reusing tools, I guess my approach goes
> even farther than Ryppl. Example: In the prior Boost/CMake effort there
> was a function to sort all modules depending on their dependancies. I
> completely removed that. 

I don't think we are doing that anymore.  At least, I hope not!

> Build dependancies are handled by the build tool (Visual Studio, GNU
> Make) while install dependancies are handled by the installer (NSIS,
> APT, ...).

Awesome.

> Ryppl unites [...] package management [...]. These are awesome features,
> no question. However, I am happy with APT on Debian, Pacman on Arch, RPM
> on Fedora, Ports on Mac... OK, I am not that happy on Windows, but that
> is a different story. Even if Ryppl becomes the best distributed
> cross-platform software management system you could imagine, it will not
> replace any of the package management tools in place. 

Correct.  However, I believe it could become a de-facto standard for
people who are interested in developing and building from source.

> All you could expect is that someone creates a Ryppl-based Linux
> destro, or that Ryppl causes confusion for end users (Should I
> install that Python module with aptitude or with easy_install?
> Should I install that Ruby module with yum or with gem? Should I
> install that boost library with pacman or with ryppl?)

Yeah, that's annoying.  Most of what's installed by ryppl I expect to
be done in a virtual environment that's set up just for development
and testing, but when you get to the point of installation on a
system with an existing package manager, I agree it would be much
better to let that handle the install.

> So my approach would be a little bit different: I want a system that is
> able to serve the currently existing package management systems. I
> started with DEB, because I am a Ubuntu user. I also made it work on
> Windows and Visual Studio because I think this is still the most
> important platform for most end users. I want to add support for RPM etc
> in the future. This sounds like much more work (supporting many PMS
> versus just one), but this work has to be done anyways. There already
> are package managers that create Boost packages for different PMS. All
> we need is to pull them on bord like: "Help us help you to automate your
> work."

That's fine; you don't need to buy into that part of Ryppl in order to
join forces with the Ryppl developers working on boost modularization.
In fact, if I understand what you're saying, your vision really
doesn't conflict with Ryppl's at all; it's just a (relatively simple)
extra layer.  So, if what you're saying is that by default "ryppl
install" on a debian system should just build and install .debs, I buy
it.

> > > In the next couple of days I was going to write some
> > > documentation and a tutorial how to migrate the libraries to
> > > CMake.
> > 
> > I'm a little confused.  From the results you've reported, it sounds
> > like you've already done that migration.  So what's left to do?
> 
> You should see Boost.CMake as four parts:

,----
| Aside: should we call this something else, that includes a reference
| to the word "modularized" somehow, just to distinguish it from the
| several unmodularized boost-cmakes?  We'll have to deal with these
| naming issues eventually.
`----

> 1. A set of CMake functions to simplify writing libraries for Boost.
> These include functions to generate documentation, define testcases, and
> then some thin wrappers around corresponding functions already in CMake
> (eg add_library -> boost_add_library). The goal of these functions is to
> provide naming consistency and so on. These functions are not intended
> to be reused for projects outside of Boost. All features, that might be
> of interest for other projects too, should rather be pushed upstream to
> CMake than implemented here.

KK.  But they'll probably have to be implemented here before we can
propose them as part of CMake.  Fortunately, the Kitware guys are keen
to help us out here.

> 2. A set of CMake functions to aggregate modules from differenct
> sources.
> 
> 3. A database of Boost modules and where to get the sources.
> 
> 4. All those CMakeLists.txt files for projects that do not provide their
> own.
> 
> I volunteer to take responsibility for 1 and 2. In my vision 3 will be
> maintainded either by the individual authors of the modules or by the
> release managers. Currently this part is quite simple since all modules
> are just copied from a boost release. Once all modules provide their own
> listfiles there will be no 4 any more.
> 
> What belongs into a projects listfile:
> a) The name of the project must be set: 'boost_project("Date Time")'
> b) Header files should be declared: 'boost_add_headers(<list>)'
> c) boost_add_library / boost_add_executable
> d) boost_add_documentation
> e) boost_add_test
> 
> I did a), b) and c) for the most of the projects. I did d) and e) for at
> least one project (which is Accumulators, since I started at the top).
> I cannot migrate all projects completely because
> * there are some headers where I do not know to which project they
> belong to,

Take a look at Eric's modularization work at
https://github.com/ryppl/ryppl/tree/master/boost.  That should give
you pretty good guidance.

> * I don't know all dependancies of each project (talking about the
> headers here, compiling deps are easy) and

Ditto.

> * it is huge amount of work.

> The interface of the CMake functions I talked about in 1. of the "four
> parts of Boost.CMake" is now more or less stable. I thought I document
> them and then maybe call for help from the Guild. I will do what I can
> in this process, but I require some help from others. Maybe I could
> participate in this years Summer of Code, but I fear there will be not
> enough work left for a whole summer :-).

It's my aim to get things in decent shape before BoostCon, but if you
sign up for GSoC I will gladly work to find ways to expand this
project for you :-)

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com