Re: [boost] Update on status of C++ 11 only Boost fork/modular distro

Subject: Re: [boost] Update on status of C++ 11 only Boost fork/modular distro
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2014-10-08 07:28:33

On 7 Oct 2014 at 16:55, Mikael Persson wrote:

> Personally, I find this proposal very scary. I'm just gonna give my opinion
> on this, make what you want of it.
> 
> One little thing, I'm not sure that STL11 is a particularly good name for
> this, since most of the new C++11 libraries would not be classified as
> "STL" libraries (except for unordered containers and <array>). But that's
> just a trivial issue of terminology.

Interestingly, one of the most obvious things you see in the bindings 
output is the word 'template' due to how template heavy the C++ 11 
additions are. But I mainly chose it for being short and descriptive.

> You state in your readme for your generator:
> ".. or write some brittle bindings which manually replicate STL items .."
> In my experience, what can make those kinds of manual bindings brittle are
> all those subtle incompatibilities (often undocumented) between analogous
> components of the alternative implementations.

In the use case situation of providing some/any C++ 11 STL 
implementation, I would see Boost as just one of Dinkumware, 
libstdc++ and libc++. If your code already doesn't mind which STL it 
runs on, it is highly unlikely to get suddenly brittle with Boost, 
and moreover if it does, there is probably a bug in there.

Of course, existing Boost libraries (with a few exceptions like ASIO) 
have only ever been tested against one C++ 11 STL implementation, the 
one coming with Boost. That is probably a bad thing. I would expect 
any surprises to emerge from binding to another C++ 11 STL to be a 
good thing.

Even with AFIO, I found it broke in useful ways when I ported to 
libc++. That found sloppiness in my lack of use of noexcept which 
merely caused silent poor performance with libstdc++.

> From a distribution point of view, I see some big problems too. I guess the
> idea, if I understand correctly, is to distribute your binding generator
> along with the select-few boost header-only libraries, and then, the user
> would have to run the binding generator for his particular platform to
> generate the implementation-detail binding headers of those boost library
> ports. Or, if you want to spare the user from having to do this, then you
> must distribute a specific version of all the headers that will work for
> their particular platform (OS, compiler and std-lib implementation) and the
> particular versions of those platform components. Either way, this is a
> distribution nightmare.

The idea is that the bindings generator makes a starting set of 
bindings. The library maintainer then adjusts those to fit their 
library and bundles them with their library. The bindings are then 
fixed, and are part of the library implementation. Remember that 
using Boost also goes through bindings, we basically allow the end 
user to choose which STL 11 implementation for a Boost library to 
use. I am proposing that we "virtualise" the STL implementation used 
by Boost libraries so it is end user selectable.

> As for "forking" boost based on this, I don't understand why this is
> necessary, or maybe I don't quite understand what it is meant to really
> involve. Most boost libraries already contain a significant amount of
> conditional code and workarounds to deal with platform-specific issues or
> to use beneficial features that only some platforms support (including
> C++11 features). Why would it be necessary to have a completely separate
> version or distribution of those libraries that are targeted at platforms
> with the (mythical) feature-complete C++11 library implementation? That is,
> if you really mean to "fork" the boost libraries in that way, there is
> going to be enormous costs associated to doing that, mostly in terms of
> maintenance and man-power.

There is plenty of discussion in the archives of this mailing list to 
explain all that, and I would prefer to not repeat all that yet 
again. I will say this: this is part of one low cost path to 
achieving modular Boost. If you consider actual useful modularity 
desirable, this is one step along that path as somehow one needs to 
create separation between a Boost library and its dependencies, and 
it's not like we've been making progress on this for some years now 
other than endlessly debating header analysed dependencies, which 
while helpful doesn't actually achieve real modularisation.

The local namespace binding technique lets the library maintainer 
explicitly mark up which dependencies are substitutable (e.g. STL 
ones), which are mandatory, and which are optional.

We have yet to think of a formal set of guidelines for marking up 
mandatory dependencies and optional dependencies. Baby steps.

> I think that a much better alternative strategy is to do what boost has
> done with Boost.TR1. Why not create a library wrapper like Boost.TR1 but
> for C++11/14? Isn't that what users want, i.e., to be able to use standard
> C++11/14 library components and not have to worry about running their code
> on platforms that don't support it or only partially? I mean, people who
> use Boost.Thread today, instead of <thread>, use it because it works
> everywhere. But if they could include something like
> <boost/cpp11/thread.hpp>, and use "std::thread" and friends, then the only
> remaining platform-specific work would be to install and link against
> libboost_thread for platforms that do not provide the standard <thread>
> library.

Thing is, TR1 was a set of new libraries, so it was straightforward 
to make a Boost.TR1 and it was contained. C++11/14 involves an 
enormous number of changes to *existing* Boost libraries, so we have 
been enhancing Boost libraries with C++ 11/14 features with varying 
rates of progress.

Speaking only as someone on the Boost.Thread team as I can't speak 
much for other Boost libraries ... Boost.Thread already provides a 
significant superset of the C++ 11 STL, and the intent is that by 
next year it shall provide a significant portion of a potential C++ 
17 threading STL too. However, we have a problem: what if a user 
library wants to use Boost.Thread's potential C++ 17 facilities with 
std:: not boost:: threading facilities - a perfectly reasonable 
request, especially if a user library only and exclusively uses 
std::thread et al and has zero wish to buy into the boost ecosystem 
apart from the 17 functionality?

My opinion on this is that the potential C++ 17 features we add to 
Thread also need to be capable of existing in std::experimental, so 
as an example of a current work item of mine, the proposed 
expected<T, E> based non-allocating future-promise could appear in 
either or both of:

1. namespace boost { namespace expected_future { inline namespace v1 
{ ...

2. namespace std { namespace experimental { inline namespace 
expected_future { inline namespace v1 { ...

However, if you think it through, expected_future::future<T, E> needs 
to understand both std::exception_ptr, std::chrono etc as well as 
boost::exception_ptr, boost::chrono etc. If a user configures a 
std::experimental target they may still wish to feed it boost STL 
items in addition to std STL items. They also may wish both 
expected_future implementations to coexist in the same code base, and 
to interoperate seamlessly.

These are some of the issues we face moving forwards. C++ is complex, 
not helped by our collective historical failure to deal properly with 
ODR violation resolution or solving object componentisation as my C++ 
Now 2014 paper describes.

> An additional way of doing this is to simply require that the maintainers
> of the libraries that have been standardized to update their library such
> that it binds to the standard C++11 library implementation (as thinly as
> possible) whenever available and sufficiently compatible (I think that most
> of the boost libraries are today, they seem to have recovered from all the
> mayhem following the C++11 standard and all its slight differences w.r.t.
> Boost). In other words, if you include <boost/thread.hpp> on a
> feature-complete C++11 platform, it should do little more than include
> <thread> and bind its components to the boost namespace.

Requiring maintainers to do anything, including do any maintenance, 
has proven to be hard. We already have many unmaintained libraries, 
some with decade old unfixed issues, so enforcing raised maintenance 
requirements just makes more unmaintained libraries. As a personal 
opinion, I would personally welcome that as an opportunity to 
deprecate and eliminate a large number of deteriorating Boost 
libraries, but mine is not the consensus opinion on this, and is 
definitely not the opinion of the steering committee.

> In both cases, the responsibility of maintaining the C++11/14 compatibility
> in those boost library rests on the shoulder of the individual maintainers.
> No forks required.

At no stage did I ever propose the forked code stops being compatible 
with legacy monolithic Boost. Forked code simply becomes dual use.

Why use the word fork then? It's still a fork in approach, 
methodology, intent, practice and most especially in Boost 
philosophy. It makes possible my alternative method of ranking 
library quality via automated scripts. That is definitely a fork in 
the definition of what qualifies a library to gain the Boost stamp of 
approval.

> And the local bindings (generated or not, from boost to
> std or vice versa) can be maintained more easily. Incompatibilities and
> other platform-specific can be dealt with more practically by manually
> maintaining the bindings. And users simply need to be advised about what
> features a platform should have to allow a specific boost library to be
> used standalone and / or without having to install and link to a
> pre-compiled library.
> 
> Maybe I completely misunderstood your goals, and all I said are things you
> had already considered. I hope so.

I think what I am proposing actually is mostly what you are looking 
for.

> This is just the way I see things. Hearing the words "fork" in the same
> sentence as "boost", gives me chills... (unless it means "fork" only in the
> sense of a temporary developement branch that is meant to be merged back
> quickly).

The other aspect of this being a fork is that new Boost libraries 
joining my modular distro simply wouldn't bother joining the 
monolithic Boost distro. That is already happening anyway to an 
extent - as my C++ Now presentation showed, exciting new C++ 
libraries are increasingly not trying to enter Boost any more and the 
Boost participation rate has been declining steadily for two years 
now.

I am hoping that through modularation and my alternative automated 
ranking of quality, the archaic buy ins demanded by Boost to enter 
Boost go away and thus shall boost (!) the greenfield library 
participation rate. I in particular want 
the boost namespace to go away where possible, it screams 
monolithicism and 
cathedral instead of bazaar. And besides with inline namespaces and 
full fat namespace binding in 11/14, separate namespaces for 
everything is now very straightforward to do - let the user bind 
dependencies into their local namespace instead of Boost 
monolithically deciding for you.

Niall

-- 
ned Productions Limited Consulting
http://www.nedproductions.biz/ 
http://ie.linkedin.com/in/nialldouglas/