The Future of Boost

From: Vinnie Falco (vinnie.falco_at_[hidden])
Date: 2023-05-08 15:46:03

Greetings!

I’m Vinnie Falco, Boost library author, C++ enthusiast, and the
founder of The C++ Alliance, a 501(c)(3) non-profit. While some of you
are enjoying the C++Now conference this week, I’d like to share some
background on our organization, provide some history, outline a vision
and goals for C++ and Boost, and solicit your feedback and support.

How It Started

I took notice of the C++ Standards Committee (“WG21”) while I was
writing Boost.Beast in 2016. Howard Hinnant, a co-workers at Ripple,
taught me about writing papers and committee meetings. Beast used
Boost.Asio (portable networking for C++) and I was and still am a huge
fan of this network library. I learned that Asio was being proposed
for standardization. There was even a “Networking TS” document: Asio
was very close to becoming part of C++ officially! But the author
Christopher Kohlhoff always seemed to not have the time to attend the
meetings and push this proposal though.

Something which should not surprise anyone is that I despise paying
taxes. In 2017, I had an idea: create a charitable organization which
I can donate pre-tax income to, and then I could hire Christopher
Kohlhoff as a “staff engineer” to work full time on C++
standardization, and Boost things! I would find the very best C++
people who are already doing open source work, then hire them
full-time so they could focus on their open source C++ open work from
home, instead of traveling to a boring job in order to make a living.

A Few Setbacks

In 2018 I offered this opportunity to Chris and he surprisingly turned
it down. He actually liked going into an office and interacting with
customers and users. He explained that the evolution of Asio and his
WG21 work is not bottlenecked by time. Instead, he prefers to “think
deeply about things over a long period, and then write something.”
Basically the opposite of my strategy, which is to write a bunch of
code quickly and then throw out the bad parts.

This is a setback but I am not so easily deterred so I offered the
same opportunity to Peter Dimov, an engineer of immense talent whose
libraries are legendary. He also declined, explaining that taking a
salary would transform a hobby into an obligation, affecting the
quality and enjoyment of the work.

Now I’m thinking, well this is a disaster! We had the non-profit in
operation officially since March of 2018 (the IRS approved us in
September of 2019). We had the C++ language Slack workspace as of
November of 2017, transitioned to a paid plan with full history. Our
strategy shifted to focus on supporting the Boost Library Collection
directly. We hired our first Staff Engineer, Marshall Clow, in April
of 2018.

Fast forward and today we have 11 staff members. We have a great
CTO/DevOps genius Sam Darwin. And we have Louis Tatta, our CEO that
keeps things running smoothly and helps get the most out of every
dollar donated. At some point I’ll share a complete list of everything
that The C++ Alliance has done since the beginning, but that is the
subject of another missive. Today I would like to talk about a vision
for Boost.

The Boost Library Collection

Long-timers know Boost’s history but for those that don’t, Beman Dawes
and Robert Klarer came up with the idea of a website offering curated,
high quality C++ libraries in May of 1998. They described the “Formal
Review,” a social technology where a group of peers would go over a
proposed library at an agreed-upon time. They could interrogate the
author about the library on the mailing list, and debate things. The
outcome is a collection of posts where each reviewer summarizes their
critique of the library, including whether or not to “accept” the
library (sometimes with conditions). The founding documenting evokes a
feeling of something big:

https://www.boost.org/users/proposal.pdf

The collection was named “Boost” and received many great
contributions. The authors and reviewers were active in the
standardization committee. In December of 2005, Boost.Asio was added
after being developed since 2003. In September of 2011 the C++11
standard was published, containing many library components modeled
closely or identically to their Boost counterparts. In my opinion,
Asio’s efforts at standardization were thwarted by the growth of
politics; an inevitable consequence of the bureaucratic ISO structure.

Boost launched its own conference called BoostCon in 2007 on the heels
of its success. Speakers included Scott Meyers, Dave Abrahams, Eric
Niebler, Howard Hinnant, and other juggernauts of C++. A new
conference called CppCon was launched in 2014 and attracted even
larger crowds, as it was focused on C++ in general.

Trouble Brewing

With the release of C++11, there were now components in Boost which
were duplicated in the Standard Library. The C++ committee became more
popular and valuable owing to the success of C++11, made possible in
part by years of lead-up from the talented Boost engineers. The
conferences turned some people into the equivalent of pop stars,
appearing as staple keynote speakers.

Library writers discovered it was easier to get a proposal into the
C++ standard than it was to get a library through the Formal Review
process. They discovered that there was more glory to have a proposal
accepted into the official C++ language, than to have their library
accepted into Boost. And once their proposal became part of C++, they
no longer had to “maintain their code” the way they would if their
library got in Boost. A Formal Review evaluates the author somewhat in
addition to the library. Because once a library is accepted, it must
be maintained and evolved. When an author abandons their Boost
library, the community is impoverished as the knowledge and expertise
leaves with them. And someone else must take over the maintenance.

In December of 2020, Beman Dawes passed away and Boost suffered a loss
which can never be replaced. Beman had an enormous impact on not just
the libraries but also C++. He was in WG21 from the very beginning,
chaired LWG (the “Library Working Group”) for quite some time, and
achieved a long history of open source contributions.

Boost had other problems. Fewer libraries were being proposed, and it
took longer to find a volunteer review manager. Mailing list volume
declined steadily. At 160+ libraries, users complained that “Boost is
too large.” They complained that “many of the libraries are outdated”,
that “the documentation is of varying quality”, and that “Boost takes
too long to compile.” They complained about the obscure build system
and lack of cmake support. The archetype of “never Booster” appeared:
individuals or corporations who ban the use of Boost entirely.

Beman was the closest thing resembling a leader, despite Boost being a
federation of authors with each having final word over their own
library. Beman would solve problems, help with the direction of
things, and even “beat the bushes” when it was time for a review by
reaching out to his network of contacts and soliciting their
participation. Without Beman, Boost lost its leader. Boost lost its
Great Founder. And no one has since filled the role.

The C++ Alliance

At this point, a vision for what our non-profit could do crystallized.
We would help C++ in general by refreshing the foundations of Boost,
restoring Boost’s prominence in the community, and helping Boost
become a leader once again as innovators in C++. To do this, I'll
share what we feel are the problems facing Boost, and ways to address
some of them. Finally I'd like you to weigh in on all of this and help
figure out what is important and what successful execution might look
like.

We believe Boost faces these obstacles:

Stagnation
* There are fewer new libraries proposed.
* Formal reviews get less participation.
* Review managers are typically scarce now.
* The mailing list volume is thinning; younger folks don’t use lists.
* There is no second order effect: new libraries rarely use Boost.

Quality
* Some libraries are unmaintained and create a negative user experience.
* Users open issues, and no one replies to them.
* Pull requests are submitted to abandoned repositories.
* Scant financial resources for infrastructure or staff.

Documentation
* The quality of documentation varies greatly across libraries.
* The rendered pages and content of some documentation looks dated.
* Some toolchains used are obscure and unmaintained.

Perception
* Boost causes long compile times.
* The libraries have too many interdependencies
* Supporting old C++ versions is a weakness not a strength.
* The duplication of std components is wasteful and causes friction.
* The “Monolithic” distribution of Boost is obsolete.

Messaging
* The website is outdated and never receives updates.
* Boost’s value proposition is not clear (“why use boost?”)
* There is no clear voice countering misconceptions and irrational phobias.
* Users receive no guidance about the future, or what is maintained.
* The libraries have no representation at conferences.

Some users have also weighed in with thoughts on Boost:

https://www.reddit.com/r/cpp/comments/gfowpq/why_you_dont_use_boost/

A Plan

I love C++, supporting users, and the Boost author experience. I think
these problems can be solved. But not by demanding that “everyone who
maintains a Boost library must do X.” In Boost culture when you want
something done you need to do it yourself, then convince the mailing
list of the merits of your proposal.

As a library author and contributor, I know that whatever I do will
never rise to the same level as the original act of the creation of
the Boost Library Collection. But I will be satisfied if I can stoke
its fires and bring them back to a roar. To this end the resources of
the non-profit are directed into projects we believe will positively
affect Boost:

Website Renovation

Our vision for an updated Boost website is clean and stylish, which
speaks to a large and diverse audience. This site will have a design
and content that effectively communicates the value proposition of
using the Boost Library Collection: that you will write better C++
code, become more productive, and achieve greater success in your
career or personal projects. Features will foster participation and
revisits, with content updated regularly. The library presentation is
elevated with a new visual design language that evokes distinction and
appeal, and credits the authors, maintainers, and contributors that
bring it to life.

To achieve this vision, you have probably heard that we contracted an
outside software firm to build something precisely tailored for our
needs. We care too much about Boost to use an ill-fitted, off the
shelf product. This website has a lot of software behind it (written
in Python as part of a Django framework application) and like most
software projects it is late and over budget. I’ll refrain from saying
“it’ll be ready soon” and just post a link to the new site instead,
hopefully in a few weeks.

I have been personally involved in the design, presentation, and
execution of the features of the website, most of which have been cut
from the initial release in order to speed things along. The goal is
to show the library collection in a way that highlights its strengths
and speaks to a desire of every C++ programmer: to find the perfect
library they can add as a dependency to help complete their next
project.

The Boost website and the site documentation can be illustrated by
retaining a talented digital artist to produce custom assets that are
unified in style, colors, and messaging, so that the entire site feels
purposeful. This artist will also provide imagery used for our social
media campaigns such as the announcements we make on Twitter which
some of you might have already seen

https://twitter.com/Boost_Libraries.

I strive  to give every tweet an image to enhance the Boost brand
(tweets with images have significantly increased engagement).

Recently an animated discussion on the mailing list took place about
adding a forum which does not replace the mailing list but is
integrated to work with it. Posts in the forum become posts to the
mailing list, and vice versa. Users of the mailing list and users of
the forum will have no idea they are interacting, even though they
are. This can only be possible if we write the software ourselves,
from the ground up, with exactly one constraint: the mailing list will
continue to operate exactly as it does today, on an unmodified version
of Mailman 3. The mailing list users stay happy, and we can attract
new people who prefer a web-based interface.

The C++ Alliance prioritizes its allocation of resources to ensure not
only the website’s completion, but also dedicated staff for ongoing
maintenance and improvement. The Boost website will rise over time to
the same level of quality expected of every Boost library. Community
members should feel free to open issues on the website repository with
bugs or features, knowing that every issue will be looked at, triaged,
and addressed appropriately.

Documentation Improvement

Our vision for documentation is to ensure that every Boost library has
the option to adopt a well-maintained toolchain that is easily
deployed, produces high-quality output befitting the Boost brand, is
itself well-documented and easy to use, and has behind it full-time
staff working continuously to make improvements and provide technical
support.

After researching the domain extensively (by just asking Peter Dimov)
we have discovered that the markdown format Asciidoc is a very popular
format with a simple and well maintained toolchain. Several regularly
active Boost authors have already switched their libraries to using
Asciidoctor. The authors of the Asciidoctor tool are also the authors
of “Antora,” a modular, multi-repository documentation site generator:

https://docs.antora.org/antora/latest/

We have built a new, modern set of additional scripts capable of
building the Boost release and documentation, including the capability
of rendering “Antora-enabled Boost library repositories” using this
Antora system. The results are beautiful and modern, and the
Asciidoctor / Antora toolchain holds the promise of being popular and
well-maintained for a long time. The use of Asciidoc or Antora is
optional; this is just an additional choice.

Peter Turcan is our full-time Senior Technical Writer who is
modernizing the instructions for users, maintainers, contributors, and
formal review participants. You can see Peter’s work along with the
quality of Antora’s output here (note that the user-interface is stock
and will be restyled soon):

https://docs.cppalliance.org/

The website above has a new full-text search feature (try it!). We are
investing in a  search experience which includes the site docs,
library docs, library references, and even the public header files. We
are also investing in the deployment of a large language model
(ChatGPT-style AI) trained in Boost and C++ specifics to answer
questions for users. We have a new talented and eager staff engineer
working full-time exclusively on this, and I don’t want to steal his
thunder so I will let him explain further soon.

Some Boost libraries currently generate their documentation reference
pages using Doxygen combined with other obscure tools such as xsltproc
or Saxon-HE to render into Boost Quickbook, an obsolete form of
markdown which only we use. This Quickbook is rendered into BoostBook,
which is a flavor of DocBook. The BoostBook is converted into HTML by
a DocBook renderer. This rapidly obsolescing toolchain is painful to
work with and is a form of technical debt which costs us.

I have begun work on a new command-line tool called MrDox (“mister
docs”) which uses the unstable clang libtooling API to extract the
documentation comments and declarations from C++ programs, and turn
them into beautiful Asciidoc reference pages. You can see that work
here:

https://github.com/cppalliance/mrdox

The core principles of the design of MrDox is to always understand the
very latest C++ constructs and extract them with high fidelity. For
example it recognizes conditional noexcept, constexpr, deduction
guides, all attributes, and many other things that other documentation
toolchains cannot fathom. In a nutshell I intend to bring the same
level of Boost quality to the documentation toolchain that Boost has
brought to the C++ libraries themselves.

MrDox intends to completely replace Doxygen, xsltproc, Saxon-HE,
Quickbook, Boostbook, and Docbook, as the only requirement to render
its results is to run the Asciidoctor tool, which has no other
dependencies. This toolchain offers modernization and simplification
for anyone who opts-in to it, which reduces long-term risks and
improves results. This unfortunately delays the development of my
other libraries, but enhancements in the documentation toolchain are a
force multiplier; many Boost libraries can benefit.

Continuous Integration

Our vision for continuous integration is to bring the most talented
individuals together and combine that with state of the art technology
and resources to ensure that every library has at its disposal, access
to robust cloud services for continuous integration. These services
are the lifeblood of maintaining and communicating the quality of a
library. We aim to provide dedicated staff and technical support to
fortify Boost in the ever-shifting landscape of free CI services for
open source projects.

The infrastructures providing our continuous integration services are
the lifeblood of maintaining the high quality of the Boost collection.
Library authors test against many versions of C++ and many different
compiler versions. And we have many libraries; over 160 of them which
all compete for the finite public resources offered by GitHub through
GHA, through Azure Pipelines, or Appveyor.

When Travis discontinued its free service, our CTO Sam Darwin deployed
Drone (https://www.drone.io/) instances and offered every Boost
library a pull request which compiles and runs their tests on our new
infrastructure. Although this service is still active and offered
today, we are not content to leave it at that. CI services are
volatile over time. Some come, some go, and some become overloaded
which is the current situation with the public GitHub Actions runners
during peak times. The Boost organization GitHub account has over one
hundred and sixty libraries each submitting sometimes enormous numbers
of jobs which take multiple hours to complete.

Although the GHA environment resources are subjected to recurring
oversubscription, we feel that it offers the best framework for
composable actions and flexibility. Sam is exploring the possibility
of having self-hosted C++ Alliance runners dedicated only to Boost
jobs during peak times. Ensuring high availability of CI resources is
an ongoing project for us, and we are always evaluating existing and
new solutions to provide the best-of-class choices for libraries.

Library Enhancements

Our vision for the libraries themselves is to preserve unchanged the
amazing social technologies invented by the Boost founders which
include the Formal Review process, the Release Schedule, the mailing
list discussions, and the federated library ownership model. We want
to ensure that no library is unmaintained and that every opened issue
receives a response. We want the community to respect and admire the
formal review process and participate with eagerness not only as
reviewers but also as volunteer review managers and participants in
the sometimes-heated list discussions. Library membership in the Boost
library collection should be seen as the highest level of honor and
recognition of effort.

The C++ Alliance has ongoing direct investments in improving existing
Boost libraries and writing new ones to be submitted for formal
review. Many folks are already aware of the optimization efforts being
applied to the Boost.Unordered library, whose plan was written up by
Peter Dimov. Joaquín M López Muñoz is providing his mathematical
expertise and container experience, while Christian Mazakas (one of
our full-time staff engineers) is writing the implementation, tests,
and documentation according to specification.

People following Boost.Math might recognize Matt Borland as a regular
contributor. He has joined us as a staff engineer and is currently
working on a new library to be proposed for Boost: charconv, which is
a C++11 port of the eponymous C++17 feature. This library will help
libraries and users who may not have access to C++17 enjoy the same
features through Boost instead.

Messaging and Direction

Our vision for Boost includes clear messaging to inform the public on
the status of the libraries, the challenges we are facing, and what
our future direction might be. We believe in robust two-way
communication between library authors and maintainers, and the
stakeholders which are largely the people and companies which use the
Boost libraries. We believe in having a social media presence that
helps convey the prestige and status that comes with the quality Boost
libraries offer.

Currently we have only anecdotal evidence of Boost’s adoption (or lack
thereof) in various companies and projects. We only hear from the
people who complain or open issues, or post to the mailing list. We do
not have a concise list of companies using Boost, when new companies
adopt Boost, or when companies stop using Boost. We do not have
feedback from stakeholders about which Boost libraries they rely on
the most, what they would like to see in future versions, or in some
cases even if they are having problems with a library or its
documentation.

The decentralized model of Boost library development works great for
the problems it tries to solve but offers no overall directional
guidance for Boost. Today the C++ language is facing unprecedented
challenges: the popularity of Rust, the demands for “memory safety”,
the rise of Artificial Intelligence capable of writing software
independently, and possibility that the bureaucratic structure of WG21
renders it incapable of meeting these challenges in a lively or
effective manner.

We believe that Boost can offer the greatest value by focusing in the
areas where C++ is strong and without meaningful competition. These
include space exploration, game development, high-performance
computing, embedded systems, the Internet of Things, robotics and
industrial process control, financial services, computer vision and
graphics, scientific simulation, and more.

Furthermore the stunning and continued lack of networking in the
standard library creates an opportunity for Boost to offer full-stack
solutions in areas that speak to the strengths of C++. This is made
possible because Boost already offers portable networking through
Asio, HTTP and Websocket through Beast, excellent JSON parsing and
serialization tailored for network programs, URLs, and more recently a
Redis client (Boost.Aedis) and even a MySQL / MariaDB client. We
intend to sponsor the development of non-Boost, open source
applications and services that target specific underserved markets
that would benefit from C++ solutions which use the excellent
libraries that exist in Boost.

Where Do You Fit In?

Our vision, our activity, and our deployed solutions are all “opt-in.”
No one controls Boost or its libraries. Change is only possible with
consensus of the folks that matter: authors, maintainers, and release
managers. If Robert Ramey wants to keep his documentation in
hand-written HTML that is entirely his choice; no one dictates what
libraries do. We can only offer choices, and hope they will be seen as
valuable.

This has been a long read, and I appreciate your investment of time.
How do you feel about this vision? What would you change, or add, and
what needs work? We welcome feedback, and value the volunteers who
share our vision and help us execute it.

I invite you to stay tuned for more great news, coming soon!

Respectfully Yours

Vinnie Falco