Re: Interest for C++20 modules support of boost officially

From: Chuanqi Xu (chuanqi.xcq_at_[hidden])
Date: 2024-12-23 02:59:42

Hi Ruben,
 thanks for your detailed explanation and now I get your points and concerns. 
> Following this approach, dependencies are automatic. But this has a
number of downsides:
> 1. At the moment, BMIs are built with the settings used to build Boost
in the first place, which causes a lot of trouble.
> 2. The installation generates many more binaries than in the headers
world. It is uncertain whether these binaries will be compatible if
build flags change.
> 3. Some libraries support configuration macros. For instance,
Boost::throw_exception supports BOOST_NO_EXCEPTIONS. This is used by
other Boost libraries to support builds that disable exceptions. In
the current world, the end user defines BOOST_NO_EXCEPTIONS in their
code, includes the library, and everything works. Using a scheme like
the one I suggested requires the user to build all the Boost modules
using BOOST_NO_EXCEPTIONS, which is more involved than what we have
today.
Understood your concerns. For 1, it is (or meant to be the responsibility) of build systems. They should take care of it. But as you mentioned, here is not ETA for it.
For 2, IIRC, it won’t be a problem since all the installed binaries really contains an initializers for modules only. And if not, I guess it would be problematic already.
For 3, I am wondering if this is similar with 1.
And also, I think your suggestion to ask the customers to build and consume the object files for consumers make since too. Although there was opinions that this strategy may produce ODR violations, it should be fine for a lot of existing libraries that provide module interfaces as wrappers of headers. The above theory for ODR violations is, every *.cppm file should map to a single and unique object file. So that we won’t have multiple definitions for the same declarations in a program. But for interfaces of modules of header wrappers, the object file should contain a single initializer. So it may be fine for the specific case.
So my conclusion is, we may have following alternatives:
(1) Just install the binraries. And wait for CMake to handle the problem of different settings. Note that all the libraries including the standard library face the same issue now.
(2) Ask the consumers to build and consume the object files. Achieve this by asking and waiting CMake to implement a new feature for this.
(3) Ask the consumers to build and consume the object files.. Achieve this by writing some cmake scripts? Sorry I am not a Cmake expert so I can’t give a concrete suggestion. I am just wondering if it is possible to make it by some cmake scripts.
(4) Do not install modules right now but provide a way for users to consume modules with local build. The libc++ has similar period: https://libcxx.llvm.org/Modules.html#using-the-local-build <https://libcxx.llvm.org/Modules.html#using-the-local-build > . (there was almost 1 year between libc++ support modules and cmake support import installed libc++ modules, if I remember correctly.) This sounds not ideal. And as pointed by other thread, a lot of users care about installed uses. But if our time are limited, and we don’t write cmake scripts again and again, maybe it is fine to refactor the source only and providing a way to use modules for local builds. e.g., we’re using bazel and it is good enough for users like us (and probably of build2 and cmake users who like fetch_contents). It is not ideal for sure, but it works for some user. And we can add the support of installataion when every thing has a solution.
Thanks,
Chuanqi
------------------------------------------------------------------
From:Ruben Perez <rubenperez038_at_[hidden]>
Send Time:2024 Dec. 22 (Sun.) 17:39
To:Chuanqi<chuanqi.xcq_at_[hidden]>
Cc:Ruben Perez via Boost<boost_at_[hidden]>
Subject:Re: [boost] Interest for C++20 modules support of boost officially
On Fri, 20 Dec 2024 at 03:15, Chuanqi Xu <chuanqi.xcq_at_[hidden]> wrote:
>
> Hi Ruben,
>
> I think we’re in the same page except a few wording issues. e.g, I won’t call it as pre-compiled headers. But this doesn’t matter.
>
> > 2. Boost.Url in the module world would install:
> > * libboost_url.a, containing the same function definitions as above.
> > * A boost_url.cppm file, akin to what headers are today.
> > CMake would build a BMI from this file when a user needs to
> import boost.url.
> > In practice, this cppm file will likely be implemented in terms of the
> > header files that we have today.
>
> The libboost_url.a with modules support may have the same entities with original libboost_url.a **plus** the module initializer.
>
> And also, as discussed in other mail, the headers will be installed too.
Yes, completely. I'm building a small prototype to show how things
would work, as discussing with concrete code is much more fruitful.
>
> > Unfortunately, CMake doesn't seem to have first-class support
> for this, meaning that the user would need to figure out dependencies
> manually.
>
> What do you mean by figuring out dependencies manually? Do you mean, in cmake, the dependencies will be calculated if the libraries are linked? If yes, I feel it might be fine. Or did I misunderstand it?
Consider Boost.variant2. It's a header-only library that depends on
Boost.assert, Boost.config and Boost.mp11, which are also header-only.
This is what the CMake code to build these as modules could look like
(simplified - I will post the entire prototype when ready):
add_library(boost_variant2)
target_sources(boost_variant2 PUBLIC FILE_SET CXX_MODULES FILES
modules/variant2.cxx)
target_include_directories(boost_variant2 PUBLIC include)
target_link_libraries(boost_variant2
 PUBLIC
 Boost::assert
 Boost::config
 Boost::mp11
)
Here, Boost::assert and Boost::mp11 have been created with similar
code as the one shown above. Boost::config is an interface library
with only headers (as it only exports macros).
The targets are installed by the Boost.CMake infrastructure, with a
call akin to:
install(TARGETS ${LIB} EXPORT ${LIB}-targets
 RUNTIME DESTINATION "${CMAKE_INSTALL_BINDIR}"
 LIBRARY DESTINATION "${CMAKE_INSTALL_LIBDIR}"
 ARCHIVE DESTINATION "${CMAKE_INSTALL_LIBDIR}"
 PRIVATE_HEADER DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}"
 PUBLIC_HEADER DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}"
 FILE_SET CXX_MODULES DESTINATION .
)
When building Boost with CMake, for each module (Boost::assert,
Boost::mp11 and Boost::variant2), a BMI and a static library with the
module initializer is built.
When installing Boost (with CMake), for each module (Boost::assert,
Boost::mp11 and Boost::variant2), the headers, module code and static
library is installed.
When consuming Boost from another project, e.g. using
find_package(Boost COMPONENTS Boost::variant2), the headers, module
code and static library for Boost::variant2 and its dependencies are
found and used. The BMIs are built in the consumer project. Right now,
the flags used originally to build Boost are used to re-build the
BMIs. This causes trouble, but is intended to change in the future,
with an unknown time frame [1].
Following this approach, dependencies are automatic. But this has a
number of downsides:
1. At the moment, BMIs are built with the settings used to build Boost
in the first place, which causes a lot of trouble.
2. The installation generates many more binaries than in the headers
world. It is uncertain whether these binaries will be compatible if
build flags change.
3. Some libraries support configuration macros. For instance,
Boost::throw_exception supports BOOST_NO_EXCEPTIONS. This is used by
other Boost libraries to support builds that disable exceptions. In
the current world, the end user defines BOOST_NO_EXCEPTIONS in their
code, includes the library, and everything works. Using a scheme like
the one I suggested requires the user to build all the Boost modules
using BOOST_NO_EXCEPTIONS, which is more involved than what we have
today.
For this reason, it makes sense to consider whether there is a way to
completely avoid the generation and installation of these binaries for
header-only libraries, and follow a model similar to standard modules.
In this ideal model, installing Boost with CMake would only install
the headers and module code for each header-only library, and no
binary artifacts. The consuming project would build both the BMIs and
the objects containing the module initializers with the adequate build
flags, as many times as required by the different targets.
Ideally, the following syntax would work and do what I'm suggesting:
# This does NOT work as of today
add_library(boost_variant2 INTERFACE) # Don't build the library in the
current project
target_sources(boost_variant2 INTERFACE FILE_SET CXX_MODULES FILES
modules/variant2.cxx)
target_include_directories(boost_variant2 INTERFACE include)
target_link_libraries(boost_variant2
 INTERFACE
 Boost::assert
 Boost::config
 Boost::mp11
)
In the ideal world, this wouldn't build any binary library or BMI per
se. When linking to the boost_variant2 target, a BMI and a static
library would be built with the flags required by the consuming
target. Note that this does NOT work as of today [2]. I've raised a
feature request to the CMake team with this suggestion [3], but I
don't think this is happening in the short-term.
Given that this does not work, the other option would be installing
the headers and the module code manually. For instance:
# This is what we have today. Note that we have removed the target_sources
add_library(boost_variant2 INTERFACE)
target_include_directories(boost_variant2 INTERFACE include)
target_link_libraries(boost_variant2
 INTERFACE
 Boost::assert
 Boost::config
 Boost::mp11
)
# Install the module code separately
install(DIRECTORY modules DESTINATION .)
With this setup, it is the user's responsibility to issue the
target_sources calls required to build the boost_variant2 module.
There is also nothing telling CMake that the boost_variant2 module
requires building boost_assert and boost_mp11, though. Thus, it
becomes the user's responsibility to know this dependency chain and
add the relevant target_sources calls. This is what I mean by
"figuring out dependencies manually". While it may not seem that hard
in this case, things get much worse as you study libraries with more
dependencies. I'm the author of Boost.MySQL, and I wouldn't be able to
spell you the current dependency tree without an automated tool to
help.
I hope this helps clarify what I meant.
If I have missed anything obvious, or anyone has ideas on how to make
this better, please let me know.
>
> Thanks,
> Chuanqi
>
Regards,
Ruben.
[1] https://discourse.cmake.org/t/advice-on-c-20-modules-boost/10641/13 <https://discourse.cmake.org/t/advice-on-c-20-modules-boost/10641/13 >
[2] https://cmake.org/cmake/help/latest/command/target_sources.html#file-sets <https://cmake.org/cmake/help/latest/command/target_sources.html#file-sets >
[3] https://discourse.cmake.org/t/distributing-c-20-modules-as-source/13246 <https://discourse.cmake.org/t/distributing-c-20-modules-as-source/13246 >