Re: So.... what are we going to do about modules?

From: Stephan T. Lavavej (stl_at_[hidden])
Date: 2022-04-05 22:40:07

[John Maddock]
> 7) Does anyone have a handle on the current state of std library 
> modularization?

[Daniela Engert]
> It is voted into the C++23 draft.

Slight correction: P2465R3 has been approved by LEWG, LWG, and CWG, and is ready for the next plenary, but it has not yet been voted into C++23.

> Stephan (STL) has this currently on his plate and is actively working on it.
> I expect this to see this Real Soon Now in msvc.

Yep! :-) I am auditing every line of our headers (~147k lines) and am currently 66.3% done with that. The majority of the work is minimally invasive (adding an internal macro that expands to `export` or nothing for everything that should be part of the module*) and semi-mechanical. However, we're finding and fixing compiler bugs along the way, and we still have some questions to resolve around how to deal with our separately compiled components (we are especially constrained due to bincompat; I expect Boost to have an easier time as it can break bincompat at will), exactly how our .ixx files should ship and be built, etc.

* As I understand it, there is a less invasive but more repetitive alternative: adding declarations of everything that needs to be exported. This would be fairly easy for things that mostly consist of classes or even class templates, but when there are lots of functions and function templates, it would involve a fair amount of duplication and having to keep things in sync. 

Regarding C++20 Standard Library Header Units, they should work (and we have active test coverage for this), although you may need build system work (as I understand it, MSBuild is ready, CMake/Ninja is not yet ready), and you may encounter compiler bugs as you use the library more intensively (as this is very new - our tests validate the basic scenario, but the total amount of usage is still small).

Getting the STL to be compatible with header units didn't require auditing every line of the headers, but it did involve writing test coverage to exercise every header, and reporting many many bugs found along the way. Now that they're resolved, Boost should have an easier time, although I would still expect some compiler issues to be found (as Boost is both larger than MSVC's STL, and has different patterns of code). You may also need to refactor certain patterns that are incompatible with header units (notably any expectation that source files can define macros to affect the behavior of headers).

There are two ways to build Standard Library Header Units - building each header independently is easier, but results in a lot of duplication. You can instead build them in a deduplicated manner, by asking the compiler to report (via JSON) how X includes Y, and then topologically sorting the graph. I have Python code in our test harness that does this (which could be used as a reference for a real build system - see https://github.com/microsoft/STL/blob/52505b9d3249c50b33eb22152a1f18f208ea2959/tests/std/tests/P1502R1_standard_library_header_units/custom_format.py#L55 ). The topo sort results in the smallest (i.e. most efficient) IFC files that you can get, short of the proposed `import std;` that I'm currently working on.

The tracking issue (originally for Standard Library Header Units, now also for Standard Library Modules) is https://github.com/microsoft/STL/issues/1694 . It primarily records the compiler bugs we've found and fixed along the way.

> From their Github I have gleaned the info that there is a file in there 
> that tells the compiler which includes can be implicitly translated into 
> imports.

Yes, this is header-units.json (see https://github.com/microsoft/STL/blob/52505b9d3249c50b33eb22152a1f18f208ea2959/stl/inc/header-units.json ). This was designed so that libraries like Boost can also opt-in by adding such files.

Thanks,
STL