Re: A proposal for a modern build-system

From: Marcin Copik (mcopik_at_[hidden])
Date: 2025-05-07 11:24:54

Hi Hassan,

Hi Marcin,
>
>
>> Has this feature been implemented in any compiler? Upstreaming such a
>> change to LLVM would demonstrate that it is mature enough to be used in
>> production software and also provide a stamp of approval.
>>
>
> Not yet. This is the next step. I can not add no-scanning build support
> without this, without which there can not be a useful drop-in header-units
> replacement.
>

I'm not a Boost developer - only user - so I won't be able to comment on
what this community will expect from your tool. However, as a potential
user, I would be very wary of investing time in a tool that requires a
custom and modified fork of an existing compiler. If you wanted to convince
me, I would expect that you already have a fork of Clang that is
up-to-date with the upstream, tested, and demonstrates the promised
performance claims. Furthermore, I would hope to see initial discussions
with LLVM developers that show (a) they are interested in this work and (b)
they are open to merging it with the main distribution at some point.

> It won't be easy to convince the community to switch to a new build system
>> that requires modification to the C/C++ compiler. These are often unlikely
>> to be adopted unless there is a significant demand for that feature in the
>> community.
>>
>
> The changes required are minimal.
>

Yes, I've seen several times the claim that the changes to the compiler are
minimal and would require 1-2 days of work. If that is the case, then why
not create a fork of Clang that shows it?

> Check the README in this repo
> https://github.com/HassanSajjad-302/ipc2978api. I also added small
> modifications to the paper. I am confident that Clang will accept the
> changes. Here is a comment from Clang dev and Tooling Study Group chair
> https://www.reddit.com/r/cpp/comments/1jb8acg/comment/mhvvfva/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button
>
> I would also like to add that this isn't related to the design of modules.
>> Despite lots of claims, I have never seen a proposed design that would
>> actually be any easier to implement in reality. You can make things easier
>> by not supporting headers, but then no existing code can use it. You can
>> also do a lot of things by restricting how they can be used, but then most
>> projects would have to change (often in major ways) to use them. The
>> fundamental problem is that C++ sits on 50+ years of textual inclusion and
>> build system legacy, and modules requires changing that. There's no easy
>> fix that's going to have high perf with a build system designed almost 50
>> years ago. Things like a module build server are the closest, but nobody is
>> actually working on that from what I can tell.
>
>
>
I am not confident that Clang will accept such a change, and the provided
comment seems unrelated to your proposal.

Best,
Marcin

Best,
> Hassan Sajjad
>
> On Tue, May 6, 2025 at 7:07 PM Marcin Copik <mcopik_at_[hidden]> wrote:
>
>>  Hi Hassan,
>>
>> 8) I wrote this paper
>>>
>>> https://htmlpreview.github.io/?https://github.com/HassanSajjad-302/iso-papers/blob/main/generated/my-paper.html
>>> .
>>> It is at most a 2-day work for a compile-dev to support this. This will
>>> allow 4x compilation speed improvement for Boost and even bigger gains
>>> for
>>> bigger projects.
>>>
>>
>> Has this feature been implemented in any compiler? Upstreaming such a
>> change to LLVM would demonstrate that it is mature enough to be used in
>> production software and also provide a stamp of approval.
>>
>> It won't be easy to convince the community to switch to a new build
>> system that requires modification to the C/C++ compiler. These are often
>> unlikely to be adopted unless there is a significant demand for that
>> feature in the community.
>>
>> Best,
>> Marcin
>>
>>
>>> 9) HMake will support a feature called BigHu (described in previous
>>> email),
>>> which makes rebuilds incredibly fast as well. Also, I think header-units
>>> can be backported for older C++ versions as well, since they change
>>> semantics very little ( mostly around the preprocessor ).
>>> 10) HMake will also support
>>> https://lists.isocpp.org/sg15/2023/11/2146.php
>>> and https://lists.isocpp.org/sg15/2023/11/2106.php if these get
>>> implemented.
>>> 11) compile-commands.json support. HMake will generate these at configure
>>> time for the configurations being configured.
>>> 12) CMake integration support through CPS. Some features, like
>>> header-units
>>> might not work.
>>> 13) HMake will support multiple other programming languages, frameworks
>>> like iOS and Android app generation, different installers, etc.
>>>
>>> HMake is still a novel build-system that is under heavy development. A
>>> project of Boost size using this in production will provide highly
>>> valuable
>>> insights that will tremendously improve the HMake itself as well. HMake
>>> is
>>> written in C++20, follows best practices, is extensively tested, uses
>>> TDD,
>>> and the core library is around 15k C++ lines.
>>>
>>> I can also do a video call presentation of my software. This might be a
>>> quicker way to make yourself familiar with my software capabilities than
>>> reading documentation. Because I can do live examples and explain details
>>> accordingly.
>>>
>>> Best,
>>> Hassan Sajjad
>>>
>>> On Wed, Apr 16, 2025 at 5:21 PM Hassan Sajjad <
>>> hassan.sajjad069_at_[hidden]>
>>> wrote:
>>>
>>> > The interactions to reproduce the results.
>>> >
>>> https://github.com/HassanSajjad-302/HMake/blob/main/Projects/Boost/hmake.cpp
>>> >
>>> > On Wed, Apr 16, 2025, 16:34 Hassan Sajjad <hassan.sajjad069_at_[hidden]>
>>> > wrote:
>>> >
>>> >> Hi.
>>> >>
>>> >> I compiled 25 Boost libraries with C++20 header-units, including
>>> examples
>>> >> and Tests of a few.
>>> >> This resulted in a 1.44x speed-up if we exclude the scanning. Scanning
>>> >> was as slow as the header-unit build.
>>> >>
>>> >> I have a solution for that. I am sending this paper (Making some final
>>> >> edits)
>>> >>
>>> https://htmlpreview.github.io/?https://github.com/HassanSajjad-302/iso-papers/blob/main/generated/my-paper.html
>>> >> ,
>>> >> which allows for a build-system to use a scanning-less approach to
>>> build
>>> >> header-units.This method requires only minimal modifications to the
>>> >> compiler and enables memory-mapped file sharing, which significantly
>>> >> reduces filesystem overhead and further boosts speed.
>>> >>
>>> >> My proposal for a scanning-less approach will allow HMake to have
>>> another
>>> >> amazing feature, monolithic Big header-units. BigHu will allow you to
>>> map
>>> >> all the headers from a directory to a single header-unit. This
>>> header-unit
>>> >> includes all these header-files and consumes them as header-files.
>>> Now,
>>> >> whenever the compiler requests a header-unit from this directory, the
>>> >> build-system instead sends this consolidated BigHu. This will not only
>>> >> improve the clean build speed, because of fewer process launches. But
>>> also
>>> >> massively boosts the rebuild speed as few header-units need to be read
>>> >> instead of lots of files. No source-code editing will be needed to
>>> use this
>>> >> feature. And you will have the option to either build a library as
>>> BigHu
>>> >> (if you don't mean to edit it) or as small header-units (if you are
>>> working
>>> >> on it).
>>> >>
>>> >> HMake will soon support compile_commands.json, which I guess can work
>>> >> fine with header-units. Thus you can benefit from both, faster
>>> compilation
>>> >> and working intelliscence.
>>> >>
>>> >> The API shown in hmake.cpp is a work in progress. HMake build-system
>>> >> benefits are numerous, even if we leave out the header-units support.
>>> This
>>> >> makes it a perfect replacement for B2. I believe we can complete the
>>> switch
>>> >> in the next 2 months. And if some compiler supports my paper, we get
>>> >3x
>>> >> faster compilation as well.
>>> >>
>>> >> Best,
>>> >> Hassan Sajjad
>>> >>
>>> >>
>>> >>
>>>
>>> _______________________________________________
>>> Unsubscribe & other changes:
>>> http://lists.boost.org/mailman/listinfo.cgi/boost
>>>
>>