Re: Boost.Compute broken in 1.86.0 because of UUID changes, 1.86.1?

From: Boris Kolpackov (boris_at_[hidden])
Date: 2024-08-23 09:29:11

Peter Dimov <pdimov_at_[hidden]> writes:

> Boris Kolpackov wrote:
> >
> > Any sense why this was not detected (e.g., by CI) when the change was made
> > or during pre-release testing? It seems to me compiling any translation unit
> > that included <boost/compute.hpp> would have picked this up, no?
> 
> It would have, but no such translation unit was compiled by anyone before
> the release went out.
> 
> There's no CI that runs the entire Boost test suite (that would take many hours)
> but even if there were, Boost.Compute's tests require OpenCL and automatically
> disable themselves when it's not present.
> 
> This is why, in principle, we do a beta release, to catch these issues before the
> real release, but it doesn't seem to be effective.

I've been thinking about this for some time now so maybe now is a
good time to bring it up: as you may know, we've packaged most[1]
of Boost libraries for build2 (as individual packages) and are
building them continuously on our CI, which covers all the major
platforms and compilers (there is currently a 100 distinct build
configurations[2]).

Currently, we package releases, usually waiting for some time for
any bugs (like this one) to shake out. But we could do that during
the Boost pre-release stage to help detect any regressions if the
Boost community would find this helpful.

Specifically, all the build results are stored in an SQL database and
we can easily run queries like "compare all build results between
these two versions of Boost and list discrepancies" or "compare, say,
Clang 18 build results to Clang 19 and list discrepancies", where the
discrepancy can be errors but also warnings[3].

So, concretely, before a release, we could produce a report that lists
Boost libraries that used to build in a specific build configuration
but which no longer do so (and which likely indicates a regression).

Now, there are a couple of "catches":

Firstly, we haven't yet ported each library's tests to build2. Instead,
for each library we have a smoke test that includes all the public
headers, and, for non-header-only libraries, also calls some non-inline
functions (to make sure the built library is linkable). This should at
least cover things like compilation errors and unresolved symbols.

Secondly, updating our Boost packages to the next release takes some
time, depending on the extent of changes (if there are new libraries,
then we need to port them, etc). Likewise, it will take some time
(days) to build everything on our CI. Currently, we have 7K builds
per Boost version[4]. Putting theses two together, we will need at
least a couple of weeks head start.

Of course, ideally, one wouldn't want to wait until the release to
detect such regressions, instead staging a snapshot of Boost
throughout the development window. But that would require quite
a bit more effort from our side and also consume quite a bit of
our CI resources. But we could consider doing this if there is
funding/help (we can do the before release check for free/ourselves).

[1] The list of excluded libraries (and the rationale for excluding
    them) can be found here:

    https://github.com/build2-packaging/boost/blob/master/exclude

    Ironically, this list includes libboost-compute for the same
    reason: OpenCL dependency. But this can be addressed if desired
    by using a system-installed version of OpenCL (in build2 we
    have system package manager integration for Debian and Fedora-
    derived distributions). It will mean we only test libboost-compute
    in a small subset of build configurations, but it's better than
    nothing.

[2] https://ci.cppget.org/?build-configs

[3] We use a similar approach to test for regressions in build2 itself.
    That is, we rebuild all the packages in the repository with the
    staged version of build2 and compare the result to the released
    version.

    Currently this is done by crafting and running an SQL query but
    in the future we plan to add a web interface for diff'ing build
    results.

[4] https://cppget.org/?builds=libboost-*&pv=1.83.0