Github Actions CI updates - e.g. Node 20
Dear maintainers, I made a bunch of PRs to fix the Github Actions CI for almost 40 repositories. Opening a PR in GitHub is manual work while I was able to apply similar changes over all those repositories using a script which commits to a single branch (and hence PR) per repo. Hence the PRs contain a few related changes that in combination allow the CI to pass again. To avoid repeating this in each PR I wanted to shortly explain the changes here: 1. macos-12 runners are dropped by github and jobs won't start anymore. at least macos-13 needs to be used 2. Ubuntu 23.* doesn't work anymore as e.g. some APT repositories don't exist anymore. Ubuntu 24.04 can be used instead 3. clang 17 (and partly 16) don't work in C++23/2b mode with the stdlibc++ (12) installed on ubuntu 24.04, clang 18+ and/or the other C++ standards can still be used 4. Support for Node 16 was recently fully dropped by Github Actions so the @v3 actions must be upgraded to @v4, which require Node 20. The workaround using $ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION has no effect anymore 5. The node 20 version available by default conflicts with the glibc version in the Ubuntu 16/18 container images used for testing older compilers. I addressed all of them with last one being the major one: Installing a compatible version inside a custom image to be used instead doesn't work because GHA mounts the node 20 installation from the host os. The solution I implemented downloads an unofficial node 20 built against the correct/older glibc and creates 2 mount points such that we can put that node onto the host runner and mount that instead of what github gives us using the idea from https://github.com/dixyes/ghactionsplay/blob/bf889ee2fed8dcd72cc8e32bfd4789b... To avoid overloading the unofficial-builds.nodejs.org server we use a copy hosted at boost.io Download and extraction requires curl and xz-utils to be installed which I added where required. Additionally I noticed additional failures due to use of C++03 where the dependencies require C++11, or of C++0x using GCC < 4.9 where e.g. C++11 `alignof`, required by the Boost.SmartPtr dependency, is not available. For (only) those I removed the C++03 and/or GCC 4.7/4.8 jobs (which cannot succeed). I think combining those changes into a single PR avoids a lot of additional manual work by me and allows to verify that the changes are sufficient to make CI pass again. Keeping e.g. C++03 builds, that cannot succeed due to dependencies, makes verifying the success of the Node 20 fix harder/less clear If anyone prefers to separate those changes into multiple PRs please mention that in the relevant PR for me to keep track of, or cherry-pick the individual changes. However it might be sufficient to check the status of the CI for the individual commits to compare the state (success/failure) for the jobs fixed or removed by the next commit. I realize maintaining the CI is a PITA and hope this helps. Regards, Alex
On Monday, December 9th, 2024 at 2:04 PM, Alexander Grund via Boost <boost@lists.boost.org> wrote:
Dear maintainers,
I made a bunch of PRs to fix the Github Actions CI for almost 40 repositories. Opening a PR in GitHub is manual work while I was able to apply similar changes over all those repositories using a script which commits to a single branch (and hence PR) per repo. Hence the PRs contain a few related changes that in combination allow the CI to pass again.
To avoid repeating this in each PR I wanted to shortly explain the changes here:
1. macos-12 runners are dropped by github and jobs won't start anymore. at least macos-13 needs to be used
2. Ubuntu 23.* doesn't work anymore as e.g. some APT repositories don't exist anymore. Ubuntu 24.04 can be used instead 3. clang 17 (and partly 16) don't work in C++23/2b mode with the stdlibc++ (12) installed on ubuntu 24.04, clang 18+ and/or the other C++ standards can still be used 4. Support for Node 16 was recently fully dropped by Github Actions so the @v3 actions must be upgraded to @v4, which require Node 20. The workaround using $ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION has no effect anymore 5. The node 20 version available by default conflicts with the glibc version in the Ubuntu 16/18 container images used for testing older compilers.
I addressed all of them with last one being the major one: Installing a compatible version inside a custom image to be used instead doesn't work because GHA mounts the node 20 installation from the host os.
The solution I implemented downloads an unofficial node 20 built against the correct/older glibc and creates 2 mount points such that we can put that node onto the host runner and mount that instead of what github gives us using the idea from https://github.com/dixyes/ghactionsplay/blob/bf889ee2fed8dcd72cc8e32bfd4789b... To avoid overloading the unofficial-builds.nodejs.org server we use a copy hosted at boost.io
Download and extraction requires curl and xz-utils to be installed which I added where required.
Additionally I noticed additional failures due to use of C++03 where the dependencies require C++11, or of C++0x using GCC < 4.9 where e.g. C++11 `alignof`, required by the Boost.SmartPtr dependency, is not available. For (only) those I removed the C++03 and/or GCC 4.7/4.8 jobs (which cannot succeed).
I think combining those changes into a single PR avoids a lot of additional manual work by me and allows to verify that the changes are sufficient to make CI pass again. Keeping e.g. C++03 builds, that cannot succeed due to dependencies, makes verifying the success of the Node 20 fix harder/less clear
If anyone prefers to separate those changes into multiple PRs please mention that in the relevant PR for me to keep track of, or cherry-pick the individual changes. However it might be sufficient to check the status of the CI for the individual commits to compare the state (success/failure) for the jobs fixed or removed by the next commit.
I realize maintaining the CI is a PITA and hope this helps.
Regards, Alex
Alex, Thank you for the explanation and hard work on this. It is certainly appreciated. Matt
On Mon, 9 Dec 2024, 20:04 Alexander Grund via Boost, <boost@lists.boost.org> wrote:
Dear maintainers,
I made a bunch of PRs to fix the Github Actions CI for almost 40 repositories. Opening a PR in GitHub is manual work while I was able to apply similar changes over all those repositories using a script which commits to a single branch (and hence PR) per repo. Hence the PRs contain a few related changes that in combination allow the CI to pass again.
To avoid repeating this in each PR I wanted to shortly explain the changes here:
1. macos-12 runners are dropped by github and jobs won't start anymore. at least macos-13 needs to be used
2. Ubuntu 23.* doesn't work anymore as e.g. some APT repositories don't exist anymore. Ubuntu 24.04 can be used instead 3. clang 17 (and partly 16) don't work in C++23/2b mode with the stdlibc++ (12) installed on ubuntu 24.04, clang 18+ and/or the other C++ standards can still be used 4. Support for Node 16 was recently fully dropped by Github Actions so the @v3 actions must be upgraded to @v4, which require Node 20. The workaround using $ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION has no effect anymore 5. The node 20 version available by default conflicts with the glibc version in the Ubuntu 16/18 container images used for testing older compilers.
I addressed all of them with last one being the major one: Installing a compatible version inside a custom image to be used instead doesn't work because GHA mounts the node 20 installation from the host os.
Does using ubuntu-latest as host OS and ubuntu 16.04 as container not work?
Am 09.12.24 um 20:29 schrieb Ruben Perez:
Does using ubuntu-latest as host OS and ubuntu 16.04 as container not work? Unfortunately not. Seems I forgot to mention this clarification:
The action will run inside the container with the Node binary mounted from the host OS. That Node (version 20) is built against a newer glibc than available inside the Ubuntu 16/18 container. So using the Node from an even newer OS makes the problem worse as it might have used/require an even newer glibc. Besides dropping those containers or the GitHub actions (used for checkouts with setting useful config params; coverage uploads and maybe more) the only solution is to use a Node 20 specifically built against a glibc supported by those containers. This is also done conditionally by using the mounts only when an ubuntu:1* container is detected and the download script also checks that condition which should avoid any side effects for recent-enough OS/containers.
On Tue, 10 Dec 2024, 10:41 Alexander Grund, <alexander.grund@tu-dresden.de> wrote:
Am 09.12.24 um 20:29 schrieb Ruben Perez:
Does using ubuntu-latest as host OS and ubuntu 16.04 as container not work? Unfortunately not. Seems I forgot to mention this clarification:
The action will run inside the container with the Node binary mounted from the host OS. That Node (version 20) is built against a newer glibc than available inside the Ubuntu 16/18 container.
So using the Node from an even newer OS makes the problem worse as it might have used/require an even newer glibc.
I see. Thanks for your detailed explanation.
On Mon, 9 Dec 2024 at 20:04, Alexander Grund via Boost <boost@lists.boost.org> wrote:
Dear maintainers,
I made a bunch of PRs to fix the Github Actions CI for almost 40 repositories. [...] I realize maintaining the CI is a PITA and hope this helps.
Alex, on behalf of GIL maintainers, thank you so very much for helping us with this! Best regards, -- Mateusz Loskot, http://mateusz.loskot.net
You have saved me and many others hours of work. Thank you. On Mon, Dec 9, 2024 at 11:04 AM Alexander Grund via Boost < boost@lists.boost.org> wrote:
Dear maintainers,
I made a bunch of PRs to fix the Github Actions CI for almost 40 repositories. Opening a PR in GitHub is manual work while I was able to apply similar changes over all those repositories using a script which commits to a single branch (and hence PR) per repo. Hence the PRs contain a few related changes that in combination allow the CI to pass again.
To avoid repeating this in each PR I wanted to shortly explain the changes here:
1. macos-12 runners are dropped by github and jobs won't start anymore. at least macos-13 needs to be used
2. Ubuntu 23.* doesn't work anymore as e.g. some APT repositories don't exist anymore. Ubuntu 24.04 can be used instead 3. clang 17 (and partly 16) don't work in C++23/2b mode with the stdlibc++ (12) installed on ubuntu 24.04, clang 18+ and/or the other C++ standards can still be used 4. Support for Node 16 was recently fully dropped by Github Actions so the @v3 actions must be upgraded to @v4, which require Node 20. The workaround using $ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION has no effect anymore 5. The node 20 version available by default conflicts with the glibc version in the Ubuntu 16/18 container images used for testing older compilers.
I addressed all of them with last one being the major one: Installing a compatible version inside a custom image to be used instead doesn't work because GHA mounts the node 20 installation from the host os.
The solution I implemented downloads an unofficial node 20 built against the correct/older glibc and creates 2 mount points such that we can put that node onto the host runner and mount that instead of what github gives us using the idea from https://github.com/dixyes/ghactionsplay/blob/bf889ee2fed8dcd72cc8e32bfd4789b... To avoid overloading the unofficial-builds.nodejs.org server we use a copy hosted at boost.io
Download and extraction requires curl and xz-utils to be installed which I added where required.
Additionally I noticed additional failures due to use of C++03 where the dependencies require C++11, or of C++0x using GCC < 4.9 where e.g. C++11 `alignof`, required by the Boost.SmartPtr dependency, is not available. For (only) those I removed the C++03 and/or GCC 4.7/4.8 jobs (which cannot succeed).
I think combining those changes into a single PR avoids a lot of additional manual work by me and allows to verify that the changes are sufficient to make CI pass again. Keeping e.g. C++03 builds, that cannot succeed due to dependencies, makes verifying the success of the Node 20 fix harder/less clear
If anyone prefers to separate those changes into multiple PRs please mention that in the relevant PR for me to keep track of, or cherry-pick the individual changes. However it might be sufficient to check the status of the CI for the individual commits to compare the state (success/failure) for the jobs fixed or removed by the next commit.
I realize maintaining the CI is a PITA and hope this helps.
Regards, Alex
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
On Mon, Dec 9, 2024 at 1:04 PM Alexander Grund via Boost < boost@lists.boost.org> wrote:
I made a bunch of PRs to fix the Github Actions CI for almost 40 repositories.
Can you point to one, or more, of those PRs? So I can study them for applying in other repos (like B2 and Predef). -- -- René Ferdinand Rivera Morell -- Don't Assume Anything -- No Supongas Nada -- Robot Dreams - http://robot-dreams.net
Can you point to one, or more, of those PRs? So I can study them for applying in other repos (like B2 and Predef).
https://github.com/boostorg/wave/pull/235/files should be a good example The template in Boost.CI (https://github.com/boostorg/boost-ci/blob/master/.github/workflows/ci.yml) is also updated: https://github.com/boostorg/boost-ci/commit/d6ccde90dc7c5adac3fabc018f70dab0... https://github.com/boostorg/boost-ci/commit/cd30b7b67cc4fa366731932a616d5c90... That changeset is larger though as the conditional use of v3 or v4 actions was also removed. Maintainers are invited to copy that workflow file at any time. We try to keep it generic and up to date. Maybe we could make use of "Reusable workflows" [1] such that the steps can be in a single place so changes for multiple repos can be made much easier. Not sure how many maintainers would be willing to give up a bit of control for that. [1] https://docs.github.com/en/actions/sharing-automations/reusing-workflows#usi...
participants (6)
-
Alexander Grund -
Jeff Trull -
Mateusz Loskot -
Matt Borland -
René Ferdinand Rivera Morell -
Ruben Perez