End of regression runners?
(Try 3 at getting accepted by the mailing list) All- I just noticed that there was an issue causing regression runs to fail on both develop and master: the results end with this: boost-test(RUN) "math/test_autodiff_8" : "libs/math/test/test_autodiff_ 8.cpp" boost-test(COMPILE) "math/diff_autodiff_incl_test" : "libs/math/test/compile_test/diff_autodiff_incl_test.cpp" boost-test(COMPILE) "math/diff_finite_difference_incl_test" : "libs/math/test/compile_test/diff_finite_difference_incl_t est.cpp" boost-test(COMPILE) "math/diff_lanczos_smoothing_incl_test" : "libs/math/test/compile_test/diff_lanczos_smoothing_incl_t est.cpp" error: Unable to find file or target named error: 'my_autodiff_reverse_test.cpp' error: referred to from project at error: '../libs/math/test' In trying to figure out how long this had been happening, I went to the website(s) to see when the last good runs were, but pulling up those, it appears that the job that parses the results into the website hasn't been running since mid-june! https://regression.boost.io/master/developer/summary.html https://regression.boost.io/develop/developer/summary.html I had been happily running these runners for over a decade, even though they don't match the immediate feedback that a library author gets from a github actions job, running across *a lot* of different compilers seems useful, but apparently no one cares? Frankly (and obviously), I haven't spent as much time on care and feeding of the runners as I have in the past. It mostly comes down to rebooting the machines every couple months and updating to new gcc/clang/vs versions when they appear. **I'm looking for feedback from the community:** Let it go, not worth the electricity spent on running these (maybe move the infrastructure to github runners?) -or- Hey this could be useful, let's spend a couple cycles getting it back working again. Thanks, Tom
El 12/09/2025 a las 12:55, Tom Kent via Boost escribió:
[...]
**I'm looking for feedback from the community:**
Let it go, not worth the electricity spent on running these (maybe move the infrastructure to github runners?)
-or-
Hey this could be useful, let's spend a couple cycles getting it back working again.
I would like to thank you for your ongoing support through the years and vote in favor of having this resource up and running again. (I took a look at it a couple of days ago and noticed the problem but didn't get around to notifying about it, my apologies.) Joaquín M López Muñoz
On 9/12/2025 1:00 PM, Joaquin M López Muñoz via Boost wrote:
[...] I would like to thank you for your ongoing support through the years and vote in favor of having this resource up and running again.
I fully echo Joaquín’s appreciation and support for reinstating this resource. Regarding DynamicBitset, it likely won’t compile under MSVC 10, so you may need to disable or remove the corresponding tests for that compiler. -- Gennaro Prota <https://prota.dev>
Tom Kent wrote:
**I'm looking for feedback from the community:**
Let it go, not worth the electricity spent on running these (maybe move the infrastructure to github runners?)
-or-
Hey this could be useful, let's spend a couple cycles getting it back working again.
Thank you Tom for keeping the lights on all these years. My personal opinion is that it's probably time to let it go. I never look at the matrix anymore; CI infrastructure has entirely superseded it. Our test matrix was, once upon a time, miles ahead of the rest of the open source community, but the world has moved on since then. :-)
the results end with this: boost-test(RUN) "math/test_autodiff_8" : "libs/math/test/test_autodiff_ 8.cpp" boost-test(COMPILE) "math/diff_autodiff_incl_test" : "libs/math/test/compile_test/diff_autodiff_incl_test.cpp" boost-test(COMPILE) "math/diff_finite_difference_incl_test" : "libs/math/test/compile_test/diff_finite_difference_incl_t est.cpp" boost-test(COMPILE) "math/diff_lanczos_smoothing_incl_test" : "libs/math/test/compile_test/diff_lanczos_smoothing_incl_t est.cpp" error: Unable to find file or target named error: 'my_autodiff_reverse_test.cpp' error: referred to from project at error: '../libs/math/test'
Reverse mode autodiff is a brand new section of the Math library for this cycle, and we have had failures on develop for ~2 days now. Some code we thought was being tested actually wasn't, and then once it actually was we had some breakage. The author is actively working on the repair. Sorry for the noise. Matt
On 12 Sep 2025 13:55, Tom Kent via Boost wrote:
**I'm looking for feedback from the community:**
Let it go, not worth the electricity spent on running these (maybe move the infrastructure to github runners?)
-or-
Hey this could be useful, let's spend a couple cycles getting it back working again.
I'm thankful for all the efforts on regression testing, especially during the pre-GitHub era, but personally I haven't looked at the test matrix for years. (Well, I have looked recently, when the URL in the readme were updated, but not other than that.) My main complaints are the same as always have been, and are (a) lack of feedback notifications to the library authors, (b) problematic diagnostics (I remember times when build/test logs were unavailable or the link pointed to an unrelated log; not sure if this got fixed) and (c) problems and misconfiguration on the test runners often go unnoticed. So, in its current state, I don't find the test matrix very useful. However, I do think that Boost could benefit from more diverse test runners than what GitHub Actions offer (more hardware architectures, more OSes, more compilers). I'm also concerned about depending solely on GitHub Actions for testing, since this is a single point of failure that historically does tend to fail from time to time and is difficult to replace. I would prefer if the current test matrix evolved into something more useful, an alternative to GHA, but I realize this is probably a lot of work. If not that, I think the resources could be better spent on improving GHA quality, e.g. as new GHA runners. BTW, on the topic of improving GHA quality, there is this collection of GHA plugins that run the CI on various OSes in VMs: https://github.com/vmactions/ It worked well enough for me in Boost.Atomic, so maybe it will be useful to others. The problem with this is that VM setup is being performed on every job run, and it takes some time and may occasionally fail. A native runner would have been faster and more reliable.
However, I do think that Boost could benefit from more diverse test runners than what GitHub Actions offer (more hardware architectures, more OSes, more compilers). I'm also concerned about depending solely on GitHub Actions for testing, since this is a single point of failure that historically does tend to fail from time to time and is difficult to replace.
Drone in the boostorg has native s390x and aarch64. I would prefer if the current test matrix evolved into
something more useful, an alternative to GHA, but I realize this is probably a lot of work. If not that, I think the resources could be better spent on improving GHA quality, e.g. as new GHA runners.
BTW, on the topic of improving GHA quality, there is this collection of GHA plugins that run the CI on various OSes in VMs:
There's also: https://github.com/jeking3/bdde which plugs right into boost.ci. I use it for testing ppc64le since the cost of testing on a native machine was astronomical. Matt
El 12/09/2025 a las 13:37, Andrey Semashev via Boost escribió:
BTW, on the topic of improving GHA quality, there is this collection of GHA plugins that run the CI on various OSes in VMs:
It worked well enough for me in Boost.Atomic, so maybe it will be useful to others. The problem with this is that VM setup is being performed on every job run, and it takes some time and may occasionally fail. A native runner would have been faster and more reliable.
Thanks for pointing this Andrey, testing BSD locally was quite a pain for libraries like Interprocess. I should take inspiration from Atomic's test workflows... Best, Ion
Am 12.09.25 um 12:55 schrieb Tom Kent via Boost:
**I'm looking for feedback from the community:**
Let it go, not worth the electricity spent on running these (maybe move the infrastructure to github runners?)
-or-
Hey this could be useful, let's spend a couple cycles getting it back working again. The main benefit here is to have reports on the granularity of individual tests.
I.e. if anything fails one can see if it fails always just a single test or a specific test only one a specific platform or something major affecting multiple/all tests. However one usually wants that before merging to develop/master where GHA is better at. So I haven't looked at these tables for a while although that rare instances when I needed it, it was quite helpful/easier to work with than GHA. So maybe there is worth in either having it update only in longer intervals and/or integrate it with other CI, optionally with notifications / GitHub integration. I can imagine it open an issue on new, consistent failures. Anyway: Thanks for hosting and providing this info so far! Best, Alex
I had been happily running these runners for over a decade, even though they don't match the immediate feedback that a library author gets from a github actions job, running across *a lot* of different compilers seems useful, but apparently no one cares?
First of all, many thanks for maintaining these tests for all these years. One thing that github runners are missing is different MSVC versions. I test msvc-10.0, msvc-11.0, msvc-12.0 locally. I remember reviewing the regression matrixes to test past year Boost releases. But I no longer follow those regression tests due to the long testing periods. I usually combine local and github tests. Best, Ion
The test matrix was triggered by a job on wowbagger, every 30 minutes. 30 * * * * $HOME/www.boost.org/testing/trigger_test_reports_new_server.sh This was enabled. Going further, the script checks if there are new results to process. Bizarrely, that test is always "successful" (the sha matches) which means there is nothing to process: [root@wowbagger testing]# cat results.lst | shasum -c results.sum [root@wowbagger testing]# echo $? 0 [root@wowbagger testing]# cat test.txt | shasum -c test.txt [root@wowbagger testing]# echo $? 0 The server is an obsolete CentOS release 6.10 (Final) which hasn't been rebooted in years. Something very peculiar is glitching in the "shasum -c" command. I have commented out the cron, and moved it to the newer regression server. CircleCI ran: https://app.circleci.com/pipelines/github/boostorg/regression The job results in this error: error: Unable to find file or target named error: '/boost/conversion//boost_conversion' error: referred to from project at error: '/root/project/boost-reports/boost_root/libs/numeric/conversion'
On 12 Sep 2025 15:12, Ion Gaztañaga via Boost wrote:
One thing that github runners are missing is different MSVC versions. I test msvc-10.0, msvc-11.0, msvc-12.0 locally.
You can use AppVeyor for older MSVC versions. Although many Boost libraries no longer support those compilers anyway.
On Fri, Sep 12, 2025 at 3:57 AM Tom Kent via Boost <boost@lists.boost.org> wrote:
**I'm looking for feedback from the community:**
The runners were always a mixed bag for me for the reasons that we all know about discoverability, consistency of reporting, and so on. However... I find the idea of a distributed system for running tests appealing. That is, that someone offers a computing resource and all they need to do is run a self-contained executable or container on their system, and then work is performed and the results reported back to a central server which collates the results and presents them for display. This is a very cool concept and it would be great if there was some way to leverage it into a system that delivers a lot more value for the project. Unfortunately I have no idea how to do that >.< Thanks
participants (10)
-
Alexander Grund -
Andrey Semashev -
Gennaro Prota -
Ion Gaztañaga -
Joaquin M López Muñoz -
Matt Borland -
Peter Dimov -
Sam Darwin -
Tom Kent -
Vinnie Falco