Re: Have we become too Intel x64 centered?

From: Tom Kent (lists_at_[hidden])
Date: 2021-01-23 20:30:17

On Sat, Jan 23, 2021 at 1:32 PM Andrey Semashev via Boost <
boost_at_[hidden]> wrote:

> On 1/23/21 10:07 PM, Robert Ramey via Boost wrote:
> > On 1/23/21 7:38 AM, Andrey Semashev via Boost wrote:
> >> On 1/23/21 3:47 PM, Tom Kent via Boost wrote:
> >>>
> >>> I've got a couple raspberry pi 4's that are running tests (slowly,
> takes
> >>> 20+hrs to run the test suite...any earlier models just didn't have
> >>> enough
> >>> ram). Look at the teeks99-05* (armv71/armhf) and teeks99-06* (aarch64).
> >>
> >> I appreciate your and all other testers efforts in running the test
> >> suite, but I must confess that I pay almost no attention to the
> >> official test matrix these days because:
> >
> > The boost test matrix is the most complete and reliable display of the
> > current state of the the boost packages.
>
> Sure, if you want to see the status of the whole Boost. I, as a library
> maintainer, am more interested in my specific library status, after
> having pushed a commit. This information is not readily provided by the
> test matrix.
>
> > I do run the more moder CI
> > stuff, but it's often failing for some issue or another totally
> > unrelated to my library.  It's the gold standard as far as I'm concerned.
>
> In my experience, the official test matrix is not more reliable than CI,
> when it comes to random failures. More than once I've seen failures
> caused by configuration errors on tester's machine (e.g. compiler
> executable not found). There were also weird failures for no apparent
> reason, which turned out to be the result of updates on the tester's
> machine. CI images are more stable, and you also can install necessary
> dependencies for testing. There are some quirks, and the installation
> can fail from time to time, but in general I would say the CI errors are
> more actionable.
>

I agree that the current setup is far from ideal, and there are lots of off
the
shelf CI setups that library authors *should* be depending on for per-commit
testing.

However there are two aspects that are often missed in per-library CI
testing:

1.  Integration with the rest of boost. If a change is made in one of the
libraries
     up near the top of the dependency graph, how does the library author
know
     if it breaks some feature in something that depends on it? For most
     changes, especially ones that don't affect the libraries API, I'd hope
this isn't
     common. However since boost is an integrated product, as a community it
     is something we should think about.
2.  Most CI setups I've seen run a very limited number of compilers. Boost's
     matrix has dozens of different compiler versions, several different
sets of
     compiler options, and multiple architectures (this used to be more
vibrant).

I think there is a need for both types of testing in Boost.

>
> >> - Problematic debugging. Often the report shows a failure, but the
> >> error log is not accessible. This seems to be a long standing problem.
> >> This makes the whole testing process pointless, as I cannot do
> >> anything about the failures.
> >
> > I have difficulties sifting through the test output on all platforms.
>
> The problem is not too much output (that wouldn't be a problem at all).
> The problem is that you often get no output at all.
>

There are lots of problems with the current regression test setup. Output
is one of them. Runners with bad configs is another one.

<aside>We also
have users with bad configs, depending on what config problem there is
boost library authors need to do a much better job and making config
problems apparent to users.
</aside>

>From my side, the actual regression test tools are barely-held-together-
with-duct-tape bad. They only work in python 2, lots of wasted time with
git overhead and log processing, janky ftp uploads.

> > (I've been roundly ridiculed for this complaint.  But it means nothing
> > to me - I wear their ridicule as a badge of honor.)  I have my own
> > solution which I run on my own machine - library_status which presents a
> > table which is actually more useful to me than the official boost one
> > not to mention the Appveyor one.  Now If I could get library_status to
> > run as part of the CI solutions ...
>
> A short status table might be nice, but that is not my complaint. I can
> do without such a table just fine. I can't do without the build and test
> output in case of failure.
>
> >> I wish the current testing infrastructure was replaced with something
> >> more modern, CI-style, as I don't believe the above issues will be
> >> fixed any time soon.
> >
> > I've made a worthy proposal for that (to be used in addition to the
> > current boost test matrix).  Again, got a lot of ridicule on that one
> too.
>
> I haven't seen this, sorry.
>

I'd love to see proposals for:

1.  Fixing the boost-wide regression test system.
2.  Getting CI best-practices for boost that can be easily pulled into
library's
     standalone testing system.

Tom