Re: Have we become too Intel x64 centered?

From: Andrey Semashev (andrey.semashev_at_[hidden])
Date: 2021-01-23 19:32:28

On 1/23/21 10:07 PM, Robert Ramey via Boost wrote:
> On 1/23/21 7:38 AM, Andrey Semashev via Boost wrote:
>> On 1/23/21 3:47 PM, Tom Kent via Boost wrote:
>>>
>>> I've got a couple raspberry pi 4's that are running tests (slowly, takes
>>> 20+hrs to run the test suite...any earlier models just didn't have 
>>> enough
>>> ram). Look at the teeks99-05* (armv71/armhf) and teeks99-06* (aarch64).
>>
>> I appreciate your and all other testers efforts in running the test 
>> suite, but I must confess that I pay almost no attention to the 
>> official test matrix these days because:
> 
> The boost test matrix is the most complete and reliable display of the 
> current state of the the boost packages.

Sure, if you want to see the status of the whole Boost. I, as a library 
maintainer, am more interested in my specific library status, after 
having pushed a commit. This information is not readily provided by the 
test matrix.

> I do run the more moder CI 
> stuff, but it's often failing for some issue or another totally 
> unrelated to my library.  It's the gold standard as far as I'm concerned.

In my experience, the official test matrix is not more reliable than CI, 
when it comes to random failures. More than once I've seen failures 
caused by configuration errors on tester's machine (e.g. compiler 
executable not found). There were also weird failures for no apparent 
reason, which turned out to be the result of updates on the tester's 
machine. CI images are more stable, and you also can install necessary 
dependencies for testing. There are some quirks, and the installation 
can fail from time to time, but in general I would say the CI errors are 
more actionable.

>> - Problematic debugging. Often the report shows a failure, but the 
>> error log is not accessible. This seems to be a long standing problem. 
>> This makes the whole testing process pointless, as I cannot do 
>> anything about the failures.
> 
> I have difficulties sifting through the test output on all platforms.

The problem is not too much output (that wouldn't be a problem at all). 
The problem is that you often get no output at all.

> (I've been roundly ridiculed for this complaint.  But it means nothing 
> to me - I wear their ridicule as a badge of honor.)Â  I have my own 
> solution which I run on my own machine - library_status which presents a 
> table which is actually more useful to me than the official boost one 
> not to mention the Appveyor one.  Now If I could get library_status to 
> run as part of the CI solutions ...

A short status table might be nice, but that is not my complaint. I can 
do without such a table just fine. I can't do without the build and test 
output in case of failure.

>> I wish the current testing infrastructure was replaced with something 
>> more modern, CI-style, as I don't believe the above issues will be 
>> fixed any time soon.
> 
> I've made a worthy proposal for that (to be used in addition to the 
> current boost test matrix).  Again, got a lot of ridicule on that one too.

I haven't seen this, sorry.