Re: Do we still want develop branch?

From: Andrey Semashev (andrey.semashev_at_[hidden])
Date: 2023-11-06 13:13:27

On 11/6/23 03:09, Robert Ramey via Boost wrote:
> On 11/5/23 1:10 PM, Andrey Semashev via Boost wrote:
>>
>> With the adoption of git and CI, the relevance of the test matrix has
>> diminished, 
> 
> Personally, I would dispute that.  I think that it's the only part of
> our testing infrastructure that still works.
> 
> and, presumably, people have shifted towards monitoring
>> their libraries' CI. 
> 
> The CI for the boost serialization does not and never has worked.  This
> is easy to veryify just by looking at any of the CI output for this
> library.  And this demonstrable fact has been pointed out numerous
> times. This situation is never addressed.
> 
> I doubt that the serialization library is the only one with this issue.
> 
> The sadist part of all this is that even if it did "work" it would still
> be useless.  It's not uncommon for a large and/or complex library to
> fail one or more tests on one or more compilers due to issues in the
> compiler itself.  These can't be "fixed" in the library.

While it is true that tests sometimes fail because of the environment
issues, this problem is not specific to CI; it also happens with our
volunteer testers. But, with CI, it may be easier to work around - for
example, by using a standard Docker image or installing missing
packages, or configuring the environment in the CI scripts. With the
test matrix, your only option is to contact the tester and try and solve
the problem with him. Which may not be easy to do, since not all testers
share their contacts. I remember this being a major problem back when I
was relying on the test matrix.

Now, if you mean compiler bugs and such that affect your library, then
the failure is legitimate, as it will likely affect your users that also
use that compiler. The solution here would be to implement workarounds
in the library or, if not feasible, declare that compiler unsupported
(preferably, in the library docs) and remove it from testing. Note that
this course of action would be the same regardless whether you're using
CI or the test matrix. Of course, there may be middle cases, when the
compiler bug affects only a minor feature of the library, but the point
remains - you either work around it in the feature implementation, or
declare that the feature is not supported on that compiler and remove it
from testing.

> The test
> matrix shows all the tests x all the environments.  One can easily see
> if any failure is general or isolated to a particular environment.  The
> current CI just registers pass/fail for the whole library and all the
> environments.

The only benefit that the test matrix provides is the breakdown by
individual tests. With CI, you typically have to search though the logs
to know which test has failed. You do get the breakdown by jobs (which
are equivalent to environments). It may not be presented the same way as
in the matrix, and you may prefer one or the other presentation, but the
information is there.

> Some times someone will suggest skipping a particular
> test for a particular library so the CI comes of clean.  This is
> basically hiding the error.  Users considering using a library in their
> own enviroment are basically mislead that their the library works
> everywhere - which is demonstrably untrue.  It's a fraud.

As I said above, it's not about hiding a failure and trying to deceive
the user. It's about supporting or not supporting a configuration. A CI
is supposed to test the configurations that you intend to support and
highlight problems with those configurations, should they appear. The
configurations you don't support need not be present in the CI, as you
don't care whether they are failing or not.

Furthermore, as a user, CI (or the test matrix) is not going to be the
first place where I look to see whether the library supports my
configuration. My first choice would probably be the documentation, and
if it lacks the information, I will probably try the library myself. The
reason why CI or the test matrix is not useful for this is because, as a
user, I have no idea what the test results show. If a test is failing, I
have no idea what it is testing, and why it is failing. Is the core
functionality broken? Is it some corner case that I will never hit? Is
it some intermittent failure e.g. due to network issues? Even
discovering the testing configuration (compiler version, installed
libraries, build options, etc.) may not be a trivial task for an outside
user - even less trivial if the test results are in a less commonly used
CI service.

Bottom line is, CI (or the test matrix) is a tool primarily for the
library maintainer, who can interpret its results and act accordingly.
For me, as a maintainer, CI does a better job than the test matrix, even
despite the shortcomings it has. And I'm not denying that there are
shortcomings.

However, CI vs. test matrix is a bit off-topic in this discussion.

>> Which means, people are less likely to notice that
>> something broke downstream, unless the downstream developer notices and
>> reports the problem. Although I have been that downstream developer
>> myself quite a few times, I have to admit that this role of the develop
>> branch of "an integration testing field" is not well filled in the
>> current workflow. Noticing the breakage is mostly luck (i.e. a matter of
>> me making a commit to my library and noticing the CI failure caused by
>> an upstream dependency) rather than a rule.
> 
> A useful observation.  The problem is that the current CI tests the
> development branch of one's library agains the develop branch of all the
> other libraries.  So now my library looks like it "works" but when (and
> only when) it could fail when run against all master branches.  So the
> testing shows pass when it actually fails when shipped with the master.
> Note that as we speak the test matrix for the master branch isn't
> working so it seems we never test the software which is actually being
> shipped.

Technically, you can configure CI to test however you like, including
testing your feature branches and PRs against master. You can even test
both against develop and master, if you like, although it will double
the testing time. You write the CI scripts, including checkout commands,
so you decide.

However, as long as our workflow includes the develop branch, it doesn't
make sense to test against master, as you will be merging your changes
into develop, not master. If you want to test whether your changes will
not break master, you could create a PR merging develop to master and
run the CI on that, but that is separate from feature branches and other
PRs that target develop.

> Personally, on my own machine, I
> test my development or feature branch agains the master branch of all
> the other libraries.  It's the only way to know that when the changes
> are merged into the master the software will still work.

But then you merge the feature branch to develop, right? In that case,
you could be breaking downstream libraries in develop, or your own
library in develop because some dependency of yours has changed in develop.