Re: [boost] [testing][regressions] All my regressions testers failing due to connection errors to GitHub

Subject: Re: [boost] [testing][regressions] All my regressions testers failing due to connection errors to GitHub
From: Tom Kent (lists_at_[hidden])
Date: 2018-05-26 17:08:28

On Sat, May 26, 2018 at 10:49 AM, Ion Gaztañaga via Boost <
boost_at_[hidden]> wrote:

> Hi,
>
> All my 4 testers (I have 4 machines running several Mingw targets) are
> always failing pulling new versions from GitHub. This problem started some
> days ago. I've tried to reset all machines, network equipment, disabling
> antivirus, firewall, modified http attributes, modified the regression
> script to use git protocol instead of https, etc. with no luck.
>
> I'm using Windows machines with official Git. Any clue on what's going on?
> Maybe Github recently does not like so many submodules?
>
> Example of failure (several libraries can fail during "git pull
> "--recurse-submodules""):
>
> -------------------
> -------------------
> -------------------
>
> Fetching submodule libs/io
> fatal: unable to access 'https://github.com/boostorg/io.git/': OpenSSL
> SSL_read: SSL_ERROR_SYSCALL, errno 10054
>
> Then:
>
> # git_source_checkout failed with message "GIT command "git pull
> "--recurse-submodules"" failed with code 1"
> # Retrying (5 more attempts).
> # Executing GIT command: c:\boost\develop\boost_root> git submodule
> foreach "git submodule deinit ."
>
> -------------------
> -------------------
>
> All retries fail and the script fails with error:
>
> -------------------
> -------------------
> -------------------
>
> # git_source_checkout failed with message "GIT command "git pull
> "--recurse-submodules"" failed with code 1"
> # Giving up.
> Traceback (most recent call last):
>    File "run.py", line 71, in <module>
>      runner(root)
>    File "c:\boost\develop\boost_regression_src\regression.py", line 286,
> in __init__
>      self.main()
>    File "c:\boost\develop\boost_regression_src\regression.py", line 623,
> in main
>      getattr(self,action_m)()
>    File "c:\boost\develop\boost_regression_src\regression.py", line 578,
> in command_regression
>      self.command_get_source()
>    File "c:\boost\develop\boost_regression_src\regression.py", line 330,
> in command_get_source
>      self.retry(self.git_source_checkout)
>    File "c:\boost\develop\boost_regression_src\regression.py", line 662,
> in retry
>      return f()
>    File "c:\boost\develop\boost_regression_src\regression.py", line 875,
> in git_source_checkout
>      self.git_checkout(git_info['boost'], self.git_branch(), clean)
>    File "c:\boost\develop\boost_regression_src\regression.py", line 849,
> in git_checkout
>      self.git_command( 'pull', '--recurse-submodules' )
>    File "c:\boost\develop\boost_regression_src\regression.py", line 834,
> in git_command
>      raise Exception( 'GIT command "%s" failed with code %d' % (git_cli,
> rc) )
> Exception: GIT command "git pull "--recurse-submodules"" failed with code 1
> END   24/05/2018 15:28:37,46
>
> -------------------
> -------------------
> -------------------
>
> Errors are never in the same libraries, very few times the checkout
> succeeds and a regression is run. Anyone else having these problems?
>
> I've attached the log of one failure in case someone can see any clue. I
> really don't see how to solve this, so I won't be able to run regressions.
>
> Best,
>
> Ion
>
I've been having horrible problems getting all the submodules from github,
that look similar to what you've got going. I'm not sure where the issue
is, but I see it across linux and windows boxes at my house, but not when I
run in a cloud data center. I'm thinking it has something to do with my
connection (AT&T), but I'm not sure.

I ended up mirroring all the repos to a local machine, with a cron job to
update them every 15 min) and then cloning from that for the regression
runs. The script I use for that can be found here:
https://github.com/teeks99/boost-build/blob/master/Regression/mirror_boostorg.py

For more backgroun, I was seeing the same failed connections over https or
ssh. I got the script failure when even one of the submoduels failed to
update...which I guess is necessary. Now with my script almost all runs one
of the repos fails to fetch, but since they are all just standalone repos,
that failures doesn't cause an overall failure.

Tom