Re: Boost CI Infrastructure

From: Boris Kolpackov (boris_at_[hidden])
Date: 2023-05-03 12:58:25

Vinnie Falco via Boost <boost_at_[hidden]> writes:

> I guess what I'm trying to say here is that we care deeply about long
> turnaround times on CI ("they suck) have been working on the problem.

This email may give an impression that the CI suckage is an inherent
problem that one just has to live with so I want to provide a different
perspective.

As part of the build2 project we have implemented our own CI 
infrastructure from the grounds up: from managing our own hardware,
to VM-based CI job handling, to custom web interface for result
viewing. This was based on concluding that all the existing off-
the-shelf solutions are primarily focused on "web development"
and will result in the subpar experience for large C/C++ projects
(Boost, Qt, etc). Some key differences of our approach compared 
to the commonly used setups (e.g., GitHub Actions):

1. "Push CI" explicitly triggered during development as opposed to
   "pull CI" triggered on commit/PR/etc. This allows specifying
   the target configurations to build in, alternative project
   configurations, etc.[1]

2. Standard "CI script"[2] that is used by all the C/C++ projects
   instead of each projects doing their own thing.

   This "CI script" is written in C++ and has evolved to cover
   a fairly elaborate set of scenarios, including:

   1. Testing of preparation of source distributions (to make sure
      there are no missing files, etc).

   2. Testing of libraries as installed (to make sure no public
      headers were omitted during installation, etc).

   3. Ability to test projects with build-time dependencies (e.g.,
      source code generators, build system modules, etc) in cross-
      compiled environments.
  
3. Interactive CI support.

   The push CI can request an interactive session where the above
   "CI script" is stopped at one of the pre-defined breakpoints
   allowing the user to login into the machine via VNC or SSH and
   investigate the problem.

4. Large number of build configurations covering all the major
   platforms/compilers/versions[3] with a well-defined obsolescence
   policy.

To give a sense of how this works in practice, here is my typical
workflow:

1. Hack on some feature in a branch, build and test locally.

2. When need to test on other platforms, push the branch to remote
   and submit the CI request. Normally it's just:

   $ bdep ci

   The result is a URL with the interface like this:

   https://ci.stage.build2.org/@04e5ffb7-cf96-42f9-bc91-94d3ffe48de1

   One nice touch is that it detects compiler warnings, as can be
   seen in this result:

   https://ci.stage.build2.org/@23da340d-cb4d-4105-8474-bb870e1a1380

   (Drop the UUID from the above URL to see some other CI jobs.)

3. Say everything works fine except on Windows with an old MSVC version.

   If the fix is clear, I push it and re-CI but just on this specific
   configuration (this is both faster for me and saves resources for
   others):

   $ bdep ci --build-config windows_10-msvc_14.3

   If the fix is not clear and I need to play with it or run the
   debugger, I can request an interactive CI session:

   $ bdep ci --interactive windows_10-msvc_14.3/error

   That `/error` part is a breakpoint, meaning that the CI job
   will execute the standard "CI script" until the first error
   (compile error, test failure, etc) and pause.

4. Once all is good I clean up my feature branch (squash commits,
   etc), rebase on master, CI one more time for good measure, and
   merge to master.

   When I push my changes to master, there can be a hook that submits
   a "verification" CI job.

[1] https://build2.org/bdep/doc/bdep-ci.xhtml
[2] https://build2.org/bbot/doc/build2-build-bot-manual.xhtml#arch-worker
[3] https://ci.cppget.org/?build-configs