Re: [boost] testing model

Subject: Re: [boost] testing model
From: Julian Gonggrijp (j.gonggrijp_at_[hidden])
Date: 2012-02-13 11:19:07

Thomas Heller wrote:

> On 02/12/2012 12:34 PM, Julian Gonggrijp wrote:
> 
>> [...] My guess is that,
>> given a branching model like gitflow, you envision something along the
>> following lines:

>> There is a "testing image" which the testing volunteers run their
>> periodic tests on. By default, the image mirrors the latest commit on
>> the develop branch. When working on a feature branch, library authors
>> can reversibly and transparently substitute the version of their
>> library from the feature branch for the default version in the image.
>> All of this is facilitated by some kind of automated tool which has
>> yet to be found.
>> 
>> The result is that most libraries in the image are from develop, but
>> some are from feature branches, without the testing volunteers needing
>> to worry about it. They can run a single testing cycle on the
>> heterogeneous image. For library authors, this brings the advantage
>> that they can already have their changes tested on all platforms
>> before they feel confident that it should be possible to make them
>> work. (Or while they emply a long-term feature branch specifically in
>> order to support a platform, per Daniel James.)

> *nod* looks like you absorbed it and summarized my ideas, thanks ;)

That's good to know. I think I figured out a way to do this. It's a 
more elegant variant of the outline provided by Daniel in [1], quoted 
here.

Daniel James wrote:

> Or maybe, rather than being a traditional branch, the test repo could
> be created by a script which merges in active feature branches. It'd
> have to deal with conflicts though, maybe just keeping track of
> commits that are known to be good and sticking with them when a
> conflict is found. But something like that could be developed later -
> especially after having some experience of how the process works.
> 
> This testing mechanism would be fairly separate from the main flow
> structure so it could be provided by an additional tool to gitflow.
> Hopefully wouldn't be too hard to develop. Does that make any sense? I
> know it isn't ideal, but it seems plausible given our current
> infrastructure. And we could work on improving it as we go.

I think it would be both wise and feasible to require (1) that authors 
edit a given library either on the develop branch or on any number of 
feature branches, but not on both in parallel, and (2) that a public 
feature branch always affects only one library. Apart from avoiding 
confusion, this offers a strong guarantee that a feature branch can 
always be merged back into develop without conflicts.

Given such a convention, I think the test imaging tool is almost 
trivial to implement. The core implementation takes only two 
additional small files in the /status directory: a list of feature 
branches that authors want to be substituted for the default in the 
develop branch, and a small script that will be run by the volunteers. 
Assuming that the list is a newline-separated text file like this 
(just an example, take it with some grains of salt),

substitute.txt (must be edited on the develop branch only):
> newlibs/application
> math/octonion/hardwareaccel
> data/heap

the core of the script would be something like this,

testcomposite.sh:
> # must be run from $BOOST_ROOT/status
> git checkout develop
> git pull
> awk 'begin { print "git merge --no-commit --no-rerere-autoupdate" } ; { printf "remote/%s" , $0 }' substitute.txt >mergecommand.sh
> sh mergecommand.sh
> rm mergecommand.sh
> bjam
> # an additional command to report the results?
> bjam --clean
> git checkout develop

Note that the last line undoes the custom merge, so the test image is 
only a temporary state of the volunteer's working directory.

The script I outlined and the existing testing tools will need some 
adjustments to make the summary correctly display which commits have 
been tested. I believe this shouldn't be an obstacle.
It should be easy to port to most platforms, as it's a plain command 
list and the only external tools used are portable themselves, i.e. 
git, awk and bjam.

I suspect it would be possible to realize this with a svn-based 
implementation of the branching model as well. That would however have 
some drawbacks with regard to the merge operation, in terms of 
robustness and speed:

- substitute.txt and the awk command would be more complex, since svn 
  merge requires authors to not only list the name of the branch but 
  also the path to the library and the revision number;
- it would require multiple calls to svn merge, because svn can merge 
  only two branches at a time;
- it would involve more server interaction because the volunteer has 
  no local copy of the history.

Putting the VCS considerations aside, it seems like we have an 
exciting opportunity here. I'm looking forward to all feedback.

-Julian

1. http://lists.boost.org/Archives/boost/2012/02/190215.php