[boost] testing model (was: Re: [git] neglected aspects)

Subject: [boost] testing model (was: Re: [git] neglected aspects)
From: Julian Gonggrijp (j.gonggrijp_at_[hidden])
Date: 2012-02-12 06:34:49

Thomas Heller wrote:

> On 02/11/2012 03:48 PM, Julian Gonggrijp wrote:
> 
>> Thomas Heller wrote:
>> 
>>> What would make this whole branching very cool is if we could somehow tie it with testing such that we are able to tell the testrunners to use our feature branch to run the tests (of course this must be automated in a way). I think in this scenario, a modularized boost will come in very handy ... Think of the VHDL model,
>> 
>> Do you mean this? http://en.wikipedia.org/wiki/VHDL
>> 
>>> you have unit tests, but the module under test might be very different, though the unit tests stay the same. This is something i would really like to see.
>> 
>> Could you clarify? I don't see why we would want to test a different
>> module with the same unit tests.
> 
> Yes, I mean the hardware description language. Let me explain:
> In VHDL you usually start prototyping your new design by writing some behavioral description at a very high level, almost like you do with a "normal" programming language. After you are done, you start to write your unit tests for that. You do that by writing a very generic unit test and refer to your new design as "device under test". After you successfully passed all the tests, you start by getting down one level in the abstraction, lets say RTL level (Register transfer level). After that you plug that new device into your unit tests to ensure correct behavior.
> Now, imagine that different branches are seen as different devices, so to say. Instead of letting the test runners run the unit tests defined in the main branch, you let'em run the unit tests in your new feature branch, completely transparent for them. This means that you don't need to disrupt regular service by committing a half done feature to the main branch, but get the full advantage of the different test runners.

Thomas, I've given this some thought and I can't help but think that 
the VHDL comparison doesn't really work. That's mainly because Boost 
libraries or hypothetical Boost modules are not like subdevices that 
cooperate on a common task, while that does seem to be the case with 
the VHDL example.

The VHDL business is causing us to miss your point, which is a waste 
since your idea does seem to be very interesting. My guess is that, 
given a branching model like gitflow, you envision something along the 
following lines:

There is a "testing image" which the testing volunteers run their 
periodic tests on. By default, the image mirrors the latest commit on 
the develop branch. When working on a feature branch, library authors 
can reversibly and transparently substitute the version of their 
library from the feature branch for the default version in the image. 
All of this is facilitated by some kind of automated tool which has 
yet to be found.

The result is that most libraries in the image are from develop, but 
some are from feature branches, without the testing volunteers needing 
to worry about it. They can run a single testing cycle on the 
heterogeneous image. For library authors, this brings the advantage 
that they can already have their changes tested on all platforms 
before they feel confident that it should be possible to make them 
work. (Or while they emply a long-term feature branch specifically in 
order to support a platform, per Daniel James.)

Please let us know to what extend this resembles what you had in mind. 
I suggest that you omit VHDL-related examples for the time being.

(After reading Daniel's early post again -- i.e. the second link in 
the PS -- I realize Daniel already proposed something very similar. I 
might have messed up by attributing ideas to the wrong persons; my 
apologies.)

Daniel, I would like to invite you specifically to take part in this 
discussion/brainstorm session.

-Julian

PS: some context for those who need it, in chronological order.

http://nvie.com/posts/a-successful-git-branching-model/
Introduction of the gitflow branching model by the inventor.

http://lists.boost.org/Archives/boost/2012/02/190215.php
Post by Daniel James, in which he raised his concerns about testing in 
gitflow. The post contained an idea similar to the one presented here.

http://lists.boost.org/Archives/boost/2012/02/190269.php
Further clarification of Daniel's concern.

http://dl.dropbox.com/u/3512486/boost_branching_model.pdf
An illustration that compares the current Boost workflow to one in 
which gitflow is applied. It doesn't cover testing, however. I 
introduced it in the following post:
http://lists.boost.org/Archives/boost/2012/02/190312.php

http://lists.boost.org/Archives/boost/2012/02/190327.php
Thomas' concern about testing in a gitflow-like branching model.

http://lists.boost.org/Archives/boost/2012/02/190352.php
My reply to Daniel, arguing that gitflow wouldn't hurt the existing 
test cycle.

http://lists.boost.org/Archives/boost/2012/02/190355.php
My reply to Thomas, arguing that the svn-based workflow that he 
describes is like a stripped down version of the workflow he would get 
with gitflow. Note that Thomas' interesting idea discussed in the 
present post was not taken into account.
The phrase "as observed by Julien" refers to this post:
http://lists.boost.org/Archives/boost/2012/02/190324.php

Daniel sent a reply to my reply to Thomas, which made me aware of 
Daniel's earlier similar proposal. At the time of writing, this post 
cannot be retrieved from the archives yet.