Re: [boost] testing model

Subject: Re: [boost] testing model
From: Thomas Heller (thom.heller_at_[hidden])
Date: 2012-02-12 06:59:39

On 02/12/2012 12:34 PM, Julian Gonggrijp wrote:
> Thomas Heller wrote:
>
>> On 02/11/2012 03:48 PM, Julian Gonggrijp wrote:
>>
>>> Thomas Heller wrote:
>>>
>>>> What would make this whole branching very cool is if we could somehow tie it with testing such that we are able to tell the testrunners to use our feature branch to run the tests (of course this must be automated in a way). I think in this scenario, a modularized boost will come in very handy ... Think of the VHDL model,
>>>
>>> Do you mean this? http://en.wikipedia.org/wiki/VHDL
>>>
>>>> you have unit tests, but the module under test might be very different, though the unit tests stay the same. This is something i would really like to see.
>>>
>>> Could you clarify? I don't see why we would want to test a different
>>> module with the same unit tests.
>>
>> Yes, I mean the hardware description language. Let me explain:
>> In VHDL you usually start prototyping your new design by writing some behavioral description at a very high level, almost like you do with a "normal" programming language. After you are done, you start to write your unit tests for that. You do that by writing a very generic unit test and refer to your new design as "device under test". After you successfully passed all the tests, you start by getting down one level in the abstraction, lets say RTL level (Register transfer level). After that you plug that new device into your unit tests to ensure correct behavior.
>> Now, imagine that different branches are seen as different devices, so to say. Instead of letting the test runners run the unit tests defined in the main branch, you let'em run the unit tests in your new feature branch, completely transparent for them. This means that you don't need to disrupt regular service by committing a half done feature to the main branch, but get the full advantage of the different test runners.
>
> Thomas, I've given this some thought and I can't help but think that
> the VHDL comparison doesn't really work. That's mainly because Boost
> libraries or hypothetical Boost modules are not like subdevices that
> cooperate on a common task, while that does seem to be the case with
> the VHDL example.
>
> The VHDL business is causing us to miss your point, which is a waste
> since your idea does seem to be very interesting. My guess is that,
> given a branching model like gitflow, you envision something along the
> following lines:

Yes you are right, the VHDL way of testing different components was just 
an inspiration to what could be created. Of course, a new feature or a 
refactoring is not the same as different architectural descriptions of a 
piece of hardware ... Anyway ...

> There is a "testing image" which the testing volunteers run their
> periodic tests on. By default, the image mirrors the latest commit on
> the develop branch. When working on a feature branch, library authors
> can reversibly and transparently substitute the version of their
> library from the feature branch for the default version in the image.
> All of this is facilitated by some kind of automated tool which has
> yet to be found.
>
> The result is that most libraries in the image are from develop, but
> some are from feature branches, without the testing volunteers needing
> to worry about it. They can run a single testing cycle on the
> heterogeneous image. For library authors, this brings the advantage
> that they can already have their changes tested on all platforms
> before they feel confident that it should be possible to make them
> work. (Or while they emply a long-term feature branch specifically in
> order to support a platform, per Daniel James.)

Just for reference, and for better understanding of the current test 
model i suggest to get familiar with it:
http://www.boost.org/development/running_regression_tests.html

> Please let us know to what extend this resembles what you had in mind.
> I suggest that you omit VHDL-related examples for the time being.

*nod* looks like you absorbed it and summarized my ideas, thanks ;)

> (After reading Daniel's early post again -- i.e. the second link in
> the PS -- I realize Daniel already proposed something very similar. I
> might have messed up by attributing ideas to the wrong persons; my
> apologies.)
>
> Daniel, I would like to invite you specifically to take part in this
> discussion/brainstorm session.
>
> -Julian
>
>
> PS: some context for those who need it, in chronological order.
>
> http://nvie.com/posts/a-successful-git-branching-model/
> Introduction of the gitflow branching model by the inventor.
>
> http://lists.boost.org/Archives/boost/2012/02/190215.php
> Post by Daniel James, in which he raised his concerns about testing in
> gitflow. The post contained an idea similar to the one presented here.
>
> http://lists.boost.org/Archives/boost/2012/02/190269.php
> Further clarification of Daniel's concern.
>
> http://dl.dropbox.com/u/3512486/boost_branching_model.pdf
> An illustration that compares the current Boost workflow to one in
> which gitflow is applied. It doesn't cover testing, however. I
> introduced it in the following post:
> http://lists.boost.org/Archives/boost/2012/02/190312.php
>
> http://lists.boost.org/Archives/boost/2012/02/190327.php
> Thomas' concern about testing in a gitflow-like branching model.
>
> http://lists.boost.org/Archives/boost/2012/02/190352.php
> My reply to Daniel, arguing that gitflow wouldn't hurt the existing
> test cycle.
>
> http://lists.boost.org/Archives/boost/2012/02/190355.php
> My reply to Thomas, arguing that the svn-based workflow that he
> describes is like a stripped down version of the workflow he would get
> with gitflow. Note that Thomas' interesting idea discussed in the
> present post was not taken into account.
> The phrase "as observed by Julien" refers to this post:
> http://lists.boost.org/Archives/boost/2012/02/190324.php
>
> Daniel sent a reply to my reply to Thomas, which made me aware of
> Daniel's earlier similar proposal. At the time of writing, this post
> cannot be retrieved from the archives yet.
>
>
> _______________________________________________
> Unsubscribe&  other changes: http://lists.boost.org/mailman/listinfo.cgi/boost