Re: [boost] [Git] Moving beyond arm waving?

Subject: Re: [boost] [Git] Moving beyond arm waving?
From: Rene Rivera (grafikrobot_at_[hidden])
Date: 2011-02-08 16:10:03

On 2/8/2011 1:17 PM, Dave Abrahams wrote:
> At Mon, 07 Feb 2011 20:30:54 -0600,
> Rene Rivera wrote:
>>
>> On 2/7/2011 7:57 PM, Dave Abrahams wrote:
>>> At Mon, 07 Feb 2011 09:53:52 -0600,
>>> Rene Rivera wrote:
>>>>
>>>>> What about a library developer? What does the tree structure they work
>>>>> with look like? How does integrate with their development repo? I
>>>>> guess the non-version controlled tree produced by the above could be
>>>>> used as a "complete (integrated) release tree", but I'd like to know
>>>>> the specifics, and give them a try.
>>>>
>>>> I would also like to know:
>>>>
>>>> 1. How does that non-versioned complete integrated tree work as regards to
>>>>      updates/pulls?
>>>
>>> Today, you call GenHeaders explicitly, but if you check out Marcus' work you
>>> don't have to.
>>
>> I tried finding where&  how Marcus did what you say, by going to the github link
>> you give in the other response.. But I don't see what there you are referring
>> to.
>
> I've asked him to follow up here with details, but I believe it's in the commits
> from 2011-01-19 at https://github.com/boost-lib/cmake/commits/master

OK, I see that now.. So the answer is that it still generates forwarding 
headers. But now does it "automatically" using Cmake.

>>>> 2. What does it mean for testing? Specifically, complete incremental testing?
>>>
>>> Nothing?  What, specifically, are you worried about?
>>
>> OK.. How would we run the existing testing
>
> Hmm, "existing testing" is a little vague, but...

I didn't think it was.. Since I'm talking about the context of this 
thread which is about using Git. And not about using Cmake, or Ryppl, or 
anything else. Hence the "existing testing" is using the current scripts 
but with Git sources.

>> in incremental mode with a "forwarding headers tree plus separate sources"?
>> What would testers need to rely on? Cmake? Python? Ryppl? Ctest? SSH? Git?
>> FTP? HTTP? HTTPS? Git protocol?
>
> Here's where I think we'll end up:
>
> - testers need to set up and register a buildbot buildslave.  I believe we can
>    make that relatively trivial for people

Even though I like buildbot.. I'm less than thrilled about adding setup 
steps for testers. As one of the testing goals is to make it simple for 
someone to provide testing resources, and hopefully fine-grained 
resources. Which means...

> - that would imply a reliance on, at a minimum:
>
>    - Python
>    - A net connection
>
> Other things they will probably need to have, or the testing process will need
> to get for them:
>
>    - CMake
>    - C and C++ compilers
>    - BuildBot
>    - Twisted
>    - Git or LibGit2 or Dulwich
>    - Git or HTTP or HTTPS protocol capability

...that any additional required programs/scripts would ideally be 
automatically installed to a sandbox (i.e. locally to not need system 
administrator access). For example; I've had a good deal of experience 
dealing with installing Twisted and it doesn't approach being nice about 
being installed anywhere other than in the system location. But my 
experience might be dated by now, since I haven't looked at it in more 
than a year. And as you can see from my other posts, I'm trying to 
figure out what the experience is like of installing Git is :-) With 
less than pleasurable results so far :-(

>>>> 3. Or is there no way to get a complete with source integrated tree?
>>>
>>> Please define "complete with source integrated tree" so I can answer that
>>> question.
>>
>> I mean a tree in the form we have it now, or equivalent to what we intend to
>> *release* as the *single* Boost C++ Libraries package.
>
> Well, we need to decide the details, but I can imagine two possibilities for a
> Boost distro:
>
> 1. It is equivalent to the result of doing a "git clone" of the Boost superproject
>     plus "git submodule update --init" to get all the subprojects (possibly with
>     the .git directories deleted)
>
> 2. It's like #1 plus a generated forwarding headers directory.
>
> Either of these is pretty trivial to generate.

I see. Is there a way to do the same without Git? I.e. do the tar/zip 
archives available from github contain the same files as the 
clone+submodules?

>> I ask from a testing manager POV. I need to know what the changes might be for
>> testers and me managing the testing infrastructure. And testers need to know
>> what will be required of them.
>
> My plan is to require as little as possible.  I've had some good experiences
> with writing BuildBot recipes that get all the pieces needed into the
> environment before running the tests, and I think that can be done
> straightforwardly.

OK. A comment on that at the end...

>> My fear is that we end up with a system that is conceptually hard to understand
>> how to put together and hence hard to make work. For example, testing at the
>> moment doesn't depend on SVN. It's possible to download the tree as a tar
>> archive through the almost universally available HTTP protocol.
>
> Yes.  Well, *incremental* testing today implies either having SVN or some kind
> of rsync-like mechanism to avoid updating unchanged files.

Currently yes. But it's possible to make it not be required. It would be 
straightforward to change the regression scripts to download a new tree, 
using the tar archive option and assuming the time stamps are correctly 
preserved (it does already IIRC), and move over the "bin.v2" directory 
from the old tree to the fresh one. And since all the generated files 
and timestamps in the bin.v2 dir are appropriate one could do an 
incremental test run on the new tree.

>> This is just one example. I'd have to see what the entire testing pipeline
>> looks like to figure out where the "locked in" points are. I.e. I need to know
>> what's actually going on, rather than just a "run this and that's it" answer.
>
> Provided I can automate everything else, I think all you need to know is what
> ports have to be open, right?

...And the comment I promised above...

No, there's more. The assumption is that testers have the capability to 
open a port. Also assumes that we want them open a port. I'd argue that 
we don't want them to open a port because it's yet another manual setup 
step they have to do. Essentially if I want to make it possible for 
someone at home, with their cable/DSL connected PC, to runs tests, I 
need to make it as simple as: 1. Have Python >= 2.4; 2. download 
"test.py"; 3. run "python test.py". In my dreams it would be even 
simpler, and just be download an installer, run it, and nothing else. 
Which brings up another item; Your description so far also assumes the 
tester will need to install/setup a variety of programs. Which is moving 
in the wrong direction, IMO.

-- 
-- Grafik - Don't Assume Anything
-- Redshift Software, Inc. - http://redshift-software.com
-- rrivera/acm.org (msn) - grafik/redshift-software.com
-- 102708583/icq - grafikrobot/aim,yahoo,skype,efnet,gmail