Re: [boost] [Git] Moving beyond arm waving?

Subject: Re: [boost] [Git] Moving beyond arm waving?
From: Dave Abrahams (dave_at_[hidden])
Date: 2011-02-08 14:17:04

At Mon, 07 Feb 2011 20:30:54 -0600,
Rene Rivera wrote:
> 
> On 2/7/2011 7:57 PM, Dave Abrahams wrote:
> > At Mon, 07 Feb 2011 09:53:52 -0600,
> > Rene Rivera wrote:
> >>
> >>> What about a library developer? What does the tree structure they work
> >>> with look like? How does integrate with their development repo? I
> >>> guess the non-version controlled tree produced by the above could be
> >>> used as a "complete (integrated) release tree", but I'd like to know
> >>> the specifics, and give them a try.
> >>
> >> I would also like to know:
> >>
> >> 1. How does that non-versioned complete integrated tree work as regards to
> >>     updates/pulls?
> >
> > Today, you call GenHeaders explicitly, but if you check out Marcus' work you
> > don't have to.
> 
> I tried finding where & how Marcus did what you say, by going to the github link
> you give in the other response.. But I don't see what there you are referring
> to.

I've asked him to follow up here with details, but I believe it's in the commits
from 2011-01-19 at https://github.com/boost-lib/cmake/commits/master

> >> 2. What does it mean for testing? Specifically, complete incremental testing?
> >
> > Nothing?  What, specifically, are you worried about?
> 
> OK.. How would we run the existing testing 

Hmm, "existing testing" is a little vague, but...

> in incremental mode with a "forwarding headers tree plus separate sources"?
> What would testers need to rely on? Cmake? Python? Ryppl? Ctest? SSH? Git?
> FTP? HTTP? HTTPS? Git protocol?

Here's where I think we'll end up:  

- testers need to set up and register a buildbot buildslave.  I believe we can
  make that relatively trivial for people

- that would imply a reliance on, at a minimum:

  - Python
  - A net connection

Other things they will probably need to have, or the testing process will need
to get for them:

  - CMake
  - C and C++ compilers
  - BuildBot
  - Twisted
  - Git or LibGit2 or Dulwich
  - Git or HTTP or HTTPS protocol capability

> >> 3. Or is there no way to get a complete with source integrated tree?
> >
> > Please define "complete with source integrated tree" so I can answer that
> > question.
> 
> I mean a tree in the form we have it now, or equivalent to what we intend to
> *release* as the *single* Boost C++ Libraries package. 

Well, we need to decide the details, but I can imagine two possibilities for a
Boost distro:

1. It is equivalent to the result of doing a "git clone" of the Boost superproject
   plus "git submodule update --init" to get all the subprojects (possibly with
   the .git directories deleted)

2. It's like #1 plus a generated forwarding headers directory.

Either of these is pretty trivial to generate.

> I ask from a testing manager POV. I need to know what the changes might be for
> testers and me managing the testing infrastructure. And testers need to know
> what will be required of them. 

My plan is to require as little as possible.  I've had some good experiences
with writing BuildBot recipes that get all the pieces needed into the
environment before running the tests, and I think that can be done
straightforwardly.

> >> I'm worried as not having an easy way to get that would make testing rather
> >> difficult.
> >>
> >> Note, I dont' consider "use ryppl", or "use cmake", as an acceptable answer
> >> ;-) As being locked into any particular tool is something I'm vehemently
> >> against.
> >
> > I don't think I'd give you an answer like that to any of these questions.  But
> > that said, being "locked in" somehow is unavoidable; we're locked into SVN now
> > aren't we?  We're certainly locked into C++ and Python and Boost.Build and
> > DocBook and BoostBook and QuickBook and... the list goes on.  Or do you mean
> > something distinct by "locked in?"
> 
> Note, I'm ignoring doc tools with this..
> 
> I guess I do mean something distinct by "locked in". The current set of tools, a
> good portion of them just scripts, do not in my view lock us in because they are
> relatively straightforward to move to something else that provides the basic
> equivalent functionality. This is because they map essentially to operations one
> could do by hand. But I do concede that is a rather thin definitional line ;-)

Yes.  And since you can do anything at all by hand... ;-)

> My fear is that we end up with a system that is conceptually hard to understand
> how to put together and hence hard to make work. For example, testing at the
> moment doesn't depend on SVN. It's possible to download the tree as a tar
> archive through the almost universally available HTTP protocol. 

Yes.  Well, *incremental* testing today implies either having SVN or some kind
of rsync-like mechanism to avoid updating unchanged files.  For that, I think we
should simply require Dulwich or LibGit2, or, failing that, Git.  Clean-slate
testing should only require HTTP and zip to get the code, since GitHub can
supply zip- (or tar-)balls for any revision
(http://github.com/ryppl/ryppl/zipball/master and
http://github.com/ryppl/ryppl/tarball/master for example).

> This is because
> we have testers, or at least have had testers, that can't do anything else. So
> if there's a requirement to have Git protocol access, that's likely a killer for
> testing. 

Git already can use HTTP/HTTPS, so there's no reason Git protocol should be
required.

> This is just one example. I'd have to see what the entire testing pipeline
> looks like to figure out where the "locked in" points are. I.e. I need to know
> what's actually going on, rather than just a "run this and that's it" answer.

Provided I can automate everything else, I think all you need to know is what
ports have to be open, right?

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com