Re: [Boost-build] bjam 4.0.. in C++

Subject: Re: [Boost-build] bjam 4.0.. in C++
From: Anthony Foglia (AFoglia_at_[hidden])
Date: 2010-05-24 11:43:59

Robert Ramey wrote:
> Rene Rivera wrote:
>> Once upon a time I started thinking, investigating, sketching, and
>> doing some minor coding for a rewrite of bjam in C++ to address the
>> various problems. Mainly the horrible memory management and hence
>> speed. Unfortunately at the time I had decided that it would not
>> depend on Bosot libraries in the hope of preventing a rather nasty
>> boot-strapping problem. This presented problems as I couldn't find
>> any good lexer/parser that played nice with C++ and wasn't a big PITA
>> to use. It also loomed on my how much of Boost I would end up writing
>> anyway. So I've just about decided to abandon my assertion that a new
>> bjam would not depend on Boost. But I thought I would ask for
>> feedback on this before continuing down this path. The new plan would
>> be to:
>>   * Have bjam 4 depend on a *released* version of Boost.
>>   * Written with Spirit as the base lexer/parser.
>>   * Have a variety of syntax improvements and support for
>> UTF8/Unicode.
>> I'm cleaning up the branch where I started this work at the moment.
>> But I'd like to hear ideas and comments on the new approach. And if
>> you have feature request for such a rewrite now would be a good time
>> to voice them, so we can add them to trac.
> 
> I've complained about bjam from time to time.  But ...
> 
> a) I see the need for such a tool
> b) Although there have been efforts to move to another tool, I'm still
> not convinced that the proposed alternatives are all the way there yet.
> c) you and Vodoya have been there every single time I've need you.  This
> means a lot to me.
> d) bjam does take some time to run - but this is not a huge issue for
> me. Making it 3 times faster wouldn't make me want to use it more
> nor less.
> 
> To me, there are a few big problems with bjam.
> 
> a) There isn't a real good manual for it.  There might be now - but
> when I have a question I can never find the answer without asking
> you guys directly.  The fact that I have to do this should be considered
> a bug.

+1.  The documentation needs to be improved.  As the most egregious 
example that comes to mind, go to 
<http://www.boost.org/doc/tools/build/doc/html/bbv2/reference/rules.html> 
and follow the links in the unit-test description to "the section called 
'Testing'".  See the ticket at 
<https://trac.lvk.cs.msu.su/boost.build/ticket/186>, which has been open 
for 2 years.

There are other minor things, like this line from the testing section 
"If the preserve-test-targets feature has the value off[...]".  AFAICT, 
preserve-test-targets isn't a feature.  I can't use 
"<preserve-test-targets>on" in my requirements for the testing rules, 
and you don't pass it from the command line like other features (i.e 
"preserve-test-targets=on"), but as a command-line option.  Also, I 
don't know what the "<source>" feature does, but it doesn't do quite 
what the documentation says.  It does something similar in most cases 
though.

This, and the lack of third-party resources, doesn't give the impression 
of an active project.

Boris Schaling's tutorial is a good start though.

> b) the behavior is "too smart" in that it does things for me - which is
> fine if it works - but if something is out of whack  I have no clue as to
> what to fix.  Some time ago in a fit of frustration, I wrote
> "The BJAM coke machine" to express my feelings.

Yes.  We chose bjam nearly three years ago for our build system.  We're 
a group of about ten, and nearly everyone is afraid to touch the 
jamfiles because they don't understand them.  I spent most of a week 
recently diagnosing build issues.  Yes, we are probably doing things 
wrong, or in unrecommended ways, but bjam does not help diagnose 
problems.  It either outputs too little, or pages and pages of debug 
output.  There needs to be a middle ground.

And there needs to be consistency.  One of the problems I stumbled on, 
the Gevorg Voskanyan helped me with, is that library "requirements" can 
be overridden by usage-requirements of their sources.  If that's the 
case why are they required?  I think that's a problem somewhere  Until 
he confirmed the same results, and also thought it was a problem, I 
thought it was another case of me not understanding some deep concept 
hidden in the documentation.

(Note that his message hasn't gotten a response yet from a developer 
confirming or denying this is a bug.)

I feel that boost-build is very powerful, but if you're doing anything 
more complex than "hello world", you're going to eventually be in a 
world of pain.

Like I said, we chose boost-build nearly three years ago, and every time 
we have a minor problem with it, the person who made the choice feels he 
must apologize.  I explain, that I think it's powerful, it just feels 
like it needs some work to become more user-friendly, polished, and 
build up a community of users who know stuff.  But I feel know progress 
has been made on that front in the past three years.

Would a faster boost-build help?  Of course, but there are probably 
hacks that can be done to speed things up short of a rewrite.  (For 
instance, it looks like boost-build spends a lot of time parsing system 
and third party headers.  Why can't boost-build store a parsed 
dependency tree for everything outside of the Jamroot, and rely on that? 
  For concerns about not having a clean build, have it turned on with a 
command-line parameter.  It's clumsy, but it should work 90% of the time.)

Boost-build wants to people to think of it as something they don't need 
boost for, something they can use in their own projects.  But they don't 
even make it easy to use boost in boost-build projects.  Look at the out 
-of-date boost.jam file: <https://svn.boost.org/trac/boost/ticket/3173> 
  (I started doing some work on my own to update, but handling the 
switch in boost library naming is beyond my meager jam skills.  And we 
already have a working solution, so why risk screwing it up and spending 
hours debugging our jamfiles for no improvement.  I don't have enough 
work already.)

The best way to get more people to use it, is to make it easy to use in 
the common cases.  You have a bunch of boost users.  And instead of them 
thinking of bjam as some unique one-use requirement for building boost, 
if you can make them think of it also as an easy way to use boost in 
their own projects, that will help a lot.

I'm sorry if this comes off as harsh.  I do think boost-build has a lot 
of potential.  I just haven't seen any movement towards it reaching that 
potential.  Speed improvements will help, but don't lose focus on the 
smaller changes that could have a much greater return.

-- 
Anthony Foglia
Princeton Consultants
[phone] x233