Re: [Boost-build] bjam 4.0.. in C++

Subject: Re: [Boost-build] bjam 4.0.. in C++
From: Rene Rivera (grafikrobot_at_[hidden])
Date: 2010-05-23 11:55:24

On 5/23/2010 1:00 AM, Spencer E. Olson wrote:
> My two, three, or four cents on the matter:

Thank you for these.. It's making me think hard about the choices..

> 1.  Issue with Boost dependence:
> One of the main advantages of bjam currently is that it is written in ansi c
> and compiles very, very easily and very quickly for every platform I have to
> work with.  Currently, bjam is not a heavy weight piece of code that is
> difficult to "carry" from machine to machine, let alone compile.  As a contrast,
> think of the "other" main contender for boost attention: CMake.  CMake is both
> gigantic, a pain in the neck to compile (yes, Kitware does provide prebuilt
> versions for many platforms already--but I prefer the small footprint, easy-
> to-move-and-compile bjam).

I certainly prefer a small footprint with quick compiles.. But I have 
yet to find a lightweight parsing solution for C++ (i.e. that plays 
nicely with C++) that is lightweight enough to be packaged with bjam. 
They either all require some largish external program or don't play nice 
with C++.

> If you have bjam rely on any part of boost that can't be repackaged and
> shipped with bjam code, the portability of bjam quickly becomes a nightmare.

True. Which makes it harder since under the plan of depending on Boost I 
would not prepackage it with bjam. Instead depending on the user to have 
it around. Which indeed does complicate the setup.

> Boost does not build completely on all the platform-compiler combinations that
> I work on (IBM AIX cluster machines, Cray XT[345...], ..., Pathscale, PGI,
> ICC, GCC, XLC, ...).  Complicating either the footprint or compilation of
> bjam/Boost.Build will cause me for one to reexamine the build system choice.
>
> I have absolutely nothing against C++, I certainly prefer it over c, but we've
> found many difficulties compiling all of boost already.  If bjam went with a c++
> underneath, I would prefer it to be portable, allowing me to continue using
> Boost.Build even where Boost is not yet tractable.

Hm, I also worry about the portability issue as you do. And it would be 
the one item that makes me rethink depending on Boost for bjam. But I do 
have one question for you.. Doesn't having something like GCC on each of 
those platforms mitigate the problem? I know it's not an ideal solution, 
but it's not unprecedented to require a "better" compiler for tools. Or 
to put it another way; Is GCC available in all the platforms you might 
have problems with?

> 2.  Issue with Boost.Spirit
> A while ago, I had to write another parser.  Plain old yacc and lex don't play
> too well with C++, especially where exceptions and errors are important.  I
> really wanted to try a C++ option, so I tried out Boost.Spirit.  It worked
> quite well and wasn't terribly difficult, although I would say that the
> documentation is not very great--especially considering that Boost.Spirit uses
> C++ in unconventional ways.
>
> Before worrying about portability issues, I had only one beef with Spirit:  it
> took forever to compile.  I ended up trying to isolate the parser in its own
> object code as much as possible (I had parsing happening all over the place)
> because it took so long.

Having done some minor Spirit grammars (1.0 though) isolation would be 
something I would do. Another one would be to use the Spirit.Lex 
component to minimize the compile complexity of the grammar parser itself.

> After spending a month testing several Boost components (Spirit, smart_ptr,
> parts of math, test, and a few smaller pieces) that I wanted to use on the
> various platform+compiler combinations, my grievance list grew.  Many things
> worked quite well:  Boost.test required a few small changes to compile
> everywhere (except PGI 8,9,10 where I exposed a bug in their math code
> generation), Boost.smart_ptr worked easily with one or two #ifdefs for
> pathscale, and Boost.math had several problems--just not with the pieces I
> needed.  Boost.Spirit was quite a bit more problematic, and depended heavily
> on the compiler.  The real stopping point was when I tried compiling with
> IBM's (on AIX) XLC [v 8,9,10].  Any real complicated tests failed to parse.
> Simple tests parsed but never finished compiling--I waited for as long as the
> sysadmin would let me (>15 minutes) without submitting the compilation to a
> batch processing queue.  Further testing showed that the IBM compilers had
> severe problems implementing the standard with respect to default template
> parameters.  I hear that during the last BoostCon, Michael Wong claimed that
> IBMs new 11.1 compiler can compile Boost 1.40, though I have to see it to
> believe it--working on getting a copy to play with.

Another option your comments have made me think about is to take the two 
parsing tasks, lex and grammar, and implement one using an existing 
library and hand-write the other. In this case it would be:

* Use lexertl (the base lexer Spirit.Lex uses 
<http://www.benhanson.net/lexertl.html>) for the lexing as it only 
depends on Boost Config and a small number of other Boost headers. And 
hence would be easy to transport with bjam.

* Implement a recursive parser for the grammar by hand. Although this is 
something I'm not fond of at this time. It's something I have enjoyed in 
the past.

The end result would be a small self contained parser with minimal 
requirements. But more work from myself.

> At this time, I did a little more research into other c++ parser options.  I
> found that GNU/Bison and GNU/Flex actually play very well with c++.  I rewrote
> my grammar/lexer in Bison/Flex including using their extensions to generate
> c++ code.  The result:  a much easier to maintain parser.  It compiles very
> quickly, is very portable, and actually turned out more robust than the Spirit
> version--I think it was easier to make robust.  There is good documentation
> for Yacc/Bison grammars.

Yacc/Bison is what I'm trying to move away from.. And I generally want 
to stay away from GNU (and the GPL). The problem with using those is 
what I stated above, they are external tools. And they have proven a bit 
of a pain at times even with bjam3. Since it makes the bootstrap process 
harder as it has to find those tools and as a developer I need to get 
them installed, etc.

> I will say that I don't simply include the bison/flex code in my source code.
> I have Boost.Build detect any installed bison/flex versions at build time.  If
> the version is new enough, I have it generate new c++ code.  Otherwise, I use
> pregenerated versions that I also package with my code.  I also have to
> package one c++ header file from Flex from the machine where the pregenerated
> versions are created.  This has been working very well and has greatly
> improved compiling and portability issues.

That is essentially what the current bjam3 does, but it doesn't have the 
luxury of BB to do it ;-) Which actually boils down to it doesn't really 
do it and assumes the developer has all his tool ducks in a row.

> Sorry about the long rant...

Again.. Thank you for the rant :-)

-- 
-- Grafik - Don't Assume Anything
-- Redshift Software, Inc. - http://redshift-software.com
-- rrivera/acm.org (msn) - grafik/redshift-software.com
-- 102708583/icq - grafikrobot/aim,yahoo,skype,efnet,gmail