Re: [Boost-build] bjam memory consumption

From: Reece Dunn (msclrhd_at_[hidden])
Date: 2006-04-15 19:36:29

Tomas Puverle wrote:
> > 
> > Python's programming facilities, including its garbage collection
> > capability (which would address the subject of this thread), are much
> > better suited than the Jam language to the sort of medium-scale
> > software project that Boost.Build has become.
> 
> I was afraid you were going to say that.
> 
> I've used another python based build system - SCons - in the past and while 
> it's great from a ease of use point of view, the performance of the build 
> process is terrible.

If you compare SCons to Boost.Build, you get similar performances for both.

> One of the really appealing things about bjam is how fast it is.  OTOH, the 
> python interpreter startup/runtime can be very slow on several platforms I've 
> used it on (Sparc/Solaris 64-bit, Cygwin).

BJam - which is a variant of Jam - and Python are both comparable here. They are the languages that Boost.Build and SCons are written in.

> Here's a link to an article that benchmarks different build systems:  
> http://www.gamesfromwithin.com/articles/0506/000092.html

This covers SCons and Jam, but not Boost.Build.

> Perhaps this is not an issue and you have some data to put my mind at rest?

I have been using Python for some months now and I really like it :). I am using the Python tar file utilities to create a simple extraction tool that rivals the best extraction tools out there in terms of performance.

The problem with bulding Boost.Build on top of BJam is that BJam only operates on strings and rules. Therefore, all the classes, lists, numbers, property pairs and so forth that Boost.Build operates on have to be simulated as strings and BJam is not very optimized for string use.

As an example, one of the major consumers of memory and time in BBv2 was the response file and printing logic. This has since been moved into the BJam language, but there are still other areas that consume memory.

Python is very good and very fast at manipulating data structures. For example you can do things like:

   lines = open( 'data.dat' ).read().split( '\n' )
   print '\n'.join( lines )

and:

   cmd = [ 'cvs' ]
   cmd.append( '-d:%(connection-type)s:%(user)s@%(server)s' % data )
   command.run( cmd )

There have been discussions about using SCons' target scaning/dependency checking logic and constructing Boost.Build on top of that, but I want to be convinced that using SCons is the right choice because of the performance issues with it. Python I have no problem with because that would allow the current Boost.Build implementation to be simplified. As an example, Python has a Set class which can be used to replace set.jam.

Dave: is there any howto/wiki-style documentation for building BJam with Python support and (1) importing a Python module into a Jamfile/*.jam file and writing a Jamfile/*.jam file in Python?

I'd like to start experimenting with replacing the lower levels of BBv2 (such as set.jam) with Python and also be involved with the Python BB conversion project.

- Reece
_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today it's FREE!
http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/