[Boost-build] Custom generator bug seen on Linux but not Windows/MacOSX

Subject: [Boost-build] Custom generator bug seen on Linux but not Windows/MacOSX
From: Matthew Chambers (matt.chambers42_at_[hidden])
Date: 2010-07-16 13:58:07

  Hi,

I have been using a custom generator to create tarballs from path globs and/or exe/lib targets and it's been working well for creating tarballs of executables 
and source tarballs. I've recently tried to add a library tarball (i.e. both precompiled libraries and header files without source files) and have run into a 
frustrating issue that I haven't been able to debug. I wasn't able to reproduce the problem with a simple test case though. I have no idea what happens between 
the simple case and the real case where it stops working.

A subset source zip (10mb) that exhibits the problem is at:
http://fenchurch.mc.vanderbilt.edu/files/pwiz-subset-src.7z

The problematic part is somewhere between:
local headers = [ path.glob-tree pwiz : *.h *.hpp : .svn bindings vendor_api ] ;
package.install install-pwiz-lib
     : <install-source-root>$(PWIZ_ROOT_PATH)
     : # executables
     : pwiz
     : $(headers)
     ;
explicit install-pwiz-lib ;

and:
tar.create pwiz-lib.tar.bz2
   : # sources are handled by the conditional
   : # requirements
<conditional>@library-tarball-requirements
   ;
explicit pwiz-lib.tar.bz2 ;

The first rule should install the library targets under the pwiz subdirectory and the headers to whatever --prefix is set to. Then the pwiz-lib.tar.bz2 target 
will create a tarball from that location (of course it's a terrible idea to build the tarball if the --prefix is set to a system-wide location, but that's a 
refinement to make later). I've been testing with a local --prefix like "build-linux-x86_64/usr" which will install it to and tar it from a subdirectory of the 
build directory.

This all works perfectly fine on MacOSX and Windows, but for some reason I haven't been able to determine, not on Linux. On Linux, specifying the 
"pwiz-lib.tar.bz2" target correctly builds the install-pwiz-lib dependency, but the tar.create action never runs. And this problem only happens on CLEAN builds. 
If I run the tarball target in a separate bjam invocation from the install-pwiz-lib target, it works as expected on all platforms. But I need it to work on a 
clean build. Another way to get it to work is to either pass in no headers or pass in just a small range of headers, like so:
package.install install-pwiz-lib
     : <install-source-root>$(PWIZ_ROOT_PATH)
     : # executables
     : pwiz
     : $(headers[1])
     ;

This leads me to my best guess of what's going wrong: somewhere in the custom generator some limit is being silently tripped and I don't know how to debug it.

We have a script to easily clean, bootstrap, and invoke bjam with a given command line. The command I've been using is:
scripts/misc/tcbuild.sh -j8 --prefix=build-linux-x86_64/usr pwiz-lib.tar.bz2

We're using a bootstrapped-from-source version of Boost.Build (libraries/boost-build) from trunk r58428. This is moderately out of date, but before I update I'd 
like to know if anyone remembers fixing anything that might solve this problem? And if not, could someone more familiar with the guts of BB take a crack at 
this? We have to build BB from source because I've added a built-in RESCAN for supporting tar.extract (which happens at parse-time, and after it runs, the 
file/timestamp caches have to be cleared from bjam).

Thanks,
-Matt