Re: [Boost-build] Tarball/archive support in bjam

Subject: Re: [Boost-build] Tarball/archive support in bjam
From: John Bito (jwbito_at_[hidden])
Date: 2009-08-11 15:47:35

In case you're interested, here is a tar.jam that will do the basic
operation if you have gzip and tar commands in the path.  First the
jamfile:

import tar : tarball ;

tarball tarballname : [ glob-tree *.?pp ] : <compression>--best ;

--- now tar.jam ---

import type ;
import targets ;
import project ;
import feature ;
import generators ;

type.register TAR : tar ;
type.register GZIP_TAR : tgz ;

feature.feature compression :
    # specify the list of valid compression options here
    : free incidental
    ;

rule tarball ( target-name : sources * : requirements * : default-build * )
{
   targets.create-typed-target GZIP_TAR : [ project.current ] : $(target-name) :
       $(sources) : $(requirements) : $(default-build) ;
}

generators.register-composing tar.tar.create : CPP HPP EXE LIB : TAR ;
# not sure how to add generic source file types
generators.register-standard tar.tgz.create : TAR : GZIP_TAR ;

rule tgz.create ( targets * : sources * : properties * )
{
    COMPRESSION on $(targets) = [ feature.get-values compression :
$(properties) ] ;
}

actions tgz.create
{
    gzip $(COMPRESSION) > $(<) < $(>)
}

actions tar.create
{
    tar -cf $(<) $(>)
}

On Tue, Aug 11, 2009 at 11:41, John Bito<jwbito_at_[hidden]> wrote:
> Right-o. AFAIK, there are no actions built into the language that
> create or move files.  The way you're thinking of going would be
> appropriate to pursue based on the Python implementation of
> Boost.Build.  There's already language support for what you want in
> Python, right?
>
> On Tue, Aug 11, 2009 at 10:36, Matthew
> Chambers<matthew.chambers_at_[hidden]> wrote:
>> The bjam language-level rules like DEPENDS, NOTFILE, and W32_GETREGNAMES
>> contain platform-specific system calls; each platform has its own file<os>.c
>> file. If a library like libarchive was available to bjam then it could
>> natively support compression without the need for external tools. So ideally
>> this functionality would not depend on the ability to find a tar/zip tool in
>> the environment. The question is whether it is feasible to add that
>> dependency to the bjam compilation (and again, it could be optional - people
>> could point the build.* scripts to the libarchive source code). If Volodya
>> is opposed to this though then it's reasonable to make do with depending on
>> finding the tools in the environment.
>>
>> The bjam language-level archive support would be low-level so Boost.Build
>> rules and actions would provide an abstraction layer on top of it so there
>> would be a convenient interface like there is for the exe, lib, and obj
>> target types.
>>
>> -Matt
>>
>>
>> John Bito wrote:
>>>
>>> The Boost Build 'native' way is to implement platform-specific rules
>>> in a toolset (tools/setname.jam).  Once the dependency network is
>>> built, bjam starts invoking 'actions' which form commands for the
>>> shell.  AFAIK, there's nothing in bjam that creates/moves target files
>>> except through actions.
>>>
>>> I'm sure I don't fully grasp the requirements for building tarballs in
>>> the specific case, but you'll depend on the ability to find a tar /
>>> zip tool somewhere in the environment.  You can look at the way
>>> actions are constructed in the various tools/*.jam files to see how
>>> the command is determined based on the environment.
>>>
>>> My thought is to start with the specific, not worrying (too much)
>>> about how to make it platform independent and once that works, add the
>>> indirection needed to make it configure automatically for the
>>> environment.
>>>
>>> On Tue, Aug 11, 2009 at 07:57, Matthew
>>> Chambers<matthew.chambers_at_[hidden]> wrote:
>>>
>>>>
>>>> That's a cool approach; it's a compromise between supporting it natively
>>>> and
>>>> using simple SHELL calls to do the archive manipulation. But it would be
>>>> tricky to get it to work on Windows. I suppose some kind of
>>>> initialization
>>>> check could be done in the tgz rule to make sure that the gzip command
>>>> exists in the current PATH and if not, to exit with an informative
>>>> warning;
>>>> linking the user to GnuWin32 on Windows and their package management
>>>> system
>>>> on *nix would go a long way.
>>>>
>>>> Thanks,
>>>> -Matt
>>>>
>>>>
>>>> John Bito wrote:
>>>>
>>>>>
>>>>> Making a tarball target is similar to the (Java) jar target that I'm
>>>>> working on.  The built-in composing generator seems that it ought to
>>>>> work. For your case, I'd start with something like:
>>>>>
>>>>> import type ;
>>>>> import targets ;
>>>>> import project ;
>>>>> import feature ;
>>>>> import generators ;
>>>>>
>>>>> type.register TAR : tar ;
>>>>> type.register GZIP_TAR : tgz ;
>>>>>
>>>>> feature.feature compression :
>>>>>   # specify the list of valid compression options here
>>>>>   : free incidental
>>>>>   ;
>>>>>
>>>>> rule tarball ( target-name : sources * : requirements * : default-build
>>>>> *
>>>>> )
>>>>> {
>>>>>  targets.create-typed-target GZIP_TAR : [ project.current ] :
>>>>> $(target-name) :
>>>>>      $(sources:G=) : $(requirements) : $(default-build) ;
>>>>> }
>>>>>
>>>>> generators.register-composing tar : CPP HPP EXE LIB : TAR ; # not sure
>>>>> how to add generic source file types
>>>>> generators.register-standard tgz : TAR : GZIP_TAR ;
>>>>>
>>>>> # Convert requirements to variables to substitute in the command line.
>>>>> rule tgz ( targets * : sources * : properties * )
>>>>> {
>>>>>   COMPRESSION on $(targets) = [ feature.get-values compression ] ;
>>>>> }
>>>>> # actual commands to run
>>>>> actions tgz
>>>>> {
>>>>>   gzip $(COMPRESSION) < $(<) >$(>) ;
>>>>> }
>>>>>
>>>>> actions tar
>>>>> {
>>>>>   tar -cf $(<) $(>) ;
>>>>> }
>>>>>
>>>>> tarball tarballname : [ glob *.cpp ] : <compression>--best ;
>>>>>
>>>>> Disappointingly, it errors out:
>>>>>
>>>>> error: properties found in the 'sources' parameter
>>>>>
>>>>> I'd like to know how to things properly so that doesn't happen.  It
>>>>> seems that the default composing generator assumes that its
>>>>> dependencies are targets coming from other generators and rather than
>>>>> non-derived files.
>>>>>
>>>>> I'll be working on this more later in the week, so maybe I'll have
>>>>> something that actually works.
>>>>>
>>>>> Good luck!
>>>>> John
>>>>>
>>>>> On Mon, Aug 10, 2009 at 13:36, Matthew
>>>>> Chambers<matthew.chambers_at_[hidden]> wrote:
>>>>>
>>>>>
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I'd like to have support for .tar.gz, .tar.bz2, .tar.lzma, and .zip in
>>>>>> bjam.
>>>>>>
>>>>>> I work on a library project with a BBv2 build system and we have
>>>>>> tarballs
>>>>>> that we use as input and tarballs that we want to be build artifacts.
>>>>>> We
>>>>>> currently package the full Boost tarball in our SVN repo and in our
>>>>>> source
>>>>>> releases. Instead of that, we'd like to have a "build release package"
>>>>>> rule
>>>>>> that will run bcp on our project (preferably only on updated
>>>>>> dependencies),
>>>>>> add/update any new files in a bcp tarball, and finally build several
>>>>>> tarballs to distribute our source code and probably some Windows
>>>>>> binaries
>>>>>> too. Some of the source tarballs would be subsets of the full source to
>>>>>> make
>>>>>> it easy for users to pick a subset of our library.
>>>>>>
>>>>>> It would be very nice to handle all this inside of the BB
>>>>>> dependency/update
>>>>>> web. When input tarballs change it would trigger all of its dependents
>>>>>> to
>>>>>> rebuild. When output tarballs have updated dependencies it would
>>>>>> trigger
>>>>>> the
>>>>>> output tarball to rebuild. The logic should quite the same as what
>>>>>> already
>>>>>> exists with the lib target. I suppose it's actually building the
>>>>>> archive
>>>>>> (libarchive?) support into bjam that would be the messy part! Perhaps
>>>>>> we
>>>>>> could adopt a system like boost::iostreams where the archive support is
>>>>>> compiled conditionally.
>>>>>>
>>>>>> I'd like to see rules with semantics like the lib target where it can
>>>>>> be
>>>>>> either an input or output (or both?).
>>>>>> Here's some example code:
>>>>>> Also pasted on codepad: http://codepad.org/iNE5kl8o
>>>>>>
>>>>>> # extract a tarball
>>>>>> tarball some-source-tarball-id
>>>>>> : # sources (empty list specifies input tarball?)
>>>>>> : # requirements
>>>>>> # specifies the filepath of an existing tarball (only applicable for
>>>>>> input
>>>>>> tarball)
>>>>>> <file>path/to/a/tarball.tgz
>>>>>>
>>>>>> # override default location to extract files;
>>>>>> # does defaulting (and, for relative paths, anchoring) to current
>>>>>> Jamfile's
>>>>>> directory make more sense than using the build path?
>>>>>> <location>path/to/extracted/files
>>>>>>
>>>>>> # no default-build
>>>>>> # no usage-requirements
>>>>>> ;
>>>>>>
>>>>>> # create/update a tarball from the source tree
>>>>>> tarball output-source-tarball-id
>>>>>> : # sources (can be files or targets; for targets, the target's output
>>>>>> files
>>>>>> are archived)
>>>>>> # add all .cpp and .hpp files, preserving their relative paths
>>>>>> [ glob-tree . : *.?pp ]
>>>>>>
>>>>>> : # requirements
>>>>>> # <location> on a <source> could override the path of the file in the
>>>>>> tarball?
>>>>>> <source>some-disjoint-header.hpp/<location>override/path/in/tarball
>>>>>>
>>>>>> # specify archive compression
>>>>>> <compression>lzma # alternatives: none, gzip, bzip2, zip; the zip would
>>>>>> not
>>>>>> actually be a tarball
>>>>>>
>>>>>> # override name of output (default is the rule name)
>>>>>> <name>source-tarball # add file extension automatically if omitted?
>>>>>>
>>>>>> # override path of output;
>>>>>> # does defaulting (and, for relative paths, anchoring) to current
>>>>>> Jamfile's
>>>>>> directory make more sense than using the build path?
>>>>>> <location>path/to/output/tarball
>>>>>>
>>>>>> # no default-build
>>>>>> # no usage-requirements
>>>>>> ;
>>>>>>
>>>>>> # create/update a tarball from the output of some build targets
>>>>>> tarball output-binary-tarball-id
>>>>>> : # sources
>>>>>> some-exe-target
>>>>>> : # requirements
>>>>>> <library>some-library-target/<with>features
>>>>>> <dependency>some-make-target
>>>>>> <name>binary-tarball.tar.bz2
>>>>>>
>>>>>> # no default-build
>>>>>> # no usage-requirements
>>>>>> ;
>>>>>>
>>>>>>
>>>>>> So, would anyone else find this helpful? I might be able to implement
>>>>>> it
>>>>>> in
>>>>>> bjam if it's approved of as a new feature.
>>>>>> _______________________________________________
>>>>>> Unsubscribe & other changes:
>>>>>> http://lists.boost.org/mailman/listinfo.cgi/boost-build
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Unsubscribe & other changes:
>>>>> http://lists.boost.org/mailman/listinfo.cgi/boost-build
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Unsubscribe & other changes:
>>>> http://lists.boost.org/mailman/listinfo.cgi/boost-build
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Unsubscribe & other changes:
>>> http://lists.boost.org/mailman/listinfo.cgi/boost-build
>>>
>>
>> _______________________________________________
>> Unsubscribe & other changes:
>> http://lists.boost.org/mailman/listinfo.cgi/boost-build
>>
>