Re: [Boost-build] Tarball/archive support in bjam

Subject: Re: [Boost-build] Tarball/archive support in bjam
From: Matthew Chambers (matthew.chambers_at_[hidden])
Date: 2009-08-11 13:36:54

The bjam language-level rules like DEPENDS, NOTFILE, and W32_GETREGNAMES 
contain platform-specific system calls; each platform has its own 
file<os>.c file. If a library like libarchive was available to bjam then 
it could natively support compression without the need for external 
tools. So ideally this functionality would not depend on the ability to 
find a tar/zip tool in the environment. The question is whether it is 
feasible to add that dependency to the bjam compilation (and again, it 
could be optional - people could point the build.* scripts to the 
libarchive source code). If Volodya is opposed to this though then it's 
reasonable to make do with depending on finding the tools in the 
environment.

The bjam language-level archive support would be low-level so 
Boost.Build rules and actions would provide an abstraction layer on top 
of it so there would be a convenient interface like there is for the 
exe, lib, and obj target types.

-Matt

John Bito wrote:
> The Boost Build 'native' way is to implement platform-specific rules
> in a toolset (tools/setname.jam).  Once the dependency network is
> built, bjam starts invoking 'actions' which form commands for the
> shell.  AFAIK, there's nothing in bjam that creates/moves target files
> except through actions.
>
> I'm sure I don't fully grasp the requirements for building tarballs in
> the specific case, but you'll depend on the ability to find a tar /
> zip tool somewhere in the environment.  You can look at the way
> actions are constructed in the various tools/*.jam files to see how
> the command is determined based on the environment.
>
> My thought is to start with the specific, not worrying (too much)
> about how to make it platform independent and once that works, add the
> indirection needed to make it configure automatically for the
> environment.
>
> On Tue, Aug 11, 2009 at 07:57, Matthew
> Chambers<matthew.chambers_at_[hidden]> wrote:
>   
>> That's a cool approach; it's a compromise between supporting it natively and
>> using simple SHELL calls to do the archive manipulation. But it would be
>> tricky to get it to work on Windows. I suppose some kind of initialization
>> check could be done in the tgz rule to make sure that the gzip command
>> exists in the current PATH and if not, to exit with an informative warning;
>> linking the user to GnuWin32 on Windows and their package management system
>> on *nix would go a long way.
>>
>> Thanks,
>> -Matt
>>
>>
>> John Bito wrote:
>>     
>>> Making a tarball target is similar to the (Java) jar target that I'm
>>> working on.  The built-in composing generator seems that it ought to
>>> work. For your case, I'd start with something like:
>>>
>>> import type ;
>>> import targets ;
>>> import project ;
>>> import feature ;
>>> import generators ;
>>>
>>> type.register TAR : tar ;
>>> type.register GZIP_TAR : tgz ;
>>>
>>> feature.feature compression :
>>>    # specify the list of valid compression options here
>>>    : free incidental
>>>    ;
>>>
>>> rule tarball ( target-name : sources * : requirements * : default-build *
>>> )
>>> {
>>>   targets.create-typed-target GZIP_TAR : [ project.current ] :
>>> $(target-name) :
>>>       $(sources:G=) : $(requirements) : $(default-build) ;
>>> }
>>>
>>> generators.register-composing tar : CPP HPP EXE LIB : TAR ; # not sure
>>> how to add generic source file types
>>> generators.register-standard tgz : TAR : GZIP_TAR ;
>>>
>>> # Convert requirements to variables to substitute in the command line.
>>> rule tgz ( targets * : sources * : properties * )
>>> {
>>>    COMPRESSION on $(targets) = [ feature.get-values compression ] ;
>>> }
>>> # actual commands to run
>>> actions tgz
>>> {
>>>    gzip $(COMPRESSION) < $(<) >$(>) ;
>>> }
>>>
>>> actions tar
>>> {
>>>    tar -cf $(<) $(>) ;
>>> }
>>>
>>> tarball tarballname : [ glob *.cpp ] : <compression>--best ;
>>>
>>> Disappointingly, it errors out:
>>>
>>> error: properties found in the 'sources' parameter
>>>
>>> I'd like to know how to things properly so that doesn't happen.  It
>>> seems that the default composing generator assumes that its
>>> dependencies are targets coming from other generators and rather than
>>> non-derived files.
>>>
>>> I'll be working on this more later in the week, so maybe I'll have
>>> something that actually works.
>>>
>>> Good luck!
>>> John
>>>
>>> On Mon, Aug 10, 2009 at 13:36, Matthew
>>> Chambers<matthew.chambers_at_[hidden]> wrote:
>>>
>>>       
>>>> Hi all,
>>>>
>>>> I'd like to have support for .tar.gz, .tar.bz2, .tar.lzma, and .zip in
>>>> bjam.
>>>>
>>>> I work on a library project with a BBv2 build system and we have tarballs
>>>> that we use as input and tarballs that we want to be build artifacts. We
>>>> currently package the full Boost tarball in our SVN repo and in our
>>>> source
>>>> releases. Instead of that, we'd like to have a "build release package"
>>>> rule
>>>> that will run bcp on our project (preferably only on updated
>>>> dependencies),
>>>> add/update any new files in a bcp tarball, and finally build several
>>>> tarballs to distribute our source code and probably some Windows binaries
>>>> too. Some of the source tarballs would be subsets of the full source to
>>>> make
>>>> it easy for users to pick a subset of our library.
>>>>
>>>> It would be very nice to handle all this inside of the BB
>>>> dependency/update
>>>> web. When input tarballs change it would trigger all of its dependents to
>>>> rebuild. When output tarballs have updated dependencies it would trigger
>>>> the
>>>> output tarball to rebuild. The logic should quite the same as what
>>>> already
>>>> exists with the lib target. I suppose it's actually building the archive
>>>> (libarchive?) support into bjam that would be the messy part! Perhaps we
>>>> could adopt a system like boost::iostreams where the archive support is
>>>> compiled conditionally.
>>>>
>>>> I'd like to see rules with semantics like the lib target where it can be
>>>> either an input or output (or both?).
>>>> Here's some example code:
>>>> Also pasted on codepad: http://codepad.org/iNE5kl8o
>>>>
>>>> # extract a tarball
>>>> tarball some-source-tarball-id
>>>> : # sources (empty list specifies input tarball?)
>>>> : # requirements
>>>> # specifies the filepath of an existing tarball (only applicable for
>>>> input
>>>> tarball)
>>>> <file>path/to/a/tarball.tgz
>>>>
>>>> # override default location to extract files;
>>>> # does defaulting (and, for relative paths, anchoring) to current
>>>> Jamfile's
>>>> directory make more sense than using the build path?
>>>> <location>path/to/extracted/files
>>>>
>>>> # no default-build
>>>> # no usage-requirements
>>>> ;
>>>>
>>>> # create/update a tarball from the source tree
>>>> tarball output-source-tarball-id
>>>> : # sources (can be files or targets; for targets, the target's output
>>>> files
>>>> are archived)
>>>> # add all .cpp and .hpp files, preserving their relative paths
>>>> [ glob-tree . : *.?pp ]
>>>>
>>>> : # requirements
>>>> # <location> on a <source> could override the path of the file in the
>>>> tarball?
>>>> <source>some-disjoint-header.hpp/<location>override/path/in/tarball
>>>>
>>>> # specify archive compression
>>>> <compression>lzma # alternatives: none, gzip, bzip2, zip; the zip would
>>>> not
>>>> actually be a tarball
>>>>
>>>> # override name of output (default is the rule name)
>>>> <name>source-tarball # add file extension automatically if omitted?
>>>>
>>>> # override path of output;
>>>> # does defaulting (and, for relative paths, anchoring) to current
>>>> Jamfile's
>>>> directory make more sense than using the build path?
>>>> <location>path/to/output/tarball
>>>>
>>>> # no default-build
>>>> # no usage-requirements
>>>> ;
>>>>
>>>> # create/update a tarball from the output of some build targets
>>>> tarball output-binary-tarball-id
>>>> : # sources
>>>> some-exe-target
>>>> : # requirements
>>>> <library>some-library-target/<with>features
>>>> <dependency>some-make-target
>>>> <name>binary-tarball.tar.bz2
>>>>
>>>> # no default-build
>>>> # no usage-requirements
>>>> ;
>>>>
>>>>
>>>> So, would anyone else find this helpful? I might be able to implement it
>>>> in
>>>> bjam if it's approved of as a new feature.
>>>> _______________________________________________
>>>> Unsubscribe & other changes:
>>>> http://lists.boost.org/mailman/listinfo.cgi/boost-build
>>>>
>>>>
>>>>         
>>> _______________________________________________
>>> Unsubscribe & other changes:
>>> http://lists.boost.org/mailman/listinfo.cgi/boost-build
>>>
>>>       
>> _______________________________________________
>> Unsubscribe & other changes:
>> http://lists.boost.org/mailman/listinfo.cgi/boost-build
>>
>>     
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost-build
>