Re: [Boost-build] Lib of libraries (using SHA-1 build property set hashes)

From: Jurko Gospodnetić (jurko.gospodnetic_at_[hidden])
Date: 2008-05-25 12:18:45

   Hi Volodya.

>> P.S.
>>    There is a different topic of whether to maintain that folder as we 
>> do now or perhaps use just use a SHA-1 hash of the used property set and 
>> not organize the build folders hierarchically. This would help avoid the 
>> problem with build folder paths becoming too long for some projects and 
>> hitting the OS path size limit.
> 
> I always wanted to use signature of build properties to determine if a target
> has to be rebuilt, but never got around to implement that.
> 
> Technically, there are two approaches
> 
> 1. Leave current directory naming intact, but compute and store SHA-1
> signatures of properties or maybe build command. Rebuild the target
> is the signature has changes. This will make sure a target is rebuild
> when you add a define, or cxxflags, or change compiler command. OTOH,
> it's not always desirable to rebuild everything in such case, so we
> might want some "just update signatures" option.
> 
> 2. Use SHA-1 for directory names. This is almost the same, with the
> added benefit of making directory names shorter. But to make
> "just update signatures" possible, we'd need to compute the SHA-1 of
> only the properties that now show up in path, and separately compute SHA-1
> of all properties for each generated file.

   As I see it, Boost Build builds a single target into separate build 
folders depending the properties used for it. This is done so that 
targets built with matching properties would not overwrite each other 
and you can more easily use different builds of the same target in the 
same project, i.e. without having to tweak your Jamfiles to make such a 
thing possible.

   Now, forcing targets to be rebuilt for properties that do not have 
their corresponding build folder seems contradictory to this original idea.

   The only general solution I can think of that supports multiple 
builds of the same target using different properties within the same 
project/build is to use build folders whose names are calculated based 
on the non-incidental properties used to build targets contained in 
those folders. For example, we can use SHA-1 hashed together with a 
description file in that folder listing the exact property values used.

   Good sides:
   -----------
     * Build folders have short paths - therefore avoiding the long path 
problems encountered on larger projects.
     * Projects with targets built using multiple property sets do not 
require any special handling for those files as they all work correctly 
out of the box.
     * Free features like cxxflags may be used when deciding which build 
folder to use. This means that solutions like ad-hoc compiler option 
tweaks will work cleanly instead of having to be implemented using 
separate Boost Build features for every such solution.

   Bad sides:
   ----------
     * Build folders become more clumsy to locate by humans when they 
know the exact property set used.

   As I see it the bad side does not seem that bad to me as I rarely 
ever need to go trudging through the Boost Build folders looking for a 
specific built target. If needed, the build system could also be 
upgraded so it can be asked: 'where will you place this target?'.

   Also, it would be relatively straightforward to add support for 
explicitly named build folders for a specific property set.

   As far as free features affecting the build go - one can say that 
they should be modeled as separate features if they are important, but 
one can also argue that this is just extra overhead imposed by the build 
system that should be able to directly understand that 'typically' 
specifying different cxxflags should cause targets to be built in 
different folders (or at least under different names). Alas, in real 
life this is a moot area since there really are such free feature values 
that are incidental (i.e. do not affect the built targets) and there are 
those that are not. Also, in real life, projects that require building 
the same target with multiple properties are minority. All things 
considered, I guess the build system should make the conservative choice 
and consider all free features as non-incidental unless explicitly 
tagged as such, which the current implementation does not.

   The current solution of using separate folder per property has the 
following good and bad sides with respect to the solution presented above:

   Good sides:
   -----------
     * Easy for a human to navigate the build folder tree.

   Bad sides:
   ----------
     * Free features must always be incidental as Boost Build can not 
construct folder names directly, mostly due to different OS restrictions 
on what characters can be included in folder names.
     * Build folder tree can easily become very large and break OS 
imposed limits on the maximum length (e.g. 260 on Windows unless using 
tricks in BJam code and any external build tools as well, which might 
not even be possible).

   I am attaching the version of this posting as a new ticket in the 
Boost Build trac (https://zigzag.cs.msu.su:7813/boost.build/ticket/183) 
so it does not get lost.

   Best regards,
     Jurko Gospodnetić