Re: [Boost-build] Sorting include headers

Subject: Re: [Boost-build] Sorting include headers
From: Phillip Seaver (phil_at_[hidden])
Date: 2008-10-15 13:09:07

Alexander Sack wrote:
> On Wed, Oct 15, 2008 at 11:44 AM, Phillip Seaver <phil_at_[hidden]> wrote:
>   
>> Alexander Sack wrote:
>>     
>>> As per FAQ:
>>>
>>> http://www.boost.org/doc/tools/build/doc/html/bbv2/faq/s04.html
>>>
>>> The boost build system seems to sort header files.  Is there any way
>>> to turn that off completely?  I know a long shot but to be honest,
>>> this is a very very (VERY) annoying feature of bjam.  It should
>>> preserve the order in which is specified on the command line (I don't
>>> see why internally it needs to do this).
>>>
>>>       
>> >From http://lists.boost.org/boost-build/2004/08/7084.php
>>
>>     
>>> Some more information. The properties are sorted for performance
>>> reasons. We do quite a lot of work with property sets and try to cache
>>> results. For caching to really help, differences in order should be
>>> erased:
>>> "<variant>release <debug-symbols>on" and "<debug-symbols>on
>>> <variant>release" are exactly the same.
>>>       
>> As I understand it, the property sets are just strings (or lists of
>> strings), so if they're not sorted, they would have to either sort
>> before comparing or check each item in one set against the items in the
>> other set.  Property sets are usually *much* longer than the simple
>> example above, so it can make a very substantial difference, especially
>> for large project trees.
>>     
>
> Maybe so but is the performance that BAD to induce build errors and
> other link nastiness?
>   

I think that it may be.  Run "bjam -n --debug-generators" to see how
large your property sets are and how many are used.  The first one that
showed up when I did that was over 900 characters.  Even if it's a list
of strings internally instead of a string that they have to parse into a
list before comparing, breaking each item into property and value and
then comparing each property in one list (using string compares) against
each property in the other list could be quite expensive.

On the other hand, I would argue that Boost.Build is just revealing the
build errors.  :-)  I think it's a really bad idea to rely on include
path order if you can possibly avoid it.

>>> In a prior thread, it was recommended I can use alias but again, how
>>> do ensure the order of includes is preserved (and for my production
>>> PROJECT it does matter - I'm not going to rewrite source just yet to
>>> avoid obvious namespace collisions, the current build just puts the
>>> includes in the right order and things compile).
>>>       
>> The same kind of problem has hit us a few times.  I usually try to fix
>> the code, because it just seems like a never-ending problem otherwise.
>> It can be a pain, but so are "random" compile problems when the order of
>> headers changes or a new library comes in with the same name.
>>     
>
> Yeah totally agree with you Philip but in order to evangelize a build
> tool I have to at least proof that bjam can build the exact pieces of
> code that autoconf and friends can (a true source of industry
> evilness).
>   

It can be a good bit of work (though it gets easier as you do more), but
you can.  We use a bunch of third-party libraries (libtiff, zlib,
libxml, wxwidgets, ...) where we had to write the Jamfiles for projects
that do things in a dozen different ways (autoconf, make, VS projects,
cake).

One of the big selling points for us was the fact that we write one
Jamfile for all the platforms we support (Window, OS X, Linux, Solaris,
AIX).  Some have OS-specific parts to them, but it's a lot cleaner than
the way we used to do it, with one Makefile per platform.

Another thing we always ran into was differences in the configuration of
Visual Studio project files -- incorrectly linking things that use
different runtimes causes all sorts of problems.  And, if you need to
change a define for one library, you have to go find all the projects
that use it to change it in those places as well (this was with VC6, so
it may have improved).  With Boost.Build, all the includes and defines
are defined by the Jamfile for the library and everything that uses them
inherits them automatically.

Automatic dependency checking is also wonderful.  It slows things down,
but it beats having to find out why the fix you made isn't in your
executable.  :-)

It's a fair bit of work switching over, but I think it's definitely
worth it.

>> Unfortunately, I don't have an easy solution for the ordering problem.
>> You could probably do something like take all the relative paths,
>> convert them to full paths (in bjam), and join them with "&&".  I don't
>> know off the top of my head how to write that, though.
>>     
>
> What's this means though is in order to preserve the order I either have to:
>
> - Move the headers in an order that works (i.e. move files around)
> - Alphabetize my directories so it magically gets sorted in a buildable way
> - Specify dozens of headers on the target rule using the && syntax (ugly as sin)
>
> Uhhh this is gonna be a tough sell!  :D
>   

I don't know if it would affect your build, but in my copy of
Boost.Build, I have a change that prevents the propagation of define's
and include paths past the main target that uses them.  So, if you have:

Jamfile in directory a:

    lib a : a.cpp : : : <include>. ;

Jamfile in directory b:

    lib b : b.cpp /a : : : <include>. ;

Jamfile in directory c:

    exe c : c.cpp /b ;

...and your Jamroot has "use-project /a : a ;", etc., the compile for
b.cpp will have "-I../a" and the compile for c.cpp will only have
"-I../b".  With the current distribution, the compile for c.cpp would
get both paths.  If I want to pass on the requirements, I can just do:

    lib b : b.cpp /a : : : <include>. <use>/a ;

IIRC, I ran into a problem where one library used a define in a
different way than another or something like that.  Also, it cuts down
on the length of compile lines and compile times.

In case you're interested, in boost-build/build/targets.jam at the end
of the "compute-usage-requirements" rule, I have:

        raw = [ property.change $(raw) : <pch-header> ] ;
        raw = [ property.change $(raw) : <pch-file> ] ;
        # don't propagate defines and includes
        raw = [ property.change $(raw) : <define> ] ;
        raw = [ property.change $(raw) : <include> ] ;

Phillip