Re: [Boost-build] BUMPITTY BUMP: [git] genrating forward headers

Subject: Re: [Boost-build] BUMPITTY BUMP: [git] genrating forward headers
From: Bjørn Roald (bjorn_at_[hidden])
Date: 2013-08-27 15:43:52

On 08/27/2013 02:48 PM, Vladimir Prus wrote:
> On 27.08.2013 10:58, Bjørn Roald wrote:
>> Unless someone is willing to bless the patch and commit it, I suggest
>> we should try get on the table what remaining work is needed.  I can
>> not commit that as I have no access and I am not sure I would want to
>> do it without blessing from Steven or Volodya anyway.  I personally
>> see no reasons to wait other than get more test coverage in SVN layout
>> to make sure we have not broken anything.  If we need more testing, I
>> will make a posting in the developer mailing list to see if others can
>> help making that happen faster.
>
> Bjørn,
>
> could you provide me with the patches that need applying, as well as
> list of whatever steps that should bring me from an empty
> directory for a build, and I'll give it a try?
>
> Yes, I know that some patches were posted in there, and that steps
> should be 'whatever is somewhere on the wiki', but it
> would help a lot if I can work from the same basis as you.

Sure - no problem, thanks for taking some time looking at it:

first off you need to make sure you have a fresh boost SVN trunk working 
directory or git clone, including the following related commits:

SVN r84524 | steven_watanabe | 2013-05-27 02:02:43 +0200
Initial supprort for creating symlinks in the git layout.

SVN r85183 | steven_watanabe | 2013-08-01 17:49:37 +0200
Apply Bjorn Roald's patch for generating header links in git.

Then you need to apply one patch, latest version I posted 2 days ago is 
fine, see:
http://lists.boost.org/boost-build/2013/08/26953.php
attachment: 201308252200-boost-trunk.patch

To test on SVN layout I do the standard:
========================================
test goal - test that none of the changes make any observable effect in 
SVN layout at all -- except the file changes, maybe b2 debug info and 
stuff like that.

svn co http://svn.boost.org/svn/boost/trunk boost-trunk
cd boost-trunk
svn apply 201308252200-boost-trunk.patch
./bootstrap.sh
./b2
check build with no errors
cd status
../b2 -j 4
check build results consistent with with
http://www.boost.org/development/tests/trunk/developer/summary.html
lately I have been getting
...failed updating 83 targets...
...skipped 146 targets...

To test on modularized layout
========================================
The https://svn.boost.org/trac/boost/wiki/TryModBoost may be used. 
Below is the step I use:

test goal - test that we are as close to normal for a b2 based build 
system as feasible before we commit patch.

git clone --recursive https://github.com/boostorg/boost.git modular-boost
cd modular-boost
git apply 201308252200-boost-trunk.patch
./bootstrap.sh
./b2 headers     ir just ./b2
check build with no errors
check build results consistent with with
http://www.boost.org/development/tests/trunk/developer/summary.html

In addition any other testing you feel like doing is probably a good 
idea for sure.

Notes:
=======

git apply seams to apply the patch created with svn diff just fine 
across the whole repo + submodules in git.  Too bad I still have not 
figured a way to create a similar diff with git diff traversing 
submodules recursively and creating one multi repo diff. I think there 
is a way as some web resources indicate - I have just not figured how.

some bits in the patch that you may care about in Jamroot:
-----------
+BOOST_MODULARLAYOUT = $(all-headers) $(numeric-headers) ;
+if $(BOOST_MODULARLAYOUT)
+{
+    echo modularized layout build ;
+}
------------
the BOOST_MODULARLAYOUT is really not used much any more, but I thought 
it was a good idea to keep it arround so it is possible to 
conditionalize in any jamfile if needed.  Also the echo line above is 
mainly cor convenience, and we can remove that before commit if you 
prefer or move it into some of the config printouts if you think that is 
better.

note that as long as the "modularized layout build" printout does not 
show in at beginning of build as you should expect in SVN layout, then the:

all-headers
numeric-headers
BOOST_MODULARLAYOUT

variables are all empty lists.  Thus the new link-directory rule is not 
called and in the call to the alias rule to create the headers target;

------------
-alias headers : : : : <include>. ;
+alias headers : $(all-headers)-headers 
numeric-$(numeric-headers)-headers : : : <include>.  ;
-------------

the sources argument does not expand to anything other than the empty 
list.  I had a condition test around alias headers in earlier patches, 
but it was not needed as the call to alias will be equal to the 
pre-patch version as the sources list is empty in SVN layout.

I decided to keep the explicit headers line unchanged, I have been back 
and forth on this but decided not to change it as it did not seem to 
have the effect I had guessed anyway.  I actually thought that was 
something Jürgen had added in his patch as a safety thing to require 
explicit:

./b2 headers
rather than just
./b2

based on request from you on the list, but I realized it used to be 
there before and git blame tells me it has been in there since a commit 
by you 2010-08-09 08:59:19.  So it stays!

Notes on other files in the patch:
----------------------------------
many of the changes is to capture the effect of the search replace 
prescribed by Steven.  I ran the

rgrep -l boost//headers * | xargs sed -i 
's%<\(use\|source\|library\)>/boost//headers%<implicit-dependency>/boost//headers%g' 

command after inspecting result of
rgrep boost//headers *

to verify that I got the files I wanted, and nothing else.  I had to do 
one or two files by hand. You may run

rgrep boost//headers *

to inspect results of the substitution.  See Stevens reasoning here for 
reference:
http://lists.boost.org/boost-build/2013/08/26882.php
some of these substitutions may be redundant as the requirement is also 
added at project level now, but that is up to library authors to clean 
up as desired I think.

I added a new project at libs/Jamfile.v2 to add 
<implicit-dependency>/boost//headers to all library and test targets. 
Likewise I added similar requirements to project level in 
tools/Jamfile.v2 and various Jamroot files I could find.   See:
http://lists.boost.org/boost-build/2013/08/26885.php
As use of <implicit-dependency>/boost//headers after substitution shall 
cause any new any effect in SVN layout, I removed the guards I had in 
earlier patches at project level.  This make things simpler, but it may 
have side effects we don't want.

known issues in modularized layout:
===================================

b2 modularized layout issue 1:
------------------------------
build in status require build in root first, at a minimum
cd $BOOST_ROOT
b2 headers

but default targets with just b2 seems to work as well.  I think all 
remaining header links not used by any of the library builds get created 
at the end.  Maybe stage target depends on headers target or something, 
I have not checked.  Logically that would make sense.

b2 modularized layout issue 2:
------------------------------
install require header links to proxy the SVN layout.  The install glob 
needs update to get the headers from the sources according to Steven.  I 
have no idea how to do that.

Best regards,
--
Bjørn