[Boost-build] RFC: problems lazy loading modules referenced using source target-ids

Subject: [Boost-build] RFC: problems lazy loading modules referenced using source target-ids
From: Jurko Gospodnetiæ (jurko.gospodnetic_at_[hidden])
Date: 2012-08-07 06:05:31

   Hi all.

   I need some more opinions on what to do with a Boost Build problem 
related to lazy loading project modules explicitly referenced using 
source target-ids, e.g as in '../some/other/project//source'.

   A text-book Boost Build run does the following:
     1. Loads the main project module, its parent modules and 
recursively any additional project modules referenced from there (e.g. 
via the use-project rule). These loaded modules contain different 
feature declarations.
     2. Processes the build request command-line (all the features used 
in the command-line need to be declared by this time).
     3. Constructs the virtual-target/action tree needed to construct 
the requested main targets and the declared features are used in this 
process, especially non-optional features.

   However, things get complicated if explicit project module references 
are allowed in source ids, e.g.:
   exe some-exe : ../some/other/project//source ;

   Such additional modules get loaded in phase 3 instead of phase 1, 
assuming nothing caused them to get loaded earlier.

   This then may (and does!) cause problems if such lazy-loaded modules 
define new non-optional features. Such features are then taken into 
consideration when constructing later targets, but were not taken into 
consideration when constructing earlier ones. There may also be other 
issues caused by this lazy loading that I am not aware of.

   I'm attaching a test case demonstrating one such issue, prepared to 
be run as a regular internal Boost Build test case. It prepares a 
project where, if you build its ccc and ddd targets in order, it fails 
due to two separate virtual-targets getting prepared for a single actual 
produced file (each has a bit different property set). If you reverse 
the order in which these two targets are built - everything goes well. 
Also, if you load its 'a' project module directly using the use-project 
rule - everything builds fine no matter which order the targets are 
built in.

   Another issue caused by the same underlying problem was reported 
recently on the list by Norbert Wenzel (see the 'Build error when adding 
Boost libraries to own projects' thread).

   Debugging issues caused by this gets real messy and requires one to 
delve deep into the Boost Build internals which does not seem like a 
'Good Thing'.

   I now ask 'what can be done about this'?

   Some ideas:

--- ignore the issue ---
   Don't like this as we are leaving behind a problem that people are 
going to keep running into time and time again, causing them to waste a 
lot of time debugging the issue and possibly even give up on using Boost 
Build all together.

--- disallow explicit project module references in source target-ids ---
   Alternative is to reference the project module earlier using the 
use-project rule, assigning a project-id for it and then use that 
project-id to reference targets from that project. Bad thing about this 
is that we are possibly complicating simpler build scenarios and 
potentially slowing down more complex one with use this to lazy load 
project modules only when and id actually needed.

--- disallow declaring features in modules loaded due to being 
explicitly references in a used source target id ---
   The only problems caused by such lazy loaded modules that I can 
currently see are caused by them declaring features. We could possibly 
report an error if this is enountered and allow such lazy module loading 
for projects not declaring additional features.

--- disallow declaring non-optional features in modules loaded due to 
being explicitly references in a used source target id ---
   It is even possible (I am not 100% certain of this though) that 
declaring optional features (i.e. those that do not get their default 
value automatically added to all built targets) does not cause any 
problems and can be allowed. Just food for though...

   Thanks in advance for your input...

   Best regards,
     Jurko Gospodnetiæ

P.S.
   There is also a bit more general design change that could be done at 
a later time which I find like a nice idea, and that is to make features 
project-specific (or at least allow them to be declared as such). Then 
we could allow project-specific features to be declared as well but all 
this is a separate issue.