[Boost-build] Issues with dependencies on targets that have an implicit-dependency on other targets

Subject: [Boost-build] Issues with dependencies on targets that have an implicit-dependency on other targets
From: Sven Van Echelpoel (sven.van.echelpoel_at_[hidden])
Date: 2009-03-13 08:53:15

Hi,

we're experiencing problems with dependencies on targets that have an
implicit dependency on other targets. Let me start by explaining our
setup.

Similar to Boost we have a project layout that is reasonable flat. It
more or less looks like this:

(Root)
  +  /jamfiles
  |    + all-aliases.jam
  |   ...
  |
  +- Jamroot
  |
  +- /software
      + /Project_X
      |   + Jamfile.v2
      |
      + /Project_Y
      |   + Jamfile.v2
     ...

First of all, from within every sub-project we include a file
all-aliases.jam (JAMFILES is a path-constant set in Jamroot)

include $(JAMFILES)/all-aliases.jam ;

This file automatically creates alias targets by globbing the project
tree. These targets can then be referred to from within the
sub-projects. So far this has been very handy. Here's how we create the
targets:

##################### all-aliases.jam ############################

# This is a variable that contains the list of
# all software projects
all-software-projects =
    [ MATCH .*software/(.*)/.* : [ glob
$(JAMROOTDIR)/software/*/Jamfile.v2 ] ]
    ;

# Declare some empty variables
all-software-components = ""
    ;
all-software-tests = ""
    ;
all-software-tools = ""
    ;

# Create aliases for every component
for local l in $(all-software-projects)
{
  # Create an alias for the component itself
  alias $(l) : $(JAMROOTDIR)/software/$(l)//Lib$(l) ;
  all-software-components += $(l) ;
  explicit $(l) ;
  # Create an alias for the component test
  alias $(l)_Test : $(JAMROOTDIR)/software/$(l)//$(l)_Test ;
  all-software-tests += $(l)_Test ;
  explicit $(l)_Test ;
  # Create an alias for the component tool(s)
  alias $(l)_Tool : $(JAMROOTDIR)/software/$(l)//$(l)_Tool ;
  all-software-tools += $(l)_Tool ;
  explicit $(l)_Tool ;
}

##################### end all-aliases.jam #########################

With this we were able to refer to other projects by the directory name
of the project we wanted to refer to. E.g.

lib prj_x : prj_x.cpp Project_Y ;

with Project_Y the directory name of the project we wanted to refer to.
The alias would refer to: $(JAMROOTDIR)/software/Project_Y//Lib
$Project_Y. Naturally in the sub-project's Jamfile.v2 a target (usually
an alias) Lib$Project_Y needs to exist.

So far this has picked up all the dependencies correctly as well as
propagating the usage requirements.

Secondly, a while ago we solved another dependency problem related to
automatically generated files. With the help from Vladimir Prus we were
able to propagate the path to the directory in which some header files
were generated from one target to a dependent target (in the same
sub-project). See
http://www.nabble.com/-generators--add-include-path-to-the-output-folder-td19046275.html for the complete thread. So in that project we have targets like these:

## /ProjectWithAProblem/Jamfile.v2

lib prj1 : ... Sources ... Project_X Project_Y ... more dependencies ...
         :
         :
         : <include>$(PUBLIC_PROJECT_INCLUDES)
         ;

lib prj2 : ... Sources ... Project_Z ... more dependencies ...
         : <implicit-dependency>prj1
         :
         : <include>$(PUBLIC_PROJECT_INCLUDES)
         ;

exe exe1 : .. Sources ... prj2 ;

alias LibProjectWithAProblem : prj2 ;

So far this worked fine. If any files were out of date, a proper build
was started of those files and all include directories required were
found, including those propagated from the implicit-dependency.

Yet, now we want to use the include files from (and possibly later link
to) the project prj2 in ProjectWithAProblem. As before we just added the
directory name (aka the alias) of the project we want to reference.

## /Project_T/Jamfile.v2

lib anotherlib : ... Sources ... ProjectWithAProblem ;

This works, as long as ProjectWithAProblem does not need to be rebuilt.
If a file is out of date, the ProjectWithAProblem will be rebuilt. This
is actually prj2 that is being rebuilt. 

However, and here's finally the problem, somehow the usage requirements
of the implicit-dependency target prj1 aren't propagated to prj2. The
usage requirements of the dependencies of prj1 (Project_X,
Project_Y, ...) aren't propagated either. This only happens when the
build is started from within the directory of Project_T and not when it
is started in the directory of ProjectWithAProblem. The result of all of
this is that the build fails. By running with -d+2 you can clearly see
that the include directories from all dependent usage requirements
aren't added to the command at all.

Interestingly, a warning is generated for each implicit-dependency
target (like prj1, but we have more of them). Notice also that there is
no double slash separating the path from the target.

warn: Unable to construct <path to ProjectWithAProblem>/prj1
                                                     ^^^^
                                     No double slash here

We were able to solve the problem for now by using a, err...,  <use>
feature instead of a POD (plain old dependency):

## /Project_T/Jamfile.v2

lib anotherlib : ... Sources ... 
   : <use>ProjectWithAProblem 
   ;

As long as we are only using header-only components we're fine. But when
we need to link, obviously this solution will no longer work. Another
thing that works is by explicitly invoking the build of
ProjectWithAProblem, like so:

## /Project_T/Jamfile.v2

build-project ../ProjectWithAProblem//prj2 ;

lib anotherlib : ... Sources ... ProjectWithAProblem ;

For now we're fine. Yet, we're still left with a couple of questions:

1. Is our approach of automatically generating targets valid or may this
be a source of confusion for BB/bjam?
2. Have we used the implicit-dependency feature correctly or did they
way we used it attribute to the problem we are seeing?
3. If we have done OK on both previous bullets, can we assume that there
is a genuine problem with the implicit-dependency feature?

Any input would be greatly appreciated.

Sorry for the very long posting, but we wanted to present a complete
case. We like BB a lot (a hellufa lot), but its lack of documentation in
relevant areas (no criticism here, at least none overt) meant that we
often had to do stuff in ways that we not fully understand. It is
therefor entirely possible that by organizing our projects the way we
did, we ended up shooting ourselves in our foot. When problems do emerge
we are often left scratching our heads, wondering why BB/bjam is
rejecting our instructions. Still it is a great tool. 

Kudos and keep up the good work!

Thanks,

Sven