[Boost-build] generator problems

Subject: [Boost-build] generator problems
From: George Georgiev (george.georgiev.sf_at_[hidden])
Date: 2012-03-21 23:49:32

Hi,

It is me again with the generator problem. It looks like it was to
early to conclude it works. When I tried it on windows it did not
work. And it looks like it works on mac and linux just because I was
lucky (or probably unlucky).

It looks like adding H files resolved the issue with adding the
generation location directory to the include path for the compiler,
but it still did not helped the actual dependencies to be computed
correctly.

The following is my latest version - just us a proof of concept. I
added every object file for a generated file to depend on any of the
sources. This makes all h and cpp files to be generated before any of
the object files and this resolves the problem. Unfortunately this
results in every generated cpp file to be recompiled even a single
compil file to be changed.

I am working on this in my "free" time and it looks like the learning
curve for developing in jam language is not as short as I wish. This
generator is part of open source to be project and I would love to
share it with whoever wants to help me with solving these issues.

To refresh the topic:

I would like to create a jam generator class for a generator part of
the project itself. The generator will allow me to create test
projects that will validate the just build generator. (This is already
done) The meta data that the generator use is structured in a
collection of files that may have dependencies between them for
example:

---------- a.compil ------------

// meta data

----------b.compil ------------

import "a.compil";

// meta data that depend on the meta data from a.compil

The generator so far, is 1-to-1 kinda - 1 COMPIL source results in one
pair of CPP and H files. This could change in the future, to be 1
compil to result in multiple pairs of CPP and H files or even other
types, but lets ignore this for the moment.

The generated header files repeat the dependencies from the compil
files, or in our case this will mean that b.h will have #include
"a.h". And this is where the current generator fails. It does not
track the fact that b.cpp includes  a.h indirectly because b.h
includes a.h. This results in the jam build to execute the b.cpp
compilation before a.compil to be (re)generated. This results in
compilation error or b.cpp build against outdated a.h depending on is
the build is initial or incremental.

To make the case even more complex the compil files could have
circular dependencies.

I tried another build systems and they are unable to handle well the
complexity of my case. It looks like Boost Build has the potential to
do it, but it seems that it requires some time to be spend with it
before to be able to develop for it. If some of the more expirenced
guys in the group is willing to help me with this I can send you the
project on you personal email - it is 123MB compressed.

Thanks,
George

import type ;
import scanner ;
import generators ;
import feature ;
import toolset ;
import "class" : new ;
import print ;

type.register COMPIL : compil ;

feature.feature <compil-binary> : : free ;
feature.feature <compil-binary-dependencies> : : free dependency ;

class compil-scanner : scanner
{
    import property-set type scanner ;

    rule __init__ ( includes * )
    {
        scanner.__init__ ;
        self.includes = $(includes) ;
    }

    rule pattern ( )
    {
        return "[ \t]*import[ \t]*\"([^\"]*)\"[ \t]*;" ;
    }

    rule process ( target : matches * : binding )
    {
        local target_path = [ NORMALIZE_PATH $(binding:D) ] ;

        INCLUDES $(target) : $(matches) ;
        SEARCH on $(matches) = $(target_path) $(self.includes:G=) ;
        ISFILE $(matches) ;

        scanner.propagate $(__name__) : $(matches) : $(target) ;
    }
}

scanner.register compil-scanner : include ;
type.set-scanner COMPIL : compil-scanner ;

class compil-generator : generator
{
    import modules path targets jamcompil ;

    rule __init__ ( * : * )
    {
        generator.__init__ $(1) : $(2) : $(3) : $(4) : $(5) : $(6) :
$(7) : $(8) : $(9) ;
    }

    rule run ( project name ? : property-set : sources * : multiple ? )
    {
        local compil-binary = [ modules.peek jamcompil : .compil-binary ] ;
        local compil-binary-dependencies ;

        local compil-main-target = [ targets.resolve-reference
/generator : $(project) ] ;

        compil-main-target =
            [ $(compil-main-target[1]).main-target generator ] ;

        compil-binary-dependencies =
            [ $(compil-main-target).generate [ $(property-set).propagated ] ] ;

        # Ignore usage-requirements returned as first element.
        compil-binary-dependencies = $(compil-binary-dependencies[2-]) ;

        for local target in $(compil-binary-dependencies)
        {
            if [ $(target).type ] = EXE
            {
                compil-binary =
                    [ path.native
                        [ path.join
                            [ $(target).path ]
                            [ $(target).name ]
                        ]
                    ] ;
            }
        }

        property-set = [ $(property-set).add-raw
            <dependency>$(compil-binary-dependencies)
            <compil-binary>$(compil-binary)
            <compil-binary-dependencies>$(compil-binary-dependencies)
        ] ;

        return [ generator.run $(project) $(name) : $(property-set) :
$(sources) : $(multiple) ] ;
    }
}

generators.register [ new compil-generator jamcompil.generate : COMPIL
: CPP H ] ;

toolset.flags jamcompil.generate COMPIL-COMMAND <compil-binary> ;
toolset.flags jamcompil.generate COMPIL-DEPENDENCIES
<compil-binary-dependencies> ;

rule generate ( targets * : source * : properties * )
{
################ HACK BEGIN ######################
    NOTFILE <src>compil ;
    DEPENDS <src>compil : $(targets) ;

    # hack this will work only on windows
    DEPENDS $(targets[1]:S=.obj) : <src>compil ;
################ HACK END ######################
    DEPENDS $(targets[1]) : [ on $(targets) return $(COMPIL-DEPENDENCIES) ] ;
}

actions generate
{
    $(COMPIL-COMMAND) $(>) -o $(<[1]:D) || exit 1
}