Re: [Boost-build] Issue running bjam with multiple jobs and custom code generator

Subject: Re: [Boost-build] Issue running bjam with multiple jobs and custom code generator
From: Christophe B. (boost_at_[hidden])
Date: 2016-11-06 05:02:27

Hi Steven,

Thanks for the fast reply. See my comments below.

On 2016-11-04 23:50 GMT+01:00 Steven Watanabe wrote:

> AMDG
>
> On 11/04/2016 10:27 AM, Christophe B. wrote:
>>
>> I developed a protoc.jam (see attachment) allowing to invoke an external
>> code generator tool (protoc from protobuf library). The tool generates
>> c++ unit files X.pb.h X.pb.cc <http://X.pb.cc> from a source X.proto.
>>
>> When I run bjam -j1 from scratch everything works like a charm.
>>
>> When I run bjam -j6 from scratch, I get compilation issues complaining
>> that the generated X.pb.h file is missing, because the compiler compiles
>> other cpp units in parallel before protoc has fully finished.
>>
>> Why it does work correctly with one job and not with multiple jobs
>> remains a mystery to me.
>
> It only works with one job by accident.
>
>> I would expect that bjam detects the
>> dependencies of cpp units with the generated X.pb.h file and waits for
>> its generation before starting compilation of depending cpp units.
>>
>
> The dependency is being missed for some reason.
>
>> I also attached the Jamfile of my project.
>>
>
> I'm not sure exactly where the problem is, but there
> are a couple of issues I can see with protoc.jam:
> - The generator should list both H and CPP as its outputs.

I thought that was the purpose of the following lines:
                 local targets ;
                 targets += [ new file-target $(name:B).pb : H : 
$(project) : $(action) ] ;   # <------------ H
                 targets += [ new file-target $(name:B).pb.cc exact : 
CPP : $(project) : $(action) ] ; # <----------- CPP

                 local result ;
                 for local t in $(targets) {
                     result += [ virtual-target.register $(t) ] ;
                 }
                 return $(result) ;

> - override generated-targets instead of run and

Not sure to follow you here. Do you mean that I should override another 
method from generator. If yes, is there some examples I could follow?

>   don't bother checking the source type.

The implementation was inspired on soap.jam generated provided in the 
examples.

>
> Minor extra note:
> - actions can access any module scope variables,
>   so there's no need to create an on TARGET variable
>   for .prefix.

Good point to know.

>
> In Christ,
> Steven Watanabe
>

I also found another implementation of protobuf generator: 
http://lipingworkshop.blogspot.com/2012/01/boostbuild-bjam-file-to-support-google.html
I made some adaptations based on it, but I am not sure it is correct, 
but seems to work (hoping this is not by coincidence). It seems that 
adding the two lines DEPENDS solve the issue, but this is not clear to 
me. Do you have explanations to provide ?

rule run ( project name ? : property-set : sources * )
{
     if ! $(sources[2]) {
         local type = [ $(sources[1]).type ] ;
         if $(type) = PROTO {
             if ! $(name) {
                 name = [ generator.determine-output-name $(sources) ] ;
             }
             local action = [ new action $(sources[1]) :
                              protoc.generate : $(property-set) ] ;

             local target1 = [ new file-target $(name:B).pb.cc exact :
                               CPP : $(project) : $(action) ] ;
             local target2 = [ new file-target $(name:B).pb.h exact :
                               H : $(project) : $(action) ] ;

             DEPENDS all : [ $(target1).actualize ] ;
             DEPENDS all : [ $(target2).actualize ] ;

             return
                 [ virtual-target.register $(target1) ]
                 [ virtual-target.register $(target2) ] ;
         }
     }
}

Many thanks for your support,

Christophe B.