Re: [jamboost] Re: Boost.Build and SCons

From: Pedro Ferreira (pedro.ferreira_at_[hidden])
Date: 2004-12-14 06:12:13

Hello Dave,

Em 13 Dec 2004, =E0s 18:49, David Abrahams escreveu:

>
> Pedro Ferreira wrote:
>> Dear all,
>>
>> I recently told Volodya I was going to spend some time experimenting
>> the integration between Boost.Build and SCons.
>> Unfortunately, I have been flooded with urgent tasks so I still=20
>> haven't
>> managed to spend more than an hour or so doing this.
>> I see interest from many of us in pursuing such an integration so I
>> think we need to debate where are we going to and which path should we
>> choose.
>>
>> My vision is:
>> - create a 100% Python Boost.Build library on top of SCons
>> - create a parser for the bjam language
>
> That's a good goal, but I think the steps outlined by Volodya in
> http://thread.gmane.org/gmane.comp.programming.tools.scons.user/5667=20
> are
> the right way to get there.

For reference, this is what Volodya proposed by then:

1. Initially, bjam will invoke SCons at the lowest level (creating=20
SCons Nodes instead of bjam's targets).

2. If that works out OK, we'd need to consider how to mix Python and=20
jam language. Say, so that one could write new tool in Python.

3. After that, we can gradually move Boost.Build code to Python. Given=20
that the languages are very similar, this should not be very hard.

My first feeling was that it was indeed the best way to proceed and I=20
implemented a prototype of 1.:

1. change build-system.jam to dump a Python map of all virtual-targets=20
and their relevant properties
2. create a Python script that reads the map an creates the appropriate=20
SCons nodes
3. map BB properties into SCons environment settings

This works fine when using simple properties, such as <define>,=20
<include> and <linkflags>, which are readily supported by SCons=20
enviroments.
But, in order to map things like <optimization> or <debug-symbols>, you=20
need to know which compiler you are using.
And, there must be a mapping between BB target type and the appropriate=20
Environment method to call in order to create a target. And you must=20
choose env.StaticObject or env.SharedObject depending on the target=20
being a DLL or a LIB.

So, I got to the conclusion that we would be better off moving directly=20
to Python.

>> This would allow people to use Jamfiles but would provide a Python
>> interface to Boost.Build, which would open way to a myriad of tools,
>> such as graphical browsers/creators, builder GUIs, etc.
>>
>> The integration with SCons can be done:
>> - at the SCons.Environment level, calling builder methods to create BB
>> virtual targets
>> - at the lower level Node ADT interface
>>
>> I really don't like SCons Enviroments because they are too cluttered
>> with functions, are too unwieldy, load all tools by default, etc.
>> So I prefer the second approach: it provides all the functionality=20
>> bjam
>> provides (and more) and fits very well with BB's architecture.
>
> I had the same feeling about Environments.
>
>> Regarding the approach, I see two possibilities:
>> - do a progressive transition from bjam to Scons, keeping it working=20
>> at
>> all times
>> - port everything from the ground up to Python until eventually it is
>> complete
>>
>> Just out of curiosity, I tried to port generators.jam to Python. It's
>> 40k worth of bjam code and I did it in afternoon, including some unit
>> tests and adapting the comments do Python doc strings.
>
> Nifty!
>
> Note that the algorithm used by generators.jam for calculating
> intermediate targets and build steps is not really general at all, but
> tuned for a specific case (Qt-related, I think -- .whl, .dlp, and .wd
> files are involved).
>
> Related messages are in the threads at: http://tinyurl.com/68xc4,
> http://tinyurl.com/5axa5
>
> As a result I created a prototype of how the generator search *ought*=20
> to
> proceed... and it's written in Python! It lives in
> tools/build/v2/generators_prototype.py
>
> Unfortunately it's not trivial, but I think it represents the "right"
> logic from a conceptual standpoint. It has some significant comments,
> but could definitely use more.

I couldn't find tools/build/v2/generators_prototype.py in the CVS. Can=20
you please point me to the right place?
Anyway, the discussion is very interesting and I think we should=20
eventually include such a tool. Still, if we do go for it, I'd suggest=20
that the first go should use the BB codebase as it is to avoid adding=20
more trouble. Then, we could enhance the generators and perhaps other=20
parts of the code could take more advantage of Python.

>> So, my feeling is that it would be more efficient to go the latter=20
>> way.
>
> Great!
>
>> I'm willing to invest a reasonable amount of time in this effort,
>> should others be interested in collaborating too.
>> Comments are welcome.
>
> I'm interested also, provided that we agree to more fully document the
> code as we convert it. I think it would be fun to do and it would help
> us to get a grip on the codebase.

Excellent! I all for enhancing the documentation too, and for adding=20
unit tests.

Regards,

Pedro --Apple-Mail-11-161482358 Content-Transfer-Encoding: quoted-printable
Content-Type: text/enriched;
charset=ISO-8859-1

Hello Dave,

Em 13 Dec 2004, =E0s 18:49, David Abrahams escreveu:

<excerpt>

Pedro Ferreira wrote:

<excerpt>Dear all,

I recently told Volodya I was going to spend some time experimenting=20

the integration between Boost.Build and SCons.

Unfortunately, I have been flooded with urgent tasks so I still
haven't=20

managed to spend more than an hour or so doing this.

I see interest from many of us in pursuing such an integration so I=20

think we need to debate where are we going to and which path should we=20=

choose.

My vision is:

- create a 100% Python Boost.Build library on top of SCons

- create a parser for the bjam language

</excerpt>

That's a good goal, but I think the steps outlined by Volodya in

http://thread.gmane.org/gmane.comp.programming.tools.scons.user/5667
are

the right way to get there.

</excerpt>

For reference, this is what Volodya proposed by then:

<color><param>FFFF,64E4,499E</param>1. Initially, bjam will invoke
SCons at the lowest level (creating SCons Nodes instead of bjam's
targets).

2. If that works out OK, we'd need to consider how to mix Python and
jam language. Say, so that one could write new tool in Python.

3. After that, we can gradually move Boost.Build code to Python. Given
that the languages are very similar, this should not be very hard.

</color>

My first feeling was that it was indeed the best way to proceed and I
implemented a prototype of 1.:

1. change build-system.jam to dump a Python map of all virtual-targets
and their relevant properties

2. create a Python script that reads the map an creates the
appropriate SCons nodes

3. map BB properties into SCons environment settings

This works fine when using simple properties, such as <<define>,
<<include> and <<linkflags>, which are readily supported by SCons
enviroments.

But, in order to map things like <<optimization> or <<debug-symbols>,
you need to know which compiler you are using.

And, there must be a mapping between BB target type and the
appropriate Environment method to call in order to create a target.
And you must choose env.StaticObject or env.SharedObject depending on
the target being a DLL or a LIB.

So, I got to the conclusion that we would be better off moving
directly to Python.

<excerpt><excerpt>This would allow people to use Jamfiles but would
provide a Python=20

interface to Boost.Build, which would open way to a myriad of tools,=20

such as graphical browsers/creators, builder GUIs, etc.

The integration with SCons can be done:

- at the SCons.Environment level, calling builder methods to create BB=20=

virtual targets

- at the lower level Node ADT interface

I really don't like SCons Enviroments because they are too cluttered=20

with functions, are too unwieldy, load all tools by default, etc.

So I prefer the second approach: it provides all the functionality
bjam=20

provides (and more) and fits very well with BB's architecture.

</excerpt>

I had the same feeling about Environments.

<excerpt>Regarding the approach, I see two possibilities:

- do a progressive transition from bjam to Scons, keeping it working
at=20

all times

- port everything from the ground up to Python until eventually it is=20

complete

Just out of curiosity, I tried to port generators.jam to Python. It's=20

40k worth of bjam code and I did it in afternoon, including some unit=20

tests and adapting the comments do Python doc strings.

</excerpt>

Nifty!

Note that the algorithm used by generators.jam for calculating

intermediate targets and build steps is not really general at all, but

tuned for a specific case (Qt-related, I think -- .whl, .dlp, and .wd

files are involved).

Related messages are in the threads at: http://tinyurl.com/68xc4,

http://tinyurl.com/5axa5

As a result I created a prototype of how the generator search *ought*
to

proceed... and it's written in Python! It lives in

tools/build/v2/generators_prototype.py

Unfortunately it's not trivial, but I think it represents the "right"

logic from a conceptual standpoint. It has some significant comments,

but could definitely use more.

</excerpt>

I couldn't find tools/build/v2/generators_prototype.py in the CVS.=20
Can you please point me to the right place?

Anyway, the discussion is very interesting and I think we should
eventually include such a tool. Still, if we do go for it, I'd suggest
that the first go should use the BB codebase as it is to avoid adding
more trouble. Then, we could enhance the generators and perhaps other
parts of the code could take more advantage of Python.

<excerpt><excerpt>So, my feeling is that it would be more efficient to
go the latter way.

</excerpt>

Great!

<excerpt>I'm willing to invest a reasonable amount of time in this
effort,=20

should others be interested in collaborating too.

Comments are welcome.

</excerpt>

I'm interested also, provided that we agree to more fully document the

code as we convert it. I think it would be fun to do and it would help

us to get a grip on the codebase.

</excerpt>

Excellent! I all for enhancing the documentation too, and for adding
unit tests.

Regards,

Pedro=
 --Apple-Mail-11-161482358--