Re: [Boost-build] Programmatic Job Count

Subject: Re: [Boost-build] Programmatic Job Count
From: Johan Nilsson (r.johan.nilsson_at_[hidden])
Date: 2008-11-21 03:51:11

Phillip Seaver wrote:
> Vladimir Prus wrote:
>> On Wednesday 19 November 2008 11:16:00 you wrote:
>>
>>> On Tuesday 18 November 2008 21:04:38 Brad Howes wrote:
>>>
>>>> Is there any way to tell Boost how many jobs to run in parallel
>>>> from within a Jamfile, a programmatic equivalent to '-j N'?
>>>>
>>> It is not possible, but hopefully will be possible later today :-)
>>>
>>
>> Brad,
>>
>> Please update and rebuild Boost.Jam. Then, put the following in your
>> Jamroot:
>>
>> import modules ;
>> modules.poke : PARALLELISM : 4 ;
>>
>> and it should have the same effect as -j4 on command line.
>> Presently, the PARALLELISM variable overrides -j option. Soonish,
>> I'll make it so:
>>
>> - you can create file project-config.jam, containing:
>>
>> option jobs : 4 ;
>>
>> and have 4 jobs by default
>>
>> - you can use --jobs=10 on command like, overriding
>> project-config.jam
>>
>> Let me know if the Boost.Jam change works for you, and if the future
>> directions seems right,
>>
>> Thanks,
>> Volodya
>
> That's cool, Volodya.  Will that work from site-config.jam?  That
> seems
> a logical place to use it for those of us that use machines with
> different numbers of processors.  We have 2-, 4-, and 8-core machines
> that we regularly build on, so it would make more sense (for us) to
> make
> it machine-specific.

Not necessarily; on our build machines we're likely to compile unrelated 
Boost.Build projects concurrently, and would rather like to allocate e.g. 2 
execution units for one of the projects and 2 for the other.

Using project-config makes lots of sense for those of us that regularly work 
on several projects at the "same time". However, I agree that it should be 
possible to specify parallellism in site-config as well (or user-config, for 
that matter).

I would guess that Volodya has already thought about this (and perhaps 
described it in this mailing list), but the precedence for options set in 
config files should probably be (highest to lowest):

project-config
user-config
site-config

> One thing that I've thought of a few times related to this is a
> special
> value to "-j" that tells it to match the number of CPU's.  So, on my
> 4-core machine, it would be as if I had typed "-j4".  If I can use it
> from site-config.jam or user-config.jam, though, that idea is a little
> less needed.  :-)

I think that this could be a nice finishing touch. What about -j0 or "jobs : 
0" for using all available hardware parallellism (for the platforms where 
this can be detected automatically)?

/ Johan