boost parallel build, will it get better?
Hi, i was wondering if there is a possibility boost (bjam?) will be supporting parallel builds better in the future, i was testing building with -j5 (-j5 is about the sweet spot at which distcc creates enough load on the helping XP3200 for this mac mini i work on) and boost was no longer able to successfully build, something it did before readding support for the user selected number of jobs. Basically im wondering what to tell users who start to notice we do not support their -jN settings ("bear with us it'll work in future" vs "we do not know if it will ever work") after having received notice that supporting make options might not be taken well by bjam (and being able to confirm this with distcc), while at the same time wishing i could take advantage of distcc myself. Is this something of concern to boost, or will it most likely remain at the state as it is in the 1.32.0 release? Thanks in advance for any information you can provide. Daniel
Daniel Goller <morfic@gentoo.org> writes:
Hi,
i was wondering if there is a possibility boost (bjam?) will be supporting parallel builds better in the future
This really belongs on the Jamboost list: http://www.boost.org/more/mailing_lists.htm#jamboost. I'm not sure any of us knew there was a problem with parallel builds.
i was testing building with -j5 (-j5 is about the sweet spot at which distcc creates enough load on the helping XP3200 for this mac mini i work on)
distcc? XP3200? mac mini? I probably don't really need to know what these mean, but I'm curious.
and boost was no longer able to successfully build,
What was the result? What errors did you see? Do you have any reason to believe the problem was connected to the -j option?
something it did before readding support for the user selected number of jobs.
Somebody re-added support for the -j option? When did support for -j ever go away? Or am I misunderstanding you?
Basically im wondering what to tell users who start to notice we do not support their -jN settings ("bear with us it'll work in future" vs "we do not know if it will ever work")
How about "we do not know that it's even broken?" ;-)
after having received notice that supporting make options might not be taken well by bjam (and being able to confirm this with distcc), while at the same time wishing i could take advantage of distcc myself.
Is this something of concern to boost, or will it most likely remain at the state as it is in the 1.32.0 release?
We're definitely concerned with making sure that parallel builds work, especially if there's currently a problem. That said, as soon as BBv2 replaces BBv1 (soon!) we are probably going to start concentrating on transitioning from the use of bjam's build engine to that of Scons, which does support parallel builds very nicely. At that point, responsibility for support of parallelism will be on the shoulders of the Scons guys ;-) HTH, -- Dave Abrahams Boost Consulting www.boost-consulting.com
David Abrahams wrote:
Daniel Goller <morfic@gentoo.org> writes:
Hi,
i was wondering if there is a possibility boost (bjam?) will be supporting parallel builds better in the future
This really belongs on the Jamboost list: http://www.boost.org/more/mailing_lists.htm#jamboost. I'm not sure any of us knew there was a problem with parallel builds.
i was testing building with -j5 (-j5 is about the sweet spot at which distcc creates enough load on the helping XP3200 for this mac mini i work on)
distcc? XP3200? mac mini?
I probably don't really need to know what these mean, but I'm curious. <snip>
Distcc is a distributed version of GCC: <http://distcc.samba.org/>. Useful if you have a bunch of mostly idle computers hanging around, I believe. "XP3200" is perhaps a very condensed version of "PC with an Athlon XP 3200+ processor". "Mac mini" is a new product of, wossname, that computer company named after some kind of fruit. Ben.
Daniel Goller wrote:
i was wondering if there is a possibility boost (bjam?) will be supporting parallel builds better in the future, i was testing building with -j5 (-j5 is about the sweet spot at which distcc creates enough load on the helping XP3200 for this mac mini i work on) and boost was no longer able to successfully build, something it did before readding support for the user selected number of jobs.
The only problem *I* know about with -j is that it sometimes misses dependencies on generated headers (with Boost.Build V2, V1 does not handle generated headers well itself). But this should not affect Boost. Can you specify exactly what kind of problems you get? Note that for distcc to work reliably, you need to have exactly the same version of the compiler on all hosts, and preferrably a stable version. I used to get segfaults otherwise. - Volodya
Hi ! On Tuesday 22 March 2005 08:09, Vladimir Prus wrote:
Daniel Goller wrote:
i was wondering if there is a possibility boost (bjam?) will be supporting parallel builds better in the future, i was testing building with -j5 (-j5 is about the sweet spot at which distcc creates enough load on the helping XP3200 for this mac mini i work on) and boost was no longer able to successfully build, something it did before readding support for the user selected number of jobs.
The only problem *I* know about with -j is that it sometimes misses dependencies on generated headers (with Boost.Build V2, V1 does not handle generated headers well itself). But this should not affect Boost.
Ah. This could possibly affect me ;-) But I'm very happy with the current -j support. I normally do bjam --v2 -j18 using SuSE's icecream and our Linux compile farm. See http://wiki.kde.org/tiki-index.php?page=icecream for more details on icecream ;-)
Can you specify exactly what kind of problems you get? Note that for distcc to work reliably, you need to have exactly the same version of the compiler on all hosts, and preferrably a stable version. I used to get segfaults otherwise.
Icecream distributes the compiler binary and libs using a tar.bz2 archive. This prevents these problems. And I even do cross-compiles for AMD-64 using debian's gcc-3.4.4 ;-) The only problem with -j18 is the link stage when linking static apps. It is, let's say unfortunate to have all my apps linking at the same time... But compiling works like a charm... Yours, Jürgen -- * Dipl.-Math. Jürgen Hunold ! Institut für Verkehrswesen, Eisenbahnbau * voice: ++49 511 762-2529 ! und -betrieb, Universität Hannover * fax : ++49 511 762-3001 ! Appelstrasse 9a, D-30167 Hannover * hunold@ive.uni-hannover.de ! www.ive.uni-hannover.de
J?rgen Hunold wrote:
Hi !
On Tuesday 22 March 2005 08:09, Vladimir Prus wrote:
Daniel Goller wrote:
i was wondering if there is a possibility boost (bjam?) will be supporting parallel builds better in the future, i was testing building with -j5 (-j5 is about the sweet spot at which distcc creates enough load on the helping XP3200 for this mac mini i work on) and boost was no longer able to successfully build, something it did before readding support for the user selected number of jobs.
The only problem *I* know about with -j is that it sometimes misses dependencies on generated headers (with Boost.Build V2, V1 does not handle generated headers well itself). But this should not affect Boost.
Ah. This could possibly affect me ;-)
But I'm very happy with the current -j support. I normally do bjam --v2 -j18 using SuSE's icecream and our Linux compile farm. See http://wiki.kde.org/tiki-index.php?page=icecream for more details on icecream ;-)
Can you specify exactly what kind of problems you get? Note that for distcc to work reliably, you need to have exactly the same version of the compiler on all hosts, and preferrably a stable version. I used to get segfaults otherwise.
Icecream distributes the compiler binary and libs using a tar.bz2 archive. This prevents these problems.
And I even do cross-compiles for AMD-64 using debian's gcc-3.4.4 ;-)
The only problem with -j18 is the link stage when linking static apps. It is, let's say unfortunate to have all my apps linking at the same time...
As I say in some other thread, there is a mechanism which allows to serialize linking. Shame on me for not activating it for gcc.link. I've just committed the relevant change -- can you verify that it works? - Volodya
participants (5)
-
Ben Hutchings -
Daniel Goller -
David Abrahams -
Jürgen Hunold -
Vladimir Prus