Subject: Re: [boost] [boost.process] 0.6 Alpha
From: Klemens Morgenstern (klemens.morgenstern_at_[hidden])
Date: 2016-06-17 05:45:34
Am 17.06.2016 um 10:36 schrieb Damien Buhl:
> On 16/06/2016 18:02, Klemens Morgenstern wrote:
>>> Hi Klemens,
>>> We have been using the Boost.Process state from Boris Schäling for more
>>> than 3 year in productive use on linux embedded devices, and one thing
>>> that we had to change is to use vfork instead of the fork function.
>>> The problem with the fork + execve is that it produce a RAM copy of the
>>> process page, while vfork+execve just don't. And when the parent process
>>> launching child is pretty big, imagine a jvm using some jni library
>>> using Boost.Process... it duplicates the whole Jvm im RAM to then
>>> deallocate all at the moment of execve. vfork doesn't have this issue on
>>> Without the use of vfork, you end up in situations where you get the
>>> following error: `boost::process::detail::posix_start: fork(2) failed:
>>> Cannot allocate memory`
>>> I think changing from fork to vfork is not much and brings alot of
>>> advantages, but one must be aware that at_fork handler won't be called.
>>> But this is not important as fork is used to do execve afterwards in
>> Hi Damien,
>> I appreciate your problem, but I am not sure it's that easy. There's
>> on major problem: vfork is removed from the posix standard as of
>> POSIX.1-2008. And I'm trying to conform to posix, not linux.
>> So I guess, I wouldn´t use vfork as default, but it might be possible
>> to add a property, which will cause the library to use that. I.e. you
>> boost::process::child c("java.exe", "overhead.jar",
>> But I'd need to be able to check if vfork is available, so I can
>> disable the property if not.
>> Would that be sufficient for your problem?
> Naturally if the code still compiles on windows even though I'm using
> boost::process::posix::use_vfork(_if_possible) then yes for me all will
> be fine.
> But on the other hand from a library design point of view, shouldn't the
> library have the best smart default in terms of performance and overhead
> on a given platform ? Instead of having a flag telling : please do it
> the same but efficiently ? Because on linux vfork is nothing but
> obsoleted and for a scenario of using execve looks better to me.
Again: obsolete and now removed in the posix-standard and that's what I
ought to go with. Please note, that I'm trying to provide two platforms:
Posix & Windows. Not Linux & Windows. Thereby I want to provide the most
common way for both platforms; and though I really appreciate your
scenario, I would not consider it the common way.
I think you underestimate the dangers of vfork; because you know, two
processes sharing memory (including the stack!) can be rather fun.
There's a reason it was removed and so it will be optional in
It would be a platform extension, so no, this would NOT compile with
windows. I thought about turning the platform extensions into NOPs ,
but that's just too weird - especially since things like signal(SIGCHLD,
...) are also provided. There you should rather use the preprocessor and
put an #ifdef there - then it's obvious what you're doing. Also I'd need
a #define to know whether vfork is available, that I would probably
provide. So you could then write something like that:
I guess I can check that via 'CLONE_VFORK'.
>> On 17/06/2016 01:21, Gavin Lambert wrote:
>>> Is this specifically for NOMMU linux?
>>> MMU architectures shouldn't have this issue, as fork does a
>>> shared-memory-copy-on-write mapping so that the pages aren't actually
>>> duplicated unless written to, and the subsequent exec* unmaps the
>>> pages so this never happens (other than a bit of stack).
>>> NOMMU architectures don't support those kinds of mappings, so have to
>>> be rewritten to use vfork instead (which is generally unsafe unless
>>> immediately followed by exec*, unless you know what you're doing).
>>> Although the last time that I played with NOMMU, fork used to just
>>> fail; perhaps it's been changed to make copies of all the pages
>>> instead? If so, that would indeed be problematic for large parent
> Hi Gavin ;)
> It's with an MMU but the parent process on this product is a
> memory-hungry-monster-jvm and the fork fails.
It seems a bit strange to me, that you would use boost.process here,
instead of - you know - java.io.process. But I guess you want
performance and your java-developers want a job, so JNI is the way to go?
> I can be wrong, and it looks like you know more from what's happening
> there, but as long as I can remember the issue arose due to virtual
> memory commit, because fork on linux even though is optimized to not
> copy the whole process pages but to do copy-on-write, still needs to
> commit virtual memory, and overcommitment is not always allowed, or by
> default is heuristically allowed.
That's probably right, because elsewise you'd get an error in the forked
process and there would be no way of getting to know it.
Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk