Boost logo

Boost :

Subject: Re: [boost] [boost.process] 0.6 Redesign
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2016-04-19 06:55:58

On 19 Apr 2016 at 1:06, Klemens Morgenstern wrote:

> > 2. You should completely eliminate all synchronous i/o as that is
> > also fundamentally broken for speaking to child processes. Everything
> > needs to be async 100% of the time, it's the only sane design choice
> > [2]. You can absolutely present publicly a device which appears to
> > quack and waddle like a synchronous pipe for compatibility purposes,
> > indeed AFIO v2 presents asynchronous i/o devices as synchronous ones
> > if you use the synchronous APIs even though underneath the i/o
> > service's run() loop gets pumped during i/o blocks. But underneath it
> > needs to be 100% async, and therefore probably ASIO.
> Actually I did not look at boost.afio, which might be a good idea.

v2 is a *very* different design to v1. The peer review here last
summer was heard.

v2's async facilities are quite minimal mainly because v2 also has no
knowledge of threads nor memory management, so no shared_ptr. Indeed
v2 doesn't even throw exceptions! It also uses gsl::span<T>
throughout including for scatter gather buffers, so it's a bit more
C++ 1z-y in look and feel.

v2 has a native_handle_type for the OS handle type, it's a very thin
and dumb value type. An afio::handle manages a native_handle_type,
basically closes it on destruction. There are increasing refinements
of afio::handle into afio::io_handle, file_handle, async_file_handle
and so on.

An io_handle works as expected with a pipe or socket. If the
native_handle_type is tagged as async/non-blocking, the synchronous
read() and write() virtual functions will hang around/pump the i/o
dispatch queue until the synchronous i/o completes, thereby quacking
and waddling like a synchronous i/o operation.

In this it's not dissimilar to ASIO, but AFIO is far lighter weight
than ASIO. We don't bother with IOCP on Windows for example, it's too
heavyweight. v2 aims for hundreds of opcodes overhead over the system
calls, no more.

> Currently you only have the async_pipe representation, which wraps
> around a either ordinary posix-pipe or a named pipe on windows.
> Now I do not think, that I need to make everything async, because I can
> think of enough scenarios where this is a complete overkill. For
> example, I might just want to pipe from one process to another:

I meant internal implementation needs to be async. It's unavoidable
unless your design is to be fundamentally broken.

> If I understand what you say here correctly, that this is what I
> originally planned. But I decided that against that, because it is much
> easier the current way and you can store all needed information in a
> simple child. And actually you can detach the child.

In this you enforce a design choice and consequence onto your users.
For example you'll force your users to use smart pointers and memory
allocation to stop the default behaviour you imposed on them. The way
I suggested does not have these problems.

> Well again: that would require the usage of asio all the time, I don't
> think that's very sensible.

As I mentioned, the biggest reason to choose a child process
management library rather than throwing one together of your own is
that someone has solved the stdout/stderr deadlock problem for me. If
you don't solve that problem for me, it's faster and easier to bodge
together my own process library because your library delivers nothing
of value to me.

> Ok now I am confused: you want me to make everything asio, so that
> you'll always need boost.asio, but you want to eliminate the dependency
> on boost.fusion? Why? That's definitely not going to happen, since I do
> a lot of meta-programming there, this would become too much work for a
> library which clearly needs other boost libraries. The only way I see
> that happening is, when I propose a similar library for the C++
> Standard, and that won't happen very soon...

There is zero need for metaprogramming in a child process library. If
you think you need it, you have the wrong design.

ASIO is not like Fusion. ASIO is entering the ISO C++ standard. That
makes it one day not-a-dependency.

I *personally* think you should use ASIO rather than duplicating
work. But you don't have to, feel free to rip bits out of AFIO v2
into your own solution if you like. Mine is certainly much easier to
borrow from.

> The async stuff is second, and is a set of features, extending it. At
> least that's the reason I need a process library, which is the reason
> I'm working on that.

I don't think you understand the stdout/stderr deadlock problem. Look
into Python's subprocess.communicate().

Python tried to use non-async i/o in its popen() for years. It never
worked right, because it can't. Async i/o is unavoidable if child i/o
is to be reliable [1]. It's a non-negotiable.


[1]: Strictly speaking background threads to pump stdout and stderr
also works just fine. I've used that a few times in a pinch as it's
quicker to deploy working code than doing proper async i/o.

ned Productions Limited Consulting

Boost list run by bdawes at, gregod at, cpdaniel at, john at