Re: [boost] [boost.process] 0.6 Redesign

Subject: Re: [boost] [boost.process] 0.6 Redesign
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2016-04-19 06:55:58

On 19 Apr 2016 at 1:06, Klemens Morgenstern wrote:

> > 2. You should completely eliminate all synchronous i/o as that is
> > also fundamentally broken for speaking to child processes. Everything
> > needs to be async 100% of the time, it's the only sane design choice
> > [2]. You can absolutely present publicly a device which appears to
> > quack and waddle like a synchronous pipe for compatibility purposes,
> > indeed AFIO v2 presents asynchronous i/o devices as synchronous ones
> > if you use the synchronous APIs even though underneath the i/o
> > service's run() loop gets pumped during i/o blocks. But underneath it
> > needs to be 100% async, and therefore probably ASIO.
> 
> Actually I did not look at boost.afio, which might be a good idea. 

v2 is a *very* different design to v1. The peer review here last 
summer was heard.

v2's async facilities are quite minimal mainly because v2 also has no 
knowledge of threads nor memory management, so no shared_ptr. Indeed 
v2 doesn't even throw exceptions! It also uses gsl::span<T> 
throughout including for scatter gather buffers, so it's a bit more 
C++ 1z-y in look and feel.

v2 has a native_handle_type for the OS handle type, it's a very thin 
and dumb value type. An afio::handle manages a native_handle_type, 
basically closes it on destruction. There are increasing refinements 
of afio::handle into afio::io_handle, file_handle, async_file_handle 
and so on.

An io_handle works as expected with a pipe or socket. If the 
native_handle_type is tagged as async/non-blocking, the synchronous 
read() and write() virtual functions will hang around/pump the i/o 
dispatch queue until the synchronous i/o completes, thereby quacking 
and waddling like a synchronous i/o operation.

In this it's not dissimilar to ASIO, but AFIO is far lighter weight 
than ASIO. We don't bother with IOCP on Windows for example, it's too 
heavyweight. v2 aims for hundreds of opcodes overhead over the system 
calls, no more.

> Currently you only have the async_pipe representation, which wraps 
> around a either ordinary posix-pipe or a named pipe on windows.
> Now I do not think, that I need to make everything async, because I can 
> think of enough scenarios where this is a complete overkill. For 
> example, I might just want to pipe from one process to another:

I meant internal implementation needs to be async. It's unavoidable 
unless your design is to be fundamentally broken.

> If I understand what you say here correctly, that this is what I 
> originally planned. But I decided that against that, because it is much 
> easier the current way and you can store all needed information in a 
> simple child. And actually you can detach the child.

In this you enforce a design choice and consequence onto your users. 
For example you'll force your users to use smart pointers and memory 
allocation to stop the default behaviour you imposed on them. The way 
I suggested does not have these problems.

> Well again: that would require the usage of asio all the time, I don't 
> think that's very sensible.

As I mentioned, the biggest reason to choose a child process 
management library rather than throwing one together of your own is 
that someone has solved the stdout/stderr deadlock problem for me. If 
you don't solve that problem for me, it's faster and easier to bodge 
together my own process library because your library delivers nothing 
of value to me.

> Ok now I am confused: you want me to make everything asio, so that 
> you'll always need boost.asio, but you want to eliminate the dependency 
> on boost.fusion? Why? That's definitely not going to happen, since I do 
> a lot of meta-programming there, this would become too much work for a 
> library which clearly needs other boost libraries. The only way I see 
> that happening is, when I propose a similar library for the C++ 
> Standard, and that won't happen very soon...

There is zero need for metaprogramming in a child process library. If 
you think you need it, you have the wrong design.

ASIO is not like Fusion. ASIO is entering the ISO C++ standard. That 
makes it one day not-a-dependency.

I *personally* think you should use ASIO rather than duplicating 
work. But you don't have to, feel free to rip bits out of AFIO v2 
into your own solution if you like. Mine is certainly much easier to 
borrow from.

> The async stuff is second, and is a set of features, extending it. At 
> least that's the reason I need a process library, which is the reason 
> I'm working on that.

I don't think you understand the stdout/stderr deadlock problem. Look 
into Python's subprocess.communicate().

Python tried to use non-async i/o in its popen() for years. It never 
worked right, because it can't. Async i/o is unavoidable if child i/o 
is to be reliable [1]. It's a non-negotiable.

Niall

[1]: Strictly speaking background threads to pump stdout and stderr 
also works just fine. I've used that a few times in a pinch as it's 
quicker to deploy working code than doing proper async i/o.

-- 
ned Productions Limited Consulting
http://www.nedproductions.biz/ 
http://ie.linkedin.com/in/nialldouglas/