Re: [boost] [thread, async] Mini design review requested for a generic extensible Boost monadic continuations framework

Subject: Re: [boost] [thread, async] Mini design review requested for a generic extensible Boost monadic continuations framework
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2014-03-05 12:49:53

On 5 Mar 2014 at 13:54, Vicente J. Botet Escriba wrote:

> Niall, thanks for this more explanatory description of what you have in
> mind. I think that our positions are closer than I believed after our
> private exchanges. 

I suspected I repeatedly failed to explain myself well to you. I 
apologise.

Before I reply to you, I was just reading through Chris Kohlhoff's 
N3964 (Library Foundations for Asynchronous Operations, Revision 1) 
at http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n3964.pdf. 
I agree with his argument that a lower level callback API without 
futures would be wise for those who need such ultra low latency they 
can't afford malloc, but I am unsure if the C++ standard is the right 
place to specify such a machinery, even if it standardises 
Boost.ASIO.  

There is much of Boost.ASIO which *ought* to be standardised (e.g. I 
don't think Google's Executors proposal is worth much compared to 
standardising the core of ASIO as ASIO's core design is very solid, 
and very popular i.e. perfect for standardisation), but I must 
confess I don't think that this specific feature of ASIO i.e. the 
extensible model of asynchronous operations proposed in N3964 - is a 
good fit for standardisation.

> > * A new when_any(...) composes a future which signals when any of its
> > input futures signals. Inputs can be futures of heterogeneous types,
> > in which case a future<tuple<...>> is returned. To avoid the linear
> > scan on return, for a homogeneous types a when_any_swapped(...) swaps
> > the last item in the return value with the ready item. Note there is
> > no way in N3857 to avoid a when_any() linear scan when input types
> > are heterogenous (something I feel is a deficiency, all you need to
> > return is an index!).
> I guess that this point could be discuss in std-proposals ML.

Erm, I think you just need to convince Artur Laksberg there is a 
better way and you're good here. I know he has personal doubts 
himself on this design choice. I'm thinking maybe a visitor callable 
might work, so some callable<T> gets instantiated and called with the 
ready future.

> > std::future<int> &a;
> > boost::future<int> &b;
> > boost::afio::async_io_op &c;
> > std::future<std::tuple<int, int, std::shared_ptr<async_io_handle>>>
> > f=when_all(a, b, c);
>
> I have some doubts on who we can decide that the result of such an 
> operation would be
> 
> std::future<std::tuple<int, int, std::shared_ptr<async_io_handle>>>

In my proposed generic framework there is some metaprogramming which 
decides what future-like type decays into. It defaults to the return 
type of future_type::get(), but it absolutely can be overriden with a 
partial template specialisation.

> > * As I mentioned earlier, when_any() is suboptimal when called with
> > heterogenous types. I don't particularly like the when_any_swapped()
> > hack either - a better solution would solve when_any() of all kinds
> > optimally.
> Make a proposal on std-proposals then.

I think that the small number of actors involved in this tiny niche 
of C++17 is small enough that personal contact works better. I only 
posted a request for design review here because what I add to AFIO 
next is likely very useful to many others using Boost. To be honest, 
I had hoped for a more vociferous feedback given the number of people 
writing async object interop boilerplate. I guess they prefer writing 
boilerplate.

> > N3857 provides no easy method for a later .then() to inspect
> > preceding .then()'s and do something about them (e.g. rewinding until
> > it finds a valid open file handle rather than error states), other
> > than by declaring special wrapper lambdas to pass through state. I
> > personally think that is ugly.
> Here is when I disagree the most on your proposal. I think that 
> transactions are orthogonal to the future class and the user could 
> return whatever she wants from its continuations.
> Maybe you need to implement some kind of Monad State.

continuations::thenable<T, ...> *is* a monadic state. It accepts any 
set of heterogeneous types so long as they mix in the 
continuations::thenable<T, ...> type.

In other words, I am using the policy class which says "this type is 
future-like then()able" as the policy implementation class *as well*. 
I suppose most would consider that bad design and insist on a 
separate set of concept classes. If that is your criticism, I accept 
that as probably valid.

> > continuations::thenable_get_placeholder<-1> last;
> > auto ret=openedfile
> > 	.then(write({ "He", "ll", "o ", "Wo", "rl", "d\n" }, 0))
> > 	.then(if_error(last, [](future<...> f){ do something with last; }))
> > 	.then(sync());
> > 	.then(write({ "He", "ll", "o ", "Wo", "rl", "d\n" }, 12))
> > 	.then(if_error(last, [](thenable<...> t){ do something more with
> > everything; }))
> > 	.then(sync());
> Could you clarify what prevents you from doing this with the current 
> proposals?

The first if_error() tag could work with the current proposals. The 
second if_error() tag can see the entire thenable chain, and 
therefore could not work with the current proposals.

My point actually was that tag dispatched filters of state I think is 
a better, more generic, more powerful design than hard coded member 
functions.

> > Which treats the continuation as a functional transformation. In
> > other words, Vicente's .recover(R(exception_ptr)) becomes tag
> > dispatched instead of explicitly specified, so it's very easy for the
> > programmer to transform state according to state.
> I have been discussing something similar lastly on c++-parallel ML. I 
> was suggesting something as if_valued/if_unexpected.
> I have implemented it first on Boost.Expected.

I know. I was trying to mirror your design. No point reinventing the 
wheel.

> I have also proposed something like that could be used as
> 
> monad_error(f).when_valued(g).when_unexpected(h);
> 
> or as
> 
> monad_error(f) & g | h;
> 
> I have a prototype implementation on one of the examples in Boost.Expected.

I don't consider myself competent to be hugely useful in designing a 
generic C++ monadic framework - I don't believe myself good enough at 
language design.

> > 2. R(..., continuations::thenable_get_placeholder<idx>, ...) inserts
> > an item from the continuation chain into the call.
> > thenable_get_placeholder can be fed a negative number to wrap from
> > the bottom, so thenable_get_placeholder<-1> is the most recently
> > preceding operation. Therefore R(prec_future_type<T>) from above is
> > actually implemented as
> > R(continuations::thenable_get_placeholder<-1>). This can be used to
> > insert error processing filters as illustrated earlier which can view
> > a preceding chunk of chain at once, and act.
> I don't understand this completely, but I suspect that what you are 
> proposing is some variation of Monad state and an adaptor that extracts 
> part of the state.

Correct.

> > 3. R(thenable<...> &&) which gives a continuation access to the
> > entire continuation chain, just for those people who really need it.
> > You saw this in the example above.
> Again, this is more a kind of monad state that accumulates the results 
> of each continuation.

Correct. thenable<T, ...> is a monad state as well as a policy class 
indicating a type is capable of being .then()ed.

> > * when_all() and when_any() gain overloads for thenable<...>. These
> > simply convert the tuple taken out of a thenable chain into a future
> > with the results - AFIO will implement this generically using its
> > closure engine, but a more intelligent solution might decompose all
> > inputs into HANDLEs and do an asynchronous WaitForMultipleObjects()
> > for example.
> As I said above, why the result will be a future?

Oh, purely because N3857 says so, and I am not trying to change 
N3857, simply extend it such that it is much more useful than now. Do 
remember a future is thenable<T, ...> under this proposal, so under 
my proposal a when_all() or when_any() simply mean that a given 
continuation chain is to have its monadic state collapsed and a new 
monadic state started.

> > Continuation monads are a very limited specialisation of general
> > monads, especially under the tight constraints of the N3857
> > requirements, so reuse of any Boost.Functional/Monads I suspect would
> > be minimal.
> I think that the Boost.Functional/Monads framework I had in mind could 
> be used for your use case. All you need to do is to define a specialized 
> Monad, that in your case is close to a Monad State.

Absolutely correct, and I would expect any code I write for AFIO now 
will be replaced later with an implementation written using your 
Boost.Functional/Monads framework. Consider my current proposal as a 
learning exercise/temporary stopgap code.

> I have no problem in proposing a different interface as IIUC your 
> proposal, the user would need to wrap the std::future, boost::expected, 
> ... or whatever AFIO monad or use specific 
> monads::when_all/monads::when_any function.

Correct. I mentioned how one would go about marking up future-like 
types with metaprogramming in my OP. Anyone can extend the 
metaprogramming with their own types.

> I hope that my comments would help you to clarify my discrepancies about 
> your design. Resuming,
> I think that you must define your own monad state that could be seen as 
> any other monad using the functional/monads framework.

Once your Boost.Functional/Monads framework is available, more than 
happy to do so. I hate writing and particularly maintaining 
boilerplate.

> > Thoughts regarding the design much appreciated! My thanks in advance.
> >
> I will inspect your code in order to see if I've understood your needs 
> as soon as I have enough time.

Thanks Vicente. I deeply appreciate your experience on this.

Niall

-- 
Currently unemployed and looking for work in Ireland.
Work Portfolio: http://careers.stackoverflow.com/nialldouglas/