Re: [boost] Flow-based programming library for Boost?

Subject: Re: [boost] Flow-based programming library for Boost?
From: Julian Gonggrijp (j.gonggrijp_at_[hidden])
Date: 2012-12-15 11:36:59

I wrote:

> I'm certainly interested in a Boost library that offers efficient 
> flow-based multithreading with an interface that is as simple as 
> possible. In fact I have some ideas of my own which I would gladly 
> present to the mailing list when I find more time.

In the meanwhile I would like to refer to FastFlow [1], which as far 
as I can tell hasn't been mentioned yet in this thread. It's similar 
to DSPatch with respect to OOD, but it offers two particular features 
that I think are more powerful and would be more fitting for a Boost 
submission.

The first is the use of producer-consumer queues as the primary means 
to transfer data between two threads. I believe this is really THE way 
to go, because the data are buffered and neither producer nor consumer 
will need to wait for the other unless the buffer is full or empty.

Note that in a sense the DSPatch wire is a special case of the 
producer-consumer queue where the buffer has only one slot, so it's 
always either full or empty, forcing the producer and consumer 
components to work strictly in tandem.

The other feature is the use of dedicated threads to create one-to-
many and many-to-one connections out of only pure one-to-one 
connections. The FastFlow authors refer to these as "emitters" and 
"collectors", respectively. This simplifies the implementation and 
improves throughput in scenarios where a thread has multiple inputs or 
multiple outputs. Moreover, in combination with generic design this 
would allow for some nifty off-the-shelf tools such as tuple 
zipping/unzipping, distributing/interlacing, etcetera. Topher Cooper 
independently came up with a similar idea in [2].

Nonetheless FastFlow still isn't my "dream library". One reason for 
that is that it isn't generic. Another reason is that to my taste it 
behaves too much like a framework instead of like a toolkit. One 
symptom of that is the requirement on users to derive from a class 
(like in DSPatch) and to override methods. Finally, while FastFlow 
makes an admirable attempt at providing abstractions for complex high-
level use cases ("skeletons"), doing simple things is not simple 
(thankfully adopting this way of talking about simplicity from [2]).

Hope this will help the discussion!

-Julian

_______

[1] http://calvados.di.unipi.it/dokuwiki/doku.php?id=ffnamespace:about
[2] http://lists.boost.org/Archives/boost/2012/12/199333.php