Subject: Re: [boost] Flow-based programming library for Boost?
From: Julian Gonggrijp (j.gonggrijp_at_[hidden])
Date: 2012-12-12 07:57:00
Dave Abrahams wrote:
> If you want such a library to be truly efficient, you should be able to
> wire together the dataflows at compile-time and not pay any overhead for
> composing large computations out of smaller ones. The accumulators
> library uses something like a dataflow model, with the "wiring" done
> completely at compile-time.
Because of your remark I studied the Accumulators library, and I must
say it's very elegant indeed. Dependencies between computations are
resolved at compile-time (wow!) so they don't add unnecessary overhead
at run-time and everything will be executed in the most efficient
However, note the word "order". Boost.Accumulators was clearly
designed under the assumption of single-threaded operation and tightly
interdependent computations. I believe it might very well be the best
possible approach under those assumptions, but when the assumptions
don't apply other considerations may call for a different solution.
"Dataflow" is a very general term, but the people who advertise with
that word typically mean something more specific, i.e. lock-free
multithreading with message passing. The underlying assumptions are
more or less opposite to the assumptions in Boost.Accumulators:
computations can run concurrently without waiting for each other, as
long as they have input available and they can dispose of their output
somewhere. If computations can take place /at the same time/ it does
not really matter whether connections are made at compile-time or run-
time; /waiting/ is eliminated altogether so the building of
connections at run-time adds only constant overhead.
That constant overhead is worth it if accepting it helps you to avoid
other problems. Note that the "wires" built by Accumulators at
compile-time are /logical/ connections between /types/ that represent
operations. Message passing "wires" are /physical/ connections between
/threads/. Instantiating the latter kind of connection at compile-time
seems very hard, if not impossible.
Actually I think the Accumulators framework and lock-free message
passing are completely orthogonal. One could use a "dataflow" library
to connect threads at run-time while composing computations for the
individual threads at compile-time with Boost.Accumulators, and get
the best of both worlds.
I hope I'm making sense. Please let me know what you think.