Subject: Re: [boost] [review] Dataflow Library
From: Stjepan Rajko (stipe_at_[hidden])
Date: 2008-09-05 02:44:16
On Thu, Sep 4, 2008 at 8:24 PM, Hervé Brönnimann
> Stjepan: I'm just curious, because my wife uses such a dataflow program for
> interactive music performance.
> The program is called MAX and is distributed by a company named Cyclic74
I really appreciate you bringing up MAX - it was actually my exposure
to MAX that eventually lead me to write the Dataflow library. I had
previously seen dataflow programming in LabVIEW, but it wasn't until
MAX that I realized how easily people (with no formal programming
training) did really cool things with such an environment. It lead me
to experiment with the dataflow paradigm within C++, and eventually
realized that Boost.Signals is a good way of connecting components
together. That's how the Dataflow.Signals layer got started.
> One of the worse quirks when flowing the data (in a push mode, I guess) is
> that components are processed in (get this) geometric order, from left to
> right (breaking ties with top to bottom, I think). I guess it's important
> to know what the order i, when you allow feedback loops (cycles). But the
> left-to-right has been a never-ending source of bugs, in my wife's
> experience :-)
> Which prompts me to ask:
> - can you create push networks with cycles?
Yes, as long as the components are designed in such a way that doesn't
propagate the signal in an infinitely recursive loop. For example,
MAX solves some situations with cycles using a convention - signals
received on the leftmost inlet are typically processed and propagated,
while those received on other inlets are just stored and not
propagated. Hence, connecting something to a non-leftmost inlet can
break the infinite recursion in cycles.
Another way of dealing with cycles is using threading (with some
components / sets of components executing in their own threads). The
example that was developed at the end of the following thread would
allow cycles through the use of threading:
I hope to expand on that example soon.
> - in the presence of cycles, how is the signal flowing to the data sinks?
In the case of the above threading example, at some point of the cycle
a component would submit a task to a thread pool corresponding to the
signal call (all data communication in the Dataflow.Signals layer
happens through Boost.Signals).
> - in the absence of cycles, do you process the components in topological
> sorting order? or is the evaluation order arbitrary?
Currently, arbitrary. In practice, I believe the signals get sent out
according to the order in which the consumers were connected to the
producer, but I don't think that Boost.Signals guarantees that.
Boost.Signals does have a way of ordering signals:
... Unfortunatelly, Dataflow.Signals doesn't take advantage of it
(yet). The library currently offers no way of customizing a
connection (e.g., to specify the signaling order), which is a serious
limitation that I need to fix.
> Surely I could deduce that from your docs if I had read carefully the
> interface of the components, but why not discuss it up front? I'm lacking a
> general high-level view of what I can and can't do with this library.
This review is helping me tremendously in understanding which parts of
the documentation are more useful than others, and what is missing. I
think the docs are definitely going to get restructured :-)
To answer your question though, I think the most useful thing about
the Dataflow.Signals layer (this layer is the focus of the review) is
that it provides tools to implement components that can be used in a
dataflow network - both specific ones (e.g., something that generates
a particular type of image), and generic ones (something that can be
used with any signal signature / any number of arguments, e.g., a
component that doubles each of the arguments it receives before
passing them on). It also makes connecting components easier, IMO.
As far as applications where a library like this might be useful,
Giovanni Piero Deretta mentioned a few examples earlier in this
thread. The specific domains would be determined by what component
one has available / is willing to develop.
> A nice feature of Max/MSP though, is that you can create a "patch" which
> recursively acts as an atomic component, and can be compiled separately, so
> you can create libraries of patches some of which are opaque to enforce
> intellectual property (not that I approve, but...). Is there an abstract
> base class for component? This would enable one conceivably to distribute
> one's own components as DLLs (i.e., not source).
Yes - the Dataflow.Blueprint layer offers such functionality (but note
that this layer is still very much a prototype). As you're familiar
with MAX, you might find interesting a sample visual editor I
developed for the Dataflow library:
If you haven't seen them, there are some videos of the editor in action:
> These are the ones I'm more familiar with. I just found the WikiPedia page
> on my own (http://en.wikipedia.org/wiki/Dataflow_programming) and *then*
> (only then) noticed the link in your description (second section of
> Introduction page documentation). It's way too discrete imho. Bring it
> out, write a short paragraph about some known examples, fire up the
> imagination, something. As it is, it's a really steep curve in the
> introduction (which is an introduction to your library, but not to the
OK, I can try to improve that part of the documentation.
> Beyond that, I've just started browsing, but I was a bit put off by the
> steepness of the documentation and the use of operator overloading. It
> would really benefit from a gentler introduction that is not based on the
> existing library but on the general idea of dataflow programming. It gets
> more concrete in the examples, but that's too late, I've already been
> hopelessly confused by that point :)
Oh oh :-( I was hoping that the "Dataflow programming in C++" section
would serve as a gentler introduction, but perhaps it does jump into
the library rather quickly and steeply. I am curious, what did you
find off-putting about the operator overloading? I am finding this to
be a rather contentious issue.
Thank you for your feedback,
Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk