Re: [boost] [review] Dataflow Library

Subject: Re: [boost] [review] Dataflow Library
From: Stjepan Rajko (stipe_at_[hidden])
Date: 2008-09-05 14:05:00

On Fri, Sep 5, 2008 at 4:30 AM, Phil Endecott
<spam_from_boost_dev_at_[hidden]> wrote:
> Stjepan Rajko wrote:
>>
>> I really appreciate you bringing up MAX - it was actually my exposure
>> to MAX that eventually lead me to write the Dataflow library. I had
>> previously seen dataflow programming in LabVIEW, but it wasn't until
>> MAX that I realized how easily people (with no formal programming
>> training) did really cool things with such an environment.
>
> This is some very interesting rationale that you hadn't previously shared
> with us ;-)
>
> A question that had been forming in my mind was, "Who is this library for?".
>  Do you see this library useful primarily for people with no (or little)
> formal programming training?

Thanks for bringing this up - it is a very important piece of
information that I completely neglected to address in the
documentation.  At it's current state, I think the library is more
geared towards people that are designing plugin-based Dataflow systems
(like MAX or LabVIEW), or developing an application / library for
which the dataflow paradigm is particularly suitable.  In the context
of Dataflow.Signals, that means providing facilities for programmers
to create new components, and providing different ways in which the
components can be connected (e.g., through GUI, using operators, using
the connect function, using the bind_mem_fn helper).

Once someone develops a reasonably complete set of components for a
particular domain, *then* a person with less programming experience
could use those components to create their own applications, assuming
they are also comfortable with whatever facility for connecting is
given to them (GUI vs. connecting in C++).

> If so, do you really think that Boost, or for
> that matter C++, is the right starting point?  Or perhaps you believe that
> the benefits that those people see would also apply to mainstream software
> developers?  Are you hoping that "regular C++ programmers like us" will
> start to use the dataflow style with the help of the library, or is there an
> existing body of dataflow programmers who currently use some other language
> who can be "converted"?
>

I have a few points related to these questions.  I can tell you from
my personal experience as a "regular C++ programmer" that, after I
started playing with MAX, I found that way of programming (meaning
dataflow, not necessarily needing a GUI editor) very well suited for
certain kinds of applications.  I worked with signal processing quite
a bit (in C++), and after seeing MAX found myself trying to implement
them in a dataflow sort of ways - developing components, and then
connecting them.  I came up with a precursor to Dataflow.Signals, and
found it very useful for signal processing (I used the operator syntax
with it).  It eased development and maintainability, simply because it
allowed me to program in a paradigm that matched the application very
well.

To address its applicability to boost, I have two angles on that.  For
one, I would assume that a good number of programmers that use boost
have clients with limited programming expertise.  They could use
Dataflow.Signals like I mentioned above - by developing a relevant set
of components, and training the client on how to connect them and use
them.  Again, from my experience, people with absolutely no
programming experience can understand the dataflow paradigm very
quickly.  I've seen it happen.  I think it is because it uses
metaphors that are readily accessible to many people, as we tend to
connect things in our daily lives (e.g., connecting headphones to an
iPod, a DVD player to a TV, a guitar to a pedal to an amp), and we are
also used to tweaking the behavior of the components to get the
desired result (e.g., turning the volume knob).

As far as the other angle... Bjarne Stroustrup gave a very interesting
talk at the last BoostCon, and it had to do with providing different
levels of accessibility / configurability trade-offs to C++
programmers.  E.g., an entry level version of a library (or set of
libraries) that is super easy to use (and debug with) but maybe not
very flexible (although providing enough features to make it useful &
fun), vs. a fully templated super powerful version of the same library
that can assume you really know what you are doing.  I actually do see
the Dataflow library as something that can address the high
accessibility / low configurability (of individual components) end of
the spectrum, especially when combined with a GUI.  Furthermore, I
think it can be conducive to leading people to learn more about C++
(because the system itself could be used without any C++ knowledge
whatsoever).  Sooner or later, many users of a GUI/dataflow system
will find the need to extend a component in some new way or develop a
new component.  It wouldn't be difficult to show someone how to take
an existing component that adds two numbers, and change it to
something that multiplies the two numbers (granted, they would have to
learn how to compile C++ / boost which can be non-trivial).  If that
sparks their interest and they learn more C++, perhaps they will
advance to wanting to develop a component that works with files, and
learn Boost.Filesystem :-)

> I also wonder how much of the benefit of a graphical environment like
> LabVIEW carry over into your textual dataflow description (even with 2D
> operator overloading).  It seems to me that one of the main benefits of a
> GUI is that the user is somewhat guided towards a "syntactically correct"
> program by the help of, for example, labelled parameter fields to fill in on
> the components.  That, and other aspects, are lost.
>

Yes, the operator syntax is a gross simplification / approximation and
probably only suitable for simple networks.  As far as labeling
fields, the library offers some help.  It is possible to provide
appropriately named member functions that return a port - for example,
the storage component has a member function send_slot(), which returns
the port corresponding to the send() member function.  So, if I do,

storage<void()> initiate_send_of_zero;
storage<void(int)> zero(0);
storage<void(int)> receiver;

connect (initiate_send_of_zero, zero.send_slot());
connect (zero, receiver);

initiate_send_of_zero.send();

what will happen is:
initiate_send_of_zero will send a void() signal to the zero.send()
function (i.e., the zero.send() function will be called);
when zero.send() is called, it will send it's stored value (0) to receiver.
receiver now holds a 0.

In addition, the library offers a way to enumerate all of the ports -
you can do things like
connect(get_port_c<0>(a), get_port_c<1>(b));

there are also mpl versions:
connect(get_port<mpl::int_<0> >(a), get_port_c<mpl::int_<1> >(b));

So, you can have member typedefs for the ports that carry the name of
the port (e.g., typedef mpl::int<0> output;), in which case you could
do:
connect(get_port<A::output>(a), get_port<B::input>(b));

Unfortunately, registering all the ports so they can be enumerated is
an advanced topic and not discussed in the documentation yet :-(

> It may be significant that systems like LabView and MAX have not escaped
> from their niche application areas.  "Real programmers prefer text" perhaps.
>  Note that over the last couple of decades, chip design has almost entirely
> moved from GUI input (schematic capture) to textual input (hardware
> description languages).
>

The Dataflow library as a whole is an attempt to provide dataflow
functionality regardless of the programming medium (text or visual).

> And while thinking about hardware description languages, note also that they
> don't expose any sort of dataflow model even though the underlying circuit
> often has that sort of structure.  Similarly, when people were building
> "dataflow computers" back in the '80s they wrote compilers that hid the
> dataflow nature of the hardware behind a more conventional programming
> language (e.g. SISAL).
>

I have found dataflow programming useful precisely when it matches the
underlying problem structure.

> The closest to what you're proposing that I have seen is the stuff that the
> GNU Radio project is doing; I mentioned them on this list once before.
>  (They have been in the news recently after operating a GSM base station
> using their software radio at Burning Man.)  It would be really helpful if
> you could perhaps try to re-implement some of their stuff using your
> library, and see how it compares in terms of ease of coding, performance
> etc.  You could ask them to compare it themselves and submit reviews here.
>

I tried suggesting a Google Summer of Code project along these lines:
http://lists.gnu.org/archive/html/discuss-gnuradio/2008-02/msg00247.html

... but got very little response.  In terms of reimplementing - when
it comes to an existing dataflow system, the purpose of the dataflow
library wouldn't be so much in reimplementing it, but helping build on
the core of the dataflow framework.  I.e., we can take the base GNU
Radio framework which provides the component / connection code, and
develop a Dataflow support layer for it.  Then, we get things like the
GUI for free.

Unfortunately, it seems like GNU radio doesn't support pure C++
programs fully yet (I will give it a try when it does).  I have done a
similar experiment with VTK though - in the dataflow library, there is
an example in libs/dataflow/example/glv_gui/glvgui_vtk which brings up
the GUI editor with the 5 components from the following example:
http://www.dancinghacker.com/code/dataflow/dataflow/support/examples/new_layer/using_support_layer.html

With the GUI editor, you can instantiate the 5 components, connect
them in the correct chain, and click on the final component to invoke
the rendered scene.

> I think you really need to justify to us why this library is useful, and to
> whom.
>

This has definitely been lacking, but I hope I did a little bit of a
better job in my response here.

>
>>> - can you create push networks with cycles?
>>
>> Yes, as long as the components are designed in such a way that doesn't
>> propagate the signal in an infinitely recursive loop.
>
> I am reminded here of the design of asynchronous circuits using handshake
> signalling.  You might like to have a look at Kees van Berkel's thesis,
> which seems to be visible at books.google.com.
>

Will do - thanks for the reference.  And thank you for the continuing
discussion about the library, it is very helpful.

Best,

Stjepan