Re: [boost] [rfc] getting the dataflow library ready for submission

From: Stjepan Rajko (stipe_at_[hidden])
Date: 2007-11-12 18:46:56

Hi Roland,

On Nov 12, 2007 4:00 AM, Roland Schwarz <roland.schwarz_at_[hidden]> wrote:
> I just skimmed over the docs, and I have some questions:
>

Great - thanks for taking the time!

>
> 1) How would I specify the control flow? I.e. How would I choose between
>     "source driven" and "sink driven"?
>

Right now, the generic layer doesn't deal with this at all - it is
completely up to the underlying mechanism (just to clarify, by
"mechanism" I mean a particular dataflow framework / data transport
mechanism, such as Boost.Signals or VTK).

The layer based on Boost.Signals is source driven - it sort of assumes
that the dominant dataflow direction is from the caller to the callee.
Although, if in actuality the data flowed the other way (via the
return value), then it would be sink driven.  The Dataflow.Signals
Example "pull-based networks" illustrates this alternative.

On the other hand, I believe the VTK mechanism is sink driven (I am
complete newb with VTK - I just learned enough to provide a small
example support layer).

> 2) Is there a mechanism that will give support for block oriented
>     processing, or will a function be called on every data item?
>     To clarify: I do not mean a couple of primitive data items blocked
>     together into a larger container which is itself a data item on
>     its own, but something like the fread() function is able to do:
>     *) request a number of items
>     *) return as much as currently available (may be less than wanted)
>     *) block process what has been received
>     *) wait for more data
>
> You are using VTK as an example and so I suspect you are using the
> demand driven processing in your library. Are you also providing
> a caching and timestamping mechanism to hold intermediary results
> that need no updating?
>
> Which special concepts for memory management are you suggesting (if
> at all)? Memory management is one of the crucial parts of data flow
> programming. Will you provide a single copy mechanism?
>

About all these - the generic dataflow layer is currently really
small.  It basically deals with connecting the dataflow network
together, and that's it.  Everything else is up to the underlying
mechanism.  I'm not sure whether the issues you mention should be
dealt with at the generic level, but I guess we'll see as things get
built on top of it.

As far as the Dataflow.Signals layer is concerned, it currently
addresses little of the above features/issues. There is a "storage"
component which can serve as cache (but offers no timestamping).  It
would be useful to add components which dealt with these issues though
(e.g., a timestamped storage, and a component (or boost::signals
Combiner) which only forwards a signal when it has changed).  There is
no memory management support here either - because of the nature of
dataflow networks built on top of boost::signals (there is no global
knowledge of the network - each component only holds connections to
its sinks), it's basically all up to the components.  There could be
some memory management components added if necessary (like now there
are some threading related components).  And I'll think about
fread-like processing components in Dataflow.Signals, seems like it
would be a good idea as well.

I am not understanding the term "single copy mechanism" - could you
help me out with a reference?

> I always found that in data flow paradigm these are the hard parts.
> It is not the hard part to come up with a "nice syntax".

A while back, Tobias Schwinger suggested a dataflow framework which
does deal with a lot of the issues you bring up here.  It's discussed
briefly in the "future directions" of the documentation.  I think that
such a framework could be supported by the generic layer of this
library, after it grows somewhat perhaps.

My personal goal in providing a generic layer is to provide a visual
programming environment on top of it.  That way, anything that has a
Dataflow support layer can have it's networks be manipulated visually,
serialized, instantiated, etc.  In the meantime, while there isn't
much built on top of the generic layer, the "nice syntax" is
unfortunately the biggest practical benefit.

Again, thanks for taking the time!

Best regards,

Stjepan