Re: [boost] Review Request: Dataflow.Signals

From: Stjepan Rajko (stipe_at_[hidden])
Date: 2008-03-16 13:12:13

Hi Paul,

On Sat, Mar 15, 2008 at 6:02 PM, Paul Baxter <pauljbaxter_at_[hidden]> wrote:
>
>  I'm really excited about this library but is it ready for review yet?
>

Thanks for your comments - I understand your concerns, please see my
thoughts below.

>  Stjepan has kindly kept us up to date with a v0.80/81 and later 0.90 with a
>  fledgling blueprint layer but, playing devil's advocate, perhaps more
>  discussion of what it's trying to solve and how it might make use of other
>  libraries might be useful prior to review. I'm concerned that its fate might
>  be similar to the first revision of the logging library otherwise. I suspect
>  there are many differing/conflicting requirements for a dataflow library.
>

I would definitely agree that there are many differing requirements
for a dataflow library.  That is why I am only proposing the Signals
layer of the review - it was the original focus of the library, and
even though the code under and over the Signals layer has been in
turmoil for a long time, the interface of the Signals layer itself has
been mostly stable for months now.  Realizing that made me think I
should focus on getting it out for review first.  It is possible that
there would also be different requirements even for a
Boost.Signals-based library - but I'm not as concerned with the fate
of the library in terms of acceptance as I am with the feedback I get.

>  When I've done similar things in the past the critical decision has been
>  whether dataflows are defined at compile time or run-time and what the
>  strategies are for information, buffer and connection management.
>

One thing that I tried to do is build the Signals layer on top of
something generic enough to (eventually) accommodate all of the above.
 For now, the generic layer is still biased towards the
characteristics of Boost.Signals - connectability is determined at
compile-time but dataflows are constructed at runtime; connection
management and the the operation of the network is left completely to
the components and "the outside world".  I hope that by review time I
can expand the generic layer so it can at least accommodate some
different scenarios.  That way anyone can adapt or implement their
ideal dataflow framework to work with the library.

>  Even in the last couple of days several areas of functionality and interface
>  have been discussed. Compiler support seems weak too on very popular
>  compilers.
>

These discussions have been great - and the features discussed would
be great additions, but I don't think that their current lack makes
the library inadequate.  Also, I can make a reasonable effort to
expand the compiler support, but I don't see this as a critical issue
for review either.  I don't want to put too much effort into doing
detail work on something that might be shown to need a complete
redesign.

>  Perhaps Stjepan could comment further on his plans between now and review.
>  Its a worthwhile library and something I'm personally very interested in but
>  I'd love to see it fleshed out a bit more with a reasonable application
>  example. This is a library that is not really core to many tasks and
>  probably needs to show its mettle through use.
>

My plans between now and review mostly have to do with expanding the
documentation, with a few implementation pushes as well.

On the documentation side it's:
* document more of what is implemented
* provide better descriptions of examples, and add more examples
* reorganize the docs for review so that it is clear what is finished
and ready for review, and what is just to be considered an
example/proof of concept)

On the implementation side it's:
* tweak the Dataflow.Signals layer so that components can be composed
at compile-time.
* address recently discussed ideas / requests.

Most of the things above are already in the works, and it might take a
long time to get a review manager + time slot anyway.  Now, the ideal
situation would be that I also manage to do the following:
* expand the generic layer to better support non-Boost.Signals-like
dataflow frameworks (e.g., determining connectability and other
properties at run-time)
* add basic support layers for existing dataflow frameworks out there
to show that the generic layer can support them.

If that is also done by review time, then the review can cover both
the Dataflow.Signals layer and the generic layer - otherwise the
generic layer can be considered at some other time.

>  Is it intended to tackle building processing sequences on the fly. Is it
>  lightweight enough to support fine-grained building blocks where connection
>  setup could end up dominating run time if not done well. Is it simply about
>  ease of use in putting together blocks? Is it's uniqueness about supporting
>  dynamic run-time sequence configurability rather than compile-time?
>

The Dataflow.Signals layer does what boost::signal does - it is not
fine grained, it's compile-time connectability-checking with run-time
connecting, it's producers with imperfect tracking of consumers and
consumers with no tracking of producers, and it couples data-transfer
with component invocation.

The generic layer, on the other hand, should eventually support all of
the scenarios you mention (and I really hope that it too will be ready
by review time).  Then anyone can take an existing dataflow framework
that has the desired characteristics (or develop a new one), make a
Dataflow support layer for it, and take advantage of anything that is
built on top of Dataflow (like the blueprint layer or the GUI, which
are still just in a proof-of-concept stage, but for review I am more
interested in showing that they /can/ be built rather than providing
full-featured versions).

>  I guess I'd like to see the rationale for what its good at and then see
>  compelling evidence (in the form of an example?)

I hope this e-mail made things a bit clearer.  If you still have
questions please let me know - but also I'd love to hear about what
your personal dataflow wishlist is, and what sorts of dataflow
frameworks/scenarios you've worked with in the past  (offline or
here).

Kind regards,

Stjepan