Re: [boost] [signal network] SoC draft of Signal Network proposal posted (based on recent "signal flow" inquiry)

From: Tobias Schwinger (tschwinger_at_[hidden])
Date: 2007-03-18 12:39:43

Stjepan Rajko wrote:
> Thanks for everyone that replied to the query for interest on a
> "signal flow" library.  I have started incorporating the feedback and
> expanding the documentation into a more formalized notion of a Signal
> Network library, which I would like to propose for Google Summer of
> Code / Boost.
> 
> http://dancinghacker.com/code/signet/ holds a draft of the
> proposal/documentation - the current implementation with
> examples/tests is available for download from that page.  It should
> work with both Boost.Signals and Frank's thread_safe_signals (thanks
> for that patch Frank it worked) for the most part.
> 
> Some of the improvements are:
>  * better choice of operators / syntax
>  * modular components for storage and flow control of signals
>  * started implementing support for threading
>  * prototype of a predefined topology (chain)
>  * ... and more....
> 
> If you have any comments / requests / suggestions on the proposal
> and/or the library, please send me a note (or feel free to post
> requests on the SoC wiki :-) )

Oh no, I promised to send you some feedback after taking a deeper look 
but didn't get around to actually post anything. I'll do so now:

IMO, the most important point from your above list is

 >  * modular components for storage and flow control of signals

If a signal network library should be used for serious signal processing 
we better have the framework manage memory of the data entities 
associated with each link (primarily for buffers):

Having one dynamic call per sample is a no-go because of the way CPUs 
implement pipelining.
Using template tricks to melt everything into one routine isn't a good 
idea either, because a) everything needs to be wired up at compile time 
b) we are likely to get instruction cache misses if there's too much 
code processed at once.
Having buffers alone doesn't solve the problem, so we also want to share 
them wherever we can to avoid data cache misses.

So basically we'll want to chain routines, that encapsulate tight loops 
that work on buffers based on control input.

The framework does not have to provide (at least not as a core 
component) or know about buffers, but rather implement memory management 
for arbitrary objects. From the framework's perspective there should be 
no distinction between buffers and control signals, but different 
properties of the individual /pins/ (I'm using this term for the 
sources&sinks here) and the /links/ between them, that control how the 
framework handles the associated data entities.

Connecting two processing nodes would simply set source and sink data 
pointer to the same memory location. Eventually (see below) we also have 
to set up a notify mechanism. Boost.Signal might be the way to go, but 
I'm not at all sure at this point.
In some cases we can go further and have a single processing node's 
source and sink share the same data object, so we can easily enable 
framework-controlled memory reuse.

So we get three basic pin types:

   - in         // processing node reads data from it
   - out        // processing node writes data to it
   - in_out     // processing node changes data in it

...

I could go on like this for a while and outline my image of how such a 
framework should look like (which is pretty clear by now - I've been 
working on a design for such a thing for several years, every once in a 
while) but I'll stop here, for now. Let's first see whether our plans 
are compatible until here. Maybe you're up to something totally else...

Further, I think we don't need an overloaded operators interface. Don't 
get me wrong, I'm not the "oh no - not so much tricky template stuff" 
kind of guy (in fact, I like template hacking a lot), but I believe the 
power of this thing lies in the runtime world.

Serialization and a careful implementation that allows to extend things 
at runtime by loading shared object files is much more important, than 
fancy compile time setup, IMO. So is multi-threading - ideally the 
framework is thread-aware and self-optimizing.

Also note that I'm not only talking about a signal processing framework 
but something more general (signal processing gives an interesting use 
case, though). One could also use it to wire up the data flow in a batch 
system, resolve GUI dependencies, create vector graphics or use it as a 
general purpose programming system.

As you can see, I'm potentially interested in collaborating on this 
project, if I haven't scared you off already ;-).

Regards,
Tobias