Re: [boost] Flow-based programming library for Boost?

Subject: Re: [boost] Flow-based programming library for Boost?
From: Julian Gonggrijp (j.gonggrijp_at_[hidden])
Date: 2012-12-09 09:18:42

Marcus, could you please write inline replies and zap all parts that 
you don't reply to? Thanks in advance.

Marcus Tomlinson wrote:

> Hi Julian, thanks for the post. I see a pattern forming :) I think I need to make my documentation clearer in a number of places.

Suggestion: describe for each member function what it does. And give a 
shorter, more conceptual overview at the front page of how the library 
works and why.

Note that you find the need to expand upon your documentation while 
it's already quite beefy. Some of your classes have lots of public 
methods. These are clear signals that your library is not simple 
enough.

> Ok, in DSPatch, circuits provide 2 main purposes: encapsulation of a component network, and parallel processing control (see more on how parallel processing is done see: http://www.adaptaudio.com/DSPatch/spec_page.html).

From that page I gather that there may be threads specific to a 
component as well as "general" (circuit) threads that take up work 
from each component in a circuit in turn. What is the motivation for 
this design? How do you avoid having two threads execute the same 
component's job on the same inputs, especially if one is a component 
thread and one is a circuit thread?

> Components don't have to be added to a circuit to be routed together, this is just recommended due to the advantages (mentioned above) gained in this approach.

Well that is good news. Am I right to assume that the circuit object 
is not necessary to enable multithreading?

> Components have ConnectInput / DisconnectInout methods that can be called directly in order to create component networks outside of a circuit.

I looked up those methods in the class reference. IIUC you have to 
"extract" the wire from the source component and manually assign it an 
index or name. Why not just take a wire object as the sole argument 
and give it an index automatically? In addition you can connect only 
one wire at a time. Why not offer a ConnectInputs method that takes a 
collection of wires?

Actually I find this interface really worrying because it seems to 
suggest that a component needs to know about other components in order 
to receive input from wires. On the specification page I read that the 
/wires/ need to know about the components as well. "Separation of 
concerns" is not being fully applied here. I would strongly prefer a 
more generic interface where wires can operate regardless of whether 
they're connected to components and vice versa.

> 
> You are correct, there isn't a way to extract multiple values from a single input. This is because each component input can only accept one wire at a time. When another wire is connected to an input that already has a connected wire, that wire is replaced with the new one.

Then
1) why can an input accept only a single wire at a time, and
2) why can't wires transport multiple values at a time?

> This is not the case with outputs though. One output can be distributed to multiple inputs. This is definitely something that should be explained explicitly (I'll add it to the list).

From the tutorial it was quite clear to me that this would be 
possible, but it appears that the user would have to implement this by 
themself every time. There is no ready facility to abstract and 
automate it. Besides, given that components can extract only a single 
value from each input at a time, the possibility to output multiple 
values at once is only marginally useful.

> 
> I don't see dynamic typing as a limitation, I see it as quite useful. Lets say your processing an input stream of audio samples. With dynamic typing, you can adjust the audio's sample size at run-time and have the component input adapt dynamically without breaking flow.

You can do polymorphism in safer, more or less "static" ways. Consider 
Boost.Variant.

> 
> I also have to disagree with you that object-oriented design has nothing to offer to dataflow programming, but perhaps this is just one man's opinion against other.

Of course it's partly a matter of taste. However, object-oriented 
design does incur overhead. If dynamic binding or superfluous pointer 
dereferencing is involved in the chain that transports a value from 
one component to another (output-signal-wire-signal-input?), it may 
well be significantly slower than it could be, especially since every 
value has to be transported in a separate invocation of the chain.

Apart from the overhead, OOD does not easily permit the same powerful 
modularity that generic programming offers. I think this explains most 
of the limitations in your library. I also think it explains why OOD 
is largely out of fashion in Boost libraries, in favour of generic 
design.

OOD is a useful tool with some uncontroversial use cases, such as game 
engines. However it should never be a goal in itself. Some areas 
benefit more from a generic design, such as containers and algorithms. 
As far as I'm concerned dataflow programming is more like the latter.

-Julian