Re: [boost] any interest in a producer consumer template?

From: Jeff Garland (jeff_at_[hidden])
Date: 2004-05-22 09:45:15

On Sat, 22 May 2004 20:20:07 +1000, Arash Partow wrote
> Hi all,
> 
> I was wondering if there would be any interest in a template
> library based around producers and consumers.

Sure, but....

> I've developed something already and have a series of working 
> examples, I propose a library that consist of 3 basic entities, a 
> producer a consumer and a hybrid entity called an interchange.
>
> the entities themselves will be threaded and will be linked together

Making each element of the processing chain a thread  is rather constraining.
 What if I want to use a leaders / followers approach to threading my chain of
transformers?  So when I get a callback on my socket I chain thru all the
transformations in the callback thread thus avoiding several thread context
switches just to process the data.

> via a linking mechanism that is itself a base template but which can
> be extended to provide data transfer between same system processes 
> and threads and also extended versions which provide data transfer over
> pipes, sockets, serial lines etc..
>
> the idea of the producer-consumer is very simple the producer 
> produces a type of data ie: string, a class, a struct etc.. and a 
> consumer that is instantiated as a consumer of the type will consume 
> the data provided by the producer and do something with it.

How does this differ from the signals and slots library?  

http://www.boost.org/doc/html/signals.html

Couldn't we just create a slot endpoint for the various types?  Can I connect
2 or more consumers to a particular interchange?  For this to be useful, I
should be able to.
 
>...snip detail...
> 
> The link mechanism is a basically a type that supports the common 
> type between a producer and its consumer, it is a one way flow of 
> data, the consumer is signaled when data is ready for it to collect 

Again, seems like signals and slots to me...

>...snip...
>
> A link has a basic get and add functionality but depending what 
> entity is being implemented they will only see it via an interface 
> and hence only be able to invoke 1 kind of method, a producer can 
> only invoke "add" where-as a consumer can only invoke a get, an 
> interchange can invoke both because it is instantiated with both a 
> consumer and producer link interface, but the links are separate for 
> an interchange, because it requires 2 links, one link for the 
> incoming data and one link for the out going data. both link maybes 
> of different types, ie:
> 
> Producer <string> ----> Interchange <string, AClass> ----> Consumer <AClass>
> 
> I need some opinions, about whether or not people will find this
> library useful and hence have it as part of Boost, if so could people
> give me their opinions about my design, its just a simple first trial
> run at designing it, i intended for the producers and interchange
> entities to be policy driven as far as production of data is
> concerned.

Well, I think it could be useful and wouldn't mind having it in boost, but it
seems like we need other basic infrastructure like a reactor, callback timers,
and sockets before there will be much advantage. 

> I think this library can be used in distributed computing, in network
> packet assembly, and also for interfacing and passing data up and 
> down threaded software layers.

Yes, it is a useful design in 'data flow' systems.  Distributed
publish-subscribe type systems which have various data transformers use this
type of design (or they should). 
 
Jeff