Re: [boost] any interest in a producer consumer template?

From: Arash Partow (arash_at_[hidden])
Date: 2004-05-22 19:08:56

Hi Jeff,

Thanx for your feedback its very much appreciated,

 >
 > Making each element of the processing chain a thread  is rather constraining.
 >  What if I want to use a leaders / followers approach to threading my chain of
 > transformers?  So when I get a callback on my socket I chain thru all the
 > transformations in the callback thread thus avoiding several thread context
 > switches just to process the data.
 >

thats a possibility but i see that in the chain of entities, each producer
entity (including interchange entities) are allowed to inject data into the
chain.

as far as event oriented sockets are concerned, lets assume you have a
producer <string> ----> consumer <string>, what i was heading towards was a
structure where the producer continually reads from a socket, whenever data
comes through it adds the data to the link when the minimum pending data limit
is reached the link signals the consumer. the consumer the goes and grabs the
data.

while this is all happening the producer can still go on its way producing
data (ie: recv from sockets)

In this way both the producer and consumer are independent of each other,
when there is no data to be consumed or produced the entities go to sleep.

I think an unthreaded mode of producer-consumers is very much possible, i
think it would be a development step for prototyping (ie: get it right with
no threads, then extend it to be concurrent )I'll have to look into how it can
be incorporated into the design.

 >
 > How does this differ from the signals and slots library?
 >
 > http://www.boost.org/doc/html/signals.html
 >
 > Couldn't we just create a slot endpoint for the various types?  Can I connect
 > 2 or more consumers to a particular interchange?  For this to be useful, I
 > should be able to.
 >

its quite possible to use signals and slots and indeed the signaling concept
I propose is very similar to the signal and slots library. however I also intend
that the link also contain the data being transported. This way concurrent access
and additions to the data is centralized from both a producer and consumer concept.

what is your ideas of a situation like this:

producer <string> [link<string,socket>] |---> TCP/IP ---> [link<string,socket>] consumer <string>

i think in this situation the link mechanism is derived from a base link entity,
but is specialized for data transfer over sockets + initial connection negotiations.

As far as linking 2 consumers to a producer you have to ask these questions before
you implement such a mechanism:

1.) will each of the consumers receive all the data that the producer produces, or
will it be like a worker situation where first come first served?

2.) when signaling consumers what priority of will be given to any one type of
consumer? if any...

3.) will the consumers have different capacities?

from a design POV I think  such a producer will contain links to each member, and
depending on the answers from the above questions data will be transported in
different manners. the production policy becomes quiet complicated at this point.

 >
 > Well, I think it could be useful and wouldn't mind having it in boost, but it
 > seems like we need other basic infrastructure like a reactor, callback timers,
 > and sockets before there will be much advantage.
 >

yeah i think it will be useful too thats why I'm asking for opinions, as far as
other infrastructure, i think the boost sockets library is coming along quiet
nicely, reactor and call-back infrastructure is something that might not be needed
if the design does not take into account full asynchronous behavior just simple
serialization of data processing.

Arash Partow

__________________________________________________
Be one who knows what they don't know,
Instead of being one who knows not what they don't know,
Thinking they know everything about all things.
http://www.partow.net

Jeff Garland wrote:
> On Sat, 22 May 2004 20:20:07 +1000, Arash Partow wrote
> 
>>Hi all,
>>
>>I was wondering if there would be any interest in a template
>>library based around producers and consumers.
> 
> 
> Sure, but....
> 
> 
>>I've developed something already and have a series of working 
>>examples, I propose a library that consist of 3 basic entities, a 
>>producer a consumer and a hybrid entity called an interchange.
>>
>>the entities themselves will be threaded and will be linked together
> 
> 
> Making each element of the processing chain a thread  is rather constraining.
>  What if I want to use a leaders / followers approach to threading my chain of
> transformers?  So when I get a callback on my socket I chain thru all the
> transformations in the callback thread thus avoiding several thread context
> switches just to process the data.
> 
> 
>>via a linking mechanism that is itself a base template but which can
>>be extended to provide data transfer between same system processes 
>>and threads and also extended versions which provide data transfer over
>>pipes, sockets, serial lines etc..
>>
>>the idea of the producer-consumer is very simple the producer 
>>produces a type of data ie: string, a class, a struct etc.. and a 
>>consumer that is instantiated as a consumer of the type will consume 
>>the data provided by the producer and do something with it.
> 
> 
> How does this differ from the signals and slots library?  
> 
> http://www.boost.org/doc/html/signals.html
> 
> Couldn't we just create a slot endpoint for the various types?  Can I connect
> 2 or more consumers to a particular interchange?  For this to be useful, I
> should be able to.
>  
> 
>>...snip detail...
>>
>>The link mechanism is a basically a type that supports the common 
>>type between a producer and its consumer, it is a one way flow of 
>>data, the consumer is signaled when data is ready for it to collect 
> 
> 
> Again, seems like signals and slots to me...
> 
> 
>>...snip...
>>
>>A link has a basic get and add functionality but depending what 
>>entity is being implemented they will only see it via an interface 
>>and hence only be able to invoke 1 kind of method, a producer can 
>>only invoke "add" where-as a consumer can only invoke a get, an 
>>interchange can invoke both because it is instantiated with both a 
>>consumer and producer link interface, but the links are separate for 
>>an interchange, because it requires 2 links, one link for the 
>>incoming data and one link for the out going data. both link maybes 
>>of different types, ie:
>>
>>Producer <string> ----> Interchange <string, AClass> ----> Consumer <AClass>
>>
>>I need some opinions, about whether or not people will find this
>>library useful and hence have it as part of Boost, if so could people
>>give me their opinions about my design, its just a simple first trial
>>run at designing it, i intended for the producers and interchange
>>entities to be policy driven as far as production of data is
>>concerned.
> 
> 
> Well, I think it could be useful and wouldn't mind having it in boost, but it
> seems like we need other basic infrastructure like a reactor, callback timers,
> and sockets before there will be much advantage. 
> 
> 
>>I think this library can be used in distributed computing, in network
>>packet assembly, and also for interfacing and passing data up and 
>>down threaded software layers.
> 
> 
> Yes, it is a useful design in 'data flow' systems.  Distributed
> publish-subscribe type systems which have various data transformers use this
> type of design (or they should). 
>  
> Jeff
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
> 
>