Re: [boost] GSoC: Boost.Pipeline -- scheduling of segments

Subject: Re: [boost] GSoC: Boost.Pipeline -- scheduling of segments
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2014-07-01 19:37:44

On 1 Jul 2014 at 23:24, Benedek Thaler wrote:

> It's not deterministic when the first segment will end, the pipeline will
> hang and won't provide any output. Also, in a slightly less strict
> scenario, when the end of input is specified, the latency could be just too
> high.
> 
> To summarize, the following options are on the table:
> 
>  1. Dedicate a thread to each segment (what to do with a fixed size
> threadpool?)
>  2. Constrain the transformations to be reentrant.
>  3. Run each transformations until there is input to be processed, from
> beginning to the end.
> 
> We are kindly asking the recipients to share their ideas or opinions,
> ideally with a matching usecse.

Mmm, you're getting into needing to implement a demand-driven 
transformation based data stream framework, something I suspected 
would always happen with this proposal.

In short, either you make all your processors coroutineable and 
resumable - which is a ton of work. Or you implement a full data 
stream implementation, which is also a ton of work. If you go with 
your approach 2 you'll murder reusability, with approach 3 you'll use 
a ton of RAM.

Given yours is a GSoC and needs to be done sooner rather than later, 
I'd actually implement a dynamic thread pool which scales up the 
worker threads to whatever is the minimum set of blocking processors. 
Or borrow the threading facilities in HPX and port your code over to 
HPX completely, then your threading and blocking problems go away 
because HPX has implemented the data stream framework for you. You'll 
obviously lose C++ genericity if you do this though.

Niall

-- 
ned Productions Limited Consulting
http://www.nedproductions.biz/ 
http://ie.linkedin.com/in/nialldouglas/