Re: [boost] GSoC: Boost.Pipeline -- scheduling of segments

Subject: Re: [boost] GSoC: Boost.Pipeline -- scheduling of segments
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2014-07-04 07:33:14

On 4 Jul 2014 at 11:55, Benedek Thaler wrote:

> Thanks for the further encouragement. I'll definitely take a closer look at
> HPX in the future. However, the deadline of GSoC is closing in, I have to
> found a simpler solution for now. Hopefully, having a more solid base will
> enable us to build something truly great later.

This is why I originally suggested HPX for this. Hartmut, who is one 
of the oldest core Boost people still here, did a really great job on 
HPX which is pretty much the final say on how to do task execution at 
large scale.

A Pipelines implementation matching your proposal wouldn't be trivial 
on HPX, but it would be tractable before GSoC ends. Your only 
alternative to deliver before GSoC ends I suspect is to fire a thread 
at every component in the pipeline as without async i/o, you can't 
leverage coroutines.

As you've probably realised by now, pipeline processors need to 
communicate in two directions, so end stage processors need to be 
able to tell the first data source "I need one more byte or else I 
can't continue" and such. Thing is, that one extra byte at the end of 
the pipeline may turn in Kb of extra bytes as each processor imposes 
its own needs on the forward propagation. This is the kind of stuff 
HPX takes care of for you.

Niall

-- 
ned Productions Limited Consulting
http://www.nedproductions.biz/ 
http://ie.linkedin.com/in/nialldouglas/