Re: [boost] Boost.Pipeline -- scheduling of segments

Subject: Re: [boost] Boost.Pipeline -- scheduling of segments
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2014-07-08 18:59:16

On 8 Jul 2014 at 21:23, christophe.j.henry_at_[hidden] wrote:

> >My only qualm, really, is that proposed Boost.Asynchronous does not
> > appear to me to implement pull-push pipeline management which is the
> >hard part of a Pipelines implementation.
> > I am under the possibly mistaken understanding that HPX does.
> 
> What do you mean with pull-push?
> Maybe I'm making it myself too easy but I'd see every pipeline stage as a 
> scheduler, say, for Asynchronous a stealing threadpool scheduler(with one or 
> more threads), every stage getting a job transforming input data and posting 
> to the queue of the next scheduler a functor doing the next stage 
> transformation, etc.
> Then I'd create a composite in one line of code to make sure work-stealing 
> happens and that would be it for the infrastructure.
> 
> Am I missing something?

We almost certainly are meaning different things. I meant something 
like:

auto p = Paragraphs << TextFile << HTMLFile << 
"http://www.boost.org/"

Paragraphs::Words::iterator it=p[4].match("Niall");

while(it)
  std::cout << "'Niall' found at offset " << it->offset << std::endl;

Here a pipeline extracting paragraphs from the text content of the 
HTML file from boost.org is constructed on the first line. No 
processing actually happens yet.

The second line configures more pipeline, an iterator matching all 
instances of the word 'Niall' in the fifth paragraph of the extracted 
text. Still no processing happens yet.

The third line loops the iterator, printing the offsets of 'Niall' in 
the fifth paragraph. At this point the Paragraphs issues a pull 
forward that it wants the fifth paragraph to TextFile, TextFile tells 
HTMLFile it only wants the fifth <p>...</p>, HTMLFile only bothers 
fetching as much of the content as will supply the fifth <p> content. 
It then pushes what content it also retrieved back down the pipeline 
as maybe empty paragraphs are to be disregarded etc and only the 
lower stage processors will know anything about that.

That's what I meant by push-pull pipeline management. Oh, and it 
should cope with live content changes, so if the source HTML changes 
the entire pipeline is flushed of state until the next pull or is 
dynamically updated with the new content and all current code using 
it is told.

Niall

-- 
ned Productions Limited Consulting
http://www.nedproductions.biz/ 
http://ie.linkedin.com/in/nialldouglas/