Subject: Re: [boost] Boost.Pipeline -- scheduling of segments
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2014-07-08 18:59:16
On 8 Jul 2014 at 21:23, christophe.j.henry_at_[hidden] wrote:
> >My only qualm, really, is that proposed Boost.Asynchronous does not
> > appear to me to implement pull-push pipeline management which is the
> >hard part of a Pipelines implementation.
> > I am under the possibly mistaken understanding that HPX does.
> What do you mean with pull-push?
> Maybe I'm making it myself too easy but I'd see every pipeline stage as a
> scheduler, say, for Asynchronous a stealing threadpool scheduler(with one or
> more threads), every stage getting a job transforming input data and posting
> to the queue of the next scheduler a functor doing the next stage
> transformation, etc.
> Then I'd create a composite in one line of code to make sure work-stealing
> happens and that would be it for the infrastructure.
> Am I missing something?
We almost certainly are meaning different things. I meant something
auto p = Paragraphs << TextFile << HTMLFile <<
std::cout << "'Niall' found at offset " << it->offset << std::endl;
Here a pipeline extracting paragraphs from the text content of the
HTML file from boost.org is constructed on the first line. No
processing actually happens yet.
The second line configures more pipeline, an iterator matching all
instances of the word 'Niall' in the fifth paragraph of the extracted
text. Still no processing happens yet.
The third line loops the iterator, printing the offsets of 'Niall' in
the fifth paragraph. At this point the Paragraphs issues a pull
forward that it wants the fifth paragraph to TextFile, TextFile tells
HTMLFile it only wants the fifth <p>...</p>, HTMLFile only bothers
fetching as much of the content as will supply the fifth <p> content.
It then pushes what content it also retrieved back down the pipeline
as maybe empty paragraphs are to be disregarded etc and only the
lower stage processors will know anything about that.
That's what I meant by push-pull pipeline management. Oh, and it
should cope with live content changes, so if the source HTML changes
the entire pipeline is flushed of state until the next pull or is
dynamically updated with the new content and all current code using
it is told.
-- ned Productions Limited Consulting http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/
Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk