Re: [boost] GSoC: Boost.Pipeline -- scheduling of segments

Subject: Re: [boost] GSoC: Boost.Pipeline -- scheduling of segments
From: Benedek Thaler (thalerbenedek_at_[hidden])
Date: 2014-07-04 05:52:58

On Thu, Jul 3, 2014 at 2:27 AM, Adam Berkan <adam.berkan_at_[hidden]> wrote:

> I was suggesting that the thread would grab a single piece of data from a
> queue, do a single transformation, and then return where it would make a
> scheduling decision about where to go next.  Then it would once again grab
> one piece of data and do one transformation before returning to scheduling.
>

In case of 1-1 transformations, I think this is done in the `development`
branch, altough the really interesting issues are not taken care of... ->

>
> I'm not suggesting that we would interrupt a thread in the middle of a
> stage...  That would require writing most of an operating system kernel :)
>
> The tricky part is knowing where there's work to be done.  The "smart
> queues" would have to update some stats in the scheduler on every push, and
> potentially wake up threads.  It's certainly a non-trivial task.
>

 -> like this. Now I think we are on the same page regarding such queues or
scheduling facility..

>
> There's also the issue of the multi-in (N:1 & N:M) transformations, i.e.
> that take input queues. They would either have to take a whole thread,
> provide some kind of yield, or something else.  I'd probably ignore this
> for now (just consume a thread) and focus on the simple 1:1 and 1:N cases
> as they naturally return after each transformation.
>
>
I think only 1:1 is an easy case. Even 1:N has an issue: consider a
transformation which duplicates its input. What if the second element
cannot be pushed to the downstream queue because it's full? The
transformation should yield (and not spin).

Thanks,
Benedek