Re: [boost] GSoC: Boost.Pipeline -- scheduling of segments

Subject: Re: [boost] GSoC: Boost.Pipeline -- scheduling of segments
From: Adam Berkan (adam.berkan_at_[hidden])
Date: 2014-07-02 20:27:17

On 1404291049408, Benedek Thaler <thalerbenedek_at_[hidden]> wrote:

> Thanks for all the responses so far!
>
> *Niall*: I'm afraid using HPX is out of scope in context of a Boost
> library. If all else fail, I'll fall back to a simple implementation (e.g:
> 1). IMO, depending on this design decision will be this library a great
> tool or just something nice.
>
> *Adam*: I'm not sure how this smart queue would solve the problem of
> reentrancy. Currently, if a thread returns from a transformation, it can
> never be resumed. This makes yielding hard/not possible.
>
>
I was suggesting that the thread would grab a single piece of data from a
queue, do a single transformation, and then return where it would make a
scheduling decision about where to go next.  Then it would once again grab
one piece of data and do one transformation before returning to scheduling.

I'm not suggesting that we would interrupt a thread in the middle of a
stage...  That would require writing most of an operating system kernel :)

The tricky part is knowing where there's work to be done.  The "smart
queues" would have to update some stats in the scheduler on every push, and
potentially wake up threads.  It's certainly a non-trivial task.

There's also the issue of the multi-in (N:1 & N:M) transformations, i.e.
that take input queues. They would either have to take a whole thread,
provide some kind of yield, or something else.  I'd probably ignore this
for now (just consume a thread) and focus on the simple 1:1 and 1:N cases
as they naturally return after each transformation.

Let me know if that makes sense.
Adam

If we make reentrancy mandatory, then we face the next challenge, which you
> described above IIUC. The think the key points are here: (1) A task can
> yield, and can point to an other task which is waited upon. (2) A task (but
> not the thread) can "block" on the upstream queue, if it's empty. If it's
> not clear, I'll describe these in greater length.
>
>

> *Oliver*: I considered using coroutines, however I don't clearly see how
> this would solve this problem exactly. IIUC, Coroutines provide a way for a
> single thread to execute multiple methods. I'd need multiples threads
> executing multiple methods in a similar fashion. I'll continue thinking
> about this.
>
> I couldn't found it in the docs. Does Coroutine have any drawbacks?
>
> Thanks,
> Benedek
>