Re: [boost] GSoC: Boost.Pipeline -- scheduling of segments

Subject: Re: [boost] GSoC: Boost.Pipeline -- scheduling of segments
From: Hartmut Kaiser (hartmut.kaiser_at_[hidden])
Date: 2014-07-03 08:19:29

> > *Niall*: I'm afraid using HPX is out of scope in context of a Boost
> > library. If all else fail, I'll fall back to a simple implementation
> (e.g:
> > 1). IMO, depending on this design decision will be this library a great
> > tool or just something nice.
> >
> > *Adam*: I'm not sure how this smart queue would solve the problem of
> > reentrancy. Currently, if a thread returns from a transformation, it can
> > never be resumed. This makes yielding hard/not possible.
> >
> >
> I was suggesting that the thread would grab a single piece of data from a
> queue, do a single transformation, and then return where it would make a
> scheduling decision about where to go next.  Then it would once again grab
> one piece of data and do one transformation before returning to
> scheduling.
> 
> I'm not suggesting that we would interrupt a thread in the middle of a
> stage...  That would require writing most of an operating system kernel :)
> 
> The tricky part is knowing where there's work to be done.  The "smart
> queues" would have to update some stats in the scheduler on every push,
> and
> potentially wake up threads.  It's certainly a non-trivial task.
> 
> There's also the issue of the multi-in (N:1 & N:M) transformations, i.e.
> that take input queues. They would either have to take a whole thread,
> provide some kind of yield, or something else.  I'd probably ignore this
> for now (just consume a thread) and focus on the simple 1:1 and 1:N cases
> as they naturally return after each transformation.

All of that functionality is available from HPX. It gives you work-queue
based scheduling (with work stealing) of suspend-able/resume-able threads
(i.e. supporting yield) with very little overhead. It also manages the 1:N
or N:M threading for you.

You could try building all of your functionality on top of HPX first. This
could allow to figure out the actual underlying mechanisms your library
would rely on. Later on you can move it to Boost after all of the required
functionality has been accepted there.

Regards Hartmut
---------------
http://boost-spirit.com
http://stellar.cct.lsu.edu

> 
> Let me know if that makes sense.
> Adam
> 
> If we make reentrancy mandatory, then we face the next challenge, which
> you
> > described above IIUC. The think the key points are here: (1) A task can
> > yield, and can point to an other task which is waited upon. (2) A task
> (but
> > not the thread) can "block" on the upstream queue, if it's empty. If
> it's
> > not clear, I'll describe these in greater length.
> >
> >
> 
> 
> > *Oliver*: I considered using coroutines, however I don't clearly see how
> > this would solve this problem exactly. IIUC, Coroutines provide a way
> for a
> > single thread to execute multiple methods. I'd need multiples threads
> > executing multiple methods in a similar fashion. I'll continue thinking
> > about this.
> >
> > I couldn't found it in the docs. Does Coroutine have any drawbacks?
> >
> > Thanks,
> > Benedek
> >
> 
> _______________________________________________
> Unsubscribe & other changes:
> http://lists.boost.org/mailman/listinfo.cgi/boost