[boost] [fiber] on schedulers and wakeups

Subject: [boost] [fiber] on schedulers and wakeups
From: Giovanni Deretta (gpderetta_at_[hidden])
Date: 2015-09-08 07:34:10

Hi Oliver,

I'm looking at the boost.fiber scheduling customization options. I have a
few comments.

My first concern is that scheduler is a global (per thread) property. This
makes sense, but it limits its usability in libraries, unless the library
completely owns a thread. It would be nice if schedulers where schedulable
entities themselves, so that they could be nested (more at the end). 

Also the description of the scheduler interface does not specify any thread
safety requirements. I assume that at least awakened must be thread safe as
the scheduling might be caused by a signal coming from another thread. Any
requirements should be specified properly. This leads to two additional points.

First of all, there should be a way to retrieve the scheduler associated
with a fiber: I haven't looked at the source, but the association must exist
internally, otherwise cross thread scheduling wouldn't work.

Second, there does not seem to be a way to allow signaled fibers to run in
the context of the signaling thread. This seems an important optimization.I
understand this is an explicit decision as currently it is not possible to
portably migrate fibers, but the option should be left to the user if he
knows it is safe in their setup. Possibly require 'awakened(fiber*x)' to
call x->get_scheduler()->awakened(x) if it does not support running the
fiber. My preference is to add a queryable boolean flag to each schedulable
entity that states whether it is allowed to move from its scheduler (this
makes a difference when you have multiple nested schedulers). Among other
things this would also prevent work stealing. The flag would by default be
set to prevent migration of course.

Now, what do nested schedulers give you? In addition to composability, you
can have the equivalent of an asio strand without explicit mutual exclusion.
Let say you have a bunch of fibers that all access the same resource; you do
not care where they run, as long as they never run concurrently. By binding
all of them to the same scheduler, the guarantee is implicit.

It should be possible to implement nested schedulers by having each fiber
set the current scheduler pointer if it differs and store the previous one.
On a scheduling event, if the current scheduler runs out of runnables, the
signal/yield code would restore the previous scheduler and try again. Of
course handling priorities and non strict fifo scheduling might be more
complex. Note that the multi scheduler logic need to be implemented by
Boost.Fiber itself.

Finally, it might be possible (but I haven't tested this) to make fibers
migrable on platforms where it normally isn't safe by allocating an actual
thread for each one. The thread would be stopped as soon as it has been
created, but a thread control block plush TLS entries would have been
allocated. The existing context switch would need to be augmented with the
ability to swap the thread context pointer. 

HTH,

-- gpd