Re: [Boost-users] [boost] Boost.Fiber mini-review September 4-13

Subject: Re: [Boost-users] [boost] Boost.Fiber mini-review September 4-13
From: Hartmut Kaiser (hartmut.kaiser_at_[hidden])
Date: 2015-09-05 12:18:06

>> Hmmm, this is not what I would have expected, frankly. One of the main
>> advantages of Fibers are their low creation/context switching/termination
>> overhead. I would have expected for Fibers to be freely movable between
>> kernel-threads (work-stealing!), even more so as you expose an interface
>> conforming std::thread... If Fibers look like threads, they should behave
>> accordingly.
> 
> but this is what C++ (incl. C++14) gives to us- compilers are free to
> optimize
> as they deserve.
> I hope you agree that is not a restriction coming from boost.fiber

The restriction that fibers can't move is imposed by the library. The question is whether the user code relies on TLS or not.

>> I think that it is too over-constraining to inhibit moving Fibers between
>> kernel-threads just to maintain NUMA awareness (not all architectures have
>> those in the first place). The decision whether to move a Fiber or not
>> should be entirely left to a scheduler/executor (which could confine a
>> Fiber
>> to a NUMA domain, for instance, if needed).
> 
> I agree but the compiler optimizations will break the application! Even if
> it looks
> like correct code!

Only if the user does not know what he/she is doing. If all my Fiber is executing is a simple self-contained operation which does not rely on any contextual data (which in my experience is a large part of code which was written with parallelism/concurrency in mind), I usually don't care which core is used to run this code on (sans NUMA placement issues, which is a completely orthogonal can of worms).

>> Also, it should be made very clear that the use of Fibers has certain
>> implications on using TLS (in short - don't use TLS, but FLS instead)
> 
> user cod, executed by fibers, might contain TLS. especially if thrid-party
> libraries
> is used you might not aware if it uses TLS or not.
> companies legacy code is not likely changed
> 
>> Thus the fact whether compilers cache the TLS or not is mostly irrelevant.
> 
> I strongly disagree!

All I can say is that allowing to move Fibers between kernel-threads works very well for us (HPX) and is one of the enabling functionalities for very high resource utilization in highly parallel applications. As I said, work-stealing (i.e. moving Fibers to different cores - and as in HPX every core has its own dedicated thread - this means moving them to different threads) ensures that cores which run out of work can be kept busy. In certain contexts this is even beneficial if you steal work across NUMA domain boundaries.

All of this certainly assumes that the user does not rely on TLS - and I agree this puts some restrictions on what can be done.

As a Boost.Fibers library however, the decision whether to allow moving Fibers or not should not only be left to a separate scheduler/executor (as mentioned before), but in the first place this decision should be left to the user of the library. If you know it's safe to move Fibers as no TLS is involved, why not allow doing so?

Regards Hartmut
---------------
http://boost-spirit.com
http://stellar.cct.lsu.edu