Re: [Boost-users] [boost] Boost.Fiber mini-review September 4-13

Subject: Re: [Boost-users] [boost] Boost.Fiber mini-review September 4-13
From: Hartmut Kaiser (hartmut.kaiser_at_[hidden])
Date: 2015-09-05 11:31:59

> > Oliver, do you mind providing the rationale for why fiber migration has
> been
> > dropped or point me to the section in the docs discussing this?
> 
> http://olk.github.io/libs/fiber/doc/html/fiber/rationale.html#fiber.ration
> ale.migrating_fibers_between_threads

I'm quoting from the link you gave:

<quote>
Support for migrating fibers between threads was removed. Especially in the
case of NUMA-architectures, it is not always advisable to migrate data
between threads. Suppose fiber f is running on logical CPU cpu0 which
belongs to NUMA node node0. The data of f are allocated on the physical
memory located at node0. Migrating the fiber from cpu0 to another logical
CPU cpuX which is part of a different NUMA node nodeX will reduce the
performance of the application because of increased latency of memory
access. 

A more important aspect is the problem with thread-local-storage (TLS).
Instead of recomputing the address of a TLS variable, a compiler might, as
an optimization, cache its previously-computed address in various function
stack frames.[2] If a fiber was running on thread t0 and then migrated to
thread t1, the cached TLS variable address(es) would continue pointing to
the TLS for thread t0. Bad things would ensue.
</quote>

Hmmm, this is not what I would have expected, frankly. One of the main
advantages of Fibers are their low creation/context switching/termination
overhead. I would have expected for Fibers to be freely movable between
kernel-threads (work-stealing!), even more so as you expose an interface
conforming std::thread... If Fibers look like threads, they should behave
accordingly.

I think that it is too over-constraining to inhibit moving Fibers between
kernel-threads just to maintain NUMA awareness (not all architectures have
those in the first place). The decision whether to move a Fiber or not
should be entirely left to a scheduler/executor (which could confine a Fiber
to a NUMA domain, for instance, if needed).

Also, it should be made very clear that the use of Fibers has certain
implications on using TLS (in short - don't use TLS, but FLS instead). Thus
the fact whether compilers cache the TLS or not is mostly irrelevant.

Regards Hartmut
---------------
http://boost-spirit.com
http://stellar.cct.lsu.edu