Re: [boost] [asio] Bug: Handlers execute on the wrong strand (Gavin Lambert).

Subject: Re: [boost] [asio] Bug: Handlers execute on the wrong strand (Gavin Lambert).
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2013-10-25 18:09:27

On 25 Oct 2013 at 19:39, Gavin Lambert wrote:

> >> upside most of the guts are entirely lock-free (though not wait-free,
> >> since it's based on Boost.LockFree's queue).

lockfree::queue isn't actually hugely performant. Most lock free code 
isn't compared to most lock based implementations because you gain in 
worst case execution times by sacrificing average case execution 
times. The only major exception is lockfree::spsc_queue which is 
indeed very fast by any metric.

> > Asio at Windows uses IOCP (default settings, using asio::io_service for task
> > scheduling) and that is the (theoretical) reason of better thread scheduling
> > for the Asio-based thread pool. Sometimes it's really visible.
> 
> It's also full of mutexes though, which is why it didn't work out for 
> me.  (Note that I was using Boost 1.53 when testing Asio; maybe this has 
> changed in future versions, although I heard that 1.54 picked up a bug 
> in the IOCP reactor.)

I saw lost wakeups during parallel writes in ASIO 1.54, so I disabled 
those for AFIO. That appears to be fixed in 1.55, so AFIO now 
parallelises everything as it was designed to. This might mean ASIO 
in 1.55 is fixed.

> I'm not sure exactly which lock triggered the slow path (my logging was 
> only sufficient to show that it was one of the ones inside Asio, but not 
> which one).  But as the prior email said, given reuse of strand 
> implementations between supposedly independent strands, that seems like 
> a likely candidate.  (Though it didn't take long for the latency spikes 
> to manifest -- typically they'd start after a couple of minutes and then 
> recur roughly every 10-30 seconds.)

ASIO is, once you compile it with optimisation, really a thin wrapper 
doing a lot of mallocs and frees around Win IO completion ports. Any 
latency spikes are surely due to either IOCP or the memory allocator 
causing a critical section to exceed its spin count, and therefore go 
to kernel sleep?

> I haven't done a head-to-head benchmark on each (and it wouldn't 
> surprise me if Asio were faster than mine for many loads -- and it's 
> definitely more flexible than I made mine) but so far my one is doing at 
> least as well as Asio on production loads but without the latency spikes 
> from the locks.  Still very early days yet though.

If you're on Haswell, you might look into my memory transaction 
implementation in AFIO. It uses Intel TSX if available according to 
runtime detection, otherwise it falls back onto a policy composed 
spin lock (yes I know I did NIH with yet another Boost spinlock 
implementation, but hey mine is policy composed so you can vary spin 
counts etc!!!). It works on Intel's TSX simulator, but I would really 
love to know if it works on real TSX hardware.

Niall

-- 
Currently unemployed and looking for work.
Work Portfolio: http://careers.stackoverflow.com/nialldouglas/