[Boost-users] dev_poll_reactor vs. strand and async_write

Subject: [Boost-users] dev_poll_reactor vs. strand and async_write
From: Chris Hite (C.Hite_at_[hidden])
Date: 2009-06-04 05:47:06

In a project using asio I noticed performance issues on Sun vs. Linux.  
It turned out that dev_poll_reactor is doing tones of system calls 
always pairs of write() and ioctl().  Stranger still is that the calls 
to write() are for 0 bytes.

I was able to avoid the issue by switching from strand to mutex.  I 
think there is some bug or logic error in dev_poll_reactor so I wrote a 
test program to narrow down when it happens:

//

Christopher Hite

 
RTS Realtime Systems Software GmbH, Rembrandtstrasse 13, D-60596 Frankfurt am Main 
T: +49.69.61009.0 / F: +49.69.61009.181 

Sitz: Frankfurt am Main - HRB 84467 Amtsgericht Frankfurt am Main
Geschäftsführer: Steffen Gemuenden, Igor Sluga

www.rtsgroup.net

This email and any attachments are for the exclusive and confidential use of the intended recipient. If you are not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, please do not read, distribute or take action in reliance upon this message. If you have received this in error, please notify me immediately by return email and promptly delete this message and its attachments from your computer system. 
-------------------------------------------------------------------------------
#include <boost/asio.hpp>
#include <iostream>

using namespace boost::asio;

io_service service;
io_service::strand strand_(service);
local::stream_protocol::socket s0(service);
local::stream_protocol::socket s1(service);

void wh(const boost::system::error_code& ec,const std::size_t){
    if(ec)
        std::cout<<ec; //never happens
}

void sendAByte(){
    static char sendBuf[256];
    async_write(s0,buffer(sendBuf,1),wh);
}

int main(){
    local::connect_pair(s0,s1);

    std::cout<<"dispatching\n";
    for(int i=0;i<100000;i++)
#if 1
        strand_.dispatch(sendAByte);
#else
        sendAByte(); //doesn't cause the sys calls
#endif

    std::cout<<"waiting on run to finish\n";
    service.run();
    return 0;
}
//-------------------------------------------------------------------------------

Here are some dtraces to help analyze what's going on $XX is the 
executable.  This one shows all the calls to write() with size=0.  
async_write uses sendmsg() so all these are attributable to 
dev_poll_reactor.

dtrace -n 'syscall::write:entry/pid==$target&&arg2==0/{ ustack(); }' -c $XX

  0   5467                      write:entry
              libc.so.1`_write+0x8
              
strandtest`__1cFboostEasioGdetailQdev_poll_reactor4b_Drun6Mb_v_+0x1b4
              
strandtest`__1cFboostEasioGdetailPtask_io_service4n0CQdev_poll_reactor4b___Gdo_one6Mrn0CLscoped_lock4n0CLposix_mutex___pn0EQidle_thread_info_rn0AGsystemKerror_code__I_+0x204
              
strandtest`__1cFboostEasioGdetailPtask_io_service4n0CQdev_poll_reactor4b___Drun6Mrn0AGsystemKerror_code__I_+0xa8
              strandtest`main+0xe8

This one counts the system calls.  What's very disturbing is that the 
longer you let it run the more sys calls are made.  Why doesn't the 
system block waiting for the socket to become write-able again (which it 
won't)?

dtrace -n 'syscall:::entry/pid == $target/ { @num[probefunc] = count(); 
}' -c $XX
  fcntl                                                             6
  brk                                                            2474
  sendmsg                                                       21505
  ioctl                                                         60194
  write                                                         60195
  lwp_sigmask                                                  120382

Similar testing on Linux(x86) using strace doesn't show so many system 
calls (except a few for locking mutexes).

Perhaps Christopher Kohlhoff or whoever implemented the thing can take a 
look.  I'm guessing there some event which keeps getting detected but 
isn't processed.  Would someone please take a look or do I need to debug 
it myself?