Re: [boost] [asio] Tutorial 7 using strdup, new

From: Giovanni P. Deretta (gpderetta_at_[hidden])
Date: 2006-02-09 10:09:34

Christopher Kohlhoff wrote:
> Hi Giovanni,
> 
> --- "Giovanni P. Deretta" <gpderetta_at_[hidden]> wrote:
> 
>>As i said, i see, in this case, move as a correctness issue
>>than an optimization.
> 
> 
> Can you explain what you mean by correctness?
> 

The same reason that auto_ptr is movable. It cannot be copied because it 
wouldn't make any sense, but it is movable because it let an owned 
object escape the scope in witch it has been created. Certanly I 
wouldn't consider auto_ptr a value type object.

With auto_ptr not copying is not a problem of efficiency, but a problem 
of not having two auto_ptr owning the same pointer. I see it the same 
way with asio sockets. You can't have have two asio sockets because they 
would own the same internal socket_impl. They can't be copied because 
that wouldn't make any sense, but if they were movable, they could 
escape the scope where they have been created. To do this now you need 
dynamic allocation.

> 
>>I do not see how you can avoid locking if you are going for
>>intrusive refcounting. For example, thread A creates a
>>stream_socket and register it in a demuxer. The demuxer might
>>be shared between threads so it needs to lock the list mutex
>>to insert it in the queue. Then thread A might destroy the
>>socket at any time, so it need to relock the queue to remove
>>the socket. Am i missing something?
> 
> 
> But this locking is currently only needed when the socket object
> is constructed or destroyed, which are not usually common
> operations. 

You need locking *every time* you traverse the list, and this is going 
to be the most expensive locking.

 > Moving is something that could occur quite a lot.

Well, right now is not possible at all :). Swap would just add another 
option to the user. Certanly if it is really going to be expensive, 
probably it is not worth it.

> 
>>I have (*) one use case that would really benefit if asio
>>moved its callback instead of copying it. As the callback owns
>>the socket, it would need the socket to be movable, or at
>>least swappable.  And no, i do not think it is a wouldn't be
>>just "a cool thing"... it is obvious from asio examples that
>>support for moving (with support from bind of course :) ) an
>>auto pointer would be obviate the need of a shared_ptr.
> 
> 
> The callbacks are an example of why I think a move operation
> would have limited application. Move support can only be used if
> there is just one chain of asynchronous operations associated
> with the object. As soon as there is a second chain (e.g.
> running an async_wait on a timer while you do an async_read, or
> doing an async_read and async_write at the same time) the move
> support becomes useless.
> 

Even if you end up not making socket movable, could you make asio move 
the callback instead of copying it (may be a asio_move() customization 
function that by default copies and can be overloaded to move for 
specific user types)?

> 
>>I do not see how and why locking a mutex could trow?
>>Boost::thread mutexes do not throw.
> 
> 
> I see that it does not, but perhaps it should :) On Windows 2000
> or earlier the EnterCriticalSection function can fail in low
> memory situations. This is not handled at all in the
> boost::mutex class as far as I can see.
> 

Hum, looks really bad. I do not really know the win32 api, but i think 
that if ECS fails with a low memory error, it might be because the 
kernel has exhausted the internal memory. I do not think that an 
application has realistically any way to recover (like freeing more 
memory on a low memory situation in userspace).  May be the mutex should 
just fail (abort the application). Conceptualy a mutex CAN'T fail, it 
locks or wait.

> 
>>Btw, you could use a lock free list (not that i know how to
>>use it :)).
> 
> 
> I would categorise this as the "difficult" bit of "difficult and
> inefficient" :)
> 
> Seriously though, the more I think about move the more problems
> and limitations I see it creating for the implementation. Some
> more examples:
> 
> - An implementation that stores state direct in the socket
>   object may have pointers to that state in other data
>   structures. All these would need to be tracked and updated if
>   a move occurred.

They would need to be tracked when the object is destroyed any way. I 
think that having a pointer inside aio that points back to a user socket 
is not going to be fun.

> 
> - The data structures used by epoll and kqueue can contain a
>   pointer to user data. For efficiency, this could (and probably
>   should) point directly to the state that is embedded in the
>   socket object. This pointer can only be updated via a system
>   call.

Same reason above.

BTW, why not doing the other way around? asio user sockets are only 
trivial objects pointing to internal asio objects. These objects are 
kept in a list owned by the demuxer and are allocated using a fast 
allocator.  When the user creates a new object, asio allocates the 
implementation and puts it in a temporary (thread safe) insert list, 
then gives the user a pointer to this implementation. At the beginning 
of run() the demuxer thread acquires the insert list lock and splices 
all objects in the main list.
Deletion is done in the same way. When the user destroys the socket, the 
implementation is put inside a delete list. At the beginning of run() 
the demuxer removes all objects in the delete list from the main list, 
then deallocates all of them.
In fact instead of an allocator you could use a free list: the same 
mutex used for the insert list could be used to lock the free list. 
Allocating would just be a matter of moving the front object from the 
free list to the insert list. When the free list is empty, you just 
allocate a batch of new objects.

Having the implementation owned by the demuxer would simplify storing 
state in the object; have the guarantee that all pointer kept by epoll 
and kqueue (or whatever api) will never be dangling; have the guarantee 
that any file descriptor you have (in an epoll set for example) has not 
be reused (you defer the destruction of the socket until you have no 
references to it); it makes it easy to reuse SOCKETs on win32 (where 
they are expensive to allocate AFAIK): you never free them, just close 
and add them to the free list.

This solution requires the same amount of locking of your intrusive list 
when creating and destroying a socket, but the lock is going to be 
contended less often (expecially if every thread has its own 
insert/delete list to the demuxer). The demuxer need to lock only once 
per run() and only for a fast splice.

Ah yes, it makes the implementation of move/swap trivial and lockless :)

This is just a wild untested idea anyway, it might not be appropriate 
for the current asio implementation.

> 
> This discussion has actually been quite helpful for me in
> confirming why the sockets should not support move and swap...
> Sorry :)
> 

Glad to have been helpful :). I will not argue any longer that asio 
socket need to be movable. In the end you know the inner working of asio 
way better than me and certanly know what can and can't be done.

> 
>>Sure, but the highest socket value seen by a demuxer is not
>>very high (in the order of MAX_OPEN on sane systems).
> 
> 
> Hmmm, I dunno. I have used asio in servers that support more
> than 50000 concurrent connections. That's potentially a lot of
> wasted space!
> 

To handle that many connections you need lots of memory anyway. 200kb 
doesn't look that bad.

I'm looking forward to try next releaso of asio.

Good work.

-- 
Giovanni P. Deretta