[Boost-bugs] [Boost C++ Libraries] #7837: Datagram packing of UDP BufferSequences

Subject: [Boost-bugs] [Boost C++ Libraries] #7837: Datagram packing of UDP BufferSequences
From: Boost C++ Libraries (noreply_at_[hidden])
Date: 2012-12-29 19:26:13

#7837: Datagram packing of UDP BufferSequences
----------------------------------------+-----------------------------------
 Reporter:  rafael_gago_81@…            |       Owner:  chris_kohlhoff
     Type:  Feature Requests            |      Status:  new           
Milestone:  To Be Determined            |   Component:  asio          
  Version:  Boost 1.52.0                |    Severity:  Showstopper   
 Keywords:  udp; asio; scatter-gatther  |  
----------------------------------------+-----------------------------------
 Hello!

 First of all to congratulate you for the great library design.

 I'm finding a limitation that makes boost asio udp use impractical.

 I'm writing an UDP based simple transfer protocol in a context where I
 want to avoid TCP latency (excessive ACK's between endpoints with high
 RTT), rate limitings and congestion control mechanisms. I also want to be
 able to set the datagram size (directly or indirectly, in my side of
 course) to minimize the bit error rate and collissions when doing wireless
 transmissions.

 I do transfer big data sizes (files) in fixed size chunks to control the
 UDP datagram size. Doing this requires that each chunk has a header with
 its transaction, chunk number, etc.

 What I'm doing now is allocating/storing the chunk headers storage/data
 and passing ConstBufferSequences in pairs (chunk header + data chunk) to
 "async_write_to". I manage the chunk headers, the data buffer in the
 sequence belongs to external buffers, so I can avoid copying using the
 scatter-gather feature.

 I have segregated storage pools both for the requests (using the
 CustomHandler api) and the message headers, the data is taken directly
 from external buffers.

 I think that this is the most memory efficient I can go with the current
 version. Unfortunately there is a missing feature for me.

 Let's hipotetically say I want to transfer a 100Mb file in chunks of 512
 bytes (payload). My custom segregated storage handler allocator tells me
 that in my platform (linux) each handler is 92 bytes.

 If I submit all the work at once that means (104857600 bytes) / (512
 bytes/chunk) = 204800 chunks, which equals 204800 * 92 = 18841600 bytes,
 18Mb of stored handlers more or less. A 18% of the transfered size.

 It would be very handy to have a way to separate individual non-blocking
 udp transfers in a ConstBufferSequence just using a single
 handler/request. That would give more reasonable memory requirements.

 With tcp this isn't a problem, it is a matter of passing all the sequence
 and forget about it. Unfortunately the typical use case of UDP is a bit
 different.

 Two ideas come to my mind.

 -To add another overload to async_sent_to, which specifies a numeric
 parameter saying how many buffers belong to each udp tranfer.
 -To a new Sequence type (would make sense when using tcp?) or a buffer
 type that is a separator.

 I was just curious and I modified the source a little using the first
 approach, unfortunately I had no time to figure out how could it work in
 windows, so I desisted because I need to support both platforms.

 What i did was a new class/file
 "detail/asio_reactive_socket_send_to_periodic.hpp" copying the source code
 of "detail/asio_reactive_socket_send_to.hpp", adding a template parameter
 named "BuffersPerDatagram" and modifying the "do_perform" function. I used
 a stl container as ConstBufferSequence:


 {{{
 static bool do_perform(reactor_op* base)
 {
         reactive_socket_sendto_periodic_op_base* o(
 static_cast<reactive_socket_sendto_periodic_op_base*>(base));

         bool result = true;
         typename ConstBufferSequence::iterator it = o->buffers_.begin();

         std::size_t bytes_transfered = 0;

         while (result && (it != o->buffers_.end()) && !o->ec_)
         {
                 boost::array<typename ConstBufferSequence::value_type,
 BuffersPerDatagram> arr;

                 for (std::size_t i = 0; i < BuffersPerDatagram; ++i)
                 {
                         std::memset (&arr[i], 0, sizeof (typename
 ConstBufferSequence::value_type));
                         if (it != o->buffers_.end())
                         {
                                 arr[i] = *it;
                         }
                 }

                 buffer_sequence_adapter<boost::asio::const_buffer,
                         ConstBufferSequence> bufs(arr);

                 result = socket_ops::non_blocking_sendto(o->socket_,
                           bufs.buffers(), bufs.count(), o->flags_,
                           o->destination_.data(), o->destination_.size(),
                           o->ec_, bytes_transfered);

                 o->bytes_transferred_ += bytes_transfered;
         }

         return result;
 }
 }}}


 After this it was a matter of call forwarding, but because I don't know
 the code base and what exactly does I preferred to submit a feature
 request, as I'm not the only one that found this limitation:

 http://comments.gmane.org/gmane.comp.lib.boost.asio.user/2987


 It starts after the third message.

 If help is needed to implement it I can do it if someone that has the
 library knowledge guides me.

 Of course I can be missing an alternate way of doing things too. I did
 some research before posting and I wasn't able to find a way.

 Best regards!
 Rafa

-- 
Ticket URL: <https://svn.boost.org/trac/boost/ticket/7837>
Boost C++ Libraries <http://www.boost.org/>
Boost provides free peer-reviewed portable C++ source libraries.