Re: Implemented support for recvmmsg / sendmmsg (on supported operating systems: Linux / *BSD / AIX / QNX)

From: Virgilio Fornazin (virgiliofornazin_at_[hidden])
Date: 2023-01-16 18:28:16

Hi Niall

The ASIO development started some work on working with 'registered buffers'
using lib io_uring (on Linux).

Windows RIO support need to ensure some thread semantics that must be
checked if they are compatible with ASIO model. For standard IOCP mode,
WSASendMsg is analog to sendmsg, and it's already implemented on ASIO.

My patch was develop because I need to implement a high-volume UDP service
and I reached the limit of sendto/recvfrom calls/CPU core/second on Linux
(about 35k  calls/datagrams on my box). With sendmmsg / recvmmsg, I easily
reaches 200k datagrams packets/second perform about 6k sendmmsg/recvmmsg
calls on same hardware). Using SO_REUSEPORT, with 12 sockets, without
sendmmsg / recvmmsg, I reached about 190k datagrams/s using 13 cores. Using
1 receive and 5 senders, reached 1 million datagrams / second (I've reached
a peak 12.5 gigabit/second transfers, without any packet loss, using 64MB
send/receive buffers).
To reach this kind of performance, very specialized programs are needed
(busy pooling udp sockets, pin each socket process to a cpu core, also pin
RSS-queue to same CPU core (you can read some material on that on
Cloudfare's blog
https://blog.cloudflare.com/how-to-receive-a-million-packets ... it's not a
trivial task).

The recvmmsg / sendmmsg also suppots TCP on some OS's, so it raised the of
I/O performance to another level.

On Mon, Jan 16, 2023 at 12:17 PM Niall Douglas via Boost <
boost_at_[hidden]> wrote:

> On 14/01/2023 18:41, Virgilio Fornazin via Boost wrote:
>
> > I finished the first version of the patch. It's able to send and receive
> > 1million packet / second on my box at same time (an intel i9 9900k /
>  ubuntu
> > 22.04 lts low latency kernel) using packet sizes of 64 / 1430 bytes
>  without
> > packet loss (using 64mb udp socket buffers)
>
> A good example of why support for one million scatter-gather buffer
> lists is a useful thing.
>
> Did you consider implementing the same support for Windows? It's more
> work than it would be on Linux as you'd need to extend the ASIO IOCP
> reactor with RIO, but you'd get similar performance.
>
> Niall
>
> _______________________________________________
> Unsubscribe & other changes:
> http://lists.boost.org/mailman/listinfo.cgi/boost
>