Re: Reimplementing ASIOs

From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2024-07-18 21:47:07

On 18/07/2024 21:58, Christian Mazakas via Boost wrote:

> Actually, it might be nice to compare implementation notes. For Fiona, I
> create a buffer
> sequence for the user by over-allocating for each buffer and making an
> intrusive doubly-linked
> list. This means that the user gets a mutable buffer sequence without the
> need for any intermediate
> allocations.

Instead of over-allocating and wasting a page, I would put the link 
pointers at the end and slightly reduce the maximum size of the i/o 
buffer. This kinda is annoying to look at because the max buffer fill is 
no longer a power of two, but in terms of efficiency it's the right call.

> It also enables nice constant time push/pop and slicing. I was heavily
> inspired by Rust and Asio when
> coming up with the buffer abstraction:
> https://github.com/cmazakas/fiona/blob/main/test/buffer_test.cpp

Surely for reading you want io_uring to tell you the buffers, and when 
you're done, you immediately push them back to io_uring? So no need to 
keep buffer lists except for the write buffers?

There is a further optimisation trick for writes: if you allocate the 
write buffers out of a memory mapped anonymous inode, you can use 
sendfile offload. This isn't actually always a win surprisingly enough, 
if your write buffers are big then it's a big win, but if they're small, 
it usually isn't. I try to aim for write buffers of 8Mb and I try to 
fill them to max before sending them.

Re: scatter-gather buffers, generally if somebody is doing more than one 
buffer at a time, they're working with wider data structures in the 
program that usually don't need to be in registered i/o buffers i.e. 
locked memory. I've found dropping support for registered i/o buffers if 
there is more than one struct iovec is generally pain free.

>   If you'd like, I'd really appreciate any implementation feedback on what's
> going on here. Not many
> seem to have the expertise in io_uring and this kind of stuff so it's rare
> when I get to really talk shop.

I can definitely agree on that!

> Maybe I need to actually sit down and write some docs because I realize
> there's a lot to the codebase here
> and it's hard to convey everything just using email.

Generally if you've made these things before, the concepts and patterns 
all are familiar and it's easy to understand each other.

If you've never done it before, yeah it can be impenetrable.

Niall