Re: [Boost-users] [Asio] Making custom std::streambuf for async_read

Subject: Re: [Boost-users] [Asio] Making custom std::streambuf for async_read
From: Gavin Lambert (gavinl_at_[hidden])
Date: 2016-08-10 20:28:33

On 11/08/2016 07:16, knarks_at_[hidden] wrote:
> Thank you for the response so far. My idea is to prevent unnecessary
> copy operations when receiving large files with tcp. For this reason I
> thought about implementing my own stream, which writes direct to file or
> memory, depending on the situation.
>
> I know that I can connect std::iostream to the asio streambuf but what
> about moving the received memory to a std::ofstream or std::stringstream
> without having the data twice in memory. Is that possible?

ASIO does provide posix::stream_descriptor and windows::stream_handle, 
which can wrap native file descriptors and handles (resp.) and provide 
async operations on them.  Thus you can read from the network into one 
asio streambuf and then pass the same streambuf to the file write 
operation.  This is probably as close to zero-copy as you can get.

Though other than scalability and blocking concerns there's not much 
difference between this and just doing a blocking write to a regular 
fstream -- at the point you're hitting disk the cost of copying memory 
is negligible.

For writing direct to memory, you can simply supply the target memory as 
a buffer to the initial read (making sure to keep track of the origin 
and size correctly as data blocks incrementally arrive, remembering that 
short reads are likely).  This will quite literally be zero-copy (bar 
any copies that occur inside the network stack); no special streams 
required.