[boost] [afio] status update (was: Re: File Handle and FILE* wrappers?)

Subject: [boost] [afio] status update (was: Re: File Handle and FILE* wrappers?)
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2014-12-02 14:23:58

On 29 Nov 2014 at 14:59, Bjorn Reese wrote:

> On 11/26/2014 08:10 PM, Olaf van der Spek wrote:
> > Not many responses...
> > Does noone use native IO?

The other problem with anything Boost.Iostreams based is that 
Iostreams has a severely flawed design in not supporting async i/o. 
This makes Iostreams useless for almost any task I've needed it for 
in recent years. It also doesn't help that it lacks a maintainer, and 
has many annoying bugs (I recently had to use Iostreams due to 
Boost.Process 0.5 forcing me, and I lost over a week of my time 
fighting Iostreams to behave when combined with ASIO. It was quite 
frankly a disaster).

> The reason I have not responded is because I would much rather have an
> asynchronous file I/O API (along the lines of Asio) with synchronous
> convenience functions. But this is a major task.

As Bjorn only knows too well but Olaf may not, proposed Boost.AFIO 
(https://boostgsoc13.github.io/boost.afio/doc/html/afio.html) 
provides portable asynchronous file i/o implementation library 
extending ASIO.

It however does not use the ASIO handler callback idiom as that is 
(in my opinion, though TBH I am right on this) unsuited for file i/o 
where _ordering_ of operations is paramount. The ability to easily 
program ordering makes an enormous difference to ease of use and 
tweaking algorithms for performance on cold and warm cache file 
systems.

Instead AFIO uses a functional result passing design currently based 
on plain old futures. Benchmarks for the v1.2 engine are more than 
plenty for even a top end SSD 
(https://boostgsoc13.github.io/boost.afio/doc/html/afio/FAQ/closure_pe
rformance.html) but latencies are in my opinion not good enough 
(https://boostgsoc13.github.io/boost.afio/doc/html/afio/FAQ/closure_la
tency.html), so the work plan is as follows:

v1.3 engine: Port Boost.AFIO to proposed Boost.BindLib, thus making 
AFIO standalone capable. This part is nearly done, it's mainly all 
dotting i's and crossing t's to go, mainly on older compilers like 
GCC 4.6. The red items on the CI dashboard 
(https://boostgsoc13.github.io/boost.afio/) are those todo fixme 
support items apart from anything before VS2013 support for which I 
am dropping.

v1.4 engine: Replace spinlocked unordered_map with proposed Boost 
concurrent_unordered_map. This should yield a substantial performance 
increase due to hugely improved concurrency and a large decrease in 
latencies due to eliminating much use of malloc.

v1.5 engine: Replace async_io_op with a custom future type based on 
the non-allocating constexpr continuations based basic_future I 
proposed on this list some time ago. This should completely eliminate 
malloc from the AFIO core engine, and I would assume we shall see 
perfectly optimal performance and latencies in AFIO. A "free" bonus 
is that AFIO becomes monadically programmable using either proposed 
Boost.Hana or proposed Boost.Expected, or indeed any monadic C++ 
programming library of your choice (if it provides basic_future 
support). This ought to make "simple" use of AFIO much more amenable 
and its current batch based API is a big gnarly for occasional users.

If approved to present at C++ Now, I shall present on all this there 
for those interested, and I currently intend to submit a talk or two 
to CppCon.

Niall

-- 
ned Productions Limited Consulting
http://www.nedproductions.biz/ 
http://ie.linkedin.com/in/nialldouglas/