Re: [boost] [afio] Formal review of Boost.AFIO

Subject: Re: [boost] [afio] Formal review of Boost.AFIO
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2015-08-25 17:39:40

On 25 Aug 2015 at 21:10, Thomas Heller wrote:

> > You're not, by any chance, saying that you think the AFIO design
> > should be 1 afio::handle = 1 fd? And when you copy construct
> > afio::handle, that calls dupfd()?
> 
> That is what I would expect from a handle copy constructor, yes. Note 
> that there are also move operations and a user is still free to store 
> handles in any "container" (including smart pointers or whatever).

Just reminding you I won't be online after this until Friday ...

You may not have considered that:

1. On POSIX file descriptors held by a process is limited, especially 
on OS X which has a crazy low limit.

2. Metadata updates are not guaranteed to be immediately consistent 
across file descriptors. E.g. fstat() on one fd can return very 
different information from fstat() on another fd even to the same 
file in the same process. This particularly happens with NTFS and 
ZFS, and is a major pain in lock free file system concurrency. Making 
this go away most of the time by reusing the same fd for end users is 
a big win.

3. In scatter gather file i/o there is no concept of file position, 
and therefore there is no good reason to needlessly duplicate fds 
when one will do.

4. AFIO has to "spend" fd's on workarounds for annoying problems on 
networked filing systems. For example, every byte range locked file 
needs to create a shadow file which is locked rather than the real 
file. This makes the problem of running out of fd's even worse.

5. Because of the need to pin lifetimes of things to one another due 
to things like shadow lock files and delete-on-close semantics, 
you're going to need a reference counting system one way or another. 
You can either hide that internally, or expose it. I chose to expose 
it, as I suspect it is more useful to end user code who also need to 
manage lifetimes.

Finally I would also suspect that dupfd() is going to be a lot slower 
than a shared_ptr increment, though granted you'd only call it 
non-frequently hopefully (unless you put a handle in a C++ 98 
container implementation).

> > Ah I think I'm beginning to understand now. You're assuming a world
> > where only std::future and std::shared_future exist and the
> > relationship between them is well understood.
> >
> > By which, I infer you're saying that one does not need anything more
> > than std::future and std::shared_future in an asynchronous filesystem
> > and file i/o library?
> 
> I didn't say there is no place for other things than std::future or 
> std::shared_future. You should know that ...
> BUT if you claim conformance to the Concurrency TS ...

I claim conformance to a suite of *extensions* to the Concurrency TS. 
AFIO needs to be able to schedule multiple continuations on a future, 
that's the single biggest extension I need. The Concurrency TS 
doesn't allow that because the first continuation is expected to 
consume the future.

The ability to avoid exception_ptr is nice as well though. 
exception_ptr is expensive.

> > And therefore, the current approach taken by AFIO of custom futures
> > is wrong?
> 
> ... a design that clearly does not conform to anything defined in the 
> standard and only the names coincide, then yes, I consider the design 
> broken. In addition, I see no shortcomings in the design of 
> std::future/std::shared_future that wouldn't fit your usecase.

Do bear in mind that up until v1.3, AFIO was using std::future and 
std::shared_future. I didn't go off and extend the Concurrency TS 
without two years of experience of what exactly needs extending in 
the TS and why.

My extensions are actually very conservative because I remain 100% 
compatible with the TS. If you look at what others are doing to 
future-promise e.g. Sean Parent, they are reworking the fundamental 
design.

> > Would this summary of your criticism be correct? If it is, I can
> > address all those points, but it'll probably be Friday as tomorrow I
> > am away and on Thursday we have no electricity all day, so no
> > internet nor computers.
> 
> It's a very short summary of it, yes. Please keep in mind that all I 
> said was under the assumption that anything "future" like in AFIO (or 
> the depend NotMonad library) is just like a shared_future/future with 
> some extensions (you repeated that multiple times on the list).
> If you have something different that's totally fine but needs other and 
> special attention because fighting an uphill battle against an ISO 
> standard is not easy. Especially after all the claims that have been made.

That makes sense. And changes my perspective on what the 
documentation needs to focus on. Thanks.

Niall

-- 
ned Productions Limited Consulting
http://www.nedproductions.biz/ 
http://ie.linkedin.com/in/nialldouglas/