Boost logo

Boost :

Subject: Re: [boost] [afio] Formal review of Boost.AFIO
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2015-08-27 21:47:08


On 27 Aug 2015 at 8:25, Sebastian Theophil wrote:

> Ok, let's revisit the original pattern code I mentioned:
>
> EXAMPLE A:
>
> shared_future h=async_file("niall.txt");
> // Perform these in any order the OS thinks best
> for(size_t n=0; n<100; n++)
> async_read(h, buffer[n], 1, n*4096);
>
> Niall,
>
> is parallel read (parallel writing maybe?) the only use-case where you want a shared_future?

The problem is continuations. You may only add one continuation to a
unique future. If you schedule 100 continuations onto the same
future, that is a problem.

> If I understand Thomas correctly, he doubts you need the shared_future
> semantics because one async operation hands down a single handle to the
> next continuation.
>
> Essentially something like:
>
> async_file(“niall.txt”)
> .async_read(buffer, length_to_read)
> .async_truncate(length_to_read)
> .async_close()
>
> Your counter example was an asynchronous *and* parallel read where you
> need to share the file handle (or rather the future<handle>) between
> parallel reads. Shouldn’t this be abstracted away in the API somehow? I
> can’t think of many file operations you want to do N times in parallel.
> Truncating a file in parallel several times doesnâ„¢t seem to make much
> sense :-)

You want to relax ordering on file i/o as much as possible for
optimum performance. It therefore should be assumed to be a very
common pattern, and during post-debugging optimisation you the
programmer is going to be _relaxing_ ordering as much as possible
without breaking your concurrency visibility.

> So why not make it:
>
> async_file(“niall.txt”)
> // Read 100 times asynchrnously and in parallel and provide lambda returning n-th buffer and offset:
> .async_parallel_read(100, [&](int nCount) { return std::make_pair(buffer[n], n*4096; })
> .async_truncate(length_to_read)
> .async_close()
>
> The 100 reads are internally not ordered but they can only begin once
> the file has been opened, they consume this handle together, and only
> after all reads are complete can we truncate.

But isn't this just shared_future?

> Is this not what you’re trying to do?

I'm all for unique futures over shared futures where the normal use
case is a 1 to 1 mapping of inputs to outputs.

However the case of one future to many futures, and many futures to
one future comes up a lot in file system programming. Sufficiently so
that I chose non-consuming (shared future) semantics as the default
for afio::future<>.get_handle(), but left consuming (unique future)
semantics for fetching any T in afio::future<T>.get().

Even in another thread with Gavin Lambert I showed how non-consuming
futures are needed to ignore errors as AFIO defaults to error
propagation. As futures either carry a handle or an error, as I
showed in that thread you can use the depends(precondition, output)
function to substitute a future output when a future precondition
becomes ready. That lets you always close or delete files even if an
error occurs.

If AFIO defaulted to unique future (consuming) semantics for
get_handle(), the user would have to explicitly convert the future to
a shared future in their code. I am questioning if that extra hassle
for the user is worth it.

Thomas has also argued that each async_* function should return only
a future specific to its return type, so instead of afio::future<T>
always transporting a future handle in addition to any future T, you
must supply (I would suppose) two futures to every async_* function,
one for the handle, the other as the precondition. I asked if he
could propose an API based on that model which is substantially
superior to the one used by AFIO, because if he can I'll take it.

I can imagine several API models based on unique futures as Thomas
advocated. Indeed, I whiteboarded design options about a year ago, as
I did genuinely try quite hard to make the current API entirely
unique future based. I came to the conclusion that it forced a whole
load of extra typing onto the library user for what I felt was very
little gain, and it made the AFIO API considerably more complex to
both understand and use correctly which had bad consequences on
client code complexity and maintainability. I therefore eliminated
that API design from consideration.

Now, Thomas and his former mentor are world experts in this field. I
take their opinions seriously, even if his former mentor has never
given my opinions or code on this stuff anything but ridicule and
mockery as you have seen in other threads. But I also suspect that
their disagreement with my approach comes from one of theory versus
boots-on-the-ground, and my view is that C++ is a dirty mongrel
language and therefore gets dirty mongel design patterns. In the end,
I ask this of C++ library design (in order of priority):

(i) Is it fun to program?

(ii) Will it bite you in the ass?

(iii) Is it very testable and easy to debug and maintainable?

(iv) Is it flexibly reusable to solve unanticipated problems?

(v) Does it perform well in space and time?

I don't care if I trample all over the priniciples of text book
proper library design if I get a "yes" to all those five questions.

In the end, the C++ compiler to me is just a fancy assembler macro
engine, and hence why I upset some people with my "mongrel" design
principles.

Niall

-- 
ned Productions Limited Consulting
http://www.nedproductions.biz/ 
http://ie.linkedin.com/in/nialldouglas/



Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk