[boost] End of AFIO review thoughts (was: Re: [Boost.AFIO] Formal Review)

Subject: [boost] End of AFIO review thoughts (was: Re: [Boost.AFIO] Formal Review)
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2015-09-01 21:50:02

On 1 Sep 2015 a 11:41, Joel FALCOU wrote:

> As a concluding note, I *really* want Niall to keep going on making AFIO 
> the reference async file I/O library. Just take the needed time to 
> reflect on the afformentionned PAI and implementation issues and 
> remember that there is no rush to get there.

Thanks for the review Joel.

This final point is an interesting one, and I think it is worth 
discussing a bit further, though everyone note I start my holidays 
away from Boost on Thursday and you won't see me again here until 
GSoC starts up again in January 2016.

Race free filesystem is far more useful to the C++ community than 
async file i/o which really does not deliver what people think it 
does without completely rearchitecting their application. Race free 
filesystem would be a big value add for reliability and 
predictability for any code using the file system because it very 
substantially reduces the chance for surprise and surface for 
security attacks.

As Beman mentioned a long time ago, for race free filesystem we need 
to standardise on some method of representing open file handles, and 
then somehow figure out how that fits nicely with STL iostreams. This 
is a big mess of difficult and hard, because everybody will have an 
opinion right from Boost up to WG21, and moreover most people's 
opinions will have no technical basis for being better than most 
other people's opinions. There are many, technical merit equivalent, 
ways to skin that cat, and piloting a compromise suitable to enough 
people to be standardisation capable would be a big multi-year 
commitment.

Nevertheless, such a library would be of general value to the C++ 
community, and moreover WG21 is keen on the idea. It would probably 
be just as slow as AFIO's file path consuming APIs i.e. a lot slower 
than Boost.Filesystem or STL iostreams path operations, and therefore 
probably cannot wholly replace the existing solutions (unless POSIX 
fixes the missing functionality we have to workaround with inode 
comparing loops - I really wish POSIX would take note of Windows more 
on file system, Windows has for the most part a far superior 
implementation). But I think the knock on consequences on modernising 
STL iostreams to better fit today's C++ world would be enormously 
valuable, and indeed the source of much disagreement and debate.

If anyone wished to volunteer to design and write such a library and 
take it eventually for standardisation, I would be more than happy to 
act as an advisor with hard experience from the coal face, but not as 
an implementor. You may be surprised to hear that I don't much care 
for conflict, and getting that library past review would involve a 
whole ton of conflict and I just don't have the energy given my plans 
for the next few years. Other fish to fry, as I will explain.

Regarding AFIO, as you all now realise it is not the library you 
thought it would be, but my big hope was and is that this perhaps too 
early review has given you all a chance to ponder what a next 
generation standardised C++ file system and file i/o library ought to 
be, and that might bear fruit in any later review of AFIO in months 
or years to come. Many of the reviews felt something like what ASIO 
provides for seekable async streams makes a lot more sense through 
virtue of being much ligher weight - yet, as I countered, if you want 
that then ASIO already has that ready to go for you.

I aim for AFIO to bring something very new to the table: a toolkit of 
fundamental portable primitives for bare metal programming of file 
systems where "bare metal" refers to the direct, unmodified exposure 
of portable filing system semantics to the programmer rather than 
"bare metal" in terms of least implementation. Where a particular 
platform differs from the standardised abstract portable model 
presented by AFIO to the programmer, emulation is performed to make 
up the shortfall, and hence the apparent complexity and overheads 
relative to a least implementation solution.

That comes, unavoidably, with a very different weighting of what 
complexities and relative tradeoffs various operations have than you 
are used to, and it will wreck you head initially. But I am confident 
that after a few months of it getting into your head space, it'll 
start to make a lot more sense especially given the relative fixed 
and variable costs of operations on the file system relative to one 
another, and the next time you see AFIO it'll hopefully all just make 
more sense why it does what it does in the way it does it.

*Especially* when you see the standardised C++ transactional 
versioned key-value store based on AFIO I have planned, and the 
enormous world of new C++ program designs you can do when you can 
just *assume* you have always reliable transactional persistent 
versioned storage built into the language runtime and which works 
perfectly on everything from embedded C++ right up to the largest 
mainframes, works during process bootstrap before main() is called, 
copes well with power loss and store damage, and has no issue with 
concurrent users on heterogeneous platforms making scaling up from 
tiny SSDs to large distributed SANs trivial.

Anyone who has ever programmed on Zope 
(https://en.wikipedia.org/wiki/Zope) will need no convincing of just 
what a paradigm shift a standardised scalable transactional 
persistent store in the runtime affords, and this is why I came out 
of lurking on boost-dev to develop these libraries after being 
adjunct to the community here since 2002 or so. I have had enough 
dealing with inherently unreliable data storage in my applications. I 
want truly reliable storage supplied as standard in every major 
programming language runtime available. That's my end goal, and AFIO 
is but a vital step along that path.  

Anyway, I'll stop boring you now. I'll be around here until Thursday 
morning British Standard Time for any questions etc, then I'll be 
deactivating my boost-dev subscriptions until Jan 2016 to take a long 
overdue break from Boost library development which has sucked down 
something close to 700 hours since February. Private email will 
continue to work of course, and I'll be tipping away at no more than 
eight hours per week on Boost.Outcome and the AFIO engine rewrite 
until they're done. I'll monitor the Fiber mini-review discussion 
starting shortly, but I can already give my vote: it's a YES to 
acceptance (I have been following Fiber closely to ensure lightweight 
futures will work with it. It's a definite yes, and my congrats to 
Oliver for doing such a great job).

Thank you *everybody* for the reviews of AFIO. They have been 
enormously valuable, and I am tremendously grateful to you all for 
taking the time to review my library.

Niall

-- 
ned Productions Limited Consulting
http://www.nedproductions.biz/ 
http://ie.linkedin.com/in/nialldouglas/