Re: [boost] NuDB: A fast key/value insert-only database for SSD drives in C++11

Subject: Re: [boost] NuDB: A fast key/value insert-only database for SSD drives in C++11
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2017-03-29 16:03:31

On 29/03/2017 16:06, Peter Dimov via Boost wrote:
>> Consistency refers to the referential integrity of the database.
> ...
>> So if transaction A updates references which are used by transaction
>> B, and if after power loss transaction A was damaged and transaction B
>> was not, if you are Consistent then you also need to throw away
>> transaction B during recovery. But if transaction B did not use any
>> inputs from modifications by transaction A, then if you are Durable
>> you MUST recover transaction B even though it occurred after the
>> damaged transaction A which is thrown away.
> 
> One definition of the letter C in ACID is that if you have issued
> transaction B after transaction A, the database is in one of three
> possible states: (none of A and B present), (A present), (both A and B
> present). It doesn't matter what references A or B update. You're
> absolutely not allowed to apply B without A on recovery.

Consistency is time invariant. If two transactions each affect totally
unrelated data, if B was committed after A in time originally then
during recovery if B was recoverable but A was not, B is indeed
recovered without A.

You'll find this effect breaking code sometimes because B will appear
without the A because the database deduced they were not related. The
cause is that you are supposed to create a reference in the database
between pieces of data which are related in the program code, that way B
always happens after A and B never appears without A. But sometimes
programmers forget, or somebody doing optimisation removes the reference
to increase speed, or sometimes the relationship is buried deep inside
libraries and is non-obvious.

> It's true that this meaning of consistency is not universally accepted.
> If you interpret C loosely, you have no letter with which to describe
> the above, which would now fall somewhere between C and D. So it
> probably makes sense to call it "delayed durability" if you consider
> "consistency" not that.
> 
> As far as I can see, NuDB claims this exact property, subject to
> 
> "NuDB's database integrity guarantees are only valid if the
> implementation of sync assures that all data is fully written to the
> underlying file before the call returns."
> 
> "NuDB's database integrity guarantees are only valid if the
> implementation of trunc assures that subsequent calls to size will
> return o, even if the program is terminated or the device is taken
> offline before calling size."
> 
> so I'm not quite sure what your objections are. You insist on him not
> calling NuDB durable, which he never did.

If he rewrote those two paragraphs above with the below I would have no
objection apart from that his design is flawed and should not enter
Boost without being substantially redesigned.

"NuDB attempts to recover a usable database, but provides no guarantees
of database integrity after unexpected power loss. Please do not use
NuDB in situations where data stored is important."

His earlier paragraphs suggest that if fsync and trunc behave according
to the semantics he gives, you are guaranteed database integrity. As Lee
showed earlier, that is a highly questionable claim in itself, and could
cause users to lose data when they did not expect to. Most developers
think fsync cannot be a noop, and don't know how delayed allocation and
trunc works. They will get this wrong, and lose data. And that is
misleading at best.

So all I am asking for is to remove the misleading claim. Don't
*guarantee* database integrity after power loss, say you try your best
to recover the database, and warn don't use NuDB for important data.
Then I'm happy.

Niall

-- 
ned Productions Limited Consulting
http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/