Re: [boost] NuDB: A fast key/value insert-only database for SSD drives in C++11

Subject: Re: [boost] NuDB: A fast key/value insert-only database for SSD drives in C++11
From: Vinnie Falco (vinnie.falco_at_[hidden])
Date: 2017-03-26 14:17:31

On Sun, Mar 26, 2017 at 7:11 AM, Niall Douglas via Boost
<boost_at_[hidden]> wrote:
> If you want durability across multiple OSs and filing systems, you need
> to assume that fsync's are reordered with respect to one another.

Does SQLite assume this?

I make no claims that NuDB works on all filing systems. It will
definitely not do well on network shares - it wasn't designed for
that. Nor will it perform well on spinning disks. It is specifically
designed for directly connected SSD drives, on mainstream operating
systems. As I said in the original post, its use-cases are niche
(hence, why I am probing for the level of interest).

> NuDB needs to remove all claims regarding durability of any kind.
> In fact, you might as well remove the fsync code path entirely

No, I don't think that I'll be doing that. If you feel that this is
important, you have my blessing to clone the repository and continue
with your own fork.

> fsync() performs pathologically awful on copy-on-write filing systems

The library is not designed for exotic file systems like the one you
describe. Its meant for simple commodity hardware and operating
systems such as what you might find on a bare metal amazon web
instance. There is no need for a copy on write file system, as long as
the invariants are met (that fsyncs aren't reordered).

> In which case you did not make a great choice.
>
> Much, much better would be Blake2b. 2 cycles/byte, cryptographically
> secure, collision probability exceeds life of the universe.

Hmm, no, it seems that you are the one who "did not make a great
choice." The requirements of Hasher do not include cryptographic
security. Blake2b is a cryptographically secure hash function which
computes digests up to 32 bytes, while xxhasher is a non
cryptographically secure hash function which computes a 64-bit digest.
NuDB requires a Hasher more like std::hash and less like SHA-1.

Blake2b can achieve almost 1Gb/s while xxhash can achieve 110Gb/s. I
think I'll be sticking with xxhash but again you have my full support
if you want to instantiate nudb::store objects with your own user
defined Hasher that uses Blake2b - here's a link to the type
requirements so you can get started with it:
http://vinniefalco.github.io/nudb/nudb/types/Hasher.html