Re: [boost] NuDB: A fast key/value insert-only database for SSD drives in C++11

Subject: Re: [boost] NuDB: A fast key/value insert-only database for SSD drives in C++11
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2017-03-21 23:36:06

On 21/03/2017 20:36, Vinnie Falco via Boost wrote:
> I just released version 1.0.0 of my header-only C++11 library NuDB.
> Its quite a mature library, haven't had to make any changes in a
> while. Its been running on production servers for 2 years or so with
> no problems, managing ever-growing databases of 2TB.
> 
> I'm wondering what the level of interest, if any, there would be for
> making this a part of Boost. You can check it out here:
> 
> https://github.com/vinniefalco/NuDB

As I said to you at CppCon, I'm very keen on the concept. It's why I
started AFIO back in 2012, is to one day deliver a low level key-value
store suitable for C++ standardisation. And I'm still at that five years
later, with Outcome being my next step to getting that achieved as the
key-value store's API uses a ton of outcome::result<T>.

But as I also said to you at CppCon, I'm not keen on your NuDB. I have
problems with its API, with its flexibility and scalability, and lots
more problems with its implementation. I am unsure what benefit it
confers over the much safer choices of SQLite or UnQLite both of which
are battle tested and written by database experts to ensure that other
people's data is never, ever lost.

I felt when I reviewed NuDB's implementation some time ago that your
implementation would be highly susceptible to data loss in a wide set of
circumstances. I also felt it wouldn't scale well to the > 1M IOPS
storage just around the corner. I appreciate that as you say, it's been
running in production for two years, but I would suspect it is being
used in a highly restricted use case where corner cases don't present.
As a library in Boost, you would find people doing all sorts of daft
things with it, and it would therefore need to be absolutely rock solid
reliable on a wide variety of operating systems, filing systems and use
cases e.g. over a Samba share. This is why it's taking me so long,
writing the testing for this stuff takes a very, very long time.

All that said, I haven't delivered a key-value store after five years of
outside-of-work effort. You have. I would therefore endorse you bringing
this to review, but be aware that in a review I would be fairly certain
it'll be rejected. That said, the feedback you'd get would be very
useful, not just to you but also to me for my planned key-value store,
so if you are up for getting this to a state ready to submit to Boost,
I'd welcome it.

Niall

-- 
ned Productions Limited Consulting
http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/