Re: [boost] NuDB: A fast key/value insert-only database for SSD drives in C++11

Subject: Re: [boost] NuDB: A fast key/value insert-only database for SSD drives in C++11
From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2017-03-27 10:57:47

>> Your hash function (which runs per thread) only needs to be as fast as
>> your storage device is at a queue depth of 1. So, taking a top of the
> 
> Why? Running that function means the core can't do other work, so I'd
> think a faster function is still beneficial.

The point I was making was not that crypto hashing isn't expensive and
doesn't consume most of a CPU when paired with a top end SSD.

Rather I was pointing out that current CPUs are plenty more than fast
enough to crypto hash for current SSDs such that maximum performance is
maintained. Obviously, if you paired a 1Ghz Atom CPU with a PCIe SSD,
crypto hashing would have a severe impact on performance, and achieving
durability there likely not worth the cost.

Copy on write filing systems typically do the hashing for you, that's
why on those filing systems you don't need to. Similarly, mounting ext4
with "journal=data" effectively applies hashing to written data so again
you don't have to.

The point being made here is that you need to use a high quality secure
hash when writing data to storage. That way after power loss you can
determine if the database store is truly in a consistent and valid state
rather than being tricked into thinking it is, and then unwittingly
destroying more user data.

>> range NVM SSD, the Samsung 960 Pro, it can write at QD1 about 50k IOPS.
>> That's around 200Mb/sec/thread. Blake2b runs at 1Gb/sec, so it should
> 
> Is that Mbit/s or MByte/s? I think MByte.

200 Megabytes per second per thread. As most i/o loads are QD1, we care
most about that performance during filing system algorithm design. The
headline figures you get for SSDs are usually at QD32, it's very
deceptive as most filing system algorithms do not parallelise well.

This is why the new XPoint SSDs from Intel are so exciting. Their QD1
performance is double anything from before. Very exciting!

Niall

-- 
ned Productions Limited Consulting
http://www.nedproductions.biz/ http://ie.linkedin.com/in/nialldouglas/