Re: AVL Databases and Boost Serialization in C++

From: Niall Douglas (s_sourceforge_at_[hidden])
Date: 2021-03-02 10:30:14

On 02/03/2021 09:30, John Maddock via Boost wrote:
> On 01/03/2021 20:25, Edward Diener via Boost wrote:
>> On 3/1/2021 11:03 AM, Benedict NNcNannara via Boost wrote:
>>> AVL Databases were invented in C# and Java in 2016. More recently (2022)
>>> AVL Databases were invented in C++. They were a port of the C# code 
>>> back to
>>> C++. In fact, the port was of a larger system - The Generic Language. 
>>> Boost
>>> Serialization Library was used to serialize a general datatype t in 
>>> set<t>.
>>> This applies to the C++ Variant of AVL Databases, The Generic Language
>>> Compiler uses AVL Databases to store its code. Execution occurs directly
>>> out of the code database, thus no loader is required. In fact the
>>> multithreaded, monolithic runtime is very fast compared to the C# 
>>> version
>>> which is now obsolete.
>>
>> Have you time-traveled from the future to tell us about AVL databases 
>> "invented" in C++ in 2022, and The Generic Language, whatever they are 
>> ? If so maybe you could tell us what the outcome of the covid virus 
>> will be, since you are from the future. I have heard of AVL trees but 
>> AVL databases is a new one on me, but the future evidently promises 
>> many new things. I hope I am writing enough in The Generic Language so 
>> that a person from the future, such as you are, can understand me.
> 
> Google is (un)-enlightening: 
> https://www.quora.com/Why-has-the-world-rejected-AVL-databases

Not wanting to carry on feeding the troll, but there are plenty of 
successful SSD and Optane based databases available, some commercial, 
some open source. So I don't buy the claims that nobody was interested, 
because for some years they've been a hot item for those niches which 
need them.

Speaking personally, I was hired by my current employer to create a 
custom DB to handle a few Petabytes of storage with a few hundred Tb 
ingressed per day. It was designed around eight NVMe SSDs running in 
RAID0, so that's about 20Gb/sec i/o bandwidth.

Due to unexpected US Federal government requirements, we had to drop 
down to a m5d AWS instance which is a mere 4x NVMe SSDs and only 384Gb 
of RAM per node. What was the impact on performance? Turns out, none 
whatsoever. In fact, we could probably drop to c6gd AWS instance with 
half the resources again and it would be barely measurable.

The reason why is that the disc i/o is not a bottleneck in most real 
world use cases. Yes it makes a huge difference to synthetic benchmarks. 
Yes it provides plenty of headroom for spikes in load. But for what our 
paying clients are currently doing, other bottlenecks such as the REST 
API overhead, and indeed the network connection between AWS and the 
client, are vastly more relevant to performance.

You'll find the same applies to most SSD-native and Optane-native 
databases like ours. Storage bottlenecks disappear, but other 
bottlenecks replace them and the overall performance barely shifts by a 
few percent at best, for most use cases.

Niall