Re: [boost] Boost B-Tree Library Status

Subject: Re: [boost] Boost B-Tree Library Status
From: Joel Young (jdy_at_[hidden])
Date: 2012-03-12 19:20:28

Beman Dawes <bdawes <at> acm.org> writes:
> 
> I'm totally distracted by other priorities.

No problem, 

I've been using the btree library heavily for the past week or so with boost
1.47 on a linux machine with 24G RAM.  I'm using 128bit keys and 196bit payload.
 I've successfully built files of 100,000,000 records.  Past that point it slows
down dramatically because RAM is exhausted and the system spends all its time
writing to disk.  I don't view this as a problem with the library.

What I am looking at doing is building the database in chunks on multiple
machines, and then merging the chunks (and then perhaps sharding the results
into smaller pieces again).

Do you have any suggestions on the most efficient way of merging two btree
files?  In your expectation, is it better to merge big into small, small into
big, or both into new?  Or should I write a low level merge that operates at the
node level?  The goal is to be able to support 1-10 Gig of records.

Also, is their a way to destroy the b-tree WITHOUT flushing the buffers to disk?
 There are times when I've built a temporary b-tree finished exploiting it and
don't need to get it written out.

I am enjoying the library.

Thanks!

Joel