Re: [boost] [btree] Status report

Subject: Re: [boost] [btree] Status report
From: Cory Nelson (phrosty_at_[hidden])
Date: 2011-05-21 07:17:07

On Thu, May 19, 2011 at 11:07 AM, Beman Dawes <bdawes_at_[hidden]> wrote:
> On Thu, May 19, 2011 at 3:24 AM, Rutger ter Borg <rutger_at_[hidden]> wrote:
>> 3) It would be nice if it would work with an asynchronous file-like
>> services, like, e.g., Asio's RandomAccessHandleService, see
>> http://www.boost.org/doc/libs/1_46_0/doc/html/boost_asio/reference/RandomAccessHandleService.html
>
> I'd need stronger motivation that "it would be nice"! What would the benefit be?
>

Real-world situation:

I've got a fuzzy full-text search running on top of my own async
b-tree code right now.

Searching needs to hit the b-tree several times depending on the
length of the query.  Being able to launch all these b-tree queries in
parallel is very important for performance, but launching one thread
for each query is tremendously sub-optimal.  In a server environment
that processes hundreds of these full-text searches at once, it might
mean asking for thousands of threads!  With async I can do all of this
very efficiently using only a single thread (or any number of threads
of my choosing).

This said, async will significantly increase the amount of code needed
to implement this library.  I wouldn't want it to be delayed for it,
especially when most people won't need async.

If it could be designed in a way that decouples the core bits from I/O
as much as possible so that it would be easy to move over later, that
would be cool.  ie. implement operations with simple I/O-agnostic
state machines -- instead of doing any potentially blocking operation
like filling buffers directly, it requests the user of the state
machine to fill the buffer for it and resume processing afterward.  A
blocking lower_bound() might be implemented something like:

lower_bound_op sm(key)

result r;

while((r = sm.run()) != done)
{
if(r == ...) ...
if(r == need_block) fill_buffer(sm.buf, sm.buf_length);
}

--
Cory Nelson
http://int64.org