Re: [boost] Boost library submission (poll for interest)

Subject: Re: [boost] Boost library submission (poll for interest)
From: Bob Walters (bob.s.walters_at_[hidden])
Date: 2010-01-08 11:14:41

On Fri, Jan 8, 2010 at 10:39 AM, Rutger ter Borg <rutger_at_[hidden]> wrote:
> Very interesting. I've written a std::map interface to Berkeley DB, which
> gives quite some of functionality. Taken from that, I have a couple of
> questions,
>
> 1) Transactional semantics: wouldn't it be easier to steal semantics from
> locks in threads? E.g., for the synchronous interface case, wouldn't
>
> map_type m_map;
> try {
>  scoped_transaction trans( m_map );
>  ..
>  .. do stuff with the map
>  ..
>  trans.commit();
> }
> catch( transaction_error ) {
> }
>
> be a easier than passing the transaction everywhere?

Yes.  I plan to add support for this, but I also think I'm going to
keep the explicit model for two reasons: 1) the odd chance that a
particular thread wants to work with multiple outstanding
transactions, and 2) makes it easy for me to write test cases for the
many integrity-related tests that I need.

i.e. I can write a test which:
transaction txn1( db );
transaction txn2( db );
map->insert( entry, txn1 );
assert( map->find( entry, txn1 ) != map->end() );
assert( map->find( entry, txn2 ) == map->end() );

>
> 2) what serialization models are you considering? I.e., for a map of int to
> doubles, serialization would be overkill, wouldn't it?

Yes.  I've thought about having specializations to support different
serialization models.  So that a complex type might use
Boost.Serialization, but concrete types can use direct byte copies,
etc.
This isn't done yet.  Currenty, I'm using map<string,string> in actual
applications, so haven't put
any work into alternatives.

> 3) have you considered things like key prefix-compression and storing keys
> and values in different files?

I have considered memory localization controls to cluster map nodes
and keys into certain
segments of a region, and leave values to use all other memory in the
region.  The point of this
being for memory maps that can't be held memory resident, to establish
hot spots which can stay
paged in thus assuring only one page in when doing find() and value
access.  But nothing more.

>
> 4) how did you solve "map[ key ] = value" vs something = map[ key ]? Here, I
> resorted to a reference object that would do the .put() in case of
> assignment, a .get() in case of an implicit conversion.

Actually, I haven't implemented that method yet.  Need an implicit
transaction passing
technique before it can be implemented.  Approach will probably be to
require an scoped lock
on the map for the duration of that call.

> 5) do you reach 200,000 transactions per second per thread? :-)

I'm assuming that you realize that the answer to this would depend on
the transaction composition, the speed of the machine, the number of
values in the map during the test. ;)  What I can say is that I will
be running a comparative test between the maps in this database and an
equivalent, multi-threaded, heap-based use of a std::map in the same
way.  All disk I/O in STLdb can be suppressed for the purpose of such
testing, allowing an apples-to-apples comparison that should show how
much overhead I am adding with the transactional infrastructure, and
how I may be negatively affecting concurrency.
The apps I use to do this will be checked in with the project, to
support repeatability.