- Boost - lists.boost.org

Re: [boost] Boost library submission (stldb - poll for interest)
by Bob Walters 08 Jan '10

08 Jan '10

On Wed, 6 Jan 2010 09:10:57 +0100, Stefan Strasser <strasser(a)uni-bremen.de> wrote: >> .... A map implementation >> which internally segmented itself into sections under independent mutexes >> in order to permit highly concurrent modifications of any kind would be a >> big help. Anyone know of any such implementations? > > any implementation based on shadowpaging, e.g. this open source one: > http://1978th.net/tokyocabinet/ > > I don't know any internals of Berkeley DB, but it also supports concurrent > modifications. > > the containers I describe here: > https://svn.boost.org/svn/boost/sandbox/persistent/libs/persistent/doc/html… > > also support concurrent modifications, since their container nodes are > implemented as MVCControlled objects. > but they are much less efficient in absolute terms than a low-level container > implementation. There's a problem with this approach to concurrency. I don't like the unavoidable chance of exceptions forcing retry. The company I work for had a problem like this with BerkeleyDB. We had a map which was typically very small, but subject to a constant load of insert/update/delete activity. Under such a load, the prospect of page-level contention increases, and in BerkeleyDB's case we got a lot of 'deadlock' exceptions, which could recurr 5+ times despite retries. The rety-based approach can be a death spiral, in that once contention occurs, the fact that threads are then having to try 2+ times to complete their transactions only increases the load against the containers, thereby also increasing the probability of contention. I think I can digress to the arguments which led to RDBMS'es doing blocking. So I probably have a bias here, but I don't like the notion of the container failing thread A simply because thread B inserted a record which causes some rebalancing which touched the node pointers on the node that A wants to modify. The container should be predictable so that end users can optimize according to its behavior. The STLdb map does not have this problem. Inserts with different key values will never cause a contention exception. Even when two processes do modify the same entry, blocking is possible, as an alternative to retry logic, because that can be more efficient overall. All of this is reinforcing my hope for trans_map<key,locator> - you address the chief persistence related problems I would have if I just put 10 gigs of values into the STLdb database, and I think I can keep these container implementations fast and predictable, and meet your desire for low-level containers.

1 0

[rdb] 0.2.04 - date, time and datetime
by Jean-Louis Leroy 08 Jan '10

08 Jan '10

I have uploaded a new version to the vault. It introduces support for three new sql types : datetime, date and time. I have been busy with other things recently, sorry for not taking part to the discussions about Persistent and other libs. Jean-Louis

1 0

How to create a shallow copy without calling a constructor?
by vicente.botet 08 Jan '10

08 Jan '10

Hello, While developing Boost.STM (Software Transactional Memory), I need to create a cache of a transactional object without using the new operator nor the copy constructor of the class. This cache needs only to copy the raw memory of the copied instance, is for that I have called it shallow_clone The following will create a deep copy and don't respect my requirements class C { public: C* shallow_clone() { return new C(*this); } }; I have looked at uninitialized_copy but if I have understood it correctly, it calls the copy constructor. I have tried with class C { public: C* shallow_clone() { C* p = reinterpret_cast<C>(malloc(sizeof(C)); if (p==0) { throw std::bad_alloc(); } memcpy(p, this, sizeof(C)); return p; } }; But I suspect that this is not correct in general. Is this correct on some particular cases? if yes on witch ones? Is there a way to create such a cache instance without calling to the constructor in a portable way using some low level C++ interface? Thanks, Vicente _____________________ Vicente Juan Botet Escribá

6 12

[Boost.Build] Unexpected MSVC linker error
by Sergey Zubkovsky 08 Jan '10

08 Jan '10

Hi, I'm using bjam and VC++2008 Express Edition to build date_time library, for example. When I try to link it with my project I have the error: LINK : fatal error LNK1104: cannot open file 'boost_date_time-vc90-mt-gd-1_40.lib' The bjam's command line options are : --layout=tagged toolset=msvc The name of the genereated lib file is "boost_date_time-mt-gd.lib". The error doesn't occur if the "--layout" option is "versioned" or "system". It seems to me that the cause of this behavior is the following text in the file "boost\config\auto_link.hpp": #ifndef BOOST_AUTO_LINK_NOMANGLE # pragma comment(lib, BOOST_LIB_PREFIX BOOST_STRINGIZE(BOOST_LIB_NAME) "-" BOOST_LIB_TOOLSET BOOST_LIB_THREAD_OPT BOOST_LIB_RT_OPT "-" BOOST_LIB_VERSION ".lib") # ifdef BOOST_LIB_DIAGNOSTIC # pragma message ("Linking to lib file: " BOOST_LIB_PREFIX BOOST_STRINGIZE(BOOST_LIB_NAME) "-" BOOST_LIB_TOOLSET BOOST_LIB_THREAD_OPT BOOST_LIB_RT_OPT "-" BOOST_LIB_VERSION ".lib") # endif #else # pragma comment(lib, BOOST_STRINGIZE(BOOST_LIB_NAME) ".lib") # ifdef BOOST_LIB_DIAGNOSTIC # pragma message ("Linking to lib file: " BOOST_STRINGIZE(BOOST_LIB_NAME) ".lib") # endif #endif "-" BOOST_LIB_VERSION ".lib" is always added in the "pragma comment" directive, if BOOST_AUTO_LINK_NOMANGLE is undefined, of cause. Is this a bug? How can I workaround it if this is not a bug? Thank you.

4 6

[test] trunk breakage
by Vladimir Prus 08 Jan '10

08 Jan '10

I went to check the current state of Boost.UUID test results on trunk, and found many tests failing. Here's a typical issue: http://tinyurl.com/y8c9w39 This looks like Boost.Test problem. The testrun is for revision 58533, while last change in boost/test is: r58526 | rogeeff | 2009-12-25 12:20:28 +0300 (Fri, 25 Dec 2009) | 1 line in preparation of production use of test tools So, it seems the test run corresponds to the current state of Boost.Test, and that state is a bit broken. In fact, it looks like a *lot* of tests are failing on trunk, and every test that uses Boost.Test is affected. Boost.Test itself appears to fail to build in may configurations: http://tinyurl.com/ybfkeb2 Gennadiy, I realize this might be a Christmas/New Year break, but at the same time we have a "major code changes" deadline on Jan 4th, and such interruption of test reporting make things hard. Is there any chance you could investigate? If this issue cannot be sorted out quickly, we might have to revert the commit for now. Thanks, Volodya

9 28

Re: [boost] [stldb] design details (was Boost library submission (poll for interest))
by Bob Walters 08 Jan '10

08 Jan '10

On Tue, 5 Jan 2010 16:54:52 +0100 Stefan Strasser <strasser(a)uni-bremen.de> wrote: > I thought the reason that requires you to load the entire dataset on startup > and save it entirely at checkpoints and recoveries is that the user needs to > access the mapped region directly through pointers, since the user types > exist in an unserialized form there. is that right? IIUC by load you mean reconstruct the memory mapped file from the checkpoints + logs? The only time that I have to reconstruct the memory mapped file or shared memory region using the checkpoints and logs is when recovery is done, NOT during routine/controlled startup and shutdown. See the architecture page: http://stldb.sourceforge.net/stldb/arch.html. I specifically wrote this page after our prior discussions. If processes disconnect normally, and recovery is not needed then I reuse the region. When recovery is required, I do have the problem of having to reconstruct the memory mapped file using the whole of the last checkpoint. Thus I agree your shadow paging of the container in the memory map would reduce recovery time. Note that I am changing my checkpoint process so that each subsequent checkpoint will only write out changes since the last checkpoint, and do so by reusing free portions of the checkpoint file. Consequently the I/O directed to both the log and checkpoints is in proportion to the rate of change, and not to the size of the database. STLdb was originally designed as a memory resident database. It should do that well since it provides direct access to containers in a shared region. My thinking was that if the user elects to use a memory mapped file for the region, they then have a paged database. The I/O directed at checkpoints and logs is based entirely on rate of change. However, that paged database then introduces redundant disk I/O since it will page out per LRU while I also occasionally write changed entries out to the checkpoints as checkpoints are done. IIUC you have a similar problem but it involves redundancy of memory. If you work with a large intrusive container, then you have the serialized storage memory mapped in, but must also cache the deserialized nodes of the container in memory to ensure fast locator traversal, and allow it to work like an index. What I am currently very intriguied by is your idea of using STLdb as indexes for a large database. i.e. The containers are map<key,locator>, and refer to a separate Boost.Persistence stored objects. Ideally, the containers will be small and tend to be held in memory (due to frequent use) so the double-write problem goes away since the OS won't be paging the memory map. A user could even mandate that the indexes stay memory resident by using a shared memory region type, rather than a memory map. There's another benefit to this - it would allow heterogenous containers, to the theme of map<key,base*> because of the locator's use of boost serialization. And also because the real value-type is not in shared memory, they can once again have virtual methods, etc. > what confuses me is the "Updating Entries" section and the fact that you do > track changes made by the user, e.g. via trans_map::update. > so if the library is aware of every change to the mapped region, what > stops you from employing a shadow paging technique and only writing to the > mapped region on transaction commit, when the modified pages are logged? > that would make your library MUCH more widely usable. Right now, the interaction between the containers and the rest of the transactional infrastructure is based off an approach inspired by the stasis library (http://tardis.cs.berkeley.edu/stasis/developers/html/main.html) There is a concept of a transactionally aware object. As modifiers on transactionally aware objects are called, those objects post 'TransactionalOperations' onto the transaction passed to them. These ops represent the means by which the transaction can ultimately commit or rollback that operation when the time comes. Containers like trans_map are transactionally aware and define their own atomic operations (i.e. in the case of trans_map, insert(), lock(), update(), erase(), clear() and swap().) When these operations are logged, it is the logical operation (insert<5,"foobar">) which is logged, not the changes done to memory, so that during recovery it can be replayed. I am not logging memory changes. A big contrast between this STLdb library and Boost.Persistence is that in your case, IIUC you are tracking the use and modification of objects but the objects themselves are unaware. One downside of this is that the objects can't collaborate to support piecemeal updates of small portion of the objects - it's always the full object which is re-serialized. Because I was starting with the notion of persisting containers, I sought to devise a model for a transaction-aware object which supported updates against portion of the object. Thus I came up with the idea of the transactional objects supporting one or more transactional operations. The tracking of changes is based on the use of these transactional operations. The only reason I went with methods like lock() and update() is that while I could have done this implicitly via non-const access to iterators (for update anyway), I thought that implicit approach would be an easy source of bugs. i.e. My 1M finds() become 1M find() and updates() because I forgot to make the getName() method on my object const. Are you requiring const correctness when accessing locators? I really like your interface in Boost.Persistent, but are you currently assuming an update when a locator is accessed non-const, or is it more clever than that, i.e. making a transaction-local copy during non-const access, but later checking for differences between the transaction-local copy and original before reserializing and storing the object? > because on the other hand, dereferencing trans_map::iterator returns a > non-const reference to value_type, indicating the opposite. Bug. I originally had the iterator always returning a const reference, to be consistent with the demand that users use update(). However, had some kind of difficulty, and never circled back to it (and can't recall why currently). Will be addressed in the final submission. Also, keep in mind that the code I have currently has not been submitted to boost. Its ugly. The interfaces need work, but what currently exists does run. I am not proud of some of what's in there currently. > what kind of tree does trans_map use internally? Red/Black per boost::interprocess::map, which is used (intact) internally for the low-level map implementation. I'm wrapping that with code to make it transactionally aware. I didn't want yet another implementation of map. > I can't wrap my head around the locking or MVCC used for isolation. > could you explain this a little more in detail? > does the library automatically > record/lock accesses? is the user supposed to lock manually on every access? The isolation behavior is based on RDBMS conventions for row-level locking. Reads do not lock entries, writes do lock entries. These entry-level locks last until commit/rollback. A row can be locked directly (for pessimistic locking convention) by using the lock() method on the entry. This was needed because I do have blocking capabilities on modifier methods, and thus needed a way to keep one thread's blocking update from overwriting another. See the section in the "quick start" which shows lock() being used. It corresponds to the RDBMS concept of "select for update" and is used for the same reasons. I have been thinking about adding optimistic locking support based on specialization of the map for a value type implementing some optimistically lockable interface. Doing so would then allow you to avoid any call to lock(), but present the chance of failure during update(). This is all per RDBMS row-update conventions, and ultimately, I planned to support both just as many RDBMS support both, giving the user the choice. Incidentally, one problem I have with Boost.Persistence is that you always handle contention by throwing. IIUC contention can occur when two threads insert records with different keys into a container simply because they would both add their node under the same parent, and despite their nodes having different keys. So in other words, there's nothing apps can do to prevent the possibility of contention errors. This can backfire badly if the probability of contention is high because of app design. For example, someone writes a transaction which does several things, one of which is to push_back something to a queue. The transactions which did the push_back would need to be completely synchronized, IIUC. Support for blocking was one of the RDBSM worlds answer to this kind of problem, and the reason I included support for it in the trans_map container design. Thus I'm drawing the conclusion that implementing containers base on the locator concept's handling of concurrency might not always be the right idea. btw... I'm still not settled on the interface for lock() / update() and have wondered if I should somehow put that onto the iterator. E.g. lock() would be a method of the iterator, and update() could be a method on the iterator which returns a non-const L-value reference that can then be updated directly. Suggestions welcome... > how are possible read-accesses over an entire range stored, e.g. by > using map::equal_range()? If you hold a shared lock on the map while accessing it via a generic algorithm you are guaranteed to see the data reflecting a consistent set of transactions. In between locks, the result of new commits can appear, leading to the various phantom scenarios defined in the ANSI isolation levels. Holding a shared lock on the map prevents other transactions from committing changes to it while the lock is in effect. You aren't meant to hold locks on the containers for a very long time if you want good concurrency, but it can be done.

3 2

Re: [boost] boost::mpl::for_each_impl() still not inlined
by Peter Foelsche 07 Jan '10

07 Jan '10

"Peter Foelsche" <peter_foelsche(a)agilent.com> wrote in message news:... > > "Peter Foelsche" <peter_foelsche(a)agilent.com> wrote in message > news:hi5bfc$gap$1@ger.gmane.org... > > > below are my patches for boost_1_41_0 -- these are dealing with inlining > and > > boost::lambda::if_then_else_return > > not being able to deal with different return types in the true vs. else > branch. > sorry -- the patch was inversed. Attached the correct one.

1 0

boost::mpl::for_each_impl() still not inlined
by Peter Foelsche 07 Jan '10

07 Jan '10

I think I published the matching patch already a couple of times. Could somebody change this please?

1 1

Is floating_point_comparison.hpp general purpose tool?
by Mateusz Loskot 07 Jan '10

07 Jan '10

Hi, I'm wondering why the floating-point algorithms [1] implementation in floating_point_comparison.hpp lives in Boost UTF. Shouldn't it go to location that would suggest it's not only usable for unit tests but it is a general purpose tool. [1] http://www.boost.org/doc/libs/1_41_0/libs/test/doc/html/utf/testing-tools/f… Best regards, -- Mateusz Loskot, http://mateusz.loskot.net Charter Member of OSGeo, http://osgeo.org

4 10

generate tuples by mpl -- should I use fusion?
by Peter Foelsche 07 Jan '10

07 Jan '10

I need to be able to convert a mpl sequence into something like a tuple: typedef boost::mpl::vector<double> CMplSequence; typedef CCreateVectorFromMpl<CMplSequence>::type CVector; I need to have runtime access with constant index: CVector s; get<0>(s) = 1.0; std::cout << get<0>(s) << "\n"; I need to be able to retrieve the type at compile time -- preferably as a mpl sequence. I already wrote something like this using std::pair as the tuple but I'm running into problems storing reference types on Windows. Peter

2 1