May 2006 - Boost-users - lists.boost.org

Re: [Boost-users] Thread-safe smart pointers, hashtables without locking
by Ray Whitmer 05 May '06

05 May '06

> Personally, I read this and said, "eeek!" (Well, not literally. My > office mates would wonder what was up.) I think any design like > this is going to be (at best) very non-portable, and (more likely) > subject to breaking at the worst possible moment (i.e. when > demonstrating to a customer). Is "relatively" thread-safe really > good enough here? Let me make it clear. I do not want relatively thread-safe. I want completely thread safe. It seems to me that boost already relies on an atomic counter for shared_ptr. I do not think the solution would be less portable than this. I certainly want it to run on Windows and Mac, 32 and 64 bit as well as major Unix and Linux including certain 32-bit-or-more embedded situations. There may be hopefully-rare C++ threading situations that may be difficult to recover from, and I will simply have to gather more information on these and see if they can be dealt with. > As a constructive suggestion, if you want to reduce lock conflicts > in hash tables, have you considered some form of finer-grained > locking? You could, for example, have one lock per "slot" in your > hash table. Computing the hash and finding the slot is thread-safe > (since it doesn't depend on the state of the hash table), so you > could then lock only the slot itself while performing list > maintenance or whatever you need to do to read/write the data. For the more-common reader of the hash table, it is not my intent to reduce lock conflicts, but, rather, to generally eliminate them, and replace them with something lighter-weight, such as the atomic counter used by shared_ptr, that in best cases may only take a few instructions to execute. I may have thousands of threads reading info from the hashtable at once and there may never be a natural time when there are no readers so I can update the table. The purpose of the threads is many operations that rely heavily on the hashtables -- every other instruction they are looking up something. For what it is worth, finding the slot is not thread-safe, because the table size and pointer can change. List maintenance involves multiple slots and I do not think it is worthwhile doing finer- grained locks because of the complexity, deadlock potential, and likely worse performance it would introduce. > This would be truly thread-safe but might perform better if you > have lots of simultaneous readers/writers. I haven't tested this, > but if you do, I'd be interested to hear the results. I don't think it even begins to be thread-safe, because it doesn't begin to deal with reallocation of the array itself, and if it did, your fine-grained locks appear to become worthless because you need a high-level lock protecting that allocation assuming you are going to use locks for readers, which seems like a bad idea from the start for me. There is a reason shared_ptr does not use locks (at least on the best-supported platforms), but uses an atomic count. > PS - I'd be even more worried in Java, btw, since Java opcodes > should map to CPU operations less well than a compiled binary. Not > to mention that in Java, the portability assumption becomes really > critical. Java classes are *supposed* to be portable. I do not want to get into Java wars here, but Java has to solve the deallocation problem in a thread-safe manner. When the garbage collector actually reclaims an allocation, the possibility cannot exist that someone else still happens to be using it or the language is badly broken. This is the hard thing to be solved in C++ to make a thread-safe hashtable that you do not have to deal with in Java. The deallocation problem is, from everything I have analyzed, the big problem in C++. When you attempt to deallocate something, for all you know there could be another thread that just barely loaded its pointer into shared memory and as soon as it unstalls will want to increment its reference count, but it has not yet been incremented, so even a smart pointer using an atomic reference count fails because the object is sharable in a multithreaded environment, but not the reference itself. Here is my first stab at a solution: Extra data (for each Hashtable): 1. Array of two atomic counts, one active and one inactive. 2. Index that flops between 0 and 1, so there is an active and inactive count. 3. A mutex that can be used for exclusive writers Each reader behaves as follows: 1. Fetch index. 2. Increment corresponding atomic count. 3. If fetched index is not current index, decrement atomic count and loop to step 1. This is rare and safely eliminates rare bad cases 4. Now the reader has free access to the entire hashtable structure, and is guaranteed by cooperation of the writer that the pointers it sees are not stale. It must only guarantee that any pointer referenced in step 4 is either destroyed or it's ref count is incremented before step 5 -- this includes the pointer to the allocated resizable array itself. 5. Decrement the same atomic count (of fetched index) incremented in step 3. Note that in Java, none of this is necessary for readers because the language safety assumes any reference is not deallocated. Each writer behaves as follows: 1. Obtain an exclusive lock on the hashtable. This lock only excludes other writers, not other readers. 2. Perform changes to the hashtable in a manner that, given atomic loads or stores of aligned pointers, never leaving the table in a state that simultaneous readers cannot use it. This is very easy compared to the allocation issues, and I have been doing this in Java for years. I would be happy to expand on this. This write operation may include releasing objects with ref-count 1, if this is supposed to be a weak hashtable that does not keep unused objects in the index. 3. This is really part of 2: any step in (2) which destroys a reference a reference keeps a private local copy to make sure it is not deallocated before all reading threads are finished with it. If the references are ref-counted, a ref-counted private reference is kept. 4. If there are no private local pointers/references being kept, go to step 9 5. Wait for inactive (1-index) atomic count to go to 0. This is rare and safely eliminates rare bad cases. 6. Change index (index = 1 - index). 7. Wait for inactive (1-index) atomic count to go to 0. This means that all readers who might have a copy of a stale pointer are finished. 8. Release all private local references with a reference count of 1. Non-ref-counted references always get released, due to single owner. If this is a weak hashtable, reinsert any references that have a reference count greater than one since these were found and incremented while we were waiting for readers to exit. 9. Release lock. Done. More notes: Although we don't want to go out of our way to make the writer inefficient, there is a single writer at any one time with an exclusive lock. But we do not have to wait for all readers to exit to perform the update, we only need to wait for current readers to exit, during which time new readers may enter, which we do not care about because we have already removed the candidates from public places in the list. This includes the pointer to the array itself, which is generally treated like any other allocated pointer, but does not require a reference count. There are many details I left out that did not directly pertain to the allocation problem, but only with atomically updating the rest of the table so that the reader view is always consistent. Ray Whitmer

1 0

pool_allocator<void> not supported?
by Strelnikov Michael 05 May '06

05 May '06

Dear all, It seem std::allocator specialized for void, but pool_allocator don't. Why? Thanks.

1 0

Re: [Boost-users] [serialization] I can't figure out what I'm doing wrong with xml_oarchive - please help
by Manos Tsagarakis 05 May '06

05 May '06

Hello Robert, I'm posting this message again (3rd time actually) as I did not see the previous two ... Thank you for the quick response. I'm glad you liked the title ... A) I did replace all BOOST_CLASS_EXPORT with BOOST_CLASS_EXPORT_GUID, but the problem with xml persists. It is really difficult for me to figure out what is wrong with xml, as both txt and bin versions are ok. Actually what worries me, is the possibility that the problem with xml somehow will show up in other formats too ... B) I don't use the version numbers at the moment but I added it just for future use. I actually have a couple of questions here. 1) BOOST_CLASS_VERSION has to be in the header file? Can I place it in the cpp file? 2) Please correct me If I'm wrong: I don't use BOOST_CLASS_VERSION so that all classes have version number assignment set to 0. I also don't use BOOST_SERIALIZATION_SPLIT_MEMBER for now. Later on, I add some members, so that the class version must change. I simply add BOOST_CLASS_VERSION(..., 1) and split serialize to save/load at that time. Previously saved data files are not affected as long as version info is handled correctly by my code. Is that correct? C) I avoid using templated code in CTCDataBase for as long as the core data that must be serialized evolve, in order to avoid frequently modification of the most included header file TC_database.h. That meens almost recompile all, and that takes time. Thank you for the suggestion any way. I'll keep it in mind. To be honest, I did try to use export feature of templates bug no luck ... D) I re-attach a couple of header files to this post and I kindly ask you to take a look at the implementation and tracking level for template class fem::CPropRepository implementation_level< fem::CPropRepository<T, SZ> > and tracking_level< fem::CPropRepository<T, SZ> >, and one utilization of it at the other header. Should it be so? Do I need it? And also is it possible that using this CPropRepository stuff (especially with MFC CString) breaks the xml code? E) After adding all headers of the serialize library compilation takes a lot more time. Are there any guidelines by you on that? I hope I'm not encroaching your time... Thank you again, Emmanuil Tsagarakis >> -----Original Message----- >> From: boost-users-bounces(a)lists.boost.org >> [mailto:boost-users-bounces@lists.boost.org] On Behalf Of >> Robert Ramey >> Sent: Friday, May 05, 2006 8:36 AM >> To: boost-users(a)lists.boost.org >> Subject: Re: [Boost-users] [boost-users] [serialization] >> I can't figure outwhatI'm doing wrong with xml_oarchive - >> please help >> >> LOL - I liked the title. >> >> I've taken a very quick look at your code and files. >> >> a) I'm a little suspicious of including a ":" character >> as the GUID for a class. I think that might be an issue. >> Try replacing >> >> BOOST_CLASS_EXPORT(fem::CTCSoil); >> >> with >> >> BOOST_CLASS_EXPORT_GUID(fem::CTCSoil, "fem_CTCSoil"); >> >> You'll have to double check the exact syntax. >> >> b) You've set version numbers to 1 but I don't see the >> version numbers used in the deserializations. Version >> number assignment is only necessary if you're going to >> use the version numbers and they are not equal to 0. >> >> c) A minor suggestion - not important here but interesting anyway. >> In fem_boost_serialize.cpp Use templates to shorten the code. >> >> template<class Archive> >> void CTCDataBase::boost_serialize_open(const char* filename) { >> try { >> std::ifstream ifs(filename); >> assert(ifs.good()); >> Archive ia(ifs); >> >> ia >> BOOST_SERIALIZATION_NVP(m_lstSoils); >> ia >> BOOST_SERIALIZATION_NVP(m_lstBrickWalls); >> ia >> BOOST_SERIALIZATION_NVP(CTCFooting::s_bspSoil); >> ia >> BOOST_SERIALIZATION_NVP(CTCFrame::s_bspBrickWall); >> } >> catch (const boost::archive::archive_exception& e) { >> e; >> } >> } >> >> and >> template<class Archive> >> void CTCDataBase::boost_serialize_saveconst char* filename) { >> try { >> std::ofstream ofs(filename); >> assert(ofs.good()); >> Archive oa(ofs); >> >> oa << BOOST_SERIALIZATION_NVP(m_lstSoils); >> oa << BOOST_SERIALIZATION_NVP(m_lstBrickWalls); >> oa << BOOST_SERIALIZATION_NVP(CTCFooting::s_bspSoil); >> oa << BOOST_SERIALIZATION_NVP(CTCFrame::s_bspBrickWall); >> } >> catch (const boost::archive::archive_exception& e) { >> e; >> } >> } >> >> void main() >> { >> fill_with_build_in(); >> boost_serialize_save<boost::archive::text_oarchive>("boost >> _serialize.txt"); >> boost_serialize_save<boost::archive::binary_oarchive>("boo >> st_serialize.bin"); >> boost_serialize_save<boost::archive::xml_oarchive>("boost_ >> serialize.xml"); >> >> // some time later ... >> boost_serialize_open<boost::archive::text_oarchive>("boost >> _serialize.txt"); >> boost_serialize_open<boost::archive::binary_oarchive>("boo >> st_serialize.bin"); >> boost_serialize_open<boost::archive::xml_oarchive>("boost_ >> serialize.xml"); >> } >> >> Robert Ramey >> >> TOL mtsagara wrote: >> >> Hi to all, >> >> >> >> This is my very first post to boost-users! I've been >> using a couple >> >> of boost libraries for a while (shared_ptr & bind >> mostly). Write now >> >> we are in the process of transforming a project based on MFC >> >> collections, (with objects derived from CObject), to >> STL, and doing >> >> some search for a library that will allow object >> persistence the way >> >> MFC does, the obvious solution was boost::serialize. >> As I'm a newbie >> >> in mpl style code the first impression was terrifying! >> But after >> >> reading the documentation and experimenting a lot, I >> did manage to >> >> start using boost::serialize. In fact there is no need >> to be familiar >> >> with mpl. The library is excellent - an exemplary >> work, thank you >> >> Robert Ramey! >> >> >> >> Now my problem is with xml files. I can write a couple >> of classes >> >> both in txt and binary files. But I can't write the >> very same data in >> >> xml format. I believe I followed all guidelines but I >> had no luck ... >> >> :( >> >> >> >> The scenario in pseudo code is this: >> >> >> >> >> >> >> ////////////////////////////////////////////////////////// >> /////////// >> >> / >> >> // file fem.h >> >> #include <boost/archive/...> >> >> #include <boost/serialize/...> >> >> >> >> namespace fem { >> >> >> >> class CTCBase >> >> { >> >> ... >> >> }; >> >> typedef boost::shared_ptr<CTCBase> CBSPBase typedef >> >> std::list<CBSPBase> CBSPBaseList >> >> >> >> >> >> >> >> class CTCSoil >> >> { >> >> ... >> >> }; >> >> typedef boost::shared_ptr<CTCSoil> CBSPSoil typedef >> >> std::list<CBSPSoil> CBSPSoilList >> >> >> >> >> >> >> >> class CTCBrickWall >> >> { >> >> ... >> >> }; >> >> typedef boost::shared_ptr<CTCBrickWall> CBSPBrickWall typedef >> >> std::list<CBSPBrickWall> CBSPBrickWallList >> >> >> >> >> >> >> >> // list of soils >> >> CBSPSoilList s_lstSoils; >> >> // list of Brick Walls >> >> CBSPBrickWallList s_lstBrickWalls; >> >> // initial soil for each new foundation beam (also >> contained in above >> >> list) CBSPSoil s_bspSoil; // initial brick wall on >> each beam (also >> >> contained in above list) CBSPBrickWall s_bspBrick; } >> >> >> >> BOOST_CLASS_VERSION(fem::CTCBase, 1); >> >> BOOST_CLASS_VERSION(fem::CTCSoil, 1); >> >> BOOST_CLASS_VERSION(fem::CTCBrickWall, 1); >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> ////////////////////////////////////////////////////////// >> /////////// >> >> / >> >> // file fem.cpp >> >> >> >> #include "fem.h" >> >> #include <boost/serialization/export.hpp> >> >> >> >> >> >> BOOST_IS_ABSTRACT(fem::CTCBaseObj); >> >> BOOST_CLASS_EXPORT(fem::CTCSoil); >> >> BOOST_CLASS_EXPORT(fem::CTCBrickWall); >> >> >> >> void fill_with_build_in() >> >> { >> >> ... >> >> } >> >> >> >> >> >> void main() >> >> { >> >> fill_with_build_in(); >> >> boost_serialize_save_txt("boost_serialize.txt"); >> >> boost_serialize_save_bin("boost_serialize.bin"); >> >> boost_serialize_save_xml("boost_serialize.xml"); >> >> >> >> >> >> // some time later ... >> >> boost_serialize_open_txt("boost_serialize.txt"); >> >> boost_serialize_open_bin("boost_serialize.bin"); >> >> boost_serialize_open_xml("boost_serialize.xml"); >> >> } >> >> >> >> >> >> >> >> The xml save & open functions are in file >> fem_boost_serialize.cpp >> >> >> >> I attached the 3 files produced this way. I can also >> read back txt & >> >> binary versions. I'm wondering if somebody - hopefully >> Robert Ramey! >> >> - can tell me just by examining the output files, >> where should I look >> >> for the problem. Or do I have to debug the code line >> by line and see >> >> what wend wrong? It seems something is wrong with the >> very first >> >> shared_ptr serialization . >> >> >> >> I include some - not stand alone I'm afraid - source >> code files with >> >> this message just in case someone might take a look at >> them. I can >> >> tell that during the transition period from MFC to STL >> (possibly >> >> after that too), our objects derive from MFC CObject. >> Also at the >> >> moment, we still use MFC CString and not std::string. >> I had to copy >> >> CString to std::string for the serialization to work >> properly. Could >> >> there be a conflict somewhere there? Could it be the >> order in hpp >> >> files are included? >> >> >> >> >> >> Finally I use MS Visual C++ .net 2003 and boost v1.33.1. >> >> >> >> >> >> Thank you in advanced, >> >> >> >> >> >> >> >> Emmanuil Tsagarakis >> >> >> >> >> > >> > >> > >> >> _______________________________________________ >> >> Boost-users mailing list >> >> Boost-users(a)lists.boost.org >> >> http://lists.boost.org/mailman/listinfo.cgi/boost-users >> >> >> >> _______________________________________________ >> Boost-users mailing list >> Boost-users(a)lists.boost.org >> http://lists.boost.org/mailman/listinfo.cgi/boost-users >> >>

1 0

Re: [Boost-users] Boost python std::set
by Nat Goodspeed 05 May '06

05 May '06

From: boost-users-bounces(a)lists.boost.org [mailto:boost-users-bounces@lists.boost.org] On Behalf Of Craig Glasgow Sent: Friday, May 05, 2006 3:55 AM To: Boost-users(a)lists.boost.org Subject: [Boost-users] Boost python std::set Does anyone know how to write the python conversions to go between the C++ STL class std::set and Python sets? I've written simialar conversions between lists/tuples and std::vector and between dict and std::map by modifying the code in: scitbx/include/scitbx/boost_python/container_conversions.h mentioned in the FAQ. Boost python, however, while supporting dict, list, tuple does not appear to support set. Any ideas? [Nat] <grin> Your timing is excellent. Check the C++-sig mailing list from two days ago: http://article.gmane.org/gmane.comp.python.c%2B%2B/9682 In general, check the C++-sig mailing list for Boost.Python questions.

1 0

Boost python std::set
by Craig Glasgow 05 May '06

05 May '06

Hi Does anyone know how to write the python conversions to go between the C++ STL class std::set and Python sets? I've written simialar conversions between lists/tuples and std::vector and between dict and std::map by modifying the code in: scitbx/include/scitbx/boost_python/container_conversions.h mentioned in the FAQ. Boost python, however, while supporting dict, list, tuple does not appear to support set. Any ideas? Regards Craig

1 0

[boost-users] [serialization] I can't figure out what I'm doing wrong with xml_oarchive - please help
by TOL mtsagara 05 May '06

05 May '06

Hi to all, This is my very first post to boost-users! I've been using a couple of boost libraries for a while (shared_ptr & bind mostly). Write now we are in the process of transforming a project based on MFC collections, (with objects derived from CObject), to STL, and doing some search for a library that will allow object persistence the way MFC does, the obvious solution was boost::serialize. As I'm a newbie in mpl style code the first impression was terrifying! But after reading the documentation and experimenting a lot, I did manage to start using boost::serialize. In fact there is no need to be familiar with mpl. The library is excellent - an exemplary work, thank you Robert Ramey! Now my problem is with xml files. I can write a couple of classes both in txt and binary files. But I can't write the very same data in xml format. I believe I followed all guidelines but I had no luck ... :( The scenario in pseudo code is this: ////////////////////////////////////////////////////////////////////// // file fem.h #include <boost/archive/...> #include <boost/serialize/...> namespace fem { class CTCBase { ... }; typedef boost::shared_ptr<CTCBase> CBSPBase typedef std::list<CBSPBase> CBSPBaseList class CTCSoil { ... }; typedef boost::shared_ptr<CTCSoil> CBSPSoil typedef std::list<CBSPSoil> CBSPSoilList class CTCBrickWall { ... }; typedef boost::shared_ptr<CTCBrickWall> CBSPBrickWall typedef std::list<CBSPBrickWall> CBSPBrickWallList // list of soils CBSPSoilList s_lstSoils; // list of Brick Walls CBSPBrickWallList s_lstBrickWalls; // initial soil for each new foundation beam (also contained in above list) CBSPSoil s_bspSoil; // initial brick wall on each beam (also contained in above list) CBSPBrickWall s_bspBrick; } BOOST_CLASS_VERSION(fem::CTCBase, 1); BOOST_CLASS_VERSION(fem::CTCSoil, 1); BOOST_CLASS_VERSION(fem::CTCBrickWall, 1); ////////////////////////////////////////////////////////////////////// // file fem.cpp #include "fem.h" #include <boost/serialization/export.hpp> BOOST_IS_ABSTRACT(fem::CTCBaseObj); BOOST_CLASS_EXPORT(fem::CTCSoil); BOOST_CLASS_EXPORT(fem::CTCBrickWall); void fill_with_build_in() { ... } void main() { fill_with_build_in(); boost_serialize_save_txt("boost_serialize.txt"); boost_serialize_save_bin("boost_serialize.bin"); boost_serialize_save_xml("boost_serialize.xml"); // some time later ... boost_serialize_open_txt("boost_serialize.txt"); boost_serialize_open_bin("boost_serialize.bin"); boost_serialize_open_xml("boost_serialize.xml"); } The xml save & open functions are in file fem_boost_serialize.cpp I attached the 3 files produced this way. I can also read back txt & binary versions. I'm wondering if somebody - hopefully Robert Ramey! - can tell me just by examining the output files, where should I look for the problem. Or do I have to debug the code line by line and see what wend wrong? It seems something is wrong with the very first shared_ptr serialization . I include some - not stand alone I'm afraid - source code files with this message just in case someone might take a look at them. I can tell that during the transition period from MFC to STL (possibly after that too), our objects derive from MFC CObject. Also at the moment, we still use MFC CString and not std::string. I had to copy CString to std::string for the serialization to work properly. Could there be a conflict somewhere there? Could it be the order in hpp files are included? Finally I use MS Visual C++ .net 2003 and boost v1.33.1. Thank you in advanced, Emmanuil Tsagarakis

2 1

[Unit-Test] How to test private functions
by Leif Gruenwoldt 05 May '06

05 May '06

Is there a way to test the private functions of a class without moving them into public scope during test? What are some strategies for this? I was hoping to add "friend class boost::unit_test::test_suite;" to my class but that didn't work. -- Leif Gruenwoldt

8 11

[shmem] New bugfix version in Boost Vault and sandbox cvs
by Ion Gaztañaga 04 May '06

04 May '06

Hi to all, I've just uploaded to Boost Vault a new version of Shmem (0.92). You can download it here: http://boost-consulting.com/vault/index.php?direction=&order=&directory=Mem… Online documentation here: http://ice.prohosting.com/newfunk/boost/libs/shmem/ Changes in 0.92: ----------------- * Solved bugs in flat_xxx insert functions. * Solved ambiguous windows native function calls with some header orderings. * Changed named_mutex to support scoped_lock. * Added missing <exceptions.hpp> include in named_mutex. * Added get_instance_type, get_instance_length and get_instance_name functions to front-ends. Warning: -------- Peter Johson has detected a problems with Vc8 and global optimization (/Og). Some tests have shown a possible compiler bug. vc7.1 has no problems with /Og. I'm investigating this but meanwhile I recommend disabling /Og (which is deprecated in vc8) when using Shmem. Special thanks to Peter Johnson, Harold Pirtle and Jan Stetka for their help on this version. Regards, Ion

2 10

[date_time] What's the proper way to convert local to UTC?
by Scott 04 May '06

04 May '06

c_local_adjustor has the static method "utc_to_local" to easily convert to local time based on the machine TZ setting. So I figured it had a utc_from_local or local_to_utc method as well. Neither compiled, so I looked at the header file c_local_time_adjustor.hpp: //! Adjust to / from utc using the C API Yet there is only that one method in the class. Is the missing ability to convert from local to UTC an oversight or is it implemented somewhere else? What's the correct way to do it? Thanks, Scott

1 0

[serialization] Bug in text_oarchive but not in xml_oarchive?
by Bill Lear 04 May '06

04 May '06

I would assume that simply changing the following from xml_oarchive: ofstream ofs(filename.c_str(), ios::binary); boost::archive::xml_oarchive oa(ofs); oa << BOOST_SERIALIZATION_NVP(my_object); to text_oarchive (with, of course, appropriate changes to include the appropriate headers): ofstream ofs(filename.c_str(), ios::binary); boost::archive::text_oarchive oa(ofs); oa << BOOST_SERIALIZATION_NVP(my_object); should, a priori, not result in a core dump in the text_oarchive version? I am running under gcc 3.4.3, and and the Intel C++ compiler version 9.0, boost version 1.33.1. The text_oarchive fails with a segmentation violation on both compilers, and the xml_oarchive works with both. Cursory inspection of the xml output shows the xml to be well-formed, etc. Before I try to narrow this down further and provide a clear test that shows this error (which, given the complexity of the code and the size of the datasets, will be a considerable effort), I would like to know if my above assumption is correct, or if it is possible I have made a mistake somehow in writing my serialization methods. BTW, I am the same guy who has posted several (confirmed) bugs to this list: I am not a beginner, I have been using boost serialization for the better part of a year or so, so in general, I know what I'm doing. I say this simply because I don't want to waste a ton of time with this if I need not --- in short, to my mind, this looks like a clear example of a bug in the boost text_oarchive, but I perhaps may be wrong. Thank you. Bill

2 9