- Boost-users - lists.boost.org

Re: [Boost-users] [MultiIndex] re-indexing on load
by Mark Westcott 28 Apr '08

28 Apr '08

Hi Joaquin. > As you have found out, loading involves some amount of reindexing, > though the algorithm is designed to minimize this, let me summarize > how serialization of multi_index_containers is performed: > > Saving a multi_index_container internally consists in: > > 1. Saving each of the elements, in the order they are traversed > with index #0. > 2. For each index other than #0, some "readjusting" info is saved if > necessary. More on this later. > > Loading on the other hand reduces to: > > 1. Loading each element and inserting them with hinted insertion > at the end of the container: for index #0, hinting the end() position > reduces the nomber of key comparisons performed: for the rest > of key based indices, hinting end() usually does not have any impact > and insertion takes as many comparison ops as a regular insertion. > 2. After step 1 the elements in index #0 are listed in the exact > order they were originally saved, as it should be. As for the > rest of the indices, they can be differences in traversal order > for non key-based indices (sequenced and random_access) > and for ordered non-unique indices (equivalent elements will be > adjacent as ordered non-unique indices guarantee, but they might > be shuffled with respect to the original situation). The "adjusting" > info stored in saving step 2 is used here to restore the exact > original traversing order for every index of the container. > > So, to try to minimize your loading time you'd need to first make > a guess about where the most overhead lies: is it in step 1 > or step 2? Step 2 can only be significant if any of your indices > *other than #0* is of type ordered_non_unique, sequenced > or random_access. Assuming you indeed have a situation where > step 2 causes some impact, you can: > > a) if affordable, move the problematic index to position #0, so that it > won't get any adjustment later. > b) replace Boost.MultiIndex built-in serialization with the a handmade > method so as to avoid step 2 altogether. Please find attached a sketchy > implementation example of this idea. Note that this method does > not guarantee exact traversal order replication for indices other > than #0. > > In any case, do not expect spectacular improvements after applying > any of the approaches above, as more likely most of the overhead > comes from step 1 anyway. A more radical solution, but also one > with a very high impact in your application code, is to put the > multi_index_container in a memory-mapped file, so that serialization > reduces to dumping the memory segment. You can find an example > of this technique at: > > http://www.boost.org/doc/libs/1_35_0/libs/multi_index/doc/examples.html#exa… Thanks for the explanation and ideas. Based on the information you provided, I guessed that moving from hashed indices (where the hash function is relatively expensive) to sorted_non_unique (with a cheap comparison function) would improve things; and indeed that change has got the load hugely quicker. I then tried the 'poor man serialization' example that you attached. As you suspected, the performance effect of this was pretty small and so I guess it is indeed step 1 that is causing the issue (I have several ordered_non_uniques). I do like the memory mapped file idea, but as you say, the impact on the code is large; I think too large for the time being, though it may be sometime I come back to if needs be in the future. I think, for the time being at least, the performance is ok now (using ordered rather than hashed indices). Is there any chance that future versions of MI might be able to store the hashes/complete ordering whilst serializing, or is that Just Not Possible (TM)? Thanks again, Mark

2 1

portable binary archive major bugfix
by Pfligersdorffer, Christian 28 Apr '08

28 Apr '08

Hello everybody, if you're using or evaluating my portable binary archives be sure to get version 2.0 from the boost vault (http://www.boost-consulting.com/vault/) (Remeber it only works with boost 1.33 and 1.34.) Version 1.x contained a serious bug that effectively transformed most of negative integral values into positive values! For example the two numbers -12 and 234 were stored in the same 8-bit pattern and later always restored to 234. This was fixed in this version in a way that does not change the interpretation of existing archives that happend to work because there were no negative numbers - as is unfortunately the case in our application or the bug would have surfaced long ago. Archives created by version 2.0 and containing negative numbers will raise an integer type size exception when reading it with version 1.x. Thanks to Markus Frohnmaier for testing the archives and finding the bug. PS @ Robert Ramey: Your portable binary example also suffers from this bug. In fact I felt free to copy it from you ;-) Ah, I see now that you fixed it in version 1.35. Didn't think you'd bother to since you wanted to remove it completely... Regards, -- Christian Pfligersdorffer Software Engineering http://www.eos.info

1 0

Re: [Boost-users] a problem about boost::regex
by Guo Xu 28 Apr '08

28 Apr '08

Hi, John I have a string splited by "\0" ( it means {'\\','0'}, the bellow is the same ), for example: "\0xxxx\0yyyy\0zzzz\0". And there is some special situation like "\0xxx\\0\0yyy\\\0zzz\0" , you can think it is similarly to c style string, the two '\\' means a single '\' and the '\\0' is not a spliter. i want to split them in to 0) xxx\\0 1) yyy\\ c) zzz how can i do it in boost::regex? thx,all > Message: 7 > Date: Sat, 26 Apr 2008 12:52:43 +0100 > From: "John Maddock" <john(a)johnmaddock.co.uk> > Subject: Re: [Boost-users] a problem about boost::regex > To: <boost-users(a)lists.boost.org> > Message-ID: <002d01c8a794$0d593e50$eeea0352@fuji> > Content-Type: text/plain; format=flowed; charset="UTF-8"; > reply-type=original > > Guo Xu wrote: > >> a regex expression: > >> (?<=[^\\](?:\\\\)*)\\0 > >> can work well in c#, but error in boost::regex > >> why? > > Variable length lookbehind isn't supported in Boost.Regex (or in Perl for > that matter). > > Regards, John. > >

1 0

re; Boost CGI
by Brad 28 Apr '08

28 Apr '08

Forget this post, as one of the guys on IRC just pointed out, Darren has recently submitted changes via SVN so is obviously still active. Sorry for the noise guys. Cheers, Brad

1 0

[serialization] Stateful serialization
by Scott Howlett 27 Apr '08

27 Apr '08

Hello, In a recent post, I asked whether it was possible to swap deserialized pointer values for already-existing ones so I could merge the incoming data with objects already in memory. I guess the answer is no... An alternative is to serialize some other identifier instead of writing pointers, and then on de-serialization I find my existing objects via those identifiers. This works fine, but it leads to another question: How do I make my internal objects available to the deserialization routines so that they have an opportunity to look up the necessary information? Or more generally, How do I do stateful serialization / deserialization? For example, maybe I want to supply options that fine-tune how my data is serialized or deserialized. The only parameter that is passed around to all the serialization functions is the Archive, so if I want to add options or other context information, it seems like I should add this to the Archive. That's simple enough to do; I just made a wrapper Archive that can sit atop any other Archive and contain any other options / context data I need. On the one hand it seems sneaky to hijack the Archive for this purpose, but on the other hand iostreams do basically the same thing via the various ios_base calls (setf, width, precision, etc.) so maybe it's not so bad. Am I perhaps overlooking some facility already present in the serialization library? Thanks, Scott

3 3

[serialization] Making my library dll serializable
by a.gray＠atlantiscyberspace.com 27 Apr '08

27 Apr '08

Hello everyone, I'm new to boost in general, as well as to the serialization library, so I expect that I'm probably overlooking a few things. I have searched the archives, and I've seen threads that are clearly related to what I'm doing, but their approaches were a bit different than mine (ie. some had the option of including concrete classes directly). General Info: I'm using boost 1.35.0. My target platform is Windows XP. I'm building with MSVC8 SP1. What I Need: I'm building an interface-driven library (all the exposed classes are pure abstract interfaces). This library resides in its own DLL, but we can assume that it is always loaded when we're serializing data. The concrete classes in this library (not exposed) implement boost serialization support. External applications should be able to link against boost, create archives, and use them to serialize my library's objects through my library's interface pointers. What I Tried: I've attached an MSVC8 solution with a sample executable and dll. The library exposes an interface, hides a concrete implementation of the interface, and exposes a factory function for retrieving an instance. The application retrieves an instance of the class, instantiates a polymorphic binary archiver, and serializes the object through the interface pointer using the archiver. What Goes Wrong: In the release configuration, the test program appears to work. In the debug configuration, the app fails an assert (extended_type_info.cpp:74 lookup(eti) == m_self->m_map.end()). This seems to be the multiple-registration issue which has been discussed in earlier threads. Why doesn't the assertion fail in release? My Workaround: I've seen it mentioned in earlier threads that I should move the actual serialization calls into the DLL. To that end, I created a MyLibrarySerializer class which simply forwards the serialization call. This seems to work, but since I just started using this library, I can't tell if I've overlooked a better solution or if I'm introducing some obscure bugs using this method. Questions: Will my solution work the way I expect it to (Am I overlooking any major pitfalls)? Am I overlooking a better solution (Is there a way to accomplish this without introducing the MyLibrarySerializer class)? Why does my original solution appear to work in the release configuration? Threads I'm using for reference: http://thread.gmane.org/gmane.comp.lib.boost.user/8281 http://thread.gmane.org/gmane.comp.lib.boost.user/13440 http://thread.gmane.org/gmane.comp.lib.boost.user/28735 http://thread.gmane.org/gmane.comp.lib.boost.user/30837 Thanks for your time and patience, Andrew Gray

3 2

math
by Hicham Mouline 27 Apr '08

27 Apr '08

Hello, Are there libraries with class templates for the following concepts: . notion of mathematical function f(x1,x2,.,xn), and perhaps derivation/integration . any-order multidim partial differential equations? . multidim stochastic differential equations? Best rds,

2 1

boost filesystem: acessing files with their extensions
by Mohan gupta 27 Apr '08

27 Apr '08

- hello every one, can anyone please tell me is there any way to find the files in a directory with some exension say .txt,.doc, .cpp etc ......and then use them ???? all help is appreciated mohan gupta

5 12

problems linking libraries in debian linux
by Shutters, Zachary A 26 Apr '08

26 Apr '08

I followed the instructions in the getting started guide but I am still having troubles with getting the libraries linked. In the guide the way it shows to compile the second program is like this: $ c++ -I path/to/boost_1_35_0 example.cpp -o example \ -L~/boost/lib/ -lboost_regex-gcc34-mt-d-1_35 Here is how I translated that to my system: $ g++ -I /usr/local/include/boost_1_35/ boostTest2.cpp -L/usr/local/lib/ -lboost_regex-gcc41-mt But when I run this I get this error: zach@debian:~/code$ ./a.out < jayne.txt ./a.out: error while loading shared libraries: libboost_regex-gcc41-mt-1_35.so.1.35.0: cannot open shared object file: No such file or directory This file does exist, here is the listing of my /usr/local/lib directory: libboost_date_time-gcc41-mt-1_35.so libboost_date_time-gcc41-mt-1_35.so.1.35.0 libboost_date_time-gcc41-mt.so libboost_filesystem-gcc41-mt-1_35.so libboost_filesystem-gcc41-mt-1_35.so.1.35.0 libboost_filesystem-gcc41-mt.so libboost_graph-gcc41-mt-1_35.so libboost_graph-gcc41-mt-1_35.so.1.35.0 libboost_graph-gcc41-mt.so libboost_prg_exec_monitor-gcc41-mt-1_35.so libboost_prg_exec_monitor-gcc41-mt-1_35.so.1.35.0 libboost_prg_exec_monitor-gcc41-mt.so libboost_program_options-gcc41-mt-1_35.so libboost_program_options-gcc41-mt-1_35.so.1.35.0 libboost_program_options-gcc41-mt.so libboost_python-gcc41-mt-1_35.so libboost_python-gcc41-mt-1_35.so.1.35.0 libboost_python-gcc41-mt.so libboost_regex-gcc41-mt-1_35.so libboost_regex-gcc41-mt-1_35.so.1.35.0 libboost_regex-gcc41-mt.so libboost_serialization-gcc41-mt-1_35.so libboost_serialization-gcc41-mt-1_35.so.1.35.0 libboost_serialization-gcc41-mt.so libboost_signals-gcc41-mt-1_35.so libboost_signals-gcc41-mt-1_35.so.1.35.0 libboost_signals-gcc41-mt.so libboost_system-gcc41-mt-1_35.so libboost_system-gcc41-mt-1_35.so.1.35.0 libboost_system-gcc41-mt.so libboost_test_exec_monitor-gcc41-mt-1_35.a libboost_test_exec_monitor-gcc41-mt.a libboost_thread-gcc41-mt-1_35.so libboost_thread-gcc41-mt-1_35.so.1.35.0 libboost_thread-gcc41-mt.so libboost_unit_test_framework-gcc41-mt-1_35.so libboost_unit_test_framework-gcc41-mt-1_35.so.1.35.0 libboost_unit_test_framework-gcc41-mt.so libboost_wave-gcc41-mt-1_35.so libboost_wave-gcc41-mt-1_35.so.1.35.0 libboost_wave-gcc41-mt.so libboost_wserialization-gcc41-mt-1_35.so libboost_wserialization-gcc41-mt-1_35.so.1.35.0 libboost_wserialization-gcc41-mt.so libpurple.la libpurple.so libpurple.so.0 libpurple.so.0.4.1 I know this is probably a linux problem and not a boost problem, but I cannot figure it out. Any ideas?

2 1

sheduling in boost::thread
by Mohan gupta 26 Apr '08

26 Apr '08

hello everyone, can anyone please tell me how is sheduling between the threads is done in boost ::thread are there any system calls involved or just its the appilcation of repeated function calls or something else thank you mohan gupta

2 1