Re: [Boost-users] [serializtion] Problem with multiplearchives andIntel compiler

From: Robert Ramey (ramey_at_[hidden])
Date: 2006-06-27 11:13:05

Bill Lear wrote:
> On Monday, June 26, 2006 at 21:13:42 (-0700) Robert Ramey writes:
>> Bill Lear wrote:
>> ...
>> I'm sorrty I can't give you a better answer.  But sometimes
>> thate is no other.  If you do find this, let us know so we
>> can roll it into the library.
>
> Not a problem.  I did do some quick profiling and all I found was that
> "operator <" was at the top of the list for the Intel compiler, but
> not for gcc.  I'll see if I can gather more info.

> I'm curious: for object tracking, do all objects that are tracked by
> pointer get put into the same STL collection (I think it's a set)?

yes

> This may sound crazy: but would it be possible to use a templated set
> based on the object type: so if I have 6,000 types of objects, I have
> 6,000 relatively small sets, instead of one set?  I was just thinking
> that there is no point in saving all pointers in one set (aside from
> simplicity, a big reason of course).  Perhaps this is just a red
> herring, but if that were possible, for very large data sets (we have
> HUUUUGE data sets), this might significantly improve performance.

My thinking is exactly the opposite.  I believe that putting all the
tracking into one collection will facilitate "load-leveling".

Here are some random observations on serialization performance
which are "on my list" but have yet to be implemented.  This is
largely because they won't effect most users and they can't just
be implemented without a lot of experimentation and profiling.
So the programming isn't the bottleneck - its determining what
to program.

a) Currently, I believe - without
the benefit of having profiled the code that for small data sets - most
of the time is consumed in setting up the STL collections used for
tracking.  This has been reported by one user who graciously took
the time to run some good tests.  So for lots of small serializations
(like for inter-process communication) there are benefits to
stream lining archive creation and perhaps re-use.

b) Currently, there are two slightly different sets used for tracking 
pointers.
One is used to implement delete_created_pointers.  These can
be consolidated into one.  This would help a lot for loading pointers.

c) For STL collections used by the archives - there is no way to
specify optimizations which can have a huge impact.   Longer term -
one would like to be able to reserve memory
in large chunks and/or also to specify a custom allocator.  This would
make a huge difference for saving/loading tracked information and
permit "tuning" based on experimentation.  This expermentation
based tuning is about the only way to do it since results are
necessarily going to depend up the standard library implementation.
Also, one size fits all can't do the job.  Setting large reserve sizes
for collections is going to conflict with other users who want to
invoke lots of tiny serializations.  Implementing this would require
lots of testing, update of API and its documentation, etc.  And
currently:
    i) testing serialization is already consuming too many resources
and we're looking to cut it back.
    ii) enhancing API and its documentation has already exhausted
my patience and I'm trying to cut back here as well.

> I'm  sure it would be TRIVIAL to try this out:-).

great - let us know how it works out.

> Next question: after serializing the object and destroying all of the
> archive variables, does boost get rid of all of the data structures
> with object pointers?

yes

> So, if I do:
>   {
>       ofstream ofs("file", ios::binary);
>
>       text_oarchive in(ifs);
>
>       in << object;
>   }
>
> when the scope exits, is the set tracking the object pointers
> destroyed?

yes

Finally, I got the impression from your previous posts that
your profiler exeperiments suggested that he problem was
with the lookup into the static variable sets which contain
one entry for each type and/or combination type-archive.
There are a lot of lookups into these tables but the tables
are small so it shouldn't consume a lot of time.  On the other
hand, the symptom you note - much slower when code
is instanticiated but never called - makes the tables bigger
(still very small) but doesn't change anything else.  Note that
this is ENTIRELY different than the issue of time consumed
by object tracking which would be effected only by the
archives actually used.

I believe that as programmers, we're too quick to fix stuff
like this based on speculation.  "To a man with a hammer
in his hand - whole world looks like a nail".  I think the
best thing is to step back and get more factual data.  I would
like to see the Jamfiles updated to add a few comprehensive
profiling tests to the serialization library tests.  These would
take too much time to be run by the tester's so they would
be commented out or conditioned on some environmenatal
variable.  Also VC 7.1 has the statistical profiler removed so
it would only work with GCC (mingw?).  This would be
a huge help.

> Thanks for all of your help.

LOL - I don't think I helped you much - but you're welcome
anyway.

>
> Bill

Robert Ramey