Re: [boost] [serialization] fast array serialization (10x speedup)

From: Robert Ramey (ramey_at_[hidden])
Date: 2005-10-11 12:09:31

troy d. straszheim wrote:
> Just to chime in, the modifications I came up with for size_t are very
> similar.  One must be able to treat size_t separately, #ifdefs and
> overloads of save() in the derived archive type won't cut it.  I
> gather that this isn't big news.
>
> These portability testing mods might come in handy if you're going to
> make these modifications.  Robert, you want this stuff?  I'm worried
> that the integration could start to become a hassle.  It's a lot of
> trivial changes to test modules, Jamfile tweaks, a modified tmpnam()
> and a modified remove(), the use of boost::random instead of
> std::rand() so that A is the same on all architectures, reseeding of
> the rng's in certain places, and the ubiquitous class "A" is either a
> portable one or a nonportable one depending on what archive you're
> testing.  If you don't specify the "I want to test portability" flag,
> the tests run as they do now, except for that they're unit tests, not
> test_main() tests.

I'm very interested in this.  I would like to see all the improvements 
you've
made incorporated into the tests - Its just that I'm bogged down on other 
stuff.
I also understand you've made headway in moving to the unit test platform
which I also want to see. It sounds like you've addressed a lot of the
pending issue regarding the tests and I want to see all this incorporated.
Don't get discouraged.

Of course we've got the problem that our consumption of testing resources
has hit its limit.

It seems to me that we're going to end up with a significantly enhanced
Jamfile or better yet several Jamfiles.  Of course this takes time but I 
would
much like to see this done.  When I get the current set of monkeys off
my back this can be looked at.

To summarize, I would like to see.

a) tests to be converted to unit_tests
b) tests improved to eliminate security issues - this really required making
another new animal - a temporary C++ file stream - which might or might
not currently exist somewhere.
c) investigation of stringstream.  It doesn't seem that stringsteam handles
code_cvt facet.  I don't know why this is but that's one reason I depend
on temporary files as much as I do.  So that's an interesting issue to
investigate.
d) a couple of different Jamfiles for selecting different kinds of testing
i) core library
ii) add-ons  - e.g stl serializations, shared_ptr, etc.
iii) portability - your name here
iv) performance - only makes sense in release mode anyway so its
handy to be separte.
v) plug-in.  I haven't been able to figure out how to make bjam do what
I want here.  But I haven't spent much time on it lately.

So, don't get discouraged because I haven't really had time to get to know 
your
stuff better.  Help me out with this and we'll get your name up there "in 
lights"

Robert Ramey

>
> The portability bit comes in when you specify
> --serialization-testdata-dir=/some/where.  I've changed tmpnam() to
> macro TESTFILE("unit_test_name") which returns
> /some/where/platform/version/compiler/archivetype.unit_test_name (for
> instance
> /path/to/Mac_OS/103300/GNU_C__version_4.0.0/portable_binary_archive.hpp.variant_A),
> and remove(), now finish("unit_test_name"), is a no-op when testing
> portability.  This allows you to afterwards run a little utility to
> walk the filesystem at /some/where and compare the checksums of the
> corresponding archivetype.unittestnames.  So one just points
> /some/where to network disk, or writes a little script to do a remote
> copy, and then runs the comparison.  My hunch is that a checksum won't
> be a good comparison for xml and text archives due to variances in the
> underlying implementations of << for primitive types, (I've only been
> hammering on a portable binary archive), but one could easily use a
> whitespace-ignorant "diff" or something.  It isn't ideal, as
> whitespace differences could still concievably trip things up, but
> this would require extensive modifications to every unit test, and I
> wasn't going to do them if there was a reasonable chance the changes
> wouldn't be used.

Keep this going - but let me have a chance to study it so we're on the same
page.  I want this stuff in there.

> Another problem is that it isn't easy to plug in your own archive
> type.  One must add files to libs/serialization/test and hack around
> with the jamfiles.  Needs a better interface.

How about run_archive_test.sh ? doesn't that do it?

Robert Ramey