Re: [boost] [serialization] fast array serialization (10x speedup)

From: David Abrahams (dave_at_[hidden])
Date: 2005-11-25 19:16:38

"Robert Ramey" <ramey_at_[hidden]> writes:

> I've take a look at your benchmark.cpp.
>
> First of all its very nice and simple and shows an understanding how
> the primitive i/o is isolated from the archives that use it.
>
> Its a step in the right direction.  But I see some problems.  The
> usage of std::vector<char> isn't what I would expect for an output
> buffer.  You arn't using this in your own archives are you?
>
> I modified the to use a simple buffer output closer to what I would
> expect to use if I were going to make a primitive buffer output.
> BTW - that would be a very nice addition.  This would be
> much faster than using strstream as is being use now.
>
> Here I the results with the program modified in this way
>
> For value type set to char I get
>
> Time using serialization library: 0.797
> Size is 100000004
> Time using direct calls to save in a loop: 0.297 (1)
> Size is 100000000
> Time using direct call to save_array: 0.203
> Size is 100000000
>
> and for value_type set to double I get
>
> Time using serialization library: 0.109 (3)
> Size is 100000004
> Time using direct calls to save in a loop: 0.078 (2)
> Size is 100000000
> Time using direct call to save_array: 0.25
> Size is 100000000

First of all, I can't reproduce those results or anything like them
with your code.  Did you run the program three times and throw out the
first result (to make sure the caches were full and you weren't seeing
the freak effect of some other process?)  On the code you posted,
using vc-7.1, I get:

Run #2
  Time using serialization library: 1.015
  Size is 100000004
  Time using direct calls to save in a loop: 0.36
  Size is 100000000
  Time using direct call to save_array: 0.25
  Size is 100000000

Run #3
  Time using serialization library: 1.078
  Size is 100000004
  Time using direct calls to save in a loop: 0.359
  Size is 100000000
  Time using direct call to save_array: 0.234
  Size is 100000000

Using MSVC-8.0 I get:

Run #2
  Time using serialization library: 1.5
  Size is 100000004
  Time using direct calls to save in a loop: 0.484
  Size is 100000000
  Time using direct call to save_array: 0.235
  Size is 100000000

Run #3
  Time using serialization library: 1.594
  Size is 100000004
  Time using direct calls to save in a loop: 0.516
  Size is 100000000
  Time using direct call to save_array: 0.234
  Size is 100000000

Using gcc-4.0.2 I get:

Run #2
  Time using serialization library: 0.547
  Size is 100000004
  Time using direct calls to save in a loop: 0.344
  Size is 100000000
  Time using direct call to save_array: 0.25
  Size is 100000000

Run #3
  Time using serialization library: 0.547
  Size is 100000004
  Time using direct calls to save in a loop: 0.343
  Size is 100000000
  Time using direct call to save_array: 0.251
  Size is 100000000

This is on a 2.00GHz Pentium M on Windows XP.

Furthermore, it's not a fair comparison unless you first measure the
number of bytes you have to save so you can preallocate the buffer.
In general the only way to do that is with a special counting archive,
so you have to account for the time taken up by the counting.  Of
course we did that test too.  The code and test results are attached.

In case it isn't obvious to you by now Matthias Troyer is a
world-class expert in high performance computing.  You don't get to be
a recognized authority in that area without developing the ability to
create tests that accurately measure performance.  You also develop
some generalized knowledge about what things will lead to slowdowns
and speedups.  It's really astounding that you manage challenge every
assertion he makes in a domain where he is an expert and you are not,
especially in a domain with so many subtle pitfalls waiting for the
naive tester.

> a) the usage of save_array does not have a huge
> effect on performance.  It IS measureable.  It seems
> that it saves about 1/3 the time over using a loop
> of saves in the best case. (1)

In the best case, even with your flawed test, it's a factor of 2 as
shown above.

> b) In the worst case, its even slower than a loop of saves!!! (2)
> and even slower than the raw serialization system (3)

That result is completely implausible.  If you can get someone else to
reproduce it using a proper test protocol I'll be *quite* impressed.

> c) the overhead of the serialization library isn't too
> bad.  It does show up when doing 100M characters
> one by one, but generally it doesn't seem to be a big
> issuues.
>
> In my view, it does support my contention that
> implementing save_array - regardless of how it is
> in fact implemented - represents a premature optimization.
> I suspect that the net benefit in the kind of scenario you
> envision using it will be very small.
>
> Obviously, this test raises more questions than it
> answers 

Like what questions?