Re: [boost] How do folks test serialization code?

Subject: Re: [boost] How do folks test serialization code?
From: Robert Ramey (ramey_at_[hidden])
Date: 2013-08-04 19:33:53

John Maddock wrote:

Very good questions - in ten years no one has asked this.

> Title almost says it all, how do you verify that your classes
> serialization code is truly platform independent,

This would not apply to binary archives as these are not guarenteed to
be portable.

The case were archives are created on one platform and
loaded on another is not explicitly tested.

>and doesn't accidently break later?

Hmmm I'm not sure how one could test that things aren't going
to break later.

> I have some trivial tests that round trip to
> a stringstream using boost::archive::text_i/oarchive, but of course
> that only verifies that platform-specific serialization works.  So
> what do folks do - generate text archives on different platforms,
> deserialize and compare?

> How does Boost.Serialization itself verify this?

It doesn't.

It also doesn't explicitly test that version x can read all
archives created with versions < x.

Generally I have not had a problem with this EXCEPT
in one glaring case which haunts me to this day.

What the serialization library test suite DOES do is:

It has tests for all serializable data types - around 50.
It runs all these tests with each archive class (4) so that's
about 200 tests (+ some more for details).  On my own
machine, I run both the debug/release and static/dynamic
versions.  and msvc and gcc compilers.  It does this via
a tricked out bjam script file.

So the total number of tests is about
50 types * 4 archives * 2 debug/release * 2 static/dynamic
* 2 compilers (gcc and msvc) or around 1600 tests.
(actually it's a little more, I forget how that happened).

This takes a while.  It has been referred to on this list
as "carpet bombing".  I took that as a complement.

I would love to see this enhanced by doing cross platform
testing.  Say there were 5 platforms- we could multiply the
above by 25.

I would also love to see backward compatibility testing -
that is test that the latest program version X could read
all archives created by previous versions of the program.

I dabbled a little in this, but didnt' make much progress.

I'm actually afraid to enhance the library due to concerns
that the testing requirements might be even bigger.  Also
I've become somewhat skeptical of the the scalability of
the boost testing methodology.  I'm actually working on
addressing this.

And although testing is indispensible, I've also become more
aware of it's limitations for validating code correctness.

I would encourage anyone who uses the serialization library
(or any boost library) to actually run the tests for that library
on his environment.  This is the only way to really address
the scalabilty demon.  Having said this, I've never run
accross anyone who actually did it.

Robert.Ramey