Re: [boost] How do folks test serialization code?

Subject: Re: [boost] How do folks test serialization code?
From: Jens Weller (JensWeller_at_[hidden])
Date: 2013-08-04 19:10:26

Hi,

Interesting topic. I've worked a few years ago for sometime at the vodafone testcenter in Düsseldorf,
so I'm kind of familiar with testing. I'll just dump my thoughts in this mail.
Its been a couple of years, so I'm not an expert in testing...
And please excuse starting this mail with top-posting... ;)

> Von: "Robert Ramey" <ramey_at_[hidden]>
> An: boost_at_[hidden]
> Betreff: Re: [boost] How do folks test serialization code?
>
> John Maddock wrote:
>
> Very good questions - in ten years no one has asked this.
Working in the industries for 6 years as freelancer on various projects/companies, only one pattern has kept true and keeps emerging:
Testing isn't standard, its the exception. At least in C++ Land.

Its nice to have all the testcases, but most companies don't cover their applications with tests.
In fact, I haven't seen a single company using tests with C++ in their application in 6 years.
Some of them do test, but not in codelevel. No unittests or anything like it.

> The case were archives are created on one platform and
> loaded on another is not explicitly tested.
>
> >and doesn't accidently break later?
>
> Hmmm I'm not sure how one could test that things aren't going
> to break later.

First one should ask what to test in this case.
The user code, which uses the library, or the library it self?
Of course you should handle the tests for your app loading the various versions of your format
with the current/next release in testing. Simply have a couple of test archives produced by your application
as a test set available for each previous version.

> > I have some trivial tests that round trip to
> > a stringstream using boost::archive::text_i/oarchive, but of course
> > that only verifies that platform-specific serialization works.  So
> > what do folks do - generate text archives on different platforms,
> > deserialize and compare?
>
> > How does Boost.Serialization itself verify this?
>
> It doesn't.
>
> It also doesn't explicitly test that version x can read all
> archives created with versions < x.
>
> Generally I have not had a problem with this EXCEPT
> in one glaring case which haunts me to this day.

So, when using serialization with boost 1.x its different then 1.y?
I know creating this kind of tests is annoying, but still people should be able to upgrade their installations.
But you seem to be stable enough, as I haven't heared complains that archives differ from one to another version of boost.

> What the serialization library test suite DOES do is:
>
> It has tests for all serializable data types - around 50.
> It runs all these tests with each archive class (4) so that's
> about 200 tests (+ some more for details).  On my own
> machine, I run both the debug/release and static/dynamic
> versions.  and msvc and gcc compilers.  It does this via
> a tricked out bjam script file.
>
> So the total number of tests is about
> 50 types * 4 archives * 2 debug/release * 2 static/dynamic
> * 2 compilers (gcc and msvc) or around 1600 tests.
> (actually it's a little more, I forget how that happened).
>
> This takes a while.  It has been referred to on this list
> as "carpet bombing".  I took that as a complement.
>
> I would love to see this enhanced by doing cross platform
> testing.  Say there were 5 platforms- we could multiply the
> above by 25.

With running tests in the cloud that could be possible indeed,
but this adds up quickly. 5 Plattforms today would maybe Win, Linux, Mac, Android & iOS.
But there is much more, so testing this - while not being impossible -
will keep growing into complexity, until your number of tests will not be able to run within a certain timeframe.
One should also test all compilers on all available Plattforms f.e.

> I would also love to see backward compatibility testing -
> that is test that the latest program version X could read
> all archives created by previous versions of the program.
>
> I dabbled a little in this, but didnt' make much progress.
>
> I'm actually afraid to enhance the library due to concerns
> that the testing requirements might be even bigger.  Also
> I've become somewhat skeptical of the the scalability of
> the boost testing methodology.  I'm actually working on
> addressing this.

That does not sound to good. Maybe its time to move on, and think about serialization II,
as we have signals2 now? Would that make sense?

kind regards,

Jens Weller