How do folks test serialization code?
Title almost says it all, how do you verify that your classes serialization code is truly platform independent, and doesn't accidently break later? I have some trivial tests that round trip to a stringstream using boost::archive::text_i/oarchive, but of course that only verifies that platform-specific serialization works. So what do folks do - generate text archives on different platforms, deserialize and compare? How does Boost.Serialization itself verify this? Cheers, John.
John Maddock wrote: Very good questions - in ten years no one has asked this.
Title almost says it all, how do you verify that your classes serialization code is truly platform independent,
This would not apply to binary archives as these are not guarenteed to be portable. The case were archives are created on one platform and loaded on another is not explicitly tested.
and doesn't accidently break later?
Hmmm I'm not sure how one could test that things aren't going to break later.
I have some trivial tests that round trip to a stringstream using boost::archive::text_i/oarchive, but of course that only verifies that platform-specific serialization works. So what do folks do - generate text archives on different platforms, deserialize and compare?
How does Boost.Serialization itself verify this?
It doesn't. It also doesn't explicitly test that version x can read all archives created with versions < x. Generally I have not had a problem with this EXCEPT in one glaring case which haunts me to this day. What the serialization library test suite DOES do is: It has tests for all serializable data types - around 50. It runs all these tests with each archive class (4) so that's about 200 tests (+ some more for details). On my own machine, I run both the debug/release and static/dynamic versions. and msvc and gcc compilers. It does this via a tricked out bjam script file. So the total number of tests is about 50 types * 4 archives * 2 debug/release * 2 static/dynamic * 2 compilers (gcc and msvc) or around 1600 tests. (actually it's a little more, I forget how that happened). This takes a while. It has been referred to on this list as "carpet bombing". I took that as a complement. I would love to see this enhanced by doing cross platform testing. Say there were 5 platforms- we could multiply the above by 25. I would also love to see backward compatibility testing - that is test that the latest program version X could read all archives created by previous versions of the program. I dabbled a little in this, but didnt' make much progress. I'm actually afraid to enhance the library due to concerns that the testing requirements might be even bigger. Also I've become somewhat skeptical of the the scalability of the boost testing methodology. I'm actually working on addressing this. And although testing is indispensible, I've also become more aware of it's limitations for validating code correctness. I would encourage anyone who uses the serialization library (or any boost library) to actually run the tests for that library on his environment. This is the only way to really address the scalabilty demon. Having said this, I've never run accross anyone who actually did it. Robert.Ramey
Hi, Interesting topic. I've worked a few years ago for sometime at the vodafone testcenter in Düsseldorf, so I'm kind of familiar with testing. I'll just dump my thoughts in this mail. Its been a couple of years, so I'm not an expert in testing... And please excuse starting this mail with top-posting... ;)
Von: "Robert Ramey" <ramey@rrsd.com> An: boost@lists.boost.org Betreff: Re: [boost] How do folks test serialization code?
John Maddock wrote:
Very good questions - in ten years no one has asked this. Working in the industries for 6 years as freelancer on various projects/companies, only one pattern has kept true and keeps emerging: Testing isn't standard, its the exception. At least in C++ Land.
Its nice to have all the testcases, but most companies don't cover their applications with tests. In fact, I haven't seen a single company using tests with C++ in their application in 6 years. Some of them do test, but not in codelevel. No unittests or anything like it.
The case were archives are created on one platform and loaded on another is not explicitly tested.
and doesn't accidently break later?
Hmmm I'm not sure how one could test that things aren't going to break later.
First one should ask what to test in this case. The user code, which uses the library, or the library it self? Of course you should handle the tests for your app loading the various versions of your format with the current/next release in testing. Simply have a couple of test archives produced by your application as a test set available for each previous version.
I have some trivial tests that round trip to a stringstream using boost::archive::text_i/oarchive, but of course that only verifies that platform-specific serialization works. So what do folks do - generate text archives on different platforms, deserialize and compare?
How does Boost.Serialization itself verify this?
It doesn't.
It also doesn't explicitly test that version x can read all archives created with versions < x.
Generally I have not had a problem with this EXCEPT in one glaring case which haunts me to this day.
So, when using serialization with boost 1.x its different then 1.y? I know creating this kind of tests is annoying, but still people should be able to upgrade their installations. But you seem to be stable enough, as I haven't heared complains that archives differ from one to another version of boost.
What the serialization library test suite DOES do is:
It has tests for all serializable data types - around 50. It runs all these tests with each archive class (4) so that's about 200 tests (+ some more for details). On my own machine, I run both the debug/release and static/dynamic versions. and msvc and gcc compilers. It does this via a tricked out bjam script file.
So the total number of tests is about 50 types * 4 archives * 2 debug/release * 2 static/dynamic * 2 compilers (gcc and msvc) or around 1600 tests. (actually it's a little more, I forget how that happened).
This takes a while. It has been referred to on this list as "carpet bombing". I took that as a complement.
I would love to see this enhanced by doing cross platform testing. Say there were 5 platforms- we could multiply the above by 25.
With running tests in the cloud that could be possible indeed, but this adds up quickly. 5 Plattforms today would maybe Win, Linux, Mac, Android & iOS. But there is much more, so testing this - while not being impossible - will keep growing into complexity, until your number of tests will not be able to run within a certain timeframe. One should also test all compilers on all available Plattforms f.e.
I would also love to see backward compatibility testing - that is test that the latest program version X could read all archives created by previous versions of the program.
I dabbled a little in this, but didnt' make much progress.
I'm actually afraid to enhance the library due to concerns that the testing requirements might be even bigger. Also I've become somewhat skeptical of the the scalability of the boost testing methodology. I'm actually working on addressing this.
That does not sound to good. Maybe its time to move on, and think about serialization II, as we have signals2 now? Would that make sense? kind regards, Jens Weller
Jens Weller wrote:
Hi,
With running tests in the cloud that could be possible indeed, but this adds up quickly. 5 Plattforms today would maybe Win, Linux, Mac, Android & iOS. But there is much more, so testing this - while not being impossible - will keep growing into complexity, until your number of tests will not be able to run within a certain timeframe. One should also test all compilers on all available Plattforms f.e.
In my not so humble opinion the "defnitive" solution to this problem is the following. a) User downloads some libraries. b) There is sort of an installation process which makes it inconvenient to skip testing as part of that installation. c) The tests are run on the users environment d) the results are posted in a common database. This a) spreads the testing throughout the known universe b) is scalable c) guarentees that libraries are tested in all environoments actually being used. d) doesn't waste time on testing platforms which aren't being used. It is a natural consequence of the (slow) evolution of boost into a more decoupled system.
I'm actually afraid to enhance the library due to concerns that the testing requirements might be even bigger. Also I've become somewhat skeptical of the the scalability of the boost testing methodology. I'm actually working on addressing this.
That does not sound to good. Maybe its time to move on, and think about serialization II, > as we have signals2 now? Would that make sense?
How would that diminishing the testing load. As boost adds more libraries and platforms proliferate, the amount of resources consumed by testing is growing faster than boost itself is. Robert Ramey
Very good questions - in ten years no one has asked this. Working in the industries for 6 years as freelancer on various projects/companies, only one pattern has kept true and keeps emerging: Testing isn't standard, its the exception. At least in C++ Land.
Its nice to have all the testcases, but most companies don't cover their applications with tests. In fact, I haven't seen a single company using tests with C++ in their application in 6 years. Some of them do test, but not in codelevel. No unittests or anything like it.
Pretty sad state of affairs that. I always find bugs in my code through "carpet-bomb" testing, generally in other folks code as well, just saying... John.
On Mon, Aug 5, 2013 at 3:10 AM, Jens Weller <JensWeller@gmx.de> wrote:
Working in the industries for 6 years as freelancer on various projects/companies, only one pattern has kept true and keeps emerging: Testing isn't standard, its the exception. At least in C++ Land.
Its nice to have all the testcases, but most companies don't cover their applications with tests. In fact, I haven't seen a single company using tests with C++ in their application in 6 years. Some of them do test, but not in codelevel. No unittests or anything like it.
I must say my experience is quite the contrary. Companies I worked in (in Russia) use unit tests in one way or another, and from what I've heard many other companies that deal with software development also use unit tests and/or manual testing where unit tests are impossible/unreasonable. Another question is how complete the test coverage is but that's another question. I would say that any company that has software development as a considerable part of its business has to use automated tests or this company is coming to its failure.
Very good questions - in ten years no one has asked this.
Oh :-(
Title almost says it all, how do you verify that your classes serialization code is truly platform independent,
This would not apply to binary archives as these are not guarenteed to be portable.
The case were archives are created on one platform and loaded on another is not explicitly tested.
and doesn't accidently break later?
Hmmm I'm not sure how one could test that things aren't going to break later.
Allow me to explain - I've experimented with adding serialization support to Boost.Multiprecision - amounst other things this would allow those types to be used with MPI for massively parrellel applications (but I digress). Now, binary archives are no issue, a simple loopback test is enough. But since I know the internals of the classes are platform specific, I've had to come up with a serialization format (for cpp_int for example) which is platform independent. Or at least I think it is. I tend to assume that anything not tested doesn't work. Maybe I'll just create some sample archives on 32 and 64 bit platforms and try reading them. John.
On 08/05/2013 09:56 AM, John Maddock wrote:
But since I know the internals of the classes are platform specific, I've had to come up with a serialization format (for cpp_int for example) which is platform independent. Or at least I think it is. I
I am interested in learning more about how this format for cpp_int is. I maintain an encoding format [1], which I would like to extend with variable-length integers.
tend to assume that anything not tested doesn't work. Maybe I'll just create some sample archives on 32 and 64 bit platforms and try reading them.
In my archives I do not use round tripping, but rather explicitly check the binary output of the oarchive, and likewise feed a binary input to the iarchive (see [2]) [1] http://transenc.sourceforge.net/ [2] http://sourceforge.net/p/protoc/code/ci/master/tree/test/transenc/
But since I know the internals of the classes are platform specific, I've had to come up with a serialization format (for cpp_int for example) which is platform independent. Or at least I think it is. I
I am interested in learning more about how this format for cpp_int is. I maintain an encoding format [1], which I would like to extend with variable-length integers.
I've just committed the code so you can try for yourself, but basically rather than storing a sequence of "limbs" (which may vary in size from one platform/compiler to another), it stores a sequence of bytes instead. The bytes are extracted using high level operations (shifts and bitmasks) so there's no issue with endianness etc. That's what I hope anyway ;-) John.
John Maddock wrote:
But since I know the internals of the classes are platform specific, I've had to come up with a serialization format (for cpp_int for example) which is platform independent. Or at least I think it is. I
I am interested in learning more about how this format for cpp_int is. I maintain an encoding format [1], which I would like to extend with variable-length integers.
I've just committed the code so you can try for yourself, but basically rather than storing a sequence of "limbs" (which may vary in size from one platform/compiler to another), it stores a sequence of bytes instead. The bytes are extracted using high level operations (shifts and bitmasks) so there's no issue with endianness etc.
As I said before - this shouldn't be necessary.
That's what I hope anyway ;-)
John.
_______________________________________________ Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
I've just committed the code so you can try for yourself, but basically rather than storing a sequence of "limbs" (which may vary in size from one platform/compiler to another), it stores a sequence of bytes instead. The bytes are extracted using high level operations (shifts and bitmasks) so there's no issue with endianness etc.
As I said before - this shouldn't be necessary.
Consider a 128-bit integer type, internally this could be: * Expressed as 8 16-bit integers (intmax_t=32 bits). * Expressed as 4 32-bit integers (intmax_t=64 bits). * Expressed as a single 128-bit (native) integer. How would you serialize from one format to a different one without breaking it down into "portably small" chunks? John.
John Maddock wrote:
I've just committed the code so you can try for yourself, but basically rather than storing a sequence of "limbs" (which may vary in size from one platform/compiler to another), it stores a sequence of bytes instead. The bytes are extracted using high level operations (shifts and bitmasks) so there's no issue with endianness etc.
As I said before - this shouldn't be necessary.
Consider a 128-bit integer type, internally this could be:
* Expressed as 8 16-bit integers (intmax_t=32 bits). * Expressed as 4 32-bit integers (intmax_t=64 bits). * Expressed as a single 128-bit (native) integer.
How would you serialize from one format to a different one without breaking it down into "portably small" chunks?
Of course you have to define the 128 bit integer in terms of some other types. But THOSE types should already be portably serializable. another way to deal with this is to work a at a more primitive level. By marking the 128 integer type as "primitive" and defining an output for output/input stream operarate - text archives should "just work". Of course these operators will have to be implemented with portability in mind so the issue is moved to somewhere else. But then one probably needs to define these operators in any case so for text archives serialization comes for free. for binary and portable binary archives, one might need to specialize the serialization operators. For binary archives there's not much if anything to do. For portable binary archives, one might have to make a minor specialization. The portable binary archive includes code for handling endianness of arbitrarily long integer types. (note that portable binary archive isn't complete in anycase in that it doesn't handle floating point types - a good GSOC project if one wants one.) So you've got a couple of ways of addressing this. Robert Ramey
How would you serialize from one format to a different one without breaking it down into "portably small" chunks?
Of course you have to define the 128 bit integer in terms of some other types. But THOSE types should already be portably serializable.
Sure, but as I said, exactly what those types are, and how many of them there are, changes from one platform/compiler to another.
another way to deal with this is to work a at a more primitive level. By marking the 128 integer type as "primitive" and defining an output for output/input stream operarate - text archives should "just work". Of course these operators will have to be implemented with portability in mind so the issue is moved to somewhere else. But then one probably needs to define these operators in any case so for text archives serialization comes for free.
You mean rely on the usual streaming << >> operators? For sure I could do that, but full string conversion is crazy expensive compared to the byte-by-byte scheme I have now.
for binary and portable binary archives, one might need to specialize the serialization operators. For binary archives there's not much if anything to do. For portable binary archives, one might have to make a minor specialization. The portable binary archive includes code for handling endianness of arbitrarily long integer types. (note that portable binary archive isn't complete in anycase in that it doesn't handle floating point types - a good GSOC project if one wants one.)
I already specialize for binary archives and just write out the bytes.
So you've got a couple of ways of addressing this.
Thanks, but now I've tested what I have (and verified that the archives are truly portable at least for 32 and 64 bit systems), I'll stick with what I have I think, Cheers, John.
John Maddock wrote:
You mean rely on the usual streaming << >> operators?
For sure I could do that, but full string conversion is crazy expensive compared to the byte-by-byte scheme I have now.
Hmm - I presume you've already had to implement these operators. So you'd be done for text based archives like text_archve and xml_archive. For these cases, I wouldn't worry about the expense as these archives are already slow enough due to all the streaming conversions they do already, I doubt anyone would notice - oh and don't forget about xml parsing. So for these archive classes: a) performance isn't really an issue. b) it's helpful to be able to read floats/double, etc as human readable strings. c) it consumes no extra development time. - it's free! I'm thinking the only place where making special overload would make sense would be for portable binary archive - which as far as I know doesn't currently support floating point types in any case. Hmmm - maybe you want to generalize your method to any size float/double and include it as part of portable binary archive? That would be great! Robert Ramey
You mean rely on the usual streaming << >> operators?
For sure I could do that, but full string conversion is crazy expensive compared to the byte-by-byte scheme I have now.
Hmm - I presume you've already had to implement these operators. So you'd be done for text based archives like text_archve and xml_archive. For these cases, I wouldn't worry about the expense as these archives are already slow enough due to all the streaming conversions they do already, I doubt anyone would notice - oh and don't forget about xml parsing. So for these archive classes:
a) performance isn't really an issue.
I hear you, but it's not quite that simple: multiprecision binary-decimal conversion is a whole minefield: * The code is terminally slow, way more than for native types. * To do it correctly for floats you need an arbitrarily large precision for the intermediate calculations in the worst case (albeit doesn't happen often). * Many libraries don't do it right anyway, so for example with mpf_t you can't round trip to and from string representations... of course that's sadly true of double on some platforms (msvc) as well :-(
b) it's helpful to be able to read floats/double, etc as human readable strings.
Can't argue with that one ;-)
c) it consumes no extra development time. - it's free!
I'm thinking the only place where making special overload would make sense would be for portable binary archive - which as far as I know doesn't currently support floating point types in any case.
Hmmm - maybe you want to generalize your method to any size float/double and include it as part of portable binary archive? That would be great!
LOL :) Not just yet, sorry! John.
John Maddock wrote: \
But since I know the internals of the classes are platform specific, I've had to come up with a serialization format (for cpp_int for example) which is platform independent. Or at least I think it is. I tend to assume that anything not tested doesn't work. Maybe I'll just create some sample archives on 32 and 64 bit platforms and try reading them.
Note that a key design feature of the serialization library is that serialization of types is meant to be totally de-coupled from archive format. That is - any serializable type is guarenteed to work with any any archive class and any archive class is guarenteed to handle any serializable type. This is the reasoning behind "carpet bombing" - testing each serializable type in the library against every archive class. There is also documentation in the serialization library on how to use bjam to run this testing on your own serializable types. Upshot is - if your type full fills the requirement of being a serializable type - it should just work. If you don't trust me on this, you can easily make one test - and run it against all known archive classes. I'm supposing that this is why in ten years no one has asked this question - because it hasn't actually come up. Also this implies that your cp_int serialization shouldn't have knowledge of the machine architecture. Note that some archives - text base ones and portable binary archives address all issues related to portability between architectures. So your type cp_int shouldn't need anything special as to 16/32/64 archive compatibility. Robert Ramey
Hi, On Mon, Aug 5, 2013 at 8:33 AM, Robert Ramey <ramey@rrsd.com> wrote:
I have some trivial tests that round trip to a stringstream using boost::archive::text_i/oarchive, but of course that only verifies that platform-specific serialization works. So what do folks do - generate text archives on different platforms, deserialize and compare?
How does Boost.Serialization itself verify this?
It doesn't.
It also doesn't explicitly test that version x can read all archives created with versions < x.
How about the following? 1. When new version of boost is released, add test archive file created by the version in the test/1.xx.x directory (in svn/git repo). 2. Try to test against all the previous archive when the full test run? This way, we can test the inter-version compatibility. If you want to test inter-platform, the test compares the created archive to the archive contained in the repogitory. (I am assuming that text archive is platform-independent.) I think that this will be rather easily implementable and better than nothing tested. Best regards, -- Ryo IGARASHI, Ph.D. rigarash@gmail.com
Ryo IGARASHI wrote:
Hi,
On Mon, Aug 5, 2013 at 8:33 AM, Robert Ramey <ramey@rrsd.com> wrote:
I have some trivial tests that round trip to a stringstream using boost::archive::text_i/oarchive, but of course that only verifies that platform-specific serialization works. So what do folks do - generate text archives on different platforms, deserialize and compare?
How does Boost.Serialization itself verify this?
It doesn't.
It also doesn't explicitly test that version x can read all archives created with versions < x.
How about the following?
1. When new version of boost is released, add test archive file created by the version in the test/1.xx.x directory (in svn/git repo). 2. Try to test against all the previous archive when the full test run?
This way, we can test the inter-version compatibility.
If you want to test inter-platform, the test compares the created archive to the archive contained in the repogitory. (I am assuming that text archive is platform-independent.)
I think that this will be rather easily implementable and better than nothing tested.
Wow - great idea! Thanks for implementing this. Robert Ramey
Best regards,
participants (6)
-
Andrey Semashev -
Bjorn Reese -
Jens Weller -
John Maddock -
Robert Ramey -
Ryo IGARASHI