Re: [boost] [test]Re: [serialization] portable_binary_archive

From: troy d. straszheim (troy_at_[hidden])
Date: 2005-09-18 09:50:05

Robert Ramey wrote:
> This is something that my test suite should test but doesn't.  Regardless of 
> what happens with your portable_binary archive, It would be very helpful to 
> have the test suite enhanced to explicitly test archive portability.
> 
> Since we're just talking, So I would like to see the following changes in 
> the test suite:
> 
> a) That output of the tests go to a particular directory.  There would be a 
> directory for each column in the test matrix.
> b) There would be a set of directories for each boost release 1.32, 1.33, 
> ...
> c) Tests would be run on the four combinations which result from:
>     i) set of archives created by the previous version
>     ii) set of archives created by other compilers (portable archives only)
> 
>     This would result in a HUGE number of tests so it couldn't be run any 
> more than on an occasional basis.  But it would  be helpful to run very 
> occasionally.
> 
>     This would require more investigation into boost test in order to find 
> out what facilities are already there for helping out with ths.

Yeah, hmm.  It's a lot of work, more tedious than difficult.  Since 
we're just talking, here's some thinking out loud on the subject, for 
sanity-check purposes:

There are a few different classes of tests.  The compile-fail ones don't 
need modification of course.  A great many contain the following or 
similar, one more more times:

int test_main( int /* argc */, char* /* argv */[] )
{
     const char * testfile = boost::archive::tmpnam(NULL);
     BOOST_REQUIRE(NULL != testfile);

     BOOST_CHECKPOINT("something");

     S s_out = create_some_serializable_data_structure();
     {
         test_ostream os(testfile, TEST_STREAM_FLAGS);
         test_oarchive oa(os);
         oa << boost::serialization::make_nvp("S", s_out);
     }
     S s_in;
     {
         test_istream is(testfile, TEST_STREAM_FLAGS);
         test_iarchive ia(is);
         ia >> boost::serialization::make_nvp("S", s_in);
     }
     BOOST_CHECK(s_out == s_in);
}

and one would concievably want to factor these out so that the 
read-in-and-compare section would happen multiple times, once per boost 
version and per compiler.  For portable archive types, it would happen 
for all boost versions, compilers *and* platforms.  An additional check 
is that at the end, all portable archive types have the same checksum. 
tmpnam() would change to, say, persistent_testfile().  Many of the test 
modules create multiple archives.  You could deal with this by keeping 
track of some index and appending this to the file name.

So you would end up with files like

$SERIALIZATION_TESTDATA_DIR/
   BOOST_PLATFORM/
     BOOST_VERSION/
       BOOST_COMPILER/
         __FILE__.INDEX.ARCHIVETYPE

for instance

(testdata)/BeOS/103300/Gnu_C++_version_4.0.1/test_something.cpp.2.xml

which would be the third file written by tests inside test_something.cpp.

> d) An alternative  to the above might be just to make some specific tests 
> for backward compatibility and for cross compiler compatibility.  This is 
> probably a better idea as it would result in tests that could be run more 
> frequently.

I would think that you would in fact want to run the full battery of 
tests periodically.  Or maybe constantly, on a couple machines dedicated 
to the purpose, but I agree that the average Joe who checks something 
out of CVS and runs the tests shouldn't be expected to sit through the 
whole lot.

All that looks doable, to me.  I wonder about storage.  If you're really 
going to keep all that data around, the question is where, and how are 
you going to get at it.   One could easily put together a little web 
interface, where the test suites use a little python script to post and 
retrieve test data.

Let me see what I can come up with.

-t