Re: [Boost-users] Known issues loading lots of objects withxml_iarchive

From: Christopher Gillett (cgillett_at_[hidden])
Date: 2006-03-28 12:26:36

See I foolishly assumed that all your code was bug free :-) lol...sorry
for the nebulous report - here is something a bit more concrete.

The scenario as stated is loading a large number of objects that have
been previously serialized into a number of files.  After around 150
files have been processed I hit an archive exception and stream error.
This occurs only when using the XML archive...if I use the binary
archive I can process as many files as I'd like (and in fact I've tested
many more than I actually need). 

The approach is "textbook" in that I've seen this same pattern many
times in sample code, tutorials, etc.  The only difference in the code
between binary and xml serialization is my load function:

XML:
#include <boost/archive/xml_oarchive.hpp>
#include <boost/archive/xml_iarchive.hpp>
void load( MyObjType &obj, const char *fileName )
{ 
  std::ifstream ifs( fileName ); 
  assert( ifs.good() );  
  boost::archive::xml_iarchive ia( ifs ); 
  ia >> BOOST_SERIALIZATION_NVP( obj );
} 

Binary:
#include <boost/archive/binary_oarchive.hpp>
#include <boost/archive/binary_iarchive.hpp>
void load( MyObjType &obj, const char *fileName )
{ 
  std::ifstream ifs( fileName ); 
  assert( ifs.good() );  
  boost::archive::binary_iarchive ia( ifs ); 
  ia >> obj;
}

Each class I'm serializing has the usual "boilerplate" in the class
definition, for example:

#include <boost/serialization/nvp.hpp>
#include <boost/serialization/utility.hpp>
#include <boost/serialization/list.hpp>
#include <boost/serialization/version.hpp>
class MyClass : public MyClassBaseObject
{
    friend class boost::serialization::access;
    friend std::ostream & operator<<(std::ostream &os, const MyClass
&MyClassObj);

    template<class Archive>
    void serialize(Archive &ar, const unsigned int version)
    {
        ar & BOOST_SERIALIZATION_BASE_OBJECT_NVP( MyClassBaseObj );
        ar & BOOST_SERIALIZATION_NVP( thing1 );
        ar & BOOST_SERIALIZATION_NVP( thing2 );
        ar & BOOST_SERIALIZATION_NVP( etc );
    }
    ...
 };

I initially thought this was a file management problem, but since I can
load as many files as I'd like with the binary archive I suspect a bug
in the XML archive code.  While I can live with binary serialization if
necessary, my goal is XML serialization.

Any thoughts on this?  I can put together a totally contrived example
program that will demonstrate the failure if necessary.  I'm hoping to
hear that it's something stupid on my part.

Thanks,
Chris Gillett
 
> Date: Mon, 13 Mar 2006 18:40:32 -0800
> From: "Robert Ramey" <ramey_at_[hidden]>
> Subject: Re: [Boost-users] Known issues loading lots of objects
>        withxml_iarchive?
> To: boost-users_at_[hidden]
> Message-ID: <dv5aeu$l6e$1_at_[hidden]>
> 
> There are no "known" issues.  In fact the  system has no known bugs
> either.  LOL - of course the function of this list is to address the
> problem.  I haven't heard of anyone having this problem.  You might
> investigate a little by seeing if occurs just in xml or in other 
> archives
> as well.
> 
> Robert Ramey
>
> Christopher Gillett wrote:
>> I am serializing a LOT of objects, one per file.  I can create 
>> XML for
>> many thousands of objects with no issue.  However, when trying to
>> load a few hundred objects (using a separate program, btw) I am
>> hitting a boost::archive::archive_exception with a stream error while
>> loading around the 161st file, which I believe based on stack traces
>> is coming from within
>> boost::archive::basic_xml_grammar<char>::parse_end_tag ().
>>
>> The approach I'm taking is fairly textbook, so I'm curious if there
>> are known issues with loading a lot of files, etc.
>>
>> Any advice appreciate.
>>
>> Thanks,
>> Chris Gillett