Re: [Boost-users] [Serialization] a couplacrashesfromhiearchycomplexity

From: Reusser, Edward (Edward.Reusser_at_[hidden])
Date: 2006-11-15 20:32:54

Robert,

 

I reported 2 crashes.  As far as the runtime crash is concerned I have performed further analysis and come up with the following conclusions:

 

A complication that I didn't mention seems to be the source of the runtime crash.

 

I have 2 separate DLL libraries, call them Library A, and library B.  Inside of A, I have what is essentially a list of key/value pairs.  Initially I serialized this data in and out to a file using xerces (someone else did this part).

 

Later, Library B comes along and asks for a value using a key (the value is just a string).  This library then uses the boost serialization library (xml_iarchive) to convert it to classes.  When I want to persist the state of Lbrary B, I then convert the class structures back to a string using xml_oarchive and store it in the master key/value map of Library A.

 

The difficulty arises because the string I am saving and retrieving is somewhat large, sometimes over a megabyte in length.  When the xerces library gets this,. It converts all of the '<' and '> characters into escape sequences like &lt and &gt.  So far so good.

 

Except there is a bug in the translation so that any string larger than about ½ megabyte do not get properly converted from XML text into the corresponding string.  Thus, the string I am feeding into the boost serialization library is not proper XML because it has been  only partially escaped.

 

So we redid the serialization in the first library to serialize the key/value pairs using boost serialization instead of xerces, completely discarding this library.

 

But the same exact problem occurs.  Even using boost serialization, a text value within 2 tags cannot exceed about ½ megabyte in length if it contains escaped character sequences, because when it is read back it will not be properly converted to XML.  I know the entire XML string is being translated out to the file because I can open the file and see that it is correct.  Furthermore, the serialization library doesn't report any errors in reading the key/value pairs.  It isn't until I parse the value using the xml_iarchive that it fails.

 

We solved this problem using text_iarchive and text_oarchive for the key/value map of Library A, preventing the XML in Library B from being touched.

 

As for the 2nd crash, using text_iarchive and text_oarchive within Library B, this still gives the internal structure overflow error.  When I get time to go back and look at it again, I will probably rewrite it so that all of the shared_ptr's are stored directly and make other simplifications.  If you have a mechanism for me to send this DLL directly to you, then I will attempt to get permission to do so.

 

Ed

 

________________________________

From: Robert Ramey [mailto:ramey_at_[hidden]] 
Sent: Monday, November 13, 2006 2:22 PM
To: boost-users_at_[hidden]
Subject: Re: [Boost-users] [Serialization] a couplacrashesfromhiearchycomplexity

 

it would seem to me that a stack trace would be interesting to see.  I'm wondering if its just a stack that is too short (I would have expected to see this sooner if it was) or some error somewhere which creates an unterminated recurssion.  With VC its easy to see the stack trace.  With the Gnu Debuger, one has to back trace from the right point.  

 

Robert Ramey

 

        "Reusser, Edward" <Edward.Reusser_at_[hidden]> wrote in message news:5E916BAE1732F344BAFD713DF2373479016EA233_at_SV-MSG-01.amer.actel.com...

        Robert,

         

        I would need permission to send you the exact library code.  I am reluctant to do this primarily because recreating my exact build environment would be a hassle (we are using gnumake on the PC).  I also would need to do this directly to you, as I could not post this code into a public forum.

         

        Having said that, any recursive descent parser has complexity limits based on stack size.  It would be pretty easy to create test cases that probe these complexity limits for the various archive types.  I would like to see how much (approximately)  stack space is allocated in the runtime for each level of xml hierarchy, and also for the text archives.

         

        The main complexity seems to be because I am using shared_ptr's extensively at all levels of the hierarchy.  Thus, I have 2 levels nested for each class, the <px> data </px> and the actual data, which is in turn being represented by a collection of shared_ptr's, etc.  I am also using std::map of shared pointers, and std::set of shared_ptr's.  I also use enable_shared_from_this<T> for my base classes, then statically cast the resultant shared pointers internally for the derived classes.  This is to get around a Linux problem which I can describe more fully if you like (would take a few pages of explanation).

         

        So for example:

         

        class Base : public boost:enable_shared_from_this<Base>

        {

              virtual boost::shared_ptr<Base> GetSharedFromThis()

            {

                return shared_from_this();

            }

         

        };

         

        class Derived1 : public Base

        {

        public:

              virtual boost::shared_ptr<Derived1> GetSharedFromThis()

            {

                return static_pointer_cast<Derived1>( shared_from_this() );

            }

        };

         

        class Derived2 : public Base

        {

        public:

              virtual boost::shared_ptr<Derived2> GetSharedFromThis()

            {

                return static_pointer_cast<Derived2>( shared_from_this() );

            }

        };

         

        I am assuming that I do not need to call boost::serialization::base_object<boost::enable_shared_from_this<Base> >( *this );  

         

        When I serialize the base class.  Could that be the problem?

         

        I think what I am going to do is go back into the program and simply eliminate all of the shared_ptr serializations.  I can serialize to raw pointers, and then just create the shared_ptr's from those pointers explicitly in my case.  I simply do not ever have any shared_ptr's with multiple reference counts to worry about.

         

        I will let you know what happens.

         

         

        
________________________________

        From: Robert Ramey [mailto:ramey_at_[hidden]] 
        Sent: Monday, November 13, 2006 8:30 AM
        To: boost-users_at_[hidden]
        Subject: Re: [Boost-users] [Serialization] a coupla crashes fromhiearchycomplexity

         

        I would be curious to see your particular case.  The fact that different archives give problems that seem related to nesting depth suggest along with the fact that this hasn't been reported suggests that there is some sort of use case or error which hasn't been considered.

         

        Robert Ramey

                "Reusser, Edward" <Edward.Reusser_at_[hidden]> wrote in message news:5E916BAE1732F344BAFD713DF2373479016EA059_at_SV-MSG-01.amer.actel.com...

                I am using the serialization library in boost 1.33.1.  I am compiling on windows using the VC 7.1 compiler.

                 

                The problem I am having is that when I use xml_iarchive and xml_oarchive, everything works just fine for simple cases, but if the xml gets very deep ( same exact class hierarchy, just more data organized a little more complexly) I am getting runtime stack overflow crashes in the spirit parser.  I figured I could solve that by simply substituting text_iarchive and text_oarchive.   Please note that I am simply substituting in these archive types instead of using the xml archive headers, which work.  There are no other code changes anywhere.

                 

                Now I am getting the following compiler error:

                 

                \boost\mpl\eval_if.hpp(40) : fatal error C1204: compiler limit : internal structure overflow

                 

                Any suggestions?  I can try increasing the stack size of my current application to eliminate the xml runtime stack overflow problem, but I am uneasy doing this until I understand how to calculate the stack sizes required, and why it is happening.  Furthermore, I would prefer using the text archives, since the data is more compact than the xml ones.

                 

                Ed Reusser 

                Principal Software Engineer

                Actel Corporation

                [phone]

                
________________________________

                The information contained in or attached to this email may be subject to the Export Administration Regulations (EAR), administered by the U.S. Department of Commerce, or the International Traffic in Arms Regulations (ITAR), administered by the U.S. Department of State, and may require an export license from the Commerce or State Department prior to its export. An export can include a release or disclosure to a foreign national inside or outside the United States. Include this notice with any reproduced portion of this information. 

                
________________________________

                _______________________________________________
                Boost-users mailing list
                Boost-users_at_[hidden]
                http://lists.boost.org/mailman/listinfo.cgi/boost-users

        
________________________________

        The information contained in or attached to this email may be subject to the Export Administration Regulations (EAR), administered by the U.S. Department of Commerce, or the International Traffic in Arms Regulations (ITAR), administered by the U.S. Department of State, and may require an export license from the Commerce or State Department prior to its export. An export can include a release or disclosure to a foreign national inside or outside the United States. Include this notice with any reproduced portion of this information. 

        
________________________________

        _______________________________________________
        Boost-users mailing list
        Boost-users_at_[hidden]
        http://lists.boost.org/mailman/listinfo.cgi/boost-users

The information contained in or attached to this email may be subject to the Export Administration Regulations (EAR), administered by the U.S. Department of Commerce, or the International Traffic in Arms Regulations (ITAR), administered by the U.S. Department of State, and may require an export license from the Commerce or State Department prior to its export.  An export can include a release or disclosure to a foreign national inside or outside the United States.  Include this notice with any reproduced portion of this information.