Re: [Boost-users] [Serialization] possible solns to vector of vectorsptr corruption

From: Robert Ramey (ramey_at_[hidden])
Date: 2006-08-10 18:47:57

Dan Notestein wrote:
> Hi Robert,
>
> Any thoughts about my ideas for fixing the vector of vectors ptr
> corruption issue? I've reiterated my early msg below. If you don't
> have
> time to investigate, I would be willing to work on fixing the problem,
> if you don't mind me bugging you with questions occassionally.

That would be just great!!!

It would be helpful to make a test case using one of the tests in the
package as a model.  Or better and easier would be to  just enhance
test_vector to include
this problem case.

This is a tricky situtaton and you've already made the investment
to understand it.  Offhand, I would think that the swap solution
below would be the best.

Of course nothing is simple.  This immediatly opens up the
probability that other collections of pointers to other collections
will have problems too.  So that should be considered also.

Its making my head hurt.

Robert Ramey

> regards,
> Dan Notestein
>
> Original Message:
> -----------------------
> Robert Ramey wrote:
>> Dan Notestein wrote:
>>> The other bug appears to be in the way moveable_objects_recent and
>>> moveable_objects_end are being set prior to calling
>>> reset_object_address. My assumption is the intent here is to modify
>>> the addresses of "trackable" sub-objects contained within the vector
>>> element being moved, so that ptrs will be hooked back up correctly,
>>> but the moveable ptrs are being set up in such a way that the vector
>>> elements (TObject) of the sub-vector (ObjectVector) are getting
>>> their addresses modified when the stack version of TObjectContainer
>>> is copied to the vector.
>>
>>> This seems wrong, because the TObjects are
>>> allocated on the heap, so their addresses should not be updated in
>>> the object_vector_id table when the ObjectVector is copied. The end
>>> result is that I'm getting bad pointers to the TObjects after
>>> deserialization.
>>
>> TObjects are de-serialized to the heap then added to the vector.
>> Since the tracking saves the address TObject is serialized to, the
>> address would be on the stack.  reset_object_address sets the tracked
>> address to the heap address after the item is appended to the vector.
>>
>> I thought I considered the case of a vector of vectors - but maybe
>> not.  I would expect that the objects get theire addresses fixed up
>> twice- once when they are moved from the stack to the heap and
>> once when the vector container itself is moved from the stack to the
>> heap. Its possible that there is something missng .
>
> Yes, the TObjects are fixed up twice, but I think the second fixup
> is an error because vector<> doesn't really contain the data elements,
> it contains a ptr to the data elements. When the vector container
> is "copied" from the stack to the heap, the newly allocated internal
> vector element array will be randomly allocated from somewhere else in
> the heap, so using an offset based off the address of the new copy
> of the vector container won't work.
>
> I can think of two possible theoretical solutions to the problem:
>
> 1) Use the container::swap function to exchange internal data ptrs of
> the stack and heap copy of the vector. This gives the heap vector the
> ownership of the stack vector's elements. This
> seems the most efficient technique since it avoid reallocating and
> copy constructing
> the elements. There's also no need to fixup the ptrs to the elements
> using this technique.
>
> 2) Continue to use the copy constructor for the vector when
> transferring the vector from stack to heap, and use the internal
> array ptr inside the new copy of the vector to fixup the elements.
> This
> definitely seems like the lesser of the two options.
>
> I would have tried to make a fix for this, but the algorithm for
> handling the moveable ptrs that define the range of addresses to
> fixup was a little complicated for me to follow. In particular, I
> wasn't sure
> how to make the fixups take place so that truly "contained" data gets
> fixed up, while avoiding
> fixing up the data that is merely "pointed" to by data members (and
> hence shouldn't be
> fixed up when the container is copied). One idea I had was to keep
> around the size
> of the data objects being moved, so that the reset_address function
> could look for ptrs in the moveable_ptr range that have addresses
> between old_address and old_address+object_size. But I also
> figured there was a good chance that there is already information
> available somewhere
> on which items in the object_id table are  "contained" vs "pointed
> to".
>
> best regards,
> Dan Notestein