Re: [Boost-users] [serialization] serializing dynamic -> serializing pointer to element before the array of elemets crashes

From: Ivan Rachev (ivanr_at_[hidden])
Date: 2005-08-08 09:54:57

>> The solution is to be sure that if objects are
>> serialized directly and through pointers, the 
>> the pointer serialization is done second.

yes, whenever a pointer serialization is done it should be defered and serialized after the direct serialization. My believe is that the lib keeps an object-address-to-object-id mapping (BoostSaveMap) when saving and an object-id-to-object-address mapping (BoostLoadMap) when loading objects from an archive so that one and the same object does not get serialized twice. Based on this assumption there is a solition for the following problematic scenario:

struct TChar { char ch; };
template <typename T> struct DynamicArray
{ int Size;
  T* Element;
};
typedef DynamicArray<TChar> TCharArray;

class A { TChar * ptr; } a;
class B { TCharArray array; } b;

b.array.Element = new TChar[3];
b.array.Size = 3;
a.ptr = b.array.Element[1];

ar & a;    // at loading, new object created here
ar & b;    // attempt to reload an already loaded object
           // to new address - throw exception - pointer conflict
patch(ar); // fix to problem that needs implementation

For the solution itself, I see the following that needs to happen at saving&loading:
When saving, I see 3 steps:
1) whenever TChar* is being saved, adding the TChar* pointer to InternalPtrVector but not sending it to the archive (sending is in step 3). This vector is a mapping of internal pointer (a.ptr) to holder id (ID of a) and contains offset of the pointer (a.ptr) in the object that holds this pointer (a). So for each TChar* in the above example, this vector knows the id of the object that holds the TChar* and its offset in this object.
2) whenever DynamicArray is being saved, adding it to ArrayRangeVector and sending it to the archive. Adding is done by writing the following into the ArrayRangeVector structure: 1.array address (b.array), 2.array size (b.array.Size), 3.array id. This structure knows the range of addresses used by a particular array and the id that boost::serialization gave to the array at saving.
3) after formal serialization (in the patch func) iterate over InternalPtrVector and for each internal pointer there, check if it's within the range of any of the arrays that are stored in ArrayRangeVector. If such an array is found, write the following InternalPtrFileStruct to the archive: 
Holder ID,  (ID of a in the example above)
Holder Offset, (offset of a.ptr in a)
Array ID, (ID of b.array)
Array Offset ( a.ptr - b.array.Element )

The structures used when saving are:
BoostLoadMap
---------------------------------
|  object ID -> object address  |
---------------------------------
InternalPtrVector
-------------------------------------------------
|  InternalPtr  |  Holder ID  |  Holder Offset  |
-------------------------------------------------
ArrayRangeVector
-----------------------------------------------
|  Array Address  |  Array Size  |  Array ID  |
-----------------------------------------------

When loading, I see 1 step:

When formal loading is going on, nothing happens to TChar* members. They 
are written into afterwords. Actual objects (TChar in the case above) 
are loaded when loading the dynamic array. After formal serialization 
(ar & a; ar & b; ) is done, TChar* members are initialized in the patch 
func. What it should do is the following:
read each InternalPtrFileStruct. Look up holder object address (&a) in 
BoostLoadMap by Holder ID (ID of a). Add Holder offset to this address 
and the result is the address of TChar* in the case above. Now we need 
to calculate the value for the TChar* member. This is done by getting 
the address of the array (&b.array.Element) from BoostLoadMap using 
Array ID (id of b.array). Add Array Offset to this address and the 
result is the value of the TChar* memeber. So write this value into the 
address of TChar*.

The structure used at loading is :
BoostLoadMap
---------------------------------
|  object address -> object ID  |
---------------------------------

In order for this to work there needs to be some way of accessing 
BoostLoadMap and BoostSaveMap from users of boost::serialization (this 
is where Robert could help). Of course, I believe the better way would 
be if all this functionality is incorporated into the lib. That way or 
another there needs to be serialization code similar to the following:

// user code below this line 
---------------------------------------------------------------------------
template <class Archive> void serialize(Archive& ar, A& obj)
{ saveInternalPtr(obj.ptr, obj, ar);
  ar & obj.other_members;
}

template <class Archive> void serialize(Archive& ar, B& obj)
{ ar & obj.array;
}

// lib code below this line 
---------------------------------------------------------------------------
template <class Archive> saveInternalPtr(void* InternalPtr , void* 
objectAddress, Archive& ar)
{ if (ar.is_saving) {
  // get object id by objectAddress from BoostSaveMap
  InternalPtrVector.insert(InternalPtr, objectID, InternalPtr - 
objectAddress);
  }
}

template <class Archive, typename ElemT> void serialize(Archive& ar, 
DynamicArray<ElemT>& obj)
{ ar & obj.Size;
  if (ar.is_saving) {
  for (int i = 0; i < obj.Size; ++i)  ar & obj.Element[i];
  // get array id by array Address from BoostSaveMap
  ArrayRangeVector.insert(&obj, obj.Size, arrayID);
  } else
  {
  assert( obj.Size >=0 );
  if (obj.Size > 0) {
    obj.Element = new ElemT[obj.Size];
    for (int i = 0; i < obj.Size; ++i)  ar & obj.Element[i];
    }
  else obj.Element = null;
  }
}

template <class Archive> void patch(Archive& ar)
{ ar & InternalPtrVector.size;
  if (ar.is_saving)
    // for each internal pointer in InternalPtrVector
    //   for each array in ArrayRangeVector
    //     if internal pointer within [ArrayRangeVector[j].arrayAddress, 
ArrayRangeVector[j].arrayAddress + ArrayRangeVector[j].arraySize]
    //       ar & InternalPtrVector[i].holderID;
    //       ar & InternalPtrVector[i].holderOffset;
    //       ar & ArrayRangeVector[j].arrayID;
    //       ar & InternalPtrVector[i].InternalPtr - 
ArrayRangeVector[j].arrayAddress;
  else
    // for (int i = 0; i < InternalPtrVector.size; ++i)
    //   ar & holderID;
    //   get holder address by holderID from BoostLoadMap
    //   ar & holderOffset;
    //   void * memberAddress = holderAddress + holderOffset; /* this is 
address of TChar* in the example above */
    //   ar & arrayID;
    //   get array address by arrayID from BoostLoadMap
    //   ar & arrayOffset;
    //   int memberValue = arrayAddress + arrayOffset; /* this is value 
of TChar* in the example above */
    //   *memberAddress = memberValue;
}

Ivan

Robert Ramey wrote:

>The problem is is that de-serializing a pointer creates a new object in a
>new address.  Now one comes along an deseriailzed the same object to a fixed
>address.  There is no way to fix this without going back to the begining.
>This situation is detected though tracking and a "pointer conflict"
>exception is thrown.  The solution is to be sure that if objects are
>serialized directly and through pointers, the the pointer serialization is
>done second.
>
>Robert Ramey
>
>Ivan Rachev wrote:
>  
>
>>Robert Ramey wrote:
>>    
>>
>>>This is a different problem.
>>>
>>>
>>>      
>>>
>>>>    DynamicArray<A> ArrayOfAs;
>>>>    ArrayOfAs.Element = new A[5];
>>>>    ArrayOfAs.Size = 5;
>>>>    A* secondElement = &ArrayOfAs.Element[1];
>>>>
>>>>    std::stringstream stream;
>>>>    {
>>>>     boost::archive::text_oarchive oa(stream);
>>>>     oa & secondElement;   // new object created here
>>>>     oa & ArrayOfAs;        // attempt to reload an already loaded
>>>>object
>>>>        
>>>>
>>>to new address - throw exception - pointer conflict
>>>
>>>      
>>>
>>>>    }
>>>>        
>>>>
>>I think over here we have a solution to try.
>>
>>I believe at saving boost::serialization creates a map
>>(object address -> object id)
>>and at loading the lib creates the reverse map
>>(object id -> object address)
>>
>>
>>How inside the archive could I write a function like this:
>>ObjID GetObjectID(void* objectAddress);
>>
>>Usage will be like this:
>>class A {
>>   template<class Archive>
>>   void save(Archive & ar, const unsigned int version) const
>>   {
>>   ...
>>   ObjID objId = ar.GetObjectID(this);
>>   ...
>>   }
>>
>>};
>>
>>
>>Also, after loading is done and the load map is created, how inside
>>the archive could I implement a function like this:
>>void* GetObjectAddress(ObjID objId);
>>
>>Usage will be like this:
>>...
>>ObjID objId;
>>std::stringstream stream;
>>boost::archive::text_iarchive ia(stream);
>>ia & someObj;
>>ia & objId;
>>void* p = ar.GetObjectAddress(objId);
>>...
>>
>>
>>Ivan
>>    
>>
>
>
>
>_______________________________________________
>Boost-users mailing list
>Boost-users_at_[hidden]
>http://lists.boost.org/mailman/listinfo.cgi/boost-users
>
>  
>