Re: [Boost-users] Usingboost::serializationinreal-timewithoutallocating memory

Subject: Re: [Boost-users] Usingboost::serializationinreal-timewithoutallocating memory
From: Peter Soetens (peter.soetens_at_[hidden])
Date: 2009-09-22 06:02:42

On Tue, Sep 22, 2009 at 00:11, Peter Soetens <peter.soetens_at_[hidden]> wrote:
> On Mon, Sep 21, 2009 at 23:44, Peter Soetens <peter.soetens_at_[hidden]> wrote:
>> On Mon, Sep 21, 2009 at 21:25, Robert Ramey <ramey_at_[hidden]> wrote:
>>> Stefan Strasser wrote:
>>>> Am Monday 21 September 2009 20:27:59 schrieb Robert Ramey:
>>>>> Stefan Strasser wrote:
>>>>>> I think to support this an other use cases like it common_archive
>>>>>> would have to be moved outside of the linked library into a header
>>>>>> and be made more generic.
>>>>>
>>>>> common_archive is a template - it's not in the linked library.
>>>>
>>>> ok, I must have mixed up some types. the point was that the ultimate
>>>> base class, that handles the type registration and object tracking
>>>> etc., is
>>>> linked .cpp and not very configurable.
>>>> the optimum case imho would be that you could pass a traits class to
>>>> that archive base class, that gets called whenever there is an object
>>>> or type to be registered and is queried for existing objects and
>>>> types later on.
>>>> that way you could disable registration, save type registration
>>>> outside of the archive itself, let type registration span multiple
>>>> archives etc, make sure type registration doesn't allocate etc.
>>>
>>> as I alluded to before - what I would like to know is why I derived
>>> trivial_archive from common_archive.  I don't remember what
>>> made me do this.  It seems that if this isn't done then save/load_object
>>> is necessary.  So the question is - why is this?  It is an oversight
>>> somewhere? or does the archive concept have to be changed.
>>>
>>> Personally, I generally don't like the "traits class" as an argument.
>>> It sort of implies that that the template has a lot of "if -" code
>>> which navigates the traits. A "policy class" is better but can
>>> still make things complex.  Personally I prefer composition
>>> of funtionality through inheritance. This can be seen in
>>> the class diagram for the serialization library. And your example
>>> makes me more convinced than ever that this is basically
>>> right.  Now the only loose end it investigate why save/load
>>> object is required.
>>
>> I found a lead to that. When the archive calls archive::save, a chain
>> of 'invoke' calls is done to select the right serialization method for
>> type T. When I pass std::vector<double> as T, it encounters a branch
>> in oserializer.hpp:245, in  struct 'save_conditional::invoke' .
>>
>> <code>
>>    // adds class information to the archive. This includes
>>    // serialization level and class version
>>    struct save_conditional {
>>        static void invoke(Archive &ar, const T &t){
>>            //if(0 == (ar.get_flags() & no_tracking))
>>                save_standard::invoke(ar, t);
>>            //else
>>            //   save_only::invoke(ar, t);
>>        }
>>    };
>> </code>
>>
>>  The alternative branch is commented out, but let that be the path
>> that needs to be taken by my code. Current code calls save_standard,
>> which calls directly ' ar.save_object'. save_object in turn depends on
>> basic_oarchive to do its job. If save_only was taken, the
>> 'non-basic_oarchive' serialization code path would have been taken.
>>
>> For some reason (I've spent an hour staring at it), my unit test did
>> not discover this flaw in my code and does as if serialisation of the
>> std::vector was fine. I'm overlooking something stupid clearly...
>>
>> Anyway, as it seems now, the code path taken by archive::save/load
>> mandates derivation of basic_?archive. Maybe I should have called
>> another 'selection' function ? like archive::save_only directly ?
>
> I tried using this instead of archive::save() :
> <code>
>  boost::archive::detail::load_non_pointer_type<binary_data_iarchive,T>::load_only::invoke(*this,t);
> </code>
>
> But the compiler (gcc 4.3.3-5ubuntu4) refuses to compile this when the
> nvt types are in use. Basically, this is not fine:
> <code>
> int count;
> ar >> BOOST_SERIALIZATION_NVP(count);
> </code>
> While this is:
> <code>
> int count;
> nvp<int> n = BOOST_SERIALIZATION_NVP(count);
> ar >> n
> </code>
>
> You can try this out in plain user code. Beats me. The errors I get
> for the first case are in the line of:
>
> usr/include/boost/serialization/access.hpp: In static member function
> ‘static void boost::serialization::access::serialize(Archive&, T&,
> unsigned int) [with Archive = RTT::marsh::binary_data_iarchive, T =
> const boost::serialization::nvp<int>]’:
> /usr/include/boost/serialization/serialization.hpp:74:   instantiated
> from ‘void boost::serialization::serialize(Archive&, T&, unsigned int)
> [with Archive = RTT::marsh::binary_data_iarchive, T = const
> boost::serialization::nvp<int>]’
> /usr/include/boost/serialization/serialization.hpp:133:   instantiated
> from ‘void boost::serialization::serialize_adl(Archive&, T&, unsigned
> int) [with Archive = RTT::marsh::binary_data_iarchive, T = const
> boost::serialization::nvp<int>]’
> /usr/include/boost/archive/detail/iserializer.hpp:316:   instantiated
> from ‘static void
> boost::archive::detail::load_non_pointer_type<Archive,
> T>::load_only::invoke(Archive&, T&) [with Archive =
> RTT::marsh::binary_data_iarchive, T = const
> boost::serialization::nvp<int>]’
> /home/kaltan/src/git/orocos-rtt/src/marsh/binary_data_archive.hpp:198:
>  instantiated from ‘RTT::marsh::binary_data_iarchive&
> RTT::marsh::binary_data_iarchive::load_a_type(T&, mpl_::false_) [with
> T = const boost::serialization::nvp<int>]’
> /home/kaltan/src/git/orocos-rtt/src/marsh/binary_data_archive.hpp:131:
>  instantiated from ‘RTT::marsh::binary_data_iarchive&
> RTT::marsh::binary_data_iarchive::operator>>(T&) [with T = const
> boost::serialization::nvp<int>]’
> /home/kaltan/src/git/orocos-rtt/tests/mqueue_archive_test.cpp:71:
> instantiated from here
> /usr/include/boost/serialization/access.hpp:109: error: passing ‘const
> boost::serialization::nvp<int>’ as ‘this’ argument of ‘void
> boost::serialization::nvp<T>::serialize(Archive&, unsigned int) [with
> Archive = RTT::marsh::binary_data_iarchive, T = int]’ discards
> qualifiers
>
> Passing to operator>> the nvt struct as a return value of make_nvt()
> makes gcc add a const qualifier, which gets propagated all the way...I
> have no idea why the other archive implementations don't suffer this.

I solved it, and came to the conclusion that things got worse. It
seems that, even when only calling
<code>
 boost::archive::detail::load_non_pointer_type<binary_data_iarchive,T>::load_only::invoke(*this,t);
</code>

I need to implement more function overloads (but not the load_object
one), which are present in the binary archive case, but not in my
class. I needed to add 3 load_override overloads in the iarchive:
<code>
            template<class T>
            void load_override(T & t, BOOST_PFTO int){
                load_a_type(t,
boost::mpl::bool_<boost::serialization::implementation_level<T>::value
== boost::serialization::primitive_type>() );
                //archive::load(* this->This(), t);
            }

            void load_override(const
boost::serialization::nvp<boost::serialization::collection_size_type>
& t, int){
                 size_t x=0;
                 * this >> x;
                 t.value() = boost::serialization::collection_size_type(x);
             }

            template<class T>
            void load_override(const boost::serialization::nvp<T> & t, int){
                 T x=0;
                 * this >> x;
                 t.value() = x;
             }

            template<class T>
            Archive & operator>>(T & t){
                this->load_override(t, 0);
                return * this;
            }
</code>

If you try to delete any of these three, serializing an std::vector
will no longer work. In some way, the basic_iarchive accomodates for
this case. So I traded one overload for load_object with three
overloads for load_override. Because my load_object function was
empty, it's very likely that the load_overrides were not required
because the marshalling was not done at all.

Someone with strong knowledge of all this needs to take my example +
the existing codebase to something that does what is promised by the
concept docs.

The attached file is how the class lives in my current project, in the
RTT::marsh namespace. I fixed my unit tests, and the class in
attachment now works 100% for:
* marshalling primitive types (non pointer)
* marshalling structs/classes similar to std::vector, but which
contain non-pointer data.

It's not looking how I had imagined/hoped for, but it does the trick
for me now. I'm happy to stick on this thread, but I'm not going to be
able to solve this or make this an example that meets your
expectations. It also looks that this thread no longer belongs on
boost-users but on the boost list.

To end with a positive note: I'm very happy how the separation between
serialization and archiving has been (or has intended to be) done. It
allows me to tap in a real-time archiver and re-using all
serialization code. That's so cool. I checked the now working
implementation again (stepped through the debugger as well) and it
truely works for my test set.

Peter