[boost] Serialization library - plug in etc

From: Robert Ramey (ramey_at_[hidden])
Date: 2002-10-04 18:05:40

>From:	Vladimir Prus [SMTP:ghost_at_[hidden]]
>Sent:	Wednesday, October 02, 2002 8:24 AM>
>To:	Robert Ramey
>Subject:	Serialization library
>Hi Robert,
>some time ago you've said that you'll think about global vs. local 
>classes registration. Did you came to any conclusions? I'm asking 
>because review of your library is scheduled to start on Nov 2, and it's 
>not so much time remaining.
> - Volodya

    Please don't think I've forgotten it - I havn't.  I have spent significant time considering it.
    Given the rigorous nature of the interchange, I can't really send something that hasn't
    been carefully thought out and this takes time.
    Its a good thing you emailed me as for some reason the date in my head was 25 Nov
    rather than 2 Nov.  I checked and it is 2 November.
    For your information I am considering the following:

    1)Peter Dimov submitted a test case that shows that classes which multiply inherit
    don't work.  This is a deficiency that I'm working to remedy.  Unfortunately, it turns
    out to be non-trivial and require a type list implementation to do in a definitive manner.
    This means I want to use mpl which isn't in the library until 1_29 which isn't yet released.
    I'm hoping it will be released before 2 Nov - I will have to check.
    
    2)Peter also compiled the package with Comeau which spit out some errors.  It seems that
    Comeau is more picky about certain C++ rules.  So I've been making tweaks to make
    it pass this test as well.
    
    3)It was also requested that I provide support of standard C arrays.  I said I would consider it.
    It turns out that is not hard and fits well with the library.  It does require coding, make a test
    and updating the documentation so that consumes some time.
    
    4)I also spent some time trying out ideas where a concept such as describe would logically fit.
    I reached no new conclusions except to reenforce my view that reflection (of which I 
    see describe as a facet) is a larger topic than first meets the eye and is better kept separate
    from serialization.
    
    I also spent time considering more carefully this issue of global vs local registration.  I see
    two issues here.
    
    a)I concede that it is inconvenient for an  to have to maintain a list of derived classes that might be
    included in an archive. 
    a) I do appreciate the fact that it is convenient for an upper level module to be 
    
    a) I do appreciate the view that an upperlevel module that works though an interface
    of base class pointers should not have any dependency on derived implementations.  The
    current archive specific registration system violates that view.
    
    I appreciate the appeal the idea that each class just register itself with a global
    singleton. .  On the other hand it does create the need for a singleton of global scope -
    something that I generally don't prefer.  Given the above however I am leaning to provide
    such a facility.
    
    b) I do not believe that this enough to necesitate program independent method of registration - 
    that is as system dependent upon type_info.  I believe the desire to implement such a thing
    is related to the view that it could be used to implement plug-ins.
    
    b) plug-ins.  It would seem that global registration of some sort is necessary to support
    plug-ins.  Indeed it is.  But I don't believe that global registration of the most derived base
    classes is going to be sufficient.  I base my view on my experience with Microsoft COM - 
    Common Object Model .  Presumably CORBA is similar.  In these systems all
    plug-ins are derived from a common base.  This common base has virtual functions
    equivalent to our save/load commands.  So when an object is serialized, the object
    schema itself is not visible - only the vtable containing the serialization
    
    I believe that any "plug-in" implementation has to look something like the following.
    Of course this is vastly over-simplified to illustrate my point.
    
    class plugin_interface
    {
            // a function which creates an object of the type requested
            // and returns a pointer to its base class
            static plugin_all *plugin_factor(GUID g);
            virtual GUID get_guid();
            // command a plug_in to save it self
            virtual void save(oarchive &ar) const;
            // or load itself
            virtual void load(iarchive &ar);
            // other interface functions for this family of plug-ins
            // ...
    }
    
    // note: the class definition for a specific plug-in does not
    // appear in the program using the plugin - all functionality is
    // accessed though its public interface as defined by the base class.
    
    class plugin_implementation : public plugin_interface
    {
            // a globally unique identifier - maybe typename or ?
            const GUID g;
            virtual GUID get_guid(){
                    return g;
            }
            virtual void save(oarchive &ar) const;
            virtual void load(iarchive &ar);
            // other interface functions for this family of plug-ins
            // ...
    };
    
    void  serialization<plugin_interface>::save_ptr(boost::basic_oarchive & ar, const T * t)
    {
        // default just derefences the pointer
        // and saves the object using the normal method
        ar << g;
        ar << *t;
    }
    void serialization<plugin_interface>::load_ptr(
        boost::basic_iarchive & ar,
        T * &t,
       boost::version_type file_version
    ){
        // If de-serialization fails due to an exception
        // this will guarentee that delete isn't called on a non-null pointer
        t = NULL;	
        try{
            // get globally unique identifier - global type identifier
            GUID guid;
            ar >> guid;
            t = plugin_base_factory(guid);
            ar >> *t;
        }
        catch(...){
            // uh-oh object in the archive could not be created - most likely
            // plugin that creates and implements it is not installed on the
            // machine reading the archive or not supported by the program
            // reading the archive.
            delete t;
            throw;
        }
    }
    
    The above illustrated (I hope) that the issue of plug-ins is not related to the desire to eliminate
    the need to explicitly register otherwise unreferened derived classes through pointers.
    
    So I anticipate that I will try find a way that addresses a) above but doesn't try to implement
    any notion of globally known type.
    
    Robert Ramey
    

- Done.

  


attached mail follows:

-----Original Message-----
From:	Vladimir Prus [SMTP:ghost_at_[hidden]]
Sent:	Wednesday, October 02, 2002 8:24 AM
To:	Robert Ramey
Subject:	Serialization library

Hi Robert,
some time ago you've said that you'll think about global vs. local 
classes registration. Did you came to any conclusions? I'm asking 
because review of your library is scheduled to start on Nov 2, and it's 
not so much time remaining.

- Volodya

Please don't think I've forgotten it - I havn't.  I have spent significant time considering it.
Given the rigorous nature of the interchange, I can't really send something that hasn't
been carefully thought out and this takes time.

Its a good thing you emailed me as for some reason the date in my head was 25 Nov
rather than 2 Nov.  I checked and it is 2 November.

For your information I am considering the following:

1)Peter Dimov submitted a test case that shows that classes which multiply inherit
don't work.  This is a deficiency that I'm working to remedy.  Unfortunately, it turns
out to be non-trivial and require a type list implementation to do in a definitive manner.
This means I want to use mpl which isn't in the library until 1_29 which isn't yet released.
I'm hoping it will be released before 2 Nov - I will have to check.

2)Peter also compiled the package with Comeau which spit out some errors.  It seems that
Comeau is more picky about certain C++ rules.  So I've been making tweaks to make
it pass this test as well.

3)It was also requested that I provide support of standard C arrays.  I said I would consider it.
It turns out that is not hard and fits well with the library.  It does require coding, make a test
and updating the documentation so that consumes some time.

4)I also spent some time trying out ideas where a concept such as describe would logically fit.
I reached no new conclusions except to reenforce my view that reflection (of which I 
see describe as a facet) is a larger topic than first meets the eye and is better kept separate
from serialization.

I also spent time considering more carefully this issue of global vs local registration.  I see
two issues here.

a)I concede that it is inconvenient for an  to have to maintain a list of derived classes that might be
included in an archive. 
a) I do appreciate the fact that it is convenient for an upper level module to be 

a) I do appreciate the view that an upperlevel module that works though an interface
of base class pointers should not have any dependency on derived implementations.  The
current archive specific registration system violates that view.

I appreciate the appeal the idea that each class just register itself with a global
singleton. .  On the other hand it does create the need for a singleton of global scope -
something that I generally don't prefer.  Given the above however I am leaning to provide
such a facility.

b) I do not believe that this enough to necesitate program independent method of registration - 
that is as system dependent upon type_info.  I believe the desire to implement such a thing
is related to the view that it could be used to implement plug-ins.

b) plug-ins.  It would seem that global registration of some sort is necessary to support
plug-ins.  Indeed it is.  But I don't believe that global registration of the most derived base
classes is going to be sufficient.  I base my view on my experience with Microsoft COM - 
Common Object Model .  Presumably CORBA is similar.  In these systems all
plug-ins are derived from a common base.  This common base has virtual functions
equivalent to our save/load commands.  So when an object is serialized, the object
schema itself is not visible - only the vtable containing the serialization

I believe that any "plug-in" implementation has to look something like the following.
Of course this is vastly over-simplified to illustrate my point.

class plugin_interface
{
        // a function which creates an object of the type requested
        // and returns a pointer to its base class
        static plugin_all *plugin_factor(GUID g);
        virtual GUID get_guid();
        // command a plug_in to save it self
        virtual void save(oarchive &ar) const;
        // or load itself
        virtual void load(iarchive &ar);
        // other interface functions for this family of plug-ins
        // ...
}

// note: the class definition for a specific plug-in does not
// appear in the program using the plugin - all functionality is
// accessed though its public interface as defined by the base class.

class plugin_implementation : public plugin_interface
{
        // a globally unique identifier - maybe typename or ?
        const GUID g;
        virtual GUID get_guid(){
                return g;
        }
        virtual void save(oarchive &ar) const;
        virtual void load(iarchive &ar);
        // other interface functions for this family of plug-ins
        // ...
};

void  serialization<plugin_interface>::save_ptr(boost::basic_oarchive & ar, const T * t)
{
    // default just derefences the pointer
    // and saves the object using the normal method
    ar << g;
    ar << *t;
}
void serialization<plugin_interface>::load_ptr(
    boost::basic_iarchive & ar,
    T * &t,
   boost::version_type file_version
){
    // If de-serialization fails due to an exception
    // this will guarentee that delete isn't called on a non-null pointer
    t = NULL;	
    try{
        // get globally unique identifier - global type identifier
        GUID guid;
        ar >> guid;
        t = plugin_base_factory(guid);
        ar >> *t;
    }
    catch(...){
        // uh-oh object in the archive could not be created - most likely
        // plugin that creates and implements it is not installed on the
        // machine reading the archive or not supported by the program
        // reading the archive.
        delete t;
        throw;
    }
}

The above illustrated (I hope) that the issue of plug-ins is not related to the desire to eliminate
the need to explicitly register otherwise unreferened derived classes through pointers.

So I anticipate that I will try find a way that addresses a) above but doesn't try to implement
any notion of globally known type.

Robert Ramey