Re: [boost] Using serialization for replication

From: Preston A. Elder (prez_at_[hidden])
Date: 2006-04-10 15:00:05

Sorry if you see this twice, but I don't think the original reply was sent.

Robert Ramey <ramey_at_[hidden]> wrote:
> create a TEE type streambuf.  This would model the std::streambuf
> that the standard library uses.  It would most likely be built with
> the i/o streams library.  All date written to the streambuf would infact
> be written to multiple stream implementations.  This would get you
> replication for free.  In fact, what would be more useful would be
> an i/o stream adapter which would take any number of streambufs
> and compose them into one TEE type streambuf.  This would permit
> one to leverage on all the streambufs already created.  It would mean
> that the the streambufs would all have to be the same type.  Some
> could be binary, others could be file bases, others could be network
> connections, etc.  This is something that could/should be added to
> the i/ostreams library - if it isn't already there.
> 
> The counter part of this - reading back one of the archives in  the same
> application would read one of the streams in the TEE.  Remember that
> all information concerning the state of the archive, addresses of created
> pointers etc, class i/d, etc is local to the archive.  So there would be
> no conflict.
If I used this method I would end up with objects being duplicated!

If I had a TEE style object and had:
 - 1 endpoint going to a local input stream
 - 1 endpoint going to disk
 - 1 endpoint going to a remote system (via. X transport method)

I would end up with multiple objects because of the first endpoint!
Every time the first endpoint saw a new object, it would allocate that
object and deserialize it, just like the remote one would (and should)
do.  This would mean every object would be there twice!

If, however, I could share the tracking map (eg. create a tracking map,
then pass it to the constructor of both the input and output
serializer, or alternatively, set it later or whatever), then this
would not be an issue.

>> Also, might I make a request to the
>> maintainer of serialization to hopefully turn boost::serialization
>> into something that can be more suitable for replication purposes in a
>> future release of boost?
> So I don't see serialization as the right place to implement such 
> functionality.
Perhaps you're correct, perhaps CORBA is more appropriate.

However AFAIK, CORBA doesn't work so well when loading from a disk.
Plus CORBA doesn't solve one of my requirements.  My requirements are
simple:
 - Be able to restore the application to the same state it was when it
 died from a file on disk (persistence).
 - Be able to keep another instance of the application up to date
 real-time and be able to fail over to that system if necessary.
 - Be able to fail BACK to the original instance if necessary (eg. it
 is re-started and ready to once again be 'primary').

Serialization can handle all of these for me with some trickery to make
it handle 'updating' objects instead of just creating them (previously
mentioned in this thread).  However there is one thing that is
dangerous, and that is the fact that each instance of the application
will have to have an iarchive and an oarchive at all times.  And they
will need to have their object tracking in sync.

> LOL - The reason the interface is "fantastic" is mainly due to my single 
> minded
> dedication to keeping it that way.  The way I've done this is to keep 
> everything
> out of it that can possible be put somewhere else.  I realize that this 
> sometimes
> might seem limiting - but in fact its liberating.  It has kept serialization 
> from
> turning into he C++ equivalent of Microsoft word - where it would do
> everything everyone wanted if anyone could ever figure out how to make it
> do what it is they want.  In spite of this, the serialization library 
> implementation
> is still quite complicated.
I know, I've looked at the code:)

However I'm not actually asking you to change much - just have the
ability to have two archives have the same object tracking backend.

A simple ability to just do CreateObjectTracker() and then pass the
result to the constructor of any archive I create after that would be
sufficient, even if the return value is completely opaque.  And of
course, if implemented as an optional argument to a constructor, the
default action could be to call that same function anyway.

I could still use boost's serialization without this functionality,
however I lose the biggest advantage (and strength) of serialization.
Namely the restoration of pointers automatically.

In other words, I could easily enough just create a new archive
instance for each time I want to serialize an object (or simply turn
off tracking for all objects), however this would mean all that book
keeping serialization does for me with previously seen objects and
restoring pointers and such would now have to be done by me, and more
importantly, done manually - increasing the possibility of missing
something.

The serialization library is VERY close to the functionality required
for replication (which is more or less a specialized form of
serialization anyway), it just has a few specific requirements that I
don't believe would change the way serialization works or complicate it
much more than it is now.

> I'm pleased you seem to like the library and have found it useful.
I just like the interface, its very clean.

-- 
PreZ :)
Death is life's way of telling you you've been fired.
		-- R. Geis