Re: [Boost-users] [Boost.MPI] question about serializingcustomobject

Subject: Re: [Boost-users] [Boost.MPI] question about serializingcustomobject
From: Matthias Troyer (troyer_at_[hidden])
Date: 2010-08-27 15:05:38

On 27 Aug 2010, at 19:44, Robert Ramey wrote:

> David Abrahams wrote:
>> At Thu, 26 Aug 2010 16:05:41 -0800,
>> Robert Ramey wrote:
>>> I just never found it to be very revealing.  The word skeleton
>>> seemed pretty suggestive. It's still not clear to me how such a
>>> think can work between heterogeneous machines.  For example, if I
>>> have an array of 2 byte integers and they each need to get
>>> transformed one by one into a 4 byte integer because that's closest
>>> MPI data type,
> 
>> I think you don't understand what MPI datatypes do.
> 
> This is true.  I suppose that's one reason why the documentation
> made no sense to me. They just looked like special types
> to make identify primitives accross differing architectures.
> 
> <snip>
> Interesting information which one should consider adding to the
> MPI documentation.
> </snip>

I disagree. We cannot and should not copy dozens of pages of MPI documentation. Boost.MPI is targeted at an audience that understands MPI but does not presume to teach MPI.

> 
> I only really have few observations/suggestions at this point.
> 
> a) It would be  helpful if there were a way to test the serialization
> of the archive classes in boost MPI without having MPI installed.
> If this is not possible, it would seem to me that the serialization is
> intertwined with the data transport - rather than be separated
> as it is in the stream/streambuf io/stream design.  This would look
> like a design flaw to me.

No, we separate transport from serialization, it is just that both transport and serialization require calls to the MPI library.
> 
> b) user experience seems to show that archive construction/destruction
> is a significant performance issue when a new archive is made for each
> data transmission.  On the otherhand, one has to do this since the
> current archive implemenation track addresses of serialized obects
> so the same archive can't e use send the same structure (maybe
> with changed data) multiple times.  Given that MPI has a focus
> on performance, I wonder if this has been considered. I looked
> a the documentation, code and examples and it wasn't obvious
> to me how this is question was addressed - if at all.

It has been considered and that is the concept behind skeleton&content and was a key reason to Boost.MPI: you set up a description of the data structures once using the "skeleton" and then can just blast data from object member to object member across the network without the use of any archive or buffer (except for any buffering in the network hardware).
> 
> You should know that all your efforts to educate me are not wasted.
> 
> a) Sometimes I pass your advice along to other people.
> 
> b) Motivated by our recent discussions on the subject as well as some
> other issues has clarified my ideas on how the archive concept
> should be refined to be more helpful to those creating their own
> archives.  I think there will be improvements in this area in part
> due to your observations.

Thanks for your efforts

Matthias