Re: [Boost-users] sending large serialized objects over network using MPI

Subject: Re: [Boost-users] sending large serialized objects over network using MPI
From: Dave Abrahams (dave_at_[hidden])
Date: 2011-10-31 12:06:45

on Fri Oct 28 2011, Jonathan Bishop <jbishop.rwc-AT-gmail.com> wrote:

> HI,
>
> I am involved in a project which requires the transport of large C++
> instances over MPI communication channels. I am using boost
> serialization to pack and unpack the objects into a buffer. The whole
> buffer is sent in one shot.

I think you should seriously look into Boost.MPI.  It too uses Boost
Serialization, but it can often avoid the extra step of packing into a
separate buffer.

>
>
> Of course for large objects this can be problematic. I have two
> questions...
>
> 1) is there any way to tell beforehand how large the buffer needs to
> be? In other words, can you run serialization in a counting only
> mode?
> 2) is there an archive implemented which streams across MPI
> communication channels (not sure this is the right terminology.)?
> Seems like it might be useful. This might be a better question for
> the boost MPI team, though.

Yes, see Boost.MPI.  You might also want to read
http://daveabrahams.com/2010/09/03/whats-so-cool-about-boost-mpi/

-- 
Dave Abrahams
BoostPro Computing
http://www.boostpro.com