Re: [Boost-mpi] mixing boost serialization with C MPI functions

Subject: Re: [Boost-mpi] mixing boost serialization with C MPI functions
From: Lorenz Hübschle-Schneider (huebschle_at_[hidden])
Date: 2015-11-19 10:56:14

Hi,

On 19/11/15 16:07, Jiaxin Han wrote:
> Hi,
>
> Could anyone point me to an example of mixing boost serialization with C
> MPI functions?

you need to send the archive size first if it's not precisely known. Try 
something like this:

mpi::packed_oarchive oa(comm);
oa << sendbuf;
auto sendptr = const_cast<void*>(oa.address());
// cast to int because MPI uses ints for sizes like it's still 1990
int sendsize = static_cast<int>(oa.size());
MPI_Send(&sendsize, 1, MPI_INT, 1, 0, comm);
MPI_Send(sendptr, sendsize, MPI_PACKED, 1, 0, comm);

The receiving side would look similar:

mpi::packed_iarchive ia(comm);
int recvsize;
MPI_Recv(&recvsize, 1, MPI_INT, 0, 0, comm, MPI_STATUS_IGNORE);
ia.resize(recvsize);
auto recvptr = ia.address();
MPI_Recv(recvptr, recvsize, MPI_PACKED, 0, 0, comm, MPI_STATUS_IGNORE);

> Do I have to make sure buffer is big enough or the oarchive will handle
> the memory automatically? If the former, what is the correct memory size
> to hold a vector? I guess it should hold not just vec.data(), but also
> vec.size().

It does this automatically for the sending size. On the receiver, you 
have to know how much data to expect and reserve enough memory 
accordingly. This is what the first transmission in my code above is for.

> And lastly, oa does not appear to be the correct variable to pass to
> MPI_Send. Then what should I pass to MPI_Send after creating the archive?

You pass in the pointer and its length -- oa.address() and oa.size() in 
the example above.

> I am asking because the boost mpi installation on our server appears to
> have a limitation on the message size. For example, the attached code
> reports an error as:
>
> terminate called after throwing an instance of
> 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::mpi::exception>
>  >'
>    what():  MPI_Alloc_mem: MPI_Alloc_mem: Out of "special" (shared) memory
> MPI Application rank 0 killed before MPI_Finalize() with signal 6
>
> This appears to be a problem only with the boost installation on the
> server. The code runs correctly on my local machine.

I'm no expert but to me this sounds like an issue with the MPI 
installation on your servers. Note that it's not Boost.MPI that's giving 
you an error - the error is from the C function that it calls.

Cheers,
Lorenz