Re: [Boost-mpi] multiple irecv tests failure with MPI_ERR_TRUNCATE

Subject: Re: [Boost-mpi] multiple irecv tests failure with MPI_ERR_TRUNCATE
From: Walter Woods (woodswalben_at_[hidden])
Date: 2014-02-23 17:00:44

Some penalty, but the largest performance penalty in my case is by far the
transmission time.  I can serialize and deserialize the data dozens of
times before I catch up to transmission penalties.

On Sun, Feb 23, 2014 at 1:58 PM, Matthias Troyer <troyer_at_[hidden]>wrote:

> Hi Walt,
>
> You are aware that you are already incurring a significant performance
> penalty by sending objects that need to be packed into an archive and
> cannot be sent directly?
>
> Matthias
>
> On 23 Feb 2014, at 22:50, Walter Woods <woodswalben_at_[hidden]> wrote:
>
> Unless there is a significant performance penalty in each
> boost::mpi::communicator holding onto a shadow version of itself, I think
> the headaches this would save would be worth it.  Simply put, the current
> behavior is just very annoying to debug, and it is a somewhat common use
> case (especially in applications that know they are receiving exactly N
> messages, and want to wait on them simultaneously).
>
> Thanks,
> Walt
>
>
> On Sun, Feb 23, 2014 at 8:18 AM, Matthias Troyer <troyer_at_[hidden]>wrote:
>
>> Indeed, that is the problem. If we don't want to reserve certain tags for
>> internal use of Boost.MPI then the only secure way of solving this problem
>> is to create a copy of the communicator, and send the actual message using
>> a unique tag in this shadow communicator. We so far hesitated to implement
>> this procedure, thinking it to be very unlikely that a user would send a
>> second message with the same tag before the first one is received. if this
>> should turn out to be a common usage case then we can consider the solution
>> I outlined. Does anyone see problems with that solution?
>>
>> Matthias
>>
>>
>>
>> On 23 Feb 2014, at 04:39, Walter Woods <woodswalben_at_[hidden]> wrote:
>>
>> > seems to indicate that MPI guarantees that sends and recvs are kept ordered
>> on a single-threaded process not using MPI_ANY_SOURCE. If that is the case
>> then boost::mpi should as well.
>>
>> Right, so they are ordered, and that's the problem.
>>
>> boost::mpi needs to know exactly the size of data that it's receiving.
>>  So, if you If you're sending / receiving a non-native type, boost::mpi
>> needs to transmit how big that data is going to be.  Then, it sends the
>> data.  So one send becomes two sends to MPI - these are ordered.
>>
>> Receiving is the opposite - it uses one receive to get the size, and then *after
>> it has the size*, issues another receive to get the data.  If you issue
>> one irecv command before another has gotten its length (and thus issued its
>> data irecv command internally), then because of message ordering, the first
>> irecv will get the length, as expected, but then the second irecv will get
>> the first's data, mistaking it for a length submission.
>>
>> Hopefully that makes sense.  It's an interleaving problem - because
>> everything is ordered, but irecvs turn into two underlying MPI irecvs, the
>> two boost::mpi irecvs interleave, causing the problem.
>>
>>
>> On Fri, Feb 21, 2014 at 5:52 PM, Roy Hashimoto <roy.hashimoto_at_[hidden]>wrote:
>>
>>> On Fri, Feb 21, 2014 at 11:49 AM, Walter Woods <woodswalben_at_[hidden]>wrote:
>>>
>>>> In Roy's case, especially the test file, the problem is having multiple
>>>> irecv's happening.  Lookat the underlying request::handle_serialized_irecv
>>>> implementation in boost/mpi/communicator.hpp - one recv is accomplished
>>>> through several MPI_IRecv requests issued in sequence.  If you have several
>>>> irecvs running at once, then one is likely to get the other's data as its
>>>> length.
>>>>
>>>
>>> Thanks for your reply and looking at the boost::mpi source - I haven't
>>> got that far. I understand what you're saying, but the first few paragraphs
>>> of this page:
>>>
>>>  http://www.mpi-forum.org/docs/mpi-1.1/mpi-11-html/node41.html
>>>
>>> seems to indicate that MPI guarantees that sends and recvs are kept
>>> ordered on a single-threaded process not using MPI_ANY_SOURCE. If that is
>>> the case then boost::mpi should as well.
>>>
>>>
>>>> In other words, if you want to receive multiple messages in the same
>>>> tag, be sure to only have one IRecv() with that tag running at a time.
>>>>  Data may only be transferred serially (not in parallel) over a single tag
>>>> anyhow.
>>>>
>>>
>>> I did change my development code to do this.
>>>
>>> Hope that helps,
>>>>
>>> Walt
>>>>
>>>
>>> It does, thanks!
>>>
>>> Roy
>>>
>>> _______________________________________________
>>> Boost-mpi mailing list
>>> Boost-mpi_at_[hidden]
>>> http://lists.boost.org/mailman/listinfo.cgi/boost-mpi
>>>
>>>
>> _______________________________________________
>> Boost-mpi mailing list
>> Boost-mpi_at_[hidden]
>> http://lists.boost.org/mailman/listinfo.cgi/boost-mpi
>>
>>
>>
>> _______________________________________________
>> Boost-mpi mailing list
>> Boost-mpi_at_[hidden]
>> http://lists.boost.org/mailman/listinfo.cgi/boost-mpi
>>
>>
> _______________________________________________
> Boost-mpi mailing list
> Boost-mpi_at_[hidden]
> http://lists.boost.org/mailman/listinfo.cgi/boost-mpi
>
>
>
> _______________________________________________
> Boost-mpi mailing list
> Boost-mpi_at_[hidden]
> http://lists.boost.org/mailman/listinfo.cgi/boost-mpi
>
>