Re: [boost] [MPI] Boost.MPI Review

From: Matthias Troyer (troyer_at_[hidden])
Date: 2006-09-15 16:40:25

On Sep 15, 2006, at 9:12 PM, Markus Blatt wrote:

>
>>
>> - What is your evaluation of the design?
>>
>
> The design is pretty straight forward. Therefore I think everybody
> familiar with the C-Interface of MPI will feel at home at once.
>
> One thing that struck me at once is the missing of tags for the
> communication. In the MPI standard they are everywhere and usually are
> used to distinguish different communications (as they might get to the
> target process in arbitrary order). How can the programmer make sure
> that the message that the process is receiving is the message that he
> wants to receive at this point of the program? If there is no such  
> means I
> would consider this a serious flaw in the design.

I don't fully understand that. There are tags in all the point-to- 
point communications, which is where the MPI standard supports them.

>
>> - What is your evaluation of the implementation?
>
> One thing I do not understand is why the collective operations use
> std:vector to collect the data. I always thought that there is now
> garanty by the standard that the values are stored in contiguous
> memory (which is what MPI needs).

Section 23.2.4 of C++03 states "The elements of a vector are stored  
contiguously"
> What if the user wants to comunicate from/to a given data structure
> represented by a standard C-Array? This might be the case very often
> in scientific computing where you tend to reuse existent libraries
> (probably implemented C or by a novice C++ programmer).

This is a good point. Doug, can we extend the interface to also allow  
a C-array?

> As I see the implementation is using the boost serialization
> framework to send data types that do not correspond to a standard
> MPI_Datatype. While in this way it is still possible to send values
> between different architectures, I fear that there is (in some cases)
> a great efficiency loss:
> E. g. with none-PODs broadcasts (and probably other collective
> operations) do not use MPI's collective operations, but independent
> asynchronous MPI send and receives. This might circumvent applications
> of special optimization based on the underlying network topology
> (e. g. on a hypercube the broadcast is just O(ld P) where P is the
> number of processes).
>
> This is a knock-out criterium for people doing
> High-Performance-Computing where efficiency is everything unless there
> is a way of sending at least 1-dimensional arrays the usual way. But I
> fear that even then what they gain by using Boost.MPI is not enough to
> persuade them to use it.

I believe that your comments are based on a misunderstanding. The  
goal of the library design was to actually make it efficient and  
useful for high performance applications. The library allows the use  
of custom MPI_Datatypes and actually creates them automatically  
whenever possible, using the serialization library. For those data  
types for which this is not possible (e.g. the pointers in a linked  
list) the serialization library is used to serialize the data  
structure, which is then packed into a message using MPI_Pack.

The  skeleton&content mechanism is essentially a simple way to create  
a custom MPI datatype. The "content" is just an MPI_Datatype created  
for the data members of the object you want to send.

Another case is when sending, e.g. a std::vector or other array-like  
data structure of a type for which a custom MPI_Datatype can be  
constructed. In that case the serialization library is called once to  
construct the needed MPI_Datatype for one element in the array, and  
then communication is done using that MPI_Datatype.

You might be worried that the use of the serialization library leads  
to inefficiencies since in the released versions each element of an  
array was serialized independently. The recent changes, which are in  
the CVS HEAD, actually address this issue by providing optimizations  
for array-like data structures.

I hope these answers address the issues you had in mind. I can  
elaborate if you want.

>
> Maybe it would be possible to use MPI_Pack or create custom
> MPI_Datatypes?

Yes, both are implemented and used automatically.

>> - What is your evaluation of the potential usefulness of the library?
>
> At the current state it will probably not be used by people doing High
> Performance Computing due to the lack of efficiency stated above.
> If people do not need best performance then the library is useful to
> them as it is by far easier to use as MPI for a C++ programmer.

I believe, as stated above, that this is just a misunderstanding.

> I don't think I am in the position to argue about that as I do not  
> have
> enough background knowledge about boost, but:
> If you want to adress people who are into High Performance Computing,
> you should address the issues above before you incorporate it into
> boost.

This was one of the main goals, and a large fraction of the design  
and implementation time was spent on optimizing the array  
serialization so that the serialization library could be used to  
create MPI_Datatypes as well as to allow efficient message passing  
using MPI_Pack.

> If you just want people, to get started with parallel programming
> more easily, go for it. But unless effiency is granted by Boost.MPI
> those people have to hand code and deal with MPI directly sooner or
> later to get the efficiency they need.
>
> I hope I could help in the evaluation process.

Thank you very much for your comments!

Matthias