Re: [boost] [MPI] Boost.MPI Review

From: Markus Blatt (Markus.Blatt_at_[hidden])
Date: 2006-09-16 06:09:48

Hi,

On Sat, Sep 16, 2006 at 08:41:53AM +0200, Matthias Troyer wrote:
> 
> To counter the mentioned overhead, there is the "skeleton&content"  
> mechanism for cases when a data structure needs to be sent multiple  
> times, with different "contents" while the "skeleton" (the sizes of  
> arrays, the values of pointers, ...) of the data structure remains  
> unchanged. In that case the structural information only (sizes,  
> types, pointers) is serialized using MPI_Pack and sent once so that  
> the receiving side can create an identical data structure to receive  
> the data. Afterwards an MPI_Datatype for the "contents" (the data  
> members) of the data structure is created, and sending the content is  
> done using this custom MPI_Datatype, which again gives optimal  
> performance.

This skeleton&content approach sounds pretty powerful to me. Is there
a way for the user to tune this behaviour?

Until now I always had the impression that you only allow the sending
and receiving of complete objects and/or their structure.
But often there are other scenarios, too:

One maybe wants to send just parts of an object. And it might be the
case that the local structure of your object is different from your
global structure, e.~g. it might need redistribution in the
communication.

Let me give you an example:

Consider a distributed array or vector. Each entry of the global
vector has a consecutive, zero starting local index l and a
coresponding global index g together with tag specifying whether the
process is owner of the value (meaning that he can compute a new value
from consistent data without communication) or not.

Process 0: 
local global tag
0     0      owner
1     3      owner
2     4      notowner

Process 1:
local global tag
0     1      owner
1     2      owner
3     3      notowner

This would represent a global array of 5 entries.

In our case we have the following communication patterns:

owner to notowner
Each process sends to each other process all entries of the array that 
are tagged as owner and on the other process as notowner in one
message. The matching is done via the global id.

In this case 0 sends to 1 the value at index 1 (global: 3) and process
1 stores is at local index 3. If that is not clear, take a look at 
http://hal.iwr.uni-heidelberg.de/~mblatt/communication.pdf (early
draft) for the more exact notion of index sets.

Would something like this be possible or can one just send and receive
objects with the same layout?

Cheers,

Markus Blatt
-- 
DUNE -- The Distributed And Unified Numerics Environment 
<http://www.dune-project.org>