Re: [Boost-users] hybrid parallelism

Subject: Re: [Boost-users] hybrid parallelism
From: Matthias Troyer (troyer_at_[hidden])
Date: 2010-11-05 17:29:28

On 5 Nov 2010, at 01:46, David Abrahams wrote:

> At Thu, 4 Nov 2010 16:34:19 +0100,
> Matthias Troyer wrote:
>> 
>> On Nov 4, 2010, at 3:14, Brian Budge <brian.budge_at_[hidden]> wrote:
>> 
>>> Do these tasks share a lot of data?  If they are really lightwieght
>>> memory-wise, heavy computationally, and don't require fine-grained
>>> communication with each other, I'd go with David's suggestion, as it
>>> will be easier to write, and the performance won't be much different.
>>> 
>>> If you use a lot of memory, need fine-grained chatter between tasks,
>>> or the tasks are pretty cheap, threads may be (much) better.
>>> 
>>> Brian
>>> 
>> 
>> I second this opinion for several reasons
>> 
>> First, mixing MPI with multithreading can be hard since many MPI
>> implementations are not thread safe. Be sure to only let the master
>> thread use MPI.
> 
> Good point; I hadn't thought of that.
> 
>> Secondly, it adds another level of complexity. Starting M*N MPI processes is much easier, unless you waste too much memory that way.
> 
> That was mostly what I had in mind.
> 
>> Third, we have just recently benchmarked several multithreaded
>> LAPACK routines in the Intel, AMD and other lapack libraries and
>> compared them to the MPI based routines in SCALAPACK. Surprisingly
>> the MPI implementations outperformed the multithreaded ones by a
>> large margin. 
> 
> Was that done running the MPI processes together on one machine?

Yes, on a 16-core SUN Blade and also on other shared memory workstations. It seems to show - at least this is my first conclusion - that the more explicit memory management and explicit barriers in the MPI based program can give faster code than the easier to write multi-threaded method.

Matthias