[boost] [futures] High-level Parallel Processing Library

From: Andrey Melnikov (melnikov_at_[hidden])
Date: 2005-08-19 03:51:37

> You might want to look at the bottom of
> http://www.osl.iu.edu/research/pbgl/documentation/graph/index.html
> 
"Parallel Processing Library"

http://www.osl.iu.edu/research/pbgl/documentation/parallel/

referenced there is exactly "a higher level framework for distributed 
computations" I suggested you to look at.

Unfortunately, it offers very little additional abstraction over raw 
MPI2 C++ API so it's advantage is doubtful.

I thought of library which offers more specialized work flow concepts.

For example, often a distributed environment is used just because there 
are a lot of data to be processed, and each piece of data can be 
processed completely independently or without major cooperation between 
computational nodes. Examples are video rendering/processing (if it's 
not I/O bound), distributed C++ compilation, SETI_at_Home/other BOINC projects.

Such tasks fit a model which I call "std::transform with a stateless 
functor". MPI has abstractions for this (like MPI_Scatter/Gather) but 
it's very limited, ineffective and practically useless in many 
circumstances. Different node speeds, different and non-predictable 
amount of resources needed to process each data unit lead to node 
stalls. And MPI_Scatter requires an in-memory array instead of just an 
input, a past-the-end and an output iterator.

Andrey