Re: [Boost-users] boost multithread use all cores in parallel ?

Subject: Re: [Boost-users] boost multithread use all cores in parallel ?
From: Gordon Woodhull (gordon_at_[hidden])
Date: 2011-08-14 14:47:49

On Aug 14, 2011, at 1:38 PM, Jack Bryan <dtustudy68_at_[hidden]> wrote:

> My program needs to run many (about 10,000+) computing tasks. 
> 
> Each task's run time is very very short (< 0.1 second or even less). 
> 
> At each iteration, all these tasks are run in parallel. Some of them
> need to exchange some data (it is very small) and then go on. 
> 
> I want to keep all 144 cores as busy as possible so that my program 
> can be done as fast as possible. 
> 
> So, I want to associate each task with a distinct thread and schedule 
>  threads as many as possible. Also, try yo make the workload balance
> among these cores.  
> 
> How can I do that from the point of programming ? 

A "thread pool" is the usual solution. You might want to try this one (based on boost):

http://threadpool.sourceforge.net/

Haven't tried it myself. 

If you mean the tasks need to exchange data *with each other*, then that's getting more complicated and you might need to look at mpi or something. However if you mean just block for I/O then you might be able to run more threads than cores.

Cheers
Gordon