Re: [boost] [OT?] SIMD and Auto-Vectorization (was Re: How to structurate libraries ?)

Subject: Re: [boost] [OT?] SIMD and Auto-Vectorization (was Re: How to structurate libraries ?)
From: Mathias Gaunard (mathias.gaunard_at_[hidden])
Date: 2009-01-18 10:24:57

Dean Michael Berris wrote:

> When you talk about threads, those are high level in the context of
> single-machine parallelism

What is a machine?
What's the difference between two machines with a CPU each and a NUMA 
machine with two CPUs? Only the protocol (and its throughput and 
latency, of course) to access memory on another CPU.

NUMA is really a form of cluster computing. And you can implement NUMA 
in software over a cluster network.

Threads are just tasks that share memory. There is no need for all those 
tasks to be on the same machine: memory be can distributed.

> -- unless your OS is a distributed OS that
> spawned processes across multiple machines and magically showed just
> one big environment to programs running in that parallel machine.
> Threads can be considered parallelism primitives in a single machine
> context, but not in a general parallel computing context where you
> might be spanning multiple distributed/independent machines.

That's a single-system image, and it actually works very well if you 
have the required low-latency network.

> BTW, you can use asynchronous communication primitives in MPI and
> still use threads on each process. I don't even think MPI and Threads
> work at the same level anyway so it depends on which level you're
> looking from which determines whether you think MPI is low-level or
> Threads are high-level. ;-)

I would personally prefer to describe parallel tasks in one way once and 
for all rather than have to express them in different ways depending on 
what level of hardware parallelism can be used to run them.