Re: [boost] Proposal: MapReduce library (single machine)

Subject: Re: [boost] Proposal: MapReduce library (single machine)
From: Craig Henderson (cdm.henderson_at_[hidden])
Date: 2009-08-31 07:09:34

> I've just joined this mailing list because of this particular
> discussion :-)
> 
> >> 2) have a plan so that the library can eventually range from
> >> working on 1 multi-core system to distributed
> 
> >This is the eventual goal, and I have designed the library to be
> extensible
> >to be able to do this in the future. With the policy based designed,
> making
> >use of the open source projects that you mention should be non-
> intrusive;
> >the library allows the user to supply a handler for the intermediate
> files
> >(to use kosmosfs or hypertable, for example)
> 
> I'm going to give it a spin as it happens I'm already having a
> half-baked distributed backend lacking a mapreduce library for both
> hypertable and KFS and report back. But this is interesting! Can't
> stop thinking about it now :-)
> 

Mateusz, did you  make any progress? I have completed the single-machine
implementation; it's performance is comparable to other libraries such as
Phoenix (http://mapreduce.stanford.edu) and has been tested by a few people
on this list. I am thinking now about how to progress the library and am
interested in your experiences.

Thanks
-- Craig