[boost] A library: out-of-core containers and algorithms

From: Roman Dementiev (dementiev_at_[hidden])
Date: 2005-02-06 16:40:02

Hello,

I am developing a library called Stxxl.
It is an implementation of  STL for external memory (out-of-core) 
computations, i.e. Stxxl implements
containers and algorithms that can process huge volumes of data that 
only fit on disks.

Currently I have implemented vector, stack, and priority_queue. External
memory map, list, and queue are coming soon. The containers
take only specified (given) fixed amount of main memory, but can contain 
more elements than can fit into the main memory.
The containers are compatible with STL algorithms, an example:

#include <stxxl>

// ten billion doubles
stxxl::vector<double>  HugeVector(10ULL * 1000000000ULL);
std::fill(HugeVector.begin(), HugeVector.end(), 0.0);

STL algorithms that rely on Output, Input, Forward and Bidirectional 
iterators will work I/O efficiently with Stxxl containers. For the 
RandomAccessIterator algorithms (sortting) I have implemented 
specialized I/O efficient implementations:

// ten billion doubles
stxxl::vector<double>  HugeVector(10ULL * 1000000000ULL);
std::generate(HugeVector.begin(), HugeVector.end(), MyRandom());
// sort the vector using only 512 MB of main memory
stxxl::sort(HugeVector.begin(), HugeVector.end(), 128*1024*1024);

or another example (sort a file):
stxxl::file myfile("../myfile.dat",stxxl::file::RDWR);
stxxl::vector<double>  HugeVector(&myfile);
// sort the vector using only 512 MB of main memory
stxxl::sort(HugeVector.begin(), HugeVector.end(), 128*1024*1024);

More details about the Stxxl library: 
http://i10www.ira.uka.de/dementiev/stxxl.shtml

Would such a library be interesting for Boost users? Does it fit here? 
Is it worth to boostify it?

I would be very glad to know your opinion.

With best regards,
Roman