[boost] Re: A library: out-of-core containers and algorithms

From: Roman Dementiev (dementiev_at_[hidden])
Date: 2005-02-08 09:01:55

Brian Braatz wrote:
> 
>>-----Original Message-----
>>On Behalf Of Roman Dementiev
>>Subject: [boost] A library: out-of-core containers and algorithms
>>
>>Hello,
>>
>>I am developing a library called Stxxl.
>>It is an implementation of  STL for external memory (out-of-core)
>>computations, i.e. Stxxl implements
>>containers and algorithms that can process huge volumes of data that
>>only fit on disks.
>>
>>Currently I have implemented vector, stack, and priority_queue.
> 
> External
> 
>>memory map, list, and queue are coming soon. The containers
>>take only specified (given) fixed amount of main memory, but can
> 
> contain
> 
>>more elements than can fit into the main memory.
>>The containers are compatible with STL algorithms, an example:
>>
>>#include <stxxl>
>>
>>// ten billion doubles
>>stxxl::vector<double>  HugeVector(10ULL * 1000000000ULL);
>>std::fill(HugeVector.begin(), HugeVector.end(), 0.0);
>>
> 
> [Brian Braatz] 
> Roman-
> 	Thank you for submitting this. I am interested in this library
> and encourage you onward in bringing this to boost.
thank you

> 
> Question:
> 	In a nutshell, if you could please describe 
> 	* How it works
> 	* what the implications\dark corners are
what exactly do want to know?
Try to look at the draft of the Stxxl tutorial:
http://i10www.ira.uka.de/dementiev/files/stxxl_tutorial.pdf

> 
> Are you using memory map files?
my library can use memory maped files, but it can also
rely on a basic I/O calls: open/read/write ...

> What other libraries or os specific
> functions does your library need.
the current implementation the library needs pthreads and
support of POSIX file system calls (open/read/write/close).

As I already mentioned in other postings, I would use
boost threads instead of pthreads to make the library
more portable and boost compliant.

What concerns I/O, the library needs:
1) large file (>2GB) support (e.g. FAT32 does not support large files)
2) avoidance of superfluios buffering (like in std::fstream)
I it also desirable to have (for performance reasons):
3*) unbuffered file access (ASAIK Windows does have it)

Roman