From: Roman Dementiev (dementiev_at_[hidden])
Date: 2005-02-08 09:01:55
Brian Braatz wrote:
>>On Behalf Of Roman Dementiev
>>Subject: [boost] A library: out-of-core containers and algorithms
>>I am developing a library called Stxxl.
>>It is an implementation of STL for external memory (out-of-core)
>>computations, i.e. Stxxl implements
>>containers and algorithms that can process huge volumes of data that
>>only fit on disks.
>>Currently I have implemented vector, stack, and priority_queue.
>>memory map, list, and queue are coming soon. The containers
>>take only specified (given) fixed amount of main memory, but can
>>more elements than can fit into the main memory.
>>The containers are compatible with STL algorithms, an example:
>>// ten billion doubles
>>stxxl::vector<double> HugeVector(10ULL * 1000000000ULL);
>>std::fill(HugeVector.begin(), HugeVector.end(), 0.0);
> [Brian Braatz]
> Thank you for submitting this. I am interested in this library
> and encourage you onward in bringing this to boost.
> In a nutshell, if you could please describe
> * How it works
> * what the implications\dark corners are
what exactly do want to know?
Try to look at the draft of the Stxxl tutorial:
> Are you using memory map files?
my library can use memory maped files, but it can also
rely on a basic I/O calls: open/read/write ...
> What other libraries or os specific
> functions does your library need.
the current implementation the library needs pthreads and
support of POSIX file system calls (open/read/write/close).
As I already mentioned in other postings, I would use
boost threads instead of pthreads to make the library
more portable and boost compliant.
What concerns I/O, the library needs:
1) large file (>2GB) support (e.g. FAT32 does not support large files)
2) avoidance of superfluios buffering (like in std::fstream)
I it also desirable to have (for performance reasons):
3*) unbuffered file access (ASAIK Windows does have it)
Boost list run by bdawes at acm.org, gregod at cs.rpi.edu, cpdaniel at pacbell.net, john at johnmaddock.co.uk